RHEL4 DomU Update Problem

[Jim Klein]

Am attempting to update a RHEL4 domU from a custom created Xen kernel (created before
RHEL5) to a supported kernel (2.6.9-55.0.9.ELxenU) and RHEL5 host without success. For
some reason, the kernel can no longer find xvda. Boots fine on the old kernel and host, so
I'm at a bit of a loss. Host has other domUs booted off the same SAN, with no security
restrictions, so I don't see how it could be hardware. Copy of configs and partial
debug log below. Ideas? 

Conf: 
name = "do5" 
memory = "1024" 
disk = [ 'phy:/dev/mpath/msa-mpathv0p1,xvda,w', ] 
vif = [ 'mac=00:16:3e:55:a5:65, bridge=xenbr2' ] 
vfb = ["type=vnc,vncunused=1"] 
bootloader="/usr/bin/pygrub" 
vcpus=1 

Grub: 
root (hd0,0) 
kernel /vmlinuz-2.6.9-55.0.9.ELxenU ro root=/dev/xvda2 rhgb quiet 
initrd /initrd-2.6.9-55.0.9.ELxenU.img 

Debug: 
[2007-10-30 12:41:21 xend.XendDomainInfo 4121] DEBUG (XendDomainInfo:190)
XendDomainInfo.create(['vm', ['name', 'do5'], ['memory',
'1024'], ['vcpus', 1], ['bootloader', '/usr/bin/pygrub'],
['image', ['linux', ['ramdisk',
'/var/lib/xen/initrd.MpxKQP'], ['kernel',
'/var/lib/xen/vmlinuz.6u2e9w'], ['args', 'ro root=/dev/xvda2 rhgb
quiet']]], ['device', ['vbd', ['uname',
'phy:/dev/mpath/msa-mpathv0p1'], ['dev', 'xvda'], ['mode',
'w']]], ['device', ['vif', ['bridge', 'xenbr2'],
['mac', '00:16:3e:55:a5:65']]], ['device', ['vkbd']],
['device', ['vfb', ['vncunused', '1'], ['type',
'vnc'], ['display', 'localhost:10.0'], ['xauthority',
'/root/.Xauthority']]]]) 
[2007-10-30 12:41:21 xend.XendDomainInfo 4121] DEBUG (XendDomainInfo:296) parseConfig:
config is ['vm', ['name', 'do5'], ['memory',
'1024'], ['vcpus', 1], ['bootloader', '/usr/bin/pygrub'],
['image', ['linux', ['ramdisk',
'/var/lib/xen/initrd.MpxKQP'], ['kernel',
'/var/lib/xen/vmlinuz.6u2e9w'], ['args', 'ro root=/dev/xvda2 rhgb
quiet']]], ['device', ['vbd', ['uname',
'phy:/dev/mpath/msa-mpathv0p1'], ['dev', 'xvda'], ['mode',
'w']]], ['device', ['vif', ['bridge', 'xenbr2'],
['mac', '00:16:3e:55:a5:65']]], ['device', ['vkbd']],
['device', ['vfb', ['vncunused', '1'], ['type',
'vnc'], ['display', 'localhost:10.0'], ['xauthority',
'/root/.Xauthority']]]] 
[2007-10-30 12:41:21 xend.XendDomainInfo 4121] DEBUG (XendDomainInfo:397) parseConfig:
result is {'shadow_memory': None, 'start_time': None, 'uuid':
None, 'on_crash': None, 'on_reboot': None, 'localtime': None,
'image': ['linux', ['ramdisk',
'/var/lib/xen/initrd.MpxKQP'], ['kernel',
'/var/lib/xen/vmlinuz.6u2e9w'], ['args', 'ro root=/dev/xvda2 rhgb
quiet']], 'on_poweroff': None, 'bootloader_args': None,
'cpus': None, 'name': 'do5', 'backend': [],
'vcpus': 1, 'cpu_weight': None, 'features': None,
'vcpu_avail': None, 'memory': 1024, 'device': [('vbd',
['vbd', ['uname', 'phy:/dev/mpath/msa-mpathv0p1'], ['dev',
'xvda'], ['mode', 'w']]), ('vif', ['vif',
['bridge', 'xenbr2'], ['mac', '00:16:3e:55:a5:65']]),
('vkbd', ['vkbd']), ('vfb', ['vfb', ['vncunused',
'1'], ['type', 'vnc'], ['display',
'localhost:10.0'], ['xauthority', '/root/.Xauthority']])],
'bootloader': '/usr/bin/pygrub', 'cpu': None, 'maxmem':
None} 
[2007-10-30 12:41:21 xend.XendDomainInfo 4121] DEBUG (XendDomainInfo:1264)
XendDomainInfo.construct: None 
[2007-10-30 12:41:21 xend.XendDomainInfo 4121] DEBUG (XendDomainInfo:1296)
XendDomainInfo.initDomain: 9 1.0 
[2007-10-30 12:41:21 xend 4121] DEBUG (balloon:127) Balloon: 1049112 KiB free; need
1048576; done. 
[2007-10-30 12:41:21 xend 4121] INFO (image:136) buildDomain os=linux dom=9 vcpus=1 
[2007-10-30 12:41:21 xend 4121] DEBUG (image:199) dom = 9 
[2007-10-30 12:41:21 xend 4121] DEBUG (image:200) image = /var/lib/xen/vmlinuz.6u2e9w 
[2007-10-30 12:41:21 xend 4121] DEBUG (image:201) store_evtchn = 1 
[2007-10-30 12:41:21 xend 4121] DEBUG (image:202) console_evtchn = 2 
[2007-10-30 12:41:21 xend 4121] DEBUG (image:203) cmdline = ro root=/dev/xvda2 rhgb quiet

[2007-10-30 12:41:21 xend 4121] DEBUG (image:204) ramdisk = /var/lib/xen/initrd.MpxKQP 
[2007-10-30 12:41:21 xend 4121] DEBUG (image:205) vcpus = 1 
[2007-10-30 12:41:21 xend 4121] DEBUG (image:206) features = 
[2007-10-30 12:41:21 xend 4121] DEBUG (blkif:24) exception looking up device number for
xvda: [Errno 2] No such file or directory: '/dev/xvda'

[Chris Lalancette]

Jim Klein wrote:
>  Am attempting to update a RHEL4 domU from a custom created Xen
> kernel
>  (created before RHEL5) to a supported kernel (2.6.9-55.0.9.ELxenU) and
>  RHEL5 host without success. For some reason, the kernel can no longer
>  find xvda. Boots fine on the old kernel and host, so I'm at a bit of a
>  loss. Host has other domUs booted off the same SAN, with no security
>  restrictions, so I don't see how it could be hardware. Copy of configs
>  and partial debug log below. Ideas? 
Hm, everything below, at a first glance, seems to be right.  What does the
console output look like from the RHEL-4 kernel (make sure to take rhgb quiet
off of the command-line)?

Chris Lalancette

[Anand Gupta]

I have had similar problems, at last i reverted to use sda instead of xvda.

Also i am getting errors about /lib/tls even when its been renamed in dom0
and domU.

Any ideas on this ?

-- 
regards,

Anand Gupta

[Jim Klein]

Interesting: 

XENBUS: Device with no driver: device/vbd/2048 
XENBUS: Device with no driver: device/vif/0 
Freeing unused kernel memory: 124k freed 
device-mapper: 4.5.5-ioctl ( 2006-12-01 ) initialised: dm-devel(a)redhat.com 
Kernel panic - not syncing: Attempted to kill init! 

Not sure where to go from here. 


----- Original Message ----- 
From: "Chris Lalancette" <clalance(a)redhat.com&gt; 
To: "Jim Klein" <jklein(a)saugus.k12.ca.us&gt; 
Cc: "fedora-xen" <fedora-xen(a)redhat.com&gt; 
Sent: Tuesday, October 30, 2007 1:23:08 PM (GMT-0800) America/Los_Angeles 
Subject: Re: [Fedora-xen] RHEL4 DomU Update Problem 

Jim Klein wrote: 
>  Am attempting to update a RHEL4 domU from a custom created Xen kernel
>
>  (created before RHEL5) to a supported kernel (2.6.9-55.0.9.ELxenU) and 
>  RHEL5 host without success. For some reason, the kernel can no longer 
>  find xvda. Boots fine on the old kernel and host, so I'm at a bit of a 
>  loss. Host has other domUs booted off the same SAN, with no security 
>  restrictions, so I don't see how it could be hardware. Copy of configs 
>  and partial debug log below. Ideas?  
Hm, everything below, at a first glance, seems to be right. What does the 
console output look like from the RHEL-4 kernel (make sure to take rhgb quiet 
off of the command-line)? 

Chris Lalancette

[Chris Lalancette]

Jim Klein wrote:
>  Interesting:
>  
>  XENBUS: Device with no driver: device/vbd/2048
>  XENBUS: Device with no driver: device/vif/0
>  Freeing unused kernel memory: 124k freed
>  device-mapper: 4.5.5-ioctl (2006-12-01) initialised: dm-devel(a)redhat.com
>  Kernel panic - not syncing: Attempted to kill init!
>  
>  Not sure where to go from here.
>   
Believe it or not, those XENBUS errors are actually expected; the xenbus driver
is loaded before the virtual disk/network drivers, and it sees that it has these
nodes without drivers.

However, that does lead me to an idea.  If I remember correctly, the Xensource
kernels have all of the drivers built-in, while we prefer to do things more
modular.  I'm guessing that you didn't have the right driver in
/etc/modprobe.conf when the kernel was installed, so the initrd doesn't have the
right drivers in it.  So what you'll want to do is:

1)  Boot the RHEL-4 domU into the (working) Xen kernel.
2)  Edit /etc/modprobe.conf, and add:

alias scsi_hostadapter xenblk

3)  Remove the RedHat RHEL-4 kernel (rpm -e), and install it again, which should
re-generate the initrd with the right stuff in it.

Chris Lalancette

[Jim Klein]

Now it's really interesting. If I just try to boot the system with the disk labeled
xvda, I get a huge series of kernel panics (see below.) If I change it to sda (in both the
config file and grub) it looks like it might work, but then I get: 

Freeing unused kernel memory: 124k freed 
SCSI subsystem initialized 
Registering block device major 8 
register_blkdev: cannot get major 8 for sd 
xen_blk: can't get major 8 with name sd 
vbd vbd-2048: 19 xlvbd_add at /local/domain/0/backend/vbd/27/2048 
Registering block device major 8 
register_blkdev: cannot get major 8 for sd 
xen_blk: can't get major 8 with name sd 
vbd vbd-2048: 19 xlvbd_add at /local/domain/0/backend/vbd/27/2048 
XENBUS: Timeout connecting to device: device/vbd/2048 (state 6) 
device-mapper: 4.5.5-ioctl (2006-12-01) initialised: dm-devel(a)redhat.com 
Kernel panic - not syncing: Attempted to kill init! 

FYI, this wasn't a Xensource kernel before, it was actually a Fedora 5 kernel
(2.6.15-1.2054_FC5xenU) I created a yum repository for RHEL4 with the Fedora kernel, and
used it to set up the box. 

Kernel panics (lots of them, all the same): 
kernel BUG at arch/i386/mm/pgtable-xen.c:306! 
invalid operand: 0000 [#1] 
SMP 
Modules linked in: ext3 jbd dm_mod xenblk sd_mod scsi_mod 
CPU: 0 
EIP: 0061:[<c011163a>] Not tainted VLI 
EFLAGS: 00010282 (2.6.9-55.0.9.ELxenU) 
EIP is at pgd_ctor+0x1d/0x26 
eax: fffffff4 ebx: 00000000 ecx: f5392000 edx: 00000000 
esi: c19fdd80 edi: eca6aaa0 ebp: 00000001 esp: ecb3cd6c 
ds: 007b es: 007b ss: 0068 
Process 10-udev.hotplug (pid: 398, threadinfo=ecb3c000 task=ecb2a070) 
Stack: c0141b69 ecb4b000 c19fdd80 00000001 ecb4b000 eca6aaa0 c19fdd80 c19fde40 
c0141ceb c19fdd80 eca6aaa0 00000001 c19fdd80 eca6aaa0 ecb4b000 00000010 
00000001 000000d0 c1a1b080 0000000c c19fde08 c19fdd80 c0141eda c19fdd80 
Call Trace: 
[<c0141b69>] cache_init_objs+0x35/0x56 
[<c0141ceb>] cache_grow+0xfb/0x187 
[<c0141eda>] cache_alloc_refill+0x163/0x19c 
[<c01420f5>] kmem_cache_alloc+0x67/0x97 
[<c0111671>] pgd_alloc+0x17/0x336 
[<c01199d4>] mm_init+0xd7/0x116 
[<c01199e4>] mm_init+0xe7/0x116 
[<c0119c8a>] copy_mm+0xbb/0x396 
[<c0268f10>] __cond_resched+0x14/0x3c 
[<c011aa5a>] copy_process+0x6b5/0xb0b 
[<c011af9d>] do_fork+0x8a/0x16b 
[<c0107507>] error_code+0x2b/0x30 
[<c0105d2c>] sys_clone+0x24/0x28 
[<c010737f>] syscall_call+0x7/0xb 
Code: 74 02 66 a5 a8 01 74 01 a4 5e 5b 5e 5f c3 80 3d 04 07 2f c0 00 75 1c 6a 20 6a 00 ff
74 24 0c e8 ce 37 00 00 83 c4 0c 85 c0 74 08 <0f> 0b 32 01 b6 31 27 c0 c3 80 3d 04
07 2f c0 00 75 0d c7 44 24 
<0>Fatal exception: panic in 5 seconds 


----- Original Message ----- 
From: "Chris Lalancette" <clalance(a)redhat.com&gt; 
To: "Jim Klein" <jklein(a)saugus.k12.ca.us&gt; 
Cc: "fedora-xen" <fedora-xen(a)redhat.com&gt; 
Sent: Tuesday, October 30, 2007 3:00:05 PM (GMT-0800) America/Los_Angeles 
Subject: Re: [Fedora-xen] RHEL4 DomU Update Problem 

Jim Klein wrote: 
>  Interesting: 
>  
>  XENBUS: Device with no driver: device/vbd/2048 
>  XENBUS: Device with no driver: device/vif/0 
>  Freeing unused kernel memory: 124k freed 
>  device-mapper: 4.5.5-ioctl (2006-12-01) initialised: dm-devel(a)redhat.com 
>  Kernel panic - not syncing: Attempted to kill init! 
>  
>  Not sure where to go from here. 
>   
Believe it or not, those XENBUS errors are actually expected; the xenbus driver 
is loaded before the virtual disk/network drivers, and it sees that it has these 
nodes without drivers. 

However, that does lead me to an idea. If I remember correctly, the Xensource 
kernels have all of the drivers built-in, while we prefer to do things more 
modular. I'm guessing that you didn't have the right driver in 
/etc/modprobe.conf when the kernel was installed, so the initrd doesn't have the 
right drivers in it. So what you'll want to do is: 

1) Boot the RHEL-4 domU into the (working) Xen kernel. 
2) Edit /etc/modprobe.conf, and add: 

alias scsi_hostadapter xenblk 

3) Remove the RedHat RHEL-4 kernel (rpm -e), and install it again, which should 
re-generate the initrd with the right stuff in it. 

Chris Lalancette

[Chris Lalancette]

Jim Klein wrote:
>  
>  Kernel panics (lots of them, all the same):
>  kernel BUG at arch/i386/mm/pgtable-xen.c:306!
>  invalid operand: 0000 [#1]
>  SMP
>  Modules linked in: ext3 jbd dm_mod xenblk sd_mod scsi_mod
>  CPU:    0
>  EIP:    0061:[<c011163a>]    Not tainted VLI
>  EFLAGS: 00010282   (2.6.9-55.0.9.ELxenU)
>  EIP is at pgd_ctor+0x1d/0x26
>  eax: fffffff4   ebx: 00000000   ecx: f5392000   edx: 00000000
>  esi: c19fdd80   edi: eca6aaa0   ebp: 00000001   esp: ecb3cd6c
>  ds: 007b   es: 007b   ss: 0068
>  Process 10-udev.hotplug (pid: 398, threadinfo=ecb3c000 task=ecb2a070)
>  Stack: c0141b69 ecb4b000 c19fdd80 00000001 ecb4b000 eca6aaa0 c19fdd80
>  c19fde40
>         c0141ceb c19fdd80 eca6aaa0 00000001 c19fdd80 eca6aaa0 ecb4b000
>  00000010
>         00000001 000000d0 c1a1b080 0000000c c19fde08 c19fdd80 c0141eda
>  c19fdd80
>  Call Trace:
>   [<c0141b69>] cache_init_objs+0x35/0x56
>   [<c0141ceb>] cache_grow+0xfb/0x187
>   [<c0141eda>] cache_alloc_refill+0x163/0x19c
>   [<c01420f5>] kmem_cache_alloc+0x67/0x97
>   [<c0111671>] pgd_alloc+0x17/0x336
>   [<c01199d4>] mm_init+0xd7/0x116
>   [<c01199e4>] mm_init+0xe7/0x116
>   [<c0119c8a>] copy_mm+0xbb/0x396
>   [<c0268f10>] __cond_resched+0x14/0x3c
>   [<c011aa5a>] copy_process+0x6b5/0xb0b
>   [<c011af9d>] do_fork+0x8a/0x16b
>   [<c0107507>] error_code+0x2b/0x30
>   [<c0105d2c>] sys_clone+0x24/0x28
>   [<c010737f>] syscall_call+0x7/0xb
>  Code: 74 02 66 a5 a8 01 74 01 a4 5e 5b 5e 5f c3 80 3d 04 07 2f c0 00 75
>  1c 6a 20 6a 00 ff 74 24 0c e8 ce 37 00 00 83 c4 0c 85 c0 74 08 <0f> 0b
>  32 01 b6 31 27 c0 c3 80 3d 04 07 2f c0 00 75 0d c7 44 24
>   <0>Fatal exception: panic in 5 seconds 
OK.  This crash is a bug in the 4.5 kernel that I've now fixed.  It should be
fixed when 4.6 comes out.  In the meantime, you probably should be able to work
around this bug by reducing the amount of memory you are assigning to that domain.

Chris Lalancette

[Jim Klein]

Thanks for the info - I'll look for the update. I still get the panic, regardless of
memory settings. 


----- Original Message ----- 
From: "Chris Lalancette" <clalance(a)redhat.com&gt; 
To: "Jim Klein" <jklein(a)saugus.k12.ca.us&gt; 
Cc: "fedora-xen" <fedora-xen(a)redhat.com&gt; 
Sent: Wednesday, October 31, 2007 9:25:52 AM (GMT-0800) America/Los_Angeles 
Subject: Re: [Fedora-xen] RHEL4 DomU Update Problem 

Jim Klein wrote: 
>  
>  Kernel panics (lots of them, all the same): 
>  kernel BUG at arch/i386/mm/pgtable-xen.c:306! 
>  invalid operand: 0000 [#1] 
>  SMP 
>  Modules linked in: ext3 jbd dm_mod xenblk sd_mod scsi_mod 
>  CPU: 0 
>  EIP: 0061:[<c011163a>] Not tainted VLI 
>  EFLAGS: 00010282 (2.6.9-55.0.9.ELxenU) 
>  EIP is at pgd_ctor+0x1d/0x26 
>  eax: fffffff4 ebx: 00000000 ecx: f5392000 edx: 00000000 
>  esi: c19fdd80 edi: eca6aaa0 ebp: 00000001 esp: ecb3cd6c 
>  ds: 007b es: 007b ss: 0068 
>  Process 10-udev.hotplug (pid: 398, threadinfo=ecb3c000 task=ecb2a070) 
>  Stack: c0141b69 ecb4b000 c19fdd80 00000001 ecb4b000 eca6aaa0 c19fdd80 
>  c19fde40 
>  c0141ceb c19fdd80 eca6aaa0 00000001 c19fdd80 eca6aaa0 ecb4b000 
>  00000010 
>  00000001 000000d0 c1a1b080 0000000c c19fde08 c19fdd80 c0141eda 
>  c19fdd80 
>  Call Trace: 
>  [<c0141b69>] cache_init_objs+0x35/0x56 
>  [<c0141ceb>] cache_grow+0xfb/0x187 
>  [<c0141eda>] cache_alloc_refill+0x163/0x19c 
>  [<c01420f5>] kmem_cache_alloc+0x67/0x97 
>  [<c0111671>] pgd_alloc+0x17/0x336 
>  [<c01199d4>] mm_init+0xd7/0x116 
>  [<c01199e4>] mm_init+0xe7/0x116 
>  [<c0119c8a>] copy_mm+0xbb/0x396 
>  [<c0268f10>] __cond_resched+0x14/0x3c 
>  [<c011aa5a>] copy_process+0x6b5/0xb0b 
>  [<c011af9d>] do_fork+0x8a/0x16b 
>  [<c0107507>] error_code+0x2b/0x30 
>  [<c0105d2c>] sys_clone+0x24/0x28 
>  [<c010737f>] syscall_call+0x7/0xb 
>  Code: 74 02 66 a5 a8 01 74 01 a4 5e 5b 5e 5f c3 80 3d 04 07 2f c0 00 75 
>  1c 6a 20 6a 00 ff 74 24 0c e8 ce 37 00 00 83 c4 0c 85 c0 74 08 <0f> 0b 
>  32 01 b6 31 27 c0 c3 80 3d 04 07 2f c0 00 75 0d c7 44 24 
>  <0>Fatal exception: panic in 5 seconds  
OK. This crash is a bug in the 4.5 kernel that I've now fixed. It should be 
fixed when 4.6 comes out. In the meantime, you probably should be able to work 
around this bug by reducing the amount of memory you are assigning to that domain. 

Chris Lalancette

[Chris Lalancette]

Jim Klein wrote:
>  Thanks for the info - I'll look for the update. I still get the
> panic,
>  regardless of memory settings.
>   
Hm, OK.  Can you try one of the kernels from here:

http://people.redhat.com/~jbaron/rhel4/

These are the pre-release 4.6 kernels (not officially supported, of
course), and should have the fix for your problem.

Chris Lalancette