rawhide guests on F-8 host

[Orion Poplawski]

I just build a F-8 x86_64 guest on a F-8 x86_64 host, then did a yum 
upgrade to rawhide.  Now the domain will not start.  Is this expected?


[2008-02-21 14:47:07 2774] DEBUG (XendBootloader:100) Launching 
bootloader as ['/usr/bin/pygrub', 
'--output=/var/run/xend/boot/xenbl.23509', 
'/export/data1/xen_disk_fdev_x86_64'].
[2008-02-21 14:47:12 2103] DEBUG (XendDomainInfo:1640) 
_initDomain:shadow_memory=0x0, memory_static_max=0x20000000, 
memory_static_min=0x0.
[2008-02-21 14:47:12 2103] DEBUG (balloon:116) Balloon: 528696 KiB free; 
need 524288; done.
[2008-02-21 14:47:12 2103] INFO (image:129) buildDomain os=linux dom=2 
vcpus=2
[2008-02-21 14:47:12 2103] DEBUG (image:198) domid          = 2
[2008-02-21 14:47:12 2103] DEBUG (image:199) memsize        = 512
[2008-02-21 14:47:12 2103] DEBUG (image:200) image          = 
/var/run/xend/boot/boot_kernel.vlVqWs
[2008-02-21 14:47:12 2103] DEBUG (image:201) store_evtchn   = 1
[2008-02-21 14:47:12 2103] DEBUG (image:202) console_evtchn = 2
[2008-02-21 14:47:12 2103] DEBUG (image:203) cmdline        = ro 
root=LABEL=/ console=xvc0
[2008-02-21 14:47:12 2103] DEBUG (image:204) ramdisk        = 
/var/run/xend/boot/boot_ramdisk.WvuY8R
[2008-02-21 14:47:12 2103] DEBUG (image:205) vcpus          = 2
[2008-02-21 14:47:12 2103] DEBUG (image:206) features       =
[2008-02-21 14:47:12 2103] INFO (XendDomainInfo:1458) createDevice: 
console : {'devid': 0, 'protocol': 'vt100', 'location':
'2', 'uuid': 
'8ee7025a-6ed3-ba11-1f3d-589420bd567b'}
[2008-02-21 14:47:12 2103] DEBUG (DevController:117) DevController: 
writing {'protocol': 'x86_64-abi', 'state': '1',
'backend-id': '0', 
'backend': '/local/domain/0/backend/console/2/0'} to 
/local/domain/2/device/console/0.
[2008-02-21 14:47:12 2103] DEBUG (DevController:119) DevController: 
writing {'domain': 'xenfdev', 'protocol': 'vt100',
'uuid': 
'8ee7025a-6ed3-ba11-1f3d-589420bd567b', 'frontend': 
'/local/domain/2/device/console/0', 'state': '1',
'location': '2', 
'online': '1', 'frontend-id': '2'} to
/local/domain/0/backend/console/2/0.
[2008-02-21 14:47:12 2103] INFO (XendDomainInfo:1458) createDevice: vbd 
: {'uuid': 'a4f5058f-f598-0c07-9bff-9a930b73c629', 'bootable': 1,

'devid': 51712, 'driver': 'paravirtualised', 'dev':
'xvda:disk', 
'uname': 'file:/export/data1/xen_disk_fdev_x86_64', 'mode':
'w', 
'backend': '0'}
[2008-02-21 14:47:12 2103] DEBUG (DevController:117) DevController: 
writing {'virtual-device': '51712', 'device-type': 'disk',
'protocol': 
'x86_64-abi', 'backend-id': '0', 'state': '1',
'backend': 
'/local/domain/0/backend/vbd/2/51712'} to /local/domain/2/device/vbd/51712.
[2008-02-21 14:47:12 2103] DEBUG (DevController:119) DevController: 
writing {'domain': 'xenfdev', 'frontend': 
'/local/domain/2/device/vbd/51712', 'uuid': 
'a4f5058f-f598-0c07-9bff-9a930b73c629', 'dev': 'xvda',
'state': '1', 
'params': '/export/data1/xen_disk_fdev_x86_64', 'mode':
'w', 'online': 
'1', 'frontend-id': '2', 'type': 'file'} to 
/local/domain/0/backend/vbd/2/51712.
[2008-02-21 14:47:12 2103] INFO (XendDomainInfo:1458) createDevice: vif 
: {'bridge': 'eth0', 'uuid':
'8c47fd72-2fe1-da66-236c-59f501b5c465', 
'script': 'vif-bridge', 'devid': 0, 'mac':
'40:00:00:00:00:01', 
'backend': '0'}
[2008-02-21 14:47:12 2103] DEBUG (DevController:117) DevController: 
writing {'mac': '40:00:00:00:00:01', 'handle': '0',
'protocol': 
'x86_64-abi', 'backend-id': '0', 'state': '1',
'backend': 
'/local/domain/0/backend/vif/2/0'} to /local/domain/2/device/vif/0.
[2008-02-21 14:47:12 2103] DEBUG (DevController:119) DevController: 
writing {'bridge': 'eth0', 'domain': 'xenfdev',
'handle': '0', 'uuid': 
'8c47fd72-2fe1-da66-236c-59f501b5c465', 'script': 
'/etc/xen/scripts/vif-bridge', 'state': '1', 'frontend': 
'/local/domain/2/device/vif/0', 'mac': '40:00:00:00:00:01',
'online': 
'1', 'frontend-id': '2'} to /local/domain/0/backend/vif/2/0.
[2008-02-21 14:47:13 2103] DEBUG (XendDomainInfo:2116) Storing VM 
details: {'on_xend_stop': 'ignore', 'shadow_memory': '0',
'uuid': 
'75d3d53b-8a9c-d5bd-cdc4-207b894c6ac6', 'on_reboot': 'restart', 
'start_time': '1203630433.0', 'on_poweroff': 'destroy',
'on_xend_start': 
'ignore', 'on_crash': 'restart', 'vcpus': '2',
'vcpu_avail': '3', 
'image': "(linux (kernel ) (notes (FEATURES 
'writable_page_tables|writable_descriptor_tables|auto_translated_physmap|pae_pgdir_above_4gb|supervisor_mode_kernel')

(VIRT_BASE 18446744071562067968) (GUEST_VERSION 2.6) (PADDR_OFFSET 
18446744071562067968) (GUEST_OS linux) (HYPERCALL_PAGE 
18446744071564189696) (LOADER generic) (SUSPEND_CANCEL 1) (ENTRY 
18446744071564165120) (XEN_VERSION xen-3.0)))", 'name': 'xenfdev'}
[2008-02-21 14:47:13 2103] DEBUG (XendDomainInfo:956) Storing domain 
details: {'console/ring-ref': '196480', 'image/entry': 
'18446744071564165120', 'console/port': '2',
'store/ring-ref': '196481', 
'image/loader': 'generic', 'vm': 
'/vm/75d3d53b-8a9c-d5bd-cdc4-207b894c6ac6', 
'control/platform-feature-multiprocessor-suspend': '1', 
'image/guest-os': 'linux', 'cpu/1/availability': 'online',

'image/features/writable-descriptor-tables': '1',
'image/virt-base': 
'18446744071562067968', 'memory/target': '524288', 
'image/guest-version': '2.6',
'image/features/supervisor-mode-kernel': 
'1', 'console/limit': '1048576', 'image/paddr-offset': 
'18446744071562067968', 'image/hypercall-page':
'18446744071564189696', 
'image/suspend-cancel': '1', 'cpu/0/availability':
'online', 
'image/features/pae-pgdir-above-4gb': '1', 
'image/features/writable-page-tables': '1', 
'image/features/auto-translated-physmap': '1', 'name':
'xenfdev', 
'domid': '2', 'image/xen-version': 'xen-3.0',
'store/port': '1'}
[2008-02-21 14:47:13 2103] DEBUG (XendDomainInfo:1040) 
XendDomainInfo.handleShutdownWatch
[2008-02-21 14:47:13 2103] WARNING (XendDomainInfo:1203) Domain has 
crashed: name=xenfdev id=2.
[2008-02-21 14:47:13 2103] ERROR (XendDomainInfo:1333) VM xenfdev 
restarting too fast (6.067748 seconds since the last restart).  Refusing 
to restart to avoid loops.

-- 
Orion Poplawski
Technical Manager                     303-415-9701 x222
NWRA/CoRA Division                    FAX: [phone]
3380 Mitchell Lane                  orion(a)cora.nwra.com
Boulder, CO 80301              http://www.cora.nwra.com

[Mark McLoughlin]

Hi,

On Thu, 2008-02-21 at 15:08 -0700, Orion Poplawski wrote:
>  I just build a F-8 x86_64 guest on a F-8 x86_64 host, then did a yum
>
>  upgrade to rawhide.  Now the domain will not start.  Is this expected? 
	Thanks for reporting, I can reproduce here. Try this older kernel for
now:

  http://koji.fedoraproject.org/koji/buildinfo?buildID=32046

	More details filed here:	

  https://bugzilla.redhat.com/show_bug.cgi?id=434556

	FWIW, I got this stack trace by setting the "on_crash" action for the
guest to "preserve" and then when it had crashed, I did:

  $> /usr/lib64/xen/bin/xenctx -s /boot/System.map-2.6.21.7-2895.fc9xen 1

Cheers,
Mark.

[Orion Poplawski]

Mark McLoughlin wrote:
>  Hi,
>  
>  On Thu, 2008-02-21 at 15:08 -0700, Orion Poplawski wrote:
> > I just build a F-8 x86_64 guest on a F-8 x86_64 host, then did a yum 
> > upgrade to rawhide.  Now the domain will not start.  Is this expected?
>  
>  	Thanks for reporting, I can reproduce here. Try this older kernel for
>  now:
>  
>    http://koji.fedoraproject.org/koji/buildinfo?buildID=32046 
Is there any way to install this into my currently dead domain?

-- 
Orion Poplawski
Technical Manager                     303-415-9701 x222
NWRA/CoRA Division                    FAX: [phone]
3380 Mitchell Lane                  orion(a)cora.nwra.com
Boulder, CO 80301              http://www.cora.nwra.com

[Mark McLoughlin]

Hi,
	This build should fix the problem for you:

  http://koji.fedoraproject.org/koji/buildinfo?buildID=39764

On Fri, 2008-02-22 at 12:07 -0700, Orion Poplawski wrote:
>  Mark McLoughlin wrote:
>  > Hi,
>  > 
>  > On Thu, 2008-02-21 at 15:08 -0700, Orion Poplawski wrote:
>  >> I just build a F-8 x86_64 guest on a F-8 x86_64 host, then did a yum 
>  >> upgrade to rawhide.  Now the domain will not start.  Is this expected?
>  > 
>  > 	Thanks for reporting, I can reproduce here. Try this older kernel for
>  > now:
>  > 
>  >   http://koji.fedoraproject.org/koji/buildinfo?buildID=32046
>  
>  Is there any way to install this into my currently dead domain? 
	Well, hopefully you still have the F-8 kernel installed and you can
just run that e.g. by doing "virsh console foo" before the grub timeout
expires and choosing the older kernel.

	Alternatively, you could mount the /boot partition and change the
default in grub.conf e.g.

  $> losetup /dev/loop0 /xen/foo.img
  $> kpartx -av /dev/loop0
  $> mount /dev/mapper/loop0p1 /tmp/foo
  $> emacs /boot/grub/grub.conf
  $> umount /tmp/foo
  $> kpartx -d /dev/loop0
  $> losetup -d /dev/loop0

	Once you've booted with a working kernel, then you can upgrade to the
newer one.

	If you don't have any working kernel in the domain, you could try
something like this:

  $> losetup /dev/loop0 /xen/foo.img
  $> kpartx -av /dev/loop0
  $> vgscan
  $> vgchange -ay VolGroup00
  $> mount /dev/VolGroup00/LogVol00 /tmp/foo
  $> mount /dev/mapper/loop0p1 /tmp/foo/boot
  $> chroot /tmp/foo yum --nogpgcheck localinstall kernel-xen-...rpm
  $> chroot mkinitrd --with xenblk -f /boot/initrd-2.6.21.7....rpm 2.6.21.7...
  $> umount /tmp/foo/boot
  $> umount /tmp/foo
  $> vgchange -an VolGroup00
  $> kpartx -d /dev/loop0
  $> losetup -d /dev/loop0

	Or maybe easier, boot the dead domain with a rescue CD.

Cheers,
Mark.