Hi,
I've been trying to test out Xen migration, but keep running into a
kernel Oops/crash. It might be my setup or I might be running into some
other problem, but I've searched mailing lists/google and haven't found
anything that has helped.
I'm using 3 FC5 boxes all running the latest kernels, updates, etc:
One box to export a mounted image file generated with
xenguest-install.py via AoE using vblade (have also tried using NFS to
export a directory containing the image file).
The other 2 boxes are running Xen. The guest starts and runs on either
Xen box without a problem, but when trying to do a migration or a live
migration between the 2 Xen boxes, in either direction, I run into the
kernel crash.
The migration actually moves the guest from one box to the other, but it
stops responding via the network, and when logging in using 'xm console'
I see a kernel call trace. The guest kinda still works other than
networking not working at all, so I tried doing a service network
restart, but this causes the guest to totally lock up. If the guest is
rebooted it works fine.
Bellow is the kernel call trace and what is in the destination Xen
server's xend.log.
Thanks
Cameron
-------------------------
BUG: unable to handle kernel NULL pointer dereference at virtual address
00000000
printing eip:
d10c2c27
*pde = ma 00000000 pa fffff000
Oops: 0002 [#1]
SMP
Modules linked in: ipv6 autofs4 sunrpc xennet ip_conntrack_netbios_ns
ipt_REJECT xt_state ip_conntrack nfnetlink xt_tcpudp iptable_filter
ip_tables x_tables dm_snapshot dm_zero dm_mirror dm_mod
CPU: 0
EIP: 0061:[<d10c2c27>] Not tainted VLI
EFLAGS: 00010046 (2.6.17-1.2145_FC5xenU #1)
EIP is at backend_changed+0x15c/0x24d [xennet]
eax: 00000000 ebx: cf608400 ecx: cf608498 edx: 00000000
esi: 00000001 edi: 00000000 ebp: 00000101 esp: c09cbf70
ds: 007b es: 007b ss: 0069
Process xenwatch (pid: 8, threadinfo=c09ca000 task=c06c6b30)
Stack: <0>c09f2200 cf608000 00000000 cf608498 cf608488 c0898dc0 c09ca000
c0898dc0
c051d45e c051cdee c09f2200 00000002 00000002 00000000 c051d5aa
c0898dc0
00000000 c06c6b30 c0429cd7 c09cbfbc c09cbfbc c0a43e7c c0429c16
00000000
Call Trace:
<c051d45e> xenwatch_thread+0x0/0x163 <c051cdee>
xenwatch_handle_callback+0x12/0x45
<c051d5aa> xenwatch_thread+0x14c/0x163 <c0429cd7>
autoremove_wake_function+0x0/0x3a
<c0429c16> kthread+0xa0/0xcd <c0429b76> kthread+0x0/0xcd
<c0402989> kernel_thread_helper+0x5/0xb
Code: 08 8b 47 60 01 43 18 ff 43 10 45 81 fd 01 01 00 00 0f 85 35 ff ff
ff 8b 44 24 08 89 43 68 8b 53 74 be 01 00 00 00 31 ff 8b 43 68 <89> 02
8b 84 b3 f8 04 00 00 3d ff ff ff bf 76 58 8b 80 98 00 00
EIP: [<d10c2c27>] backend_changed+0x15c/0x24d [xennet] SS:ESP
0069:c09cbf70
<3>BUG: sleeping function called from invalid context at
include/linux/rwsem.h:43
in_atomic():0, irqs_disabled():1
<c04242e2> blocking_notifier_call_chain+0x11/0x48 <c04198d6>
profile_task_exit+0x10/0x14
<c041b08d> do_exit+0x1b/0x78b <c040571a> die+0x1c0/0x27b
<c04057af> die+0x255/0x27b <c040ee98> do_page_fault+0x0/0x70f
<c040f4a7> do_page_fault+0x60f/0x70f <c040ee98>
do_page_fault+0x0/0x70f
<c0404d07> error_code+0x2b/0x30 <d10c2c27> backend_changed+0x15c/0x24d
[xennet]
<c051d45e> xenwatch_thread+0x0/0x163 <c051cdee>
xenwatch_handle_callback+0x12/0x45
<c051d5aa> xenwatch_thread+0x14c/0x163 <c0429cd7>
autoremove_wake_function+0x0/0x3a
<c0429c16> kthread+0xa0/0xcd <c0429b76> kthread+0x0/0xcd
<c0402989> kernel_thread_helper+0x5/0xb
BUG: xenwatch/8, lock held at task exit time!
[c06083e0] {xenwatch_mutex}
.. held by: xenwatch: 8 [c06c6b30, 110]
... acquired at: xenwatch_thread+0x95/0x163
------------------------
[2006-07-10 12:42:00 xend.XendDomainInfo] DEBUG (XendDomainInfo:260)
XendDomainInfo.restore(['domain', ['domid', '5'], ['uuid',
'08cd716d-1deb-00f4-77b2-66f14244bc16'], ['vcpus', '1'], ['vcpu_avail',
'1'], ['cpu_weight', '1.0'], ['memory', '256'], ['maxmem', '256'],
['bootloader', '/usr/bin/pygrub'], ['features'], ['name',
'xen.guest.migration.test'], ['on_poweroff', 'destroy'], ['on_reboot',
'restart'], ['on_crash', 'restart'], ['image', ['linux', ['ramdisk',
'/var/lib/xen/initrd.ihUbbw'], ['kernel',
'/var/lib/xen/vmlinuz.4kwzZ2'], ['args', 'ro
root=/dev/VolGroup00/LogVol00']]], ['device', ['vif', ['backend', '0'],
['script', 'vif-bridge'], ['mac', '00:16:3e:09:8a:dc']]], ['device',
['vbd', ['backend', '0'], ['dev', 'xvda'], ['uname', 'phy:etherd/e0.2'],
['mode', 'w']]], ['state', '-b----'], ['shutdown_reason', 'poweroff'],
['cpu_time', '0.142407299'], ['online_vcpus', '1'], ['up_time',
'508.123579025'], ['start_time', '1152498812.32'], ['store_mfn',
'613739'], ['console_mfn', '613738']])
[2006-07-10 12:42:00 xend.XendDomainInfo] DEBUG (XendDomainInfo:291)
parseConfig: config is ['domain', ['domid', '5'], ['uuid',
'08cd716d-1deb-00f4-77b2-66f14244bc16'], ['vcpus', '1'], ['vcpu_avail',
'1'], ['cpu_weight', '1.0'], ['memory', '256'], ['maxmem', '256'],
['bootloader', '/usr/bin/pygrub'], ['features'], ['name',
'xen.guest.migration.test'], ['on_poweroff', 'destroy'], ['on_reboot',
'restart'], ['on_crash', 'restart'], ['image', ['linux', ['ramdisk',
'/var/lib/xen/initrd.ihUbbw'], ['kernel',
'/var/lib/xen/vmlinuz.4kwzZ2'], ['args', 'ro
root=/dev/VolGroup00/LogVol00']]], ['device', ['vif', ['backend', '0'],
['script', 'vif-bridge'], ['mac', '00:16:3e:09:8a:dc']]], ['device',
['vbd', ['backend', '0'], ['dev', 'xvda'], ['uname', 'phy:etherd/e0.2'],
['mode', 'w']]], ['state', '-b----'], ['shutdown_reason', 'poweroff'],
['cpu_time', '0.142407299'], ['online_vcpus', '1'], ['up_time',
'508.123579025'], ['start_time', '1152498812.32'], ['store_mfn',
'613739'], ['console_mfn', '613738']]
[2006-07-10 12:42:00 xend.XendDomainInfo] DEBUG (XendDomainInfo:390)
parseConfig: result is {'uuid': '08cd716d-1deb-00f4-77b2-66f14244bc16',
'on_crash': 'restart', 'on_reboot': 'restart', 'image': ['linux',
['ramdisk', '/var/lib/xen/initrd.ihUbbw'], ['kernel',
'/var/lib/xen/vmlinuz.4kwzZ2'], ['args', 'ro
root=/dev/VolGroup00/LogVol00']], 'on_poweroff': 'destroy',
'bootloader_args': None, 'cpus': None, 'name':
'xen.guest.migration.test', 'backend': [], 'vcpus': 1, 'cpu_weight':
1.0, 'features': None, 'vcpu_avail': 1, 'memory': 256, 'device':
[('vif', ['vif', ['backend', '0'], ['script', 'vif-bridge'], ['mac',
'00:16:3e:09:8a:dc']]), ('vbd', ['vbd', ['backend', '0'], ['dev',
'xvda'], ['uname', 'phy:etherd/e0.2'], ['mode', 'w']])], 'bootloader':
'/usr/bin/pygrub', 'cpu': None, 'maxmem': 256}
[2006-07-10 12:42:00 xend.XendDomainInfo] DEBUG (XendDomainInfo:1216)
XendDomainInfo.construct: None
[2006-07-10 12:42:00 xend.XendDomainInfo] DEBUG (XendDomainInfo:701)
Storing VM details: {'name': 'xen.guest.migration.test', 'on_reboot':
'restart', 'image': "(linux (ramdisk /var/lib/xen/initrd.ihUbbw) (kernel
/var/lib/xen/vmlinuz.4kwzZ2) (args 'ro
root=/dev/VolGroup00/LogVol00'))", 'on_poweroff': 'destroy', 'uuid':
'08cd716d-1deb-00f4-77b2-66f14244bc16', 'on_crash': 'restart', 'vcpus':
'1', 'vcpu_avail': '1', 'memory': '256', 'maxmem': '256'}
[2006-07-10 12:42:00 xend] DEBUG (DevController:110) DevController:
writing {'backend-id': '0', 'mac': '00:16:3e:09:8a:dc', 'handle': '0',
'state': '1', 'backend': '/local/domain/0/backend/vif/6/0'} to
/local/domain/6/device/vif/0.
[2006-07-10 12:42:00 xend] DEBUG (DevController:112) DevController:
writing {'mac': '00:16:3e:09:8a:dc', 'state': '1', 'handle': '0',
'script': '/etc/xen/scripts/vif-bridge', 'frontend-id': '6', 'domain':
'xen.guest.migration.test', 'frontend': '/local/domain/6/device/vif/0'}
to /local/domain/0/backend/vif/6/0.
[2006-07-10 12:42:00 xend] DEBUG (blkif:24) exception looking up device
number for xvda: [Errno 2] No such file or directory: '/dev/xvda'
[2006-07-10 12:42:00 xend] DEBUG (DevController:110) DevController:
writing {'virtual-device': '51712', 'backend-id': '0', 'state': '1',
'backend': '/local/domain/0/backend/vbd/6/51712'} to
/local/domain/6/device/vbd/51712.
[2006-07-10 12:42:00 xend] DEBUG (DevController:112) DevController:
writing {'domain': 'xen.guest.migration.test', 'frontend':
'/local/domain/6/device/vbd/51712', 'dev': 'xvda', 'state': '1',
'params': 'etherd/e0.2', 'mode': 'w', 'frontend-id': '6', 'type': 'phy'}
to /local/domain/0/backend/vbd/6/51712.
[2006-07-10 12:42:00 xend.XendDomainInfo] DEBUG (XendDomainInfo:736)
Storing domain details: {'console/port': '2', 'name':
'xen.guest.migration.test', 'console/limit': '1048576', 'vm':
'/vm/08cd716d-1deb-00f4-77b2-66f14244bc16', 'domid': '6',
'cpu/0/availability': 'online', 'memory/target': '262144', 'store/port':
'1'}
[2006-07-10 12:42:00 xend] DEBUG (balloon:128) Balloon: free 746; need
265; done.
[2006-07-10 12:42:00 xend] DEBUG (XendCheckpoint:155) [xc_restore]:
/usr/lib/xen/bin/xc_restore 10 15 6 67584 1 2
[2006-07-10 12:42:00 xend] ERROR (XendCheckpoint:234) xc_linux_restore
start: max_pfn = 10800
[2006-07-10 12:42:00 xend] ERROR (XendCheckpoint:234) Increased domain
reservation by 42000 KB
[2006-07-10 12:42:00 xend] ERROR (XendCheckpoint:234) Reloading memory
pages: 0%
[2006-07-10 12:42:03 xend] ERROR (XendCheckpoint:234) Received all pages
(0 races)
[2006-07-10 12:42:03 xend] ERROR (XendCheckpoint:2100%
[2006-07-10 12:42:03 xend] ERROR (XendCheckpoint:234) Memory reloaded.
[2006-07-10 12:42:03 xend] ERROR (XendCheckpoint:234) Decreased
reservation by 2101 pages
[2006-07-10 12:42:03 xend] ERROR (XendCheckpoint:234) Domain ready to be
built.
[2006-07-10 12:42:03 xend] ERROR (XendCheckpoint:234) Restore exit with
rc=0
[2006-07-10 12:42:03 xend] DEBUG (XendCheckpoint:207) store-mfn 59373
[2006-07-10 12:42:03 xend] DEBUG (XendCheckpoint:207) console-mfn 59372
[2006-07-10 12:42:03 xend.XendDomainInfo] DEBUG (XendDomainInfo:664)
XendDomainInfo.completeRestore
[2006-07-10 12:42:03 xend.XendDomainInfo] DEBUG (XendDomainInfo:736)
Storing domain details: {'console/ring-ref': '59372', 'console/port':
'2', 'name': 'xen.guest.migration.test', 'console/limit': '1048576',
'vm': '/vm/08cd716d-1deb-00f4-77b2-66f14244bc16', 'domid': '6',
'cpu/0/availability': 'online', 'memory/target': '262144',
'store/ring-ref': '59373', 'store/port': '1'}
[2006-07-10 12:42:03 xend.XendDomainInfo] DEBUG (XendDomainInfo:674)
XendDomainInfo.completeRestore done
[2006-07-10 12:42:03 xend.XendDomainInfo] DEBUG (XendDomainInfo:919)
XendDomainInfo.handleShutdownWatch
