"Invalid kernel" error on PV creation -- what's invalid?

[JimmyT _]

I don't know if this is a Fedora, Xen, Cobbler/Koan or local config problem.
Or some combination.

I'm using cobbler+koan to create a PV VM.

hostOS = Fedora 8, guestOS = Fedora 8.

On attempt to create the PV, it chokes at (console error)

 "Error: (2, 'Invalid kernel', 'xc_dom_find_loader: no loader
found\n')"

I see the same in /var/log/xen/xend.log (detailed tail, below), so I suspect
it's a problem with xen.

The vmlinuz & initrd.img I'm using, by reference in 'cobbler distro ...'
invocation are pulled from
http://mirrors.kernel.org/fedora/releases/8/Fedora/x86_64/os/images/xen/, so
I'm not clear what's "Invalid" about the kernel.

Any suggestions?

Jimmy


[2008-01-31 18:21:13 2438] DEBUG (XendDomainInfo:84)
XendDomainInfo.create(['vm',
['name', 'f8TEST'], ['memory', '1024'], ['maxmem',
'1024'], ['vcpus', '2'],
['uuid', '30a76e12-53ef-dfb7-7acb-1aaf3fdfccff'], ['on_poweroff',
'destroy'], ['on_reboot', 'destroy'], ['on_crash',
'destroy'], ['image',
['linux', ['kernel', '/var/lib/xen/vmlinuz'], ['ramdisk',
'/var/lib/xen/initrd.img'], ['args', 'ks=
http://10.0.0.100/cblr/kickstarts/f8TEST/ks.cfg ksdevice=eth0 lang=
kssendmac syslog=10.0.0.100:25150 text  ']]], ['device', ['vbd',
['dev',
'xvda'], ['uname', 'phy:/dev/VG01/f8TEST'], ['mode',
'w']]], ['device',
['vif', ['mac', '00:15:3f:2d:68:d5'], ['bridge',
'eth0']]]])
[2008-01-31 18:21:13 2438] DEBUG (XendDomainInfo:1555)
XendDomainInfo.constructDomain
[2008-01-31 18:21:13 2438] DEBUG (balloon:116) Balloon: 6596932 KiB free;
need 2048; done.
[2008-01-31 18:21:13 2438] DEBUG (XendDomain:443) Adding Domain: 22
[2008-01-31 18:21:13 2438] DEBUG (XendDomainInfo:1609)
XendDomainInfo.initDomain: 22 256
[2008-01-31 18:21:13 2438] DEBUG (XendDomainInfo:1640)
_initDomain:shadow_memory=0x0, memory_static_max=0x40000000,
memory_static_min=0x0.
[2008-01-31 18:21:13 2438] DEBUG (balloon:116) Balloon: 6596924 KiB free;
need 1048576; done.
[2008-01-31 18:21:13 2438] INFO (image:129) buildDomain os=linux dom=22
vcpus=2
[2008-01-31 18:21:13 2438] DEBUG (image:198) domid          = 22
[2008-01-31 18:21:13 2438] DEBUG (image:199) memsize        = 1024
[2008-01-31 18:21:13 2438] DEBUG (image:200) image          =
/var/lib/xen/vmlinuz
[2008-01-31 18:21:13 2438] DEBUG (image:201) store_evtchn   = 1
[2008-01-31 18:21:13 2438] DEBUG (image:202) console_evtchn = 2
[2008-01-31 18:21:13 2438] DEBUG (image:203) cmdline        = ks=
http://10.0.0.100/cblr/kickstarts/f8TEST/ks.cfg ksdevice=eth0 lang=
kssendmac syslog=10.0.0.100:25150 text
[2008-01-31 18:21:13 2438] DEBUG (image:204) ramdisk        =
/var/lib/xen/initrd.img
[2008-01-31 18:21:13 2438] DEBUG (image:205) vcpus          = 2
[2008-01-31 18:21:13 2438] DEBUG (image:206) features       =
[2008-01-31 18:21:13 2438] ERROR (XendDomainInfo:1682)
XendDomainInfo.initDomain: exception occurred
Traceback (most recent call last):
  File "/usr/lib64/python2.5/site-packages/xen/xend/XendDomainInfo.py", line
1657, in _initDomain
    channel_details = self.image.createImage()
  File "/usr/lib64/python2.5/site-packages/xen/xend/image.py", line 112, in
createImage
    return self.createDomain()
  File "/usr/lib64/python2.5/site-packages/xen/xend/image.py", line 131, in
createDomain
    result = self.buildDomain()
  File "/usr/lib64/python2.5/site-packages/xen/xend/image.py", line 530, in
buildDomain
    return LinuxImageHandler.buildDomain(self)
  File "/usr/lib64/python2.5/site-packages/xen/xend/image.py", line 215, in
buildDomain
    features       = self.vm.getFeatures())
Error: (2, 'Invalid kernel', 'xc_dom_find_loader: no loader found\n')
[2008-01-31 18:21:13 2438] ERROR (XendDomainInfo:424) VM start failed
Traceback (most recent call last):
  File "/usr/lib64/python2.5/site-packages/xen/xend/XendDomainInfo.py", line
404, in start
    XendTask.log_progress(31, 60, self._initDomain)
  File "/usr/lib64/python2.5/site-packages/xen/xend/XendTask.py", line 209,
in log_progress
    retval = func(*args, **kwds)
  File "/usr/lib64/python2.5/site-packages/xen/xend/XendDomainInfo.py", line
1685, in _initDomain
    raise VmError(str(exn))
VmError: (2, 'Invalid kernel', 'xc_dom_find_loader: no loader found\n')
[2008-01-31 18:21:13 2438] DEBUG (XendDomainInfo:1802)
XendDomainInfo.destroy: domid=22
[2008-01-31 18:21:13 2438] DEBUG (XendDomainInfo:1821)
XendDomainInfo.destroyDomain(22)
[2008-01-31 18:21:13 2438] ERROR (XendDomainInfo:89) Domain construction
failed
Traceback (most recent call last):
  File "/usr/lib64/python2.5/site-packages/xen/xend/XendDomainInfo.py", line
87, in create
    vm.start()
  File "/usr/lib64/python2.5/site-packages/xen/xend/XendDomainInfo.py", line
404, in start
    XendTask.log_progress(31, 60, self._initDomain)
  File "/usr/lib64/python2.5/site-packages/xen/xend/XendTask.py", line 209,
in log_progress
    retval = func(*args, **kwds)
  File "/usr/lib64/python2.5/site-packages/xen/xend/XendDomainInfo.py", line
1685, in _initDomain
    raise VmError(str(exn))
VmError: (2, 'Invalid kernel', 'xc_dom_find_loader: no loader found\n')
[2008-01-31 18:21:13 2438] DEBUG (XendDomainInfo:1802)
XendDomainInfo.destroy: domid=22
[2008-01-31 18:21:13 2438] DEBUG (XendDomainInfo:1821)
XendDomainInfo.destroyDomain(22)
[2008-01-31 18:21:13 2438] ERROR (XendDomainInfo:1832)
XendDomainInfo.destroy: xc.domain_destroy failed.
Traceback (most recent call last):
  File "/usr/lib64/python2.5/site-packages/xen/xend/XendDomainInfo.py", line
1827, in destroyDomain
    xc.domain_destroy(self.domid)
Error: (3, 'No such process')
[2008-01-31 18:21:13 2438] ERROR (SrvBase:88) Request create failed.
Traceback (most recent call last):
  File "/usr/lib64/python2.5/site-packages/xen/web/SrvBase.py", line 85, in
perform
    return op_method(op, req)
  File "/usr/lib64/python2.5/site-packages/xen/xend/server/SrvDomainDir.py",
line 82, in op_create
    raise XendError("Error creating domain: " + str(ex))
XendError: Error creating domain: (2, 'Invalid kernel', 'xc_dom_find_loader:
no loader found\n')

[JimmyT _]

Digging around for instances of the "no loader found" error, I found in
/var/log/xen/domain-builder-ng.log

xc_dom_allocate: cmdline="ks=
http://10.0.0.100/cblr/kickstarts/f8_min_pv/ks.cfg ksdevice=eth0 lang=
kssendmac syslog=10.0.0.100:25150 text ", features=""
xc_dom_kernel_file: filename="/var/lib/xen/vmlinuz"
xc_dom_ramdisk_file: filename="/var/lib/xen/initrd.img"
xc_dom_boot_xen_init: ver 3.1, caps xen-3.0-x86_64 xen-3.0-x86_32p
hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64
xc_dom_parse_image: called
xc_dom_find_loader: trying ELF-generic loader ... failed
xc_dom_find_loader: trying multiboot-binary loader ... failed
xc_dom_core.c:495: panic: xc_dom_find_loader: no loader found
xc_dom_release: called

checking for the loaders in /var/lib/cobbler/settings,

---
bootloaders:
    ia64: /var/lib/cobbler/elilo-3.6-ia64.efi
    standard: /usr/lib/syslinux/pxelinux.0
...

they do exist:

ls -al /var/lib/cobbler/elilo-3.6-ia64.efi /usr/lib/syslinux/pxelinux.0
 -rw-r--r-- 1 root root  13408 2007-10-17 08:53 /usr/lib/syslinux/pxelinux.0
 -rw-rw-r-- 1 root root 374146 2008-01-31 15:04 /var/lib/cobbler/elilo-
3.6-ia64.efi

Not clear why the "xc_dom_find_loader" calls for finding the loader
"failed", unless they're the wrong loader type.

[resolved] Re: "Invalid kernel" error on PV creation -- what's invalid?

[JimmyT _]

To close the loop, this problem turns out to be a cobbler/koan issue.
the kernel really IS invalid.

with the help of "high_roller" in #rhel, i found that koan's pulling
the vmlinuz kernel from the cobbler server over an http:// connection
-- rather than an apparently required https:// -- and loading the
resultant error-page html INTO the vmlinuz file used to (attempt to)
create the PV.

i.e., the pull occurs as expected, no error about the pull is
launched, but the process populates the target DL with error code.

of course, this results in the 'kernel', now populated with garbage,
being identified as "invalid".

since i'd checked my source files, and as i read the logs to imply the
pull was happening without error, I simply didn't look at what
actually WAS pulled.

i'll communicate with the cobbler/koan author to iron this out.

hope someone else finds this helpful.

jimmy