libvirtd crashes host when autostarting

[Rich Mahn]

Summary:
   My host crashes when I use the autostart function in
libvirtd.

This command sequence works fine:

service libvirtd stop
virsh autostart mydomain --disable
service lib virtd start
virsh start mydomain

This command sequence will crash the host:

service libvirtd stop
virsh autostart mydomain
service libvirtd start

I can't find any error messages related to this.  The host system
freezes, the keyboard starts blinking the caps-lock and scroll-lock,
and as best I can tell, the kernel has stopped running.

If I starting up the host system with the VM autostarted, I sometimes
get messages that look like kernel errors on the screen, but I don't
find them in any log files so far.  They seem to be related to networking.

Here's some configuration information--I don't know really what all is
needed, but this, I guess is a start:

Host and VM both Fedora 11 with the lastest updates.
Host machine in Intel based with the virtual support turned on.

In the /var/log/libvirtd/qemu directory, the command line to start
the virtual machine is different when started from autostart.  The
difference is in this portion:

-net tap,fd=11,vlan=0	for failure (autostart) and
        -net tap,fd=18,vlan=0	for success

I am using a bridged connection.

Here's the command line that's used, split over
lines for some degree of readability:

LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin /usr/bin/qemu-kvm -S -M pc
    -m 1024 -smp 1 -name mydomain
        -uuid 5cde5a23-2abe-fcdd-04dc-5ae3a46ab51c
    -monitor pty -pidfile /var/run/libvirt/qemu//mydomain.pid
    -boot c -drive file=,if=ide,media=cdrom,index=2
    -drive file=/dev/mapper/vg_host-lv_mydomain,if=virtio,index=0,boot=on
    -drive file=/dev/mapper/vg_host-lv_mydomain_data,if=virtio,index=1
    -net nic,macaddr=54:52:00:67:91:e9,vlan=0,model=virtio
    -net tap,fd=11,vlan=0 -serial pty -parallel none
    -usb -usbdevice tablet -vnc 127.0.0.1:0

[Mark McLoughlin]

On Fri, 2009-10-09 at 09:45 -0400, Rich Mahn wrote:
> Summary:
>    My host crashes when I use the autostart function in
> libvirtd.
>
> This command sequence works fine:
>
> service libvirtd stop
> virsh autostart mydomain --disable
> service lib virtd start
> virsh start mydomain
>
> This command sequence will crash the host:
>
> service libvirtd stop
> virsh autostart mydomain
> service libvirtd start
>
> I can't find any error messages related to this.  The host system
> freezes, the keyboard starts blinking the caps-lock and scroll-lock,
> and as best I can tell, the kernel has stopped running.

Ouch, that's really bad. This is a 2.6.30 kernel, right? Is it still
reproducible with an older 2.6.29 kernel or a 2.6.31 kernel from
rawhide?

This is ultimately a kernel bug, userspace shouldn't be able to screw up
the kernel like this.

It'd be good to get this into bugzilla:

https://fedoraproject.org/wiki/Reporting virtualization bugs
> If I starting up the host system with the VM autostarted, I sometimes
> get messages that look like kernel errors on the screen, but I don't
> find them in any log files so far.  They seem to be related to networking.
>
> Here's some configuration information--I don't know really what all is
> needed, but this, I guess is a start:
>
> Host and VM both Fedora 11 with the lastest updates.
> Host machine in Intel based with the virtual support turned on.
>
> In the /var/log/libvirtd/qemu directory, the command line to start
> the virtual machine is different when started from autostart.  The
> difference is in this portion:
>    -net tap,fd=11,vlan=0	for failure (autostart) and
>    -net tap,fd=18,vlan=0	for success

Interesting, but I can't immediately think where the difference may come
from or how it could be related to the kernel lockup.

Cheers,
Mark.

[Paul Lambert]

Rich, Mark,

I experienced this problem as well when using a FE-11 host and FE-11 Guest.
I now have FE-12-alpha installed but have been afraid to try it until I know
if this issue was addressed in the new version.

Configuring a guest to start on host bootup totally hoses the host.  The
only way to recover is to boot off of a CD and then find the startup
directory and delete the link to the guest.  The host will reboot OK after
this.

By the way Mark, still no mounting for USB disks with FE-12-Alpha.  Up to
date on patches as of 10/8/2009.

Paul

On Fri, Oct 9, 2009 at 9:53 AM, Mark McLoughlin  wrote:
> On Fri, 2009-10-09 at 09:45 -0400, Rich Mahn wrote:
> > Summary:
> >    My host crashes when I use the autostart function in
> > libvirtd.
> >
> > This command sequence works fine:
> >
> > service libvirtd stop
> > virsh autostart mydomain --disable
> > service lib virtd start
> > virsh start mydomain
> >
> > This command sequence will crash the host:
> >
> > service libvirtd stop
> > virsh autostart mydomain
> > service libvirtd start
> >
> > I can't find any error messages related to this.  The host system
> > freezes, the keyboard starts blinking the caps-lock and scroll-lock,
> > and as best I can tell, the kernel has stopped running.
>
> Ouch, that's really bad. This is a 2.6.30 kernel, right? Is it still
> reproducible with an older 2.6.29 kernel or a 2.6.31 kernel from
> rawhide?
>
> This is ultimately a kernel bug, userspace shouldn't be able to screw up
> the kernel like this.
>
> It'd be good to get this into bugzilla:
>
> https://fedoraproject.org/wiki/Reporting virtualization bugs
> > If I starting up the host system with the VM autostarted, I sometimes
> > get messages that look like kernel errors on the screen, but I don't
> > find them in any log files so far.  They seem to be related to
>
> networking.
> > Here's some configuration information--I don't know really what all is
> > needed, but this, I guess is a start:
> >
> > Host and VM both Fedora 11 with the lastest updates.
> > Host machine in Intel based with the virtual support turned on.
> >
> > In the /var/log/libvirtd/qemu directory, the command line to start
> > the virtual machine is different when started from autostart.  The
> > difference is in this portion:
> >       -net tap,fd=11,vlan=0       for failure (autostart) and
> >       -net tap,fd=18,vlan=0       for success
>
> Interesting, but I can't immediately think where the difference may come
> from or how it could be related to the kernel lockup.
>
> Cheers,
> Mark.
>
> Fedora-virt mailing list
> [e-mail]
> https://www.redhat.com/mailman/listinfo/fedora-virt

[Mark McLoughlin]

On Fri, 2009-10-09 at 10:18 -0400, Paul Lambert wrote:
> Rich, Mark,
>
> I experienced this problem as well when using a FE-11 host and FE-11
> Guest.  I now have FE-12-alpha installed but have been afraid to try
> it until I know if this issue was addressed in the new version.
>
> Configuring a guest to start on host bootup totally hoses the host.
> The only way to recover is to boot off of a CD and then find the
> startup directory and delete the link to the guest.  The host will
> reboot OK after this.

Ouch. We really need to get to the bottom of this then.
> By the way Mark, still no mounting for USB disks with FE-12-Alpha.  Up
> to date on patches as of 10/8/2009.

Have you updated to selinux-policy-3.6.32-17.fc12.noarch ? Do you have
the virt_use_usb sebool enabled?

Thanks,
Mark.

[Rich Mahn]

> Rich, Mark,
> I experienced this problem as well when using a FE-11 host and FE-11 Guest.
> I now have FE-12-alpha installed but have been afraid to try it until I know
> if this issue was addressed in the new version.
> Configuring a guest to start on host bootup totally hoses the host.  The
> only way to recover is to boot off of a CD and then find the startup
> directory and delete the link to the guest.  The host will reboot OK after
> this.

Paul,
   If you are using grub, type 'a' on the kernel selection, replace
'rhbg quiet' at the end of the line with 'single' ahd boot.  Then
go ahead and remove the link.  Much easier than booting from CD.

rich

[Rich Mahn]

> On Fri, 2009-10-09 at 09:45 -0400, Rich Mahn wrote:
> > Summary:
> >    My host crashes when I use the autostart function in
> > libvirtd.
> >
> > This command sequence works fine:
> >
> > service libvirtd stop
> > virsh autostart mydomain --disable
> > service lib virtd start
> > virsh start mydomain
> >
> > This command sequence will crash the host:
> >
> > service libvirtd stop
> > virsh autostart mydomain
> > service libvirtd start
> >
> > I can't find any error messages related to this.  The host system
> > freezes, the keyboard starts blinking the caps-lock and scroll-lock,
> > and as best I can tell, the kernel has stopped running.
> Ouch, that's really bad. This is a 2.6.30 kernel, right? Is it still
> reproducible with an older 2.6.29 kernel or a 2.6.31 kernel from
> rawhide?

okay, it works correctly on 2.6.29.  I'm trying to figure out how to
install 2.6.31 from rawhide without having dozens/hundreds of other
packages modified as well.  Is there some option to 'yum' that will do
what I need?

rich

[Dale Bewley]

On Fri, 2009-10-09 at 10:39 -0400, Rich Mahn wrote:
> > On Fri, 2009-10-09 at 09:45 -0400, Rich Mahn wrote:
> > > Summary:
> > >    My host crashes when I use the autostart function in
> > > libvirtd.
> > >
> > > This command sequence works fine:
> > >
> > > service libvirtd stop
> > > virsh autostart mydomain --disable
> > > service lib virtd start
> > > virsh start mydomain
> > >
> > > This command sequence will crash the host:
> > >
> > > service libvirtd stop
> > > virsh autostart mydomain
> > > service libvirtd start
> > >
> > > I can't find any error messages related to this.  The host system
> > > freezes, the keyboard starts blinking the caps-lock and scroll-lock,
> > > and as best I can tell, the kernel has stopped running.
> > Ouch, that's really bad. This is a 2.6.30 kernel, right? Is it still
> > reproducible with an older 2.6.29 kernel or a 2.6.31 kernel from
> > rawhide?
>
> okay, it works correctly on 2.6.29.  I'm trying to figure out how to
> install 2.6.31 from rawhide without having dozens/hundreds of other
> packages modified as well.  Is there some option to 'yum' that will do
> what I need?

I am seeing a kernel oops and panic on a host running
2.6.31.6-166.fc12.x86_64 (and also at least 2.6.31.6-145) when I
autostart a f12-x86_64 qemu-kvm guest.

If I flag a guest as autoboot and reboot the host then the host starts,
the guest starts, and some seconds later (presumably when the guest is
fully up or just before) the host OOPs and hangs. The full error log is
attached, it begins as:

BUG: unable to handle kernel paging request at 0000000000200200
IP: [<ffffffff8139aad7>] destroy_conntrack+0x82/0x11f
PGD 209d9d067 PUD 214dfa067 PMD 0 
Oops: 0002 [#1] SMP 
last sysfs
file: /sys/devices/pci0000:00/0000:00:09.0/0000:01:00.0/0000:02:02.0/0000:05:00.0/0000:06:00.0/irq
CPU 0 
Modules linked in: tun bridge stp llc xt_physdev ipt_LOG xt_limit
xt_comment xt_multiport ip6t_REJECT nf_conntrack_ipv6 ip6table_filter
ip6_tables ipv6 dm_multipath kvm_amd kvm bnx2 amd64_edac_mod shpchp
edac_core i2c_piix4 serio_raw dcdbas raid1 raid10 mptsas mptscsih
mptbase sata_svw scsi_transport_sas radeon ttm drm_kms_helper drm
i2c_algo_bit i2c_core [last unloaded: freq_table]
Pid: 1962, comm: qemu-kvm Not tainted 2.6.31.6-166.fc12.x86_64 #1
PowerEdge 2970
RIP: 0010:[<ffffffff8139aad7>]  [<ffffffff8139aad7>] destroy_conntrack
+0x82/0x11f

I'm running on a quad core AMD with:

qemu-kvm-0.11.0-12.fc12.x86_64
kernel-2.6.31.6-166.fc12.x86_64
libvirt-0.7.1-15.fc12.x86_64

What can I do to help further diagnose this problem?

[Adam Huffman]

On Mon, Dec 14, 2009 at 2:04 AM, Dale Bewley  wrote:
> On Fri, 2009-10-09 at 10:39 -0400, Rich Mahn wrote:
> > > On Fri, 2009-10-09 at 09:45 -0400, Rich Mahn wrote:
> > > > Summary:
> > > >    My host crashes when I use the autostart function in
> > > > libvirtd.
> > > >
> > > > This command sequence works fine:
> > > >
> > > > service libvirtd stop
> > > > virsh autostart mydomain --disable
> > > > service lib virtd start
> > > > virsh start mydomain
> > > >
> > > > This command sequence will crash the host:
> > > >
> > > > service libvirtd stop
> > > > virsh autostart mydomain
> > > > service libvirtd start
> > > >
> > > > I can't find any error messages related to this.  The host system
> > > > freezes, the keyboard starts blinking the caps-lock and scroll-lock,
> > > > and as best I can tell, the kernel has stopped running.
> > > Ouch, that's really bad. This is a 2.6.30 kernel, right? Is it still
> > > reproducible with an older 2.6.29 kernel or a 2.6.31 kernel from
> > > rawhide?
> >
> > okay, it works correctly on 2.6.29.  I'm trying to figure out how to
> > install 2.6.31 from rawhide without having dozens/hundreds of other
> > packages modified as well.  Is there some option to 'yum' that will do
> > what I need?
>
> I am seeing a kernel oops and panic on a host running
> 2.6.31.6-166.fc12.x86_64 (and also at least 2.6.31.6-145) when I
> autostart a f12-x86_64 qemu-kvm guest.
>
> If I flag a guest as autoboot and reboot the host then the host starts,
> the guest starts, and some seconds later (presumably when the guest is
> fully up or just before) the host OOPs and hangs. The full error log is
> attached, it begins as:
>
> BUG: unable to handle kernel paging request at 0000000000200200
> IP: [<ffffffff8139aad7>] destroy_conntrack+0x82/0x11f
> PGD 209d9d067 PUD 214dfa067 PMD 0
> Oops: 0002 [#1] SMP
> last sysfs
> file: /sys/devices/pci0000:00/0000:00:09.0/0000:01:00.0/0000:02:02.0/0000:05:00.0/0000:06:00.0/irq
> CPU 0
> Modules linked in: tun bridge stp llc xt_physdev ipt_LOG xt_limit
> xt_comment xt_multiport ip6t_REJECT nf_conntrack_ipv6 ip6table_filter
> ip6_tables ipv6 dm_multipath kvm_amd kvm bnx2 amd64_edac_mod shpchp
> edac_core i2c_piix4 serio_raw dcdbas raid1 raid10 mptsas mptscsih
> mptbase sata_svw scsi_transport_sas radeon ttm drm_kms_helper drm
> i2c_algo_bit i2c_core [last unloaded: freq_table]
> Pid: 1962, comm: qemu-kvm Not tainted 2.6.31.6-166.fc12.x86_64 #1
> PowerEdge 2970
> RIP: 0010:[<ffffffff8139aad7>]  [<ffffffff8139aad7>] destroy_conntrack
> +0x82/0x11f
>
> I'm running on a quad core AMD with:
>
> qemu-kvm-0.11.0-12.fc12.x86_64
> kernel-2.6.31.6-166.fc12.x86_64
> libvirt-0.7.1-15.fc12.x86_64
>
> What can I do to help further diagnose this problem?
>
> Fedora-virt mailing list
> [e-mail]
> https://www.redhat.com/mailman/listinfo/fedora-virt

For what it's worth I've been seeing very similar crashes.  I reported it as:

https://bugzilla.redhat.com/show_bug.cgi?id=545851

and on the KVM list.  They thought it was a fault in the networking
stack, so people are now looking into it there.

On my hardware, if I disable VT-d in the BIOS, the crashes don't happen.

Adam

[Adam Huffman]

On Tue, Dec 15, 2009 at 3:04 PM, Adam Huffman  wrote:
> On Mon, Dec 14, 2009 at 2:04 AM, Dale Bewley  wrote:
> > On Fri, 2009-10-09 at 10:39 -0400, Rich Mahn wrote:
> > > > On Fri, 2009-10-09 at 09:45 -0400, Rich Mahn wrote:
> > > > > Summary:
> > > > >    My host crashes when I use the autostart function in
> > > > > libvirtd.
> > > > >
> > > > > This command sequence works fine:
> > > > >
> > > > > service libvirtd stop
> > > > > virsh autostart mydomain --disable
> > > > > service lib virtd start
> > > > > virsh start mydomain
> > > > >
> > > > > This command sequence will crash the host:
> > > > >
> > > > > service libvirtd stop
> > > > > virsh autostart mydomain
> > > > > service libvirtd start
> > > > >
> > > > > I can't find any error messages related to this.  The host system
> > > > > freezes, the keyboard starts blinking the caps-lock and scroll-lock,
> > > > > and as best I can tell, the kernel has stopped running.
> > > > Ouch, that's really bad. This is a 2.6.30 kernel, right? Is it still
> > > > reproducible with an older 2.6.29 kernel or a 2.6.31 kernel from
> > > > rawhide?
> > >
> > > okay, it works correctly on 2.6.29.  I'm trying to figure out how to
> > > install 2.6.31 from rawhide without having dozens/hundreds of other
> > > packages modified as well.  Is there some option to 'yum' that will do
> > > what I need?
> >
> > I am seeing a kernel oops and panic on a host running
> > 2.6.31.6-166.fc12.x86_64 (and also at least 2.6.31.6-145) when I
> > autostart a f12-x86_64 qemu-kvm guest.
> >
> > If I flag a guest as autoboot and reboot the host then the host starts,
> > the guest starts, and some seconds later (presumably when the guest is
> > fully up or just before) the host OOPs and hangs. The full error log is
> > attached, it begins as:
> >
> > BUG: unable to handle kernel paging request at 0000000000200200
> > IP: [<ffffffff8139aad7>] destroy_conntrack+0x82/0x11f
> > PGD 209d9d067 PUD 214dfa067 PMD 0
> > Oops: 0002 [#1] SMP
> > last sysfs
> > file: /sys/devices/pci0000:00/0000:00:09.0/0000:01:00.0/0000:02:02.0/0000:05:00.0/0000:06:00.0/irq
> > CPU 0
> > Modules linked in: tun bridge stp llc xt_physdev ipt_LOG xt_limit
> > xt_comment xt_multiport ip6t_REJECT nf_conntrack_ipv6 ip6table_filter
> > ip6_tables ipv6 dm_multipath kvm_amd kvm bnx2 amd64_edac_mod shpchp
> > edac_core i2c_piix4 serio_raw dcdbas raid1 raid10 mptsas mptscsih
> > mptbase sata_svw scsi_transport_sas radeon ttm drm_kms_helper drm
> > i2c_algo_bit i2c_core [last unloaded: freq_table]
> > Pid: 1962, comm: qemu-kvm Not tainted 2.6.31.6-166.fc12.x86_64 #1
> > PowerEdge 2970
> > RIP: 0010:[<ffffffff8139aad7>]  [<ffffffff8139aad7>] destroy_conntrack
> > +0x82/0x11f
> >
> > I'm running on a quad core AMD with:
> >
> > qemu-kvm-0.11.0-12.fc12.x86_64
> > kernel-2.6.31.6-166.fc12.x86_64
> > libvirt-0.7.1-15.fc12.x86_64
> >
> > What can I do to help further diagnose this problem?
> >
> > Fedora-virt mailing list
> > [e-mail]
> > https://www.redhat.com/mailman/listinfo/fedora-virt
>
> For what it's worth I've been seeing very similar crashes.  I reported it as:
>
> https://bugzilla.redhat.com/show_bug.cgi?id=545851
>
> and on the KVM list.  They thought it was a fault in the networking
> stack, so people are now looking into it there.
>
> On my hardware, if I disable VT-d in the BIOS, the crashes don't happen.
>
> Adam

As I've noted in the bug report, if I disable IPv6, the system is stable.

Adam

[Ján ONDREJ (SAL)]

On Wed, Jan 06, 2010 at 10:23:16AM +0000, Adam Huffman wrote:
> On Tue, Dec 15, 2009 at 3:04 PM, Adam Huffman  wrote:
> > On Mon, Dec 14, 2009 at 2:04 AM, Dale Bewley  wrote:
> > > On Fri, 2009-10-09 at 10:39 -0400, Rich Mahn wrote:
> > > > > On Fri, 2009-10-09 at 09:45 -0400, Rich Mahn wrote:
> > > > > > Summary:
> > > > > >    My host crashes when I use the autostart function in
> > > > > > libvirtd.
> > > > > >
> > > > > > This command sequence works fine:
> > > > > >
> > > > > > service libvirtd stop
> > > > > > virsh autostart mydomain --disable
> > > > > > service lib virtd start
> > > > > > virsh start mydomain
> > > > > >
> > > > > > This command sequence will crash the host:
> > > > > >
> > > > > > service libvirtd stop
> > > > > > virsh autostart mydomain
> > > > > > service libvirtd start
> > > > > >
> > > > > > I can't find any error messages related to this.  The host system
> > > > > > freezes, the keyboard starts blinking the caps-lock and scroll-lock,
> > > > > > and as best I can tell, the kernel has stopped running.
> > > > > Ouch, that's really bad. This is a 2.6.30 kernel, right? Is it still
> > > > > reproducible with an older 2.6.29 kernel or a 2.6.31 kernel from
> > > > > rawhide?
> > > >
> > > > okay, it works correctly on 2.6.29.  I'm trying to figure out how to
> > > > install 2.6.31 from rawhide without having dozens/hundreds of other
> > > > packages modified as well.  Is there some option to 'yum' that will do
> > > > what I need?
> > >
> > > I am seeing a kernel oops and panic on a host running
> > > 2.6.31.6-166.fc12.x86_64 (and also at least 2.6.31.6-145) when I
> > > autostart a f12-x86_64 qemu-kvm guest.
> > >
> > > If I flag a guest as autoboot and reboot the host then the host starts,
> > > the guest starts, and some seconds later (presumably when the guest is
> > > fully up or just before) the host OOPs and hangs. The full error log is
> > > attached, it begins as:
> > >
> > > BUG: unable to handle kernel paging request at 0000000000200200
> > > IP: [<ffffffff8139aad7>] destroy_conntrack+0x82/0x11f
> > > PGD 209d9d067 PUD 214dfa067 PMD 0
> > > Oops: 0002 [#1] SMP
> > > last sysfs
> > > file: /sys/devices/pci0000:00/0000:00:09.0/0000:01:00.0/0000:02:02.0/0000:05:00.0/0000:06:00.0/irq
> > > CPU 0
> > > Modules linked in: tun bridge stp llc xt_physdev ipt_LOG xt_limit
> > > xt_comment xt_multiport ip6t_REJECT nf_conntrack_ipv6 ip6table_filter
> > > ip6_tables ipv6 dm_multipath kvm_amd kvm bnx2 amd64_edac_mod shpchp
> > > edac_core i2c_piix4 serio_raw dcdbas raid1 raid10 mptsas mptscsih
> > > mptbase sata_svw scsi_transport_sas radeon ttm drm_kms_helper drm
> > > i2c_algo_bit i2c_core [last unloaded: freq_table]
> > > Pid: 1962, comm: qemu-kvm Not tainted 2.6.31.6-166.fc12.x86_64 #1
> > > PowerEdge 2970
> > > RIP: 0010:[<ffffffff8139aad7>]  [<ffffffff8139aad7>] destroy_conntrack
> > > +0x82/0x11f
> > >
> > > I'm running on a quad core AMD with:
> > >
> > > qemu-kvm-0.11.0-12.fc12.x86_64
> > > kernel-2.6.31.6-166.fc12.x86_64
> > > libvirt-0.7.1-15.fc12.x86_64
> > >
> > > What can I do to help further diagnose this problem?
> > >
> > > Fedora-virt mailing list
> > > [e-mail]
> > > https://www.redhat.com/mailman/listinfo/fedora-virt
> >
> > For what it's worth I've been seeing very similar crashes.  I reported it as:
> >
> > https://bugzilla.redhat.com/show_bug.cgi?id=545851
> >
> > and on the KVM list.  They thought it was a fault in the networking
> > stack, so people are now looking into it there.
> >
> > On my hardware, if I disable VT-d in the BIOS, the crashes don't happen.
> >
> > Adam
>
> As I've noted in the bug report, if I disable IPv6, the system is stable.

For me it's enough to disable iptables for bridge by adding:

net.bridge.bridge-nf-call-ip6tables = 0

to /etc/sysctl.conf

SAL