Fedora kernel crashing in virtio when running on qemu

[Richard W.M. Jones]

https://bugzilla.redhat.com/show_bug.cgi?id=1430297

This has been going on for a few weeks with no attention.  It prevents
anyone from running libguestfs on Fedora 26.

It's (IMHO) pretty serious for anyone using virtualization: "kernel
doesn't boot on qemu" should be a blocker for a Fedora release.

Can anyone see what's going on?

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine.  Supports Linux and Windows.
http://people.redhat.com/~rjones/virt-df/

[Adam Williamson]

On Tue, 2017-03-21 at 19:40 +0000, Richard W.M. Jones wrote:
>  https://bugzilla.redhat.com/show_bug.cgi?id=1430297
>  
>  This has been going on for a few weeks with no attention.  It prevents
>  anyone from running libguestfs on Fedora 26.
>  
>  It's (IMHO) pretty serious for anyone using virtualization: "kernel
>  doesn't boot on qemu" should be a blocker for a Fedora release.
>  
>  Can anyone see what's going on? 
The earlier case looks like
https://bugzilla.redhat.com/show_bug.cgi?id=1430043 , which was fixed
in rc2. I'm not sure about the latter case, though Thorsten Leemhuis
reported intermittent crash / hang during boot with rc2 as:

https://bugzilla.kernel.org/show_bug.cgi?id=194911

Since rc2 I've seen occasional failures which are probably the same as
the ones Thorsten was seeing, but usually it works OK. I run 26-on-26
VMs via virt-manager quite a lot, they certainly don't fail every time.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net

[Richard W.M. Jones]

On Tue, Mar 21, 2017 at 12:58:06PM -0700, Adam Williamson wrote:
>  On Tue, 2017-03-21 at 19:40 +0000, Richard W.M. Jones wrote:
>  > https://bugzilla.redhat.com/show_bug.cgi?id=1430297
>  > 
>  > This has been going on for a few weeks with no attention.  It prevents
>  > anyone from running libguestfs on Fedora 26.
>  > 
>  > It's (IMHO) pretty serious for anyone using virtualization: "kernel
>  > doesn't boot on qemu" should be a blocker for a Fedora release.
>  > 
>  > Can anyone see what's going on?
>  
>  The earlier case looks like
>  https://bugzilla.redhat.com/show_bug.cgi?id=1430043 , which was fixed
>  in rc2. I'm not sure about the latter case, though Thorsten Leemhuis
>  reported intermittent crash / hang during boot with rc2 as:
>  
>  https://bugzilla.kernel.org/show_bug.cgi?id=194911
>  
>  Since rc2 I've seen occasional failures which are probably the same as
>  the ones Thorsten was seeing, but usually it works OK. I run 26-on-26
>  VMs via virt-manager quite a lot, they certainly don't fail every time. 
Does libguestfs-test-tool work for you in F26?

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-builder quickly builds VMs from scratch
http://libguestfs.org/virt-builder.1.html

[Adam Williamson]

On Tue, 2017-03-21 at 20:48 +0000, Richard W.M. Jones wrote:
>  On Tue, Mar 21, 2017 at 12:58:06PM -0700, Adam Williamson wrote:
>  > On Tue, 2017-03-21 at 19:40 +0000, Richard W.M. Jones wrote:
>  > > https://bugzilla.redhat.com/show_bug.cgi?id=1430297
>  > > 
>  > > This has been going on for a few weeks with no attention.  It prevents
>  > > anyone from running libguestfs on Fedora 26.
>  > > 
>  > > It's (IMHO) pretty serious for anyone using virtualization: "kernel
>  > > doesn't boot on qemu" should be a blocker for a Fedora release.
>  > > 
>  > > Can anyone see what's going on?
>  > 
>  > The earlier case looks like
>  > https://bugzilla.redhat.com/show_bug.cgi?id=1430043 , which was fixed
>  > in rc2. I'm not sure about the latter case, though Thorsten Leemhuis
>  > reported intermittent crash / hang during boot with rc2 as:
>  > 
>  > https://bugzilla.kernel.org/show_bug.cgi?id=194911
>  > 
>  > Since rc2 I've seen occasional failures which are probably the same as
>  > the ones Thorsten was seeing, but usually it works OK. I run 26-on-26
>  > VMs via virt-manager quite a lot, they certainly don't fail every time.
>  
>  Does libguestfs-test-tool work for you in F26? 
Hum, interesting results:

If I run it as a regular user, it works.
If I run it as root as usual, it gives me a libvirt error, some
permission issue.
If I run it as root with LIBGUESTFS_BACKEND=direct , it hits the same
kernel trace you hit.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net

[Adam Williamson]

On Tue, 2017-03-21 at 15:59 -0700, Adam Williamson wrote:
>  On Tue, 2017-03-21 at 20:48 +0000, Richard W.M. Jones wrote:
>  > On Tue, Mar 21, 2017 at 12:58:06PM -0700, Adam Williamson wrote:
>  > > On Tue, 2017-03-21 at 19:40 +0000, Richard W.M. Jones wrote:
>  > > > https://bugzilla.redhat.com/show_bug.cgi?id=1430297
>  > > > 
>  > > > This has been going on for a few weeks with no attention.  It prevents
>  > > > anyone from running libguestfs on Fedora 26.
>  > > > 
>  > > > It's (IMHO) pretty serious for anyone using virtualization:
> "kernel
>  > > > doesn't boot on qemu" should be a blocker for a Fedora release.
>  > > > 
>  > > > Can anyone see what's going on?
>  > > 
>  > > The earlier case looks like
>  > > https://bugzilla.redhat.com/show_bug.cgi?id=1430043 , which was fixed
>  > > in rc2. I'm not sure about the latter case, though Thorsten Leemhuis
>  > > reported intermittent crash / hang during boot with rc2 as:
>  > > 
>  > > https://bugzilla.kernel.org/show_bug.cgi?id=194911
>  > > 
>  > > Since rc2 I've seen occasional failures which are probably the same as
>  > > the ones Thorsten was seeing, but usually it works OK. I run 26-on-26
>  > > VMs via virt-manager quite a lot, they certainly don't fail every time.
>  > 
>  > Does libguestfs-test-tool work for you in F26?
>  
>  Hum, interesting results: 
...

>  If I run it as root with LIBGUESTFS_BACKEND=direct , it hits the
> same
>  kernel trace you hit. 
Well, first time it did that. Second time, I got this, and the boot
hung:

[    0.937752] EXT4-fs (sda1): mounting ext2 file system using the ext4 subsystem
[    0.938362] general protection fault: 0000 [#1] SMP
[    0.938719] Modules linked in: kvm_intel kvm snd_pcsp snd_pcm ppdev irqbypass
ghash_clmulni_intel snd_timer snd parport_pc ata_generic soundcore joydev pata_acpi
parport i2c_piix4 serio_raw qemu_fw_cfg acpi_cpufreq tpm_tis tpm_tis_core tpm libcrc32c
crc8 crc7 crc_itu_t virtio_pci virtio_mmio virtio_input virtio_balloon virtio_scsi nd_pmem
nd_btt virtio_net virtio_crypto crypto_engine virtio_console virtio_rng virtio_blk
virtio_ring virtio nfit crc32_generic crct10dif_pclmul crc32c_intel crc32_pclmul
[    0.939350] CPU: 0 PID: 457 Comm: mount Not tainted 4.11.0-0.rc2.git2.2.fc26.x86_64 #1
[    0.939350] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.10.2-1.fc26
04/01/2014
[    0.939350] task: ffff94981d290000 task.stack: ffffae8780374000
[    0.939350] RIP: 0010:__kmalloc+0xa4/0x210
[    0.939350] RSP: 0018:ffffae8780377c30 EFLAGS: 00010246
[    0.939350] RAX: 2d326f6974726976 RBX: 00000000014000c0 RCX: 000000000000000a
[    0.939350] RDX: 0000000000000015 RSI: 0000000000000000 RDI: 000000000001cae0
[    0.939350] RBP: ffffae8780377c60 R08: ffff94981f01cae0 R09: ffff94981ec02280
[    0.939350] R10: 00000000000005ac R11: ffff94981b640400 R12: 2d326f6974726976
[    0.939350] R13: 00000000014000c0 R14: 0000000000002000 R15: ffff94981ec02280
[    0.939350] FS:  00007fe249b47480(0000) GS:ffff94981f000000(0000)
knlGS:0000000000000000
[    0.939350] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[    0.939350] CR2: 000055ac86a6ac08 CR3: 000000001d329000 CR4: 00000000000406f0
[    0.939350] Call Trace:
[    0.939350]  ? mb_cache_create+0x75/0x110
[    0.939350]  mb_cache_create+0x75/0x110
[    0.939350]  ext4_xattr_create_cache+0x13/0x20
[    0.939350]  ext4_fill_super+0x1cad/0x3600
[    0.939350]  ? __kmalloc+0x1d1/0x210
[    0.939350]  ? snprintf+0x49/0x60
[    0.939350]  mount_bdev+0x17c/0x1b0
[    0.939350]  ? ext4_calculate_overhead+0x490/0x490
[    0.939350]  ext4_mount+0x15/0x20
[    0.939350]  mount_fs+0x32/0x160
[    0.939350]  ? __alloc_percpu+0x15/0x20
[    0.939350]  vfs_kern_mount.part.19+0x5d/0x120
[    0.939350]  do_mount+0x520/0xc30
[    0.939350]  ? _copy_from_user+0x4e/0x80
[    0.939350]  ? memdup_user+0x4f/0x80
[    0.939350]  SyS_mount+0x98/0xe0
[    0.939350]  entry_SYSCALL_64_fastpath+0x1a/0xa9
[    0.939350] RIP: 0033:0x7fe248bcc4fa
[    0.939350] RSP: 002b:00007ffef410b938 EFLAGS: 00000246 ORIG_RAX: 00000000000000a5
[    0.939350] RAX: ffffffffffffffda RBX: 00007fe249741522 RCX: 00007fe248bcc4fa
[    0.939350] RDX: 000055ac86a69c80 RSI: 000055ac86a62f50 RDI: 000055ac86a60240
[    0.939350] RBP: 00007fe249951184 R08: 0000000000000000 R09: 00007ffef410a838
[    0.939350] R10: 00000000c0ed0000 R11: 0000000000000246 R12: 000055ac86a60120
[    0.939350] R13: 00007ffef410bc48 R14: 0000000000000000 R15: 00000000ffffffff
[    0.939350] Code: 49 83 78 10 00 4d 8b 20 0f 84 03 01 00 00 4d 85 e4 0f 84 fa 00 00 00
49 63 41 20 49 8b 39 4c 01 e0 40 f6 c7 0f 0f 85 5f 01 00 00 <48> 8b 18 48 8d 4a 01
4c 89 e0 65 48 0f c7 0f 0f 94 c0 84 c0 74 
[    0.939350] RIP: __kmalloc+0xa4/0x210 RSP: ffffae8780377c30
[    0.958773] ---[ end trace 900f41ab5fa73f40 ]---
guestfsd: error: /dev/sda1 on / (options: ''): 
guestfsd: main_loop: proc 1 (mount) took 0.02 seconds
libguestfs: error: mount: /dev/sda1 on / (options: ''): 
libguestfs: closing guestfs handle 0x5653569ae490 (state 2)
guestfsd: main_loop: new request, len 0x28
umount-all: /proc/mounts: fsname=/dev/root dir=/ type=ext2
opts=rw,noatime,block_validity,barrier,user_xattr,acl freq=0 passno=0
umount-all: /proc/mounts: fsname=/proc dir=/proc type=proc opts=rw,relatime freq=0
passno=0
umount-all: /proc/mounts: fsname=/sys dir=/sys type=sysfs opts=rw,relatime freq=0
passno=0
umount-all: /proc/mounts: fsname=tmpfs dir=/run type=tmpfs
opts=rw,nosuid,relatime,size=97320k,mode=755 freq=0 passno=0
umount-all: /proc/mounts: fsname=/dev dir=/dev type=devtmpfs
opts=rw,relatime,size=240912k,nr_inodes=60228,mode=755 freq=0 passno=0
umount-all: /proc/mounts: fsname=/dev/pts dir=/dev/pts type=devpts
opts=rw,relatime,mode=600,ptmxmode=000 freq=0 passno=0
[    1.120063] tsc: Refined TSC clocksource calibration: 3502.622 MHz
[    1.120580] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x327cfdbd6b4,
max_idle_ns: 440795243113 ns

Third, fourth and fifth tries I got the virtcons_probe trace again.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net

[Richard W.M. Jones]

On Tue, Mar 21, 2017 at 04:02:19PM -0700, Adam Williamson wrote:
>  Well, first time it did that. Second time, I got this, and the boot
>  hung:
>  
>  [    0.937752] EXT4-fs (sda1): mounting ext2 file system using the ext4 subsystem
>  [    0.938362] general protection fault: 0000 [#1] SMP
>  [    0.938719] Modules linked in: kvm_intel kvm snd_pcsp snd_pcm ppdev irqbypass
> ghash_clmulni_intel snd_timer snd parport_pc ata_generic soundcore joydev pata_acpi
> parport i2c_piix4 serio_raw qemu_fw_cfg acpi_cpufreq tpm_tis tpm_tis_core tpm libcrc32c
> crc8 crc7 crc_itu_t virtio_pci virtio_mmio virtio_input virtio_balloon virtio_scsi nd_pmem
> nd_btt virtio_net virtio_crypto crypto_engine virtio_console virtio_rng virtio_blk
> virtio_ring virtio nfit crc32_generic crct10dif_pclmul crc32c_intel crc32_pclmul
>  [    0.939350] CPU: 0 PID: 457 Comm: mount Not tainted 4.11.0-0.rc2.git2.2.fc26.x86_64
> #1
>  [    0.939350] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.10.2-1.fc26
> 04/01/2014
>  [    0.939350] task: ffff94981d290000 task.stack: ffffae8780374000
>  [    0.939350] RIP: 0010:__kmalloc+0xa4/0x210
>  [    0.939350] RSP: 0018:ffffae8780377c30 EFLAGS: 00010246
>  [    0.939350] RAX: 2d326f6974726976 RBX: 00000000014000c0 RCX: 000000000000000a
>  [    0.939350] RDX: 0000000000000015 RSI: 0000000000000000 RDI: 000000000001cae0
>  [    0.939350] RBP: ffffae8780377c60 R08: ffff94981f01cae0 R09: ffff94981ec02280
>  [    0.939350] R10: 00000000000005ac R11: ffff94981b640400 R12: 2d326f6974726976
>  [    0.939350] R13: 00000000014000c0 R14: 0000000000002000 R15: ffff94981ec02280
>  [    0.939350] FS:  00007fe249b47480(0000) GS:ffff94981f000000(0000)
> knlGS:0000000000000000
>  [    0.939350] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>  [    0.939350] CR2: 000055ac86a6ac08 CR3: 000000001d329000 CR4: 00000000000406f0
>  [    0.939350] Call Trace:
>  [    0.939350]  ? mb_cache_create+0x75/0x110
>  [    0.939350]  mb_cache_create+0x75/0x110
>  [    0.939350]  ext4_xattr_create_cache+0x13/0x20
>  [    0.939350]  ext4_fill_super+0x1cad/0x3600
>  [    0.939350]  ? __kmalloc+0x1d1/0x210
>  [    0.939350]  ? snprintf+0x49/0x60
>  [    0.939350]  mount_bdev+0x17c/0x1b0
>  [    0.939350]  ? ext4_calculate_overhead+0x490/0x490
>  [    0.939350]  ext4_mount+0x15/0x20
>  [    0.939350]  mount_fs+0x32/0x160
>  [    0.939350]  ? __alloc_percpu+0x15/0x20
>  [    0.939350]  vfs_kern_mount.part.19+0x5d/0x120
>  [    0.939350]  do_mount+0x520/0xc30
>  [    0.939350]  ? _copy_from_user+0x4e/0x80
>  [    0.939350]  ? memdup_user+0x4f/0x80
>  [    0.939350]  SyS_mount+0x98/0xe0
>  [    0.939350]  entry_SYSCALL_64_fastpath+0x1a/0xa9
>  [    0.939350] RIP: 0033:0x7fe248bcc4fa
>  [    0.939350] RSP: 002b:00007ffef410b938 EFLAGS: 00000246 ORIG_RAX: 00000000000000a5
>  [    0.939350] RAX: ffffffffffffffda RBX: 00007fe249741522 RCX: 00007fe248bcc4fa
>  [    0.939350] RDX: 000055ac86a69c80 RSI: 000055ac86a62f50 RDI: 000055ac86a60240
>  [    0.939350] RBP: 00007fe249951184 R08: 0000000000000000 R09: 00007ffef410a838
>  [    0.939350] R10: 00000000c0ed0000 R11: 0000000000000246 R12: 000055ac86a60120
>  [    0.939350] R13: 00007ffef410bc48 R14: 0000000000000000 R15: 00000000ffffffff
>  [    0.939350] Code: 49 83 78 10 00 4d 8b 20 0f 84 03 01 00 00 4d 85 e4 0f 84 fa 00 00 00
> 49 63 41 20 49 8b 39 4c 01 e0 40 f6 c7 0f 0f 85 5f 01 00 00 <48> 8b 18 48 8d 4a 01
> 4c 89 e0 65 48 0f c7 0f 0f 94 c0 84 c0 74 
>  [    0.939350] RIP: __kmalloc+0xa4/0x210 RSP: ffffae8780377c30
>  [    0.958773] ---[ end trace 900f41ab5fa73f40 ]--- 
That's a (another, different) bug.  Does it happen often?  You
said one time in 6 ...

If it's repeatable, then I guess it's another candidate to go into BZ.

Unfortunately filing Fedora kernel BZs is IME not productive.  They
sit there forever until a "MASS BUG UPDATE" message is added to all
kernel bugs, and then they're closed, so really it's a waste of
everyone's time.  I only bothered to file 1430297 because I needed a
bug number to go into my spec file so I could remember why I disabled
the tests.

If I have time today I'll try to see if I can bisect the 1430297
issue.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-builder quickly builds VMs from scratch
http://libguestfs.org/virt-builder.1.html

[Justin Forbes]

On Wed, Mar 22, 2017 at 7:28 AM, Richard W.M. Jones <rjones(a)redhat.com&gt; wrote:
>  On Tue, Mar 21, 2017 at 04:02:19PM -0700, Adam Williamson wrote:
> > Well, first time it did that. Second time, I got this, and the boot
> > hung:
> >
> > [    0.937752] EXT4-fs (sda1): mounting ext2 file system using the ext4 subsystem
> > [    0.938362] general protection fault: 0000 [#1] SMP
> > [    0.938719] Modules linked in: kvm_intel kvm snd_pcsp snd_pcm ppdev irqbypass
> ghash_clmulni_intel snd_timer snd parport_pc ata_generic soundcore joydev pata_acpi
> parport i2c_piix4 serio_raw qemu_fw_cfg acpi_cpufreq tpm_tis tpm_tis_core tpm libcrc32c
> crc8 crc7 crc_itu_t virtio_pci virtio_mmio virtio_input virtio_balloon virtio_scsi nd_pmem
> nd_btt virtio_net virtio_crypto crypto_engine virtio_console virtio_rng virtio_blk
> virtio_ring virtio nfit crc32_generic crct10dif_pclmul crc32c_intel crc32_pclmul
> > [    0.939350] CPU: 0 PID: 457 Comm: mount Not tainted
> 4.11.0-0.rc2.git2.2.fc26.x86_64 #1
> > [    0.939350] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS
> 1.10.2-1.fc26 04/01/2014
> > [    0.939350] task: ffff94981d290000 task.stack: ffffae8780374000
> > [    0.939350] RIP: 0010:__kmalloc+0xa4/0x210
> > [    0.939350] RSP: 0018:ffffae8780377c30 EFLAGS: 00010246
> > [    0.939350] RAX: 2d326f6974726976 RBX: 00000000014000c0 RCX: 000000000000000a
> > [    0.939350] RDX: 0000000000000015 RSI: 0000000000000000 RDI: 000000000001cae0
> > [    0.939350] RBP: ffffae8780377c60 R08: ffff94981f01cae0 R09: ffff94981ec02280
> > [    0.939350] R10: 00000000000005ac R11: ffff94981b640400 R12: 2d326f6974726976
> > [    0.939350] R13: 00000000014000c0 R14: 0000000000002000 R15: ffff94981ec02280
> > [    0.939350] FS:  00007fe249b47480(0000) GS:ffff94981f000000(0000)
> knlGS:0000000000000000
> > [    0.939350] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > [    0.939350] CR2: 000055ac86a6ac08 CR3: 000000001d329000 CR4: 00000000000406f0
> > [    0.939350] Call Trace:
> > [    0.939350]  ? mb_cache_create+0x75/0x110
> > [    0.939350]  mb_cache_create+0x75/0x110
> > [    0.939350]  ext4_xattr_create_cache+0x13/0x20
> > [    0.939350]  ext4_fill_super+0x1cad/0x3600
> > [    0.939350]  ? __kmalloc+0x1d1/0x210
> > [    0.939350]  ? snprintf+0x49/0x60
> > [    0.939350]  mount_bdev+0x17c/0x1b0
> > [    0.939350]  ? ext4_calculate_overhead+0x490/0x490
> > [    0.939350]  ext4_mount+0x15/0x20
> > [    0.939350]  mount_fs+0x32/0x160
> > [    0.939350]  ? __alloc_percpu+0x15/0x20
> > [    0.939350]  vfs_kern_mount.part.19+0x5d/0x120
> > [    0.939350]  do_mount+0x520/0xc30
> > [    0.939350]  ? _copy_from_user+0x4e/0x80
> > [    0.939350]  ? memdup_user+0x4f/0x80
> > [    0.939350]  SyS_mount+0x98/0xe0
> > [    0.939350]  entry_SYSCALL_64_fastpath+0x1a/0xa9
> > [    0.939350] RIP: 0033:0x7fe248bcc4fa
> > [    0.939350] RSP: 002b:00007ffef410b938 EFLAGS: 00000246 ORIG_RAX:
> 00000000000000a5
> > [    0.939350] RAX: ffffffffffffffda RBX: 00007fe249741522 RCX: 00007fe248bcc4fa
> > [    0.939350] RDX: 000055ac86a69c80 RSI: 000055ac86a62f50 RDI: 000055ac86a60240
> > [    0.939350] RBP: 00007fe249951184 R08: 0000000000000000 R09: 00007ffef410a838
> > [    0.939350] R10: 00000000c0ed0000 R11: 0000000000000246 R12: 000055ac86a60120
> > [    0.939350] R13: 00007ffef410bc48 R14: 0000000000000000 R15: 00000000ffffffff
> > [    0.939350] Code: 49 83 78 10 00 4d 8b 20 0f 84 03 01 00 00 4d 85 e4 0f 84 fa 00
> 00 00 49 63 41 20 49 8b 39 4c 01 e0 40 f6 c7 0f 0f 85 5f 01 00 00 <48> 8b 18 48 8d
> 4a 01 4c 89 e0 65 48 0f c7 0f 0f 94 c0 84 c0 74
> > [    0.939350] RIP: __kmalloc+0xa4/0x210 RSP: ffffae8780377c30
> > [    0.958773] ---[ end trace 900f41ab5fa73f40 ]---
>
>  That's a (another, different) bug.  Does it happen often?  You
>  said one time in 6 ...
>
>  If it's repeatable, then I guess it's another candidate to go into BZ.
>
>  Unfortunately filing Fedora kernel BZs is IME not productive.  They
>  sit there forever until a "MASS BUG UPDATE" message is added to all
>  kernel bugs, and then they're closed, so really it's a waste of
>  everyone's time.  I only bothered to file 1430297 because I needed a
>  bug number to go into my spec file so I could remember why I disabled
>  the tests.
>
>  If I have time today I'll try to see if I can bisect the 1430297
>  issue.
>  
It isn't a waste of time, but there are only 2 of us.  We do look at
everything that comes in, but there is only so much that 2 people can
actually chase down in a week. Right now Laura is trying to work
through F26 blockers and I am chasing down the CRDA issue in 4.10
(that will be a blocker for F26 release if it isn't solved).

Justin

[Andrew Lutomirski]

On Tue, Mar 21, 2017 at 3:59 PM, Adam Williamson
<adamwill(a)fedoraproject.org&gt; wrote:
>  On Tue, 2017-03-21 at 20:48 +0000, Richard W.M. Jones wrote:
> > On Tue, Mar 21, 2017 at 12:58:06PM -0700, Adam Williamson wrote:
> > > On Tue, 2017-03-21 at 19:40 +0000, Richard W.M. Jones wrote:
> > > > https://bugzilla.redhat.com/show_bug.cgi?id=1430297
> > > >
> > > > This has been going on for a few weeks with no attention.  It prevents
> > > > anyone from running libguestfs on Fedora 26.
> > > >
> > > > It's (IMHO) pretty serious for anyone using virtualization:
> "kernel
> > > > doesn't boot on qemu" should be a blocker for a Fedora release.
> > > >
> > > > Can anyone see what's going on?
> > >
> > > The earlier case looks like
> > > https://bugzilla.redhat.com/show_bug.cgi?id=1430043 , which was fixed
> > > in rc2. I'm not sure about the latter case, though Thorsten Leemhuis
> > > reported intermittent crash / hang during boot with rc2 as:
> > >
> > > https://bugzilla.kernel.org/show_bug.cgi?id=194911
> > >
> > > Since rc2 I've seen occasional failures which are probably the same as
> > > the ones Thorsten was seeing, but usually it works OK. I run 26-on-26
> > > VMs via virt-manager quite a lot, they certainly don't fail every time.
> >
> > Does libguestfs-test-tool work for you in F26?
>
>  Hum, interesting results:
>
>  If I run it as a regular user, it works.
>  If I run it as root as usual, it gives me a libvirt error, some
>  permission issue.
>  If I run it as root with LIBGUESTFS_BACKEND=direct , it hits the same
>  kernel trace you hit. 
Off the top of my head, I'd guess that the failing cases are using
vhost on the host.  I've never used vhost in my life, so I have no
clue about it.

--Andy