GCC crash on Fedora Rawhide armv7

[Richard W.M. Jones]

  gcc -Ilibqemuutil.a.p -I. -I.. -Isubprojects/libvhost-user
-I../subprojects/libvhost-user -Iqapi -Itrace -Iui -Iui/shader -I/usr/include/glib-2.0
-I/usr/lib/glib-2.0/include -I/usr/include/sysprof-4 -I/usr/include/libmount
-I/usr/include/blkid -I/usr/include/gio-unix-2.0 -I/usr/include/p11-kit-1
-I/usr/include/pixman-1 -fdiagnostics-color=auto -Wall -Winvalid-pch -std=gnu11 -O2 -g
-isystem /builddir/build/BUILD/qemu-6.1.0/linux-headers -isystem linux-headers -iquote .
-iquote /builddir/build/BUILD/qemu-6.1.0 -iquote /builddir/build/BUILD/qemu-6.1.0/include
-iquote /builddir/build/BUILD/qemu-6.1.0/disas/libvixl -iquote
/builddir/build/BUILD/qemu-6.1.0/tcg/arm -pthread -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2
-D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes
-Wredundant-decls -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing
-fno-common -fwrapv -O2 -fexceptions -g -grecord-gcc-switches -pipe -Wall
-Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS
-specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -march=armv7-a
-mfpu=vfpv3-d16 -mtune=generic-armv7-a -mabi=aapcs-linux -mfloat-abi=hard
-Wold-style-declaration -Wold-style-definition -Wtype-limits -Wformat-security
-Wformat-y2k -Winit-self -Wignored-qualifiers -Wempty-body -Wnested-externs -Wendif-labels
-Wexpansion-to-defined -Wimplicit-fallthrough=2 -Wno-missing-include-dirs
-Wno-shift-negative-value -Wno-psabi -fstack-protector-strong -DSTAP_SDT_V2 -fPIE -MD -MQ
libqemuutil.a.p/util_vfio-helpers.c.o -MF libqemuutil.a.p/util_vfio-helpers.c.o.d -o
libqemuutil.a.p/util_vfio-helpers.c.o -c ../util/vfio-helpers.c
  during RTL pass: mach
  ../util/vfio-helpers.c: In function 'qemu_vfio_open_pci':
  ../util/vfio-helpers.c:523:1: internal compiler error: in create_fix_barrier, at
config/arm/arm.c:17891
    523 | }
        | ^
  Please submit a full bug report,
  with preprocessed source if appropriate.
  See <http://bugzilla.redhat.com/bugzilla> for instructions.
  Preprocessed source stored into /tmp/ccOFlWwZ.out file, please attach this to your
bugreport.

Unfortunately the crash happened in Koji:
https://koji.fedoraproject.org/koji/taskinfo?taskID=79160330

I'm not able to reproduce it locally, which is strange.  I've got an
armv7 guest which has the same version of gcc, binutils and annobin,
yet it does not crash compiling the same file.

I wonder if anyone can manage to reproduce this and capture the
/tmp/*.out file?  In theory all you need to do is build qemu from
dist-git on armv7.

Alternatively, can anyone think of a reason why this crash would
happen in Koji but not in an armv7 guest (on aarch64 host)?

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-builder quickly builds VMs from scratch
http://libguestfs.org/virt-builder.1.html

[Kaleb Keithley]

On Mon, Nov 22, 2021 at 12:56 PM Richard W.M. Jones <rjones(a)redhat.com&gt;
wrote:
...

>   libqemuutil.a.p/util_vfio-helpers.c.o -c ../util/vfio-helpers.c
>    during RTL pass: mach
>    ../util/vfio-helpers.c: In function 'qemu_vfio_open_pci':
>    ../util/vfio-helpers.c:523:1: internal compiler error: in
>  create_fix_barrier, at config/arm/arm.c:17891
>      523 | }
>          | ^
>    Please submit a full bug report,
>    with preprocessed source if appropriate.
>    See <http://bugzilla.redhat.com/bugzilla> for instructions.
>    Preprocessed source stored into /tmp/ccOFlWwZ.out file, please attach
>  this to your bugreport.
>
>  
FWIW a few days ago I received a "started to fail in F35" email for
ceph.
I ran a couple of scratch builds on f35 and f36/rawhide.  The f35 build ran
successfully to completion, the f36 build failed with an internal compiler
error.

Since then I've received another "started to fail in rawhide" email for
ceph with a similar internal compiler error, i.e.: internal compiler error:
maximum number of generated reload insns per insn achieved (90)

Both f35 and f36 have the same NVR of gcc.

And mine are all on x86_64. I'm wondering if there isn't a sick x86_64
builder.

-- 

Kaleb

qemu ExcludeArch %{arm} (was: Re: GCC crash on Fedora Rawhide armv7)

[Richard W.M. Jones]

[Adding qemu maintainers]

On Mon, Nov 22, 2021 at 05:55:06PM +0000, Richard W.M. Jones wrote:
>  
>    gcc -Ilibqemuutil.a.p -I. -I.. -Isubprojects/libvhost-user
> -I../subprojects/libvhost-user -Iqapi -Itrace -Iui -Iui/shader -I/usr/include/glib-2.0
> -I/usr/lib/glib-2.0/include -I/usr/include/sysprof-4 -I/usr/include/libmount
> -I/usr/include/blkid -I/usr/include/gio-unix-2.0 -I/usr/include/p11-kit-1
> -I/usr/include/pixman-1 -fdiagnostics-color=auto -Wall -Winvalid-pch -std=gnu11 -O2 -g
> -isystem /builddir/build/BUILD/qemu-6.1.0/linux-headers -isystem linux-headers -iquote .
> -iquote /builddir/build/BUILD/qemu-6.1.0 -iquote /builddir/build/BUILD/qemu-6.1.0/include
> -iquote /builddir/build/BUILD/qemu-6.1.0/disas/libvixl -iquote
> /builddir/build/BUILD/qemu-6.1.0/tcg/arm -pthread -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2
> -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes
> -Wredundant-decls -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing
> -fno-common -fwrapv -O2 -fexceptions -g -grecord-gcc-switches -pipe -Wall
> -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS
> -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -march=armv7-a
> -mfpu=vfpv3-d16 -mtune=generic-armv7-a -mabi=aapcs-linux -mfloat-abi=hard
> -Wold-style-declaration -Wold-style-definition -Wtype-limits -Wformat-security
> -Wformat-y2k -Winit-self -Wignored-qualifiers -Wempty-body -Wnested-externs -Wendif-labels
> -Wexpansion-to-defined -Wimplicit-fallthrough=2 -Wno-missing-include-dirs
> -Wno-shift-negative-value -Wno-psabi -fstack-protector-strong -DSTAP_SDT_V2 -fPIE -MD -MQ
> libqemuutil.a.p/util_vfio-helpers.c.o -MF libqemuutil.a.p/util_vfio-helpers.c.o.d -o
> libqemuutil.a.p/util_vfio-helpers.c.o -c ../util/vfio-helpers.c
>    during RTL pass: mach
>    ../util/vfio-helpers.c: In function 'qemu_vfio_open_pci':
>    ../util/vfio-helpers.c:523:1: internal compiler error: in create_fix_barrier, at
> config/arm/arm.c:17891
>      523 | }
>          | ^
>    Please submit a full bug report,
>    with preprocessed source if appropriate.
>    See <http://bugzilla.redhat.com/bugzilla> for instructions.
>    Preprocessed source stored into /tmp/ccOFlWwZ.out file, please attach this to your
> bugreport.
>  
>  Unfortunately the crash happened in Koji:
>  https://koji.fedoraproject.org/koji/taskinfo?taskID=79160330
>  
>  I'm not able to reproduce it locally, which is strange.  I've got an
>  armv7 guest which has the same version of gcc, binutils and annobin,
>  yet it does not crash compiling the same file.
>  
>  I wonder if anyone can manage to reproduce this and capture the
>  /tmp/*.out file?  In theory all you need to do is build qemu from
>  dist-git on armv7.
>  
>  Alternatively, can anyone think of a reason why this crash would
>  happen in Koji but not in an armv7 guest (on aarch64 host)? 
I'm still unable to reproduce this bug locally.

But as a result, qemu is currently ExcludeArch %{arm}.  This means
that various other packages will start to be affected (I had to add
the same to virt-v2v in Fedora just now).

I don't know how to proceed here - it seems like it's a GCC bug, so
simply reverting the ExcludeArch commit will NOT help us.

I would favour permanently excluding armv7, as it is unlikely anyone
is really running VMs on 32 bit ARM, but obviously that will have
knock-on effects through the rest of the stack.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-p2v converts physical machines to virtual machines.  Boot with a
live CD or over the network (PXE) and turn machines into KVM guests.
http://libguestfs.org/virt-v2v

[Dan Horák]

On Mon, 22 Nov 2021 17:55:06 +0000
"Richard W.M. Jones" <rjones(a)redhat.com&gt; wrote:

>  
>    gcc -Ilibqemuutil.a.p -I. -I.. -Isubprojects/libvhost-user
> -I../subprojects/libvhost-user -Iqapi -Itrace -Iui -Iui/shader -I/usr/include/glib-2.0
> -I/usr/lib/glib-2.0/include -I/usr/include/sysprof-4 -I/usr/include/libmount
> -I/usr/include/blkid -I/usr/include/gio-unix-2.0 -I/usr/include/p11-kit-1
> -I/usr/include/pixman-1 -fdiagnostics-color=auto -Wall -Winvalid-pch -std=gnu11 -O2 -g
> -isystem /builddir/build/BUILD/qemu-6.1.0/linux-headers -isystem linux-headers -iquote .
> -iquote /builddir/build/BUILD/qemu-6.1.0 -iquote /builddir/build/BUILD/qemu-6.1.0/include
> -iquote /builddir/build/BUILD/qemu-6.1.0/disas/libvixl -iquote
> /builddir/build/BUILD/qemu-6.1.0/tcg/arm -pthread -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2
> -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes
> -Wredundant-decls -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing
> -fno-common -fwrapv -O2 -fexceptions -g -grecord-gcc-switches -pipe -Wall
> -Werror=format-security -Wp,-D_FORTIF 
 Y_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS
-specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -march=armv7-a
-mfpu=vfpv3-d16 -mtune=generic-armv7-a -mabi=aapcs-linux -mfloat-abi=hard
-Wold-style-declaration -Wold-style-definition -Wtype-limits -Wformat-security
-Wformat-y2k -Winit-self -Wignored-qualifiers -Wempty-body -Wnested-externs -Wendif-labels
-Wexpansion-to-defined -Wimplicit-fallthrough=2 -Wno-missing-include-dirs
-Wno-shift-negative-value -Wno-psabi -fstack-protector-strong -DSTAP_SDT_V2 -fPIE -MD -MQ
libqemuutil.a.p/util_vfio-helpers.c.o -MF libqemuutil.a.p/util_vfio-helpers.c.o.d -o
libqemuutil.a.p/util_vfio-helpers.c.o -c ../util/vfio-helpers.c
>    during RTL pass: mach
>    ../util/vfio-helpers.c: In function 'qemu_vfio_open_pci':
>    ../util/vfio-helpers.c:523:1: internal compiler error: in create_fix_barrier, at
> config/arm/arm.c:17891
>      523 | }
>          | ^
>    Please submit a full bug report,
>    with preprocessed source if appropriate.
>    See <http://bugzilla.redhat.com/bugzilla> for instructions.
>    Preprocessed source stored into /tmp/ccOFlWwZ.out file, please attach this to your
> bugreport.
>  
>  Unfortunately the crash happened in Koji:
>  https://koji.fedoraproject.org/koji/taskinfo?taskID=79160330
>  
>  I'm not able to reproduce it locally, which is strange.  I've got an
>  armv7 guest which has the same version of gcc, binutils and annobin,
>  yet it does not crash compiling the same file.
>  
>  I wonder if anyone can manage to reproduce this and capture the
>  /tmp/*.out file?  In theory all you need to do is build qemu from
>  dist-git on armv7. 
I was able to get the preprocessed file from the builder, it's
available at https://fedora.danny.cz/tmp/ccOFlWwZ.out now

Do I get it right, that you have reproduced the issue in koji at least
twice (same error, same file)?

>  
>  Alternatively, can anyone think of a reason why this crash would
>  happen in Koji but not in an armv7 guest (on aarch64 host)? 
it could be the difference in the host HW (or in the hypervisor), but
that's a wild idea :-)


		Dan

[Jakub Jelinek]

On Tue, Nov 23, 2021 at 05:19:09PM +0100, Dan Horák wrote:
>  >         | ^
>  >   Please submit a full bug report,
>  >   with preprocessed source if appropriate.
>  >   See <http://bugzilla.redhat.com/bugzilla> for instructions.
>  >   Preprocessed source stored into /tmp/ccOFlWwZ.out file, please attach this to your
> bugreport.
>  > 
>  > Unfortunately the crash happened in Koji:
>  > https://koji.fedoraproject.org/koji/taskinfo?taskID=79160330
>  > 
>  > I'm not able to reproduce it locally, which is strange.  I've got an
>  > armv7 guest which has the same version of gcc, binutils and annobin,
>  > yet it does not crash compiling the same file.
>  > 
>  > I wonder if anyone can manage to reproduce this and capture the
>  > /tmp/*.out file?  In theory all you need to do is build qemu from
>  > dist-git on armv7.
>  
>  I was able to get the preprocessed file from the builder, it's
>  available at https://fedora.danny.cz/tmp/ccOFlWwZ.out now 
Reproduced with current GCC trunk, will reduce/bisect it tomorrow and file an
upstream bug.

	Jakub

[Richard W.M. Jones]

On Tue, Nov 23, 2021 at 05:27:36PM +0100, Jakub Jelinek wrote:
>  On Tue, Nov 23, 2021 at 05:19:09PM +0100, Dan Horák wrote:
>  > >         | ^
>  > >   Please submit a full bug report,
>  > >   with preprocessed source if appropriate.
>  > >   See <http://bugzilla.redhat.com/bugzilla> for instructions.
>  > >   Preprocessed source stored into /tmp/ccOFlWwZ.out file, please attach this to
> your bugreport.
>  > > 
>  > > Unfortunately the crash happened in Koji:
>  > > https://koji.fedoraproject.org/koji/taskinfo?taskID=79160330
>  > > 
>  > > I'm not able to reproduce it locally, which is strange.  I've got an
>  > > armv7 guest which has the same version of gcc, binutils and annobin,
>  > > yet it does not crash compiling the same file.
>  > > 
>  > > I wonder if anyone can manage to reproduce this and capture the
>  > > /tmp/*.out file?  In theory all you need to do is build qemu from
>  > > dist-git on armv7.
>  > 
>  > I was able to get the preprocessed file from the builder, it's
>  > available at https://fedora.danny.cz/tmp/ccOFlWwZ.out now
>  
>  Reproduced with current GCC trunk, will reduce/bisect it tomorrow and file an
>  upstream bug. 
Thanks so much to both of you.  Could you please CC me on the upstream
bug once you've filed it so I can track this for qemu.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-top is 'top' for virtual machines.  Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://people.redhat.com/~rjones/virt-top

[Richard W.M. Jones]

For the record:
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=103395

Jakub came up with a very cute reproducer :-)

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-top is 'top' for virtual machines.  Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://people.redhat.com/~rjones/virt-top

qemu ExcludeArch %{arm} (was: Re: GCC crash on Fedora Rawhide armv7)

[Richard W.M. Jones]

I've removed the ExcludeArch %{arm} now, and added this temporary
patch to make things compile on armv7:

https://src.fedoraproject.org/rpms/qemu/c/f9f5de99949d58a308ae03ab2a0f4fd...
https://koji.fedoraproject.org/koji/taskinfo?taskID=79236809

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-builder quickly builds VMs from scratch
http://libguestfs.org/virt-builder.1.html