I've been talking to Richard Jones about this privately, and he suggested that a message to fedora-devel could be helpful.
I've been talking for weeks about updating the entire coq stack, badgering people into swapping reviews with me, etc. Today I finally had all the bits in place to do the updates. I built the whole stack for Rawhide with no issues.
The same builds need to happen for F32, where parts of the stack currently have broken deps. But I can't get coq to build successfully, even though it did in Rawhide. It went like this:
Attempt 1: the x86_64 build segfaulted while running "bin/coqc -coqlib . -q -native-compiler yes theories/FSets/FMapFullAVL.v", where coqc is a binary built from the coq sources. Coq is a theorem prover. The coqc run is essentially checking that a series of proof steps indeed prove a theorem. The "-native-compiler yes" part means that the OCaml compiler will be invoked to generate native code for parts of this process.
I was puzzled by the segfault. I decided to see if the segfault is deterministic, so I launched...
Attempt 2: the x86_64 build succeeded. The s390x build failed with an illegal instruction error while running *the same command*.
Attempt 3: The s390x build again failed with an illegal instruction error while running the same command, and aarch64 segfaulted while running the same command. All other arches succeeded. I saved the URL for this one:
https://koji.fedoraproject.org/koji/taskinfo?taskID=42768075
Either I got lucky when I launched the Rawhide build, or something is fundamentally different between F32 and Rawhide. Clearly something nondeterministic is at work. I have been unable to reproduce this failure in mock so far, but will keep trying.
If anybody has any ideas, I am all ears. Thank you,
Le Wed, 25 Mar 2020 16:58:33 -0600, Jerry James loganjerry@gmail.com a écrit :
Either I got lucky when I launched the Rawhide build, or something is fundamentally different between F32 and Rawhide. Clearly something nondeterministic is at work. I have been unable to reproduce this failure in mock so far, but will keep trying.
If anybody has any ideas, I am all ears. Thank you,
Well... It ring a bell for me, because I kind of seeing what seems to be weird bugs in F32... I am kind of suspecting a gcc bug giving some weird things... but I am not a developer, know a bit of assembler but not enough to make the link between C code and assembler.
Anyway, one hour a go, I got a segment violation error while doing updates with dnfdragora... and it seems that the code just jump to a low address outside of the program: rh bug #1817269
The other bug that make me suspect something might be wrong with gcc was a bug that make a segment violation (seems to access a bit lower address than a .png font file) in a game package that have not been updated in many years as far as I know: rh bug #1816471 but in that case, frankly the C code seems a bit hard for me to take time to try to understand it.
What puzzle me in this case, is why a very old code would have stop to work... that's why I came to the idea that maybe this is the generated code itself that changed.
... yeah, I know, might have no link at all with your Coq build... I don't know.
On Wed, Mar 25, 2020 at 8:57 PM Paul Dufresne via devel devel@lists.fedoraproject.org wrote:
... yeah, I know, might have no link at all with your Coq build... I don't know.
Maybe ... but that sure is suspicious. I recall hearing that OpenJDK is also suffering from weird segfaults. Hmmmm.....
Thanks for the input, Paul. I'll keep investigating.
On Wed, Mar 25, 2020 at 4:58 PM Jerry James loganjerry@gmail.com wrote:
Either I got lucky when I launched the Rawhide build, or something is fundamentally different between F32 and Rawhide. Clearly something nondeterministic is at work. I have been unable to reproduce this failure in mock so far, but will keep trying.
I did a scratch build in Rawhide, just to see if this morning's successful build was a fluke. It succeeded on all architectures:
https://koji.fedoraproject.org/koji/taskinfo?taskID=42770190
So far, 2 out of 2 Rawhide builds succeeded, and 3 out of 3 F32 builds ended with either a segfault or a bad instruction on at least 1 architecture. Here are the side-by-side differences in the packages in the aarch64 buildroots (since aarch64 was one of the failing architectures on my last attempt), Rawhide on the left, F32 on the right. You'll want to view this with a fixed-width font.
adwaita-cursor-theme-3.36.0-1.fc33.noarch adwaita-cursor-theme-3.36.0-1.fc32.noarch adwaita-icon-theme-3.36.0-1.fc33.noarch adwaita-icon-theme-3.36.0-1.fc32.noarch annobin-9.12-3.fc33.aarch64 annobin-9.06-4.fc32.aarch64 at-spi2-atk-2.34.2-1.fc33.aarch64 at-spi2-atk-2.34.2-1.fc32.aarch64 at-spi2-atk-devel-2.34.2-1.fc33.aarch64 at-spi2-atk-devel-2.34.2-1.fc32.aarch64 at-spi2-core-2.36.0-1.fc33.aarch64 at-spi2-core-2.36.0-1.fc32.aarch64 at-spi2-core-devel-2.36.0-1.fc33.aarch64 at-spi2-core-devel-2.36.0-1.fc32.aarch64 audit-libs-3.0-0.19.20191104git1c2f876.fc33.aarch64 audit-libs-3.0-0.18.20191104git1c2f876.fc32.aarch64 avahi-libs-0.8-2.fc33.aarch64 avahi-libs-0.7-23.fc32.aarch64 binutils-2.34-2.fc33.aarch64 binutils-2.34-2.fc32.aarch64 binutils-gold-2.34-2.fc33.aarch64 binutils-gold-2.34-2.fc32.aarch64 cmake-filesystem-3.17.0-1.fc33.aarch64 cmake-filesystem-3.16.4-1.fc32.aarch64 coreutils-8.32-3.fc33.aarch64 coreutils-8.31-10.fc32.aarch64 coreutils-common-8.32-3.fc33.aarch64 coreutils-common-8.31-10.fc32.aarch64 cpp-10.0.1-0.9.fc33.aarch64 cpp-10.0.1-0.9.fc32.aarch64 crypto-policies-20200312-1.git3ae59d2.fc33.noarch crypto-policies-20191128-5.gitcd267a5.fc32.noarch cryptsetup-libs-2.3.1-1.fc33.aarch64 cryptsetup-libs-2.3.0-1.fc32.aarch64 curl-7.69.0-2.fc33.aarch64 curl-7.69.1-1.fc32.aarch64 cyrus-sasl-lib-2.1.27-4.fc33.aarch64 cyrus-sasl-lib-2.1.27-3.fc32.aarch64 dbus-1:1.12.16-5.fc33.aarch64 dbus-1:1.12.16-4.fc32.aarch64 dbus-broker-22-1.fc33.aarch64 dbus-broker-22-1.fc32.aarch64 dbus-common-1:1.12.16-5.fc33.noarch dbus-common-1:1.12.16-4.fc32.noarch dbus-devel-1:1.12.16-5.fc33.aarch64 dbus-devel-1:1.12.16-4.fc32.aarch64 dbus-libs-1:1.12.16-5.fc33.aarch64 dbus-libs-1:1.12.16-4.fc32.aarch64 dejavu-sans-fonts-2.37-7.fc33.noarch dejavu-sans-fonts-2.37-7.fc32.noarch fedora-gpg-keys-33-0.3.noarch fedora-gpg-keys-32-0.7.noarch fedora-release-33-0.3.noarch fedora-release-32-0.8.noarch fedora-release-common-33-0.3.noarch fedora-release-common-32-0.8.noarch fedora-repos-33-0.3.noarch fedora-repos-32-0.7.noarch fedora-repos-rawhide-33-0.3.noarch file-5.38-3.fc33.aarch64 file-5.38-2.fc32.aarch64 file-libs-5.38-3.fc33.aarch64 file-libs-5.38-2.fc32.aarch64 fontconfig-2.13.92-8.fc33.aarch64 fontconfig-2.13.92-8.fc32.aarch64 fontconfig-devel-2.13.92-8.fc33.aarch64 fontconfig-devel-2.13.92-8.fc32.aarch64 fonts-filesystem-2.0.3-1.fc33.noarch fonts-filesystem-2.0.3-1.fc32.noarch fonts-srpm-macros-2.0.3-1.fc33.noarch fonts-srpm-macros-2.0.3-1.fc32.noarch fribidi-1.0.9-1.fc33.aarch64 fribidi-1.0.9-1.fc32.aarch64 fribidi-devel-1.0.9-1.fc33.aarch64 fribidi-devel-1.0.9-1.fc32.aarch64 gcc-10.0.1-0.9.fc33.aarch64 gcc-10.0.1-0.9.fc32.aarch64 gdb-minimal-9.1-4.fc33.aarch64 gdb-minimal-9.1-3.fc32.aarch64 ghostscript-9.50-1.fc33.aarch64 ghostscript-9.50-1.fc32.aarch64 ghostscript-core-9.50-1.fc33.aarch64 ghostscript-core-9.50-1.fc32.aarch64 ghostscript-tools-fonts-9.50-1.fc33.aarch64 ghostscript-tools-fonts-9.50-1.fc32.aarch64 ghostscript-tools-printing-9.50-1.fc33.aarch64 ghostscript-tools-printing-9.50-1.fc32.aarch64 glib-networking-2.64.0-1.fc33.aarch64 glib-networking-2.64.0-1.fc32.aarch64 glib2-2.64.1-1.fc33.aarch64 glib2-2.64.1-1.fc32.aarch64 glib2-devel-2.64.1-1.fc33.aarch64 glib2-devel-2.64.1-1.fc32.aarch64 glibc-2.31.9000-4.fc33.aarch64 glibc-2.31-1.fc32.aarch64 glibc-common-2.31.9000-4.fc33.aarch64 glibc-common-2.31-1.fc32.aarch64 glibc-devel-2.31.9000-4.fc33.aarch64 glibc-devel-2.31-1.fc32.aarch64 glibc-headers-2.31.9000-4.fc33.aarch64 glibc-headers-2.31-1.fc32.aarch64 glibc-minimal-langpack-2.31.9000-4.fc33.aarch64 glibc-minimal-langpack-2.31-1.fc32.aarch64 gmp-1:6.2.0-1.fc33.aarch64 gmp-1:6.1.2-13.fc32.aarch64 google-droid-sans-fonts-20200215-3.fc33.noarch google-droid-sans-fonts-20200215-3.fc32.noarch google-roboto-slab-fonts-1.100263-0.12.20150923git.fc33.noarch google-roboto-slab-fonts-1.100263-0.11.20150923git.fc32.noarch groff-base-1.22.3-24.fc33.aarch64 groff-base-1.22.3-21.fc32.aarch64 gsettings-desktop-schemas-3.36.0-1.fc33.aarch64 gsettings-desktop-schemas-3.36.0-1.fc32.aarch64 gtk-update-icon-cache-3.24.14-1.fc33.aarch64 gtk-update-icon-cache-3.24.14-1.fc32.aarch64 gtk3-3.24.14-1.fc33.aarch64 gtk3-3.24.14-1.fc32.aarch64 gtk3-devel-3.24.14-1.fc33.aarch64 gtk3-devel-3.24.14-1.fc32.aarch64
guile-5:2.0.14-19.fc32.aarch64 harfbuzz-2.6.4-4.fc33.aarch64 harfbuzz-2.6.4-3.fc32.aarch64 harfbuzz-devel-2.6.4-4.fc33.aarch64 harfbuzz-devel-2.6.4-3.fc32.aarch64 harfbuzz-icu-2.6.4-4.fc33.aarch64 harfbuzz-icu-2.6.4-3.fc32.aarch64 hevea-2.33-2.fc33.aarch64 hevea-2.33-2.fc32.1.aarch64 info-6.7-7.fc33.aarch64 info-6.7-6.fc32.aarch64 iptables-libs-1.8.4-7.fc33.aarch64 iptables-libs-1.8.4-7.fc32.aarch64 java-1.8.0-openjdk-headless-1:1.8.0.242.b08-2.fc33.aarch64 java-1.8.0-openjdk-headless-1:1.8.0.242.b08-1.fc32.aarch64 json-c-0.13.1-9.fc33.aarch64 json-c-0.13.1-9.fc32.aarch64 json-glib-1.4.4-4.fc32.aarch64 json-glib-1.4.4-4.fc32.aarch64 jsonp-1.0.4-11.fc32.noarch jsonp-1.0.4-11.fc32.noarch kernel-headers-5.6.0-0.rc7.git0.1.fc33.aarch64 kernel-headers-5.6.0-0.rc7.git0.1.fc32.aarch64 kmod-libs-27-2.fc33.aarch64 kmod-libs-27-1.fc32.aarch64 krb5-libs-1.18-7.fc33.aarch64 krb5-libs-1.18-1.fc32.aarch64 langpacks-core-font-en-3.0-3.fc33.noarch langpacks-core-font-en-3.0-3.fc32.noarch latexmk-4.69-1.fc33.noarch latexmk-4.69-1.fc32.noarch libarchive-3.4.2-1.fc33.aarch64 libarchive-3.4.2-1.fc32.aarch64 libasan-10.0.1-0.9.fc33.aarch64 libasan-10.0.1-0.9.fc32.aarch64 libatomic-10.0.1-0.9.fc33.aarch64 libatomic-10.0.1-0.9.fc32.aarch64 libblkid-2.35.1-7.fc33.aarch64 libblkid-2.35.1-4.fc32.aarch64 libblkid-devel-2.35.1-7.fc33.aarch64 libblkid-devel-2.35.1-4.fc32.aarch64 libcom_err-1.45.6-1.fc33.aarch64 libcom_err-1.45.5-3.fc32.aarch64 libcurl-7.69.0-2.fc33.aarch64 libcurl-7.69.1-1.fc32.aarch64 libevent-2.1.8-8.fc32.aarch64 libfdisk-2.35.1-7.fc33.aarch64 libfdisk-2.35.1-4.fc32.aarch64 libgcc-10.0.1-0.9.fc33.aarch64 libgcc-10.0.1-0.9.fc32.aarch64 libgfortran-10.0.1-0.9.fc33.aarch64 libgfortran-10.0.1-0.9.fc32.aarch64 libglvnd-1:1.3.1-1.fc33.aarch64 libglvnd-1:1.3.1-1.fc32.aarch64 libglvnd-core-devel-1:1.3.1-1.fc33.aarch64 libglvnd-core-devel-1:1.3.1-1.fc32.aarch64 libglvnd-devel-1:1.3.1-1.fc33.aarch64 libglvnd-devel-1:1.3.1-1.fc32.aarch64 libglvnd-egl-1:1.3.1-1.fc33.aarch64 libglvnd-egl-1:1.3.1-1.fc32.aarch64 libglvnd-gles-1:1.3.1-1.fc33.aarch64 libglvnd-gles-1:1.3.1-1.fc32.aarch64 libglvnd-glx-1:1.3.1-1.fc33.aarch64 libglvnd-glx-1:1.3.1-1.fc32.aarch64 libglvnd-opengl-1:1.3.1-1.fc33.aarch64 libglvnd-opengl-1:1.3.1-1.fc32.aarch64 libgomp-10.0.1-0.9.fc33.aarch64 libgomp-10.0.1-0.9.fc32.aarch64 libgpg-error-1.37-1.fc33.aarch64 libgpg-error-1.36-3.fc32.aarch64 libgs-9.50-1.fc33.aarch64 libgs-9.50-1.fc32.aarch64 libgusb-0.3.4-1.fc33.aarch64 libgusb-0.3.4-1.fc32.aarch64 libmount-2.35.1-7.fc33.aarch64 libmount-2.35.1-4.fc32.aarch64 libmount-devel-2.35.1-7.fc33.aarch64 libmount-devel-2.35.1-4.fc32.aarch64 libpcap-14:1.9.1-3.fc33.aarch64 libpcap-14:1.9.1-3.fc32.aarch64 libpq-12.2-1.fc33.aarch64 libpq-12.2-1.fc32.aarch64 libproxy-0.4.15-18.fc33.aarch64 libproxy-0.4.15-17.fc32.aarch64 libselinux-3.0-4.fc33.aarch64 libselinux-3.0-3.fc32.aarch64 libselinux-devel-3.0-4.fc33.aarch64 libselinux-devel-3.0-3.fc32.aarch64 libsemanage-3.0-3.fc32.aarch64 libsemanage-3.0-3.fc32.aarch64 libsmartcols-2.35.1-7.fc33.aarch64 libsmartcols-2.35.1-4.fc32.aarch64 libsoup-2.70.0-1.fc33.aarch64 libsoup-2.70.0-1.fc32.aarch64 libstdc++-10.0.1-0.9.fc33.aarch64 libstdc++-10.0.1-0.9.fc32.aarch64 libtirpc-1.2.5-1.rc2.fc33.aarch64 libtirpc-1.2.5-1.rc2.fc32.aarch64 libubsan-10.0.1-0.9.fc33.aarch64 libubsan-10.0.1-0.9.fc32.aarch64 libuuid-2.35.1-7.fc33.aarch64 libuuid-2.35.1-4.fc32.aarch64 libwayland-client-1.18.0-1.fc33.aarch64 libwayland-client-1.18.0-1.fc32.aarch64 libwayland-cursor-1.18.0-1.fc33.aarch64 libwayland-cursor-1.18.0-1.fc32.aarch64 libwayland-egl-1.18.0-1.fc33.aarch64 libwayland-egl-1.18.0-1.fc32.aarch64 libwayland-server-1.18.0-1.fc33.aarch64 libwayland-server-1.18.0-1.fc32.aarch64 libxcrypt-4.4.15-1.fc33.aarch64 libxcrypt-4.4.15-1.fc32.aarch64 libxcrypt-devel-4.4.15-1.fc33.aarch64 libxcrypt-devel-4.4.15-1.fc32.aarch64 libxml2-2.9.10-4.fc33.aarch64 libxml2-2.9.10-3.fc32.aarch64 libxml2-devel-2.9.10-4.fc33.aarch64 libxml2-devel-2.9.10-3.fc32.aarch64 libxslt-1.1.34-1.fc33.aarch64 libxslt-1.1.34-1.fc32.aarch64 make-1:4.3-1.fc33.aarch64 make-1:4.2.1-16.fc32.aarch64 mesa-libEGL-20.0.2-1.fc33.aarch64 mesa-libEGL-20.0.2-1.fc32.aarch64 mesa-libGL-20.0.2-1.fc33.aarch64 mesa-libGL-20.0.2-1.fc32.aarch64 mesa-libgbm-20.0.2-1.fc33.aarch64 mesa-libgbm-20.0.2-1.fc32.aarch64 mesa-libglapi-20.0.2-1.fc33.aarch64 mesa-libglapi-20.0.2-1.fc32.aarch64 ncurses-6.2-1.20200222.fc33.aarch64 ncurses-6.1-15.20191109.fc32.aarch64 ncurses-base-6.2-1.20200222.fc33.noarch ncurses-base-6.1-15.20191109.fc32.noarch ncurses-libs-6.2-1.20200222.fc33.aarch64 ncurses-libs-6.1-15.20191109.fc32.aarch64 netpbm-10.89.00-3.fc33.aarch64 netpbm-10.89.00-2.fc32.aarch64 netpbm-progs-10.89.00-3.fc33.aarch64 netpbm-progs-10.89.00-2.fc32.aarch64 nspr-4.25.0-1.fc33.aarch64 nspr-4.25.0-1.fc32.aarch64 nss-3.50.0-2.fc33.aarch64 nss-3.50.0-2.fc32.aarch64 nss-softokn-3.50.0-2.fc33.aarch64 nss-softokn-3.50.0-2.fc32.aarch64 nss-softokn-freebl-3.50.0-2.fc33.aarch64 nss-softokn-freebl-3.50.0-2.fc32.aarch64 nss-sysinit-3.50.0-2.fc33.aarch64 nss-sysinit-3.50.0-2.fc32.aarch64 nss-util-3.50.0-2.fc33.aarch64 nss-util-3.50.0-2.fc32.aarch64 ocaml-4.10.0-2.fc33.aarch64 ocaml-4.10.0-2.fc32.aarch64 ocaml-cairo-2:0.6.1-3.fc33.aarch64 ocaml-cairo-2:0.6.1-3.fc32.1.aarch64 ocaml-cairo-devel-2:0.6.1-3.fc33.aarch64 ocaml-cairo-devel-2:0.6.1-3.fc32.1.aarch64 ocaml-compiler-libs-4.10.0-2.fc33.aarch64 ocaml-compiler-libs-4.10.0-2.fc32.aarch64 ocaml-findlib-1.8.1-12.fc33.aarch64 ocaml-findlib-1.8.1-12.fc32.1.aarch64 ocaml-lablgtk3-3.1.0-2.fc33.aarch64 ocaml-lablgtk3-3.1.0-2.fc32.aarch64 ocaml-lablgtk3-devel-3.1.0-2.fc33.aarch64 ocaml-lablgtk3-devel-3.1.0-2.fc32.aarch64 ocaml-lablgtk3-sourceview3-3.1.0-2.fc33.aarch64 ocaml-lablgtk3-sourceview3-3.1.0-2.fc32.aarch64 ocaml-lablgtk3-sourceview3-devel-3.1.0-2.fc33.aarch64 ocaml-lablgtk3-sourceview3-devel-3.1.0-2.fc32.aarch64 ocaml-num-1.1-21.fc33.aarch64 ocaml-num-1.1-21.fc32.1.aarch64 ocaml-num-devel-1.1-21.fc33.aarch64 ocaml-num-devel-1.1-21.fc32.1.aarch64 ocaml-runtime-4.10.0-2.fc33.aarch64 ocaml-runtime-4.10.0-2.fc32.aarch64 ocaml-srpm-macros-6-2.fc32.noarch ocaml-srpm-macros-6-2.fc32.noarch openblas-0.3.9-1.fc33.aarch64 openblas-0.3.9-1.fc32.aarch64 openblas-serial-0.3.9-1.fc33.aarch64 openblas-serial-0.3.9-1.fc32.aarch64 openjpeg2-2.3.1-6.fc33.aarch64 openjpeg2-2.3.1-6.fc32.aarch64 openssl-libs-1:1.1.1e-1.fc33.aarch64 openssl-libs-1:1.1.1d-7.fc32.aarch64 pam-1.3.1-24.fc33.aarch64 pam-1.3.1-23.fc32.aarch64 pari-2.11.3-1.fc33.aarch64 pari-2.11.3-1.fc32.aarch64 pcre-8.44-1.fc33.aarch64 pcre-8.44-1.fc32.aarch64 pcre-cpp-8.44-1.fc33.aarch64 pcre-cpp-8.44-1.fc32.aarch64 pcre-devel-8.44-1.fc33.aarch64 pcre-devel-8.44-1.fc32.aarch64 pcre-utf16-8.44-1.fc33.aarch64 pcre-utf16-8.44-1.fc32.aarch64 pcre-utf32-8.44-1.fc33.aarch64 pcre-utf32-8.44-1.fc32.aarch64 pcre2-10.34-9.fc33.aarch64 pcre2-10.34-7.fc32.aarch64 pcre2-devel-10.34-9.fc33.aarch64 pcre2-devel-10.34-7.fc32.aarch64 pcre2-syntax-10.34-9.fc33.noarch pcre2-utf16-10.34-9.fc33.aarch64 pcre2-utf16-10.34-7.fc32.aarch64 pcre2-utf32-10.34-9.fc33.aarch64 pcre2-utf32-10.34-7.fc32.aarch64 perl-B-1.76-452.fc33.aarch64 perl-Class-Struct-0.65-452.fc33.noarch perl-Data-Dumper-2.174-443.fc32.aarch64 perl-Data-Dumper-2.174-443.fc32.aarch64 perl-Encode-4:3.05-444.fc33.aarch64 perl-Encode-4:3.02-441.fc32.aarch64 perl-English-1.10-452.fc33.noarch perl-Errno-1.30-452.fc33.aarch64 perl-Errno-1.30-451.fc32.aarch64 perl-Fcntl-1.13-452.fc33.aarch64 perl-File-Basename-2.85-452.fc33.noarch perl-File-Copy-2.34-452.fc33.noarch perl-File-Find-1.36-452.fc33.noarch perl-File-stat-1.08-452.fc33.noarch perl-FileHandle-2.03-452.fc33.noarch perl-Getopt-Std-1.12-452.fc33.noarch perl-IO-1.40-452.fc33.aarch64 perl-IO-1.40-451.fc32.aarch64 perl-IPC-Open3-1.20-452.fc33.noarch perl-POSIX-1.88-452.fc33.aarch64 perl-Pod-Perldoc-3.28.01-444.fc33.noarch perl-Pod-Perldoc-3.28.01-443.fc32.noarch perl-Pod-Usage-4:1.70-1.fc33.noarch perl-Pod-Usage-4:1.69-440.fc32.noarch perl-SelectSaver-1.02-452.fc33.noarch perl-Symbol-1.08-452.fc33.noarch perl-Tie-4.6-452.fc33.noarch
perl-Unicode-Normalize-1.26-440.fc32.aarch64 perl-XML-Parser-2.46-3.fc33.aarch64 perl-XML-Parser-2.46-2.fc32.aarch64 perl-encoding-4:2.22-444.fc33.aarch64 perl-encoding-4:2.22-441.fc32.aarch64 perl-if-0.60.800-452.fc33.noarch perl-interpreter-4:5.30.2-452.fc33.aarch64 perl-interpreter-4:5.30.2-451.fc32.aarch64 perl-libs-4:5.30.2-452.fc33.aarch64 perl-libs-4:5.30.2-451.fc32.aarch64 perl-macros-4:5.30.2-452.fc33.aarch64 perl-macros-4:5.30.2-451.fc32.aarch64 perl-mro-1.22-452.fc33.aarch64 perl-open-1.11-452.fc33.noarch perl-open-1.11-451.fc32.noarch perl-overload-1.30-452.fc33.noarch perl-overloading-0.02-452.fc33.noarch perl-subs-1.03-452.fc33.noarch
perl-threads-1:2.22-442.fc32.aarch64
perl-threads-shared-1.60-441.fc32.aarch64 perl-vars-1.05-452.fc33.noarch python-pip-wheel-20.0.2-2.fc33.noarch python-pip-wheel-19.3.1-2.fc32.noarch python-setuptools-wheel-46.0.0-1.fc33.noarch python-setuptools-wheel-41.6.0-2.fc32.noarch python3-3.8.2-2.fc33.aarch64 python3-3.8.2-2.fc32.aarch64 python3-devel-3.8.2-2.fc33.aarch64 python3-devel-3.8.2-2.fc32.aarch64 python3-libs-3.8.2-2.fc33.aarch64 python3-libs-3.8.2-2.fc32.aarch64 python3-orderedset-2.0.3-1.fc33.aarch64 python3-orderedset-2.0.3-1.fc32.aarch64 python3-packaging-20.3-1.fc33.noarch python3-packaging-20.1-2.fc32.noarch python3-pyyaml-5.3.1-1.fc33.aarch64 python3-pyyaml-5.3-4.fc32.aarch64 python3-requests-2.23.0-1.fc33.noarch python3-requests-2.22.0-8.fc32.noarch python3-setuptools-46.0.0-1.fc33.noarch python3-setuptools-41.6.0-2.fc32.noarch python3-sphinx_rtd_theme-0.4.3-11.fc33.noarch python3-sphinx_rtd_theme-0.4.3-10.fc32.noarch python3-urllib3-1.25.8-1.fc33.noarch python3-urllib3-1.25.7-3.fc32.noarch redhat-rpm-config-153-1.fc33.noarch redhat-rpm-config-150-1.fc32.noarch rust-srpm-macros-13-2.fc33.noarch rust-srpm-macros-13-2.fc32.noarch sed-4.8-1.fc33.aarch64 sed-4.5-5.fc32.aarch64 shadow-utils-2:4.8.1-2.fc33.aarch64 shadow-utils-2:4.8-4.fc32.aarch64 shared-mime-info-1.15-3.fc33.aarch64 shared-mime-info-1.15-3.fc32.aarch64 systemd-245.2-1.fc33.aarch64 systemd-245.2-1.fc32.aarch64 systemd-libs-245.2-1.fc33.aarch64 systemd-libs-245.2-1.fc32.aarch64 systemd-pam-245.2-1.fc33.aarch64 systemd-pam-245.2-1.fc32.aarch64 systemd-rpm-macros-245.2-1.fc33.noarch systemd-rpm-macros-245.2-1.fc32.noarch texlive-adjustbox-9:svn49596-19.fc33.noarch texlive-adjustbox-9:svn49596-19.fc32.noarch texlive-ae-9:svn15878.1.4-19.fc33.noarch texlive-ae-9:svn15878.1.4-19.fc32.noarch texlive-algorithms-9:svn42428-19.fc33.noarch texlive-algorithms-9:svn42428-19.fc32.noarch texlive-amscls-9:svn46099-19.fc33.noarch texlive-amscls-9:svn46099-19.fc32.noarch texlive-amsfonts-9:svn29208.3.04-19.fc33.noarch texlive-amsfonts-9:svn29208.3.04-19.fc32.noarch texlive-amsmath-9:svn49390-19.fc33.noarch texlive-amsmath-9:svn49390-19.fc32.noarch texlive-anysize-9:svn15878.0-19.fc33.noarch texlive-anysize-9:svn15878.0-19.fc32.noarch texlive-attachfile-9:svn42099-19.fc33.noarch texlive-attachfile-9:svn42099-19.fc32.noarch texlive-auto-pst-pdf-9:svn23723.0.6-19.fc33.noarch texlive-auto-pst-pdf-9:svn23723.0.6-19.fc32.noarch texlive-avantgar-9:svn31835.0-19.fc33.noarch texlive-avantgar-9:svn31835.0-19.fc32.noarch texlive-babel-9:svn51681-19.fc33.noarch texlive-babel-9:svn51681-19.fc32.noarch texlive-babel-english-9:svn44495-19.fc33.noarch texlive-babel-english-9:svn44495-19.fc32.noarch texlive-babelbib-9:svn50354-19.fc33.noarch texlive-babelbib-9:svn50354-19.fc32.noarch texlive-beamer-9:svn51711-19.fc33.noarch texlive-beamer-9:svn51711-19.fc32.noarch texlive-bera-9:svn20031.0-19.fc33.noarch texlive-bera-9:svn20031.0-19.fc32.noarch texlive-beton-9:svn15878.0-19.fc33.noarch texlive-beton-9:svn15878.0-19.fc32.noarch texlive-bidi-9:svn51377-19.fc33.noarch texlive-bidi-9:svn51377-19.fc32.noarch texlive-bookman-9:svn31835.0-19.fc33.noarch texlive-bookman-9:svn31835.0-19.fc32.noarch texlive-booktabs-9:svn40846-19.fc33.noarch texlive-booktabs-9:svn40846-19.fc32.noarch texlive-breakurl-9:svn29901.1.40-19.fc33.noarch texlive-breakurl-9:svn29901.1.40-19.fc32.noarch texlive-breqn-9:svn43071-19.fc33.noarch texlive-breqn-9:svn43071-19.fc32.noarch texlive-capt-of-9:svn29803.0-19.fc33.noarch texlive-capt-of-9:svn29803.0-19.fc32.noarch texlive-caption-9:svn47968-19.fc33.noarch texlive-caption-9:svn47968-19.fc32.noarch texlive-carlisle-9:svn47876-19.fc33.noarch texlive-carlisle-9:svn47876-19.fc32.noarch texlive-changepage-9:svn15878.1.0c-19.fc33.noarch texlive-changepage-9:svn15878.1.0c-19.fc32.noarch texlive-charter-9:svn15878.0-19.fc33.noarch texlive-charter-9:svn15878.0-19.fc32.noarch texlive-chngcntr-9:svn47577-19.fc33.noarch texlive-chngcntr-9:svn47577-19.fc32.noarch texlive-cite-9:svn36428.5.5-19.fc33.noarch texlive-cite-9:svn36428.5.5-19.fc32.noarch texlive-cm-9:svn49028-19.fc33.noarch texlive-cm-9:svn49028-19.fc32.noarch texlive-cm-super-9:svn15878.0-19.fc33.noarch texlive-cm-super-9:svn15878.0-19.fc32.noarch texlive-cmap-9:svn42428-19.fc33.noarch texlive-cmap-9:svn42428-19.fc32.noarch texlive-cmextra-9:svn42428-19.fc33.noarch texlive-cmextra-9:svn42428-19.fc32.noarch texlive-collectbox-9:svn26557.0.4b-19.fc33.noarch texlive-collectbox-9:svn26557.0.4b-19.fc32.noarch texlive-collection-basic-9:svn51558-19.fc33.noarch texlive-collection-basic-9:svn51558-19.fc32.noarch texlive-collection-fontsrecommended-9:svn35830.0-19.fc33.noarch texlive-collection-fontsrecommended-9:svn35830.0-19.fc32.noarch texlive-collection-latex-9:svn41614-19.fc33.noarch texlive-collection-latex-9:svn41614-19.fc32.noarch texlive-collection-latexrecommended-9:svn51780-19.fc33.noarch texlive-collection-latexrecommended-9:svn51780-19.fc32.noarch texlive-colorprofiles-9:svn49086-19.fc33.noarch texlive-colorprofiles-9:svn49086-19.fc32.noarch texlive-colortbl-9:svn49404-19.fc33.noarch texlive-colortbl-9:svn49404-19.fc32.noarch texlive-comment-9:svn41927-19.fc33.noarch texlive-comment-9:svn41927-19.fc32.noarch texlive-courier-9:svn35058.0-19.fc33.noarch texlive-courier-9:svn35058.0-19.fc32.noarch texlive-crop-9:svn15878.1.5-19.fc33.noarch texlive-crop-9:svn15878.1.5-19.fc32.noarch texlive-csquotes-9:svn51091-19.fc33.noarch texlive-csquotes-9:svn51091-19.fc32.noarch texlive-ctable-9:svn38672-19.fc33.noarch texlive-ctable-9:svn38672-19.fc32.noarch texlive-currfile-9:svn40725-19.fc33.noarch texlive-currfile-9:svn40725-19.fc32.noarch texlive-dehyph-9:svn48599-19.fc33.noarch texlive-dehyph-9:svn48599-19.fc32.noarch texlive-ec-9:svn25033.1.0-19.fc33.noarch texlive-ec-9:svn25033.1.0-19.fc32.noarch texlive-eepic-9:svn15878.1.1e-19.fc33.noarch texlive-eepic-9:svn15878.1.1e-19.fc32.noarch texlive-enctex-9:svn34957.0-19.fc33.noarch texlive-enctex-9:svn34957.0-19.fc32.noarch texlive-enumitem-9:svn51423-19.fc33.noarch texlive-enumitem-9:svn51423-19.fc32.noarch texlive-environ-9:svn33821.0.3-19.fc33.noarch texlive-environ-9:svn33821.0.3-19.fc32.noarch texlive-eso-pic-9:svn47694-19.fc33.noarch texlive-eso-pic-9:svn47694-19.fc32.noarch texlive-etex-9:svn37057.0-19.fc33.noarch texlive-etex-9:svn37057.0-19.fc32.noarch texlive-etex-pkg-9:svn41784-19.fc33.noarch texlive-etex-pkg-9:svn41784-19.fc32.noarch texlive-etoolbox-9:svn46602-19.fc33.noarch texlive-etoolbox-9:svn46602-19.fc32.noarch texlive-euenc-9:svn19795.0.1h-19.fc33.noarch texlive-euenc-9:svn19795.0.1h-19.fc32.noarch texlive-euler-9:svn42428-19.fc33.noarch texlive-euler-9:svn42428-19.fc32.noarch texlive-euro-9:svn22191.1.1-19.fc33.noarch texlive-euro-9:svn22191.1.1-19.fc32.noarch texlive-euro-ce-9:svn25714-19.fc33.noarch texlive-euro-ce-9:svn25714-19.fc32.noarch texlive-eurosym-9:svn17265.1.4_subrfix-19.fc33.noarch texlive-eurosym-9:svn17265.1.4_subrfix-19.fc32.noarch texlive-extsizes-9:svn17263.1.4a-19.fc33.noarch texlive-extsizes-9:svn17263.1.4a-19.fc32.noarch texlive-fancybox-9:svn18304.1.4-19.fc33.noarch texlive-fancybox-9:svn18304.1.4-19.fc32.noarch texlive-fancyhdr-9:svn49886-19.fc33.noarch texlive-fancyhdr-9:svn49886-19.fc32.noarch texlive-fancyref-9:svn15878.0.9c-19.fc33.noarch texlive-fancyref-9:svn15878.0.9c-19.fc32.noarch texlive-fancyvrb-9:svn49717-19.fc33.noarch texlive-fancyvrb-9:svn49717-19.fc32.noarch texlive-filecontents-9:svn47890-19.fc33.noarch texlive-filecontents-9:svn47890-19.fc32.noarch texlive-filehook-9:svn24280.0.5d-19.fc33.noarch texlive-filehook-9:svn24280.0.5d-19.fc32.noarch texlive-finstrut-9:svn21719.0.5-19.fc33.noarch texlive-finstrut-9:svn21719.0.5-19.fc32.noarch texlive-fix2col-9:svn38770-19.fc33.noarch texlive-fix2col-9:svn38770-19.fc32.noarch texlive-float-9:svn15878.1.3d-19.fc33.noarch texlive-float-9:svn15878.1.3d-19.fc32.noarch texlive-fncychap-9:svn20710.v1.34-19.fc33.noarch texlive-fncychap-9:svn20710.v1.34-19.fc32.noarch texlive-fontspec-9:svn50387-19.fc33.noarch texlive-fontspec-9:svn50387-19.fc32.noarch texlive-footmisc-9:svn23330.5.5b-19.fc33.noarch texlive-footmisc-9:svn23330.5.5b-19.fc32.noarch texlive-fp-9:svn49719-19.fc33.noarch texlive-fp-9:svn49719-19.fc32.noarch texlive-fpl-9:svn49603-19.fc33.noarch texlive-fpl-9:svn49603-19.fc32.noarch texlive-framed-9:svn26789.0.96-19.fc33.noarch texlive-framed-9:svn26789.0.96-19.fc32.noarch texlive-geometry-9:svn47638-19.fc33.noarch texlive-geometry-9:svn47638-19.fc32.noarch texlive-gnu-freefont-9:svn29349.0-19.fc33.noarch texlive-gnu-freefont-9:svn29349.0-19.fc32.noarch texlive-graphics-9:svn51733-19.fc33.noarch texlive-graphics-9:svn51733-19.fc32.noarch texlive-graphics-cfg-9:svn40269-19.fc33.noarch texlive-graphics-cfg-9:svn40269-19.fc32.noarch texlive-graphics-def-9:svn46267-19.fc33.noarch texlive-graphics-def-9:svn46267-19.fc32.noarch texlive-helvetic-9:svn31835.0-19.fc33.noarch texlive-helvetic-9:svn31835.0-19.fc32.noarch texlive-hyperref-9:svn51742-19.fc33.noarch texlive-hyperref-9:svn51742-19.fc32.noarch texlive-hyph-utf8-9:svn51186-19.fc33.noarch texlive-hyph-utf8-9:svn51186-19.fc32.noarch texlive-hyphen-base-9:svn51200-19.fc33.noarch texlive-hyphen-base-9:svn51200-19.fc32.noarch texlive-hyphenex-9:svn37354.0-19.fc33.noarch texlive-hyphenex-9:svn37354.0-19.fc32.noarch texlive-ifetex-9:svn47231-19.fc33.noarch texlive-ifetex-9:svn47231-19.fc32.noarch texlive-ifluatex-9:svn51770-19.fc33.noarch texlive-ifluatex-9:svn51770-19.fc32.noarch texlive-ifmtarg-9:svn47544-19.fc33.noarch texlive-ifmtarg-9:svn47544-19.fc32.noarch texlive-ifoddpage-9:svn40726-19.fc33.noarch texlive-ifoddpage-9:svn40726-19.fc32.noarch texlive-ifplatform-9:svn45533-19.fc33.noarch texlive-ifplatform-9:svn45533-19.fc32.noarch texlive-iftex-9:svn29654.0.2-19.fc33.noarch texlive-iftex-9:svn29654.0.2-19.fc32.noarch texlive-ifxetex-9:svn19685.0.5-19.fc33.noarch texlive-ifxetex-9:svn19685.0.5-19.fc32.noarch texlive-index-9:svn24099.4.1beta-19.fc33.noarch texlive-index-9:svn24099.4.1beta-19.fc32.noarch texlive-jknapltx-9:svn19440.0-19.fc33.noarch texlive-jknapltx-9:svn19440.0-19.fc32.noarch texlive-kastrup-9:svn15878.0-19.fc33.noarch texlive-kastrup-9:svn15878.0-19.fc32.noarch texlive-knuth-lib-9:svn35820.0-19.fc33.noarch texlive-knuth-lib-9:svn35820.0-19.fc32.noarch texlive-knuth-local-9:svn38627-19.fc33.noarch texlive-knuth-local-9:svn38627-19.fc32.noarch texlive-koma-script-9:svn49929-19.fc33.noarch texlive-koma-script-9:svn49929-19.fc32.noarch texlive-l3backend-9:svn52343-19.fc33.noarch texlive-l3backend-9:svn52343-19.fc32.noarch texlive-l3experimental-9:svn51546-19.fc33.noarch texlive-l3experimental-9:svn51546-19.fc32.noarch texlive-l3kernel-9:svn51745-19.fc33.noarch texlive-l3kernel-9:svn51745-19.fc32.noarch texlive-l3packages-9:svn51546-19.fc33.noarch texlive-l3packages-9:svn51546-19.fc32.noarch texlive-latex-base-dev-9:svn51730-19.fc33.noarch texlive-latex-base-dev-9:svn51730-19.fc32.noarch texlive-latex-bin-dev-9:svn51839-19.fc33.noarch texlive-latex-bin-dev-9:svn51839-19.fc32.noarch texlive-latex-fonts-9:svn28888.0-19.fc33.noarch texlive-latex-fonts-9:svn28888.0-19.fc32.noarch texlive-latex-graphics-dev-9:svn51730-19.fc33.noarch texlive-latex-graphics-dev-9:svn51730-19.fc32.noarch texlive-latexbug-9:svn49573-19.fc33.noarch texlive-latexbug-9:svn49573-19.fc32.noarch texlive-latexconfig-9:svn45777-19.fc33.noarch texlive-latexconfig-9:svn45777-19.fc32.noarch texlive-lineno-9:svn21442.4.41-19.fc33.noarch texlive-lineno-9:svn21442.4.41-19.fc32.noarch texlive-listings-9:svn50175-19.fc33.noarch texlive-listings-9:svn50175-19.fc32.noarch texlive-lm-9:svn48145-19.fc33.noarch texlive-lm-9:svn48145-19.fc32.noarch texlive-lm-math-9:svn36915.1.959-19.fc33.noarch texlive-lm-math-9:svn36915.1.959-19.fc32.noarch texlive-ltabptch-9:svn17533.1.74d-19.fc33.noarch texlive-ltabptch-9:svn17533.1.74d-19.fc32.noarch texlive-ltxmisc-9:svn21927.0-19.fc33.noarch texlive-ltxmisc-9:svn21927.0-19.fc32.noarch texlive-lua-alt-getopt-9:svn29349.0.7.0-19.fc33.noarch texlive-lua-alt-getopt-9:svn29349.0.7.0-19.fc32.noarch texlive-luabidi-9:svn30790.0.2-19.fc33.noarch texlive-luabidi-9:svn30790.0.2-19.fc32.noarch texlive-lualatex-math-9:svn49904-19.fc33.noarch texlive-lualatex-math-9:svn49904-19.fc32.noarch texlive-lualibs-9:svn51642-19.fc33.noarch texlive-lualibs-9:svn51642-19.fc32.noarch texlive-luatexbase-9:svn38550-19.fc33.noarch texlive-luatexbase-9:svn38550-19.fc32.noarch texlive-makecmds-9:svn15878.0-19.fc33.noarch texlive-makecmds-9:svn15878.0-19.fc32.noarch texlive-manfnt-font-9:svn45777-19.fc33.noarch texlive-manfnt-font-9:svn45777-19.fc32.noarch texlive-marginnote-9:svn48383-19.fc33.noarch texlive-marginnote-9:svn48383-19.fc32.noarch texlive-marvosym-9:svn29349.2.2a-19.fc33.noarch texlive-marvosym-9:svn29349.2.2a-19.fc32.noarch texlive-mathpazo-9:svn15878.1.003-19.fc33.noarch texlive-mathpazo-9:svn15878.1.003-19.fc32.noarch texlive-mathspec-9:svn42773-19.fc33.noarch texlive-mathspec-9:svn42773-19.fc32.noarch texlive-mathtools-9:svn51788-19.fc33.noarch texlive-mathtools-9:svn51788-19.fc32.noarch texlive-mdwtools-9:svn15878.1.05.4-19.fc33.noarch texlive-mdwtools-9:svn15878.1.05.4-19.fc32.noarch texlive-memoir-9:svn49394-19.fc33.noarch texlive-memoir-9:svn49394-19.fc32.noarch texlive-metalogo-9:svn18611.0.12-19.fc33.noarch texlive-metalogo-9:svn18611.0.12-19.fc32.noarch texlive-mflogo-9:svn42428-19.fc33.noarch texlive-mflogo-9:svn42428-19.fc32.noarch texlive-mflogo-font-9:svn36898.1.002-19.fc33.noarch texlive-mflogo-font-9:svn36898.1.002-19.fc32.noarch texlive-mfnfss-9:svn46036-19.fc33.noarch texlive-mfnfss-9:svn46036-19.fc32.noarch texlive-microtype-9:svn50187-19.fc33.noarch texlive-microtype-9:svn50187-19.fc32.noarch texlive-mnsymbol-9:svn18651.1.4-19.fc33.noarch texlive-mnsymbol-9:svn18651.1.4-19.fc32.noarch texlive-moreverb-9:svn22126.2.3a-19.fc33.noarch texlive-moreverb-9:svn22126.2.3a-19.fc32.noarch texlive-mparhack-9:svn15878.1.4-19.fc33.noarch texlive-mparhack-9:svn15878.1.4-19.fc32.noarch texlive-ms-9:svn51784-19.fc33.noarch texlive-ms-9:svn51784-19.fc32.noarch texlive-multido-9:svn18302.1.42-19.fc33.noarch texlive-multido-9:svn18302.1.42-19.fc32.noarch texlive-multirow-9:svn51278-19.fc33.noarch texlive-multirow-9:svn51278-19.fc32.noarch texlive-natbib-9:svn20668.8.31b-19.fc33.noarch texlive-natbib-9:svn20668.8.31b-19.fc32.noarch texlive-ncntrsbk-9:svn31835.0-19.fc33.noarch texlive-ncntrsbk-9:svn31835.0-19.fc32.noarch texlive-needspace-9:svn29601.1.3d-19.fc33.noarch texlive-needspace-9:svn29601.1.3d-19.fc32.noarch texlive-ntgclass-9:svn15878.2.1a-19.fc33.noarch texlive-ntgclass-9:svn15878.2.1a-19.fc32.noarch texlive-palatino-9:svn31835.0-19.fc33.noarch texlive-palatino-9:svn31835.0-19.fc32.noarch texlive-paralist-9:svn43021-19.fc33.noarch texlive-paralist-9:svn43021-19.fc32.noarch texlive-parallel-9:svn15878.0-19.fc33.noarch texlive-parallel-9:svn15878.0-19.fc32.noarch texlive-parskip-9:svn49730-19.fc33.noarch texlive-parskip-9:svn49730-19.fc32.noarch texlive-pdfpages-9:svn45659-19.fc33.noarch texlive-pdfpages-9:svn45659-19.fc32.noarch texlive-pgf-9:svn51817-19.fc33.noarch texlive-pgf-9:svn51817-19.fc32.noarch texlive-placeins-9:svn19848.2.2-19.fc33.noarch texlive-placeins-9:svn19848.2.2-19.fc32.noarch texlive-plain-9:svn43076-19.fc33.noarch texlive-plain-9:svn43076-19.fc32.noarch texlive-polyglossia-9:svn50787-19.fc33.noarch texlive-polyglossia-9:svn50787-19.fc32.noarch texlive-preprint-9:svn30447.2011-19.fc33.noarch texlive-preprint-9:svn30447.2011-19.fc32.noarch texlive-psfrag-9:svn15878.3.04-19.fc33.noarch texlive-psfrag-9:svn15878.3.04-19.fc32.noarch texlive-pslatex-9:svn16416.0-19.fc33.noarch texlive-pslatex-9:svn16416.0-19.fc32.noarch texlive-psnfss-9:svn33946.9.2a-19.fc33.noarch texlive-psnfss-9:svn33946.9.2a-19.fc32.noarch texlive-pspicture-9:svn15878.0-19.fc33.noarch texlive-pspicture-9:svn15878.0-19.fc32.noarch texlive-pst-3d-9:svn17257.1.10-19.fc33.noarch texlive-pst-3d-9:svn17257.1.10-19.fc32.noarch texlive-pst-coil-9:svn37377.1.07-19.fc33.noarch texlive-pst-coil-9:svn37377.1.07-19.fc32.noarch texlive-pst-eps-9:svn15878.1.0-19.fc33.noarch texlive-pst-eps-9:svn15878.1.0-19.fc32.noarch texlive-pst-fill-9:svn15878.1.01-19.fc33.noarch texlive-pst-fill-9:svn15878.1.01-19.fc32.noarch texlive-pst-grad-9:svn15878.1.06-19.fc33.noarch texlive-pst-grad-9:svn15878.1.06-19.fc32.noarch texlive-pst-math-9:svn49425-19.fc33.noarch texlive-pst-math-9:svn49425-19.fc32.noarch texlive-pst-node-9:svn50215-19.fc33.noarch texlive-pst-node-9:svn50215-19.fc32.noarch texlive-pst-ovl-9:svn45506-19.fc33.noarch texlive-pst-ovl-9:svn45506-19.fc32.noarch texlive-pst-plot-9:svn51650-19.fc33.noarch texlive-pst-plot-9:svn51650-19.fc32.noarch texlive-pst-text-9:svn49542-19.fc33.noarch texlive-pst-text-9:svn49542-19.fc32.noarch texlive-pst-tools-9:svn45978-19.fc33.noarch texlive-pst-tools-9:svn45978-19.fc32.noarch texlive-pst-tree-9:svn43272-19.fc33.noarch texlive-pst-tree-9:svn43272-19.fc32.noarch texlive-pstricks-9:svn51102-19.fc33.noarch texlive-pstricks-9:svn51102-19.fc32.noarch texlive-pstricks-add-9:svn49680-19.fc33.noarch texlive-pstricks-add-9:svn49680-19.fc32.noarch texlive-pxfonts-9:svn15878.0-19.fc33.noarch texlive-pxfonts-9:svn15878.0-19.fc32.noarch texlive-qstest-9:svn15878.0-19.fc33.noarch texlive-qstest-9:svn15878.0-19.fc32.noarch texlive-ragged2e-9:svn51780-19.fc33.noarch texlive-ragged2e-9:svn51780-19.fc32.noarch texlive-rcs-9:svn15878.0-19.fc33.noarch texlive-rcs-9:svn15878.0-19.fc32.noarch texlive-realscripts-9:svn39706-19.fc33.noarch texlive-realscripts-9:svn39706-19.fc32.noarch texlive-rsfs-9:svn15878.0-19.fc33.noarch texlive-rsfs-9:svn15878.0-19.fc32.noarch texlive-sansmath-9:svn17997.1.1-19.fc33.noarch texlive-sansmath-9:svn17997.1.1-19.fc32.noarch texlive-sansmathaccent-9:svn30187.0-19.fc33.noarch texlive-sansmathaccent-9:svn30187.0-19.fc32.noarch texlive-sauerj-9:svn15878.0-19.fc33.noarch texlive-sauerj-9:svn15878.0-19.fc32.noarch texlive-section-9:svn20180.0-19.fc33.noarch texlive-section-9:svn20180.0-19.fc32.noarch texlive-seminar-9:svn34011.1.62-19.fc33.noarch texlive-seminar-9:svn34011.1.62-19.fc32.noarch texlive-sepnum-9:svn20186.2.0-19.fc33.noarch texlive-sepnum-9:svn20186.2.0-19.fc32.noarch texlive-setspace-9:svn24881.6.7a-19.fc33.noarch texlive-setspace-9:svn24881.6.7a-19.fc32.noarch texlive-showexpl-9:svn42677-19.fc33.noarch texlive-showexpl-9:svn42677-19.fc32.noarch texlive-soul-9:svn15878.2.4-19.fc33.noarch texlive-soul-9:svn15878.2.4-19.fc32.noarch texlive-stmaryrd-9:svn22027.0-19.fc33.noarch texlive-stmaryrd-9:svn22027.0-19.fc32.noarch texlive-subfig-9:svn15878.1.3-19.fc33.noarch texlive-subfig-9:svn15878.1.3-19.fc32.noarch texlive-symbol-9:svn31835.0-19.fc33.noarch texlive-symbol-9:svn31835.0-19.fc32.noarch texlive-tabulary-9:svn34368.0.10-19.fc33.noarch texlive-tabulary-9:svn34368.0.10-19.fc32.noarch texlive-tetex-7:20190410-12.fc32.noarch texlive-tetex-7:20190410-12.fc32.noarch texlive-tex-gyre-9:svn48058-19.fc33.noarch texlive-tex-gyre-9:svn48058-19.fc32.noarch texlive-tex-gyre-math-9:svn41264-19.fc33.noarch texlive-tex-gyre-math-9:svn41264-19.fc32.noarch texlive-tex-ini-files-9:svn40533-19.fc33.noarch texlive-tex-ini-files-9:svn40533-19.fc32.noarch texlive-texlive-common-doc-9:svn50466-19.fc33.noarch texlive-texlive-common-doc-9:svn50466-19.fc32.noarch texlive-texlive-docindex-9:svn51813-19.fc33.noarch texlive-texlive-docindex-9:svn51813-19.fc32.noarch texlive-texlive-msg-translations-9:svn51750-19.fc33.noarch texlive-texlive-msg-translations-9:svn51750-19.fc32.noarch texlive-textcase-9:svn15878.0-19.fc33.noarch texlive-textcase-9:svn15878.0-19.fc32.noarch texlive-times-9:svn35058.0-19.fc33.noarch texlive-times-9:svn35058.0-19.fc32.noarch texlive-tipa-9:svn29349.1.3-19.fc33.noarch texlive-tipa-9:svn29349.1.3-19.fc32.noarch texlive-titlesec-9:svn51664-19.fc33.noarch texlive-titlesec-9:svn51664-19.fc32.noarch texlive-tools-9:svn49619-19.fc33.noarch texlive-tools-9:svn49619-19.fc32.noarch texlive-translator-9:svn51279-19.fc33.noarch texlive-translator-9:svn51279-19.fc32.noarch texlive-trimspaces-9:svn15878.1.1-19.fc33.noarch texlive-trimspaces-9:svn15878.1.1-19.fc32.noarch texlive-txfonts-9:svn15878.0-19.fc33.noarch texlive-txfonts-9:svn15878.0-19.fc32.noarch texlive-typehtml-9:svn17134.0-19.fc33.noarch texlive-typehtml-9:svn17134.0-19.fc32.noarch texlive-ucharcat-9:svn38907-19.fc33.noarch texlive-ucharcat-9:svn38907-19.fc32.noarch texlive-ucs-9:svn35853.2.2-19.fc33.noarch texlive-ucs-9:svn35853.2.2-19.fc32.noarch texlive-underscore-9:svn18261.0-19.fc33.noarch texlive-underscore-9:svn18261.0-19.fc32.noarch texlive-unicode-data-9:svn50303-19.fc33.noarch texlive-unicode-data-9:svn50303-19.fc32.noarch texlive-unicode-math-9:svn50245-19.fc33.noarch texlive-unicode-math-9:svn50245-19.fc32.noarch texlive-updmap-map-9:svn52454-19.fc33.noarch texlive-updmap-map-9:svn52454-19.fc32.noarch texlive-upquote-9:svn26059.v1.3-19.fc33.noarch texlive-upquote-9:svn26059.v1.3-19.fc32.noarch texlive-url-9:svn32528.3.4-19.fc33.noarch texlive-url-9:svn32528.3.4-19.fc32.noarch texlive-utopia-9:svn15878.0-19.fc33.noarch texlive-utopia-9:svn15878.0-19.fc32.noarch texlive-varwidth-9:svn24104.0.92-19.fc33.noarch texlive-varwidth-9:svn24104.0.92-19.fc32.noarch texlive-wasy-9:svn35831.0-19.fc33.noarch texlive-wasy-9:svn35831.0-19.fc32.noarch texlive-wasy2-ps-9:svn35830.0-19.fc33.noarch texlive-wasy2-ps-9:svn35830.0-19.fc32.noarch texlive-wasysym-9:svn15878.2.0-19.fc33.noarch texlive-wasysym-9:svn15878.2.0-19.fc32.noarch texlive-wrapfig-9:svn22048.3.6-19.fc33.noarch texlive-wrapfig-9:svn22048.3.6-19.fc32.noarch texlive-xcolor-9:svn41044-19.fc33.noarch texlive-xcolor-9:svn41044-19.fc32.noarch texlive-xetexconfig-9:svn45845-19.fc33.noarch texlive-xetexconfig-9:svn45845-19.fc32.noarch texlive-xifthen-9:svn38929-19.fc33.noarch texlive-xifthen-9:svn38929-19.fc32.noarch texlive-xkeyval-9:svn35741.2.7a-19.fc33.noarch texlive-xkeyval-9:svn35741.2.7a-19.fc32.noarch texlive-xltxtra-9:svn49555-19.fc33.noarch texlive-xltxtra-9:svn49555-19.fc32.noarch texlive-xunicode-9:svn30466.0.981-19.fc33.noarch texlive-xunicode-9:svn30466.0.981-19.fc32.noarch texlive-zapfchan-9:svn31835.0-19.fc33.noarch texlive-zapfchan-9:svn31835.0-19.fc32.noarch texlive-zapfding-9:svn31835.0-19.fc33.noarch texlive-zapfding-9:svn31835.0-19.fc32.noarch util-linux-2.35.1-7.fc33.aarch64 util-linux-2.35.1-4.fc32.aarch64 vim-filesystem-2:8.2.444-1.fc33.noarch vim-filesystem-2:8.2.348-1.fc32.noarch wayland-devel-1.18.0-1.fc33.aarch64 wayland-devel-1.18.0-1.fc32.aarch64 wayland-protocols-devel-1.20-1.fc33.noarch wayland-protocols-devel-1.20-1.fc32.noarch which-2.21-18.fc33.aarch64 which-2.21-17.fc32.aarch64 xxhash-libs-0.7.3-1.fc33.aarch64 xxhash-libs-0.7.3-1.fc32.aarch64
I notice that curl has a higher version in F32 than in Rawhide. Is that a known issue?
The only packages on this list that really jump out at me are annobin, glibc, gmp, and redhat-rpm-config. I am out of time tonight, but tomorrow I am going to take a careful look at what changed for each of those 4 packages between the F32 version and the Rawhide version.
On Wed, Mar 25, 2020 at 9:24 PM Jerry James loganjerry@gmail.com wrote:
You'll want to view this with a fixed-width font.
Thank you, gmail, for completely destroying my formatting work. Oh well, you can probably tell what I was trying to do.
Hi Jerry! Long time no see.
Jerry James writes:
I built the whole stack for Rawhide with no issues.
Zero compiler warnings? No fiddling to suppress compiler warnings *anywhere* in the build chain (whether options or pragmas)?
Not suggesting you go looking for pragmas (probably have 10 million lines of code :-( ), but you could check for warnings in the build logs. I didn't see any in the tail, though.
Attempt 1: the x86_64 build segfaulted while running "bin/coqc -coqlib . -q -native-compiler yes theories/FSets/FMapFullAVL.v", where coqc is a binary built from the coq sources. Coq is a theorem prover. The coqc run is essentially checking that a series of proof steps indeed prove a theorem. The "-native-compiler yes" part means that the OCaml compiler will be invoked to generate native code for parts of this process.
Where is the segfault? Is it in coqc or is it in the generated code or is it in the OCaml compiler or is it somewhere else?
If you can't rule out the OCaml compiler (or other parts of the OCaml toolchain) and the generated code: - what happens if you use -native-compiler=no? - How is the native code used? Compiled and linked as a .so, then dynamically loaded and called? Compiled and linked as an executable, then run as a separate process?
If you can't rule out OCaml: does it produce native code directly, or does it produce C or similar and then use GCC or similar to produce the native code?
What optimizations to gcc are being used where it's being used?
Are other compilers (LLVM) supported in lieu of GCC for any parts of the build chain?
I was puzzled by the segfault. I decided to see if the segfault is deterministic, so I launched...
Attempt 2: the x86_64 build succeeded. The s390x build failed with an illegal instruction error while running *the same command*.
Attempt 3: The s390x build again failed with an illegal instruction error while running the same command, and aarch64 segfaulted while running the same command. All other arches succeeded. I saved the URL for this one:
https://koji.fedoraproject.org/koji/taskinfo?taskID=42768075
From the tail of the S390 build log:
rm -f theories/FSets/FMapFullAVL.glob bin/coqc -coqlib . -q -native-compiler yes theories/FSets/FMapFullAVL.v make[1]: *** [Makefile.build:880: theories/FSets/FMapFullAVL.vo] Illegal instruction (core dumped) make[1]: *** [theories/FSets/FMapFullAVL.vo] Deleting file 'theories/FSets/FMapFullAVL.glob' make[1]: Leaving directory '/builddir/build/BUILD/coq-8.11.0' make: *** [Makefile:177: submake] Error 2
That seems pretty unhelpful, as Makefile.build:880 is presumably just the rule that invokes bin/coqc, which evidently is actually a compiler driver that invokes various executables. Do you have access to the core that was allegedly dumped, to at least determine the identity of the executable that segfaulted? Is it possible to reproduce outside of Koji?
Stay healthy!
On Thu, Mar 26, 2020 at 01:20:58PM +0900, Stephen J. Turnbull wrote:
Where is the segfault? Is it in coqc or is it in the generated code or is it in the OCaml compiler or is it somewhere else?
I managed to reproduce this on my mostly Rawhide x86-64 development machine by running fedpkg build in a loop overnight. This also captured a core dump.
Unfortunately the stack trace, even with full debuginfo installed, was not very helpful. I'm now trying to run the unstable bin/coqc command over and over again under gdb, but the bug has not reproduced there so far.
One thing I did learn from the stack trace is that it appears to fail in C code (in the garbage collector). However that might be because the OCaml heap is corrupted by an earlier event. Another thing I learned is that it happens when doing something or other with weak references, which can be non-deterministic.
If you can't rule out OCaml: does it produce native code directly, or does it produce C or similar and then use GCC or similar to produce the native code?
OCaml produces native code directly. GCC is only used to link the final object files together.
Rich.
Core was generated by `bin/coqc -coqlib . -q -native-compiler yes theories/FSets/FMapFullAVL.v'. Program terminated with signal SIGSEGV, Segmentation fault. #0 0x0000556989736c94 in ?? () (gdb) t a a bt
Thread 1 (Thread 0x7f7453e1ec00 (LWP 2710541)): #0 0x0000556989736c94 in ?? () #1 0x0000000000000001 in ?? () #2 0x0000000000000001 in ?? () #3 0x00007f7453440ec8 in ?? () #4 0xfffffffffffffec0 in ?? () #5 0x00005569c42daba8 in ?? () #6 0x0000000000000004 in ?? () #7 0x0000000000000002 in ?? () #8 0x000000042d7f6c00 in ?? () #9 0x0000000000000001 in ?? ()
^------ probable bad stack frame
#10 0x00005569c35adc63 in caml_alloc_shr ()
^------ garbage collector https://github.com/ocaml/ocaml/blob/0756052841d0d3e6266c8c1ca4a608dc16c11441...
#11 0x00005569c35c00bc in caml_ephe_set_key ()
^------ weak ephemerons https://github.com/ocaml/ocaml/blob/0756052841d0d3e6266c8c1ca4a608dc16c11441...
#12 0x00005569c35419e8 in camlStdlib__obj__set_key_306 () #13 0x00005569c358da66 in camlStdlib__ephemeron__create_1033 ()
^------ OCaml functions Stdlib.Obj.set_key & Stdlib.Ephemeron.create
#14 0x00007ffd45de5508 in ?? () #15 0x00007ffd45de5500 in ?? () #16 0x00007f744b7cf7b0 in ?? () #17 0x0000000030124a25 in ?? () #18 0x00007f7453745e88 in ?? () #19 0x0000000000000225 in ?? () #20 0x4917d3992e9b8500 in ?? () #21 0x00005569c3a3c858 in camlCsymtable__9 () #22 0x0000000000000000 in ?? ()
This bug is VERY annoying! Although not very reproducible, I can usually hit it after about 4-6 hours of looping ‘fedpkg build’.
Considering the whole Coq build is quite lengthy you'd think that one of these commands would hit it much more quickly:
$ while true ; do gdb -ex 'set args -coqlib . -q -native-compiler yes theories/FSets/FMapFullAVL.v' -ex run bin/coqc -ex quit ; done
or
$ while bin/coqc -coqlib . -q -native-compiler yes theories/FSets/FMapFullAVL.v ; do date; done
But nope ...
My latest thing is to run the gdb command above _and_ run fedpkg build (in another directory) at the same time, in the hope that it's caused by system load.
Rich.
On Thu, Mar 26, 2020 at 12:00:14PM +0000, Richard W.M. Jones wrote:
This bug is VERY annoying! Although not very reproducible, I can usually hit it after about 4-6 hours of looping ‘fedpkg build’.
I mean ‘fedpkg local’. Local builds, not building in Koji.
Rich.
I don't know, now I can't even get the ‘fedpkg local’ to reproduce it :-(
Jerry, I suggest this bug is real, but is also likely to be a bug in Coq (most likely) or the OCaml runtime, possibly in the Weak module. You might have more luck asking the upstream developers for help.
As for what to do about Fedora 32, how about disabling _smp_mflags, on the basis that it might be load-related? That should at least reduce the non-determinism.
Rich.
On Thu, Mar 26, 2020 at 10:28 AM Richard W.M. Jones rjones@redhat.com wrote:
I don't know, now I can't even get the ‘fedpkg local’ to reproduce it :-(
Jerry, I suggest this bug is real, but is also likely to be a bug in Coq (most likely) or the OCaml runtime, possibly in the Weak module. You might have more luck asking the upstream developers for help.
As for what to do about Fedora 32, how about disabling _smp_mflags, on the basis that it might be load-related? That should at least reduce the non-determinism.
Okay, I will give that a try. I'm worried that even if I get a "good" build, that it's going to explode on the hapless Fedora users who install it.
Thank you for all the work you put into this, Richard. I appreciate the investigation. I'll keep poking at it, too.
On Fri, Mar 27, 2020 at 11:03 AM Jerry James loganjerry@gmail.com wrote:
Okay, I will give that a try.
Except I can't. We're already not using %_smp_mflags. Ugh.
On Fri, Mar 27, 2020 at 11:15:01AM -0600, Jerry James wrote:
On Fri, Mar 27, 2020 at 11:03 AM Jerry James loganjerry@gmail.com wrote:
Okay, I will give that a try.
Except I can't. We're already not using %_smp_mflags. Ugh.
We are. Did you trying putting '%global _smp_mflags -j1' somewhere near the top of the spec file?
Zbyszek
On Fri, Mar 27, 2020 at 11:30 AM Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl wrote:
We are. Did you trying putting '%global _smp_mflags -j1' somewhere near the top of the spec file?
We are? Where? The spec file does this to build:
make world VERBOSE=1
On Fri, Mar 27, 2020 at 11:36:38AM -0600, Jerry James wrote:
On Fri, Mar 27, 2020 at 11:30 AM Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl wrote:
We are. Did you trying putting '%global _smp_mflags -j1' somewhere near the top of the spec file?
We are? Where?
That is what the packaging guidelines say [1]:
Whenever possible, invocations of make should be done as %make_build
(which is defined as $ rpmbuild --showrc |grep make_build make_build %{__make} %{_make_output_sync} %{?_smp_mflags} %{_make_verbose}).
[1] https://docs.fedoraproject.org/en-US/packaging-guidelines/#_parallel_make
The spec file does this to build:
make world VERBOSE=1
So it seems that this particular Makefile enables parallelization internally. (make defaults -j1 unless told otherwise). But it should be possible to somehow disable it wherever it is enabled.
Zbyszek
On Fri, Mar 27, 2020 at 11:15:01AM -0600, Jerry James wrote:
On Fri, Mar 27, 2020 at 11:03 AM Jerry James loganjerry@gmail.com wrote:
Okay, I will give that a try.
Except I can't. We're already not using %_smp_mflags. Ugh.
I realized that my build machine has MAKEFLAGS=-j24 which is why it's building coq in parallel (and quickly). So it seems like you could be using _smp_mflags.
Another thing I tried today was enabling debugging in the GC (see patch below). However it also didn't find any problems, so I'm still at a loss.
Rich.
diff --git a/coq.spec b/coq.spec index df2a847..1e590cc 100644 --- a/coq.spec +++ b/coq.spec @@ -164,6 +164,10 @@ done %global opt_option -bytecode-compiler yes -byte-only -coqide byte %endif
+# Use debug runtime. +sed -i 's/^(OCAMLC :=.*)/\1 -runtime-variant d/' Makefile.build +sed -i 's/^(OCAMLOPT :=.*)/\1 -runtime-variant d/' Makefile.build + %global coqdocdir %{?_pkgdocdir}%{!?_pkgdocdir:%{_docdir}/coq-%{version}} %global coqdatadir %{_libdir}/coq
On Thu, Mar 26, 2020 at 10:28 AM Richard W.M. Jones rjones@redhat.com wrote:
Jerry, I suggest this bug is real, but is also likely to be a bug in Coq (most likely) or the OCaml runtime, possibly in the Weak module. You might have more luck asking the upstream developers for help.
Coq issue: https://github.com/coq/coq/issues/11939
I looked around in the ocaml issue tracker and found a possibility:
https://github.com/ocaml/ocaml/issues/9391
Coq uses ephemerons.
On Fri, Mar 27, 2020 at 11:54:44AM -0600, Jerry James wrote:
On Thu, Mar 26, 2020 at 10:28 AM Richard W.M. Jones rjones@redhat.com wrote:
Jerry, I suggest this bug is real, but is also likely to be a bug in Coq (most likely) or the OCaml runtime, possibly in the Weak module. You might have more luck asking the upstream developers for help.
Coq issue: https://github.com/coq/coq/issues/11939
I looked around in the ocaml issue tracker and found a possibility:
https://github.com/ocaml/ocaml/issues/9391
Coq uses ephemerons.
Yup, and the stack trace shows the failure happening when creating an ephemeron.
Rich.
On Fri, Mar 27, 2020 at 2:41 PM Richard W.M. Jones rjones@redhat.com wrote:
Yup, and the stack trace shows the failure happening when creating an ephemeron.
On the other hand, a scratch build with coq upstream's final patch for ocaml 4.10.0 succeeded on the first try:
https://koji.fedoraproject.org/koji/taskinfo?taskID=42808115
Where this is an intermittent problem, I'll have to try more builds to be sure, but this looks promising.
On Fri, Mar 27, 2020 at 04:20:16PM -0600, Jerry James wrote:
On Fri, Mar 27, 2020 at 2:41 PM Richard W.M. Jones rjones@redhat.com wrote:
Yup, and the stack trace shows the failure happening when creating an ephemeron.
On the other hand, a scratch build with coq upstream's final patch for ocaml 4.10.0 succeeded on the first try:
https://koji.fedoraproject.org/koji/taskinfo?taskID=42808115
Where this is an intermittent problem, I'll have to try more builds to be sure, but this looks promising.
I'll run it in a loop overnight and see if I can make it crash.
Rich.
On Fri, Mar 27, 2020 at 4:41 PM Richard W.M. Jones rjones@redhat.com wrote:
I'll run it in a loop overnight and see if I can make it crash.
Good idea. Thank you!
On Fri, Mar 27, 2020 at 4:42 PM Jerry James loganjerry@gmail.com wrote:
Good idea. Thank you!
I got a segfault on s390x on the second build attempt. I'd like to investigate the ephemeron angle a bit, I think.
On Fri, Mar 27, 2020 at 05:55:42PM -0600, Jerry James wrote:
On Fri, Mar 27, 2020 at 4:42 PM Jerry James loganjerry@gmail.com wrote:
Good idea. Thank you!
I got a segfault on s390x on the second build attempt. I'd like to investigate the ephemeron angle a bit, I think.
No crashes here overnight, but I'll leave it running.
Rich.
On Wed, Mar 25, 2020 at 10:21 PM Stephen J. Turnbull stephen@xemacs.org wrote:
Hi Jerry! Long time no see.
Hi Stephen! It's good to hear from yet another person I've worked with via network for years.
I think Richard addressed most of the points you raised.
Where is the segfault? Is it in coqc or is it in the generated code or is it in the OCaml compiler or is it somewhere else?
It's in coqc. Richard's evidence seems to point to garbage collector involvement, with weak references possibly playing a role.
Stay healthy!
You, too, and likewise for everybody in the Fedora community.
Jerry James writes:
I think Richard addressed most of the points you raised.
Yes, and no new questions, either. So I think I'm done here (well, I'll spectate, but the internals of the OCaml compiler are well beyond my skillset :-).
Good luck!
Steve
-
Jerry James -
Paul Dufresne -
Richard W.M. Jones -
Stephen J. Turnbull -
Zbigniew Jędrzejewski-Szmek