Coq build issues in F32

[Jerry James]

I've been talking to Richard Jones about this privately, and he
suggested that a message to fedora-devel could be helpful.

I've been talking for weeks about updating the entire coq stack,
badgering people into swapping reviews with me, etc.  Today I finally
had all the bits in place to do the updates.  I built the whole stack
for Rawhide with no issues.

The same builds need to happen for F32, where parts of the stack
currently have broken deps.  But I can't get coq to build
successfully, even though it did in Rawhide.  It went like this:

Attempt 1: the x86_64 build segfaulted while running "bin/coqc -coqlib . -q
-native-compiler yes   theories/FSets/FMapFullAVL.v", where coqc is a
binary built from the coq sources.  Coq is a theorem prover.  The coqc
run is essentially checking that a series of proof steps indeed prove
a theorem.  The "-native-compiler yes" part means that the OCaml
compiler will be invoked to generate native code for parts of this
process.

I was puzzled by the segfault.  I decided to see if the segfault is
deterministic, so I launched...

Attempt 2: the x86_64 build succeeded.  The s390x build failed with an
illegal instruction error while running *the same command*.

Attempt 3: The s390x build again failed with an illegal instruction
error while running the same command, and aarch64 segfaulted while
running the same command.  All other arches succeeded.  I saved the
URL for this one:

https://koji.fedoraproject.org/koji/taskinfo?taskID=42768075

Either I got lucky when I launched the Rawhide build, or something is
fundamentally different between F32 and Rawhide.  Clearly something
nondeterministic is at work.  I have been unable to reproduce this
failure in mock so far, but will keep trying.

If anybody has any ideas, I am all ears.  Thank you,
-- 
Jerry James
http://www.jamezone.org/

[Paul Dufresne]

Le Wed, 25 Mar 2020 16:58:33 -0600,
Jerry James <loganjerry(a)gmail.com&gt; a écrit :
>  Either I got lucky when I launched the Rawhide build, or something
> is
>  fundamentally different between F32 and Rawhide.  Clearly something
>  nondeterministic is at work.  I have been unable to reproduce this
>  failure in mock so far, but will keep trying.
>  
>  If anybody has any ideas, I am all ears.  Thank you, 
Well... It ring a bell for me, because I kind of seeing what seems to
be weird bugs in F32... I am kind of suspecting a gcc bug giving some
weird things... but I am not a developer, know a bit of assembler but
not enough to make the link between C code and assembler.

Anyway, one hour a go, I got a segment violation error while doing
updates with dnfdragora... and it seems that the code just jump to a
low address outside of the program:
rh bug #1817269

The other bug that make me suspect something might be wrong with gcc
was a bug that make a segment violation (seems to access a bit lower
address than a .png font file) in a game package that have not been
updated in many years as far as I know:
rh bug #1816471
but in that case, frankly the C code seems a bit hard for me to take
time to try to understand it.

What puzzle me in this case, is why a very old code would have stop to
work... that's why I came to the idea that maybe this is the generated
code itself that changed.

... yeah, I know, might have no link at all with your Coq build... I
don't know.

[Jerry James]

On Wed, Mar 25, 2020 at 8:57 PM Paul Dufresne via devel
<devel(a)lists.fedoraproject.org&gt; wrote:
>  ... yeah, I know, might have no link at all with your Coq build... I
>  don't know. 
Maybe ... but that sure is suspicious.  I recall hearing that OpenJDK
is also suffering from weird segfaults.  Hmmmm.....

Thanks for the input, Paul.  I'll keep investigating.
-- 
Jerry James
http://www.jamezone.org/

[Jerry James]

On Wed, Mar 25, 2020 at 4:58 PM Jerry James <loganjerry(a)gmail.com&gt; wrote:
>  Either I got lucky when I launched the Rawhide build, or something
> is
>  fundamentally different between F32 and Rawhide.  Clearly something
>  nondeterministic is at work.  I have been unable to reproduce this
>  failure in mock so far, but will keep trying. 
I did a scratch build in Rawhide, just to see if this morning's
successful build was a fluke.  It succeeded on all architectures:

https://koji.fedoraproject.org/koji/taskinfo?taskID=42770190

So far, 2 out of 2 Rawhide builds succeeded, and 3 out of 3 F32 builds
ended with either a segfault or a bad instruction on at least 1
architecture.  Here are the side-by-side differences in the packages
in the aarch64 buildroots (since aarch64 was one of the failing
architectures on my last attempt), Rawhide on the left, F32 on the
right.  You'll want to view this with a fixed-width font.

adwaita-cursor-theme-3.36.0-1.fc33.noarch
adwaita-cursor-theme-3.36.0-1.fc32.noarch
adwaita-icon-theme-3.36.0-1.fc33.noarch
adwaita-icon-theme-3.36.0-1.fc32.noarch
annobin-9.12-3.fc33.aarch64
annobin-9.06-4.fc32.aarch64
at-spi2-atk-2.34.2-1.fc33.aarch64
at-spi2-atk-2.34.2-1.fc32.aarch64
at-spi2-atk-devel-2.34.2-1.fc33.aarch64
at-spi2-atk-devel-2.34.2-1.fc32.aarch64
at-spi2-core-2.36.0-1.fc33.aarch64
at-spi2-core-2.36.0-1.fc32.aarch64
at-spi2-core-devel-2.36.0-1.fc33.aarch64
at-spi2-core-devel-2.36.0-1.fc32.aarch64
audit-libs-3.0-0.19.20191104git1c2f876.fc33.aarch64
audit-libs-3.0-0.18.20191104git1c2f876.fc32.aarch64
avahi-libs-0.8-2.fc33.aarch64
avahi-libs-0.7-23.fc32.aarch64
binutils-2.34-2.fc33.aarch64
binutils-2.34-2.fc32.aarch64
binutils-gold-2.34-2.fc33.aarch64
binutils-gold-2.34-2.fc32.aarch64
cmake-filesystem-3.17.0-1.fc33.aarch64
cmake-filesystem-3.16.4-1.fc32.aarch64
coreutils-8.32-3.fc33.aarch64
coreutils-8.31-10.fc32.aarch64
coreutils-common-8.32-3.fc33.aarch64
coreutils-common-8.31-10.fc32.aarch64
cpp-10.0.1-0.9.fc33.aarch64
cpp-10.0.1-0.9.fc32.aarch64
crypto-policies-20200312-1.git3ae59d2.fc33.noarch
crypto-policies-20191128-5.gitcd267a5.fc32.noarch
cryptsetup-libs-2.3.1-1.fc33.aarch64
cryptsetup-libs-2.3.0-1.fc32.aarch64
curl-7.69.0-2.fc33.aarch64
curl-7.69.1-1.fc32.aarch64
cyrus-sasl-lib-2.1.27-4.fc33.aarch64
cyrus-sasl-lib-2.1.27-3.fc32.aarch64
dbus-1:1.12.16-5.fc33.aarch64
dbus-1:1.12.16-4.fc32.aarch64
dbus-broker-22-1.fc33.aarch64
dbus-broker-22-1.fc32.aarch64
dbus-common-1:1.12.16-5.fc33.noarch
dbus-common-1:1.12.16-4.fc32.noarch
dbus-devel-1:1.12.16-5.fc33.aarch64
dbus-devel-1:1.12.16-4.fc32.aarch64
dbus-libs-1:1.12.16-5.fc33.aarch64
dbus-libs-1:1.12.16-4.fc32.aarch64
dejavu-sans-fonts-2.37-7.fc33.noarch
dejavu-sans-fonts-2.37-7.fc32.noarch
fedora-gpg-keys-33-0.3.noarch
fedora-gpg-keys-32-0.7.noarch
fedora-release-33-0.3.noarch
fedora-release-32-0.8.noarch
fedora-release-common-33-0.3.noarch
fedora-release-common-32-0.8.noarch
fedora-repos-33-0.3.noarch
fedora-repos-32-0.7.noarch
fedora-repos-rawhide-33-0.3.noarch
file-5.38-3.fc33.aarch64
file-5.38-2.fc32.aarch64
file-libs-5.38-3.fc33.aarch64
file-libs-5.38-2.fc32.aarch64
fontconfig-2.13.92-8.fc33.aarch64
fontconfig-2.13.92-8.fc32.aarch64
fontconfig-devel-2.13.92-8.fc33.aarch64
fontconfig-devel-2.13.92-8.fc32.aarch64
fonts-filesystem-2.0.3-1.fc33.noarch
fonts-filesystem-2.0.3-1.fc32.noarch
fonts-srpm-macros-2.0.3-1.fc33.noarch
fonts-srpm-macros-2.0.3-1.fc32.noarch
fribidi-1.0.9-1.fc33.aarch64
fribidi-1.0.9-1.fc32.aarch64
fribidi-devel-1.0.9-1.fc33.aarch64
fribidi-devel-1.0.9-1.fc32.aarch64
gcc-10.0.1-0.9.fc33.aarch64
gcc-10.0.1-0.9.fc32.aarch64
gdb-minimal-9.1-4.fc33.aarch64
gdb-minimal-9.1-3.fc32.aarch64
ghostscript-9.50-1.fc33.aarch64
ghostscript-9.50-1.fc32.aarch64
ghostscript-core-9.50-1.fc33.aarch64
ghostscript-core-9.50-1.fc32.aarch64
ghostscript-tools-fonts-9.50-1.fc33.aarch64
ghostscript-tools-fonts-9.50-1.fc32.aarch64
ghostscript-tools-printing-9.50-1.fc33.aarch64
ghostscript-tools-printing-9.50-1.fc32.aarch64
glib-networking-2.64.0-1.fc33.aarch64
glib-networking-2.64.0-1.fc32.aarch64
glib2-2.64.1-1.fc33.aarch64
glib2-2.64.1-1.fc32.aarch64
glib2-devel-2.64.1-1.fc33.aarch64
glib2-devel-2.64.1-1.fc32.aarch64
glibc-2.31.9000-4.fc33.aarch64
glibc-2.31-1.fc32.aarch64
glibc-common-2.31.9000-4.fc33.aarch64
glibc-common-2.31-1.fc32.aarch64
glibc-devel-2.31.9000-4.fc33.aarch64
glibc-devel-2.31-1.fc32.aarch64
glibc-headers-2.31.9000-4.fc33.aarch64
glibc-headers-2.31-1.fc32.aarch64
glibc-minimal-langpack-2.31.9000-4.fc33.aarch64
glibc-minimal-langpack-2.31-1.fc32.aarch64
gmp-1:6.2.0-1.fc33.aarch64
gmp-1:6.1.2-13.fc32.aarch64
google-droid-sans-fonts-20200215-3.fc33.noarch
google-droid-sans-fonts-20200215-3.fc32.noarch
google-roboto-slab-fonts-1.100263-0.12.20150923git.fc33.noarch
google-roboto-slab-fonts-1.100263-0.11.20150923git.fc32.noarch
groff-base-1.22.3-24.fc33.aarch64
groff-base-1.22.3-21.fc32.aarch64
gsettings-desktop-schemas-3.36.0-1.fc33.aarch64
gsettings-desktop-schemas-3.36.0-1.fc32.aarch64
gtk-update-icon-cache-3.24.14-1.fc33.aarch64
gtk-update-icon-cache-3.24.14-1.fc32.aarch64
gtk3-3.24.14-1.fc33.aarch64
gtk3-3.24.14-1.fc32.aarch64
gtk3-devel-3.24.14-1.fc33.aarch64
gtk3-devel-3.24.14-1.fc32.aarch64

guile-5:2.0.14-19.fc32.aarch64
harfbuzz-2.6.4-4.fc33.aarch64
harfbuzz-2.6.4-3.fc32.aarch64
harfbuzz-devel-2.6.4-4.fc33.aarch64
harfbuzz-devel-2.6.4-3.fc32.aarch64
harfbuzz-icu-2.6.4-4.fc33.aarch64
harfbuzz-icu-2.6.4-3.fc32.aarch64
hevea-2.33-2.fc33.aarch64
hevea-2.33-2.fc32.1.aarch64
info-6.7-7.fc33.aarch64
info-6.7-6.fc32.aarch64
iptables-libs-1.8.4-7.fc33.aarch64
iptables-libs-1.8.4-7.fc32.aarch64
java-1.8.0-openjdk-headless-1:1.8.0.242.b08-2.fc33.aarch64
java-1.8.0-openjdk-headless-1:1.8.0.242.b08-1.fc32.aarch64
json-c-0.13.1-9.fc33.aarch64
json-c-0.13.1-9.fc32.aarch64
json-glib-1.4.4-4.fc32.aarch64
json-glib-1.4.4-4.fc32.aarch64
jsonp-1.0.4-11.fc32.noarch
jsonp-1.0.4-11.fc32.noarch
kernel-headers-5.6.0-0.rc7.git0.1.fc33.aarch64
kernel-headers-5.6.0-0.rc7.git0.1.fc32.aarch64
kmod-libs-27-2.fc33.aarch64
kmod-libs-27-1.fc32.aarch64
krb5-libs-1.18-7.fc33.aarch64
krb5-libs-1.18-1.fc32.aarch64
langpacks-core-font-en-3.0-3.fc33.noarch
langpacks-core-font-en-3.0-3.fc32.noarch
latexmk-4.69-1.fc33.noarch
latexmk-4.69-1.fc32.noarch
libarchive-3.4.2-1.fc33.aarch64
libarchive-3.4.2-1.fc32.aarch64
libasan-10.0.1-0.9.fc33.aarch64
libasan-10.0.1-0.9.fc32.aarch64
libatomic-10.0.1-0.9.fc33.aarch64
libatomic-10.0.1-0.9.fc32.aarch64
libblkid-2.35.1-7.fc33.aarch64
libblkid-2.35.1-4.fc32.aarch64
libblkid-devel-2.35.1-7.fc33.aarch64
libblkid-devel-2.35.1-4.fc32.aarch64
libcom_err-1.45.6-1.fc33.aarch64
libcom_err-1.45.5-3.fc32.aarch64
libcurl-7.69.0-2.fc33.aarch64
libcurl-7.69.1-1.fc32.aarch64
libevent-2.1.8-8.fc32.aarch64
libfdisk-2.35.1-7.fc33.aarch64
libfdisk-2.35.1-4.fc32.aarch64
libgcc-10.0.1-0.9.fc33.aarch64
libgcc-10.0.1-0.9.fc32.aarch64
libgfortran-10.0.1-0.9.fc33.aarch64
libgfortran-10.0.1-0.9.fc32.aarch64
libglvnd-1:1.3.1-1.fc33.aarch64
libglvnd-1:1.3.1-1.fc32.aarch64
libglvnd-core-devel-1:1.3.1-1.fc33.aarch64
libglvnd-core-devel-1:1.3.1-1.fc32.aarch64
libglvnd-devel-1:1.3.1-1.fc33.aarch64
libglvnd-devel-1:1.3.1-1.fc32.aarch64
libglvnd-egl-1:1.3.1-1.fc33.aarch64
libglvnd-egl-1:1.3.1-1.fc32.aarch64
libglvnd-gles-1:1.3.1-1.fc33.aarch64
libglvnd-gles-1:1.3.1-1.fc32.aarch64
libglvnd-glx-1:1.3.1-1.fc33.aarch64
libglvnd-glx-1:1.3.1-1.fc32.aarch64
libglvnd-opengl-1:1.3.1-1.fc33.aarch64
libglvnd-opengl-1:1.3.1-1.fc32.aarch64
libgomp-10.0.1-0.9.fc33.aarch64
libgomp-10.0.1-0.9.fc32.aarch64
libgpg-error-1.37-1.fc33.aarch64
libgpg-error-1.36-3.fc32.aarch64
libgs-9.50-1.fc33.aarch64
libgs-9.50-1.fc32.aarch64
libgusb-0.3.4-1.fc33.aarch64
libgusb-0.3.4-1.fc32.aarch64
libmount-2.35.1-7.fc33.aarch64
libmount-2.35.1-4.fc32.aarch64
libmount-devel-2.35.1-7.fc33.aarch64
libmount-devel-2.35.1-4.fc32.aarch64
libpcap-14:1.9.1-3.fc33.aarch64
libpcap-14:1.9.1-3.fc32.aarch64
libpq-12.2-1.fc33.aarch64
libpq-12.2-1.fc32.aarch64
libproxy-0.4.15-18.fc33.aarch64
libproxy-0.4.15-17.fc32.aarch64
libselinux-3.0-4.fc33.aarch64
libselinux-3.0-3.fc32.aarch64
libselinux-devel-3.0-4.fc33.aarch64
libselinux-devel-3.0-3.fc32.aarch64
libsemanage-3.0-3.fc32.aarch64
libsemanage-3.0-3.fc32.aarch64
libsmartcols-2.35.1-7.fc33.aarch64
libsmartcols-2.35.1-4.fc32.aarch64
libsoup-2.70.0-1.fc33.aarch64
libsoup-2.70.0-1.fc32.aarch64
libstdc++-10.0.1-0.9.fc33.aarch64
libstdc++-10.0.1-0.9.fc32.aarch64
libtirpc-1.2.5-1.rc2.fc33.aarch64
libtirpc-1.2.5-1.rc2.fc32.aarch64
libubsan-10.0.1-0.9.fc33.aarch64
libubsan-10.0.1-0.9.fc32.aarch64
libuuid-2.35.1-7.fc33.aarch64
libuuid-2.35.1-4.fc32.aarch64
libwayland-client-1.18.0-1.fc33.aarch64
libwayland-client-1.18.0-1.fc32.aarch64
libwayland-cursor-1.18.0-1.fc33.aarch64
libwayland-cursor-1.18.0-1.fc32.aarch64
libwayland-egl-1.18.0-1.fc33.aarch64
libwayland-egl-1.18.0-1.fc32.aarch64
libwayland-server-1.18.0-1.fc33.aarch64
libwayland-server-1.18.0-1.fc32.aarch64
libxcrypt-4.4.15-1.fc33.aarch64
libxcrypt-4.4.15-1.fc32.aarch64
libxcrypt-devel-4.4.15-1.fc33.aarch64
libxcrypt-devel-4.4.15-1.fc32.aarch64
libxml2-2.9.10-4.fc33.aarch64
libxml2-2.9.10-3.fc32.aarch64
libxml2-devel-2.9.10-4.fc33.aarch64
libxml2-devel-2.9.10-3.fc32.aarch64
libxslt-1.1.34-1.fc33.aarch64
libxslt-1.1.34-1.fc32.aarch64
make-1:4.3-1.fc33.aarch64
make-1:4.2.1-16.fc32.aarch64
mesa-libEGL-20.0.2-1.fc33.aarch64
mesa-libEGL-20.0.2-1.fc32.aarch64
mesa-libGL-20.0.2-1.fc33.aarch64
mesa-libGL-20.0.2-1.fc32.aarch64
mesa-libgbm-20.0.2-1.fc33.aarch64
mesa-libgbm-20.0.2-1.fc32.aarch64
mesa-libglapi-20.0.2-1.fc33.aarch64
mesa-libglapi-20.0.2-1.fc32.aarch64
ncurses-6.2-1.20200222.fc33.aarch64
ncurses-6.1-15.20191109.fc32.aarch64
ncurses-base-6.2-1.20200222.fc33.noarch
ncurses-base-6.1-15.20191109.fc32.noarch
ncurses-libs-6.2-1.20200222.fc33.aarch64
ncurses-libs-6.1-15.20191109.fc32.aarch64
netpbm-10.89.00-3.fc33.aarch64
netpbm-10.89.00-2.fc32.aarch64
netpbm-progs-10.89.00-3.fc33.aarch64
netpbm-progs-10.89.00-2.fc32.aarch64
nspr-4.25.0-1.fc33.aarch64
nspr-4.25.0-1.fc32.aarch64
nss-3.50.0-2.fc33.aarch64
nss-3.50.0-2.fc32.aarch64
nss-softokn-3.50.0-2.fc33.aarch64
nss-softokn-3.50.0-2.fc32.aarch64
nss-softokn-freebl-3.50.0-2.fc33.aarch64
nss-softokn-freebl-3.50.0-2.fc32.aarch64
nss-sysinit-3.50.0-2.fc33.aarch64
nss-sysinit-3.50.0-2.fc32.aarch64
nss-util-3.50.0-2.fc33.aarch64
nss-util-3.50.0-2.fc32.aarch64
ocaml-4.10.0-2.fc33.aarch64
ocaml-4.10.0-2.fc32.aarch64
ocaml-cairo-2:0.6.1-3.fc33.aarch64
ocaml-cairo-2:0.6.1-3.fc32.1.aarch64
ocaml-cairo-devel-2:0.6.1-3.fc33.aarch64
ocaml-cairo-devel-2:0.6.1-3.fc32.1.aarch64
ocaml-compiler-libs-4.10.0-2.fc33.aarch64
ocaml-compiler-libs-4.10.0-2.fc32.aarch64
ocaml-findlib-1.8.1-12.fc33.aarch64
ocaml-findlib-1.8.1-12.fc32.1.aarch64
ocaml-lablgtk3-3.1.0-2.fc33.aarch64
ocaml-lablgtk3-3.1.0-2.fc32.aarch64
ocaml-lablgtk3-devel-3.1.0-2.fc33.aarch64
ocaml-lablgtk3-devel-3.1.0-2.fc32.aarch64
ocaml-lablgtk3-sourceview3-3.1.0-2.fc33.aarch64
ocaml-lablgtk3-sourceview3-3.1.0-2.fc32.aarch64
ocaml-lablgtk3-sourceview3-devel-3.1.0-2.fc33.aarch64
ocaml-lablgtk3-sourceview3-devel-3.1.0-2.fc32.aarch64
ocaml-num-1.1-21.fc33.aarch64
ocaml-num-1.1-21.fc32.1.aarch64
ocaml-num-devel-1.1-21.fc33.aarch64
ocaml-num-devel-1.1-21.fc32.1.aarch64
ocaml-runtime-4.10.0-2.fc33.aarch64
ocaml-runtime-4.10.0-2.fc32.aarch64
ocaml-srpm-macros-6-2.fc32.noarch
ocaml-srpm-macros-6-2.fc32.noarch
openblas-0.3.9-1.fc33.aarch64
openblas-0.3.9-1.fc32.aarch64
openblas-serial-0.3.9-1.fc33.aarch64
openblas-serial-0.3.9-1.fc32.aarch64
openjpeg2-2.3.1-6.fc33.aarch64
openjpeg2-2.3.1-6.fc32.aarch64
openssl-libs-1:1.1.1e-1.fc33.aarch64
openssl-libs-1:1.1.1d-7.fc32.aarch64
pam-1.3.1-24.fc33.aarch64
pam-1.3.1-23.fc32.aarch64
pari-2.11.3-1.fc33.aarch64
pari-2.11.3-1.fc32.aarch64
pcre-8.44-1.fc33.aarch64
pcre-8.44-1.fc32.aarch64
pcre-cpp-8.44-1.fc33.aarch64
pcre-cpp-8.44-1.fc32.aarch64
pcre-devel-8.44-1.fc33.aarch64
pcre-devel-8.44-1.fc32.aarch64
pcre-utf16-8.44-1.fc33.aarch64
pcre-utf16-8.44-1.fc32.aarch64
pcre-utf32-8.44-1.fc33.aarch64
pcre-utf32-8.44-1.fc32.aarch64
pcre2-10.34-9.fc33.aarch64
pcre2-10.34-7.fc32.aarch64
pcre2-devel-10.34-9.fc33.aarch64
pcre2-devel-10.34-7.fc32.aarch64
pcre2-syntax-10.34-9.fc33.noarch
pcre2-utf16-10.34-9.fc33.aarch64
pcre2-utf16-10.34-7.fc32.aarch64
pcre2-utf32-10.34-9.fc33.aarch64
pcre2-utf32-10.34-7.fc32.aarch64
perl-B-1.76-452.fc33.aarch64
perl-Class-Struct-0.65-452.fc33.noarch
perl-Data-Dumper-2.174-443.fc32.aarch64
perl-Data-Dumper-2.174-443.fc32.aarch64
perl-Encode-4:3.05-444.fc33.aarch64
perl-Encode-4:3.02-441.fc32.aarch64
perl-English-1.10-452.fc33.noarch
perl-Errno-1.30-452.fc33.aarch64
perl-Errno-1.30-451.fc32.aarch64
perl-Fcntl-1.13-452.fc33.aarch64
perl-File-Basename-2.85-452.fc33.noarch
perl-File-Copy-2.34-452.fc33.noarch
perl-File-Find-1.36-452.fc33.noarch
perl-File-stat-1.08-452.fc33.noarch
perl-FileHandle-2.03-452.fc33.noarch
perl-Getopt-Std-1.12-452.fc33.noarch
perl-IO-1.40-452.fc33.aarch64
perl-IO-1.40-451.fc32.aarch64
perl-IPC-Open3-1.20-452.fc33.noarch
perl-POSIX-1.88-452.fc33.aarch64
perl-Pod-Perldoc-3.28.01-444.fc33.noarch
perl-Pod-Perldoc-3.28.01-443.fc32.noarch
perl-Pod-Usage-4:1.70-1.fc33.noarch
perl-Pod-Usage-4:1.69-440.fc32.noarch
perl-SelectSaver-1.02-452.fc33.noarch
perl-Symbol-1.08-452.fc33.noarch
perl-Tie-4.6-452.fc33.noarch

perl-Unicode-Normalize-1.26-440.fc32.aarch64
perl-XML-Parser-2.46-3.fc33.aarch64
perl-XML-Parser-2.46-2.fc32.aarch64
perl-encoding-4:2.22-444.fc33.aarch64
perl-encoding-4:2.22-441.fc32.aarch64
perl-if-0.60.800-452.fc33.noarch
perl-interpreter-4:5.30.2-452.fc33.aarch64
perl-interpreter-4:5.30.2-451.fc32.aarch64
perl-libs-4:5.30.2-452.fc33.aarch64
perl-libs-4:5.30.2-451.fc32.aarch64
perl-macros-4:5.30.2-452.fc33.aarch64
perl-macros-4:5.30.2-451.fc32.aarch64
perl-mro-1.22-452.fc33.aarch64
perl-open-1.11-452.fc33.noarch
perl-open-1.11-451.fc32.noarch
perl-overload-1.30-452.fc33.noarch
perl-overloading-0.02-452.fc33.noarch
perl-subs-1.03-452.fc33.noarch

perl-threads-1:2.22-442.fc32.aarch64

perl-threads-shared-1.60-441.fc32.aarch64
perl-vars-1.05-452.fc33.noarch
python-pip-wheel-20.0.2-2.fc33.noarch
python-pip-wheel-19.3.1-2.fc32.noarch
python-setuptools-wheel-46.0.0-1.fc33.noarch
python-setuptools-wheel-41.6.0-2.fc32.noarch
python3-3.8.2-2.fc33.aarch64
python3-3.8.2-2.fc32.aarch64
python3-devel-3.8.2-2.fc33.aarch64
python3-devel-3.8.2-2.fc32.aarch64
python3-libs-3.8.2-2.fc33.aarch64
python3-libs-3.8.2-2.fc32.aarch64
python3-orderedset-2.0.3-1.fc33.aarch64
python3-orderedset-2.0.3-1.fc32.aarch64
python3-packaging-20.3-1.fc33.noarch
python3-packaging-20.1-2.fc32.noarch
python3-pyyaml-5.3.1-1.fc33.aarch64
python3-pyyaml-5.3-4.fc32.aarch64
python3-requests-2.23.0-1.fc33.noarch
python3-requests-2.22.0-8.fc32.noarch
python3-setuptools-46.0.0-1.fc33.noarch
python3-setuptools-41.6.0-2.fc32.noarch
python3-sphinx_rtd_theme-0.4.3-11.fc33.noarch
python3-sphinx_rtd_theme-0.4.3-10.fc32.noarch
python3-urllib3-1.25.8-1.fc33.noarch
python3-urllib3-1.25.7-3.fc32.noarch
redhat-rpm-config-153-1.fc33.noarch
redhat-rpm-config-150-1.fc32.noarch
rust-srpm-macros-13-2.fc33.noarch
rust-srpm-macros-13-2.fc32.noarch
sed-4.8-1.fc33.aarch64
sed-4.5-5.fc32.aarch64
shadow-utils-2:4.8.1-2.fc33.aarch64
shadow-utils-2:4.8-4.fc32.aarch64
shared-mime-info-1.15-3.fc33.aarch64
shared-mime-info-1.15-3.fc32.aarch64
systemd-245.2-1.fc33.aarch64
systemd-245.2-1.fc32.aarch64
systemd-libs-245.2-1.fc33.aarch64
systemd-libs-245.2-1.fc32.aarch64
systemd-pam-245.2-1.fc33.aarch64
systemd-pam-245.2-1.fc32.aarch64
systemd-rpm-macros-245.2-1.fc33.noarch
systemd-rpm-macros-245.2-1.fc32.noarch
texlive-adjustbox-9:svn49596-19.fc33.noarch
texlive-adjustbox-9:svn49596-19.fc32.noarch
texlive-ae-9:svn15878.1.4-19.fc33.noarch
texlive-ae-9:svn15878.1.4-19.fc32.noarch
texlive-algorithms-9:svn42428-19.fc33.noarch
texlive-algorithms-9:svn42428-19.fc32.noarch
texlive-amscls-9:svn46099-19.fc33.noarch
texlive-amscls-9:svn46099-19.fc32.noarch
texlive-amsfonts-9:svn29208.3.04-19.fc33.noarch
texlive-amsfonts-9:svn29208.3.04-19.fc32.noarch
texlive-amsmath-9:svn49390-19.fc33.noarch
texlive-amsmath-9:svn49390-19.fc32.noarch
texlive-anysize-9:svn15878.0-19.fc33.noarch
texlive-anysize-9:svn15878.0-19.fc32.noarch
texlive-attachfile-9:svn42099-19.fc33.noarch
texlive-attachfile-9:svn42099-19.fc32.noarch
texlive-auto-pst-pdf-9:svn23723.0.6-19.fc33.noarch
texlive-auto-pst-pdf-9:svn23723.0.6-19.fc32.noarch
texlive-avantgar-9:svn31835.0-19.fc33.noarch
texlive-avantgar-9:svn31835.0-19.fc32.noarch
texlive-babel-9:svn51681-19.fc33.noarch
texlive-babel-9:svn51681-19.fc32.noarch
texlive-babel-english-9:svn44495-19.fc33.noarch
texlive-babel-english-9:svn44495-19.fc32.noarch
texlive-babelbib-9:svn50354-19.fc33.noarch
texlive-babelbib-9:svn50354-19.fc32.noarch
texlive-beamer-9:svn51711-19.fc33.noarch
texlive-beamer-9:svn51711-19.fc32.noarch
texlive-bera-9:svn20031.0-19.fc33.noarch
texlive-bera-9:svn20031.0-19.fc32.noarch
texlive-beton-9:svn15878.0-19.fc33.noarch
texlive-beton-9:svn15878.0-19.fc32.noarch
texlive-bidi-9:svn51377-19.fc33.noarch
texlive-bidi-9:svn51377-19.fc32.noarch
texlive-bookman-9:svn31835.0-19.fc33.noarch
texlive-bookman-9:svn31835.0-19.fc32.noarch
texlive-booktabs-9:svn40846-19.fc33.noarch
texlive-booktabs-9:svn40846-19.fc32.noarch
texlive-breakurl-9:svn29901.1.40-19.fc33.noarch
texlive-breakurl-9:svn29901.1.40-19.fc32.noarch
texlive-breqn-9:svn43071-19.fc33.noarch
texlive-breqn-9:svn43071-19.fc32.noarch
texlive-capt-of-9:svn29803.0-19.fc33.noarch
texlive-capt-of-9:svn29803.0-19.fc32.noarch
texlive-caption-9:svn47968-19.fc33.noarch
texlive-caption-9:svn47968-19.fc32.noarch
texlive-carlisle-9:svn47876-19.fc33.noarch
texlive-carlisle-9:svn47876-19.fc32.noarch
texlive-changepage-9:svn15878.1.0c-19.fc33.noarch
texlive-changepage-9:svn15878.1.0c-19.fc32.noarch
texlive-charter-9:svn15878.0-19.fc33.noarch
texlive-charter-9:svn15878.0-19.fc32.noarch
texlive-chngcntr-9:svn47577-19.fc33.noarch
texlive-chngcntr-9:svn47577-19.fc32.noarch
texlive-cite-9:svn36428.5.5-19.fc33.noarch
texlive-cite-9:svn36428.5.5-19.fc32.noarch
texlive-cm-9:svn49028-19.fc33.noarch
texlive-cm-9:svn49028-19.fc32.noarch
texlive-cm-super-9:svn15878.0-19.fc33.noarch
texlive-cm-super-9:svn15878.0-19.fc32.noarch
texlive-cmap-9:svn42428-19.fc33.noarch
texlive-cmap-9:svn42428-19.fc32.noarch
texlive-cmextra-9:svn42428-19.fc33.noarch
texlive-cmextra-9:svn42428-19.fc32.noarch
texlive-collectbox-9:svn26557.0.4b-19.fc33.noarch
texlive-collectbox-9:svn26557.0.4b-19.fc32.noarch
texlive-collection-basic-9:svn51558-19.fc33.noarch
texlive-collection-basic-9:svn51558-19.fc32.noarch
texlive-collection-fontsrecommended-9:svn35830.0-19.fc33.noarch
texlive-collection-fontsrecommended-9:svn35830.0-19.fc32.noarch
texlive-collection-latex-9:svn41614-19.fc33.noarch
texlive-collection-latex-9:svn41614-19.fc32.noarch
texlive-collection-latexrecommended-9:svn51780-19.fc33.noarch
texlive-collection-latexrecommended-9:svn51780-19.fc32.noarch
texlive-colorprofiles-9:svn49086-19.fc33.noarch
texlive-colorprofiles-9:svn49086-19.fc32.noarch
texlive-colortbl-9:svn49404-19.fc33.noarch
texlive-colortbl-9:svn49404-19.fc32.noarch
texlive-comment-9:svn41927-19.fc33.noarch
texlive-comment-9:svn41927-19.fc32.noarch
texlive-courier-9:svn35058.0-19.fc33.noarch
texlive-courier-9:svn35058.0-19.fc32.noarch
texlive-crop-9:svn15878.1.5-19.fc33.noarch
texlive-crop-9:svn15878.1.5-19.fc32.noarch
texlive-csquotes-9:svn51091-19.fc33.noarch
texlive-csquotes-9:svn51091-19.fc32.noarch
texlive-ctable-9:svn38672-19.fc33.noarch
texlive-ctable-9:svn38672-19.fc32.noarch
texlive-currfile-9:svn40725-19.fc33.noarch
texlive-currfile-9:svn40725-19.fc32.noarch
texlive-dehyph-9:svn48599-19.fc33.noarch
texlive-dehyph-9:svn48599-19.fc32.noarch
texlive-ec-9:svn25033.1.0-19.fc33.noarch
texlive-ec-9:svn25033.1.0-19.fc32.noarch
texlive-eepic-9:svn15878.1.1e-19.fc33.noarch
texlive-eepic-9:svn15878.1.1e-19.fc32.noarch
texlive-enctex-9:svn34957.0-19.fc33.noarch
texlive-enctex-9:svn34957.0-19.fc32.noarch
texlive-enumitem-9:svn51423-19.fc33.noarch
texlive-enumitem-9:svn51423-19.fc32.noarch
texlive-environ-9:svn33821.0.3-19.fc33.noarch
texlive-environ-9:svn33821.0.3-19.fc32.noarch
texlive-eso-pic-9:svn47694-19.fc33.noarch
texlive-eso-pic-9:svn47694-19.fc32.noarch
texlive-etex-9:svn37057.0-19.fc33.noarch
texlive-etex-9:svn37057.0-19.fc32.noarch
texlive-etex-pkg-9:svn41784-19.fc33.noarch
texlive-etex-pkg-9:svn41784-19.fc32.noarch
texlive-etoolbox-9:svn46602-19.fc33.noarch
texlive-etoolbox-9:svn46602-19.fc32.noarch
texlive-euenc-9:svn19795.0.1h-19.fc33.noarch
texlive-euenc-9:svn19795.0.1h-19.fc32.noarch
texlive-euler-9:svn42428-19.fc33.noarch
texlive-euler-9:svn42428-19.fc32.noarch
texlive-euro-9:svn22191.1.1-19.fc33.noarch
texlive-euro-9:svn22191.1.1-19.fc32.noarch
texlive-euro-ce-9:svn25714-19.fc33.noarch
texlive-euro-ce-9:svn25714-19.fc32.noarch
texlive-eurosym-9:svn17265.1.4_subrfix-19.fc33.noarch
texlive-eurosym-9:svn17265.1.4_subrfix-19.fc32.noarch
texlive-extsizes-9:svn17263.1.4a-19.fc33.noarch
texlive-extsizes-9:svn17263.1.4a-19.fc32.noarch
texlive-fancybox-9:svn18304.1.4-19.fc33.noarch
texlive-fancybox-9:svn18304.1.4-19.fc32.noarch
texlive-fancyhdr-9:svn49886-19.fc33.noarch
texlive-fancyhdr-9:svn49886-19.fc32.noarch
texlive-fancyref-9:svn15878.0.9c-19.fc33.noarch
texlive-fancyref-9:svn15878.0.9c-19.fc32.noarch
texlive-fancyvrb-9:svn49717-19.fc33.noarch
texlive-fancyvrb-9:svn49717-19.fc32.noarch
texlive-filecontents-9:svn47890-19.fc33.noarch
texlive-filecontents-9:svn47890-19.fc32.noarch
texlive-filehook-9:svn24280.0.5d-19.fc33.noarch
texlive-filehook-9:svn24280.0.5d-19.fc32.noarch
texlive-finstrut-9:svn21719.0.5-19.fc33.noarch
texlive-finstrut-9:svn21719.0.5-19.fc32.noarch
texlive-fix2col-9:svn38770-19.fc33.noarch
texlive-fix2col-9:svn38770-19.fc32.noarch
texlive-float-9:svn15878.1.3d-19.fc33.noarch
texlive-float-9:svn15878.1.3d-19.fc32.noarch
texlive-fncychap-9:svn20710.v1.34-19.fc33.noarch
texlive-fncychap-9:svn20710.v1.34-19.fc32.noarch
texlive-fontspec-9:svn50387-19.fc33.noarch
texlive-fontspec-9:svn50387-19.fc32.noarch
texlive-footmisc-9:svn23330.5.5b-19.fc33.noarch
texlive-footmisc-9:svn23330.5.5b-19.fc32.noarch
texlive-fp-9:svn49719-19.fc33.noarch
texlive-fp-9:svn49719-19.fc32.noarch
texlive-fpl-9:svn49603-19.fc33.noarch
texlive-fpl-9:svn49603-19.fc32.noarch
texlive-framed-9:svn26789.0.96-19.fc33.noarch
texlive-framed-9:svn26789.0.96-19.fc32.noarch
texlive-geometry-9:svn47638-19.fc33.noarch
texlive-geometry-9:svn47638-19.fc32.noarch
texlive-gnu-freefont-9:svn29349.0-19.fc33.noarch
texlive-gnu-freefont-9:svn29349.0-19.fc32.noarch
texlive-graphics-9:svn51733-19.fc33.noarch
texlive-graphics-9:svn51733-19.fc32.noarch
texlive-graphics-cfg-9:svn40269-19.fc33.noarch
texlive-graphics-cfg-9:svn40269-19.fc32.noarch
texlive-graphics-def-9:svn46267-19.fc33.noarch
texlive-graphics-def-9:svn46267-19.fc32.noarch
texlive-helvetic-9:svn31835.0-19.fc33.noarch
texlive-helvetic-9:svn31835.0-19.fc32.noarch
texlive-hyperref-9:svn51742-19.fc33.noarch
texlive-hyperref-9:svn51742-19.fc32.noarch
texlive-hyph-utf8-9:svn51186-19.fc33.noarch
texlive-hyph-utf8-9:svn51186-19.fc32.noarch
texlive-hyphen-base-9:svn51200-19.fc33.noarch
texlive-hyphen-base-9:svn51200-19.fc32.noarch
texlive-hyphenex-9:svn37354.0-19.fc33.noarch
texlive-hyphenex-9:svn37354.0-19.fc32.noarch
texlive-ifetex-9:svn47231-19.fc33.noarch
texlive-ifetex-9:svn47231-19.fc32.noarch
texlive-ifluatex-9:svn51770-19.fc33.noarch
texlive-ifluatex-9:svn51770-19.fc32.noarch
texlive-ifmtarg-9:svn47544-19.fc33.noarch
texlive-ifmtarg-9:svn47544-19.fc32.noarch
texlive-ifoddpage-9:svn40726-19.fc33.noarch
texlive-ifoddpage-9:svn40726-19.fc32.noarch
texlive-ifplatform-9:svn45533-19.fc33.noarch
texlive-ifplatform-9:svn45533-19.fc32.noarch
texlive-iftex-9:svn29654.0.2-19.fc33.noarch
texlive-iftex-9:svn29654.0.2-19.fc32.noarch
texlive-ifxetex-9:svn19685.0.5-19.fc33.noarch
texlive-ifxetex-9:svn19685.0.5-19.fc32.noarch
texlive-index-9:svn24099.4.1beta-19.fc33.noarch
texlive-index-9:svn24099.4.1beta-19.fc32.noarch
texlive-jknapltx-9:svn19440.0-19.fc33.noarch
texlive-jknapltx-9:svn19440.0-19.fc32.noarch
texlive-kastrup-9:svn15878.0-19.fc33.noarch
texlive-kastrup-9:svn15878.0-19.fc32.noarch
texlive-knuth-lib-9:svn35820.0-19.fc33.noarch
texlive-knuth-lib-9:svn35820.0-19.fc32.noarch
texlive-knuth-local-9:svn38627-19.fc33.noarch
texlive-knuth-local-9:svn38627-19.fc32.noarch
texlive-koma-script-9:svn49929-19.fc33.noarch
texlive-koma-script-9:svn49929-19.fc32.noarch
texlive-l3backend-9:svn52343-19.fc33.noarch
texlive-l3backend-9:svn52343-19.fc32.noarch
texlive-l3experimental-9:svn51546-19.fc33.noarch
texlive-l3experimental-9:svn51546-19.fc32.noarch
texlive-l3kernel-9:svn51745-19.fc33.noarch
texlive-l3kernel-9:svn51745-19.fc32.noarch
texlive-l3packages-9:svn51546-19.fc33.noarch
texlive-l3packages-9:svn51546-19.fc32.noarch
texlive-latex-base-dev-9:svn51730-19.fc33.noarch
texlive-latex-base-dev-9:svn51730-19.fc32.noarch
texlive-latex-bin-dev-9:svn51839-19.fc33.noarch
texlive-latex-bin-dev-9:svn51839-19.fc32.noarch
texlive-latex-fonts-9:svn28888.0-19.fc33.noarch
texlive-latex-fonts-9:svn28888.0-19.fc32.noarch
texlive-latex-graphics-dev-9:svn51730-19.fc33.noarch
texlive-latex-graphics-dev-9:svn51730-19.fc32.noarch
texlive-latexbug-9:svn49573-19.fc33.noarch
texlive-latexbug-9:svn49573-19.fc32.noarch
texlive-latexconfig-9:svn45777-19.fc33.noarch
texlive-latexconfig-9:svn45777-19.fc32.noarch
texlive-lineno-9:svn21442.4.41-19.fc33.noarch
texlive-lineno-9:svn21442.4.41-19.fc32.noarch
texlive-listings-9:svn50175-19.fc33.noarch
texlive-listings-9:svn50175-19.fc32.noarch
texlive-lm-9:svn48145-19.fc33.noarch
texlive-lm-9:svn48145-19.fc32.noarch
texlive-lm-math-9:svn36915.1.959-19.fc33.noarch
texlive-lm-math-9:svn36915.1.959-19.fc32.noarch
texlive-ltabptch-9:svn17533.1.74d-19.fc33.noarch
texlive-ltabptch-9:svn17533.1.74d-19.fc32.noarch
texlive-ltxmisc-9:svn21927.0-19.fc33.noarch
texlive-ltxmisc-9:svn21927.0-19.fc32.noarch
texlive-lua-alt-getopt-9:svn29349.0.7.0-19.fc33.noarch
texlive-lua-alt-getopt-9:svn29349.0.7.0-19.fc32.noarch
texlive-luabidi-9:svn30790.0.2-19.fc33.noarch
texlive-luabidi-9:svn30790.0.2-19.fc32.noarch
texlive-lualatex-math-9:svn49904-19.fc33.noarch
texlive-lualatex-math-9:svn49904-19.fc32.noarch
texlive-lualibs-9:svn51642-19.fc33.noarch
texlive-lualibs-9:svn51642-19.fc32.noarch
texlive-luatexbase-9:svn38550-19.fc33.noarch
texlive-luatexbase-9:svn38550-19.fc32.noarch
texlive-makecmds-9:svn15878.0-19.fc33.noarch
texlive-makecmds-9:svn15878.0-19.fc32.noarch
texlive-manfnt-font-9:svn45777-19.fc33.noarch
texlive-manfnt-font-9:svn45777-19.fc32.noarch
texlive-marginnote-9:svn48383-19.fc33.noarch
texlive-marginnote-9:svn48383-19.fc32.noarch
texlive-marvosym-9:svn29349.2.2a-19.fc33.noarch
texlive-marvosym-9:svn29349.2.2a-19.fc32.noarch
texlive-mathpazo-9:svn15878.1.003-19.fc33.noarch
texlive-mathpazo-9:svn15878.1.003-19.fc32.noarch
texlive-mathspec-9:svn42773-19.fc33.noarch
texlive-mathspec-9:svn42773-19.fc32.noarch
texlive-mathtools-9:svn51788-19.fc33.noarch
texlive-mathtools-9:svn51788-19.fc32.noarch
texlive-mdwtools-9:svn15878.1.05.4-19.fc33.noarch
texlive-mdwtools-9:svn15878.1.05.4-19.fc32.noarch
texlive-memoir-9:svn49394-19.fc33.noarch
texlive-memoir-9:svn49394-19.fc32.noarch
texlive-metalogo-9:svn18611.0.12-19.fc33.noarch
texlive-metalogo-9:svn18611.0.12-19.fc32.noarch
texlive-mflogo-9:svn42428-19.fc33.noarch
texlive-mflogo-9:svn42428-19.fc32.noarch
texlive-mflogo-font-9:svn36898.1.002-19.fc33.noarch
texlive-mflogo-font-9:svn36898.1.002-19.fc32.noarch
texlive-mfnfss-9:svn46036-19.fc33.noarch
texlive-mfnfss-9:svn46036-19.fc32.noarch
texlive-microtype-9:svn50187-19.fc33.noarch
texlive-microtype-9:svn50187-19.fc32.noarch
texlive-mnsymbol-9:svn18651.1.4-19.fc33.noarch
texlive-mnsymbol-9:svn18651.1.4-19.fc32.noarch
texlive-moreverb-9:svn22126.2.3a-19.fc33.noarch
texlive-moreverb-9:svn22126.2.3a-19.fc32.noarch
texlive-mparhack-9:svn15878.1.4-19.fc33.noarch
texlive-mparhack-9:svn15878.1.4-19.fc32.noarch
texlive-ms-9:svn51784-19.fc33.noarch
texlive-ms-9:svn51784-19.fc32.noarch
texlive-multido-9:svn18302.1.42-19.fc33.noarch
texlive-multido-9:svn18302.1.42-19.fc32.noarch
texlive-multirow-9:svn51278-19.fc33.noarch
texlive-multirow-9:svn51278-19.fc32.noarch
texlive-natbib-9:svn20668.8.31b-19.fc33.noarch
texlive-natbib-9:svn20668.8.31b-19.fc32.noarch
texlive-ncntrsbk-9:svn31835.0-19.fc33.noarch
texlive-ncntrsbk-9:svn31835.0-19.fc32.noarch
texlive-needspace-9:svn29601.1.3d-19.fc33.noarch
texlive-needspace-9:svn29601.1.3d-19.fc32.noarch
texlive-ntgclass-9:svn15878.2.1a-19.fc33.noarch
texlive-ntgclass-9:svn15878.2.1a-19.fc32.noarch
texlive-palatino-9:svn31835.0-19.fc33.noarch
texlive-palatino-9:svn31835.0-19.fc32.noarch
texlive-paralist-9:svn43021-19.fc33.noarch
texlive-paralist-9:svn43021-19.fc32.noarch
texlive-parallel-9:svn15878.0-19.fc33.noarch
texlive-parallel-9:svn15878.0-19.fc32.noarch
texlive-parskip-9:svn49730-19.fc33.noarch
texlive-parskip-9:svn49730-19.fc32.noarch
texlive-pdfpages-9:svn45659-19.fc33.noarch
texlive-pdfpages-9:svn45659-19.fc32.noarch
texlive-pgf-9:svn51817-19.fc33.noarch
texlive-pgf-9:svn51817-19.fc32.noarch
texlive-placeins-9:svn19848.2.2-19.fc33.noarch
texlive-placeins-9:svn19848.2.2-19.fc32.noarch
texlive-plain-9:svn43076-19.fc33.noarch
texlive-plain-9:svn43076-19.fc32.noarch
texlive-polyglossia-9:svn50787-19.fc33.noarch
texlive-polyglossia-9:svn50787-19.fc32.noarch
texlive-preprint-9:svn30447.2011-19.fc33.noarch
texlive-preprint-9:svn30447.2011-19.fc32.noarch
texlive-psfrag-9:svn15878.3.04-19.fc33.noarch
texlive-psfrag-9:svn15878.3.04-19.fc32.noarch
texlive-pslatex-9:svn16416.0-19.fc33.noarch
texlive-pslatex-9:svn16416.0-19.fc32.noarch
texlive-psnfss-9:svn33946.9.2a-19.fc33.noarch
texlive-psnfss-9:svn33946.9.2a-19.fc32.noarch
texlive-pspicture-9:svn15878.0-19.fc33.noarch
texlive-pspicture-9:svn15878.0-19.fc32.noarch
texlive-pst-3d-9:svn17257.1.10-19.fc33.noarch
texlive-pst-3d-9:svn17257.1.10-19.fc32.noarch
texlive-pst-coil-9:svn37377.1.07-19.fc33.noarch
texlive-pst-coil-9:svn37377.1.07-19.fc32.noarch
texlive-pst-eps-9:svn15878.1.0-19.fc33.noarch
texlive-pst-eps-9:svn15878.1.0-19.fc32.noarch
texlive-pst-fill-9:svn15878.1.01-19.fc33.noarch
texlive-pst-fill-9:svn15878.1.01-19.fc32.noarch
texlive-pst-grad-9:svn15878.1.06-19.fc33.noarch
texlive-pst-grad-9:svn15878.1.06-19.fc32.noarch
texlive-pst-math-9:svn49425-19.fc33.noarch
texlive-pst-math-9:svn49425-19.fc32.noarch
texlive-pst-node-9:svn50215-19.fc33.noarch
texlive-pst-node-9:svn50215-19.fc32.noarch
texlive-pst-ovl-9:svn45506-19.fc33.noarch
texlive-pst-ovl-9:svn45506-19.fc32.noarch
texlive-pst-plot-9:svn51650-19.fc33.noarch
texlive-pst-plot-9:svn51650-19.fc32.noarch
texlive-pst-text-9:svn49542-19.fc33.noarch
texlive-pst-text-9:svn49542-19.fc32.noarch
texlive-pst-tools-9:svn45978-19.fc33.noarch
texlive-pst-tools-9:svn45978-19.fc32.noarch
texlive-pst-tree-9:svn43272-19.fc33.noarch
texlive-pst-tree-9:svn43272-19.fc32.noarch
texlive-pstricks-9:svn51102-19.fc33.noarch
texlive-pstricks-9:svn51102-19.fc32.noarch
texlive-pstricks-add-9:svn49680-19.fc33.noarch
texlive-pstricks-add-9:svn49680-19.fc32.noarch
texlive-pxfonts-9:svn15878.0-19.fc33.noarch
texlive-pxfonts-9:svn15878.0-19.fc32.noarch
texlive-qstest-9:svn15878.0-19.fc33.noarch
texlive-qstest-9:svn15878.0-19.fc32.noarch
texlive-ragged2e-9:svn51780-19.fc33.noarch
texlive-ragged2e-9:svn51780-19.fc32.noarch
texlive-rcs-9:svn15878.0-19.fc33.noarch
texlive-rcs-9:svn15878.0-19.fc32.noarch
texlive-realscripts-9:svn39706-19.fc33.noarch
texlive-realscripts-9:svn39706-19.fc32.noarch
texlive-rsfs-9:svn15878.0-19.fc33.noarch
texlive-rsfs-9:svn15878.0-19.fc32.noarch
texlive-sansmath-9:svn17997.1.1-19.fc33.noarch
texlive-sansmath-9:svn17997.1.1-19.fc32.noarch
texlive-sansmathaccent-9:svn30187.0-19.fc33.noarch
texlive-sansmathaccent-9:svn30187.0-19.fc32.noarch
texlive-sauerj-9:svn15878.0-19.fc33.noarch
texlive-sauerj-9:svn15878.0-19.fc32.noarch
texlive-section-9:svn20180.0-19.fc33.noarch
texlive-section-9:svn20180.0-19.fc32.noarch
texlive-seminar-9:svn34011.1.62-19.fc33.noarch
texlive-seminar-9:svn34011.1.62-19.fc32.noarch
texlive-sepnum-9:svn20186.2.0-19.fc33.noarch
texlive-sepnum-9:svn20186.2.0-19.fc32.noarch
texlive-setspace-9:svn24881.6.7a-19.fc33.noarch
texlive-setspace-9:svn24881.6.7a-19.fc32.noarch
texlive-showexpl-9:svn42677-19.fc33.noarch
texlive-showexpl-9:svn42677-19.fc32.noarch
texlive-soul-9:svn15878.2.4-19.fc33.noarch
texlive-soul-9:svn15878.2.4-19.fc32.noarch
texlive-stmaryrd-9:svn22027.0-19.fc33.noarch
texlive-stmaryrd-9:svn22027.0-19.fc32.noarch
texlive-subfig-9:svn15878.1.3-19.fc33.noarch
texlive-subfig-9:svn15878.1.3-19.fc32.noarch
texlive-symbol-9:svn31835.0-19.fc33.noarch
texlive-symbol-9:svn31835.0-19.fc32.noarch
texlive-tabulary-9:svn34368.0.10-19.fc33.noarch
texlive-tabulary-9:svn34368.0.10-19.fc32.noarch
texlive-tetex-7:20190410-12.fc32.noarch
texlive-tetex-7:20190410-12.fc32.noarch
texlive-tex-gyre-9:svn48058-19.fc33.noarch
texlive-tex-gyre-9:svn48058-19.fc32.noarch
texlive-tex-gyre-math-9:svn41264-19.fc33.noarch
texlive-tex-gyre-math-9:svn41264-19.fc32.noarch
texlive-tex-ini-files-9:svn40533-19.fc33.noarch
texlive-tex-ini-files-9:svn40533-19.fc32.noarch
texlive-texlive-common-doc-9:svn50466-19.fc33.noarch
texlive-texlive-common-doc-9:svn50466-19.fc32.noarch
texlive-texlive-docindex-9:svn51813-19.fc33.noarch
texlive-texlive-docindex-9:svn51813-19.fc32.noarch
texlive-texlive-msg-translations-9:svn51750-19.fc33.noarch
texlive-texlive-msg-translations-9:svn51750-19.fc32.noarch
texlive-textcase-9:svn15878.0-19.fc33.noarch
texlive-textcase-9:svn15878.0-19.fc32.noarch
texlive-times-9:svn35058.0-19.fc33.noarch
texlive-times-9:svn35058.0-19.fc32.noarch
texlive-tipa-9:svn29349.1.3-19.fc33.noarch
texlive-tipa-9:svn29349.1.3-19.fc32.noarch
texlive-titlesec-9:svn51664-19.fc33.noarch
texlive-titlesec-9:svn51664-19.fc32.noarch
texlive-tools-9:svn49619-19.fc33.noarch
texlive-tools-9:svn49619-19.fc32.noarch
texlive-translator-9:svn51279-19.fc33.noarch
texlive-translator-9:svn51279-19.fc32.noarch
texlive-trimspaces-9:svn15878.1.1-19.fc33.noarch
texlive-trimspaces-9:svn15878.1.1-19.fc32.noarch
texlive-txfonts-9:svn15878.0-19.fc33.noarch
texlive-txfonts-9:svn15878.0-19.fc32.noarch
texlive-typehtml-9:svn17134.0-19.fc33.noarch
texlive-typehtml-9:svn17134.0-19.fc32.noarch
texlive-ucharcat-9:svn38907-19.fc33.noarch
texlive-ucharcat-9:svn38907-19.fc32.noarch
texlive-ucs-9:svn35853.2.2-19.fc33.noarch
texlive-ucs-9:svn35853.2.2-19.fc32.noarch
texlive-underscore-9:svn18261.0-19.fc33.noarch
texlive-underscore-9:svn18261.0-19.fc32.noarch
texlive-unicode-data-9:svn50303-19.fc33.noarch
texlive-unicode-data-9:svn50303-19.fc32.noarch
texlive-unicode-math-9:svn50245-19.fc33.noarch
texlive-unicode-math-9:svn50245-19.fc32.noarch
texlive-updmap-map-9:svn52454-19.fc33.noarch
texlive-updmap-map-9:svn52454-19.fc32.noarch
texlive-upquote-9:svn26059.v1.3-19.fc33.noarch
texlive-upquote-9:svn26059.v1.3-19.fc32.noarch
texlive-url-9:svn32528.3.4-19.fc33.noarch
texlive-url-9:svn32528.3.4-19.fc32.noarch
texlive-utopia-9:svn15878.0-19.fc33.noarch
texlive-utopia-9:svn15878.0-19.fc32.noarch
texlive-varwidth-9:svn24104.0.92-19.fc33.noarch
texlive-varwidth-9:svn24104.0.92-19.fc32.noarch
texlive-wasy-9:svn35831.0-19.fc33.noarch
texlive-wasy-9:svn35831.0-19.fc32.noarch
texlive-wasy2-ps-9:svn35830.0-19.fc33.noarch
texlive-wasy2-ps-9:svn35830.0-19.fc32.noarch
texlive-wasysym-9:svn15878.2.0-19.fc33.noarch
texlive-wasysym-9:svn15878.2.0-19.fc32.noarch
texlive-wrapfig-9:svn22048.3.6-19.fc33.noarch
texlive-wrapfig-9:svn22048.3.6-19.fc32.noarch
texlive-xcolor-9:svn41044-19.fc33.noarch
texlive-xcolor-9:svn41044-19.fc32.noarch
texlive-xetexconfig-9:svn45845-19.fc33.noarch
texlive-xetexconfig-9:svn45845-19.fc32.noarch
texlive-xifthen-9:svn38929-19.fc33.noarch
texlive-xifthen-9:svn38929-19.fc32.noarch
texlive-xkeyval-9:svn35741.2.7a-19.fc33.noarch
texlive-xkeyval-9:svn35741.2.7a-19.fc32.noarch
texlive-xltxtra-9:svn49555-19.fc33.noarch
texlive-xltxtra-9:svn49555-19.fc32.noarch
texlive-xunicode-9:svn30466.0.981-19.fc33.noarch
texlive-xunicode-9:svn30466.0.981-19.fc32.noarch
texlive-zapfchan-9:svn31835.0-19.fc33.noarch
texlive-zapfchan-9:svn31835.0-19.fc32.noarch
texlive-zapfding-9:svn31835.0-19.fc33.noarch
texlive-zapfding-9:svn31835.0-19.fc32.noarch
util-linux-2.35.1-7.fc33.aarch64
util-linux-2.35.1-4.fc32.aarch64
vim-filesystem-2:8.2.444-1.fc33.noarch
vim-filesystem-2:8.2.348-1.fc32.noarch
wayland-devel-1.18.0-1.fc33.aarch64
wayland-devel-1.18.0-1.fc32.aarch64
wayland-protocols-devel-1.20-1.fc33.noarch
wayland-protocols-devel-1.20-1.fc32.noarch
which-2.21-18.fc33.aarch64
which-2.21-17.fc32.aarch64
xxhash-libs-0.7.3-1.fc33.aarch64
xxhash-libs-0.7.3-1.fc32.aarch64

I notice that curl has a higher version in F32 than in Rawhide.  Is
that a known issue?

The only packages on this list that really jump out at me are annobin,
glibc, gmp, and redhat-rpm-config.  I am out of time tonight, but
tomorrow I am going to take a careful look at what changed for each of
those 4 packages between the F32 version and the Rawhide version.
-- 
Jerry James
http://www.jamezone.org/

[Jerry James]

On Wed, Mar 25, 2020 at 9:24 PM Jerry James <loganjerry(a)gmail.com&gt; wrote:
>  You'll want to view this with a fixed-width font. 
Thank you, gmail, for completely destroying my formatting work.  Oh
well, you can probably tell what I was trying to do.
-- 
Jerry James
http://www.jamezone.org/

[Stephen J. Turnbull]

Hi Jerry!  Long time no see.

Jerry James writes:

>  I built the whole stack for Rawhide with no issues. 
Zero compiler warnings?  No fiddling to suppress compiler warnings
*anywhere* in the build chain (whether options or pragmas)?

Not suggesting you go looking for pragmas (probably have 10 million
lines of code :-( ), but you could check for warnings in the build
logs.  I didn't see any in the tail, though.

>  Attempt 1: the x86_64 build segfaulted while running "bin/coqc
> -coqlib . -q
>  -native-compiler yes   theories/FSets/FMapFullAVL.v", where coqc is a
>  binary built from the coq sources.  Coq is a theorem prover.  The coqc
>  run is essentially checking that a series of proof steps indeed prove
>  a theorem.  The "-native-compiler yes" part means that the OCaml
>  compiler will be invoked to generate native code for parts of this
>  process. 
Where is the segfault?  Is it in coqc or is it in the generated code
or is it in the OCaml compiler or is it somewhere else?

If you can't rule out the OCaml compiler (or other parts of the OCaml
toolchain) and the generated code:
- what happens if you use -native-compiler=no?
- How is the native code used?  Compiled and linked as a .so, then
  dynamically loaded and called?  Compiled and linked as an
  executable, then run as a separate process?

If you can't rule out OCaml: does it produce native code directly, or
does it produce C or similar and then use GCC or similar to produce
the native code?

What optimizations to gcc are being used where it's being used?

Are other compilers (LLVM) supported in lieu of GCC for any parts of
the build chain?

>  
>  I was puzzled by the segfault.  I decided to see if the segfault is
>  deterministic, so I launched...
>  
>  Attempt 2: the x86_64 build succeeded.  The s390x build failed with an
>  illegal instruction error while running *the same command*.
>  
>  Attempt 3: The s390x build again failed with an illegal instruction
>  error while running the same command, and aarch64 segfaulted while
>  running the same command.  All other arches succeeded.  I saved the
>  URL for this one:
>  
>  https://koji.fedoraproject.org/koji/taskinfo?taskID=42768075 
From the tail of the S390 build log:

rm -f theories/FSets/FMapFullAVL.glob
bin/coqc -coqlib . -q -native-compiler yes   theories/FSets/FMapFullAVL.v  
make[1]: *** [Makefile.build:880: theories/FSets/FMapFullAVL.vo] Illegal instruction (core
dumped)
make[1]: *** [theories/FSets/FMapFullAVL.vo] Deleting file
'theories/FSets/FMapFullAVL.glob'
make[1]: Leaving directory '/builddir/build/BUILD/coq-8.11.0'
make: *** [Makefile:177: submake] Error 2

That seems pretty unhelpful, as Makefile.build:880 is presumably just
the rule that invokes bin/coqc, which evidently is actually a compiler
driver that invokes various executables.  Do you have access to the
core that was allegedly dumped, to at least determine the identity of
the executable that segfaulted?  Is it possible to reproduce outside
of Koji?

Stay healthy!

-- 
Associate Professor              Division of Policy and Planning Science
http://turnbull.sk.tsukuba.ac.jp/     Faculty of Systems and Information
Email: turnbull(a)sk.tsukuba.ac.jp                   University of Tsukuba
Tel: [phone]                 Tennodai 1-1-1, Tsukuba 305-8573 JAPAN

[Richard W.M. Jones]

On Thu, Mar 26, 2020 at 01:20:58PM +0900, Stephen J. Turnbull wrote:
>  Where is the segfault?  Is it in coqc or is it in the generated code
>  or is it in the OCaml compiler or is it somewhere else? 
I managed to reproduce this on my mostly Rawhide x86-64 development
machine by running fedpkg build in a loop overnight.  This also
captured a core dump.

Unfortunately the stack trace, even with full debuginfo installed, was
not very helpful.  I'm now trying to run the unstable bin/coqc command
over and over again under gdb, but the bug has not reproduced there so
far.

One thing I did learn from the stack trace is that it appears to fail
in C code (in the garbage collector).  However that might be because
the OCaml heap is corrupted by an earlier event.  Another thing I
learned is that it happens when doing something or other with weak
references, which can be non-deterministic.

>  If you can't rule out OCaml: does it produce native code
> directly, or
>  does it produce C or similar and then use GCC or similar to produce
>  the native code? 
OCaml produces native code directly.  GCC is only used to link the
final object files together.

Rich.


Core was generated by `bin/coqc -coqlib . -q -native-compiler yes
theories/FSets/FMapFullAVL.v'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x0000556989736c94 in ?? ()
(gdb) t a a bt

Thread 1 (Thread 0x7f7453e1ec00 (LWP 2710541)):
#0  0x0000556989736c94 in ?? ()
#1  0x0000000000000001 in ?? ()
#2  0x0000000000000001 in ?? ()
#3  0x00007f7453440ec8 in ?? ()
#4  0xfffffffffffffec0 in ?? ()
#5  0x00005569c42daba8 in ?? ()
#6  0x0000000000000004 in ?? ()
#7  0x0000000000000002 in ?? ()
#8  0x000000042d7f6c00 in ?? ()
#9  0x0000000000000001 in ?? ()

    ^------ probable bad stack frame

#10 0x00005569c35adc63 in caml_alloc_shr ()

    ^------ garbage collector
           
https://github.com/ocaml/ocaml/blob/0756052841d0d3e6266c8c1ca4a608dc16c11...

#11 0x00005569c35c00bc in caml_ephe_set_key ()

    ^------ weak ephemerons
           
https://github.com/ocaml/ocaml/blob/0756052841d0d3e6266c8c1ca4a608dc16c11...

#12 0x00005569c35419e8 in camlStdlib__obj__set_key_306 ()
#13 0x00005569c358da66 in camlStdlib__ephemeron__create_1033 ()

    ^------ OCaml functions Stdlib.Obj.set_key & Stdlib.Ephemeron.create

#14 0x00007ffd45de5508 in ?? ()
#15 0x00007ffd45de5500 in ?? ()
#16 0x00007f744b7cf7b0 in ?? ()
#17 0x0000000030124a25 in ?? ()
#18 0x00007f7453745e88 in ?? ()
#19 0x0000000000000225 in ?? ()
#20 0x4917d3992e9b8500 in ?? ()
#21 0x00005569c3a3c858 in camlCsymtable__9 ()
#22 0x0000000000000000 in ?? ()

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
libguestfs lets you edit virtual machines.  Supports shell scripting,
bindings from many languages.  http://libguestfs.org

[Richard W.M. Jones]

This bug is VERY annoying!  Although not very reproducible, I can
usually hit it after about 4-6 hours of looping ‘fedpkg build’.

Considering the whole Coq build is quite lengthy you'd think that one
of these commands would hit it much more quickly:

  $ while true ; do gdb -ex 'set args -coqlib . -q -native-compiler yes
theories/FSets/FMapFullAVL.v' -ex run bin/coqc -ex quit ; done 

or

  $ while bin/coqc -coqlib . -q -native-compiler yes theories/FSets/FMapFullAVL.v ; do
date; done

But nope ...

My latest thing is to run the gdb command above _and_ run fedpkg build
(in another directory) at the same time, in the hope that it's caused
by system load.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-builder quickly builds VMs from scratch
http://libguestfs.org/virt-builder.1.html

[Richard W.M. Jones]

On Thu, Mar 26, 2020 at 12:00:14PM +0000, Richard W.M. Jones wrote:
>  This bug is VERY annoying!  Although not very reproducible, I can
>  usually hit it after about 4-6 hours of looping ‘fedpkg build’. 
I mean ‘fedpkg local’.  Local builds, not building in Koji.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-p2v converts physical machines to virtual machines.  Boot with a
live CD or over the network (PXE) and turn machines into KVM guests.
http://libguestfs.org/virt-v2v

[Richard W.M. Jones]

I don't know, now I can't even get the ‘fedpkg local’ to reproduce it :-(

Jerry, I suggest this bug is real, but is also likely to be a bug in
Coq (most likely) or the OCaml runtime, possibly in the Weak module.
You might have more luck asking the upstream developers for help.

As for what to do about Fedora 32, how about disabling _smp_mflags, on
the basis that it might be load-related?  That should at least reduce
the non-determinism.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-p2v converts physical machines to virtual machines.  Boot with a
live CD or over the network (PXE) and turn machines into KVM guests.
http://libguestfs.org/virt-v2v

[Jerry James]

On Thu, Mar 26, 2020 at 10:28 AM Richard W.M. Jones <rjones(a)redhat.com&gt; wrote:
>  I don't know, now I can't even get the ‘fedpkg local’ to
> reproduce it :-(
>
>  Jerry, I suggest this bug is real, but is also likely to be a bug in
>  Coq (most likely) or the OCaml runtime, possibly in the Weak module.
>  You might have more luck asking the upstream developers for help.
>
>  As for what to do about Fedora 32, how about disabling _smp_mflags, on
>  the basis that it might be load-related?  That should at least reduce
>  the non-determinism. 
Okay, I will give that a try.  I'm worried that even if I get a "good"
build, that it's going to explode on the hapless Fedora users who
install it.

Thank you for all the work you put into this, Richard.  I appreciate
the investigation.  I'll keep poking at it, too.
-- 
Jerry James
http://www.jamezone.org/

[Jerry James]

On Fri, Mar 27, 2020 at 11:03 AM Jerry James <loganjerry(a)gmail.com&gt; wrote:
>  Okay, I will give that a try. 
Except I can't.  We're already not using %_smp_mflags.  Ugh.
-- 
Jerry James
http://www.jamezone.org/

[Zbigniew Jędrzejewski-Szmek]

On Fri, Mar 27, 2020 at 11:15:01AM -0600, Jerry James wrote:
>  On Fri, Mar 27, 2020 at 11:03 AM Jerry James
> <loganjerry(a)gmail.com&gt; wrote:
>  > Okay, I will give that a try.
>  
>  Except I can't.  We're already not using %_smp_mflags.  Ugh. 
We are. Did you trying putting '%global _smp_mflags -j1' somewhere
near the top of the spec file?

Zbyszek

[Jerry James]

On Fri, Mar 27, 2020 at 11:30 AM Zbigniew Jędrzejewski-Szmek
<zbyszek(a)in.waw.pl&gt; wrote:
>  We are. Did you trying putting '%global _smp_mflags -j1'
> somewhere
>  near the top of the spec file? 
We are?  Where?  The spec file does this to build:

make world VERBOSE=1

-- 
Jerry James
http://www.jamezone.org/

[Zbigniew Jędrzejewski-Szmek]

On Fri, Mar 27, 2020 at 11:36:38AM -0600, Jerry James wrote:
>  On Fri, Mar 27, 2020 at 11:30 AM Zbigniew Jędrzejewski-Szmek
>  <zbyszek(a)in.waw.pl&gt; wrote:
>  > We are. Did you trying putting '%global _smp_mflags -j1' somewhere
>  > near the top of the spec file?
>  
>  We are?  Where? 
That is what the packaging guidelines say [1]:
>  Whenever possible, invocations of make should be done as
>  %make_build 
(which is defined as
$ rpmbuild --showrc |grep make_build
make_build %{__make} %{_make_output_sync} %{?_smp_mflags} %{_make_verbose}).

[1] https://docs.fedoraproject.org/en-US/packaging-guidelines/#_parallel_make

>  The spec file does this to build:
>  
>  make world VERBOSE=1 
So it seems that this particular Makefile enables parallelization
internally. (make defaults -j1 unless told otherwise). But it should
be possible to somehow disable it wherever it is enabled.

Zbyszek

[Richard W.M. Jones]

On Fri, Mar 27, 2020 at 11:15:01AM -0600, Jerry James wrote:
>  On Fri, Mar 27, 2020 at 11:03 AM Jerry James
> <loganjerry(a)gmail.com&gt; wrote:
>  > Okay, I will give that a try.
>  
>  Except I can't.  We're already not using %_smp_mflags.  Ugh. 
I realized that my build machine has MAKEFLAGS=-j24 which is why it's
building coq in parallel (and quickly).  So it seems like you could be
using _smp_mflags.

Another thing I tried today was enabling debugging in the GC (see
patch below).  However it also didn't find any problems, so I'm still
at a loss.

Rich.

diff --git a/coq.spec b/coq.spec
index df2a847..1e590cc 100644
--- a/coq.spec
+++ b/coq.spec
@@ -164,6 +164,10 @@ done
 %global opt_option -bytecode-compiler yes -byte-only -coqide byte
 %endif
 
+# Use debug runtime.
+sed -i 's/^\(OCAMLC :=.*\)/\1 -runtime-variant d/' Makefile.build
+sed -i 's/^\(OCAMLOPT :=.*\)/\1 -runtime-variant d/' Makefile.build
+
 %global coqdocdir %{?_pkgdocdir}%{!?_pkgdocdir:%{_docdir}/coq-%{version}}
 %global coqdatadir %{_libdir}/coq
 


-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-top is 'top' for virtual machines.  Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://people.redhat.com/~rjones/virt-top

[Jerry James]

On Thu, Mar 26, 2020 at 10:28 AM Richard W.M. Jones <rjones(a)redhat.com&gt; wrote:
>  Jerry, I suggest this bug is real, but is also likely to be a bug in
>  Coq (most likely) or the OCaml runtime, possibly in the Weak module.
>  You might have more luck asking the upstream developers for help. 
Coq issue: https://github.com/coq/coq/issues/11939

I looked around in the ocaml issue tracker and found a possibility:

https://github.com/ocaml/ocaml/issues/9391

Coq uses ephemerons.
-- 
Jerry James
http://www.jamezone.org/

[Richard W.M. Jones]

On Fri, Mar 27, 2020 at 11:54:44AM -0600, Jerry James wrote:
>  On Thu, Mar 26, 2020 at 10:28 AM Richard W.M. Jones
> <rjones(a)redhat.com&gt; wrote:
>  > Jerry, I suggest this bug is real, but is also likely to be a bug in
>  > Coq (most likely) or the OCaml runtime, possibly in the Weak module.
>  > You might have more luck asking the upstream developers for help.
>  
>  Coq issue: https://github.com/coq/coq/issues/11939
>  
>  I looked around in the ocaml issue tracker and found a possibility:
>  
>  https://github.com/ocaml/ocaml/issues/9391
>  
>  Coq uses ephemerons. 
Yup, and the stack trace shows the failure happening when creating an
ephemeron.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-p2v converts physical machines to virtual machines.  Boot with a
live CD or over the network (PXE) and turn machines into KVM guests.
http://libguestfs.org/virt-v2v

[Jerry James]

On Fri, Mar 27, 2020 at 2:41 PM Richard W.M. Jones <rjones(a)redhat.com&gt; wrote:
>  Yup, and the stack trace shows the failure happening when creating
> an
>  ephemeron. 
On the other hand, a scratch build with coq upstream's final patch for
ocaml 4.10.0 succeeded on the first try:

https://koji.fedoraproject.org/koji/taskinfo?taskID=42808115

Where this is an intermittent problem, I'll have to try more builds to
be sure, but this looks promising.
-- 
Jerry James
http://www.jamezone.org/

[Richard W.M. Jones]

On Fri, Mar 27, 2020 at 04:20:16PM -0600, Jerry James wrote:
>  On Fri, Mar 27, 2020 at 2:41 PM Richard W.M. Jones
> <rjones(a)redhat.com&gt; wrote:
>  > Yup, and the stack trace shows the failure happening when creating an
>  > ephemeron.
>  
>  On the other hand, a scratch build with coq upstream's final patch for
>  ocaml 4.10.0 succeeded on the first try:
>  
>  https://koji.fedoraproject.org/koji/taskinfo?taskID=42808115
>  
>  Where this is an intermittent problem, I'll have to try more builds to
>  be sure, but this looks promising. 
I'll run it in a loop overnight and see if I can make it crash.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine.  Supports Linux and Windows.
http://people.redhat.com/~rjones/virt-df/

[Jerry James]

On Fri, Mar 27, 2020 at 4:41 PM Richard W.M. Jones <rjones(a)redhat.com&gt; wrote:
>  I'll run it in a loop overnight and see if I can make it crash.
Good idea.  Thank you!
-- 
Jerry James
http://www.jamezone.org/

[Jerry James]

On Fri, Mar 27, 2020 at 4:42 PM Jerry James <loganjerry(a)gmail.com&gt; wrote:
>  Good idea.  Thank you! 
I got a segfault on s390x on the second build attempt.  I'd like to
investigate the ephemeron angle a bit, I think.
-- 
Jerry James
http://www.jamezone.org/

[Richard W.M. Jones]

On Fri, Mar 27, 2020 at 05:55:42PM -0600, Jerry James wrote:
>  On Fri, Mar 27, 2020 at 4:42 PM Jerry James
> <loganjerry(a)gmail.com&gt; wrote:
>  > Good idea.  Thank you!
>  
>  I got a segfault on s390x on the second build attempt.  I'd like to
>  investigate the ephemeron angle a bit, I think. 
No crashes here overnight, but I'll leave it running.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-top is 'top' for virtual machines.  Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://people.redhat.com/~rjones/virt-top

[Jerry James]

On Wed, Mar 25, 2020 at 10:21 PM Stephen J. Turnbull <stephen(a)xemacs.org&gt; wrote:
>  Hi Jerry!  Long time no see. 
Hi Stephen!  It's good to hear from yet another person I've worked
with via network for years.

I think Richard addressed most of the points you raised.

>  Where is the segfault?  Is it in coqc or is it in the generated code
>  or is it in the OCaml compiler or is it somewhere else? 
It's in coqc.  Richard's evidence seems to point to garbage collector
involvement, with weak references possibly playing a role.

>  Stay healthy! 
You, too, and likewise for everybody in the Fedora community.
-- 
Jerry James
http://www.jamezone.org/

[Stephen J. Turnbull]

Jerry James writes:

>  I think Richard addressed most of the points you raised. 
Yes, and no new questions, either.  So I think I'm done here (well,
I'll spectate, but the internals of the OCaml compiler are well beyond
my skillset :-).

Good luck!

Steve

-- 
Associate Professor              Division of Policy and Planning Science
http://turnbull/sk.tsukuba.ac.jp/     Faculty of Systems and Information
Email: turnbull(a)sk.tsukuba.ac.jp                   University of Tsukuba
Tel: [phone]                 Tennodai 1-1-1, Tsukuba 305-8573 JAPAN