1:02 p.m.
I was trying to chase down why modules.dep is no longer built in
Rawhide (which affects supermin and therefore libguestfs), but it
looks like there's a much more serious problem:
https://kojipkgs.fedoraproject.org//work/tasks/8242/44698242/root.log
(from https://koji.fedoraproject.org/koji/taskinfo?taskID=44698142)
DEBUG util.py:602: Running transaction
DEBUG util.py:602: /usr/bin/systemctl: error while loading shared libraries:
libibverbs.so.1: cannot open shared object file: No such file or directory
DEBUG util.py:602: /sbin/udevadm: error while loading shared libraries: libibverbs.so.1:
cannot open shared object file: No such file or directory
DEBUG util.py:602: /sbin/udevadm: error while loading shared libraries: libibverbs.so.1:
cannot open shared object file: No such file or directory
DEBUG util.py:602: /sbin/udevadm: error while loading shared libraries: libibverbs.so.1:
cannot open shared object file: No such file or directory
DEBUG util.py:602: /var/tmp/rpm-tmp.pYK3f6: line 6: 2005996 Segmentation fault (core
dumped) /usr/bin/systemctl --no-reload preset dbus.socket
DEBUG util.py:602: /var/tmp/rpm-tmp.pYK3f6: line 13: 2005999 Segmentation fault
(core dumped) /usr/bin/systemctl --no-reload preset \--global dbus.socket
DEBUG util.py:602: /var/tmp/rpm-tmp.heiSf7: line 6: 2006029 Segmentation fault (core
dumped) /usr/bin/systemctl --no-reload preset dbus-broker.service
DEBUG util.py:602: /var/tmp/rpm-tmp.heiSf7: line 13: 2006032 Segmentation fault
(core dumped) /usr/bin/systemctl --no-reload preset \--global dbus-broker.service
DEBUG util.py:602: /var/tmp/rpm-tmp.NScHA9: line 9: 2006061 Segmentation fault (core
dumped) /usr/bin/systemctl --no-reload preset polkit.service
DEBUG util.py:602: /var/tmp/rpm-tmp.05AcI8: line 6: 2006069 Segmentation fault (core
dumped) /usr/bin/systemctl --no-reload preset dm-event.socket
DEBUG util.py:602: /var/tmp/rpm-tmp.05AcI8: line 11: 2006072 Segmentation fault
(core dumped) systemctl enable dm-event.socket
DEBUG util.py:602: /var/tmp/rpm-tmp.05AcI8: line 12: 2006080 Segmentation fault
(core dumped) systemctl start dm-event.socket > /dev/null 2>&1
DEBUG util.py:602: /var/tmp/rpm-tmp.4iXDa7: line 6: 2006113 Segmentation fault (core
dumped) /usr/bin/systemctl --no-reload preset saslauthd.service
DEBUG util.py:602: /var/tmp/rpm-tmp.8CYlZ7: line 6: 2006121 Segmentation fault (core
dumped) /usr/bin/systemctl --no-reload preset rdisc.service
DEBUG util.py:602: /var/tmp/rpm-tmp.mIKYA7: line 6: 2006130 Segmentation fault (core
dumped) /usr/bin/systemctl --no-reload preset mdmonitor.service
raid-check.{service,timer}
DEBUG util.py:602: /var/tmp/rpm-tmp.mIKYA7: line 8: 2006133 Segmentation fault (core
dumped) /usr/bin/systemctl disable mdmonitor-takeover.service > /dev/null 2>&1
DEBUG util.py:602: /var/tmp/rpm-tmp.pk6AR5: line 6: 2006141 Segmentation fault (core
dumped) /usr/bin/systemctl --no-reload preset numad.service
DEBUG util.py:602: /var/tmp/rpm-tmp.eLxfL5: line 7: 2006179 Segmentation fault (core
dumped) /usr/bin/systemctl --no-reload preset virtlockd.socket virtlockd-admin.socket
DEBUG util.py:602: /var/tmp/rpm-tmp.eLxfL5: line 14: 2006182 Segmentation fault
(core dumped) /usr/bin/systemctl --no-reload preset virtlogd.socket virtlogd-admin.socket
DEBUG util.py:602: /var/tmp/rpm-tmp.eLxfL5: line 21: 2006189 Segmentation fault
(core dumped) /usr/bin/systemctl --no-reload preset libvirtd.socket libvirtd-ro.socket
libvirtd-admin.socket
DEBUG util.py:602: /var/tmp/rpm-tmp.eLxfL5: line 28: 2006196 Segmentation fault
(core dumped) /usr/bin/systemctl --no-reload preset libvirtd-tcp.socket
libvirtd-tls.socket
DEBUG util.py:602: /var/tmp/rpm-tmp.eLxfL5: line 35: 2006203 Segmentation fault
(core dumped) /usr/bin/systemctl --no-reload preset libvirtd.service
DEBUG util.py:602: /var/tmp/rpm-tmp.bk1ZR5: line 6: 2006222 Segmentation fault (core
dumped) /usr/bin/systemctl --no-reload preset blk-availability.service
lvm2-monitor.service
DEBUG util.py:602: /var/tmp/rpm-tmp.bk1ZR5: line 13: 2006225 Segmentation fault
(core dumped) systemctl enable lvm2-monitor.service
DEBUG util.py:602: /var/tmp/rpm-tmp.bk1ZR5: line 13: 2006232 Segmentation fault
(core dumped) systemctl start lvm2-monitor.service > /dev/null 2>&1
DEBUG util.py:602: /var/tmp/rpm-tmp.bk1ZR5: line 20: 2006239 Segmentation fault
(core dumped) /usr/bin/systemctl --no-reload preset lvm2-lvmpolld.socket
DEBUG util.py:602: /var/tmp/rpm-tmp.bk1ZR5: line 25: 2006246 Segmentation fault
(core dumped) systemctl enable lvm2-lvmpolld.socket
DEBUG util.py:602: /var/tmp/rpm-tmp.bk1ZR5: line 26: 2006254 Segmentation fault
(core dumped) systemctl start lvm2-lvmpolld.socket > /dev/null 2>&1
DEBUG util.py:602: /var/tmp/rpm-tmp.M5VjN9: line 6: 2006725 Segmentation fault (core
dumped) /usr/bin/systemctl --no-reload preset brltty.service
DEBUG util.py:602: /bin/kernel-install: line 137: /etc/kernel/install.d/: Is a directory
DEBUG util.py:600: Error in POSTTRANS scriptlet in rpm package kernel-core
DEBUG util.py:602: /var/tmp/rpm-tmp.6RiAo6: line 34: 2006782 Segmentation fault
(core dumped) /bin/systemctl is-active libvirtd.service > /dev/null 2>&1
DEBUG util.py:602: /var/tmp/rpm-tmp.LMHzS9: line 4: 2006829 Segmentation fault (core
dumped) /usr/bin/systemd-run /usr/bin/systemctl start man-db-cache-update > /dev/null
2>&1
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-builder quickly builds VMs from scratch
http://libguestfs.org/virt-builder.1.html
5:16 p.m.
On Tue, 2020-05-19 at 19:02 +0100, Richard W.M. Jones wrote:
I was trying to chase down why modules.dep is no longer built in
Rawhide (which affects supermin and therefore libguestfs), but it
looks like there's a much more serious problem:
https://kojipkgs.fedoraproject.org//work/tasks/8242/44698242/root.log
(from https://koji.fedoraproject.org/koji/taskinfo?taskID=44698142)
DEBUG util.py:602: Running transaction
DEBUG util.py:602: /usr/bin/systemctl: error while loading shared
libraries: libibverbs.so.1: cannot open shared object file: No such
file or directory
DEBUG util.py:602: /sbin/udevadm: error while loading shared
libraries: libibverbs.so.1: cannot open shared object file: No such
file or directory
DEBUG util.py:602: /sbin/udevadm: error while loading shared
libraries: libibverbs.so.1: cannot open shared object file: No such
file or directory
DEBUG util.py:602: /sbin/udevadm: error while loading shared
libraries: libibverbs.so.1: cannot open shared object file: No such
file or directory
Just to double check, I installed a rawhide installation and then ran
ldd on both systemctl and udevadm. Neither of these are linked to
libibverbs as far as I can tell. If they use libibverbs, they do it via
dlopen, not static linking. So that being the case, these errors make
no sense.
DEBUG util.py:602: /var/tmp/rpm-tmp.pYK3f6: line 6: 2005996
Segmentation fault (core dumped) /usr/bin/systemctl --no-reload
preset dbus.socket
Starting here, the error changes to a core dump instead of a library not
found, so maybe this is after the libibverbs package is finally
installed and libibverbs.so.1 is now found?
DEBUG util.py:602: /var/tmp/rpm-tmp.pYK3f6: line 13: 2005999
Segmentation fault (core dumped) /usr/bin/systemctl --no-reload
preset \--global dbus.socket
DEBUG util.py:602: /var/tmp/rpm-tmp.heiSf7: line 6: 2006029
Segmentation fault (core dumped) /usr/bin/systemctl --no-reload
preset dbus-broker.service
DEBUG util.py:602: /var/tmp/rpm-tmp.heiSf7: line 13: 2006032
Segmentation fault (core dumped) /usr/bin/systemctl --no-reload
preset \--global dbus-broker.service
DEBUG util.py:602: /var/tmp/rpm-tmp.NScHA9: line 9: 2006061
Segmentation fault (core dumped) /usr/bin/systemctl --no-reload
preset polkit.service
DEBUG util.py:602: /var/tmp/rpm-tmp.05AcI8: line 6: 2006069
Segmentation fault (core dumped) /usr/bin/systemctl --no-reload
preset dm-event.socket
DEBUG util.py:602: /var/tmp/rpm-tmp.05AcI8: line 11: 2006072
Segmentation fault (core dumped) systemctl enable dm-event.socket
DEBUG util.py:602: /var/tmp/rpm-tmp.05AcI8: line 12: 2006080
Segmentation fault (core dumped) systemctl start dm-event.socket
> /dev/null 2>&1
DEBUG util.py:602: /var/tmp/rpm-tmp.4iXDa7: line 6: 2006113
Segmentation fault (core dumped) /usr/bin/systemctl --no-reload
preset saslauthd.service
DEBUG util.py:602: /var/tmp/rpm-tmp.8CYlZ7: line 6: 2006121
Segmentation fault (core dumped) /usr/bin/systemctl --no-reload
preset rdisc.service
DEBUG util.py:602: /var/tmp/rpm-tmp.mIKYA7: line 6: 2006130
Segmentation fault (core dumped) /usr/bin/systemctl --no-reload
preset mdmonitor.service raid-check.{service,timer}
DEBUG util.py:602: /var/tmp/rpm-tmp.mIKYA7: line 8: 2006133
Segmentation fault (core dumped) /usr/bin/systemctl disable
mdmonitor-takeover.service > /dev/null 2>&1
DEBUG util.py:602: /var/tmp/rpm-tmp.pk6AR5: line 6: 2006141
Segmentation fault (core dumped) /usr/bin/systemctl --no-reload
preset numad.service
DEBUG util.py:602: /var/tmp/rpm-tmp.eLxfL5: line 7: 2006179
Segmentation fault (core dumped) /usr/bin/systemctl --no-reload
preset virtlockd.socket virtlockd-admin.socket
DEBUG util.py:602: /var/tmp/rpm-tmp.eLxfL5: line 14: 2006182
Segmentation fault (core dumped) /usr/bin/systemctl --no-reload
preset virtlogd.socket virtlogd-admin.socket
DEBUG util.py:602: /var/tmp/rpm-tmp.eLxfL5: line 21: 2006189
Segmentation fault (core dumped) /usr/bin/systemctl --no-reload
preset libvirtd.socket libvirtd-ro.socket libvirtd-admin.socket
DEBUG util.py:602: /var/tmp/rpm-tmp.eLxfL5: line 28: 2006196
Segmentation fault (core dumped) /usr/bin/systemctl --no-reload
preset libvirtd-tcp.socket libvirtd-tls.socket
DEBUG util.py:602: /var/tmp/rpm-tmp.eLxfL5: line 35: 2006203
Segmentation fault (core dumped) /usr/bin/systemctl --no-reload
preset libvirtd.service
DEBUG util.py:602: /var/tmp/rpm-tmp.bk1ZR5: line 6: 2006222
Segmentation fault (core dumped) /usr/bin/systemctl --no-reload
preset blk-availability.service lvm2-monitor.service
DEBUG util.py:602: /var/tmp/rpm-tmp.bk1ZR5: line 13: 2006225
Segmentation fault (core dumped) systemctl enable lvm2-
monitor.service
DEBUG util.py:602: /var/tmp/rpm-tmp.bk1ZR5: line 13: 2006232
Segmentation fault (core dumped) systemctl start lvm2-
monitor.service > /dev/null 2>&1
DEBUG util.py:602: /var/tmp/rpm-tmp.bk1ZR5: line 20: 2006239
Segmentation fault (core dumped) /usr/bin/systemctl --no-reload
preset lvm2-lvmpolld.socket
DEBUG util.py:602: /var/tmp/rpm-tmp.bk1ZR5: line 25: 2006246
Segmentation fault (core dumped) systemctl enable lvm2-
lvmpolld.socket
DEBUG util.py:602: /var/tmp/rpm-tmp.bk1ZR5: line 26: 2006254
Segmentation fault (core dumped) systemctl start lvm2-
lvmpolld.socket > /dev/null 2>&1
DEBUG util.py:602: /var/tmp/rpm-tmp.M5VjN9: line 6: 2006725
Segmentation fault (core dumped) /usr/bin/systemctl --no-reload
preset brltty.service
Up until here, these have all been the same. Failed %pre or %post
scriptlets because of systemctl failures.
DEBUG util.py:602: /bin/kernel-install: line 137:
/etc/kernel/install.d/: Is a directory
DEBUG util.py:600: Error in POSTTRANS scriptlet in rpm package
kernel-core
But this is different, and it's the cause of your problem (well, it's
the immediate cause anyway). The kernel-install script is failing
because it's passing /etc/kernel/install.d/ to something that wants
something other than a directory. That could be because a failed
scriptlet elsewhere has resulted in that directory being empty, so some
glob is returning the directory instead of the files in the directory,
so it could still be the systemctl issue, but it would be worth looking
into install-kernel to see.
OK, so install-kernel is part of systemd (I'm seeing a pattern here).
It generates an array of plugins that should be called for the new
kernel. It does all plugins in .install, /etc/kernel/install.d/, and
/usr/lib/kernel/install.d. My guess here is that the %POSTTRANS of the
kernel-core package is calling install-kernel, which is failing with the
above error, causing the kernel-core %POSTTRANS to error out
prematurely, resulting in the missing modules.dep file that is breaking
your build. At this point, I can't see what the problem can be other
than either a bad build of systemd, or if the recent upgrade to rdma-
core-29, with the new libibverbs-29 package, has caused a failure in
systemd because it needs relinked or something. But that can only be
the case if it's some sort of weak dependency based on dlopen as both
the rpm tools and ldd are not picking up the libibverbs dependency. If
that's the case, the systemd and/or udev rpm packages should have an
explicit requires on libibverbs I think.
Anyway, at this point, I don't know if the rdma-core-owner people can
help. I think this is first in the hands of the systemd folks.
DEBUG util.py:602: /var/tmp/rpm-tmp.6RiAo6: line 34: 2006782
Segmentation fault (core dumped) /bin/systemctl is-active
libvirtd.service > /dev/null 2>&1
DEBUG util.py:602: /var/tmp/rpm-tmp.LMHzS9: line 4: 2006829
Segmentation fault (core dumped) /usr/bin/systemd-run
/usr/bin/systemctl start man-db-cache-update > /dev/null 2>&1
Rich.
--
Doug Ledford <dledford(a)redhat.com>
GPG KeyID: B826A3330E572FDD
Fingerprint = AE6B 1BDA 122B 23B4 265B 1274 B826 A333 0E57 2FDD
5:45 p.m.
On Tue, 2020-05-19 at 18:16 -0400, Doug Ledford wrote:
But this is different, and it's the cause of your problem (well, it's
the immediate cause anyway). The kernel-install script is failing
because it's passing /etc/kernel/install.d/ to something that wants
something other than a directory. That could be because a failed
scriptlet elsewhere has resulted in that directory being empty, so some
glob is returning the directory instead of the files in the directory,
so it could still be the systemctl issue, but it would be worth looking
into install-kernel to see.
OK, so install-kernel is part of systemd (I'm seeing a pattern here).
It generates an array of plugins that should be called for the new
kernel. It does all plugins in .install, /etc/kernel/install.d/, and
/usr/lib/kernel/install.d. My guess here is that the %POSTTRANS of the
kernel-core package is calling install-kernel, which is failing with the
above error, causing the kernel-core %POSTTRANS to error out
prematurely, resulting in the missing modules.dep file that is breaking
your build. At this point, I can't see what the problem can be other
than either a bad build of systemd, or if the recent upgrade to rdma-
core-29, with the new libibverbs-29 package, has caused a failure in
systemd because it needs relinked or something. But that can only be
the case if it's some sort of weak dependency based on dlopen as both
the rpm tools and ldd are not picking up the libibverbs dependency. If
that's the case, the systemd and/or udev rpm packages should have an
explicit requires on libibverbs I think.
Anyway, at this point, I don't know if the rdma-core-owner people can
help. I think this is first in the hands of the systemd folks.
Well, systemd hasn't changed for nearly a month - it was last built on
2020-04-21. nbdkit was built successfully a few days ago, so blaming
systemd here feels wrong.
It's notable that the last couple of Rawhide composes have also failed
with an odd kernel-related error:
https://koji.fedoraproject.org/koji/taskinfo?taskID=44651583
DEBUG util.py:602: Traceback (most recent call last):
DEBUG util.py:602: File "/usr/sbin/lorax", line 223, in <module>
DEBUG util.py:602: main()
DEBUG util.py:602: File "/usr/sbin/lorax", line 204, in main
DEBUG util.py:602: lorax.run(dnfbase, opts.product, opts.version, opts.release,
DEBUG util.py:602: File
"/usr/lib/python3.8/site-packages/pylorax/__init__.py", line 354, in run
DEBUG util.py:602: treebuilder.rebuild_initrds(add_args=anaconda_args)
DEBUG util.py:602: File
"/usr/lib/python3.8/site-packages/pylorax/treebuilder.py", line 308, in
rebuild_initrds
DEBUG util.py:602: raise Exception("No kernels found, cannot
rebuild_initrds")
DEBUG util.py:602: Exception: No kernels found, cannot rebuild_initrds
still, having trouble pinning down a culprit; it's not kernel-5.7.0-
0.rc6.1.fc33 as the 20200518.n.0 compose failed, and that was run with
the previous kernel build, which *succeeded* in the 20200517.n.0
compose...
I guess we get to poke through everything built around the 17th and try
to find a relevant change? :)
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
5:55 p.m.
On Tue, 2020-05-19 at 15:45 -0700, Adam Williamson wrote:
On Tue, 2020-05-19 at 18:16 -0400, Doug Ledford wrote:
> But this is different, and it's the cause of your problem (well,
> it's
> the immediate cause anyway). The kernel-install script is failing
> because it's passing /etc/kernel/install.d/ to something that wants
> something other than a directory. That could be because a failed
> scriptlet elsewhere has resulted in that directory being empty, so
> some
> glob is returning the directory instead of the files in the
> directory,
> so it could still be the systemctl issue, but it would be worth
> looking
> into install-kernel to see.
>
> OK, so install-kernel is part of systemd (I'm seeing a pattern
> here).
> It generates an array of plugins that should be called for the new
> kernel. It does all plugins in .install, /etc/kernel/install.d/,
> and
> /usr/lib/kernel/install.d. My guess here is that the %POSTTRANS of
> the
> kernel-core package is calling install-kernel, which is failing with
> the
> above error, causing the kernel-core %POSTTRANS to error out
> prematurely, resulting in the missing modules.dep file that is
> breaking
> your build. At this point, I can't see what the problem can be
> other
> than either a bad build of systemd, or if the recent upgrade to
> rdma-
> core-29, with the new libibverbs-29 package, has caused a failure in
> systemd because it needs relinked or something. But that can only
> be
> the case if it's some sort of weak dependency based on dlopen as
> both
> the rpm tools and ldd are not picking up the libibverbs
> dependency. If
> that's the case, the systemd and/or udev rpm packages should have an
> explicit requires on libibverbs I think.
>
> Anyway, at this point, I don't know if the rdma-core-owner people
> can
> help. I think this is first in the hands of the systemd folks.
Well, systemd hasn't changed for nearly a month - it was last built on
2020-04-21. nbdkit was built successfully a few days ago, so blaming
systemd here feels wrong.
Ok, then something else has changed that is causing systemd problems
then.
It's notable that the last couple of Rawhide composes have also
failed
with an odd kernel-related error:
https://koji.fedoraproject.org/koji/taskinfo?taskID=44651583
DEBUG util.py:602: Traceback (most recent call last):
DEBUG util.py:602: File "/usr/sbin/lorax", line 223, in <module>
DEBUG util.py:602: main()
DEBUG util.py:602: File "/usr/sbin/lorax", line 204, in main
DEBUG util.py:602: lorax.run(dnfbase, opts.product, opts.version,
opts.release,
DEBUG util.py:602: File "/usr/lib/python3.8/site-
packages/pylorax/__init__.py", line 354, in run
DEBUG
util.py:602: treebuilder.rebuild_initrds(add_args=anaconda_args)
DEBUG util.py:602: File "/usr/lib/python3.8/site-
packages/pylorax/treebuilder.py", line 308, in rebuild_initrds
DEBUG util.py:602: raise Exception("No kernels found, cannot
rebuild_initrds")
DEBUG util.py:602: Exception: No kernels found, cannot
rebuild_initrds
still, having trouble pinning down a culprit; it's not kernel-5.7.0-
0.rc6.1.fc33 as the 20200518.n.0 compose failed, and that was run with
the previous kernel build, which *succeeded* in the 20200517.n.0
compose...
I guess we get to poke through everything built around the 17th and
try
to find a relevant change? :)
Aren't highly complex, interdependent systems with different owners of
different components fun? :-)
--
Doug Ledford <dledford(a)redhat.com>
GPG KeyID: B826A3330E572FDD
Fingerprint = AE6B 1BDA 122B 23B4 265B 1274 B826 A333 0E57 2FDD
5:58 p.m.
On Tue, 2020-05-19 at 18:55 -0400, Doug Ledford wrote:
> I guess we get to poke through everything built around the 17th and
> try
> to find a relevant change? :)
Aren't highly complex, interdependent systems with different owners of
different components fun? :-)
well, they pay my mortgage...:P
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
6:03 p.m.
On Tue, 2020-05-19 at 15:58 -0700, Adam Williamson wrote:
On Tue, 2020-05-19 at 18:55 -0400, Doug Ledford wrote:
> > I guess we get to poke through everything built around the 17th
> > and
> > try
> > to find a relevant change? :)
>
> Aren't highly complex, interdependent systems with different owners
> of
> different components fun? :-)
well, they pay my mortgage...:P
And some people's therapist's mortgage :-)
--
Doug Ledford <dledford(a)redhat.com>
GPG KeyID: B826A3330E572FDD
Fingerprint = AE6B 1BDA 122B 23B4 265B 1274 B826 A333 0E57 2FDD
6:34 p.m.
On Tue, 2020-05-19 at 15:45 -0700, Adam Williamson wrote:
still, having trouble pinning down a culprit; it's not kernel-5.7.0-
0.rc6.1.fc33 as the 20200518.n.0 compose failed, and that was run with
the previous kernel build, which *succeeded* in the 20200517.n.0
compose...
I guess we get to poke through everything built around the 17th and try
to find a relevant change? :)
This should be *more or less* the list of candidate culprits, which I
just heath robinson'd out of Bodhi (yet again I wish we got compose
reports for doomed composes, it'd make this easier...). grub2 and
openssl are the ones that jump out at me right away...ceph is also in
there, and ceph is one of the things that *is* linked to libibverbs ,
though the change in ceph doesn't seem like a significant one...
aha! This looks juicy. systemctl is linked against libpcap.so.1 , part
of libpcap, which is in the list below, and this is the changelog for
it:
* Fri May 15 2020 Michal Ruprich <michalruprich(a)gmail.com> - 14:1.9.1-4
- Enabling rdma support in libpcap
the changelog date is May 15 but the pcap build actually ran on 2020-
05-18, and indeed the last successful Rawhide compose (20200517.n.1)
had 1.9.1-3.fc33, and the first failed compose (20200518.n.0) has
1.9.1-4.fc33.
I'm betting we have a circular dependency problem or something here...
awscli-1.18.61-1.fc33
python-boto3-1.13.11-1.fc33
khotkeys-5.18.90-1.fc33
python-botocore-1.16.11-1.fc33
plasma-workspace-wallpapers-5.18.90-1.fc33
cascadia-code-fonts-2005.15-1.fc33
breeze-gtk-5.18.90-1.fc33
plasma-pa-5.18.90-1.fc33
plasma-oxygen-5.18.90-1.fc33
plasma-vault-5.18.90-1.fc33
ksshaskpass-5.18.90-1.fc33
kwayland-integration-5.18.90-1.fc33
kscreenlocker-5.18.90-1.fc33
plymouth-theme-breeze-5.18.90-1.fc33
libsysstat-0.4.3-1.fc33
plasma-browser-integration-5.18.90-1.fc33
liblxqt-0.15.0-1.fc33
plasma-sdk-5.18.90-1.fc33
bluedevil-5.18.90-1.fc33
plasma-milou-5.18.90-1.fc33
kdecoration-5.18.90-1.fc33
polkit-kde-5.18.90-1.fc33
kactivitymanagerd-5.18.90-1.fc33
pam-kwallet-5.18.90-1.fc33
sddm-kcm-5.18.90-1.fc33
kgamma-5.18.90-1.fc33
libkscreen-qt5-5.18.90-1.fc33
grub2-breeze-theme-5.18.90-1.fc33
python-readme-renderer-26.0-2.fc33
kwrited-5.18.90-1.fc33
vim-8.2.789-1.fc33
phd2-2.6.8-1.fc33
libqtxdg-3.5.0-1.fc33
python-readme-renderer-26.0-1.fc33
grub2-2.04-20.fc33
perl-DateTime-Format-Natural-1.09-1.fc33
openssl-1.1.1g-3.fc33
rust-isahc-0.9.2-1.fc33
lxqt-build-tools-0.7.0-1.fc33
onedrive-2.4.1-1.fc33
rust-zincati-0.0.10-3.fc33
rust-afterburn-4.3.3-3.fc33
rust-reqwest-0.10.4-2.fc33
dummy-test-package-crested-0-512.fc33
dummy-test-package-rubino-0-512.fc33
java-11-openjdk-11.0.7.10-2.fc33
rust-procfs-0.7.9-1.fc33
rust-libflate_lz77-1.0.0-2.fc33
libyang-1.0.167-1.fc33
quota-4.05-12.fc33
libpcap-1.9.1-4.fc33
rust-libflate-1.0.0-1.fc33
rust-mockito-0.25.1-1.fc33
php-zmq-1.1.3-14.fc33
libpq-12.3-1.fc33
perl-Metrics-Any-0.05-1.fc33
rust-libflate_lz77-1.0.0-1.fc33
rust-testing_logger-0.1.1-1.fc33
ghostscript-9.52-2.fc33
ruby-2.7.1-131.fc33
rust-y4m-0.6.0-1.fc33
rust-datetime-0.5.0-2.fc33
licensecheck-3.0.47-1.fc33
mosquitto-1.6.9-4.fc33
rust-resize-0.4.0-1.fc33
rust-iso8601-0.4.0-1.fc33
rust-arbitrary-0.4.4-1.fc33
libwebp-1.1.0-3.fc33
perl-PkgConfig-LibPkgConf-0.11-1.fc33
iputils-20190515-7.fc33
perl-Alien-Build-2.23-1.fc33
dummy-test-package-gloster-0-538.fc33
xeus-0.23.14-1.fc33
python-rasterio-1.1.4-1.fc33
supertux-0.6.2-1.fc33
perl-Regexp-Pattern-License-3.3.1-1.fc33
kbibtex-0.9-6.fc33
unar-1.10.1-17.fc33
calibre-4.16.0-2.fc33
php-bartlett-php-compatinfo-db-2.14.0-1.fc33
libwebsockets-4.0.10-1.fc33
spill-0.8-21.fc33
memcached-1.6.6-1.fc33
dummy-test-package-crested-0-511.fc33
dummy-test-package-rubino-0-511.fc33
rust-quote-1.0.6-1.fc33
rust-mailparse-0.12.1-1.fc33
rubygem-rspec-expectations-3.9.2-1.fc33
libreoffice-6.4.3.2-2.fc33
python-cartopy-0.18.0-1.fc33
open-vm-tools-11.0.5-4.fc33
adobe-afdko-3.3.0-1.fc33
nuspell-3.1.1-1.fc33
dummy-test-package-gloster-0-537.fc33
grsync-1.2.8-1.fc33
sakura-3.7.1-1.fc33
bindfs-1.14.7-1.fc33
ghc-rpm-macros-2.0.8-1.fc33
redis-6.0.3-1.fc33
dummy-test-package-gloster-0-536.fc33
dummy-test-package-crested-0-509.fc33
dummy-test-package-rubino-0-509.fc33
python-scikit-learn-0.23.0-1.fc33
0ad-0.0.23b-14.fc33
389-ds-base-1.4.4.2-1.fc33.1
bes-3.20.6-3.fc33
boost-1.69.0-19.fc33
cfdg-3.3-2.fc33
community-mysql-8.0.20-2.fc33
cyrus-imapd-3.0.13-7.fc33
evolution-data-server-3.37.1-2.fc33
freerdp-2.0.0-57.20200207git245fc60.fc33
gnucash-3.10-2.fc33
harfbuzz-2.6.6-2.fc33
ibus-qt-1.3.3-25.fc33
icecat-68.8.0-4.rh1.fc33
icu-67.1-1.fc33
idzebra-2.1.4-8.fc33
imv-4.1.0-2.fc33
kiwix-lib-9.1.2-2.fc33
libcdr-0.1.6-2.fc33
libe-book-0.1.3-13.fc33
libical-3.0.8-2.fc33
libicu65-65.1-1.fc33
libmspub-0.1.4-12.fc33
libphonenumber-8.12.1-2.fc33
libqalculate-3.10.0-2.fc33
libqxp-0.0.2-7.fc33
libtoml-0-15.20161213git03e8a3a.fc33
libtranslit-0.0.3-27.fc33
libvisio-0.1.7-4.fc33
libzmf-0.0.2-13.fc33
mapnik-3.0.23-3.fc33
mongo-c-driver-1.17.0~beta-2.fc33
msort-8.53-36.fc33
ncmpcpp-0.8.2-14.fc33
nodejs-node-stringprep-0.8.0-15.fc33
nuspell-3.0.0-6.fc33
openttd-1.10.1-2.fc33
php-7.4.6-2.fc33
php-pecl-http-3.2.3-2.fc33
poedit-2.3.1-3.fc33
postfix-3.5.1-2.fc33
postgresql-12.2-3.fc33
prelude-lml-5.1.0-3.fc33
pyicu-2.4.3-2.fc33
python-mapnik-3.0.23-2.20200224git7da019c.fc33
qt5-qtbase-5.14.2-5.fc33
qt5-qtlocation-5.14.2-2.fc33
qt5-qtwebkit-5.212.0-0.46.alpha4.fc33
raptor2-2.0.15-23.fc33
samba-4.12.2-1.fc33.2
seamonkey-2.53.2-2.fc33
slop-7.4-10.fc33
sword-1.8.1-20.fc33
tarantool-2.2.2.62-2.fc33
tesseract-4.1.1-3.fc33
tin-2.4.4-3.fc33
tracker-2.3.4-2.fc33
tracker-miners-2.3.3-2.fc33
v8-314-3.14.5.10-20.fc33
vte291-0.60.2-2.fc33
webkit2gtk3-2.28.2-3.fc33
widelands-0-0.78.build20.fc33
xfsprogs-5.6.0-2.fc33
xiphos-4.2.1-3.fc33
yaz-5.14.11-21.fc33
zimlib-6.1.1-2.fc33
znc-1.7.5-5.fc33
grass-7.8.3-3.fc33
python-pytest-benchmark-3.2.3-1.fc33
python-guessit-3.1.1-1.fc33
python-subliminal-2.1.0-1.fc33
python-rebulk-2.0.1-1.fc33
python-faker-4.1.0-1.fc33
dummy-test-package-gloster-0-535.fc33
cpu-x-4.0.0-1.fc33
rust-ffsend-0.2.64-1.fc33
xpra-4.0.1-1.fc33
vimiv-qt-0.7.0-1.fc33
rust-ffsend-api-0.6.0-1.fc33
nekohtml-1.9.22-12.fc33
dummy-test-package-crested-0-508.fc33
dummy-test-package-rubino-0-508.fc33
python-joblib-0.15.1-1.fc33
video-downloader-0.2.9-1.fc33
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
6:52 p.m.
On Tue, 2020-05-19 at 16:34 -0700, Adam Williamson wrote:
aha! This looks juicy. systemctl is linked against libpcap.so.1 , part
of libpcap, which is in the list below, and this is the changelog for
it:
* Fri May 15 2020 Michal Ruprich <michalruprich(a)gmail.com> - 14:1.9.1-4
- Enabling rdma support in libpcap
the changelog date is May 15 but the pcap build actually ran on 2020-
05-18, and indeed the last successful Rawhide compose (20200517.n.1)
had 1.9.1-3.fc33, and the first failed compose (20200518.n.0) has
1.9.1-4.fc33.
I'm betting we have a circular dependency problem or something here...
Just to expand on this a bit, what I meant is, I bet adding a
libibverbs.so.0 dependency to libpcap gives dnf an irresolvable
dependency loop, because libibverbs depends on rdma-core and rdma-core
depends on quite a lot of other stuff, including libsystemd and
systemd. So now dnf is in a corner, because systemd needs libpcap which
(now) needs libibverbs which needs rdma-core which needs...systemd.
In fact it seems to be a bit more complex than that, because systemd
doesn't actually list its dependency on libpcap:
[adamw@adam libpcap (master)]$ rpm -q --requires systemd | grep pcap
[adamw@adam libpcap (master)]$
but it must be encoded indirectly somehow, because if you try and
remove libpcap, dnf refuses to because it'd remove systemd:
[adamw@adam libpcap (master)]$ sudo dnf remove libpcap
[sudo] password for adamw:
Error:
Problem: The operation would result in removing the following
protected packages: systemd
anyhow, bottom line is, I think dnf winds up installing systemd and
then a bunch of other packages whose scriptlets call systemctl well
before libibverbs gets installed, and that's where all the failures
come from, as it tries to run systemctl (which is linked against
libpcap which is now linked against libibverbs) before libibverbs is
there.
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
2:49 a.m.
On Tue, May 19, 2020 at 04:52:55PM -0700, Adam Williamson wrote:
In fact it seems to be a bit more complex than that, because systemd
doesn't actually list its dependency on libpcap:
[adamw@adam libpcap (master)]$ rpm -q --requires systemd | grep pcap
[adamw@adam libpcap (master)]$
but it must be encoded indirectly somehow, because if you try and
remove libpcap, dnf refuses to because it'd remove systemd:
[adamw@adam libpcap (master)]$ sudo dnf remove libpcap
[sudo] password for adamw:
Error:
Problem: The operation would result in removing the following
protected packages: systemd
systemd requires libip4tc.so.2()(64bit) provided by iptables-libs,
and iptables-libs requires libpcap.so.1()(64bit) provided by libpcap.
Zbyszek
7:19 p.m.
On Tue, 2020-05-19 at 16:34 -0700, Adam Williamson wrote:
On Tue, 2020-05-19 at 15:45 -0700, Adam Williamson wrote:
> still, having trouble pinning down a culprit; it's not kernel-5.7.0-
> 0.rc6.1.fc33 as the 20200518.n.0 compose failed, and that was run with
> the previous kernel build, which *succeeded* in the 20200517.n.0
> compose...
>
> I guess we get to poke through everything built around the 17th and try
> to find a relevant change? :)
This should be *more or less* the list of candidate culprits, which I
just heath robinson'd out of Bodhi (yet again I wish we got compose
reports for doomed composes, it'd make this easier...). grub2 and
openssl are the ones that jump out at me right away...ceph is also in
there, and ceph is one of the things that *is* linked to libibverbs ,
though the change in ceph doesn't seem like a significant one...
aha! This looks juicy. systemctl is linked against libpcap.so.1 , part
of libpcap, which is in the list below, and this is the changelog for
it:
* Fri May 15 2020 Michal Ruprich <michalruprich(a)gmail.com> - 14:1.9.1-4
- Enabling rdma support in libpcap
the changelog date is May 15 but the pcap build actually ran on 2020-
05-18, and indeed the last successful Rawhide compose (20200517.n.1)
had 1.9.1-3.fc33, and the first failed compose (20200518.n.0) has
1.9.1-4.fc33.
I'm betting we have a circular dependency problem or something here...
Well, seems like I may be half right. We untagged libcap-1.9.1-4.fc33
and I re-ran the nbdkit build. It seems that this has fixed the "error
while loading shared libraries: libibverbs.so.1: cannot open shared
object file: No such file or directory" errors...but not the segfaults
:( we now just get segfaults all the way:
DEBUG util.py:602: Running transaction
DEBUG util.py:602: /var/tmp/rpm-tmp.yHBGhZ: line 1: 1976807 Segmentation fault (core
dumped) systemd-machine-id-setup &> /dev/null
DEBUG util.py:602: /var/tmp/rpm-tmp.yHBGhZ: line 21: 1976809 Segmentation fault
(core dumped) systemctl daemon-reexec &> /dev/null
DEBUG util.py:602: /var/tmp/rpm-tmp.yHBGhZ: line 23: 1976811 Segmentation fault
(core dumped) journalctl --update-catalog &> /dev/null
DEBUG util.py:602: /var/tmp/rpm-tmp.yHBGhZ: line 24: 1976813 Segmentation fault
(core dumped) systemd-tmpfiles --create &> /dev/null
DEBUG util.py:602: /var/tmp/rpm-tmp.yHBGhZ: line 50: 1976821 Segmentation fault
(core dumped) systemctl preset-all &> /dev/null
DEBUG util.py:602: /var/tmp/rpm-tmp.yHBGhZ: line 50: 1976823 Segmentation fault
(core dumped) systemctl --global preset-all &> /dev/null
DEBUG util.py:602: /var/tmp/rpm-tmp.TL9iz0: line 6: 1976826 Segmentation fault (core
dumped) /usr/bin/systemctl --no-reload preset dbus.socket
so there might be two different problems here, possibly?
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
8:32 p.m.
On Tue, 2020-05-19 at 17:19 -0700, Adam Williamson wrote:
On Tue, 2020-05-19 at 16:34 -0700, Adam Williamson wrote:
> On Tue, 2020-05-19 at 15:45 -0700, Adam Williamson wrote:
> > still, having trouble pinning down a culprit; it's not kernel-5.7.0-
> > 0.rc6.1.fc33 as the 20200518.n.0 compose failed, and that was run with
> > the previous kernel build, which *succeeded* in the 20200517.n.0
> > compose...
> >
> > I guess we get to poke through everything built around the 17th and try
> > to find a relevant change? :)
>
> This should be *more or less* the list of candidate culprits, which I
> just heath robinson'd out of Bodhi (yet again I wish we got compose
> reports for doomed composes, it'd make this easier...). grub2 and
> openssl are the ones that jump out at me right away...ceph is also in
> there, and ceph is one of the things that *is* linked to libibverbs ,
> though the change in ceph doesn't seem like a significant one...
>
> aha! This looks juicy. systemctl is linked against libpcap.so.1 , part
> of libpcap, which is in the list below, and this is the changelog for
> it:
>
> * Fri May 15 2020 Michal Ruprich <michalruprich(a)gmail.com> - 14:1.9.1-4
> - Enabling rdma support in libpcap
>
> the changelog date is May 15 but the pcap build actually ran on 2020-
> 05-18, and indeed the last successful Rawhide compose (20200517.n.1)
> had 1.9.1-3.fc33, and the first failed compose (20200518.n.0) has
> 1.9.1-4.fc33.
>
> I'm betting we have a circular dependency problem or something here...
Well, seems like I may be half right. We untagged libcap-1.9.1-4.fc33
and I re-ran the nbdkit build. It seems that this has fixed the "error
while loading shared libraries: libibverbs.so.1: cannot open shared
object file: No such file or directory" errors...but not the segfaults
:( we now just get segfaults all the way:
DEBUG util.py:602: Running transaction
DEBUG util.py:602: /var/tmp/rpm-tmp.yHBGhZ: line 1: 1976807 Segmentation fault
(core dumped) systemd-machine-id-setup &> /dev/null
DEBUG util.py:602: /var/tmp/rpm-tmp.yHBGhZ: line 21: 1976809 Segmentation fault
(core dumped) systemctl daemon-reexec &> /dev/null
DEBUG util.py:602: /var/tmp/rpm-tmp.yHBGhZ: line 23: 1976811 Segmentation fault
(core dumped) journalctl --update-catalog &> /dev/null
DEBUG util.py:602: /var/tmp/rpm-tmp.yHBGhZ: line 24: 1976813 Segmentation fault
(core dumped) systemd-tmpfiles --create &> /dev/null
DEBUG util.py:602: /var/tmp/rpm-tmp.yHBGhZ: line 50: 1976821 Segmentation fault
(core dumped) systemctl preset-all &> /dev/null
DEBUG util.py:602: /var/tmp/rpm-tmp.yHBGhZ: line 50: 1976823 Segmentation fault
(core dumped) systemctl --global preset-all &> /dev/null
DEBUG util.py:602: /var/tmp/rpm-tmp.TL9iz0: line 6: 1976826 Segmentation fault
(core dumped) /usr/bin/systemctl --no-reload preset dbus.socket
so there might be two different problems here, possibly?
OK, so I dug into the history of the segfaults a bit. Turns out they've
been going on for a few days. They only seem to happen on builds where
the systemd package gets pulled into the buildroot. The closest delta
I've got is this:
GOOD: https://koji.fedoraproject.org/koji/buildinfo?buildID=1507622 (Fri, 15 May 2020
18:01:07 UTC)
BAD: https://koji.fedoraproject.org/koji/buildinfo?buildID=1507650 (Fri, 15 May 2020
19:49:31 UTC)
"GOOD" is the latest build I can find where systemd was pulled in as a
build dep (so the bug "should have" happened) and the bug didn't
happen. "BAD" is the earliest build I can find where the bug *did*
happen.
The most suspicious change between the two build envs that I can see is
openssl. GOOD has openssl-1.1.1g-1.fc33.x86_64 , and BAD has
openssl-1.1.1g-2.fc33.x86_64 . I'm gonna try doing an openssl build
with the patch from -2 reverted and see where that gets us.
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
10:24 p.m.
On Tue, 2020-05-19 at 18:32 -0700, Adam Williamson wrote:
The most suspicious change between the two build envs that I can see is
openssl. GOOD has openssl-1.1.1g-1.fc33.x86_64 , and BAD has
openssl-1.1.1g-2.fc33.x86_64 . I'm gonna try doing an openssl build
with the patch from -2 reverted and see where that gets us.
OK, I think that did the trick. nbdkit's x86_64 build succeeded this
time, and the log shows no systemctl crashes:
https://kojipkgs.fedoraproject.org//work/tasks/7666/44717666/root.log
I'll file a bug on openssl to let tmraz know about the problem.
Hopefully between the two fixes, the next Rawhide may also be fixed...
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
10:43 p.m.
On Tue, 2020-05-19 at 20:24 -0700, Adam Williamson wrote:
On Tue, 2020-05-19 at 18:32 -0700, Adam Williamson wrote:
> The most suspicious change between the two build envs that I can see is
> openssl. GOOD has openssl-1.1.1g-1.fc33.x86_64 , and BAD has
> openssl-1.1.1g-2.fc33.x86_64 . I'm gonna try doing an openssl build
> with the patch from -2 reverted and see where that gets us.
OK, I think that did the trick. nbdkit's x86_64 build succeeded this
time, and the log shows no systemctl crashes:
https://kojipkgs.fedoraproject.org//work/tasks/7666/44717666/root.log
I'll file a bug on openssl to let tmraz know about the problem.
Hopefully between the two fixes, the next Rawhide may also be fixed...
Bug reports:
openssl - https://bugzilla.redhat.com/show_bug.cgi?id=1837809
libpcap - https://bugzilla.redhat.com/show_bug.cgi?id=1837812
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
2:39 a.m.
On Tue, May 19, 2020 at 08:43:01PM -0700, Adam Williamson wrote:
On Tue, 2020-05-19 at 20:24 -0700, Adam Williamson wrote:
> On Tue, 2020-05-19 at 18:32 -0700, Adam Williamson wrote:
> > The most suspicious change between the two build envs that I can see is
> > openssl. GOOD has openssl-1.1.1g-1.fc33.x86_64 , and BAD has
> > openssl-1.1.1g-2.fc33.x86_64 . I'm gonna try doing an openssl build
> > with the patch from -2 reverted and see where that gets us.
>
> OK, I think that did the trick. nbdkit's x86_64 build succeeded this
> time, and the log shows no systemctl crashes:
>
> https://kojipkgs.fedoraproject.org//work/tasks/7666/44717666/root.log
>
> I'll file a bug on openssl to let tmraz know about the problem.
> Hopefully between the two fixes, the next Rawhide may also be fixed...
Bug reports:
openssl - https://bugzilla.redhat.com/show_bug.cgi?id=1837809
libpcap - https://bugzilla.redhat.com/show_bug.cgi?id=1837812
Adam, thanks for tracking this down!
Zbyszek
4:26 a.m.
On Tue, May 19, 2020 at 08:43:01PM -0700, Adam Williamson wrote:
On Tue, 2020-05-19 at 20:24 -0700, Adam Williamson wrote:
> On Tue, 2020-05-19 at 18:32 -0700, Adam Williamson wrote:
> > The most suspicious change between the two build envs that I can see is
> > openssl. GOOD has openssl-1.1.1g-1.fc33.x86_64 , and BAD has
> > openssl-1.1.1g-2.fc33.x86_64 . I'm gonna try doing an openssl build
> > with the patch from -2 reverted and see where that gets us.
>
> OK, I think that did the trick. nbdkit's x86_64 build succeeded this
> time, and the log shows no systemctl crashes:
>
> https://kojipkgs.fedoraproject.org//work/tasks/7666/44717666/root.log
>
> I'll file a bug on openssl to let tmraz know about the problem.
> Hopefully between the two fixes, the next Rawhide may also be fixed...
Bug reports:
openssl - https://bugzilla.redhat.com/show_bug.cgi?id=1837809
libpcap - https://bugzilla.redhat.com/show_bug.cgi?id=1837812
Excellent detective work, thanks!
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
libguestfs lets you edit virtual machines. Supports shell scripting,
bindings from many languages. http://libguestfs.org
2:03 a.m.
On Tue, May 19, 2020 at 08:24:34PM -0700, Adam Williamson wrote:
On Tue, 2020-05-19 at 18:32 -0700, Adam Williamson wrote:
>
> The most suspicious change between the two build envs that I can see is
> openssl. GOOD has openssl-1.1.1g-1.fc33.x86_64 , and BAD has
> openssl-1.1.1g-2.fc33.x86_64 . I'm gonna try doing an openssl build
> with the patch from -2 reverted and see where that gets us.
OK, I think that did the trick. nbdkit's x86_64 build succeeded this
time, and the log shows no systemctl crashes:
https://kojipkgs.fedoraproject.org//work/tasks/7666/44717666/root.log
I'll file a bug on openssl to let tmraz know about the problem.
Hopefully between the two fixes, the next Rawhide may also be fixed...
Nice detective work. You could have save your work with Koschei.
<https://koschei.fedoraproject.org/package/nbdkit> nbdkit history,
<https://koschei.fedoraproject.org/build/8351572> first failure with this test
failure on x86_64:
supermin: exception:
Sys_error("/lib/modules/5.7.0-0.rc5.20200515git1ae7efb38854.1.fc33.x86_64/modules.dep:
No such file or directory")
with only 5 source packages changed and openssl going from 1:1.1.1g-1.fc33 to
1:1.1.1g-2.fc33 being one of them.
-- Petr
10:48 a.m.
On Wed, 2020-05-20 at 09:03 +0200, Petr Pisar wrote:
On Tue, May 19, 2020 at 08:24:34PM -0700, Adam Williamson wrote:
> On Tue, 2020-05-19 at 18:32 -0700, Adam Williamson wrote:
> > The most suspicious change between the two build envs that I can see is
> > openssl. GOOD has openssl-1.1.1g-1.fc33.x86_64 , and BAD has
> > openssl-1.1.1g-2.fc33.x86_64 . I'm gonna try doing an openssl build
> > with the patch from -2 reverted and see where that gets us.
>
> OK, I think that did the trick. nbdkit's x86_64 build succeeded this
> time, and the log shows no systemctl crashes:
>
> https://kojipkgs.fedoraproject.org//work/tasks/7666/44717666/root.log
>
> I'll file a bug on openssl to let tmraz know about the problem.
> Hopefully between the two fixes, the next Rawhide may also be fixed...
Nice detective work. You could have save your work with Koschei.
<https://koschei.fedoraproject.org/package/nbdkit> nbdkit history,
<https://koschei.fedoraproject.org/build/8351572> first failure with this test
failure on x86_64:
supermin: exception:
Sys_error("/lib/modules/5.7.0-0.rc5.20200515git1ae7efb38854.1.fc33.x86_64/modules.dep:
No such file or directory")
with only 5 source packages changed and openssl going from 1:1.1.1g-1.fc33 to
1:1.1.1g-2.fc33 being one of them.
Good point! Somehow Koschei just didn't come to mind as something that
would be helpful here, but it would indeed have saved me a bit of time.
I did look for packages with frequent 'official' builds to try and find
one which would show a narrow window where the bug appeared, but of
course Koschei would've been the ideal place to look :)
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
1434
days inactive
1435
days old
16 comments
5 participants
participants (5)
-
Adam Williamson -
Doug Ledford -
Petr Pisar -
Richard W.M. Jones -
Zbigniew Jędrzejewski-Szmek