Re: Lots of systemctl segfaults in Koji Rawhide

[Doug Ledford]

On Tue, 2020-05-19 at 15:45 -0700, Adam Williamson wrote:
>  On Tue, 2020-05-19 at 18:16 -0400, Doug Ledford wrote:
>  > But this is different, and it's the cause of your problem (well,
>  > it's
>  > the immediate cause anyway).  The kernel-install script is failing
>  > because it's passing /etc/kernel/install.d/ to something that wants
>  > something other than a directory.  That could be because a failed
>  > scriptlet elsewhere has resulted in that directory being empty, so
>  > some
>  > glob is returning the directory instead of the files in the
>  > directory,
>  > so it could still be the systemctl issue, but it would be worth
>  > looking
>  > into install-kernel to see.
>  > 
>  > OK, so install-kernel is part of systemd (I'm seeing a pattern
>  > here). 
>  > It generates an array of plugins that should be called for the new
>  > kernel.  It does all plugins in .install, /etc/kernel/install.d/,
>  > and
>  > /usr/lib/kernel/install.d.  My guess here is that the %POSTTRANS of
>  > the
>  > kernel-core package is calling install-kernel, which is failing with
>  > the
>  > above error, causing the kernel-core %POSTTRANS to error out
>  > prematurely, resulting in the missing modules.dep file that is
>  > breaking
>  > your build.  At this point, I can't see what the problem can be
>  > other
>  > than either a bad build of systemd, or if the recent upgrade to
>  > rdma-
>  > core-29, with the new libibverbs-29 package, has caused a failure in
>  > systemd because it needs relinked or something.  But that can only
>  > be
>  > the case if it's some sort of weak dependency based on dlopen as
>  > both
>  > the rpm tools and ldd are not picking up the libibverbs
>  > dependency.  If
>  > that's the case, the systemd and/or udev rpm packages should have an
>  > explicit requires on libibverbs I think.
>  > 
>  > Anyway, at this point, I don't know if the rdma-core-owner people
>  > can
>  > help.  I think this is first in the hands of the systemd folks.
>  
>  Well, systemd hasn't changed for nearly a month - it was last built on
>  2020-04-21. nbdkit was built successfully a few days ago, so blaming
>  systemd here feels wrong. 
Ok, then something else has changed that is causing systemd problems
then.

>  It's notable that the last couple of Rawhide composes have also
> failed
>  with an odd kernel-related error:
>  
>  https://koji.fedoraproject.org/koji/taskinfo?taskID=44651583
>  
>  DEBUG util.py:602:  Traceback (most recent call last):
>  DEBUG util.py:602:    File "/usr/sbin/lorax", line 223, in <module>
>  DEBUG util.py:602:      main()
>  DEBUG util.py:602:    File "/usr/sbin/lorax", line 204, in main
>  DEBUG util.py:602:      lorax.run(dnfbase, opts.product, opts.version,
>  opts.release,
>  DEBUG util.py:602:    File "/usr/lib/python3.8/site-
>  packages/pylorax/__init__.py", line 354, in run
>  DEBUG
>  util.py:602:      treebuilder.rebuild_initrds(add_args=anaconda_args)
>  DEBUG util.py:602:    File "/usr/lib/python3.8/site-
>  packages/pylorax/treebuilder.py", line 308, in rebuild_initrds
>  DEBUG util.py:602:      raise Exception("No kernels found, cannot
>  rebuild_initrds")
>  DEBUG util.py:602:  Exception: No kernels found, cannot
>  rebuild_initrds
>  
>  still, having trouble pinning down a culprit; it's not kernel-5.7.0-
>  0.rc6.1.fc33 as the 20200518.n.0 compose failed, and that was run with
>  the previous kernel build, which *succeeded* in the 20200517.n.0
>  compose...
>  
>  I guess we get to poke through everything built around the 17th and
>  try
>  to find a relevant change? :) 
Aren't highly complex, interdependent systems with different owners of
different components fun? :-)

-- 
Doug Ledford <dledford(a)redhat.com&gt;
    GPG KeyID: B826A3330E572FDD
    Fingerprint = AE6B 1BDA 122B 23B4 265B  1274 B826 A333 0E57 2FDD