ELN rebuild ordering fail (was Re: userspace-rcu 0.13.0 soname bump)

[Daniel P. Berrangé]

On Tue, Jun 08, 2021 at 02:30:42PM -0400, Michael Jeanson wrote:
>  I have started the process to update userspace-rcu to 0.13 in rawhide
> which
>  implies a soname bump to 8.
>  
>  From what I understand, the following packages will need to be rebuilt:
>  
>  device-mapper-multipath
>  glusterfs
>  knot
>  libntirpc
>  lttng-tools
>  lttng-ust
>  netsniff-ng
>  nfs-ganesha
>  
>  
>  I have created a side tag 'f35-build-side-42347' and built userspace-rcu,
>  lttng-ust and lttng-tools. At this point I'm unsure what the rest of the
>  procedure is to get the other packages built in this tag and then get them
>  pushed to rawhide once it's done. 
Using the side tag for the builds of lttng-ust did the job fine for
Fedora rawhide builds, but I'm seeing problems with the ELN rebuilds
I'm getting spammed frequently by the failing libvirt builds for the
ELN rebuilds. 8 failed libvirt builds, and another related 18 failed
libvirt-python builds just since last night

Most recent is

  https://koji.fedoraproject.org/koji/taskinfo?taskID=71422991

Looking at the failed s390x build logs:

  https://kojipkgs.fedoraproject.org//work/tasks/2991/71422991/root.log

I see the tell tail sign of the soname bump

Error: 
 Problem 1: package librados-devel-2:16.2.4-5.eln112.s390x requires
librados.so.2()(64bit), but none of the providers can be installed
  - package librados-devel-2:16.2.4-5.eln112.s390x requires librados_tp.so.2()(64bit), but
none of the providers can be installed
  - package librados-devel-2:16.2.4-5.eln112.s390x requires librados2 = 2:16.2.4-5.eln112,
but none of the providers can be installed
  - package librados2-2:16.2.4-5.eln112.s390x requires liblttng-ust.so.0()(64bit), but
none of the providers can be installed
  - conflicting requests
  - nothing provides liburcu-bp.so.6()(64bit) needed by lttng-ust-2.12.2-4.eln112.s390x
  - nothing provides liburcu-cds.so.6()(64bit) needed by lttng-ust-2.12.2-4.eln112.s390x


librados-devel can't be installed because librados2 can't be installed
because liblttng-ust can't be installed, because it depends on the
old soname of userspace-rcu.

Looking at koji logs, I can see the most recent build of userspace-rcu
version 0.13.0-2 which has the soname bump:

 - rawhide: 2021-06-08 16:10:21
 - eln112: 2021-06-23 15:17:36

while the rebuilds of lttng-ust were:

 - rawhide: 2021-06-08 17:31:22
 - eln112: 2021-06-11 19:52:32

So the rebuilds in ELN were not only in the wrong order, they were
weeks apart in the wrong order.

According to this proposal to fix ELN, the side tag builds  are all
run in parallel. Obviously this relies on luck to work, but in this
particular case I don't see any parallelism. The userspace-rcu
build didn't run till 12 days later.

https://lists.fedoraproject.org/archives/list/[e-mail].o...

Anyway, it is good to see there's a proposal to fix ELN schedular
but I'm wondering what the right way to fix this immediate problem
is ?

I presume we need a bogus release bump and rebuild of lttng-ust to
be run in rawhide in order to trigger ELN into fixing itself ?

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|