Fedora 25 Koji buildroots broken…

[Björn 'besser82' Esser]

Hi there!

The Fedora 25 buildroot on Koji is b0rk3n…  :(

DEBUG util.py:435:  Error: nothing provides libGL.so.1()(64bit) needed by muffin-devel-3.2.1-1.fc25.x86_64.
DEBUG util.py:435:  nothing provides libEGL.so.1()(64bit) needed by clutter-1.26.0-1.fc25.x86_64.
DEBUG util.py:435:  nothing provides libGL.so.1()(64bit) needed by cairo-gobject-1.14.8-1.fc25.x86_64.
DEBUG util.py:435:  nothing provides libGL.so.1()(64bit) needed by cairo-1.14.8-1.fc25.x86_64.
DEBUG util.py:435:  nothing provides libGL.so.1()(64bit) needed by cairo-gobject-1.14.8-1.fc25.x86_64.
DEBUG util.py:435:  nothing provides libGL.so.1()(64bit) needed by cairo-1.14.8-1.fc25.x86_64.
DEBUG util.py:435:  nothing provides libGL.so.1()(64bit) needed by cairo-1.14.8-1.fc25.x86_64.
DEBUG util.py:435:  nothing provides libglvnd-glx(x86-64) needed by mesa-libGL-13.0.3-4.fc25.x86_64

Can the one who broke fix it, please?

Cheers
   Björn

[Björn 'besser82' Esser]

Allrighty…  Looks like the override for libglvnd somehow got untagged…  
Just re-tagged in f25-build…  Should be fixed now.

Am 29.01.2017 um 17:19 schrieb Björn 'besser82' Esser:
> Hi there!
>
> The Fedora 25 buildroot on Koji is b0rk3n…  :(
>
> DEBUG util.py:435:  Error: nothing provides libGL.so.1()(64bit) needed by muffin-devel-3.2.1-1.fc25.x86_64.
> DEBUG util.py:435:  nothing provides libEGL.so.1()(64bit) needed by clutter-1.26.0-1.fc25.x86_64.
> DEBUG util.py:435:  nothing provides libGL.so.1()(64bit) needed by cairo-gobject-1.14.8-1.fc25.x86_64.
> DEBUG util.py:435:  nothing provides libGL.so.1()(64bit) needed by cairo-1.14.8-1.fc25.x86_64.
> DEBUG util.py:435:  nothing provides libGL.so.1()(64bit) needed by cairo-gobject-1.14.8-1.fc25.x86_64.
> DEBUG util.py:435:  nothing provides libGL.so.1()(64bit) needed by cairo-1.14.8-1.fc25.x86_64.
> DEBUG util.py:435:  nothing provides libGL.so.1()(64bit) needed by cairo-1.14.8-1.fc25.x86_64.
> DEBUG util.py:435:  nothing provides libglvnd-glx(x86-64) needed by mesa-libGL-13.0.3-4.fc25.x86_64
>
> Can the one who broke fix it, please?
>
> Cheers
>    Björn

[Hans de Goede]

Hi,

On 01/29/2017 05:23 PM, Björn 'besser82' Esser wrote:
> Allrighty…  Looks like the override for libglvnd somehow got untagged…  Just re-tagged in f25-build…  Should be fixed now.

Well, that is a fix, but the real problem is that either the new libglvnd enabled
mesa should not be in updates-stable and thus not in the buildroot; or
both the new libglvnd enabled mesa and the new libglvnd should be in updates-stable.

What happened is I created an update in bodhi for libglvnd-0.2.999-7.gitdc16f8c.fc25
and mesa-13.0.3-3.fc25.

Then airlied did an unrelated bug-fix to mesa and created an update in bodhi for
mesa-13.0.3-4.fc25. This update did not obsolete my previous update, did not
inherit it bugs and did not inherit the libglvnd package which was only in my
update.

When I noticed things both updates were in perpetual locked state, when they
were finally unlocked airlied's update got auto-pushed to stable because of
karma (I had auto-karma disabled on mine), so now we have a mesa depending on
the latest libglvnd in updates-stable, without the latest libglvnd.

When I tried to also push my update to stable, I could not as bodhi complained
there was a newer mesa already in stable, so I had to remove mesa from my
update (why does it detect conflicts like this on push and not when someone
creates a conflicting update?), loosing all karma??? and now it is pending for
testing.

Frankly I blame this whole mess on bodhi, it should not allow creating
updates which partly obsolete other updates, there likely is a good reason
the partly obsoleted update bundled multiple packages in one update; or
it should allow it and then simply add all non obsoleted packages from
the obsoleted update to the new update and obsolete the old one.

I've had trouble with bodhi not doing proper obsoleting more often
lately, as well as having trouble with updates which involve obsoleting
getting in a perpetual locked state. Again frankly I've the feeling that
bodhi has regressed and that the old bodhi was more stable. We really need
to do better here, both with obsoleting and with the locked state thing.

Why does the admin side of bodhi not run a check to see everything is
unlocked again once the push is complete ? That should at least catch the
lock issue ?

For the 2 updates in question see:

https://bodhi.fedoraproject.org/updates/FEDORA-2017-9f6383547e
https://bodhi.fedoraproject.org/updates/FEDORA-2017-9c9c0899f9

Regards,

Hans
> Am 29.01.2017 um 17:19 schrieb Björn 'besser82' Esser:
> > Hi there!
> >
> > The Fedora 25 buildroot on Koji is b0rk3n…  :(
> >
> > DEBUG util.py:435:  Error: nothing provides libGL.so.1()(64bit) needed by muffin-devel-3.2.1-1.fc25.x86_64.
> > DEBUG util.py:435:  nothing provides libEGL.so.1()(64bit) needed by clutter-1.26.0-1.fc25.x86_64.
> > DEBUG util.py:435:  nothing provides libGL.so.1()(64bit) needed by cairo-gobject-1.14.8-1.fc25.x86_64.
> > DEBUG util.py:435:  nothing provides libGL.so.1()(64bit) needed by cairo-1.14.8-1.fc25.x86_64.
> > DEBUG util.py:435:  nothing provides libGL.so.1()(64bit) needed by cairo-gobject-1.14.8-1.fc25.x86_64.
> > DEBUG util.py:435:  nothing provides libGL.so.1()(64bit) needed by cairo-1.14.8-1.fc25.x86_64.
> > DEBUG util.py:435:  nothing provides libGL.so.1()(64bit) needed by cairo-1.14.8-1.fc25.x86_64.
> > DEBUG util.py:435:  nothing provides libglvnd-glx(x86-64) needed by mesa-libGL-13.0.3-4.fc25.x86_64
> >
> > Can the one who broke fix it, please?
> >
> > Cheers
> >   Björn
>
> devel mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]

[Kevin Fenzi]

On Sun, 29 Jan 2017 18:12:59 +0100
Hans de Goede  wrote:
> Hi,
>
> On 01/29/2017 05:23 PM, Björn 'besser82' Esser wrote:
> > Allrighty…  Looks like the override for libglvnd somehow got
> > untagged…  Just re-tagged in f25-build…  Should be fixed now.
>
> Well, that is a fix, but the real problem is that either the new
> libglvnd enabled mesa should not be in updates-stable and thus not in
> the buildroot; or both the new libglvnd enabled mesa and the new
> libglvnd should be in updates-stable.
>
> What happened is I created an update in bodhi for
> libglvnd-0.2.999-7.gitdc16f8c.fc25 and mesa-13.0.3-3.fc25.
>
> Then airlied did an unrelated bug-fix to mesa and created an update
> in bodhi for mesa-13.0.3-4.fc25. This update did not obsolete my
> previous update, did not inherit it bugs and did not inherit the
> libglvnd package which was only in my update.

Yeah, currently bodhi doesn't obsolete if the other update is locked or
if the other update has a different packageset.
> When I noticed things both updates were in perpetual locked state,
> when they were finally unlocked airlied's update got auto-pushed to
> stable because of karma (I had auto-karma disabled on mine), so now
> we have a mesa depending on the latest libglvnd in updates-stable,
> without the latest libglvnd.

:(
> When I tried to also push my update to stable, I could not as bodhi
> complained there was a newer mesa already in stable, so I had to
> remove mesa from my update (why does it detect conflicts like this on
> push and not when someone creates a conflicting update?), loosing all
> karma??? and now it is pending for testing.
>
> Frankly I blame this whole mess on bodhi, it should not allow creating
> updates which partly obsolete other updates, there likely is a good
> reason the partly obsoleted update bundled multiple packages in one
> update; or it should allow it and then simply add all non obsoleted
> packages from the obsoleted update to the new update and obsolete the
> old one.
>
> I've had trouble with bodhi not doing proper obsoleting more often
> lately, as well as having trouble with updates which involve
> obsoleting getting in a perpetual locked state. Again frankly I've
> the feeling that bodhi has regressed and that the old bodhi was more
> stable. We really need to do better here, both with obsoleting and
> with the locked state thing.

I'll leave the above for bodhi developers to comment on.
This has always been a complicated area, but I agree we could do
better.
> Why does the admin side of bodhi not run a check to see everything is
> unlocked again once the push is complete ? That should at least catch
> the lock issue ?

It does. The "lock issue" was just because we couldn't get a
f25-updates-testing push to complete due to rpm-ostree issues. So, IMHO
it was completely right to lock those until the push finished.

I guess of your suggestions it might indeed be best to reject a new
update when there's already one in existance that it cannot obsolete
(because it's locked or has a different packageset). I'm sure that will
annoy some people, but seems cleanest to me and then allows people to
discuss and modify the in-flight update or whatever.

kevin

[Hans de Goede]

Hi,

On 01/29/2017 08:00 PM, Kevin Fenzi wrote:
> On Sun, 29 Jan 2017 18:12:59 +0100
> Hans de Goede  wrote:
> > Hi,
> >
> > On 01/29/2017 05:23 PM, Björn 'besser82' Esser wrote:
> > > Allrighty…  Looks like the override for libglvnd somehow got
> > > untagged…  Just re-tagged in f25-build…  Should be fixed now.
> >
> > Well, that is a fix, but the real problem is that either the new
> > libglvnd enabled mesa should not be in updates-stable and thus not in
> > the buildroot; or both the new libglvnd enabled mesa and the new
> > libglvnd should be in updates-stable.
> >
> > What happened is I created an update in bodhi for
> > libglvnd-0.2.999-7.gitdc16f8c.fc25 and mesa-13.0.3-3.fc25.
> >
> > Then airlied did an unrelated bug-fix to mesa and created an update
> > in bodhi for mesa-13.0.3-4.fc25. This update did not obsolete my
> > previous update, did not inherit it bugs and did not inherit the
> > libglvnd package which was only in my update.
>
> Yeah, currently bodhi doesn't obsolete if the other update is locked or
> if the other update has a different packageset.
> > When I noticed things both updates were in perpetual locked state,
> > when they were finally unlocked airlied's update got auto-pushed to
> > stable because of karma (I had auto-karma disabled on mine), so now
> > we have a mesa depending on the latest libglvnd in updates-stable,
> > without the latest libglvnd.
>
> :(

Yeah, this is quite unfortunate :(
> > When I tried to also push my update to stable, I could not as bodhi
> > complained there was a newer mesa already in stable, so I had to
> > remove mesa from my update (why does it detect conflicts like this on
> > push and not when someone creates a conflicting update?), loosing all
> > karma??? and now it is pending for testing.
> >
> > Frankly I blame this whole mess on bodhi, it should not allow creating
> > updates which partly obsolete other updates, there likely is a good
> > reason the partly obsoleted update bundled multiple packages in one
> > update; or it should allow it and then simply add all non obsoleted
> > packages from the obsoleted update to the new update and obsolete the
> > old one.
> >
> > I've had trouble with bodhi not doing proper obsoleting more often
> > lately, as well as having trouble with updates which involve
> > obsoleting getting in a perpetual locked state. Again frankly I've
> > the feeling that bodhi has regressed and that the old bodhi was more
> > stable. We really need to do better here, both with obsoleting and
> > with the locked state thing.
>
> I'll leave the above for bodhi developers to comment on.
> This has always been a complicated area, but I agree we could do
> better.

I've filed:

https://github.com/fedora-infra/bodhi/issues/1254

To track this.

Regaards,

Hans
> > Why does the admin side of bodhi not run a check to see everything is
> > unlocked again once the push is complete ? That should at least catch
> > the lock issue ?
>
> It does. The "lock issue" was just because we couldn't get a
> f25-updates-testing push to complete due to rpm-ostree issues. So, IMHO
> it was completely right to lock those until the push finished.
>
> I guess of your suggestions it might indeed be best to reject a new
> update when there's already one in existance that it cannot obsolete
> (because it's locked or has a different packageset). I'm sure that will
> annoy some people, but seems cleanest to me and then allows people to
> discuss and modify the in-flight update or whatever.
>
> kevin
>
> devel mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]

[Kevin Kofler]

Hans de Goede wrote:
> Well, that is a fix, but the real problem is that either the new libglvnd
> enabled mesa should not be in updates-stable and thus not in the
> buildroot; or both the new libglvnd enabled mesa and the new libglvnd
> should be in updates-stable.

This is the result of 4 very poor decisions, by different people/groups:

1. the decision to enable libglvnd in an update to a stable release. IMHO,
   such a change is totally unsuitable for a stable release update and
   should have been done in Rawhide only.

2. the decision to block Bodhi pushes on ostree failures. Without that,
   Bodhi would not have been stuck for days with all updates locked and this
   fiasco might possibly have been avoided. If the ostree compose fails, the
   Bodhi push should just proceed with an empty or old ostree directory
   (whatever is easier to implement). Rarely used experimental delivery
   methods should not hold the entire distribution hostage.

3. the decision to use autokarma. This is just yet another broken update
   that went stable due to autokarma. Autokarma is an absolutely
   unacceptable practice and should not be allowed. All push requests should
   be issued by a human after reviewing the status.

4. the decision to disallow direct stable pushes. The right way to fix the
   issue quickly, limiting the damage once done, would have been to push
   libglvnd directly to stable. But Bodhi won't allow that. Direct stable
   pushes are an essential mechanism to fix regressions and to limit the
   number of affected users by minimizing the exposure time to the
   regression.

Kevin Kofler

[Kevin Fenzi]

On Mon, 30 Jan 2017 00:53:05 +0100
Kevin Kofler  wrote:
> Hans de Goede wrote:
> > Well, that is a fix, but the real problem is that either the new
> > libglvnd enabled mesa should not be in updates-stable and thus not
> > in the buildroot; or both the new libglvnd enabled mesa and the new
> > libglvnd should be in updates-stable.
>
> This is the result of 4 very poor decisions, by different
> people/groups:
> the decision to enable libglvnd in an update to a stable release.
>
> IMHO, such a change is totally unsuitable for a stable release update
> and should have been done in Rawhide only.

Perhaps so. I agree it definitely shouldn't have been pushed without
fixing all the packages that are broken by the update.
> the decision to block Bodhi pushes on ostree failures. Without
>
> that, Bodhi would not have been stuck for days with all updates
> locked and this fiasco might possibly have been avoided. If the
> ostree compose fails, the Bodhi push should just proceed with an
> empty or old ostree directory (whatever is easier to implement).
> Rarely used experimental delivery methods should not hold the entire
> distribution hostage.

I disagree with you here. I think ostree is important and people are
using it. However, there's two (IMHO better) ways forward here. 
First we could (and should) just back out
rpm-ostree/ostree/bublewrap/whatever updates breaks composing and debug
it on the side. Second, the atomic sig is looking at changing how the
updates stream is made and only push it every 2 weeks. That would
remove it from being done by bodhi.
> the decision to use autokarma. This is just yet another broken
>
> update that went stable due to autokarma. Autokarma is an absolutely
>    unacceptable practice and should not be allowed. All push requests
> should be issued by a human after reviewing the status.

I disagree. If autokarma wasn't used then the update could well have
been pushed again after the maintainer saw that it was able to be
pushed. Also, it would result it a bunch of updates stalling in
updates-testing as maintainers forget to push things stable. Also, it
would result in maintainers spending a bunch more time looking at
things and seeing... that they are all ready to go to stable.
> the decision to disallow direct stable pushes. The right way to
>
> fix the issue quickly, limiting the damage once done, would have been
> to push libglvnd directly to stable. But Bodhi won't allow that.
> Direct stable pushes are an essential mechanism to fix regressions
> and to limit the number of affected users by minimizing the exposure
> time to the regression.

I disagree again. Right now, mesa in stable updates has a broken dep.
dnf and gnome-software will not update it because of that. So, right
now, no stable updates users are broken, they just see an unsightly
broken dep on trying to update.

If we push libglvnd to stable right now, everyone will be able to
update mesa and it, and all users of sway will get a black screen and
be broken. So, we should _not_ push libglvnd until sway is fixed.

kevin

[Kevin Kofler]

Kevin Fenzi wrote:
> I disagree again. Right now, mesa in stable updates has a broken dep.
> dnf and gnome-software will not update it because of that. So, right
> now, no stable updates users are broken, they just see an unsightly
> broken dep on trying to update.

That is not true. Mesa is composed of multiple subpackages. The updater I 
used (plasma-pk-updates) happily updated mesa-dri-drivers to the new build 
and kept the old builds of mesa-libGL and mesa-libGLES that provide the 
libGL libraries. Though to be fair that does not seem to have actually 
broken anything.

It does create a problem for the people using my patched builds from the 
QtWebEngine Copr (with the Nouveau locking patches that were rejected 
upstream and in Fedora, which make QtWebEngine usable with Nouveau), because 
mesa-dri-drivers gets replaced by the newer version from stock F25 updates 
and I cannot build my patched version due to no libglvnd-devel available.

Kevin Kofler

[Rex Dieter]

Kevin Kofler wrote:
> That is not true. Mesa is composed of multiple subpackages. The updater I
> used (plasma-pk-updates) happily updated mesa-dri-drivers to the new build
> and kept the old builds of mesa-libGL and mesa-libGLES that provide the
> libGL libraries. Though to be fair that does not seem to have actually
> broken anything.

This issue could be fixed in mesa packaging (that would enforce that all 
subpkg components be updated together).  I can help implement that, if there 
is a consensus that would be a "good thing(tm)"

-- rex

[Hans de Goede]

Hi,

On 30-01-17 20:41, Rex Dieter wrote:
> Kevin Kofler wrote:
> > That is not true. Mesa is composed of multiple subpackages. The updater I
> > used (plasma-pk-updates) happily updated mesa-dri-drivers to the new build
> > and kept the old builds of mesa-libGL and mesa-libGLES that provide the
> > libGL libraries. Though to be fair that does not seem to have actually
> > broken anything.
>
> This issue could be fixed in mesa packaging (that would enforce that all
> subpkg components be updated together).  I can help implement that, if there
> is a consensus that would be a "good thing(tm)"

Igor Gnatenko co-maintains mesa now a days, I'm pretty sure he will
happily accept a patch tightening up the inter-pkg dependencies.

Igor, can you confirm (or deny) this please ?

Regards,

Hans

[Igor Gnatenko]

On Tue, 2017-01-31 at 09:42 +0100, Hans de Goede wrote:
> Hi,
>
> On 30-01-17 20:41, Rex Dieter wrote:
> > Kevin Kofler wrote:
> > > That is not true. Mesa is composed of multiple subpackages. The
> > > updater I
> > > used (plasma-pk-updates) happily updated mesa-dri-drivers to the
> > > new build
> > > and kept the old builds of mesa-libGL and mesa-libGLES that
> > > provide the
> > > libGL libraries. Though to be fair that does not seem to have
> > > actually
> > > broken anything.
> >
> > This issue could be fixed in mesa packaging (that would enforce
> > that all
> > subpkg components be updated together).  I can help implement that,
> > if there
> > is a consensus that would be a "good thing(tm)"
>
> Igor Gnatenko co-maintains mesa now a days, I'm pretty sure he will
> happily accept a patch tightening up the inter-pkg dependencies.
>
> Igor, can you confirm (or deny) this please ?

Sure, just send a patch and if it looks good -- I will happily apply
it! :)
> Regards,
>
> Hans

[Jan Pokorný]

On 30/01/17 00:53 +0100, Kevin Kofler wrote:
> This is the result of 4 very poor decisions, by different people/groups:
> the decision to enable libglvnd in an update to a stable release. IMHO,
> such a change is totally unsuitable for a stable release update and
> should have been done in Rawhide only.

Big +1 as you can see in my posts:
https://bodhi.fedoraproject.org/updates/libglvnd-0.2.999-7.gitdc16f8c.fc25#c...
https://bugzilla.redhat.com/show_bug.cgi?id=1413579#c14
and on.

TL;DR:
Another Fedora package (sway and possibly more) gets broken in return.

[Jan Pokorný]

On 30/01/17 14:44 +0100, Jan Pokorný wrote:
> On 30/01/17 00:53 +0100, Kevin Kofler wrote:
> > This is the result of 4 very poor decisions, by different people/groups:
> > the decision to enable libglvnd in an update to a stable release. IMHO,
> > such a change is totally unsuitable for a stable release update and
> > should have been done in Rawhide only.
>
> Big +1 as you can see in my posts:
> https://bodhi.fedoraproject.org/updates/libglvnd-0.2.999-7.gitdc16f8c.fc25#c...
> https://bugzilla.redhat.com/show_bug.cgi?id=1413579#c14
> and on.
>
> TL;DR:
> Another Fedora package (sway and possibly more) gets broken in return.

No longer the case for sway, thanks to Hans' fix of wlc, merged upstream:
https://github.com/Cloudef/wlc/pull/233
and coming to Fedora along the same update.