Qt update and packages rebuild

[Germano Massullo]

Recent CentOS Stream Qt update broke some EPEL packages like keepassxc 
that needed a rebuild against the new Qt version.
Can we talk about a way to prevent this from happening again?

Best regards

[1]: https://bugzilla.redhat.com/show_bug.cgi?id=2077742

[Stephen John Smoogen]

On Fri, 29 Apr 2022 at 14:28, Germano Massullo 
wrote:
> Recent CentOS Stream Qt update broke some EPEL packages like keepassxc
> that needed a rebuild against the new Qt version.
> Can we talk about a way to prevent this from happening again?

This is the current situation of events for dealing with CentOS Stream and
EPEL
0. Packages get put into stream at the rate of internal developers doing
things and getting stuff put into GIT. There is no communication to know
when this will happen so knowing what packages to build before this drops
isn't happening.
1. The QT packages in Stream have taken a week to be fixed due to various
issues found in them. [Mostly they were built in the wrong order and linked
against each other poorly.]
2. This means rebuilding packages have to wait until that is fixed as some
people found when they jumped on it sooner. Either they could not rebuild
anything or when they rebuilt it they needed to do it again when the
updated packages with the right library links came out.
3. Packages in EPEL are maintained by a lot of people who may not know that
centos-stream have updated rapidly and do not have the spare capacity to
update sooner than the weekend spare time they had allotted to do it.

What are ways that this could be improved?
> Best regards
>
> epel-devel mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]...
> Do not reply to spam on the list, report it:
> https://pagure.io/fedora-infrastructure

[Alex Iribarren]

Hi,

On 4/29/22 21:17, Stephen John Smoogen wrote:
> On Fri, 29 Apr 2022 at 14:28, Germano Massullo 
> <[e-mail] > wrote:
> Recent CentOS Stream Qt update broke some EPEL packages like keepassxc
> that needed a rebuild against the new Qt version.
> Can we talk about a way to prevent this from happening again?
>
>
> This is the current situation of events for dealing with CentOS Stream 
> and EPEL
> 0. Packages get put into stream at the rate of internal developers doing 
> things and getting stuff put into GIT. There is no communication to know 
> when this will happen so knowing what packages to build before this 
> drops isn't happening.
> The QT packages in Stream have taken a week to be fixed due to
>
> various issues found in them. [Mostly they were built in the wrong order 
> and linked against each other poorly.]

So how could we stop this from happening in the future? The 
50_test_comps[0] test caught some of the problems, but not all because 
not all qt5 packages seem to be listed in a comps group. `dnf install 
qt5-*` is unlikely to work, though I haven't tried.

How could we test that all qt5 packages have been correctly rebuilt? If 
that's done right, the following steps will be a lot less painful.

Cheers,
Alex
> This means rebuilding packages have to wait until that is fixed as
>
> some people found when they jumped on it sooner. Either they could not 
> rebuild anything or when they rebuilt it they needed to do it again when 
> the updated packages with the right library links came out.
> 3. Packages in EPEL are maintained by a lot of people who may not know 
> that centos-stream have updated rapidly and do not have the spare 
> capacity to update sooner than the weekend spare time they had allotted 
> to do it.
>
> What are ways that this could be improved?
>
> -- 
> Stephen J Smoogen.
> Let us be kind to one another, for most of us are fighting a hard 
> battle. -- Ian MacClaren

[0] 
https://github.com/CentOS/sig-core-t_functional/blob/master/tests/0_common/5...

[Stephen John Smoogen]

On Mon, 2 May 2022 at 04:22, Alex Iribarren  wrote:
> Hi,
>
> On 4/29/22 21:17, Stephen John Smoogen wrote:
> > On Fri, 29 Apr 2022 at 14:28, Germano Massullo
> > <[e-mail] > wrote:
> > Recent CentOS Stream Qt update broke some EPEL packages like
>
> keepassxc
> > that needed a rebuild against the new Qt version.
> > Can we talk about a way to prevent this from happening again?
> >
> >
> > This is the current situation of events for dealing with CentOS Stream
> > and EPEL
> > 0. Packages get put into stream at the rate of internal developers doing
> > things and getting stuff put into GIT. There is no communication to know
> > when this will happen so knowing what packages to build before this
> > drops isn't happening.
> > The QT packages in Stream have taken a week to be fixed due to
> >
> > various issues found in them. [Mostly they were built in the wrong order
> > and linked against each other poorly.]
>
> So how could we stop this from happening in the future? The

We can't stop it from happening in the future. Pretty much every time I
have seen it is because something changed in the 'way things have been done
previously' and so it looked like a new problem with the same similarities.
I think we can offer solutions to make this better and possibly help
implement prototypes to show that they work or not.
> 50_test_comps[0] test caught some of the problems, but not all because
> not all qt5 packages seem to be listed in a comps group. `dnf install
> qt5-*` is unlikely to work, though I haven't tried.
>
> How could we test that all qt5 packages have been correctly rebuilt? If
> that's done right, the following steps will be a lot less painful.

I think the primary issue is that the crafting of binary packages is
'fairly' manual. Someone has to put the src.rpm in the meat grinder (koji)
in the right order with the right spices (flags) to make the sausage at the
other end. We rely on the cook to remember how they did it the last 10
times and that the taster (functional and ci) says it works. This normally
works well but then it turns out that something swapped out somewhere and
once 'fully cooked' (composed) that the sausage explodes.

I think we would need to study how the build system really works, and how
the recipe of 'order of builds' is determined. From that we can then
suggest improvements which the CentOS Release Engineering can implement. It
could be that coming up with some tool which makes the order of builds more
automated may help, but without knowing how the workplace actually runs.. I
am just making suggestions from the restaurant table :).
> Cheers,
> Alex
> > This means rebuilding packages have to wait until that is fixed as
> >
> > some people found when they jumped on it sooner. Either they could not
> > rebuild anything or when they rebuilt it they needed to do it again when
> > the updated packages with the right library links came out.
> > 3. Packages in EPEL are maintained by a lot of people who may not know
> > that centos-stream have updated rapidly and do not have the spare
> > capacity to update sooner than the weekend spare time they had allotted
> > to do it.
> >
> > What are ways that this could be improved?
> >
> > --
> > Stephen J Smoogen.
> > Let us be kind to one another, for most of us are fighting a hard
> > battle. -- Ian MacClaren
>
> [0]
>
> https://github.com/CentOS/sig-core-t_functional/blob/master/tests/0_common/5...
> _______________________________________________
> epel-devel mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]...
> Do not reply to spam on the list, report it:
> https://pagure.io/fedora-infrastructure

[CentOS-devel] Re: Qt update and packages rebuild

[Troy Dawson]

On Mon, May 2, 2022 at 1:22 AM Alex Iribarren 
wrote:
> Hi,
>
> On 4/29/22 21:17, Stephen John Smoogen wrote:
> > On Fri, 29 Apr 2022 at 14:28, Germano Massullo
> > <[e-mail] > wrote:
> > Recent CentOS Stream Qt update broke some EPEL packages like
>
> keepassxc
> > that needed a rebuild against the new Qt version.
> > Can we talk about a way to prevent this from happening again?
> >
> >
> > This is the current situation of events for dealing with CentOS Stream
> > and EPEL
> > 0. Packages get put into stream at the rate of internal developers doing
> > things and getting stuff put into GIT. There is no communication to know
> > when this will happen so knowing what packages to build before this
> > drops isn't happening.
> > The QT packages in Stream have taken a week to be fixed due to
> >
> > various issues found in them. [Mostly they were built in the wrong order
> > and linked against each other poorly.]
>
> So how could we stop this from happening in the future? The
> 50_test_comps[0] test caught some of the problems, but not all because
> not all qt5 packages seem to be listed in a comps group. `dnf install
> qt5-*` is unlikely to work, though I haven't tried.
>
> How could we test that all qt5 packages have been correctly rebuilt? If
> that's done right, the following steps will be a lot less painful.
>
> Cheers,
> Alex

When CentOS Stream 8 flips from "after" rhel8 to "before" rhel8, that
should fix this problem.

Right now, the CentOS Stream team just get a list of packages.  No order at
all.
They try their best, but sometimes things get built out of order.

Although they've fixed all the out of order problems, there is currently a
python27 module problem in CentOS Stream 8.
It's been broken since February, but nobody noticed due to a bug/feature in
dnf.
The EPEL build system notices because it pulls in ONLY the latest module.
Thus anyone trying to build with python2 (qt5-qtwebengine) get's failure.
> > This means rebuilding packages have to wait until that is fixed as
> >
> > some people found when they jumped on it sooner. Either they could not
> > rebuild anything or when they rebuilt it they needed to do it again when
> > the updated packages with the right library links came out.
> > 3. Packages in EPEL are maintained by a lot of people who may not know
> > that centos-stream have updated rapidly and do not have the spare
> > capacity to update sooner than the weekend spare time they had allotted
> > to do it.
> >
> > What are ways that this could be improved?
> >
> > --
> > Stephen J Smoogen.
> > Let us be kind to one another, for most of us are fighting a hard
> > battle. -- Ian MacClaren
>
> [0]
>
> https://github.com/CentOS/sig-core-t_functional/blob/master/tests/0_common/5...
> _______________________________________________
> CentOS-devel mailing list
> [e-mail]
> https://lists.centos.org/mailman/listinfo/centos-devel

[Troy Dawson]

On Fri, Apr 29, 2022 at 11:28 AM Germano Massullo <
[e-mail]> wrote:
> Recent CentOS Stream Qt update broke some EPEL packages like keepassxc
> that needed a rebuild against the new Qt version.
> Can we talk about a way to prevent this from happening again?
>
> Best regards

The qt5 version went from 5.15.2 to 5.15.3 in CentOS Stream, thus packages
that depend on specific version of qt5 need to be rebuilt, in just
epel8-next.
That means that in RHEL 8.7 those same versions are going to change and
need a rebuild again.

What do you suggest would be a good way to prevent this in the future.

Troy

[Troy Dawson]

On Fri, Apr 29, 2022 at 12:19 PM Troy Dawson  wrote:
> On Fri, Apr 29, 2022 at 11:28 AM Germano Massullo <
> [e-mail]> wrote:
> > Recent CentOS Stream Qt update broke some EPEL packages like keepassxc
> > that needed a rebuild against the new Qt version.
> > Can we talk about a way to prevent this from happening again?
> >
> > Best regards
>
> The qt5 version went from 5.15.2 to 5.15.3 in CentOS Stream, thus packages
> that depend on specific version of qt5 need to be rebuilt, in just
> epel8-next.
> That means that in RHEL 8.7 those same versions are going to change and
> need a rebuild again.
>
> What do you suggest would be a good way to prevent this in the future.
>
> Troy

So, my mind isn't letting this go.
There are going to be updates in RHEL and CentOS Stream, that is just the
nature of software and operating systems.

But what if we got an early warning system?  Would it be worth it?

The RHEL 9 workflow (and hopefully RHEL 8 sometime reasonably soon) starts
with a build on CentOS Stream 9.
That build is tagged with c9s-gate, and then it goes through a variety of
tests.
If everything goes well it get's tagged into c9s-pending.
At that point it goes into the CentOS Stream 9 development compose (once a
day).
That development compose goes through more tests and if everything looks
good a production compose is made.
Again, more tests, then signing and it get's pushed to the mirrors and all
the other places.

Anyway, all the CentOS STream 9 (hopefully 8 soon) is publically visible.
It would be possible for someone to create a type of early warning system
that let's maintainers know that a change is coming down the road and they
will need a rebuild or something.

And ... now I just realized that this sounds alot like taking my
Will-It-Install and pointing it at the CentOS Stream 9 development compose.

But if someone wants to design their own things that does this, that would
be cool.

Troy

[Miro Hrončok]

On 29. 04. 22 22:01, Troy Dawson wrote:
> And ... now I just realized that this sounds alot like taking my 
> Will-It-Install and pointing it at the CentOS Stream 9 development compose.

I am afraid that we would need to point it to a repository that contains all 
the builds immediately as they happen, before they pass gating. And I am not 
sure such repository exists. Is there a Koji repo from the c9s-gate tag?

[Germano Massullo]

any update on this matter? A few days ago I got another bugreport 
concerning CentOS Stream Qt update breaking an application

[Troy Dawson]

There hasn't been a qt update in CentOS Stream 8 or 9 since May 2022, so
you'll have to give more information.

[Germano Massullo]

> There hasn't been a qt update in CentOS Stream 8 or 9 since May 2022, 
> so you'll have to give more information.

I double checked and I found out the origin of my misunderstanding. I 
wrote everything in this comment
https://bugzilla.redhat.com/show_bug.cgi?id=2129662#c8

[Ian Laurie]

On 10/4/22 08:16, Germano Massullo wrote:
> > There hasn't been a qt update in CentOS Stream 8 or 9 since May 2022, 
> > so you'll have to give more information.
>
> I double checked and I found out the origin of my misunderstanding. I 
> wrote everything in this comment
> https://bugzilla.redhat.com/show_bug.cgi?id=2129662#c8

I guess the issue here is that if a package is in EPEL9, but not in 
EPEL9-NEXT, and it can install, then "sudo dnf install keepassxc" is 
going to work just fine, and back when I did it, it did.

As a an unrelated issue, running "sudo dnf upgrade" from the command 
line, as I like to do, in GNOME results in gnome-shell crashing quite 
often, leaving updates in an indeterminate state and a possibly a 
corrupted database, so one is forced to use gnome-shell when in GNOME. 
However gnome-shell doesn't inform you of an updates conflict, so my 
problem may have dated back to May for all I know.

I became aware of the QT conflict when I used the command line to do 
updates from a virtual console, but the issue could have been there from 
any time in the past.

[Germano Massullo]

As I wrote in
https://bugzilla.redhat.com/show_bug.cgi?id=2129662#c11
I did not know that epel packages are used also in CentOS Stream, 
sometimes causing issues like the one previously mentioned.

In fact this behaviour obliges epel maintainers that wanted to maintain 
RHEL only, to work to maintain CentOS Stream too.
This an additional source of troubles for package maintainers

[Petr Pisar]

V Tue, Oct 04, 2022 at 10:17:34AM +0200, Germano Massullo napsal(a):
> As I wrote in
> https://bugzilla.redhat.com/show_bug.cgi?id=2129662#c11
> I did not know that epel packages are used also in CentOS Stream, sometimes
> causing issues like the one previously mentioned.
>
> In fact this behaviour obliges epel maintainers that wanted to maintain RHEL
> only, to work to maintain CentOS Stream too.
> This an additional source of troubles for package maintainers

I guess this strategy was selected
https://lists.fedoraproject.org/archives/list/[e-mail]/thread/NH4CM6MAVUTUH35NDM53PTKCHODSEP6F/
because most EPEL packages keep working in CentOS Stream. Hence EPEL-next
delivers only the few execptions which need a rebuild/patch.

Originally when RHEL 9 did not exist, EPEL-next was a completly separate
repository, but EPEL maintainers were forgetting to build (and debug) for
EPEL-next, hence EPEL-next became completely useless. Thus EPEL-next was
repurposed as an overlay for EPEL.

While EPEL maintainers might see EPEL-next as a nuisance, they should realize
that what's CentOS Stream now, will become RHEL in 6 months. If their EPEL
package breaks in CentOS Stream now, it will become broken in RHEL in
6 months. Ignoring CentOS Stream only postpones an inevitable maintenance
work. I think EPEL maintainers should rather perceive EPEL-next the same way
as Fedora maintainers handle Rawhide.

The only problem is that there is no automatic move of EPEL-next packages to
EPEL after the 6 months. A maintainer needs to repeat the work in EPEL and, if
possible, to remove the unnecessary EPEL-next build manually
https://lists.fedoraproject.org/archives/list/[e-mail]/thread/QMVRM4MQM4DHWUEUN65YYNU7UJADAT6Z/#MU4LXGKDUQCKKFET22J36GKLY4UFKIT2.

-- Petr

[Troy Dawson]

On Fri, Apr 29, 2022 at 11:28 AM Germano Massullo <
[e-mail]> wrote:
> Recent CentOS Stream Qt update broke some EPEL packages like keepassxc
> that needed a rebuild against the new Qt version.
> Can we talk about a way to prevent this from happening again?
>
> Best regards

Looking at  keepassxc specifically, I'd say take the following line out of
the spec file.
  %{?_qt5:Requires: %{_qt5}%{?_isa} = %{_qt5_version}}

If you let rpmbuild figure out the dependencies (which it is already doing
fairly well), then it will know when there is a real ABI change and you
need to rebuild the package.
Looking at
  "dnf repoquery --requires keepassxc | grep -i qt"
and then looking at the changes that happened in qt5, it looks like it
wouldn't have needed a rebuild if the Requires wasn't manually set.

I'm not saying rpmbuild is 100% perfect for figuring out requires, but in
this case it was doing it right.

Troy

[Neal Gompa]

On Fri, Apr 29, 2022 at 3:42 PM Troy Dawson  wrote:
> On Fri, Apr 29, 2022 at 11:28 AM Germano Massullo  wrote:
> > Recent CentOS Stream Qt update broke some EPEL packages like keepassxc
> > that needed a rebuild against the new Qt version.
> > Can we talk about a way to prevent this from happening again?
> >
> > Best regards
>
> Looking at  keepassxc specifically, I'd say take the following line out of the spec file.
>   %{?_qt5:Requires: %{_qt5}%{?_isa} = %{_qt5_version}}

This macro is only supposed to be used if the program uses private Qt5
APIs. Unfortunately, according to BRs, it does, so this macro makes
sense to keep.

But someone should talk to upstream and ask why it's using private APIs...

--
真実はいつも一つ！/ Always, there's only one truth!

[Germano Massullo]

Il 29/04/22 21:42, Troy Dawson ha scritto:
> On Fri, Apr 29, 2022 at 11:28 AM Germano Massullo 
>  wrote:
> Recent CentOS Stream Qt update broke some EPEL packages like
> keepassxc
> that needed a rebuild against the new Qt version.
> Can we talk about a way to prevent this from happening again?
>
> Best regards
>
>
> Looking at  keepassxc specifically, I'd say take the following line 
> out of the spec file.
>   %{?_qt5:Requires: %{_qt5}%{?_isa} = %{_qt5_version}}
>
> If you let rpmbuild figure out the dependencies (which it is already 
> doing fairly well), then it will know when there is a real ABI change 
> and you need to rebuild the package.
> Looking at
>   "dnf repoquery --requires keepassxc | grep -i qt"
> and then looking at the changes that happened in qt5, it looks like it 
> wouldn't have needed a rebuild if the Requires wasn't manually set.
>
> I'm not saying rpmbuild is 100% perfect for figuring out requires, but 
> in this case it was doing it right.

That macro is needed to avoid situations where users somehow manage to 
install keepassxc on a system with old Qt version like in following 
bugreport
https://bugzilla.redhat.com/show_bug.cgi?id=2068981