Fedora 37 mass rebuild complete

[Kevin Fenzi]

Hi all,

Per the Fedora 37 schedule[1] we started a mass rebuild for Fedora
37 on 2022/07/20. We did a mass rebuild for Fedora 37 for:

https://pagure.io/releng/issues?status=Open&tags=mass+rebuild

The mass rebuild was done in a side tag (f37-rebuild) and moved over to
f37. Failures can be seen
https://kojipkgs.fedoraproject.org/mass-rebuild/f37-failures.html Things
still needing rebuilding
https://kojipkgs.fedoraproject.org/mass-rebuild/f37-need-rebuild.html

21713 builds have been tagged into f37, there is currently 1144 failed
builds that need to be addressed by the package maintainers. FTBFS bugs
will be filed shortly. Please be sure to let releng know if you see any
bugs in the reporting. You can contact releng in #fedora-releng on
libera.chat, by dropping an email to our list[2], joining
#releng:fedoraproject.org on Matrix, or filing an issue in pagure[3]

Regards,
Fedora Release Engineering

[1] https://fedorapeople.org/groups/schedule/f-N/f-N-key-tasks.html
[2] https://lists.fedoraproject.org/admin/lists/rel-eng.lists.fedoraproject.org/
[3] https://pagure.io/releng/


_______________________________________________
devel-announce mailing list -- devel-announce(a)lists.fedoraproject.org
To unsubscribe send an email to devel-announce-leave(a)lists.fedoraproject.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/[e-mail]...
Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure

[Richard W.M. Jones]

On Mon, Jul 25, 2022 at 08:57:39AM -0700, Kevin Fenzi wrote:
>  Hi all,
>  
>  Per the Fedora 37 schedule[1] we started a mass rebuild for Fedora
>  37 on 2022/07/20. We did a mass rebuild for Fedora 37 for:
>  
>  https://pagure.io/releng/issues?status=Open&tags=mass+rebuild
>  
>  The mass rebuild was done in a side tag (f37-rebuild) and moved over to
>  f37. Failures can be seen
>  https://kojipkgs.fedoraproject.org/mass-rebuild/f37-failures.html Things
>  still needing rebuilding
>  https://kojipkgs.fedoraproject.org/mass-rebuild/f37-need-rebuild.html 
I could be being stupid here, but what's the difference between
packages that failed to build and packages that need rebuilding?

Rich.

>  21713 builds have been tagged into f37, there is currently 1144
> failed
>  builds that need to be addressed by the package maintainers. FTBFS bugs
>  will be filed shortly. Please be sure to let releng know if you see any
>  bugs in the reporting. You can contact releng in #fedora-releng on
>  libera.chat, by dropping an email to our list[2], joining
>  #releng:fedoraproject.org on Matrix, or filing an issue in pagure[3]
>  
>  Regards,
>  Fedora Release Engineering
>  
>  [1] https://fedorapeople.org/groups/schedule/f-N/f-N-key-tasks.html
>  [2] https://lists.fedoraproject.org/admin/lists/rel-eng.lists.fedoraproject.org/
>  [3] https://pagure.io/releng/
>   


>  _______________________________________________
>  devel-announce mailing list -- devel-announce(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-announce-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]...
>  Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]
>  Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-builder quickly builds VMs from scratch
http://libguestfs.org/virt-builder.1.html

[Miro Hrončok]

On 25. 07. 22 23:38, Richard W.M. Jones wrote:
>  On Mon, Jul 25, 2022 at 08:57:39AM -0700, Kevin Fenzi wrote:
> > Hi all,
> >
> > Per the Fedora 37 schedule[1] we started a mass rebuild for Fedora
> > 37 on 2022/07/20. We did a mass rebuild for Fedora 37 for:
> >
> > https://pagure.io/releng/issues?status=Open&tags=mass+rebuild
> >
> > The mass rebuild was done in a side tag (f37-rebuild) and moved over to
> > f37. Failures can be seen
> > https://kojipkgs.fedoraproject.org/mass-rebuild/f37-failures.html  Things
> > still needing rebuilding
> > https://kojipkgs.fedoraproject.org/mass-rebuild/f37-need-rebuild.html
>  I could be being stupid here, but what's the difference between
>  packages that failed to build and packages that need rebuilding? 
Considering "need-rebuild" is a superset of "failures" (I've just
checked [1]) 
the 2 differences I can think of are:

1) Only builds that manage to get past buildSRPMFromSCM are associated to the 
particular package by Koji. Hence, packages that failed buildSRPMFromSCM or 
when the Koji build was not even submitted (i.e. there was no spec file, 
missing source, RPM parse error, noautobuild file, etc.) -- such package swill 
onyl be listed in "need-rebuild" as there is no known associated build failure 
with them.

2) Immediately after I submitted a newer build of a package that was initially 
listed in "failures", it got removed form that list. Assuming my build would 
fail, it would probbaly only stayed in


Unfortunately, only failures from "failures" are reported in bugzilla [2], but 
I think they should be opened for all packages from "need-rebuild".


[1]
$ comm -23 <(grep -F '<dd><a 
href="http://koji.fedoraproject.org/koji/packageinfo?packageID=' 
f37-need-rebuild.html | sort | uniq) <(grep -F '<dd><a 
href="http://koji.fedoraproject.org/koji/packageinfo?packageID=' 
f37-failures.html | sort | uniq) | wc -l
1392
$ comm -13 ...snip... | wc -l
0

[2] https://pagure.io/releng/issue/8601

-- 
Miro Hrončok
-- 
Phone: [phone]
IRC: mhroncok

[Richard Shaw]

On Mon, Jul 25, 2022 at 4:47 PM Miro Hrončok <mhroncok(a)redhat.com&gt; wrote:

>  On 25. 07. 22 23:38, Richard W.M. Jones wrote:
>  > On Mon, Jul 25, 2022 at 08:57:39AM -0700, Kevin Fenzi wrote:
>  >> Hi all,
>  >>
>  >> Per the Fedora 37 schedule[1] we started a mass rebuild for Fedora
>  >> 37 on 2022/07/20. We did a mass rebuild for Fedora 37 for:
>  >>
>  >> https://pagure.io/releng/issues?status=Open&tags=mass+rebuild
>  >>
>  >> The mass rebuild was done in a side tag (f37-rebuild) and moved over to
>  >> f37. Failures can be seen
>  >> https://kojipkgs.fedoraproject.org/mass-rebuild/f37-failures.html
>  Things
>  >> still needing rebuilding
>  >> https://kojipkgs.fedoraproject.org/mass-rebuild/f37-need-rebuild.html
>  > I could be being stupid here, but what's the difference between
>  > packages that failed to build and packages that need rebuilding?
>
>  Considering "need-rebuild" is a superset of "failures" (I've just
> checked
>  [1])
>  the 2 differences I can think of are:
>
>  1) Only builds that manage to get past buildSRPMFromSCM are associated to
>  the
>  particular package by Koji. Hence, packages that failed buildSRPMFromSCM
>  or
>  when the Koji build was not even submitted (i.e. there was no spec file,
>  missing source, RPM parse error, noautobuild file, etc.) -- such package
>  swill
>  onyl be listed in "need-rebuild" as there is no known associated build
>  failure
>  with them.
>  
Exactly, almost all of the packages for me are not direct failures, but
related to vtk not building (and probably cascades down from there but I
stopped drilling down).

A couple are due to libusb-devel being retired. I'm surprised there weren't
bugs already filed for that. I would think detecting packages using retired
dependencies would be somewhat trivial to discover.

Thanks,
Richard

[Sérgio Basto]

On Mon, 2022-07-25 at 21:55 -0500, Richard Shaw wrote:
>  On Mon, Jul 25, 2022 at 4:47 PM Miro Hrončok
> <mhroncok(a)redhat.com&gt;
>  wrote:
>  > On 25. 07. 22 23:38, Richard W.M. Jones wrote:
>  > > On Mon, Jul 25, 2022 at 08:57:39AM -0700, Kevin Fenzi wrote:
>  > >> Hi all,
>  > >>
>  > >> Per the Fedora 37 schedule[1] we started a mass rebuild for
>  > Fedora
>  > >> 37 on 2022/07/20. We did a mass rebuild for Fedora 37 for:
>  > >>
>  > >> https://pagure.io/releng/issues?status=Open&tags=mass+rebuild
>  > >>
>  > >> The mass rebuild was done in a side tag (f37-rebuild) and moved
>  > over to
>  > >> f37. Failures can be seen
>  > >>
>  > https://kojipkgs.fedoraproject.org/mass-rebuild/f37-failures.html 
>  > Things
>  > >> still needing rebuilding
>  > >>
>  > https://kojipkgs.fedoraproject.org/mass-rebuild/f37-need-rebuild.html
>  > > I could be being stupid here, but what's the difference between
>  > > packages that failed to build and packages that need rebuilding?
>  > 
>  > Considering "need-rebuild" is a superset of "failures" (I've
> just
>  > checked [1]) 
>  > the 2 differences I can think of are:
>  > 
>  > 1) Only builds that manage to get past buildSRPMFromSCM are
>  > associated to the 
>  > particular package by Koji. Hence, packages that failed
>  > buildSRPMFromSCM or 
>  > when the Koji build was not even submitted (i.e. there was no spec
>  > file, 
>  > missing source, RPM parse error, noautobuild file, etc.) -- such
>  > package swill 
>  > onyl be listed in "need-rebuild" as there is no known associated
>  > build failure 
>  > with them.
>  > 
>  
>  
>  Exactly, almost all of the packages for me are not direct failures,
>  but related to vtk not building (and probably cascades down from
>  there but I stopped drilling down). 

I already did one PR to fix the this (vtk) issue  

https://bugzilla.redhat.com/show_bug.cgi?id=2104097


>  A couple are due to libusb-devel being retired. I'm surprised
> there
>  weren't bugs already filed for that. I would think detecting packages
>  using retired dependencies would be somewhat trivial to discover.
>  
>  Thanks,
>  Richard
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct:
>  https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines:
>  https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
>  https://lists.fedoraproject.org/archives/list/[e-mail]
>  Do not reply to spam on the list, report it:
>  https://pagure.io/fedora-infrastructure 
-- 
Sérgio M. B.

[Petr Pisar]

V Mon, Jul 25, 2022 at 11:47:24PM +0200, Miro Hrončok napsal(a):
>  On 25. 07. 22 23:38, Richard W.M. Jones wrote:
>  > On Mon, Jul 25, 2022 at 08:57:39AM -0700, Kevin Fenzi wrote:
>  > > Hi all,
>  > > 
>  > > Per the Fedora 37 schedule[1] we started a mass rebuild for Fedora
>  > > 37 on 2022/07/20. We did a mass rebuild for Fedora 37 for:
>  > > 
>  > > https://pagure.io/releng/issues?status=Open&tags=mass+rebuild
>  > > 
>  > > The mass rebuild was done in a side tag (f37-rebuild) and moved over to
>  > > f37. Failures can be seen
>  > > https://kojipkgs.fedoraproject.org/mass-rebuild/f37-failures.html  Things
>  > > still needing rebuilding
>  > > https://kojipkgs.fedoraproject.org/mass-rebuild/f37-need-rebuild.html
>  > I could be being stupid here, but what's the difference between
>  > packages that failed to build and packages that need rebuilding?
>  
>  Considering "need-rebuild" is a superset of "failures" (I've just
> checked
>  [1]) the 2 differences I can think of are:
>  
>  1) Only builds that manage to get past buildSRPMFromSCM are associated to
>  the particular package by Koji. Hence, packages that failed buildSRPMFromSCM
>  or when the Koji build was not even submitted (i.e. there was no spec file,
>  missing source, RPM parse error, noautobuild file, etc.) -- such package
>  swill onyl be listed in "need-rebuild" as there is no known associated build
>  failure with them.
>   
And then there are the mysterious packages like perl-Math-Cartesian-Product
which even do not have a mass rebuild commit.

I guess because of TCP rejects when pushing/cloning to/from dist-git. It would
be great if relengs retried these network failures next time.

-- Petr

[Sérgio Basto]

On Tue, 2022-07-26 at 10:12 +0200, Petr Pisar wrote:
>  V Mon, Jul 25, 2022 at 11:47:24PM +0200, Miro Hrončok napsal(a):
>  > On 25. 07. 22 23:38, Richard W.M. Jones wrote:
>  > > On Mon, Jul 25, 2022 at 08:57:39AM -0700, Kevin Fenzi wrote:
>  > > > Hi all,
>  > > > 
>  > > > Per the Fedora 37 schedule[1] we started a mass rebuild for
>  > > > Fedora
>  > > > 37 on 2022/07/20. We did a mass rebuild for Fedora 37 for:
>  > > > 
>  > > > https://pagure.io/releng/issues?status=Open&tags=mass+rebuild
>  > > > 
>  > > > The mass rebuild was done in a side tag (f37-rebuild) and moved
>  > > > over to
>  > > > f37. Failures can be seen
>  > > > https://kojipkgs.fedoraproject.org/mass-rebuild/f37-failures.html
>  > > >   Things
>  > > > still needing rebuilding
>  > > > https://kojipkgs.fedoraproject.org/mass-rebuild/f37-need-rebuild.html
>  > > I could be being stupid here, but what's the difference between
>  > > packages that failed to build and packages that need rebuilding?
>  > 
>  > Considering "need-rebuild" is a superset of "failures" (I've
> just
>  > checked
>  > [1]) the 2 differences I can think of are:
>  > 
>  > 1) Only builds that manage to get past buildSRPMFromSCM are
>  > associated to
>  > the particular package by Koji. Hence, packages that failed
>  > buildSRPMFromSCM
>  > or when the Koji build was not even submitted (i.e. there was no
>  > spec file,
>  > missing source, RPM parse error, noautobuild file, etc.) -- such
>  > package
>  > swill onyl be listed in "need-rebuild" as there is no known
>  > associated build
>  > failure with them.
>  > 
>  And then there are the mysterious packages like perl-Math-Cartesian-
>  Product
>  which even do not have a mass rebuild commit.
>  
>  I guess because of TCP rejects when pushing/cloning to/from dist-git.
>  It would
>  be great if relengs retried these network failures next time.
>   
also debootstrap package doesn't have the mass rebuild commit [1] ,
what we should do in these cases ? 


[1]
https://src.fedoraproject.org/rpms/debootstrap/commits/rawhide


>  -- Petr
>  
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct:
>  https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines:
>  https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
>  https://lists.fedoraproject.org/archives/list/[e-mail]
>  Do not reply to spam on the list, report it:
>  https://pagure.io/fedora-infrastructure 
-- 
Sérgio M. B.

[Omair Majid]

Kevin Fenzi <kevin(a)scrye.com&gt; writes:

>  Failures can be seen
>  https://kojipkgs.fedoraproject.org/mass-rebuild/f37-failures.html 
dotnet-build-reference-packages and dotnet3.1 have a cyclic build
dependency on each other. dotnet3.1 was intentionally retired. This
package should have been retired along with it. Done now:

https://src.fedoraproject.org/rpms/dotnet-build-reference-packages/c/f987...

Thanks,
Omair

--
PGP Key: B157A9F0 (http://pgp.mit.edu/)
Fingerprint = 9DB5 2F0B FD3E C239 E108  E7BD DF99 7AF8 B157 A9F0

[Miro Hrončok]

On 25. 07. 22 17:57, Kevin Fenzi wrote:
>  21713 builds have been tagged into f37, there is currently 1144
> failed
>  builds that need to be addressed by the package maintainers. FTBFS bugs
>  will be filed shortly. 
Is there any place we can track the progress for this?

We need to link ~70 bugzillas to https://bugzilla.redhat.com/2107826

Thanks,
-- 
Miro Hrončok
-- 
Phone: [phone]
IRC: mhroncok

[Kevin Fenzi]

On Mon, Aug 01, 2022 at 03:04:50PM +0200, Miro Hrončok wrote:
>  On 25. 07. 22 17:57, Kevin Fenzi wrote:
>  > 21713 builds have been tagged into f37, there is currently 1144 failed
>  > builds that need to be addressed by the package maintainers. FTBFS bugs
>  > will be filed shortly.
>  
>  Is there any place we can track the progress for this?
>  
>  We need to link ~70 bugzillas to https://bugzilla.redhat.com/2107826 
Not really. I guess I'll go see about doing it today. 

kevin

[Kevin Fenzi]

On Mon, Aug 01, 2022 at 03:43:39PM -0700, Kevin Fenzi wrote:
>  On Mon, Aug 01, 2022 at 03:04:50PM +0200, Miro Hrončok wrote:
>  > On 25. 07. 22 17:57, Kevin Fenzi wrote:
>  > > 21713 builds have been tagged into f37, there is currently 1144 failed
>  > > builds that need to be addressed by the package maintainers. FTBFS bugs
>  > > will be filed shortly.
>  > 
>  > Is there any place we can track the progress for this?
>  > 
>  > We need to link ~70 bugzillas to https://bugzilla.redhat.com/2107826
>  
>  Not really. I guess I'll go see about doing it today.  
They should now be filed. 

This is just ones that failed to build. If someone wants to take a stab
and improving the script so we can also file bugs on the ones that
didn't make a valid src.rpm, that would be great. 

kevin

[Miro Hrončok]

On 02. 08. 22 2:23, Kevin Fenzi wrote:
>  On Mon, Aug 01, 2022 at 03:43:39PM -0700, Kevin Fenzi wrote:
> > On Mon, Aug 01, 2022 at 03:04:50PM +0200, Miro Hrončok wrote:
> >> On 25. 07. 22 17:57, Kevin Fenzi wrote:
> >>> 21713 builds have been tagged into f37, there is currently 1144 failed
> >>> builds that need to be addressed by the package maintainers. FTBFS bugs
> >>> will be filed shortly.
> >>
> >> Is there any place we can track the progress for this?
> >>
> >> We need to link ~70 bugzillas to https://bugzilla.redhat.com/2107826
> >
> > Not really. I guess I'll go see about doing it today.
>  
>  They should now be filed. 
Thank you.

The following packages are listed at 
https://kojipkgs.fedoraproject.org/mass-rebuild/f37-failures.html but we see no 
F37FTBFS bugzilla:

fedmsg
lldb
pungi
pybind11
python-fasjson-client
python-nose2
python-pikepdf
python-psutil
python-pycdlib
python-pynwb
restview

>  This is just ones that failed to build. If someone wants to take a
> stab
>  and improving the script so we can also file bugs on the ones that
>  didn't make a valid src.rpm, that would be great. 
Could you please share a link to the existing script?

-- 
Miro Hrončok
-- 
Phone: [phone]
IRC: mhroncok

[Miro Hrončok]

On 02. 08. 22 10:26, Miro Hrončok wrote:
>  On 02. 08. 22 2:23, Kevin Fenzi wrote:
> > On Mon, Aug 01, 2022 at 03:43:39PM -0700, Kevin Fenzi wrote:
> >> On Mon, Aug 01, 2022 at 03:04:50PM +0200, Miro Hrončok wrote:
> >>> On 25. 07. 22 17:57, Kevin Fenzi wrote:
> >>>> 21713 builds have been tagged into f37, there is currently 1144 failed
> >>>> builds that need to be addressed by the package maintainers. FTBFS bugs
> >>>> will be filed shortly.
> >>>
> >>> Is there any place we can track the progress for this?
> >>>
> >>> We need to link ~70 bugzillas to https://bugzilla.redhat.com/2107826
> >>
> >> Not really. I guess I'll go see about doing it today.
> >
> > They should now be filed.
>  
>  Thank you.
>  
>  The following packages are listed at 
>  https://kojipkgs.fedoraproject.org/mass-rebuild/f37-failures.html but we see no 
>  F37FTBFS bugzilla:
>  
>  fedmsg
>  lldb
>  pungi 
This one actually has the bugzilla, sorry about that.

>  pybind11
>  python-fasjson-client
>  python-nose2
>  python-pikepdf
>  python-psutil
>  python-pycdlib
>  python-pynwb 
As well as this one.

>  restview
>  
> > This is just ones that failed to build. If someone wants to take a stab
> > and improving the script so we can also file bugs on the ones that
> > didn't make a valid src.rpm, that would be great.
>  
>  Could you please share a link to the existing script?
>   
-- 
Miro Hrončok
-- 
Phone: [phone]
IRC: mhroncok

[Kevin Fenzi]

On Tue, Aug 02, 2022 at 01:32:07PM +0200, Miro Hrončok wrote:
>  > 
>  > Could you please share a link to the existing script?
>  >  
https://pagure.io/releng/blob/main/f/scripts/mass_rebuild_file_bugs.py

I wonder if it's hitting a limit from bugzilla now, or perhaps a paging
issue? It definitely seems like it missed some. 

kevin

[Miro Hrončok]

On 02. 08. 22 18:31, Kevin Fenzi wrote:
>  On Tue, Aug 02, 2022 at 01:32:07PM +0200, Miro Hrončok wrote:
> >>
> >> Could you please share a link to the existing script?
> >>
>  
>  https://pagure.io/releng/blob/main/f/scripts/mass_rebuild_file_bugs.py
>  
>  I wonder if it's hitting a limit from bugzilla now, or perhaps a paging
>  issue? It definitely seems like it missed some. 
The pagination code seems OK to me.

When I comment out the releng user stuff and the actual report_failure(...) 
calls, I see the sripts reports that the bugs were already opened for the 
missing packages.

When I add a print here and there, I see it considers closed F37FTBFS bugzillas 
as already reported. That might be tru for the stuff that was closed after the 
script run, but not for bugzillas closed before the mass rebuild.

I can prepare a fix.



-- 
Miro Hrončok
-- 
Phone: [phone]
IRC: mhroncok

[Miro Hrončok]

On 02. 08. 22 18:55, Miro Hrončok wrote:
>  On 02. 08. 22 18:31, Kevin Fenzi wrote:
> > On Tue, Aug 02, 2022 at 01:32:07PM +0200, Miro Hrončok wrote:
> >>>
> >>> Could you please share a link to the existing script?
> >>>
> >
> > https://pagure.io/releng/blob/main/f/scripts/mass_rebuild_file_bugs.py
> >
> > I wonder if it's hitting a limit from bugzilla now, or perhaps a paging
> > issue? It definitely seems like it missed some.
>  
>  The pagination code seems OK to me.
>  
>  When I comment out the releng user stuff and the actual report_failure(...) 
>  calls, I see the sripts reports that the bugs were already opened for the 
>  missing packages.
>  
>  When I add a print here and there, I see it considers closed F37FTBFS bugzillas 
>  as already reported. That might be tru for the stuff that was closed after the 
>  script run, but not for bugzillas closed before the mass rebuild.
>  
>  I can prepare a fix. 
This should do: https://pagure.io/releng/pull-request/10944

-- 
Miro Hrončok
-- 
Phone: [phone]
IRC: mhroncok