Fedora 34 Mass Rebuild

[Mohan Boddu]

Hi all,

Per the Fedora 34 schedule[1] we will start a mass rebuild for Fedora 34
on Jan 20th 2021. We will run a mass rebuild for Fedora 34 for the
changes listed in:

https://pagure.io/releng/issues?status=Open&tags=mass+rebuild

Mass rebuild will be done in a side tag (f34-rebuild) and moved over
when completed.

Failures can be seen
https://kojipkgs.fedoraproject.org/mass-rebuild/f34-failures.html

Things still needing rebuilt
https://kojipkgs.fedoraproject.org/mass-rebuild/f34-need-rebuild.html

FTBFS bugs will be filed shortly.

Please be sure to let releng know if you see any bugs in the
reporting. You can contact releng in #fedora-releng on freenode, by
dropping an email to our list[2] or filing an issue in pagure[3]

Regards,
Mohan Boddu.

[1] https://fedorapeople.org/groups/schedule/f-34/f-34-key-tasks.html
[2] https://lists.fedoraproject.org/admin/lists/rel-eng.lists.fedoraproject.org/
[3] https://pagure.io/releng/
_______________________________________________
devel-announce mailing list -- devel-announce(a)lists.fedoraproject.org
To unsubscribe send an email to devel-announce-leave(a)lists.fedoraproject.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/[e-mail]...

[Fabio Valentini]

On Fri, Jan 15, 2021 at 8:35 PM Mohan Boddu <mboddu(a)bhujji.com&gt; wrote:
>  Hi all,
>
>  Per the Fedora 34 schedule[1] we will start a mass rebuild for Fedora 34
>  on Jan 20th 2021. We will run a mass rebuild for Fedora 34 for the
>  changes listed in:
>
>  https://pagure.io/releng/issues?status=Open&tags=mass+rebuild
>
>  Mass rebuild will be done in a side tag (f34-rebuild) and moved over
>  when completed.
>
>  Failures can be seen
>  https://kojipkgs.fedoraproject.org/mass-rebuild/f34-failures.html
>
>  Things still needing rebuilt
>  https://kojipkgs.fedoraproject.org/mass-rebuild/f34-need-rebuild.html
>
>  FTBFS bugs will be filed shortly.
>
>  Please be sure to let releng know if you see any bugs in the
>  reporting. You can contact releng in #fedora-releng on freenode, by
>  dropping an email to our list[2] or filing an issue in pagure[3]
>
>  Regards,
>  Mohan Boddu. 
What system will be used to run the mass rebuild script?
If it's a fedora 33 system, I suggest modifying the script to invoke
rpmdev-bumpspec with the "-D" flag for better compatibiltiy with older
RPM versions (for people who care about that sort of thing, e.g.
maintainers who merge specs back into epel branches).

Fabio

[Kevin Fenzi]

On Sat, Jan 16, 2021 at 12:43:56AM +0100, Fabio Valentini wrote:
>  
>  What system will be used to run the mass rebuild script? 
They run on compose-x86-01 I think. 

>  If it's a fedora 33 system, I suggest modifying the script to
> invoke
>  rpmdev-bumpspec with the "-D" flag for better compatibiltiy with older
>  RPM versions (for people who care about that sort of thing, e.g.
>  maintainers who merge specs back into epel branches). 
It's currently f32, but we were going to upgrade it. 
We could do that and add this in I think...

Thanks for the suggestion!

kevin

[Mohan Boddu]

On Sat, Jan 16, 2021 at 2:44 PM Kevin Fenzi <kevin(a)scrye.com&gt; wrote:
>  On Sat, Jan 16, 2021 at 12:43:56AM +0100, Fabio Valentini wrote:
>  >
>  > What system will be used to run the mass rebuild script?
>
>  They run on compose-x86-01 I think. 
We run mass rebuild on compose-branched01.iad2.fp.o which is updated to F33.

>  > If it's a fedora 33 system, I suggest modifying the script to invoke
>  > rpmdev-bumpspec with the "-D" flag for better compatibiltiy with older
>  > RPM versions (for people who care about that sort of thing, e.g.
>  > maintainers who merge specs back into epel branches). 
I updated the mass-rebuild.py script to use '-D' flag
https://pagure.io/releng/pull-request/9951

>
> It's currently f32, but we were going to upgrade it.
> We could do that and add this in I think...
>
> Thanks for the suggestion!
>
> kevin
> _______________________________________________
> devel mailing list -- devel(a)lists.fedoraproject.org
> To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
> Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
https://lists.fedoraproject.org/archives/list/[e-mail]

[Miro Hrončok]

On 15. 01. 21 20:31, Mohan Boddu wrote:
>  Hi all,
>  
>  Per the Fedora 34 schedule[1] we will start a mass rebuild for Fedora 34
>  on Jan 20th 2021. We will run a mass rebuild for Fedora 34 for the
>  changes listed in:
>  
>  https://pagure.io/releng/issues?status=Open&tags=mass+rebuild 
I've noticed https://pagure.io/releng/issue/9829 is not listed here.
Could you please not start the build until make is removed?

Also, https://pagure.io/releng/issue/9910 was rejected.

-- 
Miro Hrončok
--
Phone: [phone]
IRC: mhroncok

[Kevin Fenzi]

On Sat, Jan 16, 2021 at 08:39:17AM +0100, Miro Hrončok wrote:
>  On 15. 01. 21 20:31, Mohan Boddu wrote:
>  > Hi all,
>  > 
>  > Per the Fedora 34 schedule[1] we will start a mass rebuild for Fedora 34
>  > on Jan 20th 2021. We will run a mass rebuild for Fedora 34 for the
>  > changes listed in:
>  > 
>  > https://pagure.io/releng/issues?status=Open&tags=mass+rebuild
>  
>  I've noticed https://pagure.io/releng/issue/9829 is not listed here.
>  Could you please not start the build until make is removed? 
Yes, as mentioned in that ticket we are planning on removing make from
the mass rebuild side tag before starting it, and then remove it from
the main buildroot after the mass rebuild if all looks good. 

>  Also, https://pagure.io/releng/issue/9910 was rejected. 
Right, thanks for the reminder. 

kevin

[Miro Hrončok]

On 16. 01. 21 20:46, Kevin Fenzi wrote:
> >>
> https://pagure.io/releng/issues?status=Open&tags=mass+rebuild
> > I've noticedhttps://pagure.io/releng/issue/9829  is not listed here.
> > Could you please not start the build until make is removed?
>  Yes, as mentioned in that ticket we are planning on removing make from
>  the mass rebuild side tag before starting it, and then remove it from
>  the main buildroot after the mass rebuild if all looks good. 
I've only pointed it out because the issue is not tagged with mass-rebuild, so I 
was afraid it might be forgotten. Could somebody with permissiond please tag it?

-- 
Miro Hrončok
--
Phone: [phone]
IRC: mhroncok

[Daniel Pocock]

On 15/01/2021 20:31, Mohan Boddu wrote:
>  Hi all,
>  
>  Per the Fedora 34 schedule[1] we will start a mass rebuild for Fedora 34
>  on Jan 20th 2021. We will run a mass rebuild for Fedora 34 for the
>  changes listed in:
>  
>  https://pagure.io/releng/issues?status=Open&tags=mass+rebuild 

The ppc64le 4k page size is listed there, do you need any pull request
from me to adapt the ppc64le kernel config or anything else or it is
already in hand?

Feel free to reply here or through the Pagure issue

[Kevin Fenzi]

On Sun, Jan 17, 2021 at 01:19:34AM +0100, Daniel Pocock wrote:
>  
>  
>  On 15/01/2021 20:31, Mohan Boddu wrote:
>  > Hi all,
>  > 
>  > Per the Fedora 34 schedule[1] we will start a mass rebuild for Fedora 34
>  > on Jan 20th 2021. We will run a mass rebuild for Fedora 34 for the
>  > changes listed in:
>  > 
>  > https://pagure.io/releng/issues?status=Open&tags=mass+rebuild
>  
>  
>  The ppc64le 4k page size is listed there, do you need any pull request
>  from me to adapt the ppc64le kernel config or anything else or it is
>  already in hand?
>  
>  Feel free to reply here or through the Pagure issue 
Well, given that it's not yet discussed and approved by fesco, and that
it needs builders to be running the changed kernel, I do now think this
is likely to get in this mass rebuild. 

We need a f33 kernel for the builders, and time to update them to it
before the mass rebuild and shake out any issues. 

So I think this should be retargeted for f35.

kevin

[Daniel Pocock]

On 17/01/2021 23:10, Kevin Fenzi wrote:
>  On Sun, Jan 17, 2021 at 01:19:34AM +0100, Daniel Pocock wrote:
> >
> >
> > On 15/01/2021 20:31, Mohan Boddu wrote:
> >> Hi all,
> >>
> >> Per the Fedora 34 schedule[1] we will start a mass rebuild for Fedora 34
> >> on Jan 20th 2021. We will run a mass rebuild for Fedora 34 for the
> >> changes listed in:
> >>
> >> https://pagure.io/releng/issues?status=Open&tags=mass+rebuild
> >
> >
> > The ppc64le 4k page size is listed there, do you need any pull request
> > from me to adapt the ppc64le kernel config or anything else or it is
> > already in hand?
> >
> > Feel free to reply here or through the Pagure issue
>  
>  Well, given that it's not yet discussed and approved by fesco, and that
>  it needs builders to be running the changed kernel, I do now think this
>  is likely to get in this mass rebuild. 
>  
>  We need a f33 kernel for the builders, and time to update them to it
>  before the mass rebuild and shake out any issues. 
>  
>  So I think this should be retargeted for f35. 
Thanks for the feedback.  I understand that is the normal way to proceed
at this point in the schedule.  It doesn't really impact me a lot
personally as I build my own kernels anyway.

Given there are not so many users on this architecture, is it feasible
to take a more do-or-die approach, to try and put that f33 kernel in
there and if it doesn't come together, f34 is simply released without
any ppc64le?

The upside is that if it does work, all those new GPUs will be working,
plug and play.  I'm talking about all the new RX 6800 cards and also the
previous generation, RX 5700 and friends.

If f34 is status quo, 64k kernel, people have to either:

a) use a GPU that is much older now, like Radeon RX580 Polaris, or

b) compile a kernel with 4k, or

c) download a kernel with 4k from somebody

I don't mind too much whether it is f34 or f35 and I understand you
probably have higher priority things to do at this point in the release
cycle.

Regards,

Daniel

[Kevin Fenzi]

On Sun, Jan 17, 2021 at 11:31:32PM +0100, Daniel Pocock wrote:
>  
>  Thanks for the feedback.  I understand that is the normal way to proceed
>  at this point in the schedule.  It doesn't really impact me a lot
>  personally as I build my own kernels anyway. 
Fair enough. 

>  Given there are not so many users on this architecture, is it
> feasible
>  to take a more do-or-die approach, to try and put that f33 kernel in
>  there and if it doesn't come together, f34 is simply released without
>  any ppc64le? 
I'm not in favor of that. For example, we use Fedora on our builders, if
there's no f34 for them they will be out of sync with the rest and
possibly missing features people will depend on for builds. 

>  The upside is that if it does work, all those new GPUs will be
> working,
>  plug and play.  I'm talking about all the new RX 6800 cards and also the
>  previous generation, RX 5700 and friends.
>  
>  If f34 is status quo, 64k kernel, people have to either:
>  
>  a) use a GPU that is much older now, like Radeon RX580 Polaris, or
>  
>  b) compile a kernel with 4k, or
>  
>  c) download a kernel with 4k from somebody
>  
>  I don't mind too much whether it is f34 or f35 and I understand you
>  probably have higher priority things to do at this point in the release
>  cycle. 
Given that it's not passed through the process yet, I would want it to
land in f35. For example, I haven't heard any feedback from ibm yet... 

kevin

[Dan Horák]

On Mon, 18 Jan 2021 17:31:25 -0800
Kevin Fenzi <kevin(a)scrye.com&gt; wrote:

>  On Sun, Jan 17, 2021 at 11:31:32PM +0100, Daniel Pocock wrote:
>  > 
>  > Thanks for the feedback.  I understand that is the normal way to proceed
>  > at this point in the schedule.  It doesn't really impact me a lot
>  > personally as I build my own kernels anyway.
>  
>  Fair enough. 
>  
>  > Given there are not so many users on this architecture, is it feasible
>  > to take a more do-or-die approach, to try and put that f33 kernel in
>  > there and if it doesn't come together, f34 is simply released without
>  > any ppc64le?
>  
>  I'm not in favor of that. For example, we use Fedora on our builders, if
>  there's no f34 for them they will be out of sync with the rest and
>  possibly missing features people will depend on for builds. 
>  
>  > The upside is that if it does work, all those new GPUs will be working,
>  > plug and play.  I'm talking about all the new RX 6800 cards and also the
>  > previous generation, RX 5700 and friends.
>  > 
>  > If f34 is status quo, 64k kernel, people have to either:
>  > 
>  > a) use a GPU that is much older now, like Radeon RX580 Polaris, or
>  > 
>  > b) compile a kernel with 4k, or
>  > 
>  > c) download a kernel with 4k from somebody 
I believe we could provide a semi-official 4k kernels and document the
issue in the "known bugs" page

>  > 
>  > I don't mind too much whether it is f34 or f35 and I understand you
>  > probably have higher priority things to do at this point in the release
>  > cycle.
>  
>  Given that it's not passed through the process yet, I would want it to
>  land in f35. For example, I haven't heard any feedback from ibm yet...  
I have asked them again thru a different channel, so hopefully we will
hear back soon.


		Dan

[Tom Stellard]

On 1/15/21 11:31 AM, Mohan Boddu wrote:
>  Hi all,
>  
>  Per the Fedora 34 schedule[1] we will start a mass rebuild for Fedora 34
>  on Jan 20th 2021. We will run a mass rebuild for Fedora 34 for the
>  changes listed in:
>  
>  https://pagure.io/releng/issues?status=Open&tags=mass+rebuild
>  
>  Mass rebuild will be done in a side tag (f34-rebuild) and moved over
>  when completed.
>   
Once a package is built in the side-tag, does it enter the side-tag 
buildroot so that other packages built in the side-tag may use it?

-Tom

>  Failures can be seen
>  https://kojipkgs.fedoraproject.org/mass-rebuild/f34-failures.html
>  
>  Things still needing rebuilt
>  https://kojipkgs.fedoraproject.org/mass-rebuild/f34-need-rebuild.html
>  
>  FTBFS bugs will be filed shortly.
>  
>  Please be sure to let releng know if you see any bugs in the
>  reporting. You can contact releng in #fedora-releng on freenode, by
>  dropping an email to our list[2] or filing an issue in pagure[3]
>  
>  Regards,
>  Mohan Boddu.
>  
>  [1] https://fedorapeople.org/groups/schedule/f-34/f-34-key-tasks.html
>  [2] https://lists.fedoraproject.org/admin/lists/rel-eng.lists.fedoraproject.org/
>  [3] https://pagure.io/releng/
>  _______________________________________________
>  devel-announce mailing list -- devel-announce(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-announce-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]...
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]
>

[Miro Hrončok]

On 20. 01. 21 17:35, Tom Stellard wrote:
>  On 1/15/21 11:31 AM, Mohan Boddu wrote:
> > Hi all,
> >
> > Per the Fedora 34 schedule[1] we will start a mass rebuild for Fedora 34
> > on Jan 20th 2021. We will run a mass rebuild for Fedora 34 for the
> > changes listed in:
> >
> > https://pagure.io/releng/issues?status=Open&tags=mass+rebuild
> >
> > Mass rebuild will be done in a side tag (f34-rebuild) and moved over
> > when completed.
> >
>  
>  Once a package is built in the side-tag, does it enter the side-tag buildroot so 
>  that other packages built in the side-tag may use it? 
Usually yes, but not with the mass rebuild site tag (AFAIK).

-- 
Miro Hrončok
--
Phone: [phone]
IRC: mhroncok

[Kevin Fenzi]

On Wed, Jan 20, 2021 at 08:35:54AM -0800, Tom Stellard wrote:
>  On 1/15/21 11:31 AM, Mohan Boddu wrote:
>  > Hi all,
>  > 
>  > Per the Fedora 34 schedule[1] we will start a mass rebuild for Fedora 34
>  > on Jan 20th 2021. We will run a mass rebuild for Fedora 34 for the
>  > changes listed in:
>  > 
>  > https://pagure.io/releng/issues?status=Open&tags=mass+rebuild
>  > 
>  > Mass rebuild will be done in a side tag (f34-rebuild) and moved over
>  > when completed.
>  > 
>  
>  Once a package is built in the side-tag, does it enter the side-tag
>  buildroot so that other packages built in the side-tag may use it? 
Nope... or well, you shouldn't count on it. 

Basically things are submitted as fast as they can be, if we waited for
newrepo between each build it would take months to rebuild things. 

In some kind of shiny future, it would be cool to be able to see what
things need bootstrapping, bootstrap them into the rebuild and then
rebuild everything else in dependecy order. Thats gonna be a lot of work
though. 

kevin

[Mohan Boddu]

On Wed, Jan 20, 2021 at 11:36 AM Tom Stellard <tstellar(a)redhat.com&gt; wrote:
>  On 1/15/21 11:31 AM, Mohan Boddu wrote:
>  > Hi all,
>  >
>  > Per the Fedora 34 schedule[1] we will start a mass rebuild for Fedora 34
>  > on Jan 20th 2021. We will run a mass rebuild for Fedora 34 for the
>  > changes listed in:
>  >
>  > https://pagure.io/releng/issues?status=Open&tags=mass+rebuild
>  >
>  > Mass rebuild will be done in a side tag (f34-rebuild) and moved over
>  > when completed.
>  >
>
>  Once a package is built in the side-tag, does it enter the side-tag
>  buildroot so that other packages built in the side-tag may use it? 
Nope, it wont, mass rebuild side tag uses the general release
buildroot (for ex: f34-build), so in case if you really want something
to be available in the buildroot, you can run a normal build and the
mass rebuild will pick up the new buildroot, but only some mass
rebuild builds will pick up the new buildroot depending on where
exactly the mass rebuild is at that point of time. That's why we ask
the maintainers to land all their changes in rawhide before we start
the mass rebuild.

>
> -Tom
>
> > Failures can be seen
> > https://kojipkgs.fedoraproject.org/mass-rebuild/f34-failures.html
> >
> > Things still needing rebuilt
> > https://kojipkgs.fedoraproject.org/mass-rebuild/f34-need-rebuild.html
> >
> > FTBFS bugs will be filed shortly.
> >
> > Please be sure to let releng know if you see any bugs in the
> > reporting. You can contact releng in #fedora-releng on freenode, by
> > dropping an email to our list[2] or filing an issue in pagure[3]
> >
> > Regards,
> > Mohan Boddu.
> >
> > [1] https://fedorapeople.org/groups/schedule/f-34/f-34-key-tasks.html
> > [2]
https://lists.fedoraproject.org/admin/lists/rel-eng.lists.fedoraproject.org/
> > [3] https://pagure.io/releng/
> > _______________________________________________
> > devel-announce mailing list -- devel-announce(a)lists.fedoraproject.org
> > To unsubscribe send an email to devel-announce-leave(a)lists.fedoraproject.org
> > Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives:
https://lists.fedoraproject.org/archives/list/[e-mail]...
> > _______________________________________________
> > devel mailing list -- devel(a)lists.fedoraproject.org
> > To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
> > Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives:
https://lists.fedoraproject.org/archives/list/[e-mail]
> >
>

[Mohan Boddu]

Hi all,

We are delaying the mass rebuild by a day as of now due to bugs in gcc
and dwz. As of now, we are expecting to start mass rebuild tomorrow,
Jan 21st 2021. There is a new build of gcc running currently which has
fixes to gcc bugs, but we are still figuring out the dwz fixes. We
will keep you posted with any further developments.

Thanks.

On Fri, Jan 15, 2021 at 2:31 PM Mohan Boddu <mboddu(a)bhujji.com&gt; wrote:
>  Hi all,
>
>  Per the Fedora 34 schedule[1] we will start a mass rebuild for Fedora 34
>  on Jan 20th 2021. We will run a mass rebuild for Fedora 34 for the
>  changes listed in:
>
>  https://pagure.io/releng/issues?status=Open&tags=mass+rebuild
>
>  Mass rebuild will be done in a side tag (f34-rebuild) and moved over
>  when completed.
>
>  Failures can be seen
>  https://kojipkgs.fedoraproject.org/mass-rebuild/f34-failures.html
>
>  Things still needing rebuilt
>  https://kojipkgs.fedoraproject.org/mass-rebuild/f34-need-rebuild.html
>
>  FTBFS bugs will be filed shortly.
>
>  Please be sure to let releng know if you see any bugs in the
>  reporting. You can contact releng in #fedora-releng on freenode, by
>  dropping an email to our list[2] or filing an issue in pagure[3]
>
>  Regards,
>  Mohan Boddu.
>
>  [1] https://fedorapeople.org/groups/schedule/f-34/f-34-key-tasks.html
>  [2] https://lists.fedoraproject.org/admin/lists/rel-eng.lists.fedoraproject.org/
>  [3] https://pagure.io/releng/ 
_______________________________________________
devel-announce mailing list -- devel-announce(a)lists.fedoraproject.org
To unsubscribe send an email to devel-announce-leave(a)lists.fedoraproject.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/[e-mail]...

[Jakub Jelinek]

On Wed, Jan 20, 2021 at 02:17:28PM -0500, Mohan Boddu wrote:
>  We are delaying the mass rebuild by a day as of now due to bugs in
> gcc
>  and dwz. As of now, we are expecting to start mass rebuild tomorrow,
>  Jan 21st 2021. There is a new build of gcc running currently which has
>  fixes to gcc bugs, but we are still figuring out the dwz fixes. We
>  will keep you posted with any further developments. 
Both gcc-11.0.0-0.16.fc34 and dwz-0.13-6.fc34 with the fixes
are now in f34-build (and in eln-build too with s/fc34/eln108/).
Those who had their builds fail in the last 2 days because of s390x
rpm crashes, errors about strip failures of LTO debug sections or dwz
crashes can retry their builds, sorry for the inconvenience.

AFAIK we are waiting now for boost and maybe binutils.

	Jakub

[Fabio Valentini]

On Thu, Jan 21, 2021 at 5:10 PM Jakub Jelinek <jakub(a)redhat.com&gt; wrote:
>  On Wed, Jan 20, 2021 at 02:17:28PM -0500, Mohan Boddu wrote:
>  > We are delaying the mass rebuild by a day as of now due to bugs in gcc
>  > and dwz. As of now, we are expecting to start mass rebuild tomorrow,
>  > Jan 21st 2021. There is a new build of gcc running currently which has
>  > fixes to gcc bugs, but we are still figuring out the dwz fixes. We
>  > will keep you posted with any further developments.
>
>  Both gcc-11.0.0-0.16.fc34 and dwz-0.13-6.fc34 with the fixes
>  are now in f34-build (and in eln-build too with s/fc34/eln108/).
>  Those who had their builds fail in the last 2 days because of s390x
>  rpm crashes, errors about strip failures of LTO debug sections or dwz
>  crashes can retry their builds, sorry for the inconvenience.
>
>  AFAIK we are waiting now for boost and maybe binutils. 
boost 1.75 was merged yesterday, and binutils 2.36 was postponed to F35.

I'm wondering something else: Will automation trigger a "second mass
rebuild" for ELN once all the builds from the Fedora mass rebuild get
tagged into f34?
There's apparently issues with ELN right now, it looks like the base
buildroot is not installable because boost 1.75 builds were done in
the wrong order.

Fabio

[Kevin Fenzi]

On Mon, Jan 25, 2021 at 03:16:11PM +0100, Fabio Valentini wrote:
>  On Thu, Jan 21, 2021 at 5:10 PM Jakub Jelinek
> <jakub(a)redhat.com&gt; wrote:
>  >
>  > On Wed, Jan 20, 2021 at 02:17:28PM -0500, Mohan Boddu wrote:
>  > > We are delaying the mass rebuild by a day as of now due to bugs in gcc
>  > > and dwz. As of now, we are expecting to start mass rebuild tomorrow,
>  > > Jan 21st 2021. There is a new build of gcc running currently which has
>  > > fixes to gcc bugs, but we are still figuring out the dwz fixes. We
>  > > will keep you posted with any further developments.
>  >
>  > Both gcc-11.0.0-0.16.fc34 and dwz-0.13-6.fc34 with the fixes
>  > are now in f34-build (and in eln-build too with s/fc34/eln108/).
>  > Those who had their builds fail in the last 2 days because of s390x
>  > rpm crashes, errors about strip failures of LTO debug sections or dwz
>  > crashes can retry their builds, sorry for the inconvenience.
>  >
>  > AFAIK we are waiting now for boost and maybe binutils.
>  
>  boost 1.75 was merged yesterday, and binutils 2.36 was postponed to F35.
>  
>  I'm wondering something else: Will automation trigger a "second mass
>  rebuild" for ELN once all the builds from the Fedora mass rebuild get
>  tagged into f34?
>  There's apparently issues with ELN right now, it looks like the base
>  buildroot is not installable because boost 1.75 builds were done in
>  the wrong order. 
We will be disabling the ELN automation just before we merge the
f34-rebuild tag into f34. Then the ELN sig will do their own mass
rebuild manually. 

So, hopefully this will avoid any ordering issues. 

kevin

[Jonathan Wakely]

On 25/01/21 15:16 +0100, Fabio Valentini wrote:
> On Thu, Jan 21, 2021 at 5:10 PM Jakub Jelinek <jakub(a)redhat.com&gt;
> wrote:
> >
> > On Wed, Jan 20, 2021 at 02:17:28PM -0500, Mohan Boddu wrote:
> > > We are delaying the mass rebuild by a day as of now due to bugs in gcc
> > > and dwz. As of now, we are expecting to start mass rebuild tomorrow,
> > > Jan 21st 2021. There is a new build of gcc running currently which has
> > > fixes to gcc bugs, but we are still figuring out the dwz fixes. We
> > > will keep you posted with any further developments.
> >
> > Both gcc-11.0.0-0.16.fc34 and dwz-0.13-6.fc34 with the fixes
> > are now in f34-build (and in eln-build too with s/fc34/eln108/).
> > Those who had their builds fail in the last 2 days because of s390x
> > rpm crashes, errors about strip failures of LTO debug sections or dwz
> > crashes can retry their builds, sorry for the inconvenience.
> >
> > AFAIK we are waiting now for boost and maybe binutils.
>
> boost 1.75 was merged yesterday, and binutils 2.36 was postponed to F35.
>
> I'm wondering something else: Will automation trigger a "second mass
> rebuild" for ELN once all the builds from the Fedora mass rebuild get
> tagged into f34?
> There's apparently issues with ELN right now, it looks like the base
> buildroot is not installable because boost 1.75 builds were done in
> the wrong order. 
Wrong in what way?

[Fabio Valentini]

On Tue, Jan 26, 2021 at 4:47 PM Jonathan Wakely
<jwakely(a)fedoraproject.org&gt; wrote:
>  On 25/01/21 15:16 +0100, Fabio Valentini wrote:
>  >On Thu, Jan 21, 2021 at 5:10 PM Jakub Jelinek <jakub(a)redhat.com&gt; wrote:
>  >>
>  >> On Wed, Jan 20, 2021 at 02:17:28PM -0500, Mohan Boddu wrote:
>  >> > We are delaying the mass rebuild by a day as of now due to bugs in gcc
>  >> > and dwz. As of now, we are expecting to start mass rebuild tomorrow,
>  >> > Jan 21st 2021. There is a new build of gcc running currently which has
>  >> > fixes to gcc bugs, but we are still figuring out the dwz fixes. We
>  >> > will keep you posted with any further developments.
>  >>
>  >> Both gcc-11.0.0-0.16.fc34 and dwz-0.13-6.fc34 with the fixes
>  >> are now in f34-build (and in eln-build too with s/fc34/eln108/).
>  >> Those who had their builds fail in the last 2 days because of s390x
>  >> rpm crashes, errors about strip failures of LTO debug sections or dwz
>  >> crashes can retry their builds, sorry for the inconvenience.
>  >>
>  >> AFAIK we are waiting now for boost and maybe binutils.
>  >
>  >boost 1.75 was merged yesterday, and binutils 2.36 was postponed to F35. 
(snip)

>  >I'm wondering something else: Will automation trigger a
> "second mass
>  >rebuild" for ELN once all the builds from the Fedora mass rebuild get
>  >tagged into f34?
>  >There's apparently issues with ELN right now, it looks like the base
>  >buildroot is not installable because boost 1.75 builds were done in
>  >the wrong order.
>
>  Wrong in what way? 
Looks like this was a transient issue. koschei is no longer
complaining about boost.
The problem was some dependency chain involving rpm-build ->
source-highlight -> boost 1.73, but it appears to have been a
temporary problem, if it was a problem at all.

Fabio

[Jonathan Wakely]

On 26/01/21 16:52 +0100, Fabio Valentini wrote:
> On Tue, Jan 26, 2021 at 4:47 PM Jonathan Wakely
> <jwakely(a)fedoraproject.org&gt; wrote:
> >
> > On 25/01/21 15:16 +0100, Fabio Valentini wrote:
> > >On Thu, Jan 21, 2021 at 5:10 PM Jakub Jelinek <jakub(a)redhat.com&gt; wrote:
> > >>
> > >> On Wed, Jan 20, 2021 at 02:17:28PM -0500, Mohan Boddu wrote:
> > >> > We are delaying the mass rebuild by a day as of now due to bugs in gcc
> > >> > and dwz. As of now, we are expecting to start mass rebuild tomorrow,
> > >> > Jan 21st 2021. There is a new build of gcc running currently which has
> > >> > fixes to gcc bugs, but we are still figuring out the dwz fixes. We
> > >> > will keep you posted with any further developments.
> > >>
> > >> Both gcc-11.0.0-0.16.fc34 and dwz-0.13-6.fc34 with the fixes
> > >> are now in f34-build (and in eln-build too with s/fc34/eln108/).
> > >> Those who had their builds fail in the last 2 days because of s390x
> > >> rpm crashes, errors about strip failures of LTO debug sections or dwz
> > >> crashes can retry their builds, sorry for the inconvenience.
> > >>
> > >> AFAIK we are waiting now for boost and maybe binutils.
> > >
> > >boost 1.75 was merged yesterday, and binutils 2.36 was postponed to F35.
>
> (snip)
>
> > >I'm wondering something else: Will automation trigger a "second mass
> > >rebuild" for ELN once all the builds from the Fedora mass rebuild get
> > >tagged into f34?
> > >There's apparently issues with ELN right now, it looks like the base
> > >buildroot is not installable because boost 1.75 builds were done in
> > >the wrong order.
> >
> > Wrong in what way?
>
> Looks like this was a transient issue. koschei is no longer
> complaining about boost.
> The problem was some dependency chain involving rpm-build ->
> source-highlight -> boost 1.73, but it appears to have been a
> temporary problem, if it was a problem at all. 
Possible the ELN buildroot contains gdb-headless which depends on
source-highlight which depends on boost, which causes a bootstrapping
problem (you can't update boost, because then you can't install the
buildroot to rebuild anything else). It might have been solved by
untagging the new boost. I thought that was a problem for RHEL builds,
but maybe it's ELN too.

The Fedora buildroot appears to contain gdb-minimal instead of
gdb-headless, and that doesn't depend on source-highlight.

[Jeff Law]

On 1/26/21 9:13 AM, Jonathan Wakely wrote:
>  On 26/01/21 16:52 +0100, Fabio Valentini wrote:
> > On Tue, Jan 26, 2021 at 4:47 PM Jonathan Wakely
> > <jwakely(a)fedoraproject.org&gt; wrote:
> >>
> >> On 25/01/21 15:16 +0100, Fabio Valentini wrote:
> >> >On Thu, Jan 21, 2021 at 5:10 PM Jakub Jelinek <jakub(a)redhat.com&gt;
> >> wrote:
> >> >>
> >> >> On Wed, Jan 20, 2021 at 02:17:28PM -0500, Mohan Boddu wrote:
> >> >> > We are delaying the mass rebuild by a day as of now due to bugs
> >> in gcc
> >> >> > and dwz. As of now, we are expecting to start mass rebuild
> >> tomorrow,
> >> >> > Jan 21st 2021. There is a new build of gcc running currently
> >> which has
> >> >> > fixes to gcc bugs, but we are still figuring out the dwz fixes. We
> >> >> > will keep you posted with any further developments.
> >> >>
> >> >> Both gcc-11.0.0-0.16.fc34 and dwz-0.13-6.fc34 with the fixes
> >> >> are now in f34-build (and in eln-build too with s/fc34/eln108/).
> >> >> Those who had their builds fail in the last 2 days because of s390x
> >> >> rpm crashes, errors about strip failures of LTO debug sections or
> >> dwz
> >> >> crashes can retry their builds, sorry for the inconvenience.
> >> >>
> >> >> AFAIK we are waiting now for boost and maybe binutils.
> >> >
> >> >boost 1.75 was merged yesterday, and binutils 2.36 was postponed to
> >> F35.
> >
> > (snip)
> >
> >> >I'm wondering something else: Will automation trigger a "second
> mass
> >> >rebuild" for ELN once all the builds from the Fedora mass rebuild get
> >> >tagged into f34?
> >> >There's apparently issues with ELN right now, it looks like the base
> >> >buildroot is not installable because boost 1.75 builds were done in
> >> >the wrong order.
> >>
> >> Wrong in what way?
> >
> > Looks like this was a transient issue. koschei is no longer
> > complaining about boost.
> > The problem was some dependency chain involving rpm-build ->
> > source-highlight -> boost 1.73, but it appears to have been a
> > temporary problem, if it was a problem at all.
>
>  Possible the ELN buildroot contains gdb-headless which depends on
>  source-highlight which depends on boost, which causes a bootstrapping
>  problem (you can't update boost, because then you can't install the
>  buildroot to rebuild anything else). It might have been solved by
>  untagging the new boost. I thought that was a problem for RHEL builds,
>  but maybe it's ELN too.
>
>  The Fedora buildroot appears to contain gdb-minimal instead of
>  gdb-headless, and that doesn't depend on source-highlight. 
See:
https://src.fedoraproject.org/rpms/gdb/pull-request/8#request_diff

It's on the agenda for the team call today.

jeff

[Mohan Boddu]

On Tue, Jan 26, 2021 at 10:52 AM Fabio Valentini <decathorpe(a)gmail.com&gt; wrote:
>  On Tue, Jan 26, 2021 at 4:47 PM Jonathan Wakely
>  <jwakely(a)fedoraproject.org&gt; wrote:
>  >
>  > On 25/01/21 15:16 +0100, Fabio Valentini wrote:
>  > >On Thu, Jan 21, 2021 at 5:10 PM Jakub Jelinek <jakub(a)redhat.com&gt; wrote:
>  > >>
>  > >> On Wed, Jan 20, 2021 at 02:17:28PM -0500, Mohan Boddu wrote:
>  > >> > We are delaying the mass rebuild by a day as of now due to bugs in
> gcc
>  > >> > and dwz. As of now, we are expecting to start mass rebuild tomorrow,
>  > >> > Jan 21st 2021. There is a new build of gcc running currently which
> has
>  > >> > fixes to gcc bugs, but we are still figuring out the dwz fixes. We
>  > >> > will keep you posted with any further developments.
>  > >>
>  > >> Both gcc-11.0.0-0.16.fc34 and dwz-0.13-6.fc34 with the fixes
>  > >> are now in f34-build (and in eln-build too with s/fc34/eln108/).
>  > >> Those who had their builds fail in the last 2 days because of s390x
>  > >> rpm crashes, errors about strip failures of LTO debug sections or dwz
>  > >> crashes can retry their builds, sorry for the inconvenience.
>  > >>
>  > >> AFAIK we are waiting now for boost and maybe binutils.
>  > >
>  > >boost 1.75 was merged yesterday, and binutils 2.36 was postponed to F35.
>
>  (snip)
>
>  > >I'm wondering something else: Will automation trigger a "second mass
>  > >rebuild" for ELN once all the builds from the Fedora mass rebuild get
>  > >tagged into f34?
>  > >There's apparently issues with ELN right now, it looks like the base
>  > >buildroot is not installable because boost 1.75 builds were done in
>  > >the wrong order.
>  >
>  > Wrong in what way?
>
>  Looks like this was a transient issue. koschei is no longer
>  complaining about boost.
>  The problem was some dependency chain involving rpm-build ->
>  source-highlight -> boost 1.73, but it appears to have been a
>  temporary problem, if it was a problem at all. 
Also, boost 1.75 builds were built in a side tag, when the side tag
was merged it didn't merge boost-1.75 properly and the eln builds
started using older boost, that also caused some issues with deps.

>
> Fabio
> _______________________________________________
> devel mailing list -- devel(a)lists.fedoraproject.org
> To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
> Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
https://lists.fedoraproject.org/archives/list/[e-mail]

[Mohan Boddu]

Fedora 34 mass rebuild of rpms is done and we started running the mass
rebuild of modules. FTBFS tickets [0] are being filed now.

[0] https://bugzilla.redhat.com/show_bug.cgi?id=1868278



On Tue, Jan 26, 2021 at 1:25 PM Mohan Boddu <mboddu(a)bhujji.com&gt; wrote:
>
> On Tue, Jan 26, 2021 at 10:52 AM Fabio Valentini <decathorpe(a)gmail.com&gt; wrote:
> >
> > On Tue, Jan 26, 2021 at 4:47 PM Jonathan Wakely
> > <jwakely(a)fedoraproject.org&gt; wrote:
> > >
> > > On 25/01/21 15:16 +0100, Fabio Valentini wrote:
> > > >On Thu, Jan 21, 2021 at 5:10 PM Jakub Jelinek <jakub(a)redhat.com&gt;
wrote:
> > > >>
> > > >> On Wed, Jan 20, 2021 at 02:17:28PM -0500, Mohan Boddu wrote:
> > > >> > We are delaying the mass rebuild by a day as of now due to
bugs in gcc
> > > >> > and dwz. As of now, we are expecting to start mass rebuild
tomorrow,
> > > >> > Jan 21st 2021. There is a new build of gcc running currently
which has
> > > >> > fixes to gcc bugs, but we are still figuring out the dwz
fixes. We
> > > >> > will keep you posted with any further developments.
> > > >>
> > > >> Both gcc-11.0.0-0.16.fc34 and dwz-0.13-6.fc34 with the fixes
> > > >> are now in f34-build (and in eln-build too with s/fc34/eln108/).
> > > >> Those who had their builds fail in the last 2 days because of
s390x
> > > >> rpm crashes, errors about strip failures of LTO debug sections or
dwz
> > > >> crashes can retry their builds, sorry for the inconvenience.
> > > >>
> > > >> AFAIK we are waiting now for boost and maybe binutils.
> > > >
> > > >boost 1.75 was merged yesterday, and binutils 2.36 was postponed to
F35.
> >
> > (snip)
> >
> > > >I'm wondering something else: Will automation trigger a
"second mass
> > > >rebuild" for ELN once all the builds from the Fedora mass rebuild
get
> > > >tagged into f34?
> > > >There's apparently issues with ELN right now, it looks like the
base
> > > >buildroot is not installable because boost 1.75 builds were done in
> > > >the wrong order.
> > >
> > > Wrong in what way?
> >
> > Looks like this was a transient issue. koschei is no longer
> > complaining about boost.
> > The problem was some dependency chain involving rpm-build ->
> > source-highlight -> boost 1.73, but it appears to have been a
> > temporary problem, if it was a problem at all.
>
> Also, boost 1.75 builds were built in a side tag, when the side tag
> was merged it didn't merge boost-1.75 properly and the eln builds
> started using older boost, that also caused some issues with deps.
>
> >
> > Fabio
> > _______________________________________________
> > devel mailing list -- devel(a)lists.fedoraproject.org
> > To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
> > Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives:
https://lists.fedoraproject.org/archives/list/[e-mail]

[Stephen Gallagher]

On Mon, Feb 1, 2021 at 3:04 PM Mohan Boddu <mboddu(a)bhujji.com&gt; wrote:
>  Fedora 34 mass rebuild of rpms is done and we started running the mass
>  rebuild of modules. FTBFS tickets [0] are being filed now.
>  
A mass-rebuild of ELN packages is also underway right now.

[Vít Ondruch]

It seems you are once again fighting with bootstrapped packages (e.g. 
rubygem-actionpack [1]). I would suggest if you tried the build with 
boostrapping mode [2] enabled first.


Vít


[1] https://src.fedoraproject.org/rpms/rubygem-actionpack

[2] https://docs.fedoraproject.org/en-US/packaging-guidelines/#bootstrapping


Dne 01. 02. 21 v 21:31 Stephen Gallagher napsal(a):
>  On Mon, Feb 1, 2021 at 3:04 PM Mohan Boddu <mboddu(a)bhujji.com&gt;
> wrote:
> > Fedora 34 mass rebuild of rpms is done and we started running the mass
> > rebuild of modules. FTBFS tickets [0] are being filed now.
> >
>  A mass-rebuild of ELN packages is also underway right now.
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]

[Kevin Fenzi]

On Mon, Feb 01, 2021 at 03:03:07PM -0500, Mohan Boddu wrote:
>  Fedora 34 mass rebuild of rpms is done and we started running the
> mass
>  rebuild of modules. FTBFS tickets [0] are being filed now.
>  
>  [0] https://bugzilla.redhat.com/show_bug.cgi?id=1868278 
Additionally, the eln mass rebuild is running now also. 

Since it's many fewer packages it should finish before too long... 

kevin

[Milan Crha]

On Mon, 2021-02-01 at 15:03 -0500, Mohan Boddu wrote:
>  Fedora 34 mass rebuild of rpms is done 
	Hi,
I just noticed that syncevolution built just fine on the January 30th,
but it is failing to build right now, using the same sources as releng.
I opened [1] for it.

It kind of scares me, I do not know how to decipher it. Is it that the
mass rebuild was not done against proper packages?
	Bye,
	Milan

[1] https://bugzilla.redhat.com/show_bug.cgi?id=1926239

[Jonathan Wakely]

On 08/02/21 14:40 +0100, Milan Crha wrote:
> On Mon, 2021-02-01 at 15:03 -0500, Mohan Boddu wrote:
> > Fedora 34 mass rebuild of rpms is done
>
> 	Hi,
> I just noticed that syncevolution built just fine on the January 30th,
> but it is failing to build right now, using the same sources as releng.
> I opened [1] for it.
>
> It kind of scares me, I do not know how to decipher it. Is it that the
> mass rebuild was not done against proper packages? 
No, it's simply that glib2 was changed a few days ago, after the mass
rebuild. The syncevolution sources didn't change, but some of the
headers it depends on did change.

[Milan Crha]

On Mon, 2021-02-08 at 23:17 +0100, Jonathan Wakely wrote:
>  No, it's simply that glib2 was changed a few days ago, after the
> mass
>  rebuild. The syncevolution sources didn't change, but some of the
>  headers it depends on did change. 
	Hi,
you are right, it was just a good timing thing.
	Bye,
	Milan