F37 side tag after branching point

[Iñaki Ucar]

Hi all,

We have a new R version sitting on a side tag (f37-build-side-55653)
for a few weeks now, where packages are being rebuilt as time permits.
Unfortunately, F37 is not rawhide anymore, so the question is whether
this side tag could be safely merged both in F37 and rawhide when it
is ready.

Thanks,
-- 
Iñaki Úcar

[Maxwell G]

On Tuesday, August 23, 2022 1:16:00 PM CDT Iñaki Ucar wrote:
>  We have a new R version sitting on a side tag (f37-build-side-55653)
>  for a few weeks now, where packages are being rebuilt as time permits. 
Can this perhaps be handled differently next time? I admit that I'm not 
familiar with the R ecosystem, so the answer may be no. Side tags are not 
meant to be open for this long. So far, this R rebuild has caused a lot of 
problems (see "The R stack in Rawhide is on fire"). What are the issues that 
prevent the rebuild from happening all at once? Can it be staged in COPR to 
make sure nothing will break? Can the packages be built all at once with a 
script?

>  Unfortunately, F37 is not rawhide anymore, so the question is
> whether
>  this side tag could be safely merged both in F37 and rawhide when it
>  is ready. 
I'll defer to the releng folks, but I think you should be able to merge the 
sidetag normally through the Bodhi interface, though it will be for f37 and 
not rawhide. You'll have to rebuild everything in rawhide.

-- 
Thanks,

Maxwell G (@gotmax23)
Pronouns: He/Him/His

[Iñaki Ucar]

On Wed, 24 Aug 2022 at 04:56, Maxwell G <gotmax(a)e.email&gt; wrote:
>  On Tuesday, August 23, 2022 1:16:00 PM CDT Iñaki Ucar wrote:
>  > We have a new R version sitting on a side tag (f37-build-side-55653)
>  > for a few weeks now, where packages are being rebuilt as time permits.
>
>  Can this perhaps be handled differently next time? I admit that I'm not
>  familiar with the R ecosystem, so the answer may be no. Side tags are not
>  meant to be open for this long. So far, this R rebuild has caused a lot of
>  problems (see "The R stack in Rawhide is on fire"). What are the issues that
>  prevent the rebuild from happening all at once? 
Time and provenpackager hands. The lack of them, to be precise.

>  Can it be staged in COPR to
>  make sure nothing will break? Can the packages be built all at once with a
>  script? 
Sure, but this is additional work that, again, requires time. See above.

We are in the process of creating a SIG, a FAS group (already done),
and adding commit access to this group to all the R software, so that
more time and more hands can be invested in the future. But yet again,
this requires time, and people were a bit overloaded already. So it is
what it is for now.

>  > Unfortunately, F37 is not rawhide anymore, so the question is
> whether
>  > this side tag could be safely merged both in F37 and rawhide when it
>  > is ready.
>
>  I'll defer to the releng folks, but I think you should be able to merge the
>  sidetag normally through the Bodhi interface, though it will be for f37 and
>  not rawhide. You'll have to rebuild everything in rawhide. 
We expected that much for F37, we just hoped that the latter could be avoided.

-- 
Iñaki Úcar

[Petr Pisar]

V Tue, Aug 23, 2022 at 08:16:00PM +0200, Iñaki Ucar napsal(a):
>  Hi all,
>  
>  We have a new R version sitting on a side tag (f37-build-side-55653)
>  for a few weeks now, where packages are being rebuilt as time permits.
>  Unfortunately, F37 is not rawhide anymore, so the question is whether
>  this side tag could be safely merged both in F37 and rawhide when it
>  is ready.
>   
I think you can tag any package anywhere. Therefore should be possible to get
the same build into both Fedoras.

However, it could be unsafe (e.g. a change in C toolchain to distribution
macros). To mitigate it I think you can rebuild the packages in a F38 side tag
without additional commits. Just follow the order which was used in F37 side
tag. The commits exist both in F37 and F38 git branches. F38 builds will get
unique release strings due to differing %dist tag.

-- Petr

[Fabio Valentini]

On Wed, Aug 24, 2022 at 10:39 AM Petr Pisar <ppisar(a)redhat.com&gt; wrote:
>  V Tue, Aug 23, 2022 at 08:16:00PM +0200, Iñaki Ucar napsal(a):
>  > Hi all,
>  >
>  > We have a new R version sitting on a side tag (f37-build-side-55653)
>  > for a few weeks now, where packages are being rebuilt as time permits.
>  > Unfortunately, F37 is not rawhide anymore, so the question is whether
>  > this side tag could be safely merged both in F37 and rawhide when it
>  > is ready.
>  >
>  I think you can tag any package anywhere. Therefore should be possible to get
>  the same build into both Fedoras.
>
>  However, it could be unsafe (e.g. a change in C toolchain to distribution
>  macros). To mitigate it I think you can rebuild the packages in a F38 side tag
>  without additional commits. Just follow the order which was used in F37 side
>  tag. The commits exist both in F37 and F38 git branches. F38 builds will get
>  unique release strings due to differing %dist tag. 
When I was in a similar situation around the F37 branch point, releng
told me that while it would be *possible* for them to tag builds into
f38, it is possibly ill-advisable, for some of the reasons Petr
mentioned here, but also, the packages will get signed with the f37
key and not the f38 key, which will create another set of problems
down the line.

Iñaki, do you have a list of packages that still needs to be rebuilt for f37?
I have provenpackager rights and could handle those builds for you, if
you give me

- name of the side tag
- packages that still need to be rebuilt
- which order they need to be built in (or at least how to determine an order)
- what changelog message / commit message to use for the dist-git commits

And for rawhide / f38, I'd need the same, but the list of *all*
packages that need to be built, not only the ones that are still
missing from f38.
I'll probably write a short script to handle the actual task and let
it run in the background today.

Sorry for not volunteering earlier, but I'm already at my limits wrt/
time I can spend on Fedora.

Fabio

[Iñaki Ucar]

On Wed, 24 Aug 2022 at 10:59, Fabio Valentini <decathorpe(a)gmail.com&gt; wrote:
>  On Wed, Aug 24, 2022 at 10:39 AM Petr Pisar <ppisar(a)redhat.com&gt; wrote:
>  >
>  > V Tue, Aug 23, 2022 at 08:16:00PM +0200, Iñaki Ucar napsal(a):
>  > > Hi all,
>  > >
>  > > We have a new R version sitting on a side tag (f37-build-side-55653)
>  > > for a few weeks now, where packages are being rebuilt as time permits.
>  > > Unfortunately, F37 is not rawhide anymore, so the question is whether
>  > > this side tag could be safely merged both in F37 and rawhide when it
>  > > is ready.
>  > >
>  > I think you can tag any package anywhere. Therefore should be possible to get
>  > the same build into both Fedoras.
>  >
>  > However, it could be unsafe (e.g. a change in C toolchain to distribution
>  > macros). To mitigate it I think you can rebuild the packages in a F38 side tag
>  > without additional commits. Just follow the order which was used in F37 side
>  > tag. The commits exist both in F37 and F38 git branches. F38 builds will get
>  > unique release strings due to differing %dist tag.
>
>  When I was in a similar situation around the F37 branch point, releng
>  told me that while it would be *possible* for them to tag builds into
>  f38, it is possibly ill-advisable, for some of the reasons Petr
>  mentioned here, but also, the packages will get signed with the f37
>  key and not the f38 key, which will create another set of problems
>  down the line.
>
>  Iñaki, do you have a list of packages that still needs to be rebuilt for f37?
>  I have provenpackager rights and could handle those builds for you, 
Thank you very much, but note that @spot (in cc) is slowly working
through the packages, so maybe it's best to coordinate with him.

>  if you give me
>
>  - name of the side tag 
f37-build-side-55653

>  - packages that still need to be rebuilt 
blist-37.txt attached, but note this may change if @spot sends new builds.

>  - which order they need to be built in (or at least how to determine
> an order) 
blist-37.txt contains batches of packages to be built in order, one
per line. This was obtained by

1. cloning all the packages,
2. sed'ing the specs to set "%bcond_without bootstrap" and
"%bcond_with checks" in those specs that contain the opposite (not
sure if there is any other edge case),
3. evaluating the specs to get the BuildRequires, and finally
4. using these to get these dependent batches of packages.

The script is in https://pagure.io/R/packaging (in R, sorry). Finally,
I filtered out the packages that are currently in the side tag.

>  - what changelog message / commit message to use for the dist-git
> commits 
@spot is using just "R 4.2.1".

>  And for rawhide / f38, I'd need the same, but the list of *all*
>  packages that need to be built, not only the ones that are still
>  missing from f38. 
blist-38.txt attached. Again, one batch per line. If this rebuild
could be based on the prior one, then there would be no need to do
batches though.

>  I'll probably write a short script to handle the actual task and
> let
>  it run in the background today.
>
>  Sorry for not volunteering earlier, but I'm already at my limits wrt/
>  time I can spend on Fedora. 
No need to apologise. On the contrary, it's very generous of you
considering all the things you have on your plate already.

-- 
Iñaki Úcar

[Iñaki Ucar]

And now with the attachments... Classic.

On Wed, 24 Aug 2022 at 11:28, Iñaki Ucar <iucar(a)fedoraproject.org&gt; wrote:
>  On Wed, 24 Aug 2022 at 10:59, Fabio Valentini <decathorpe(a)gmail.com&gt; wrote:
>  >
>  > On Wed, Aug 24, 2022 at 10:39 AM Petr Pisar <ppisar(a)redhat.com&gt; wrote:
>  > >
>  > > V Tue, Aug 23, 2022 at 08:16:00PM +0200, Iñaki Ucar napsal(a):
>  > > > Hi all,
>  > > >
>  > > > We have a new R version sitting on a side tag (f37-build-side-55653)
>  > > > for a few weeks now, where packages are being rebuilt as time permits.
>  > > > Unfortunately, F37 is not rawhide anymore, so the question is whether
>  > > > this side tag could be safely merged both in F37 and rawhide when it
>  > > > is ready.
>  > > >
>  > > I think you can tag any package anywhere. Therefore should be possible to get
>  > > the same build into both Fedoras.
>  > >
>  > > However, it could be unsafe (e.g. a change in C toolchain to distribution
>  > > macros). To mitigate it I think you can rebuild the packages in a F38 side tag
>  > > without additional commits. Just follow the order which was used in F37 side
>  > > tag. The commits exist both in F37 and F38 git branches. F38 builds will get
>  > > unique release strings due to differing %dist tag.
>  >
>  > When I was in a similar situation around the F37 branch point, releng
>  > told me that while it would be *possible* for them to tag builds into
>  > f38, it is possibly ill-advisable, for some of the reasons Petr
>  > mentioned here, but also, the packages will get signed with the f37
>  > key and not the f38 key, which will create another set of problems
>  > down the line.
>  >
>  > Iñaki, do you have a list of packages that still needs to be rebuilt for f37?
>  > I have provenpackager rights and could handle those builds for you,
>
>  Thank you very much, but note that @spot (in cc) is slowly working
>  through the packages, so maybe it's best to coordinate with him.
>
>  > if you give me
>  >
>  > - name of the side tag
>
>  f37-build-side-55653
>
>  > - packages that still need to be rebuilt
>
>  blist-37.txt attached, but note this may change if @spot sends new builds.
>
>  > - which order they need to be built in (or at least how to determine an order)
>
>  blist-37.txt contains batches of packages to be built in order, one
>  per line. This was obtained by
>
>  1. cloning all the packages,
>  2. sed'ing the specs to set "%bcond_without bootstrap" and
>  "%bcond_with checks" in those specs that contain the opposite (not
>  sure if there is any other edge case),
>  3. evaluating the specs to get the BuildRequires, and finally
>  4. using these to get these dependent batches of packages.
>
>  The script is in https://pagure.io/R/packaging (in R, sorry). Finally,
>  I filtered out the packages that are currently in the side tag.
>
>  > - what changelog message / commit message to use for the dist-git commits
>
>  @spot is using just "R 4.2.1".
>
>  > And for rawhide / f38, I'd need the same, but the list of *all*
>  > packages that need to be built, not only the ones that are still
>  > missing from f38.
>
>  blist-38.txt attached. Again, one batch per line. If this rebuild
>  could be based on the prior one, then there would be no need to do
>  batches though.
>
>  > I'll probably write a short script to handle the actual task and let
>  > it run in the background today.
>  >
>  > Sorry for not volunteering earlier, but I'm already at my limits wrt/
>  > time I can spend on Fedora.
>
>  No need to apologise. On the contrary, it's very generous of you
>  considering all the things you have on your plate already.
>
>  --
>  Iñaki Úcar 


-- 
Iñaki Úcar

[Iñaki Ucar]

On Wed, 24 Aug 2022 at 10:49, Petr Pisar <ppisar(a)redhat.com&gt; wrote:
>  V Tue, Aug 23, 2022 at 08:16:00PM +0200, Iñaki Ucar napsal(a):
>  > Hi all,
>  >
>  > We have a new R version sitting on a side tag (f37-build-side-55653)
>  > for a few weeks now, where packages are being rebuilt as time permits.
>  > Unfortunately, F37 is not rawhide anymore, so the question is whether
>  > this side tag could be safely merged both in F37 and rawhide when it
>  > is ready.
>  >
>  I think you can tag any package anywhere. Therefore should be possible to get
>  the same build into both Fedoras.
>
>  However, it could be unsafe (e.g. a change in C toolchain to distribution
>  macros). To mitigate it I think you can rebuild the packages in a F38 side tag
>  without additional commits. Just follow the order which was used in F37 side
>  tag. The commits exist both in F37 and F38 git branches. F38 builds will get
>  unique release strings due to differing %dist tag. 
Thanks. The main issue is that there are circular dependencies, and it
requires bootstrapping in some cases and disabling the checks in
others, and then another pass to reenable everything. So if the
rawhide rebuild can be based on the result of the F37 side tag, then
bootstrapping etc. is not required, and the rebuild is fast and
straightforward. More so if no commits are needed.

-- 
Iñaki Úcar

[Fabio Valentini]

On Wed, Aug 24, 2022 at 10:58 AM Iñaki Ucar <iucar(a)fedoraproject.org&gt; wrote:
>  On Wed, 24 Aug 2022 at 10:49, Petr Pisar <ppisar(a)redhat.com&gt; wrote:
>  >
>  > V Tue, Aug 23, 2022 at 08:16:00PM +0200, Iñaki Ucar napsal(a):
>  > > Hi all,
>  > >
>  > > We have a new R version sitting on a side tag (f37-build-side-55653)
>  > > for a few weeks now, where packages are being rebuilt as time permits.
>  > > Unfortunately, F37 is not rawhide anymore, so the question is whether
>  > > this side tag could be safely merged both in F37 and rawhide when it
>  > > is ready.
>  > >
>  > I think you can tag any package anywhere. Therefore should be possible to get
>  > the same build into both Fedoras.
>  >
>  > However, it could be unsafe (e.g. a change in C toolchain to distribution
>  > macros). To mitigate it I think you can rebuild the packages in a F38 side tag
>  > without additional commits. Just follow the order which was used in F37 side
>  > tag. The commits exist both in F37 and F38 git branches. F38 builds will get
>  > unique release strings due to differing %dist tag.
>
>  Thanks. The main issue is that there are circular dependencies, and it
>  requires bootstrapping in some cases and disabling the checks in
>  others, and then another pass to reenable everything. So if the
>  rawhide rebuild can be based on the result of the F37 side tag, then
>  bootstrapping etc. is not required, and the rebuild is fast and
>  straightforward. More so if no commits are needed. 
That might be possible, I'll look into it when it comes to that.
First the f37 update needs to be finished and submitted :)

Fabio

[Miro Hrončok]

On 24. 08. 22 10:58, Iñaki Ucar wrote:
>  Thanks. The main issue is that there are circular dependencies, and
> it
>  requires bootstrapping in some cases and disabling the checks in
>  others, and then another pass to reenable everything. So if the
>  rawhide rebuild can be based on the result of the F37 side tag, then
>  bootstrapping etc. is not required, and the rebuild is fast and
>  straightforward. More so if no commits are needed. 
Either use the f37 builds or use the existing bootstrap commits. It is not 
necessary to build from the latest commit.

-- 
Miro Hrončok
-- 
Phone: [phone]
IRC: mhroncok

[Iñaki Ucar]

On Wed, 24 Aug 2022 at 12:04, Miro Hrončok <mhroncok(a)redhat.com&gt; wrote:
>  On 24. 08. 22 10:58, Iñaki Ucar wrote:
>  > Thanks. The main issue is that there are circular dependencies, and it
>  > requires bootstrapping in some cases and disabling the checks in
>  > others, and then another pass to reenable everything. So if the
>  > rawhide rebuild can be based on the result of the F37 side tag, then
>  > bootstrapping etc. is not required, and the rebuild is fast and
>  > straightforward. More so if no commits are needed.
>
>  Either use the f37 builds or use the existing bootstrap commits. It is not
>  necessary to build from the latest commit. 
How is that done? I don't see any arguments to "fedpkg build" to
specify the commit and I assumed that it uses the remote HEAD. Does it
honor the local HEAD?

-- 
Iñaki Úcar

[Miro Hrončok]

On 24. 08. 22 12:15, Iñaki Ucar wrote:
>  On Wed, 24 Aug 2022 at 12:04, Miro Hrončok
> <mhroncok(a)redhat.com&gt; wrote:
> >
> > On 24. 08. 22 10:58, Iñaki Ucar wrote:
> >> Thanks. The main issue is that there are circular dependencies, and it
> >> requires bootstrapping in some cases and disabling the checks in
> >> others, and then another pass to reenable everything. So if the
> >> rawhide rebuild can be based on the result of the F37 side tag, then
> >> bootstrapping etc. is not required, and the rebuild is fast and
> >> straightforward. More so if no commits are needed.
> >
> > Either use the f37 builds or use the existing bootstrap commits. It is not
> > necessary to build from the latest commit.
>  
>  How is that done? I don't see any arguments to "fedpkg build" to
>  specify the commit and I assumed that it uses the remote HEAD. Does it
>  honor the local HEAD? 
I don't think it can be done by fedpkg. I use this:

koji build f37 --nowait --fail-fast 
'git+https://src.fedoraproject.org/rpms/ipsilon.git#26169c9776b8bdb62a6f3da1fe5fe6c71071502b'

-- 
Miro Hrončok
-- 
Phone: [phone]
IRC: mhroncok

[Petr Pisar]

V Wed, Aug 24, 2022 at 10:58:00AM +0200, Iñaki Ucar napsal(a):
>  On Wed, 24 Aug 2022 at 10:49, Petr Pisar <ppisar(a)redhat.com&gt;
> wrote:
>  >
>  > V Tue, Aug 23, 2022 at 08:16:00PM +0200, Iñaki Ucar napsal(a):
>  > > Hi all,
>  > >
>  > > We have a new R version sitting on a side tag (f37-build-side-55653)
>  > > for a few weeks now, where packages are being rebuilt as time permits.
>  > > Unfortunately, F37 is not rawhide anymore, so the question is whether
>  > > this side tag could be safely merged both in F37 and rawhide when it
>  > > is ready.
>  > >
>  > I think you can tag any package anywhere. Therefore should be possible to get
>  > the same build into both Fedoras.
>  >
>  > However, it could be unsafe (e.g. a change in C toolchain to distribution
>  > macros). To mitigate it I think you can rebuild the packages in a F38 side tag
>  > without additional commits. Just follow the order which was used in F37 side
>  > tag. The commits exist both in F37 and F38 git branches. F38 builds will get
>  > unique release strings due to differing %dist tag.
>  
>  Thanks. The main issue is that there are circular dependencies, and it
>  requires bootstrapping in some cases and disabling the checks in
>  others, and then another pass to reenable everything. 
One can build the packages from the historical commits where the boostraping
was in effect. That's not a technical problem.

>  So if the rawhide rebuild can be based on the result of the F37 side
> tag,
>  then bootstrapping etc. is not required, and the rebuild is fast and
>  straightforward. More so if no commits are needed.
>  
This optimization is also possible.

-- Petr

[Iñaki Ucar]

On Wed, 24 Aug 2022 at 12:39, Petr Pisar <ppisar(a)redhat.com&gt; wrote:
>  > So if the rawhide rebuild can be based on the result of the F37 side tag,
>  > then bootstrapping etc. is not required, and the rebuild is fast and
>  > straightforward. More so if no commits are needed.
>  >
>  This optimization is also possible. 
Nice! And how would this work? It is a
you-need-help-from-releng-possible or an option-to-koji-possible? The
--repo-id option to "koji build" seems promising.

-- 
Iñaki Úcar

[Petr Pisar]

V Wed, Aug 24, 2022 at 03:00:23PM +0200, Iñaki Ucar napsal(a):
>  On Wed, 24 Aug 2022 at 12:39, Petr Pisar <ppisar(a)redhat.com&gt;
> wrote:
>  >
>  > > So if the rawhide rebuild can be based on the result of the F37 side tag,
>  > > then bootstrapping etc. is not required, and the rebuild is fast and
>  > > straightforward. More so if no commits are needed.
>  > >
>  > This optimization is also possible.
>  
>  Nice! And how would this work? It is a
>  you-need-help-from-releng-possible or an option-to-koji-possible? The
>  --repo-id option to "koji build" seems promising.
>  
You tag each f37 build into the f38 side tag with:

$ koji tag-pkg THE_F38_TAG THE_PACKAGE_BUILD

Then you use koji "wait-repo THE_F38_TAG --build THE_PACKAGE_BUILD" to make
sure the builds are available in a build root. Then you can build the f38
packages in the f38 side tag.

But be ware that it also can be risky. The f37 packages could compile in f37
compiler flags and the f38 rebuilds could retrieve them and use. You need to
know the packages to be sure that it is safe to short-circuit the builds.

-- Petr

[Kevin Fenzi]

Just to chime in from a releng perspective here... 

IMHO you should do builds for f38 now also (either by making a side tag
and bootstrapping them just like was done for f37, or tagging f37 builds
you need into the f38 sidetag). 

While it's technically possible to push the f37 builds into rawhide
also, it would take releng manually tagging them in, bypassing bodhi,
gating and CI completely. It's much better to build again for
f38/rawhide and let those builds get checked by gating and CI, etc. 

If you run into any problems, let me know...

kevin

[Iñaki Ucar]

Here we go:

- F37: https://bodhi.fedoraproject.org/updates/FEDORA-2022-8414514ae6
- rawhide: https://bodhi.fedoraproject.org/updates/FEDORA-2022-0c2d48988e

After the mass rebuild in the F37 side tag, we tagged all builds also
in a rawhide side tag, rebuilt everything in one go, untagged the F37
builds, and created the update for rawhide. Quick and easy.

Thanks all for your help.
Iñaki

On Wed, 24 Aug 2022 at 19:04, Kevin Fenzi <kevin(a)scrye.com&gt; wrote:
>  Just to chime in from a releng perspective here...
>
>  IMHO you should do builds for f38 now also (either by making a side tag
>  and bootstrapping them just like was done for f37, or tagging f37 builds
>  you need into the f38 sidetag).
>
>  While it's technically possible to push the f37 builds into rawhide
>  also, it would take releng manually tagging them in, bypassing bodhi,
>  gating and CI completely. It's much better to build again for
>  f38/rawhide and let those builds get checked by gating and CI, etc.
>
>  If you run into any problems, let me know...
>
>  kevin
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]
>  Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue


-- 
Iñaki Úcar