rpkg-util v3 change, moving copr builders to F35

[Pavel Raiskup]

Hello,

we are currently preparing Copr build system for migration to Fedora 35,
including the builder workers.

There's though, at least from the builder perspective, a major `rpkg-util`
behavior change [1] (removal of `auto_pack` feature) that many of the Copr
projects rely on.  This is a bit unexpected as `auto_pack = True` has been
the default in Copr for a long time, and the rpkg-util upstream intentions
behind the feature were generally misunderstood (even in Copr team).

That being said, some of the currently working builds on F34 would fail
after the migration to F35.

In order to make the migration a bit less dramatic, we plan to migrate in
the following steps:

  1. Document the feature drop and possible fixes/workarounds see [1], and
     properly announce the change in advance (here we are _now_).

  2. Change the default, and turn off the `auto_pack` feature for
     relatively short period of time on the current F34 builders
     (ie autopack=False).  This will happen soon in December 2021.
     The change will naturally trigger the build failures, but a bit
     earlier so users will be informed sooner ... with a _temporary_
     work-around in hand => everyone can provide the file
     <git-root>/rpkg.conf upstream, and opt-in the old behavior with with
     content like:

     [rpkg]
     # temporary work-around for rpkg-util v2, auto_pack is going to be
     # removed soon! See:
     # https://docs.pagure.org/copr.copr/rpkg_util_2_vs_3.html#rpkg-util-v3
     auto_pack = True

  3. Migrate to Fedora 35, early in January 2022.  This will make the
     `auto_pack = True` work-around non-working, and users will have to
     adapt, again, see [1].

Of course, anyone is encouraged to migrate to the new syntax **right now**,
sooner the better.  If done correctly, package can be built both with rpkg-util
v2 and v3 [2].

Sorry for inconvenience!  Any comment, idea or help is welcome!

[1] https://docs.pagure.org/copr.copr/rpkg_util_2_vs_3.html#rpkg-util-v3
[2]
https://pagure.io/copr/copr-hello/c/739ff9910ee8a9c76d7e97de2f6176106dc19...

Pavel

[Pavel Raiskup]

Hello all,

just a quick update on this topic.  This took a bit more time than I
originally expected, but I was able to test that configuration.

Please expect that step 2 happens on Monday, 2021-12-20 (I don't want to
break things just when weekend starts).

For more info see the previous email, and we should be able to help you
next week with the migration on #fedora-buildsys, when needed.

Also, the step 3 will be delayed at least by 20 days, so you still have the
1-month transition period.

Sorry for inconvenience,
Pavel

On Friday, November 19, 2021 9:44:51 AM CET Pavel Raiskup wrote:
>  we are currently preparing Copr build system for migration to Fedora
> 35,
>  including the builder workers.
>  
>  There's though, at least from the builder perspective, a major `rpkg-util`
>  behavior change [1] (removal of `auto_pack` feature) that many of the Copr
>  projects rely on.  This is a bit unexpected as `auto_pack = True` has been
>  the default in Copr for a long time, and the rpkg-util upstream intentions
>  behind the feature were generally misunderstood (even in Copr team).
>  
>  That being said, some of the currently working builds on F34 would fail
>  after the migration to F35.
>  
>  In order to make the migration a bit less dramatic, we plan to migrate in
>  the following steps:
>  
>    1. Document the feature drop and possible fixes/workarounds see [1], and
>       properly announce the change in advance (here we are _now_).
>  
>    2. Change the default, and turn off the `auto_pack` feature for
>       relatively short period of time on the current F34 builders
>       (ie autopack=False).  This will happen soon in December 2021.
>       The change will naturally trigger the build failures, but a bit
>       earlier so users will be informed sooner ... with a _temporary_
>       work-around in hand => everyone can provide the file
>       <git-root>/rpkg.conf upstream, and opt-in the old behavior with with
>       content like:
>  
>       [rpkg]
>       # temporary work-around for rpkg-util v2, auto_pack is going to be
>       # removed soon! See:
>       # https://docs.pagure.org/copr.copr/rpkg_util_2_vs_3.html#rpkg-util-v3
>       auto_pack = True
>  
>    3. Migrate to Fedora 35, early in January 2022.  This will make the
>       `auto_pack = True` work-around non-working, and users will have to
>       adapt, again, see [1].
>  
>  Of course, anyone is encouraged to migrate to the new syntax **right now**,
>  sooner the better.  If done correctly, package can be built both with rpkg-util
>  v2 and v3 [2].
>  
>  Sorry for inconvenience!  Any comment, idea or help is welcome!
>  
>  [1] https://docs.pagure.org/copr.copr/rpkg_util_2_vs_3.html#rpkg-util-v3
>  [2]
> https://pagure.io/copr/copr-hello/c/739ff9910ee8a9c76d7e97de2f6176106dc19...
>  
>  Pavel
>  
>

[Pavel Raiskup]

On Friday, December 17, 2021 12:32:37 PM CET Pavel Raiskup wrote:
>  Hello all,
>  
>  just a quick update on this topic.  This took a bit more time than I
>  originally expected, but I was able to test that configuration.
>  
>  Please expect that step 2 happens on Monday, 2021-12-20 (I don't want to
>  break things just when weekend starts). 
Hello, this is deployed now, once more the link:
[1] https://docs.pagure.org/copr.copr/rpkg_util_2_vs_3.html
And feel free to ask on #fedora-buildsys.

Pavel

>  For more info see the previous email, and we should be able to help
> you
>  next week with the migration on #fedora-buildsys, when needed.
>  
>  Also, the step 3 will be delayed at least by 20 days, so you still have the
>  1-month transition period.
>  
>  Sorry for inconvenience,
>  Pavel
>  
>  On Friday, November 19, 2021 9:44:51 AM CET Pavel Raiskup wrote:
>  > we are currently preparing Copr build system for migration to Fedora 35,
>  > including the builder workers.
>  > 
>  > There's though, at least from the builder perspective, a major `rpkg-util`
>  > behavior change [1] (removal of `auto_pack` feature) that many of the Copr
>  > projects rely on.  This is a bit unexpected as `auto_pack = True` has been
>  > the default in Copr for a long time, and the rpkg-util upstream intentions
>  > behind the feature were generally misunderstood (even in Copr team).
>  > 
>  > That being said, some of the currently working builds on F34 would fail
>  > after the migration to F35.
>  > 
>  > In order to make the migration a bit less dramatic, we plan to migrate in
>  > the following steps:
>  > 
>  >   1. Document the feature drop and possible fixes/workarounds see [1], and
>  >      properly announce the change in advance (here we are _now_).
>  > 
>  >   2. Change the default, and turn off the `auto_pack` feature for
>  >      relatively short period of time on the current F34 builders
>  >      (ie autopack=False).  This will happen soon in December 2021.
>  >      The change will naturally trigger the build failures, but a bit
>  >      earlier so users will be informed sooner ... with a _temporary_
>  >      work-around in hand => everyone can provide the file
>  >      <git-root>/rpkg.conf upstream, and opt-in the old behavior with with
>  >      content like:
>  > 
>  >      [rpkg]
>  >      # temporary work-around for rpkg-util v2, auto_pack is going to be
>  >      # removed soon! See:
>  >      # https://docs.pagure.org/copr.copr/rpkg_util_2_vs_3.html#rpkg-util-v3
>  >      auto_pack = True
>  > 
>  >   3. Migrate to Fedora 35, early in January 2022.  This will make the
>  >      `auto_pack = True` work-around non-working, and users will have to
>  >      adapt, again, see [1].
>  > 
>  > Of course, anyone is encouraged to migrate to the new syntax **right now**,
>  > sooner the better.  If done correctly, package can be built both with rpkg-util
>  > v2 and v3 [2].
>  > 
>  > Sorry for inconvenience!  Any comment, idea or help is welcome!
>  > 
>  > [1] https://docs.pagure.org/copr.copr/rpkg_util_2_vs_3.html#rpkg-util-v3
>  > [2]
> https://pagure.io/copr/copr-hello/c/739ff9910ee8a9c76d7e97de2f6176106dc19...
>  > 
>  > Pavel
>  > 
>  > 
>  
>

[clime]

Pavel,

As a developer of the rpkg-util project, I want to thank you for your
good work on the Copr integration.

Best regards and enjoy Christmas!
Michal Novotny

On Mon, 20 Dec 2021 at 08:05, Pavel Raiskup <praiskup(a)redhat.com&gt; wrote:
>
> On Friday, December 17, 2021 12:32:37 PM CET Pavel Raiskup wrote:
> > Hello all,
> >
> > just a quick update on this topic.  This took a bit more time than I
> > originally expected, but I was able to test that configuration.
> >
> > Please expect that step 2 happens on Monday, 2021-12-20 (I don't want to
> > break things just when weekend starts).
>
> Hello, this is deployed now, once more the link:
> [1] https://docs.pagure.org/copr.copr/rpkg_util_2_vs_3.html
> And feel free to ask on #fedora-buildsys.
>
> Pavel
>
> > For more info see the previous email, and we should be able to help you
> > next week with the migration on #fedora-buildsys, when needed.
> >
> > Also, the step 3 will be delayed at least by 20 days, so you still have the
> > 1-month transition period.
> >
> > Sorry for inconvenience,
> > Pavel
> >
> > On Friday, November 19, 2021 9:44:51 AM CET Pavel Raiskup wrote:
> > > we are currently preparing Copr build system for migration to Fedora 35,
> > > including the builder workers.
> > >
> > > There's though, at least from the builder perspective, a major
`rpkg-util`
> > > behavior change [1] (removal of `auto_pack` feature) that many of the Copr
> > > projects rely on.  This is a bit unexpected as `auto_pack = True` has been
> > > the default in Copr for a long time, and the rpkg-util upstream intentions
> > > behind the feature were generally misunderstood (even in Copr team).
> > >
> > > That being said, some of the currently working builds on F34 would fail
> > > after the migration to F35.
> > >
> > > In order to make the migration a bit less dramatic, we plan to migrate in
> > > the following steps:
> > >
> > >   1. Document the feature drop and possible fixes/workarounds see [1], and
> > >      properly announce the change in advance (here we are _now_).
> > >
> > >   2. Change the default, and turn off the `auto_pack` feature for
> > >      relatively short period of time on the current F34 builders
> > >      (ie autopack=False).  This will happen soon in December 2021.
> > >      The change will naturally trigger the build failures, but a bit
> > >      earlier so users will be informed sooner ... with a _temporary_
> > >      work-around in hand => everyone can provide the file
> > >      <git-root>/rpkg.conf upstream, and opt-in the old behavior with
with
> > >      content like:
> > >
> > >      [rpkg]
> > >      # temporary work-around for rpkg-util v2, auto_pack is going to be
> > >      # removed soon! See:
> > >      #
https://docs.pagure.org/copr.copr/rpkg_util_2_vs_3.html#rpkg-util-v3
> > >      auto_pack = True
> > >
> > >   3. Migrate to Fedora 35, early in January 2022.  This will make the
> > >      `auto_pack = True` work-around non-working, and users will have to
> > >      adapt, again, see [1].
> > >
> > > Of course, anyone is encouraged to migrate to the new syntax **right
now**,
> > > sooner the better.  If done correctly, package can be built both with
rpkg-util
> > > v2 and v3 [2].
> > >
> > > Sorry for inconvenience!  Any comment, idea or help is welcome!
> > >
> > > [1] https://docs.pagure.org/copr.copr/rpkg_util_2_vs_3.html#rpkg-util-v3
> > > [2]
https://pagure.io/copr/copr-hello/c/739ff9910ee8a9c76d7e97de2f6176106dc19...
> > >
> > > Pavel
> > >
> > >
> >
> >
>
>
>
>

[Pavel Raiskup]

Hello maintainers,

seems like we are finally able to migrate to Fedora 35.  If all the
necessary test pass, we should be OK to migrate tomorrow, 2022-01-25.

This also means that we'll finally run against rpkg-util v3+ == no way
back to auto_pack feature.

Pavel

On Friday, December 17, 2021 12:32:37 PM CET Pavel Raiskup wrote:
>  Hello all,
>  
>  just a quick update on this topic.  This took a bit more time than I
>  originally expected, but I was able to test that configuration.
>  
>  Please expect that step 2 happens on Monday, 2021-12-20 (I don't want to
>  break things just when weekend starts).
>  
>  For more info see the previous email, and we should be able to help you
>  next week with the migration on #fedora-buildsys, when needed.
>  
>  Also, the step 3 will be delayed at least by 20 days, so you still have the
>  1-month transition period.
>  
>  Sorry for inconvenience,
>  Pavel
>  
>  On Friday, November 19, 2021 9:44:51 AM CET Pavel Raiskup wrote:
>  > we are currently preparing Copr build system for migration to Fedora 35,
>  > including the builder workers.
>  > 
>  > There's though, at least from the builder perspective, a major `rpkg-util`
>  > behavior change [1] (removal of `auto_pack` feature) that many of the Copr
>  > projects rely on.  This is a bit unexpected as `auto_pack = True` has been
>  > the default in Copr for a long time, and the rpkg-util upstream intentions
>  > behind the feature were generally misunderstood (even in Copr team).
>  > 
>  > That being said, some of the currently working builds on F34 would fail
>  > after the migration to F35.
>  > 
>  > In order to make the migration a bit less dramatic, we plan to migrate in
>  > the following steps:
>  > 
>  >   1. Document the feature drop and possible fixes/workarounds see [1], and
>  >      properly announce the change in advance (here we are _now_).
>  > 
>  >   2. Change the default, and turn off the `auto_pack` feature for
>  >      relatively short period of time on the current F34 builders
>  >      (ie autopack=False).  This will happen soon in December 2021.
>  >      The change will naturally trigger the build failures, but a bit
>  >      earlier so users will be informed sooner ... with a _temporary_
>  >      work-around in hand => everyone can provide the file
>  >      <git-root>/rpkg.conf upstream, and opt-in the old behavior with with
>  >      content like:
>  > 
>  >      [rpkg]
>  >      # temporary work-around for rpkg-util v2, auto_pack is going to be
>  >      # removed soon! See:
>  >      # https://docs.pagure.org/copr.copr/rpkg_util_2_vs_3.html#rpkg-util-v3
>  >      auto_pack = True
>  > 
>  >   3. Migrate to Fedora 35, early in January 2022.  This will make the
>  >      `auto_pack = True` work-around non-working, and users will have to
>  >      adapt, again, see [1].
>  > 
>  > Of course, anyone is encouraged to migrate to the new syntax **right now**,
>  > sooner the better.  If done correctly, package can be built both with rpkg-util
>  > v2 and v3 [2].
>  > 
>  > Sorry for inconvenience!  Any comment, idea or help is welcome!
>  > 
>  > [1] https://docs.pagure.org/copr.copr/rpkg_util_2_vs_3.html#rpkg-util-v3
>  > [2]
> https://pagure.io/copr/copr-hello/c/739ff9910ee8a9c76d7e97de2f6176106dc19...
>  > 
>  > Pavel
>  > 
>  > 
>  
>

[Pavel Raiskup]

On Tuesday, January 25, 2022 5:16:31 PM CET Pavel Raiskup wrote:
>  Hello maintainers,
>  
>  seems like we are finally able to migrate to Fedora 35.  If all the
>  necessary test pass, we should be OK to migrate tomorrow, 2022-01-25. 
We couldn't finish testing today.  There are too many issues with the new
F35 qcow2 images (rpkg-util incompatibilities and spawning delays).

This is to be deployed as soon as possible (tomorrow hopefully), I will
notify here once we are done with this.

Pavel


>  This also means that we'll finally run against rpkg-util v3+ ==
> no way
>  back to auto_pack feature.
>  
>  Pavel
>  
>  On Friday, December 17, 2021 12:32:37 PM CET Pavel Raiskup wrote:
>  > Hello all,
>  > 
>  > just a quick update on this topic.  This took a bit more time than I
>  > originally expected, but I was able to test that configuration.
>  > 
>  > Please expect that step 2 happens on Monday, 2021-12-20 (I don't want to
>  > break things just when weekend starts).
>  > 
>  > For more info see the previous email, and we should be able to help you
>  > next week with the migration on #fedora-buildsys, when needed.
>  > 
>  > Also, the step 3 will be delayed at least by 20 days, so you still have the
>  > 1-month transition period.
>  > 
>  > Sorry for inconvenience,
>  > Pavel
>  > 
>  > On Friday, November 19, 2021 9:44:51 AM CET Pavel Raiskup wrote:
>  > > we are currently preparing Copr build system for migration to Fedora 35,
>  > > including the builder workers.
>  > > 
>  > > There's though, at least from the builder perspective, a major `rpkg-util`
>  > > behavior change [1] (removal of `auto_pack` feature) that many of the Copr
>  > > projects rely on.  This is a bit unexpected as `auto_pack = True` has been
>  > > the default in Copr for a long time, and the rpkg-util upstream intentions
>  > > behind the feature were generally misunderstood (even in Copr team).
>  > > 
>  > > That being said, some of the currently working builds on F34 would fail
>  > > after the migration to F35.
>  > > 
>  > > In order to make the migration a bit less dramatic, we plan to migrate in
>  > > the following steps:
>  > > 
>  > >   1. Document the feature drop and possible fixes/workarounds see [1], and
>  > >      properly announce the change in advance (here we are _now_).
>  > > 
>  > >   2. Change the default, and turn off the `auto_pack` feature for
>  > >      relatively short period of time on the current F34 builders
>  > >      (ie autopack=False).  This will happen soon in December 2021.
>  > >      The change will naturally trigger the build failures, but a bit
>  > >      earlier so users will be informed sooner ... with a _temporary_
>  > >      work-around in hand => everyone can provide the file
>  > >      <git-root>/rpkg.conf upstream, and opt-in the old behavior with
> with
>  > >      content like:
>  > > 
>  > >      [rpkg]
>  > >      # temporary work-around for rpkg-util v2, auto_pack is going to be
>  > >      # removed soon! See:
>  > >      # https://docs.pagure.org/copr.copr/rpkg_util_2_vs_3.html#rpkg-util-v3
>  > >      auto_pack = True
>  > > 
>  > >   3. Migrate to Fedora 35, early in January 2022.  This will make the
>  > >      `auto_pack = True` work-around non-working, and users will have to
>  > >      adapt, again, see [1].
>  > > 
>  > > Of course, anyone is encouraged to migrate to the new syntax **right now**,
>  > > sooner the better.  If done correctly, package can be built both with
> rpkg-util
>  > > v2 and v3 [2].
>  > > 
>  > > Sorry for inconvenience!  Any comment, idea or help is welcome!
>  > > 
>  > > [1] https://docs.pagure.org/copr.copr/rpkg_util_2_vs_3.html#rpkg-util-v3
>  > > [2]
> https://pagure.io/copr/copr-hello/c/739ff9910ee8a9c76d7e97de2f6176106dc19...
>  > > 
>  > > Pavel
>  > > 
>  > > 
>  > 
>  > 
>  
>

F35buildersdeployedtoFedoraCopr

[Pavel Raiskup]

Finally!  All the builders were moved to Fedora 35.

Test are passing on staging, so hopefully everything goes simiarly well in
production.

Happy building!
Pavell


On Wednesday, January 26, 2022 9:50:18 PM CET Pavel Raiskup wrote:
>  On Tuesday, January 25, 2022 5:16:31 PM CET Pavel Raiskup wrote:
>  > Hello maintainers,
>  > 
>  > seems like we are finally able to migrate to Fedora 35.  If all the
>  > necessary test pass, we should be OK to migrate tomorrow, 2022-01-25.
>  
>  We couldn't finish testing today.  There are too many issues with the new
>  F35 qcow2 images (rpkg-util incompatibilities and spawning delays).
>  
>  This is to be deployed as soon as possible (tomorrow hopefully), I will
>  notify here once we are done with this.
>  
>  Pavel
>  
>  
>  > This also means that we'll finally run against rpkg-util v3+ == no way
>  > back to auto_pack feature.
>  > 
>  > Pavel
>  > 
>  > On Friday, December 17, 2021 12:32:37 PM CET Pavel Raiskup wrote:
>  > > Hello all,
>  > > 
>  > > just a quick update on this topic.  This took a bit more time than I
>  > > originally expected, but I was able to test that configuration.
>  > > 
>  > > Please expect that step 2 happens on Monday, 2021-12-20 (I don't want to
>  > > break things just when weekend starts).
>  > > 
>  > > For more info see the previous email, and we should be able to help you
>  > > next week with the migration on #fedora-buildsys, when needed.
>  > > 
>  > > Also, the step 3 will be delayed at least by 20 days, so you still have the
>  > > 1-month transition period.
>  > > 
>  > > Sorry for inconvenience,
>  > > Pavel
>  > > 
>  > > On Friday, November 19, 2021 9:44:51 AM CET Pavel Raiskup wrote:
>  > > > we are currently preparing Copr build system for migration to Fedora 35,
>  > > > including the builder workers.
>  > > > 
>  > > > There's though, at least from the builder perspective, a major
> `rpkg-util`
>  > > > behavior change [1] (removal of `auto_pack` feature) that many of the
> Copr
>  > > > projects rely on.  This is a bit unexpected as `auto_pack = True` has
> been
>  > > > the default in Copr for a long time, and the rpkg-util upstream
> intentions
>  > > > behind the feature were generally misunderstood (even in Copr team).
>  > > > 
>  > > > That being said, some of the currently working builds on F34 would fail
>  > > > after the migration to F35.
>  > > > 
>  > > > In order to make the migration a bit less dramatic, we plan to migrate in
>  > > > the following steps:
>  > > > 
>  > > >   1. Document the feature drop and possible fixes/workarounds see [1],
> and
>  > > >      properly announce the change in advance (here we are _now_).
>  > > > 
>  > > >   2. Change the default, and turn off the `auto_pack` feature for
>  > > >      relatively short period of time on the current F34 builders
>  > > >      (ie autopack=False).  This will happen soon in December 2021.
>  > > >      The change will naturally trigger the build failures, but a bit
>  > > >      earlier so users will be informed sooner ... with a _temporary_
>  > > >      work-around in hand => everyone can provide the file
>  > > >      <git-root>/rpkg.conf upstream, and opt-in the old behavior with
> with
>  > > >      content like:
>  > > > 
>  > > >      [rpkg]
>  > > >      # temporary work-around for rpkg-util v2, auto_pack is going to be
>  > > >      # removed soon! See:
>  > > >      #
> https://docs.pagure.org/copr.copr/rpkg_util_2_vs_3.html#rpkg-util-v3
>  > > >      auto_pack = True
>  > > > 
>  > > >   3. Migrate to Fedora 35, early in January 2022.  This will make the
>  > > >      `auto_pack = True` work-around non-working, and users will have to
>  > > >      adapt, again, see [1].
>  > > > 
>  > > > Of course, anyone is encouraged to migrate to the new syntax **right
> now**,
>  > > > sooner the better.  If done correctly, package can be built both with
> rpkg-util
>  > > > v2 and v3 [2].
>  > > > 
>  > > > Sorry for inconvenience!  Any comment, idea or help is welcome!
>  > > > 
>  > > > [1] https://docs.pagure.org/copr.copr/rpkg_util_2_vs_3.html#rpkg-util-v3
>  > > > [2]
> https://pagure.io/copr/copr-hello/c/739ff9910ee8a9c76d7e97de2f6176106dc19...
>  > > > 
>  > > > Pavel
>  > > > 
>  > > > 
>  > > 
>  > > 
>  > 
>  > 
>  
>