A reminder: you cannot just "revert" package version bumps

[Adam Williamson]

nirik ran a script that checks for versioning issues in Rawhide today, 
and it found several: https://pagure.io/releng/issue/11922#comment-893797

Some of these followed a pattern, so I figured a reminder was in order. 
In all these cases, a new version was pushed to Rawhide, then "reverted" 
some time later:

golang-github-nats-io-jwt - bumped to 2.4.1 in July, reverted to 1.2.2 
in September
golang-google-grpc - bumped to 1.58.0 in September, reverted to 1.48.0 
in October; no 1.58.0 build ever landed, but the revert left the 
%release much lower than before
python-mizani - bumped to 0.10.0 on September 10, reverted to 0.9.3 on 
September 12
python-pywlroots - bumped to 0.16.6 on November 4, reverted to 0.16.4 
later the same day

so the reminder is this: you cannot simply "downgrade" RPM package 
versions like this. Especially not if the upgraded version ever made it 
into a Rawhide compose.

If a Rawhide user has your package installed, and got the upgraded 
version, they will be marooned on that build forever unless they 
manually run `dnf distro-sync`. A `dnf upgrade` will not downgrade the 
package to the build you now consider to be the "current" one.

If you have to downgrade a package that made it to a Rawhide compose, 
you must use an epoch. If the package did not have an epoch before, make 
it `Epoch: 1`. If it had an epoch already, bump it by 1. People tend not 
to like epochs, so *try* not to have to do this. Be really certain 
before pushing out version bumps.

If the "upgraded" build never made it into a compose (as is likely the 
case for python-pywlroots) you're *probably* okay, but it's still 
something to be careful about - for instance you might fall into the 
trap golang-google-grpc fell into with the %release tag.

Thanks folks!

[Sérgio Basto]

On Mon, 2024-01-29 at 15:43 -0800, Adam Williamson wrote:
> nirik ran a script that checks for versioning issues in Rawhide
> today, 
> and it found several:
> https://pagure.io/releng/issue/11922#comment-893797
>
> Some of these followed a pattern, so I figured a reminder was in
> order. 
> In all these cases, a new version was pushed to Rawhide, then
> "reverted" 
> some time later:
>
> golang-github-nats-io-jwt - bumped to 2.4.1 in July, reverted to
> 1.2.2 
> in September
> golang-google-grpc - bumped to 1.58.0 in September, reverted to
> 1.48.0 
> in October; no 1.58.0 build ever landed, but the revert left the 
> %release much lower than before
> python-mizani - bumped to 0.10.0 on September 10, reverted to 0.9.3
> on 
> September 12
> python-pywlroots - bumped to 0.16.6 on November 4, reverted to 0.16.4
> later the same day
>
> so the reminder is this: you cannot simply "downgrade" RPM package 
> versions like this. Especially not if the upgraded version ever made
> it 
> into a Rawhide compose.
>
> If a Rawhide user has your package installed, and got the upgraded 
> version, they will be marooned on that build forever unless they 
> manually run `dnf distro-sync`. A `dnf upgrade` will not downgrade
> the 
> package to the build you now consider to be the "current" one.

yes rawhide user should use dnf distro-sync not dnf upgrade

https://docs.fedoraproject.org/en-US/packaging-guidelines/Versioning/#_rawhi...

you may do a new build with lower EVR
> If you have to downgrade a package that made it to a Rawhide compose,
> you must use an epoch. If the package did not have an epoch before,
> make 
> it `Epoch: 1`. If it had an epoch already, bump it by 1. People tend
> not 
> to like epochs, so *try* not to have to do this. Be really certain 
> before pushing out version bumps.
>
> If the "upgraded" build never made it into a compose (as is likely
> the 
> case for python-pywlroots) you're *probably* okay, but it's still 
> something to be careful about - for instance you might fall into the 
> trap golang-google-grpc fell into with the %release tag.
> Thanks folks!
>
> Adam Williamson (he/him/his)
> Fedora QA
> Fedora Chat: @adamwill:fedora.im | Mastodon: @[e-mail]
> https://www.happyassassin.net
> --
> _______________________________________________
> devel mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines:
> https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]
> Do not reply to spam, report it:
> https://pagure.io/fedora-infrastructure/new_issue

[Fabio Valentini]

On Tue, Jan 30, 2024 at 1:00 AM Sérgio Basto  wrote:
>

(snip)
> yes rawhide user should use dnf distro-sync not dnf upgrade

It is better, yes, but it is not *required*.
> https://docs.fedoraproject.org/en-US/packaging-guidelines/Versioning/#_rawhi...

This is a completely different problem. Package updates that don't
build in Rawhide but build on stable branches are permissible as long
as the package update will be submitted to Rawhide *eventually*.
> you may do a new build with lower EVR

No, you may not. That is exactly what Adam's email is about.

Fabio

[Kevin Kofler]

Sérgio Basto wrote:
> yes rawhide user should use dnf distro-sync not dnf upgrade

+1. Rawhide EVRs should be allowed to go backwards, that is an integral part 
of being a development branch.

Kevin Kofler

[kevin]

On Tue, Jan 30, 2024 at 01:19:18AM +0100, Kevin Kofler via devel wrote:
> Sérgio Basto wrote:
> > yes rawhide user should use dnf distro-sync not dnf upgrade
>
> +1. Rawhide EVRs should be allowed to go backwards, that is an integral part 
> of being a development branch.

distro-sync is nice and all, but it's not a silver bullet.
In cases of simple packages a downgrade may not break anything, but in
cases where other things already built upon it, where the new one
changed conguration or interface, or even where the upgrade changed
data, it can leave things in a pretty unfortunate state.

rawhide is a place for people to integrate their upsteam work with
the other working collection of packages. Of course packages upgrade
all the time and you have to adjust and fix your package to continue
to work with them. But if packages could also just downgrade all
the time it would make things much more a 'shifting sands'.

We did recently change this so that releng could untag packages that
went out already if it was judged to be a serious enough matter.

kevin

[Kevin Kofler]

kevin wrote:
> distro-sync is nice and all, but it's not a silver bullet.
> In cases of simple packages a downgrade may not break anything, but in
> cases where other things already built upon it, where the new one
> changed conguration or interface, or even where the upgrade changed
> data, it can leave things in a pretty unfortunate state.

And the proposed "solution" of bumping Epoch fixes none of that. It just 
introduces an Epoch that we will be stuck with forever. It will not 
magically make the downgrade safe in any of the 3 situations you describe.

Kevin Kofler

[Gary Buhrmaster]

On Thu, Feb 1, 2024 at 1:53 AM Kevin Kofler via devel
 wrote:
> And the proposed "solution" of bumping Epoch fixes none of that. It just
> introduces an Epoch that we will be stuck with forever. It will not
> magically make the downgrade safe in any of the 3 situations you describe.

While I don't like epochs, there is nothing intrinsically
wrong with an epoch bump when a packager
determines that they need to downgrade because
the testing for the upgrade was insufficient or
inadequately performed and the packager found
that there was no way forward with fixes to the
new versions (either from upstream, or by the
packager).

Sometimes the packager (or upstream) screws up,
and  the epoch bump is the "get out of  jail (mostly)
free card" for the packager.

If you don't want a "get out of jail (mostly) free
card", more power to you, for you are committing
to fix any/all issues.  Sometimes not every Fedora
packager has commit access to the upstream
sources.

[Kevin Kofler]

Gary Buhrmaster wrote:
> While I don't like epochs, there is nothing intrinsically
> wrong with an epoch bump when a packager
> determines that they need to downgrade because
> the testing for the upgrade was insufficient or
> inadequately performed and the packager found
> that there was no way forward with fixes to the
> new versions (either from upstream, or by the
> packager).

There are plenty of valid reasons to bump an Epoch. IMHO, reverting a 
Rawhide-only version bump that never made it to a stable release is not. I 
do not see why it cannot just be reverted.

Actually, the downgrade masquerading as an "upgrade" (due to the Epoch) only 
makes it more likely that any issues related to the downgrade (such as the 
ones the other Kevin, Kevin Fenzi, pointed out) will catch the user by 
surprise. If the distro-sync (which should be the default way to do updates 
at least on Rawhide, if not everywhere) mentions a package being downgraded, 
that is much more likely to be noticed.

Kevin Kofler

[Gary Buhrmaster]

On Thu, Feb 1, 2024 at 3:11 AM Kevin Kofler via devel
 wrote:
> If the distro-sync (which should be the default way to do updates
> at least on Rawhide, if not everywhere) mentions a package being downgraded,
> that is much more likely to be noticed.

I look forward to your formal change proposal
to replace what we know of today as upgrade
to be distro-sync.

While I will reserve judgement on the proposal
until I see the full details, I am going to say
that as today I am dubious that that is the right
way forward.

[Kevin Fenzi]

On Thu, Feb 01, 2024 at 02:51:54AM +0100, Kevin Kofler via devel wrote:
> kevin wrote:
> > distro-sync is nice and all, but it's not a silver bullet.
> > In cases of simple packages a downgrade may not break anything, but in
> > cases where other things already built upon it, where the new one
> > changed conguration or interface, or even where the upgrade changed
> > data, it can leave things in a pretty unfortunate state.
>
> And the proposed "solution" of bumping Epoch fixes none of that. It just 
> introduces an Epoch that we will be stuck with forever. It will not 
> magically make the downgrade safe in any of the 3 situations you describe.

I am unsure when I proposed Epoch's. I'm not a great fan of them either.
In addition to what you mentioned, Epochs have another problem:
Depending on how dependent packages (build)require your package, they
must be adjusted for the new Epoch too.

Anyhow, to be more clear:

I don't think we can or should say "just downgrade whenever you like",
unless/until dnf5 gets rid of update and only has distro-sync.
Nor do I think we should rush to using Epochs. In rare cases we should
go back to older versions, but it should be a discussion and other
alternatives should all be exhausted first (patch the problem and push a
newer update, push a revert of the problematic part, engage with
upstream for a solution, etc).

kevin

[Adam Williamson]

On 2024-01-29 16:00, Sérgio Basto wrote:
> https://docs.fedoraproject.org/en-US/packaging-guidelines/Versioning/#_rawhi...
>
> you may do a new build with lower EVR

That is not what that guideline says. It says the Rawhide build can be 
lower-versioned than a current build from *a different branch* 
*temporarily*. It says nothing about allowing a new Rawhide build to be 
lower versioned than the previous one.

[Sérgio Basto]

On Mon, 2024-01-29 at 16:33 -0800, Adam Williamson wrote:
> On 2024-01-29 16:00, Sérgio Basto wrote:
> > https://docs.fedoraproject.org/en-US/packaging-guidelines/Versioning/#_rawhi...
> >
> > you may do a new build with lower EVR
>
> That is not what that guideline says. It says the Rawhide build can
> be 
> lower-versioned than a current build from *a different branch* 
> *temporarily*. It says nothing about allowing a new Rawhide build to
> be 
> lower versioned than the previous one.

that is what happened

[Daniel P. Berrangé]

On Mon, Jan 29, 2024 at 03:43:39PM -0800, Adam Williamson wrote:
> nirik ran a script that checks for versioning issues in Rawhide today, and
> it found several: https://pagure.io/releng/issue/11922#comment-893797
>
> Some of these followed a pattern, so I figured a reminder was in order. In
> all these cases, a new version was pushed to Rawhide, then "reverted" some
> time later:
>
> golang-github-nats-io-jwt - bumped to 2.4.1 in July, reverted to 1.2.2 in
> September
> golang-google-grpc - bumped to 1.58.0 in September, reverted to 1.48.0 in
> October; no 1.58.0 build ever landed, but the revert left the %release much
> lower than before
> python-mizani - bumped to 0.10.0 on September 10, reverted to 0.9.3 on
> September 12
> python-pywlroots - bumped to 0.16.6 on November 4, reverted to 0.16.4 later
> the same day
>
> so the reminder is this: you cannot simply "downgrade" RPM package versions
> like this. Especially not if the upgraded version ever made it into a
> Rawhide compose.

This is the kind of rule that is a prime target for automation. Can we
have Fedora Rawhide gating validate that the NEVR doesn't go backwards,
and block bad builds from getting into the compose.

With regards,
Daniel

[kevin]

On Tue, Jan 30, 2024 at 08:08:54AM +0000, Daniel P. Berrangé wrote:
> On Mon, Jan 29, 2024 at 03:43:39PM -0800, Adam Williamson wrote:
> > nirik ran a script that checks for versioning issues in Rawhide today, and
> > it found several: https://pagure.io/releng/issue/11922#comment-893797
> >
> > Some of these followed a pattern, so I figured a reminder was in order. In
> > all these cases, a new version was pushed to Rawhide, then "reverted" some
> > time later:
> >
> > golang-github-nats-io-jwt - bumped to 2.4.1 in July, reverted to 1.2.2 in
> > September
> > golang-google-grpc - bumped to 1.58.0 in September, reverted to 1.48.0 in
> > October; no 1.58.0 build ever landed, but the revert left the %release much
> > lower than before
> > python-mizani - bumped to 0.10.0 on September 10, reverted to 0.9.3 on
> > September 12
> > python-pywlroots - bumped to 0.16.6 on November 4, reverted to 0.16.4 later
> > the same day
> >
> > so the reminder is this: you cannot simply "downgrade" RPM package versions
> > like this. Especially not if the upgraded version ever made it into a
> > Rawhide compose.
>
> This is the kind of rule that is a prime target for automation. Can we
> have Fedora Rawhide gating validate that the NEVR doesn't go backwards,
> and block bad builds from getting into the compose.

Yeah, seems like it could be a ci test... of course there may be times
when it needs to be waived, but we could do that then with full
understanding.

kevin