Direct to stable updates

[Bojan Smojver]

Quick question about direct to stable updates in bodhi, such as FF
106.0.4 and kernel 6.0.7 that are lined up for F37 right now. Such
updates often end up being in nowhere land for quite some time, because
they skip testing to go to stable directly, but the push to stable
cannot happen for whatever reason.

Would it not be better to get such update to testing in this scenario,
given stable is something that cannot be touched? In other words,
should the algorithm for pushes be changed during such times?

-- 
Bojan

[Adam Williamson]

On Tue, 2022-11-08 at 07:52 +1100, Bojan Smojver via devel wrote:
>  Quick question about direct to stable updates in bodhi, such as FF
>  106.0.4 and kernel 6.0.7 that are lined up for F37 right now. Such
>  updates often end up being in nowhere land for quite some time, because
>  they skip testing to go to stable directly, but the push to stable
>  cannot happen for whatever reason.
>  
>  Would it not be better to get such update to testing in this scenario,
>  given stable is something that cannot be touched? In other words,
>  should the algorithm for pushes be changed during such times? 
Yeah, this is a good point, though I'm not sure how easy it is to fix.
I've been leaving those updates in -testing for a while in order to try
and be really sure they're OK before we push them stable - we were
reserving the option of shipping F37 with earlier version of the
kernel, GNOME and Firefox in case the newer versions had problems. But
if the updates never make it to -testing, that does restrict the
testing they get.

I just noticed another consequence of this, too: they don't go into the
Silverblue (and Kinoite etc.) testing builds. So if you're running one
of those rpm-ostree bases, you can't really get them. I'm on kernel
6.0.6 instead of 6.0.7 for this reason, as that's the most recent
kernel build that actually made it to updates-testing.
-- 
Adam Williamson
Fedora QA
IRC: adamw | Twitter: adamw_ha
https://www.happyassassin.net

[Bojan Smojver]

Maybe push them to both, if they've never been to testing? In other
words, never skip testing.

Sure, there will be some duplication of packages for a cycle or two,
but eventually, they anything that's already in stable will be kicked
out of testing, right?

-- 
Bojan

[Kevin Kofler]

Adam Williamson wrote:
>  Yeah, this is a good point, though I'm not sure how easy it is to
> fix. 
if (state == pending && request == stable && !can_push(stable))
  push(testing);
else
  push(request);

        Kevin Kofler

[Mattia Verga]

The problem is
>  !can_push(stable))
>  
with the current workflow, Bodhi doesn't know when a release is freezed.
There is support for a "Freeze" state, but it was never used.

Mattia

[Kevin Kofler]

Mattia Verga via devel wrote:
>  with the current workflow, Bodhi doesn't know when a release is
> freezed.
>  There is support for a "Freeze" state, but it was never used. 
How do we prevent then that pushes to stable actually move forward? If rel-
eng just hits a different button / runs a different script to push testing 
only instead of both testing and stable, that is the "can we push to 
stable?" property Bodhi needs to check.

        Kevin Kofler

[Kevin Kofler]

Kevin Kofler via devel wrote:

>  Mattia Verga via devel wrote:
> > with the current workflow, Bodhi doesn't know when a release is freezed.
> > There is support for a "Freeze" state, but it was never used.
>  
>  How do we prevent then that pushes to stable actually move forward? If
>  rel- eng just hits a different button / runs a different script to push
>  testing only instead of both testing and stable, that is the "can we push
>  to stable?" property Bodhi needs to check. 
PS: The "worst mistake" that can happen then is that if we push only testing 
to a non-frozen release for whatever reason, the update will be included in 
that testing push, and then move forward to stable in the next stable push. 
I do not see this as a real issue.

        Kevin Kofler

[Mattia Verga]

Il 10/11/22 01:58, Kevin Kofler via devel ha scritto:
>  Kevin Kofler via devel wrote:
>
> > Mattia Verga via devel wrote:
> >> with the current workflow, Bodhi doesn't know when a release is freezed.
> >> There is support for a "Freeze" state, but it was never used.
> > How do we prevent then that pushes to stable actually move forward? If
> > rel- eng just hits a different button / runs a different script to push
> > testing only instead of both testing and stable, that is the "can we push
> > to stable?" property Bodhi needs to check.
>  PS: The "worst mistake" that can happen then is that if we push only testing
>  to a non-frozen release for whatever reason, the update will be included in
>  that testing push, and then move forward to stable in the next stable push.
>  I do not see this as a real issue.
>  
I'm working on fixing some bits in Bodhi before proposing to releng the
use of the 'frozen' release state. That will enable Bodhi to avoid
pushing updates directly to stable if the release is frozen, as well as
some small tweaks that were requested and would make life easier for
releng folks. It shouldn't be too hard, it's just that Bodhi code is
sometimes so contorted that by making a simple change it's easy to break
something else... moving updates from one state to another and tagging
builds in the correct way without losing the right track is one of those
contorted part.

Mattia

[Kevin Fenzi]

On Thu, Nov 10, 2022 at 05:16:44PM +0000, Mattia Verga via devel wrote:
>  Il 10/11/22 01:58, Kevin Kofler via devel ha scritto:
>  > Kevin Kofler via devel wrote:
>  >
>  >> Mattia Verga via devel wrote:
>  >>> with the current workflow, Bodhi doesn't know when a release is
> freezed.
>  >>> There is support for a "Freeze" state, but it was never used.
>  >> How do we prevent then that pushes to stable actually move forward? If
>  >> rel- eng just hits a different button / runs a different script to push
>  >> testing only instead of both testing and stable, that is the "can we push
>  >> to stable?" property Bodhi needs to check.
>  > PS: The "worst mistake" that can happen then is that if we push only
> testing
>  > to a non-frozen release for whatever reason, the update will be included in
>  > that testing push, and then move forward to stable in the next stable push.
>  > I do not see this as a real issue.
>  >
>  I'm working on fixing some bits in Bodhi before proposing to releng the
>  use of the 'frozen' release state. That will enable Bodhi to avoid
>  pushing updates directly to stable if the release is frozen, as well as
>  some small tweaks that were requested and would make life easier for
>  releng folks. 
Thanks!

To explain a bit to everyone the current workflow: 

* In non freeze times, we have a cron job that pushes "all pending
updates" at 00:14 UTC every day. This is stable and testing for anything
thats pending moving to a new state. 

* In freezes however, we have to disable the cron job and manually: 
- First push branched version updates-testing by itself.
- Then push everything else (minus the branched version). 

This sucks, because it's manual, there's a hour or so wait for
updates-testing to finish before we can do the rest, and you have to
remember to list: 
--releases
EPEL-9N,EPEL-7,EPEL-8,F36,F36C,EPEL-8M,EPEL-9,EPEL-8N,F35,F35M,F35C,F36M,F35F,F36F

If we could just mark branched frozen and have it not push stable there
(without specific builds, because we do still need to push stable
requests through the freeze) that would be great. We could also actually
note this in updates so people don't get confused why their update
didn't get pushed stable.

>  It shouldn't be too hard, it's just that Bodhi code is
>  sometimes so contorted that by making a simple change it's easy to break
>  something else... moving updates from one state to another and tagging
>  builds in the correct way without losing the right track is one of those
>  contorted part. 
Yeah. ;( 

Thanks again for working on it!

kevin

[Neal Gompa]

On Thu, Nov 10, 2022 at 3:49 PM Kevin Fenzi <kevin(a)scrye.com&gt; wrote:
>  On Thu, Nov 10, 2022 at 05:16:44PM +0000, Mattia Verga via devel wrote:
>  > It shouldn't be too hard, it's just that Bodhi code is
>  > sometimes so contorted that by making a simple change it's easy to break
>  > something else... moving updates from one state to another and tagging
>  > builds in the correct way without losing the right track is one of those
>  > contorted part.
>
>  Yeah. ;(
>
>  Thanks again for working on it!
>  
I sympathize greatly here. It was a pain to wire up "logout" to the
"relogin" property in updateinfo (the field had been in bodhi for a
decade and nobody wired it up to the appropriate rpm metadata field!).
Bodhi is an unusually difficult codebase for what it does.



-- 
真実はいつも一つ！/ Always, there's only one truth!

[Stephen Smoogen]

On Thu, Nov 10, 2022 at 18:55 Neal Gompa <ngompa13(a)gmail.com&gt; wrote:

>  On Thu, Nov 10, 2022 at 3:49 PM Kevin Fenzi <kevin(a)scrye.com&gt;
> wrote:
>  >
>  > On Thu, Nov 10, 2022 at 05:16:44PM +0000, Mattia Verga via devel wrote:
>  > > It shouldn't be too hard, it's just that Bodhi code is
>  > > sometimes so contorted that by making a simple change it's easy to
>  break
>  > > something else... moving updates from one state to another and tagging
>  > > builds in the correct way without losing the right track is one of
>  those
>  > > contorted part.
>  >
>  > Yeah. ;(
>  >
>  > Thanks again for working on it!
>  >
>
>  I sympathize greatly here. It was a pain to wire up "logout" to the
>  "relogin" property in updateinfo (the field had been in bodhi for a
>  decade and nobody wired it up to the appropriate rpm metadata field!).
>  Bodhi is an unusually difficult codebase for what it does.
>
>  
From what I can tell is that every time someone says that and then tries to
fix it they find about 20 reasons why the code is not complex enough for
all the corner cases and “obvious” requirements that people expect of it




>  --
>  真実はいつも一つ！/ Always, there's only one truth!
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct:
>  https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
>  https://lists.fedoraproject.org/archives/list/[e-mail]
>  Do not reply to spam, report it:
>  https://pagure.io/fedora-infrastructure/new_issue
>  
-- 
Stephen Smoogen, Red Hat Automotive
Let us be kind to one another, for most of us are fighting a hard battle.
-- Ian MacClaren

[Gary Buhrmaster]

On Fri, Nov 11, 2022 at 12:52 AM Stephen Smoogen <ssmoogen(a)redhat.com&gt; wrote:
>  On Thu, Nov 10, 2022 at 18:55 Neal Gompa <ngompa13(a)gmail.com&gt; wrote:
> >
> > I sympathize greatly here. It was a pain to wire up "logout" to the
> > "relogin" property in updateinfo (the field had been in bodhi for a
> > decade and nobody wired it up to the appropriate rpm metadata field!).
> > Bodhi is an unusually difficult codebase for what it does.
> >
>
>  From what I can tell is that every time someone says that and then tries to fix it they
> find about 20 reasons why the code is not complex enough for all the corner cases and
> “obvious” requirements that people expect of it
>  
The code is both too complex, and nowhere near complex
enough.

I really dislike code like that.

[Demi Marie Obenour]

On 11/10/22 21:02, Gary Buhrmaster wrote:
>  On Fri, Nov 11, 2022 at 12:52 AM Stephen Smoogen
> <ssmoogen(a)redhat.com&gt; wrote:
> > On Thu, Nov 10, 2022 at 18:55 Neal Gompa <ngompa13(a)gmail.com&gt; wrote:
> >>
> >> I sympathize greatly here. It was a pain to wire up "logout" to the
> >> "relogin" property in updateinfo (the field had been in bodhi for a
> >> decade and nobody wired it up to the appropriate rpm metadata field!).
> >> Bodhi is an unusually difficult codebase for what it does.
> >>
> >
> > From what I can tell is that every time someone says that and then tries to fix it
> they find about 20 reasons why the code is not complex enough for all the corner cases and
> “obvious” requirements that people expect of it
> >
>  
>  The code is both too complex, and nowhere near complex
>  enough.
>  
>  I really dislike code like that. 
Time for some serious refactoring?
-- 
Sincerely,
Demi Marie Obenour (she/her/hers)

[Stephen Smoogen]

On Fri, 11 Nov 2022 at 02:34, Demi Marie Obenour <demiobenour(a)gmail.com&gt;
wrote:

>  On 11/10/22 21:02, Gary Buhrmaster wrote:
>  > On Fri, Nov 11, 2022 at 12:52 AM Stephen Smoogen <ssmoogen(a)redhat.com&gt;
>  wrote:
>  >> On Thu, Nov 10, 2022 at 18:55 Neal Gompa <ngompa13(a)gmail.com&gt; wrote:
>  >>>
>  >>> I sympathize greatly here. It was a pain to wire up "logout" to
> the
>  >>> "relogin" property in updateinfo (the field had been in bodhi for
> a
>  >>> decade and nobody wired it up to the appropriate rpm metadata field!).
>  >>> Bodhi is an unusually difficult codebase for what it does.
>  >>>
>  >>
>  >> From what I can tell is that every time someone says that and then
>  tries to fix it they find about 20 reasons why the code is not complex
>  enough for all the corner cases and “obvious” requirements that people
>  expect of it
>  >>
>  >
>  > The code is both too complex, and nowhere near complex
>  > enough.
>  >
>  > I really dislike code like that.
>  Time for some serious refactoring?
>
>  
You can only refactor it when you have a steady set of requirements. The
code has been 'refactored' at least 4 times but what happens is that you
will get into about 1/3rd of the way into it and find you have now to add a
bunch of new requirements. You will still have to keep the old ones working
also because of older releases. OK you have gotten to adding that and you
find that the business logic has a major flaw in it that FPC and FESCO need
to explain when they said 'SHOULD' in something, did they want it enforced
in Bodhi or not.. oh wait it turns out they both disagree with each other.
OK go do some other refactoring while that works out. Oh look another
conflict and you now have to refactor in that the message bus is changed
AND now you need to add in containers in 3 different formats made by
different tools. And look 3 new tools need to be added into the logic. And
two parts of the things you were told you had to call out to no longer
exist..

And crap we are in freeze so nothing changes except fixing the little bugs
you can. Freeze is over, deploy the stage and find out that production and
stage don't match well enough because there isn't enough resources in
people, time and servers to do a 1:1 stage/product environment. Crap crap
crap.. need to get it all working before the mass rebuild.. put off all the
refactoring for another month. Oh new requirements to add in.

Repeat every 3-6 months until you are reassigned to a different project.


-- 
Stephen Smoogen, Red Hat Automotive
Let us be kind to one another, for most of us are fighting a hard battle.
-- Ian MacClaren

[Kevin Kofler]

Stephen Smoogen wrote:
>  You can only refactor it when you have a steady set of requirements.
> The
>  code has been 'refactored' at least 4 times but what happens is that you
>  will get into about 1/3rd of the way into it and find you have now to add
>  a bunch of new requirements. 
Sounds like pretty much what I had guessed. ;-)

So I think it would really help if Bodhi were to become more of an enabling 
tool and less of an enforcing tool again. Package maintainers have this 
wonderful organ between their ears that allows them to know better what is 
best for their packages than some piece of software, no matter how much 
complexity we force that software's maintainers to add to the latter. :-)

        Kevin Kofler

[Stephen Smoogen]

On Fri, 11 Nov 2022 at 10:19, Kevin Kofler via devel <
devel(a)lists.fedoraproject.org&gt; wrote:

>  Stephen Smoogen wrote:
>  > You can only refactor it when you have a steady set of requirements. The
>  > code has been 'refactored' at least 4 times but what happens is that you
>  > will get into about 1/3rd of the way into it and find you have now to add
>  > a bunch of new requirements.
>
>  Sounds like pretty much what I had guessed. ;-)
>
>  So I think it would really help if Bodhi were to become more of an
>  enabling
>  tool and less of an enforcing tool again. Package maintainers have this
>  wonderful organ between their ears that allows them to know better what is
>  best for their packages than some piece of software, no matter how much
>  complexity we force that software's maintainers to add to the latter. :-)
>
>  
Pretty much every one of those bodhi requirements is because either

* a developer did not use that wonderful organ for some reason, and people
said 'that should never happen again.'
* what the developer decided was not liked by other developers enough that
it was decided 'that should never happen again.'

Look back on the 15-20 years of fedora devel emails and see how many times
someone has said that something should never happen. Guess what? Enough
other developers agreed at times, and decided it needed to be automated
because the other big old complaint was about how long it took for people
to review things and how prone to failure was also true.




-- 
Stephen Smoogen, Red Hat Automotive
Let us be kind to one another, for most of us are fighting a hard battle.
-- Ian MacClaren

[Kevin Kofler]

Stephen Smoogen wrote:
>  Pretty much every one of those bodhi requirements is because either
>  
>  * a developer did not use that wonderful organ for some reason, and people
>  said 'that should never happen again.'
>  * what the developer decided was not liked by other developers enough that
>  it was decided 'that should never happen again.'
>  
>  Look back on the 15-20 years of fedora devel emails and see how many times
>  someone has said that something should never happen. Guess what? Enough
>  other developers agreed at times, and decided it needed to be automated
>  because the other big old complaint was about how long it took for people
>  to review things and how prone to failure was also true. 
Whenever something "bad" happens, people are always quick to jump to the 
conclusion that we need a law or rule banning the "bad" thing, no matter 
whether a rule is actually a workable way to prevent it. So we end up with 
overreaching laws banning even common activities, or with law texts so 
complex that everyone agrees they need to be simplified.

In Fedora, we have had a handful bad updates making it to stable due to 
questionable decisions by some maintainers, and instead of simply telling 
those people to be more careful, we instead turned pushing Fedora updates 
into a bureaucracy that just keeps getting worse and worse when invariably 
bad updates keep slipping through because the complexity actually 
discourages good practice instead of encouraging it. (E.g., many maintainers 
enable automatic pushes because they need to wait so long to be allowed to 
push an update to stable that they would forget to push it manually. But 
automatic pushes are the most common source of bad updates making it 
through, and also for issues such as broken upgrade paths between releases 
(because one release happened to get karma sooner than the other).) So of 
course the answer is that we need even stricter rules, because the 
alternative would mean to admit a mistake and step back, which nobody seems 
to be willing to do.

The original trigger for the update policy enforcement was actually an 
update that broke the bind DNS server, a package that ~99% of Fedora users 
do not even have installed. The response has always been a complete 
overreaction.

        Kevin Kofler

[Gary Buhrmaster]

On Sat, Nov 12, 2022 at 7:19 PM Kevin Kofler via devel
<devel(a)lists.fedoraproject.org&gt; wrote:

>  (E.g., many maintainers
>  enable automatic pushes because they need to wait so long to be allowed to
>  push an update to stable that they would forget to push it manually. But
>  automatic pushes are the most common source of bad updates making it
>  through, and also for issues such as broken upgrade paths between releases
>  (because one release happened to get karma sooner than the other).) 
Interesting.  I have never seen such analysis results
shared (either the part about why maintainers do it,
or the pushes being the common causes of bad
updates), and of course, anecdotal experience does
not lead to a supportable conclusion.  Where was
that analysis published so I can read it?

[Kevin Kofler]

Gary Buhrmaster wrote:
>  Interesting.  I have never seen such analysis results
>  shared (either the part about why maintainers do it,
>  or the pushes being the common causes of bad
>  updates), and of course, anecdotal experience does
>  not lead to a supportable conclusion.  Where was
>  that analysis published so I can read it? 
It is empirical evidence only. (When something bad was pushed to stable, I 
looked at how it happened, and more often than not, it was autokarma.) There 
is not much of an analysis that can be done because Bodhi does not retain 
historical data.

        Kevin Kofler

[Kevin Fenzi]

For a few years I was keeping track of updates that caused big problems:

https://fedoraproject.org/wiki/Updates_Lessons

The orig "very bad" update was a dbus update that broke everything. 

In any case I have seen our current updates system working and blocking
tons of harmfull updates over the years. I think direct to stable is a
bad idea. 

kevin

[Kevin Kofler]

Kevin Fenzi wrote:
>  For a few years I was keeping track of updates that caused big
> problems:
>  
>  https://fedoraproject.org/wiki/Updates_Lessons 
That is also very anecdotal evidence though. And with only one exception 
(the broken dependency in celt), the updates in the above list above are 
all:
* either updates that have made it through to stable despite being broken, 
in all cases since March 2010 (i.e., after the dnssec-conf one that was the 
trigger for the main Bodhi crackdown) despite (or even because of) the new 
update policies, (In other words, the strict rules have NOT prevented the 
breakage in stable. At least in one case (the 2014-03 one), you have 
explicitly documented autokarma as the cause for the breakage: 
"User:Adamwill noticed several cases where updates had been submitted to 
stable despite a valid AutoQA test failure, usually not via a manual push 
but via karma automatism.")
* or critical updates that have not made it through to stable in a timely 
manner, usually because of the karma requirements. (In other words, those 
issues were directly CAUSED by the strict policies!)

And this list is by no means complete. I remember way more bad updates and 
delayed updates (sometimes even together: bad update gets through instantly 
through autokarma, the regression fix is stuck in testing for days, because 
the testers just did not care about and/or were unable to test the regressed 
use case), too many to write down a list like this (and indeed, I have no 
such list). Your list has only the worst failures of the bureaucratic update 
policy.

>  The orig "very bad" update was a dbus update that broke
> everything. 
That was one of the issues that had triggered a perceived demand for 
stricter rules, but the Bodhi crackdown was actually implemented only 
starting from March 2010, after the dnssec-conf update (which was pushed 
directly to stable, as was still allowed in February 2010).

The D-Bus one actually affected a large range of users, but it had a trivial 
workaround (update from the CLI, i.e.: su -c "yum update" – yum because DNF 
was not a thing at the time). The dnssec-conf one affected only a very small 
percentage of Fedora users, because most users do not run their own DNS 
server software. (Not even most domain owners, because name servers are 
typically provided by the registrar.) None was as catastrophic a failure as 
the mob for stricter rules had painted them.

>  In any case I have seen our current updates system working and
> blocking
>  tons of harmfull updates over the years. 
But you do not have a list proving that. (The list you linked to is not it, 
because I see only exactly one entry in it where a bad update did not make 
it beyond testing, the 2010-07-02 celt one.) And even if you had one, it 
would not prove that the maintainers would not have used updates-testing the 
intended way without being forced to by software.

What I have seen (and no, I do not have a list either) is updates making it 
through (due to karma or because the 7 or 14 days just ran out without 
anybody bothering to test them) to stable causing a bad regression, and then 
the regression fix needlessly sitting in testing for up to two weeks, 
instead of being pushed directly to stable to undo the breakage. Not only 
does that delay the fix for those users unlucky enough to get the bad 
update, but it also largely increases the number of affected users, because 
many users do not update daily and might not have noticed the regression if 
the fix had been pushed the next day rather than 1-2 weeks later (window of 
exposure). In some cases, I was the maintainer that was unable to push the 
regression fix out any sooner due to the rules, so I definitely know that 
the rules are to blame. (And before the rules were enforced, I had always 
used direct stable pushes to fix regressions, leading to happy users.)

Due to the window of exposure effect, 7 regressions slipping through, each 
fixed the next day with a direct stable push, are not necessarily worse than 
1 regression slipping through (the other 6 having been prevented by the 
rules), fixed only 7 days later due to the update rules. And in practice, 
the testing is far from catching 6 bad updates out of 7. A lot slips 
through, because testers are unable to test, e.g., library packages 
exhaustively.

>  I think direct to stable is a bad idea. 
I think it is a good idea, under some conditions (critical regression or 
security fix, entirely new package, or package that was previously 
uninstallable or unusable). When stable is actually open for pushes.

Of course it does not work during a freeze, which (together with the fact 
that karma thresholds can be reached even before the updates reach testing, 
leading to a special case in which direct stable pushes are actually still 
possible after all) is the issue that started this thread. But that should 
be solved the way I already mentioned (automatically divert the push to 
testing instead, but keep the update queued for stable so it goes out as a 
0-day update as soon as the freeze lifts).

        Kevin Kofler

[Kevin Fenzi]

On Mon, Nov 14, 2022 at 11:54:32PM +0100, Kevin Kofler via devel wrote:
...snip...

I think we are going to just have to agree to disagree here. 

I think we have had this discussion a number of times now and aren't
going to convince the other. 

kevin

[Kevin Kofler]

Kevin Fenzi wrote:
>  I think we are going to just have to agree to disagree here.
>  
>  I think we have had this discussion a number of times now and aren't
>  going to convince the other. 
So Bodhi will continue to become more and more unmaintainable due to piling 
up more and more complicated rules that it needs to enforce, and obvious 
defects such as the one that started this thread will never get addressed.

It is really sad that you are not willing to open your eyes and see the 
amount of damage that this dead-end policy has caused and is still causing.

        Kevin Kofler

[Stephen Smoogen]

On Tue, 15 Nov 2022 at 16:04, Kevin Kofler via devel <
devel(a)lists.fedoraproject.org&gt; wrote:

>  Kevin Fenzi wrote:
>  > I think we are going to just have to agree to disagree here.
>  >
>  > I think we have had this discussion a number of times now and aren't
>  > going to convince the other.
>
>  So Bodhi will continue to become more and more unmaintainable due to
>  piling
>  up more and more complicated rules that it needs to enforce, and obvious
>  defects such as the one that started this thread will never get addressed.
>
>  It is really sad that you are not willing to open your eyes and see the
>  amount of damage that this dead-end policy has caused and is still causing.
>
>  
Could you possibly pick the fight with the right people for once? It
doesn't matter if Kevin Fenzi agreed with you because he isn't the person
who a) writes the guidelines which were asked to be implemented or b) works
on bodhi codebase in any way. He just makes sure it and the other 240
services runs and that the builds generally flow. That already takes up
about 60 hours of his work week.

If you have problems with this please bring it up with FESCO and the Fedora
Packaging Committee and probably the Council. They are the ones who over
time have listened to packagers, users, and developers about what was
expected from the build system and they are the ones who have given general
guidance over the last 10+ years of bodhi development.



-- 
Stephen Smoogen, Red Hat Automotive
Let us be kind to one another, for most of us are fighting a hard battle.
-- Ian MacClaren

[Kevin Kofler]

Stephen Smoogen wrote:
>  On Tue, 15 Nov 2022 at 16:04, Kevin Kofler via devel <
>  devel(a)lists.fedoraproject.org&gt; wrote:
>  
> > Kevin Fenzi wrote:
> > > I think we are going to just have to agree to disagree here.
> > >
> > > I think we have had this discussion a number of times now and aren't
> > > going to convince the other.
> >
> > So Bodhi will continue to become more and more unmaintainable due to
> > piling
> > up more and more complicated rules that it needs to enforce, and obvious
> > defects such as the one that started this thread will never get
> > addressed.
> >
> > It is really sad that you are not willing to open your eyes and see the
> > amount of damage that this dead-end policy has caused and is still
> > causing.
> >
> >
>  Could you possibly pick the fight with the right people for once? It
>  doesn't matter if Kevin Fenzi agreed with you because he isn't the person
>  who a) writes the guidelines which were asked to be implemented or b)
>  works on bodhi codebase in any way. He just makes sure it and the other
>  240 services runs and that the builds generally flow. That already takes
>  up about 60 hours of his work week.
>  
>  If you have problems with this please bring it up with FESCO and the
>  Fedora Packaging Committee and probably the Council. 
Kevin Fenzi is currently a member of FESCo (see 
https://docs.fedoraproject.org/en-US/fesco/) and has been in that role for 
years. So pushing the blame off to "someone else" is not going to work.

And I have brought this issue to FESCo (which is where most of the update 
policies came from, not FPC or the Council) more than once. Usually, it was 
not even brought to a vote. And whenever there was a vote about the issue, 
it was always in favor of either the status quo or even stricter rules.

>  They are the ones who over time have listened to packagers, users,
> and
>  developers about what was expected from the build system 
And that is exactly what I am disputing, that they are listening to what 
packagers expect. Because it can surely not be the packagers' wish to have 
some piece of software stubbornly dictate that no, that package can not be 
pushed to stable now because it reached testing only 6 days, 23 hours, 59 
minutes and 59.999999 seconds ago. (I believe the timestamps in Bodhi have 
microsecond resolution internally. They used to be displayed that way, at 
least.)

Nor can it be in the interest of the Bodhi developers to have to maintain 
all that complex logic: you pointed out yourself that "what happens is that 
you will get into about 1/3rd of the way into it and find you have now to 
add a bunch of new requirements" – surely that is not what the developers 
expect.

>  and they are the ones who have given general guidance over the last
> 10+
>  years of bodhi development. 
If by "general guidance" you mean skyrocketing requirements, then I agree. 
Otherwise, I don't, sorry. See above, you admitted yourself that the 
requirements keep increasing all the time, forcing a major refactoring.

These days, even Rawhide (!) gets forced through Bodhi, though with an 
entirely different workflow (but in turn, that means the Bodhi developers 
get to maintain 2 completely different workflows with completely different 
rules). That is something Bodhi was never designed for. And the policy 
changes that have been made for Rawhide over the years have also changed 
things for the worse: It used to be that you were able to just do 
development in Rawhide, without having to bother about broken dependencies 
(which would just show up in the daily dependency report and get fixed 
there: I remember provenpackagers, mainly Alex Lancaster and me, mass-
rebuilding packages to fix broken dependencies; Rawhide composes did NOT 
fail due to broken dependencies at the time, the deliverables that failed to 
compose would just be missing that day), upgrade paths (from Rawhide to 
Rawhide, really no reason to not just let the users use distro-sync there 
and allow the version to go backwards; upgrade paths make sense only for 
releases), test failures (Rawhide was expected to be broken, as is normal), 
etc. Now we just make life harder for everyone, both package maintainers and 
Bodhi developers, for no reason.

        Kevin Kofler

[Kevin Fenzi]

On Wed, Nov 16, 2022 at 02:09:47PM +0100, Kevin Kofler via devel wrote:
>  
>  Kevin Fenzi is currently a member of FESCo (see 
>  https://docs.fedoraproject.org/en-US/fesco/) and has been in that role for 
>  years. So pushing the blame off to "someone else" is not going to work. 
You're welcome to bring anything you like to me. (That goes for
everyone!)

That said, it doesn't mean I have to agree with you or do what you want.

>  And I have brought this issue to FESCo (which is where most of the
> update 
>  policies came from, not FPC or the Council) more than once. Usually, it was 
>  not even brought to a vote. And whenever there was a vote about the issue, 
>  it was always in favor of either the status quo or even stricter rules. 
Indeed.
That might be an indicator that not so many people agree with you? 
 
>  > They are the ones who over time have listened to packagers,
> users, and
>  > developers about what was expected from the build system
>  
>  And that is exactly what I am disputing, that they are listening to what 
>  packagers expect. Because it can surely not be the packagers' wish to have 
>  some piece of software stubbornly dictate that no, that package can not be 
>  pushed to stable now because it reached testing only 6 days, 23 hours, 59 
>  minutes and 59.999999 seconds ago. (I believe the timestamps in Bodhi have 
>  microsecond resolution internally. They used to be displayed that way, at 
>  least.) 
With my packager hat on, yes, I am completely fine with bodhi telling me
that. 

>  Nor can it be in the interest of the Bodhi developers to have to
> maintain 
>  all that complex logic: you pointed out yourself that "what happens is that 
>  you will get into about 1/3rd of the way into it and find you have now to 
>  add a bunch of new requirements" – surely that is not what the developers 
>  expect. 
I agree that bodhi is a complex application, but I disagree most
strongly that the solution is to just make it not do all those things
that I find useful and desireable. 

>  > and they are the ones who have given general guidance over the
> last 10+
>  > years of bodhi development.
>  
>  If by "general guidance" you mean skyrocketing requirements, then I agree. 
>  Otherwise, I don't, sorry. See above, you admitted yourself that the 
>  requirements keep increasing all the time, forcing a major refactoring.
>  
>  These days, even Rawhide (!) gets forced through Bodhi, though with an 
>  entirely different workflow (but in turn, that means the Bodhi developers 
>  get to maintain 2 completely different workflows with completely different 
>  rules). That is something Bodhi was never designed for. And the policy 
>  changes that have been made for Rawhide over the years have also changed 
>  things for the worse: It used to be that you were able to just do 
>  development in Rawhide, without having to bother about broken dependencies 
>  (which would just show up in the daily dependency report and get fixed 
>  there: I remember provenpackagers, mainly Alex Lancaster and me, mass-
>  rebuilding packages to fix broken dependencies; Rawhide composes did NOT 
>  fail due to broken dependencies at the time, the deliverables that failed to 
>  compose would just be missing that day), upgrade paths (from Rawhide to 
>  Rawhide, really no reason to not just let the users use distro-sync there 
>  and allow the version to go backwards; upgrade paths make sense only for 
>  releases), test failures (Rawhide was expected to be broken, as is normal), 
>  etc. Now we just make life harder for everyone, both package maintainers and 
>  Bodhi developers, for no reason. 
I think the current situation is much better than what we had then. 
Blocking updates that break things is desireable, they will be fixed
sooner and not break everyone else that is trying to integrate their
changes. 

We can of course make things better, but I don't have any desire to go
back to the "good old days". 

Anyhow, I am done, feel free to get the last words in the thread. :) 

kevin

[Kevin Kofler]

Neal Gompa wrote:
>  Bodhi is an unusually difficult codebase for what it does. 
IMHO, the main reason the Bodhi code is so complex is because of all the 
policy that it enforces: karma (counting), update policies (minimum 
karma/time), critical path, autopushes, gating (automatic QA), …

If we would just let Bodhi do what the maintainer asks in the common case 
(i.e., no karma, no autopushes, no gating, just two buttons "push to 
testing" and "push to stable" that the maintainer can click at any time, as

Bodhi was initially designed), it would be a lot easier to implement those 
few special cases that are actually needed, such as freezes.

        Kevin Kofler

[Mattia Verga]

Il 10/11/22 21:49, Kevin Fenzi ha scritto:
>  On Thu, Nov 10, 2022 at 05:16:44PM +0000, Mattia Verga via devel
> wrote:
> > Il 10/11/22 01:58, Kevin Kofler via devel ha scritto:
> >> Kevin Kofler via devel wrote:
> >>
> >>> Mattia Verga via devel wrote:
> >>>> with the current workflow, Bodhi doesn't know when a release is
> freezed.
> >>>> There is support for a "Freeze" state, but it was never used.
> >>> How do we prevent then that pushes to stable actually move forward? If
> >>> rel- eng just hits a different button / runs a different script to push
> >>> testing only instead of both testing and stable, that is the "can we
> push
> >>> to stable?" property Bodhi needs to check.
> >> PS: The "worst mistake" that can happen then is that if we push only
> testing
> >> to a non-frozen release for whatever reason, the update will be included in
> >> that testing push, and then move forward to stable in the next stable push.
> >> I do not see this as a real issue.
> >>
> > I'm working on fixing some bits in Bodhi before proposing to releng the
> > use of the 'frozen' release state. That will enable Bodhi to avoid
> > pushing updates directly to stable if the release is frozen, as well as
> > some small tweaks that were requested and would make life easier for
> > releng folks.
>  Thanks!
>
>  To explain a bit to everyone the current workflow:
>
>  * In non freeze times, we have a cron job that pushes "all pending
>  updates" at 00:14 UTC every day. This is stable and testing for anything
>  thats pending moving to a new state.
>
>  * In freezes however, we have to disable the cron job and manually:
>  - First push branched version updates-testing by itself.
>  - Then push everything else (minus the branched version).
>
>  This sucks, because it's manual, there's a hour or so wait for
>  updates-testing to finish before we can do the rest, and you have to
>  remember to list:
>  --releases
> EPEL-9N,EPEL-7,EPEL-8,F36,F36C,EPEL-8M,EPEL-9,EPEL-8N,F35,F35M,F35C,F36M,F35F,F36F
>
>  If we could just mark branched frozen and have it not push stable there
>  (without specific builds, because we do still need to push stable
>  requests through the freeze) that would be great. We could also actually
>  note this in updates so people don't get confused why their update
>  didn't get pushed stable.
>
> > It shouldn't be too hard, it's just that Bodhi code is
> > sometimes so contorted that by making a simple change it's easy to break
> > something else... moving updates from one state to another and tagging
> > builds in the correct way without losing the right track is one of those
> > contorted part.
>  Yeah. ;(
>
>  Thanks again for working on it!
>
>  kevin 
Kevin, the PR for Bodhi is nearly finished, it will add support for
using the 'frozen' release state to avoid pausing the push cron job and
avoid direct pending to stable push of updates when the release is frozen.

I'll open a ticket to releng for changing the usual workflow of new
releases once the PR gets merged and a new Bodhi release is drafted and
pushed to prod. About the avoidance of direct to stable pushing, do you
think it needs to be discussed by FESCo?

Mattia

[Kevin Fenzi]

On Wed, Nov 16, 2022 at 05:37:07PM +0000, Mattia Verga via devel wrote:
>  
>  Kevin, the PR for Bodhi is nearly finished, it will add support for
>  using the 'frozen' release state to avoid pausing the push cron job and
>  avoid direct pending to stable push of updates when the release is frozen. 
Awesome.

>  I'll open a ticket to releng for changing the usual workflow of
> new
>  releases once the PR gets merged and a new Bodhi release is drafted and
>  pushed to prod. About the avoidance of direct to stable pushing, do you
>  think it needs to be discussed by FESCo? 
No, I think it's a implementation detail that makes things better for
releng, but has no end visible changes, so no need to run it by FESCo. 

kevin

[Adam Williamson]

On Wed, 2022-11-16 at 17:37 +0000, Mattia Verga via devel wrote:
>  
>  Kevin, the PR for Bodhi is nearly finished, it will add support for
>  using the 'frozen' release state to avoid pausing the push cron job and
>  avoid direct pending to stable push of updates when the release is frozen.
>  
>  I'll open a ticket to releng for changing the usual workflow of new
>  releases once the PR gets merged and a new Bodhi release is drafted and
>  pushed to prod. About the avoidance of direct to stable pushing, do you
>  think it needs to be discussed by FESCo? 
Hey Mattia! Can you keep me in the loop on the new version and
deployment also? The final phase of my work on critical path status by
group is waiting on that. Thanks!
-- 
Adam Williamson
Fedora QA
IRC: adamw | Twitter: adamw_ha
https://www.happyassassin.net

[Mattia Verga]

Il 17/11/22 18:10, Adam Williamson ha scritto:
>  On Wed, 2022-11-16 at 17:37 +0000, Mattia Verga via devel wrote:
> > Kevin, the PR for Bodhi is nearly finished, it will add support for
> > using the 'frozen' release state to avoid pausing the push cron job and
> > avoid direct pending to stable push of updates when the release is frozen.
> >
> > I'll open a ticket to releng for changing the usual workflow of new
> > releases once the PR gets merged and a new Bodhi release is drafted and
> > pushed to prod. About the avoidance of direct to stable pushing, do you
> > think it needs to be discussed by FESCo?
>  Hey Mattia! Can you keep me in the loop on the new version and
>  deployment also? The final phase of my work on critical path status by
>  group is waiting on that. Thanks! 
Hello Adam,

as soon as the last batch of PRs I submitted are merged, I plan to ask
@abompard (or whoever is taking care of Bodhi now) to draft and deploy
the new version.

I can take care of drafting the new version, if they agree, but then
someone has to deploy it to stg/prod, as I'm not part of infra-sig and I
don't know ansible/OC. My wishes are to have the new version with the
new features land in prod early now, so that if a bugfix release is
needed it can be deployed before the next freeze and that releng can
adjust their SOP to start using the 'frozen' release state in the next
release cycle.

I'll keep you posted. Do you need any other change about gating pushed
before the next release is drafted?

Mattia

[Kevin Fenzi]

On Fri, Nov 18, 2022 at 08:10:56AM +0000, Mattia Verga via devel wrote:
>  Il 17/11/22 18:10, Adam Williamson ha scritto:
>  > On Wed, 2022-11-16 at 17:37 +0000, Mattia Verga via devel wrote:
>  >> Kevin, the PR for Bodhi is nearly finished, it will add support for
>  >> using the 'frozen' release state to avoid pausing the push cron job and
>  >> avoid direct pending to stable push of updates when the release is frozen.
>  >>
>  >> I'll open a ticket to releng for changing the usual workflow of new
>  >> releases once the PR gets merged and a new Bodhi release is drafted and
>  >> pushed to prod. About the avoidance of direct to stable pushing, do you
>  >> think it needs to be discussed by FESCo?
>  > Hey Mattia! Can you keep me in the loop on the new version and
>  > deployment also? The final phase of my work on critical path status by
>  > group is waiting on that. Thanks!
>  
>  Hello Adam,
>  
>  as soon as the last batch of PRs I submitted are merged, I plan to ask
>  @abompard (or whoever is taking care of Bodhi now) to draft and deploy
>  the new version. 
I can help with the deployment. :) 

It might also be nice to move bodhi-backend01 to f37.

kevin

[Mattia Verga]

@kevin I've announced on Bodhi matrix channel that I was planning to draft the new
release, but since I received no response I've pushed out Bodhi 7.0.0.

I have followed the SOP at [1] and bodhi-* RPMs are now available in both f37-infra-stg
and f36-infra-stg. Tests run smoothly in F37, so I think it's ok to move bodhi-backend
on it.

Can you take care of running the playbooks on staging so that we can start testing the new
version?

Mattia

[1]
https://fedora-infra.github.io/bodhi/develop/developer/releases.html#depl...

[Kevin Fenzi]

On Sat, Nov 26, 2022 at 05:27:30PM -0000, Mattia Verga via devel wrote:
>  @kevin I've announced on Bodhi matrix channel that I was planning
> to draft the new release, but since I received no response I've pushed out Bodhi
> 7.0.0.
>  
>  I have followed the SOP at [1] and bodhi-* RPMs are now available in both f37-infra-stg
> and f36-infra-stg. Tests run smoothly in F37, so I think it's ok to move bodhi-backend
> on it.
>  
>  Can you take care of running the playbooks on staging so that we can start testing the
> new version? 
Yep. I will do so sometime in the coming week.

Will have more of an idea when tomorrow.

kevin

[Adam Williamson]

On Fri, 2022-11-18 at 08:10 +0000, Mattia Verga via devel wrote:
>  Il 17/11/22 18:10, Adam Williamson ha scritto:
>  > On Wed, 2022-11-16 at 17:37 +0000, Mattia Verga via devel wrote:
>  > > Kevin, the PR for Bodhi is nearly finished, it will add support for
>  > > using the 'frozen' release state to avoid pausing the push cron job
> and
>  > > avoid direct pending to stable push of updates when the release is frozen.
>  > > 
>  > > I'll open a ticket to releng for changing the usual workflow of new
>  > > releases once the PR gets merged and a new Bodhi release is drafted and
>  > > pushed to prod. About the avoidance of direct to stable pushing, do you
>  > > think it needs to be discussed by FESCo?
>  > Hey Mattia! Can you keep me in the loop on the new version and
>  > deployment also? The final phase of my work on critical path status by
>  > group is waiting on that. Thanks!
>  
>  Hello Adam,
>  
>  as soon as the last batch of PRs I submitted are merged, I plan to ask
>  @abompard (or whoever is taking care of Bodhi now) to draft and deploy
>  the new version.
>  
>  I can take care of drafting the new version, if they agree, but then
>  someone has to deploy it to stg/prod, as I'm not part of infra-sig and I
>  don't know ansible/OC. My wishes are to have the new version with the
>  new features land in prod early now, so that if a bugfix release is
>  needed it can be deployed before the next freeze and that releng can
>  adjust their SOP to start using the 'frozen' release state in the next
>  release cycle.
>  
>  I'll keep you posted. Do you need any other change about gating pushed
>  before the next release is drafted? 
Thanks! I'll work with nirik on the deployment part. I don't need any
more changes merged (at least, I hope I didn't forget anything),
everything I had planned for Bodhi side has been merged; further
changes will be to our ansible configs and then the openQA scheduler.
-- 
Adam Williamson
Fedora QA
IRC: adamw | Twitter: adamw_ha
https://www.happyassassin.net