Attempting to allow for a source of truth for Fedora release cycle information: releasestream

[Adam Williamson]

Hi folks!

Before the RH holiday shutdown, I started a new project designed to
address a long-standing pain point in Fedora: we don't really have a
good source of truth for release cycle information. I'm thinking of
situations where something:

* Needs to know what the "current stable" Fedora release(s) are
* Needs to know if Fedora X is Branched
* Needs to know whether Fedora X Beta happened yet
* Needs to know whether Fedora X is frozen

...etc etc etc. Some of this information is available...sort of...from
various different systems, with various awkward caveats that I won't
dive into unless someone asks. But there isn't really a single source
of truth for it, and some of it just isn't really available in any
easily machine-consumable way.

So I wrote releasestream:
https://pagure.io/fedora-qa/releasestream

releasestream is intended to be a system that will let us do this. It
is a very very simple webapp with three capabilities:

* It can read a simple record ("stream") of "release events" for a
"release" and produce several static JSON representations of the
information
* It can write an entry to one of these streams in response to a
properly-formatted POST request
* It can publish a message when a new entry is received

that is all it does. The "releases" and "events" are entirely
arbitrary; a "release" can be any string, and so can the "state" for
a
given "event". An "event" is defined as a state being either reached
or
left; any number of events for the same state can be present for a
release.

The JSON outputs are basically "states by release" and "releases by
state", as you might want either depending on what you're doing. You
can conveniently look up "what releases are currently in state X?" or
"what states is release X currently in?".

This all works right now; the main thing that isn't implemented yet is
any form of authentication. Right now if you can talk to the server you
can submit events. I wanted to check if there is interest in moving
forward with this, and discuss various options, before working on that.
There is a ticket where I sketched out various possible approaches:
https://pagure.io/fedora-qa/releasestream/issue/2

My idea for using this is basically that we deploy an 'official'
releasestream instance in infra, and then find things that do the
actual work of moving Fedora releases into and out of given "states"
and tack on a bit at the end to tell releasestream about it. So when
e.g. Mohan is putting Fedora 34 into the Beta freeze, we would add a
"submit 'enter freeze' event to releasestream" to that process somehow
- ideally something that has to be run as part of the process anyway
would send the POST, but in a pinch a human could do it too. When
Fedora 34 is being 'released', we tweak that process to include sending
a releasestream event POST. And so on.

The thing I like about this design is that there's a low barrier to
entry, and can easily be adopted piecemeal but still be immediately
useful for some things - as long as the event your code needs to watch
for has been "onboarded", you're good. It also allows for the
implementation details of a state change to change radically - it can
go from being done by a human to being done by system X to being done
by system Y, and all that needs to happen is to ensure the same dead
simple POST request is sent to the same server.

So, what do folks think? Does this seem like a good idea? Should I go
ahead with trying to get it deployed and onboard things to it? Any
comments, ideas, potential problems? Thanks!
-- 
Adam Williamson
Fedora QA
IRC: adamw | Twitter: adamw_ha
https://www.happyassassin.net

[Kevin Fenzi]

I'm definitely in favor of this effort. ;) 

A few questions inline...

On Wed, Jan 13, 2021 at 09:53:36AM -0800, Adam Williamson wrote:
...snip...
>  
>  that is all it does. The "releases" and "events" are entirely
>  arbitrary; a "release" can be any string, and so can the "state" for
> a
>  given "event". An "event" is defined as a state being either reached
> or
>  left; any number of events for the same state can be present for a
>  release. 
Might it be good to standardize these? I mean we don't want to have: 

state: "Fedora 34 Beta Released to mirrors"
and then in f35 cycle:
state: "f35b released/mirrors"

Or at least record somewhere in a doc what we want to use after the
first cycle in use?

...snip...
>  
>  My idea for using this is basically that we deploy an 'official'
>  releasestream instance in infra, and then find things that do the
>  actual work of moving Fedora releases into and out of given "states"
>  and tack on a bit at the end to tell releasestream about it. So when
>  e.g. Mohan is putting Fedora 34 into the Beta freeze, we would add a
>  "submit 'enter freeze' event to releasestream" to that process somehow
>  - ideally something that has to be run as part of the process anyway
>  would send the POST, but in a pinch a human could do it too. When
>  Fedora 34 is being 'released', we tweak that process to include sending
>  a releasestream event POST. And so on. 
I see right now this keeps the state/messages in a file store?
I think this is a great app to just run in openshift, but if we do that
I wouldn't want to loose state, so perhaps we store things in a
database? I know thats overkill for this simple stuff, but it would mean
we have everything easily stored/backed up, etc. I guess we could also
just make a persistent volume and make sure to back it up. 

...snip...

>  
>  So, what do folks think? Does this seem like a good idea? Should I go
>  ahead with trying to get it deployed and onboard things to it? Any
>  comments, ideas, potential problems? Thanks! 
Yeah, I like it and definitely think it's worth a try!

Thanks for implementing it.

kevin

[Adam Williamson]

On Wed, 2021-01-13 at 13:42 -0800, Kevin Fenzi wrote:
>  I'm definitely in favor of this effort. ;) 
>  
>  A few questions inline...
>  
>  On Wed, Jan 13, 2021 at 09:53:36AM -0800, Adam Williamson wrote:
>  ...snip...
>  > 
>  > that is all it does. The "releases" and "events" are entirely
>  > arbitrary; a "release" can be any string, and so can the "state"
> for a
>  > given "event". An "event" is defined as a state being either
> reached or
>  > left; any number of events for the same state can be present for a
>  > release.
>  
>  Might it be good to standardize these? I mean we don't want to have: 
>  
>  state: "Fedora 34 Beta Released to mirrors"
>  and then in f35 cycle:
>  state: "f35b released/mirrors"
>  
>  Or at least record somewhere in a doc what we want to use after the
>  first cycle in use? 
I was following the lessons we learned with taskotron: make the tool
simple and flexible, standardize the usage by convention. (This also
means the tool is flexible: if it works out for Fedora, it could be
used for...pretty much any other software project that has this issue).

Typically submissions should be done by code or at least scripted, and
there shouldn't be *too* many systems submitting events, so it should
be relatively easy to make sure they follow the same names. But yes, we
definitely want everything that submits to call the same release the
same thing :)

We *could* write a "fedora helper" for submitting events that enforces
a house style, but for something this simple I'm not sure it's
worthwhile.

We could use one of the name formats Bodhi has, or we could use the
'dist' / 'short' names from productmd compose metadata, I guess.

>  ...snip...
>  > 
>  > My idea for using this is basically that we deploy an 'official'
>  > releasestream instance in infra, and then find things that do the
>  > actual work of moving Fedora releases into and out of given "states"
>  > and tack on a bit at the end to tell releasestream about it. So when
>  > e.g. Mohan is putting Fedora 34 into the Beta freeze, we would add a
>  > "submit 'enter freeze' event to releasestream" to that process
> somehow
>  > - ideally something that has to be run as part of the process anyway
>  > would send the POST, but in a pinch a human could do it too. When
>  > Fedora 34 is being 'released', we tweak that process to include sending
>  > a releasestream event POST. And so on.
>  
>  I see right now this keeps the state/messages in a file store?
>  I think this is a great app to just run in openshift, but if we do that
>  I wouldn't want to loose state, so perhaps we store things in a
>  database? I know thats overkill for this simple stuff, but it would mean
>  we have everything easily stored/backed up, etc. I guess we could also
>  just make a persistent volume and make sure to back it up.  
Yeah, it could be a database too, I guess, I just used a text file
because it was easy, honestly, and it also means you can hand edit
stuff quite easily (this would allow fixups for incorrect names, and
it'd also make it easy to 'reconstruct' historical schedules, which I
might do at some point). If you file a ticket for adding db support
I'll look at it.

>  ...snip...
>  
>  > 
>  > So, what do folks think? Does this seem like a good idea? Should I go
>  > ahead with trying to get it deployed and onboard things to it? Any
>  > comments, ideas, potential problems? Thanks!
>  
>  Yeah, I like it and definitely think it's worth a try!
>  
>  Thanks for implementing it. 
Thanks for the feedback!
-- 
Adam Williamson
Fedora QA
IRC: adamw | Twitter: adamw_ha
https://www.happyassassin.net

[Ben Cotton]

This sounds like a Particularly Delightful Concept. :-) I like the
idea and heartily endorse you doing it.

-- 
Ben Cotton
He / Him / His
Senior Program Manager, Fedora & CentOS Stream
Red Hat
TZ=America/Indiana/Indianapolis

[Debarshi Ray]

Hey,

I wanted to respond earlier, but it totally slipped my mind.

We'd totally use releasestream in Toolbox for mapping the string
"rawhide" to a numeric version:
https://github.com/containers/toolbox/issues/646

Cheers,
Rishi

[Frantisek Zatloukal]

On Wed, Feb 10, 2021 at 11:27 PM Debarshi Ray <rishi.is(a)lostca.se&gt; wrote:

>  Hey,
>
>  I wanted to respond earlier, but it totally slipped my mind.
>
>  We'd totally use releasestream in Toolbox for mapping the string
>  "rawhide" to a numeric version:
>  https://github.com/containers/toolbox/issues/646
>
>  Cheers,
>  Rishi
>  
You (and anybody else, ofc :) ) can take a look at the packager dashboard
before Adam deploys releasestream. It exposes Fedora release information on
https://packager-dashboard.fedoraproject.org/api/v1/releases . It's  based
on fedfind, with some more logic and information processing in the
background.

[Adam Williamson]

On Thu, 2021-02-11 at 08:16 +0100, Frantisek Zatloukal wrote:
>  On Wed, Feb 10, 2021 at 11:27 PM Debarshi Ray
> <rishi.is(a)lostca.se&gt; wrote:
>  
>  > Hey,
>  > 
>  > I wanted to respond earlier, but it totally slipped my mind.
>  > 
>  > We'd totally use releasestream in Toolbox for mapping the string
>  > "rawhide" to a numeric version:
>  > https://github.com/containers/toolbox/issues/646
>  > 
>  > Cheers,
>  > Rishi
>  > 
>  
>  You (and anybody else, ofc :) ) can take a look at the packager dashboard
>  before Adam deploys releasestream. It exposes Fedora release information on
>  https://packager-dashboard.fedoraproject.org/api/v1/releases . It's  based
>  on fedfind, with some more logic and information processing in the
>  background. 
In the end, though, fedfind is just getting its info from a JSON file I
hand edit, which winds up getting reprocessed into that...:D
https://fedorapeople.org/groups/qa/metadata/release.json 
-- 
Adam Williamson
Fedora QA
IRC: adamw | Twitter: adamw_ha
https://www.happyassassin.net

[Clement Verna]

On Wed, 13 Jan 2021 at 18:53, Adam Williamson <adamwill(a)fedoraproject.org&gt;
wrote:

>  Hi folks!
>
>  Before the RH holiday shutdown, I started a new project designed to
>  address a long-standing pain point in Fedora: we don't really have a
>  good source of truth for release cycle information. I'm thinking of
>  situations where something:
>
>  * Needs to know what the "current stable" Fedora release(s) are
>  * Needs to know if Fedora X is Branched
>  * Needs to know whether Fedora X Beta happened yet
>  * Needs to know whether Fedora X is frozen
>
>  ...etc etc etc. Some of this information is available...sort of...from
>  various different systems, with various awkward caveats that I won't
>  dive into unless someone asks. But there isn't really a single source
>  of truth for it, and some of it just isn't really available in any
>  easily machine-consumable way.
>
>  So I wrote releasestream:
>  https://pagure.io/fedora-qa/releasestream
>
>  releasestream is intended to be a system that will let us do this. It
>  is a very very simple webapp with three capabilities:
>
>  * It can read a simple record ("stream") of "release events" for a
>  "release" and produce several static JSON representations of the
>  information
>  * It can write an entry to one of these streams in response to a
>  properly-formatted POST request
>  * It can publish a message when a new entry is received
>
>  that is all it does. The "releases" and "events" are entirely
>  arbitrary; a "release" can be any string, and so can the "state" for
> a
>  given "event". An "event" is defined as a state being either reached
> or
>  left; any number of events for the same state can be present for a
>  release.
>
>  The JSON outputs are basically "states by release" and "releases by
>  state", as you might want either depending on what you're doing. You
>  can conveniently look up "what releases are currently in state X?" or
>  "what states is release X currently in?".
>
>  This all works right now; the main thing that isn't implemented yet is
>  any form of authentication. Right now if you can talk to the server you
>  can submit events. I wanted to check if there is interest in moving
>  forward with this, and discuss various options, before working on that.
>  There is a ticket where I sketched out various possible approaches:
>  https://pagure.io/fedora-qa/releasestream/issue/2
>
>  My idea for using this is basically that we deploy an 'official'
>  releasestream instance in infra, and then find things that do the
>  actual work of moving Fedora releases into and out of given "states"
>  and tack on a bit at the end to tell releasestream about it. So when
>  e.g. Mohan is putting Fedora 34 into the Beta freeze, we would add a
>  "submit 'enter freeze' event to releasestream" to that process somehow
>  - ideally something that has to be run as part of the process anyway
>  would send the POST, but in a pinch a human could do it too. When
>  Fedora 34 is being 'released', we tweak that process to include sending
>  a releasestream event POST. And so on.
>
>  The thing I like about this design is that there's a low barrier to
>  entry, and can easily be adopted piecemeal but still be immediately
>  useful for some things - as long as the event your code needs to watch
>  for has been "onboarded", you're good. It also allows for the
>  implementation details of a state change to change radically - it can
>  go from being done by a human to being done by system X to being done
>  by system Y, and all that needs to happen is to ensure the same dead
>  simple POST request is sent to the same server.
>
>  So, what do folks think? Does this seem like a good idea? Should I go
>  ahead with trying to get it deployed and onboard things to it? Any
>  comments, ideas, potential problems? Thanks!
>  

I am pretty sure you did :-) but have you looked at adding the missing
information to Bodhi ? Now that we have rawhide in Bodhi we should be able
to expose most the information needed.



>  --
>  Adam Williamson
>  Fedora QA
>  IRC: adamw | Twitter: adamw_ha
>  https://www.happyassassin.net
>
>
>  _______________________________________________
>  rel-eng mailing list -- rel-eng(a)lists.fedoraproject.org
>  To unsubscribe send an email to rel-eng-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct:
>  https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
>  https://lists.fedoraproject.org/archives/list/[e-mail]...
>

[Adam Williamson]

On Thu, 2021-02-11 at 09:55 +0100, Clement Verna wrote:
>  > 
>  > So, what do folks think? Does this seem like a good idea? Should I go
>  > ahead with trying to get it deployed and onboard things to it? Any
>  > comments, ideas, potential problems? Thanks!
>  > 
>  
>  
>  I am pretty sure you did :-) but have you looked at adding the missing
>  information to Bodhi ? Now that we have rawhide in Bodhi we should be able
>  to expose most the information needed. 
I didn't look at that in any technical detail but conceptually it seems
kinda wrong to me. The problem with using any existing system is that
all existing systems are *for* something. Bodhi is for dealing with
updates. There's no reason why Bodhi would track, for instance, whether
34 is under a string freeze, right? Because Bodhi doesn't have anything
to do with translations. There are a zillion other 'states' that it
would be useful to have info on which aren't relevant to any *one*
existing system, and that's why to me it makes sense to have a separate
thing for that.
-- 
Adam Williamson
Fedora QA
IRC: adamw | Twitter: adamw_ha
https://www.happyassassin.net

[Kevin Fenzi]

On Thu, Feb 11, 2021 at 09:04:12AM -0800, Adam Williamson wrote:
>  On Thu, 2021-02-11 at 09:55 +0100, Clement Verna wrote:
>  > > 
>  > > So, what do folks think? Does this seem like a good idea? Should I go
>  > > ahead with trying to get it deployed and onboard things to it? Any
>  > > comments, ideas, potential problems? Thanks!
>  > > 
>  > 
>  > 
>  > I am pretty sure you did :-) but have you looked at adding the missing
>  > information to Bodhi ? Now that we have rawhide in Bodhi we should be able
>  > to expose most the information needed.
>  
>  I didn't look at that in any technical detail but conceptually it seems
>  kinda wrong to me. The problem with using any existing system is that
>  all existing systems are *for* something. Bodhi is for dealing with
>  updates. There's no reason why Bodhi would track, for instance, whether
>  34 is under a string freeze, right? Because Bodhi doesn't have anything
>  to do with translations. There are a zillion other 'states' that it
>  would be useful to have info on which aren't relevant to any *one*
>  existing system, and that's why to me it makes sense to have a separate
>  thing for that. 
The other problem that comes from this is: 

When we are 'go' for a new release, we start pushing '0-day' updates. 
Usually this is the day after the 'go' decision and a week or two ahead
of the actual release. At that point bodhi thinks the release is
"released", even though it is not. 

kevin

[Clement Verna]

On Thu, 11 Feb 2021 at 18:04, Adam Williamson <adamwill(a)fedoraproject.org&gt;
wrote:

>  On Thu, 2021-02-11 at 09:55 +0100, Clement Verna wrote:
>  > >
>  > > So, what do folks think? Does this seem like a good idea? Should I go
>  > > ahead with trying to get it deployed and onboard things to it? Any
>  > > comments, ideas, potential problems? Thanks!
>  > >
>  >
>  >
>  > I am pretty sure you did :-) but have you looked at adding the missing
>  > information to Bodhi ? Now that we have rawhide in Bodhi we should be
>  able
>  > to expose most the information needed.
>
>  I didn't look at that in any technical detail but conceptually it seems
>  kinda wrong to me. The problem with using any existing system is that
>  all existing systems are *for* something. Bodhi is for dealing with
>  updates. There's no reason why Bodhi would track, for instance, whether
>  34 is under a string freeze, right? Because Bodhi doesn't have anything
>  to do with translations. There are a zillion other 'states' that it
>  would be useful to have info on which aren't relevant to any *one*
>  existing system, and that's why to me it makes sense to have a separate
>  thing for that.
>  
I was wondering about the 4 Needs bullet points in your original email
which could be in Bodhi but yes if we want to add more and more info a
seperate app makes sense :-)

-- 
>  Adam Williamson
>  Fedora QA
>  IRC: adamw | Twitter: adamw_ha
>  https://www.happyassassin.net
>
>
>
>