Hi, I'm Beatriz and I'm a student at the Santa Catarina State University.
Currently, I'm studying the Fedora Release Life Cycle, and would like to know if anyone could help me with some questions about this subject:
1. Is the Fedora CI infrastructure ( https://docs.fedoraproject.org/en-US/ci/) in production, i.e., is it being used to build/test official packages/images? Or is it only experimental? 2. When considering the release validation tests. I understand that each day, a full compose of the tree is attempted and successful composes are synced to the /fedora/linux/development/ directory on the mirrors. Right after this process, each successful compose is tested by openQA, that is, the release validation tests are run. Am I right, or the validation tests are run before the synchronization with mirrors?
-- Best regards, Beatriz
Santa Catarina State University - UDESC
On Wed, 2021-06-16 at 15:03 -0300, Beatriz Michelson Reichert wrote:
Hi, I'm Beatriz and I'm a student at the Santa Catarina State University.
Hi Beatriz! Glad to help out.
Currently, I'm studying the Fedora Release Life Cycle, and would like to know if anyone could help me with some questions about this subject:
- Is the Fedora CI infrastructure (
https://docs.fedoraproject.org/en-US/ci/) in production, i.e., is it being used to build/test official packages/images? Or is it only experimental?
It is in production. "Fedora CI" per se, though, isn't directly involved in compose testing/validation at present; it only runs tests at the level of individual package builds. These are reported and used for gating via Bodhi as part of the workflow of individual packages/updates being tagged stable, rather than the level of an entire compose being run from the stable package set.
- When considering the release validation tests. I understand that each
day, a full compose of the tree is attempted and successful composes are synced to the /fedora/linux/development/ directory on the mirrors. Right after this process, each successful compose is tested by openQA, that is, the release validation tests are run. Am I right, or the validation tests are run before the synchronization with mirrors?
The openQA validation tests run before the mirror sync is complete; they're triggered by the "compose finished" message. This is partly to get them done as quickly as possible and partly because, IIRC anyway, there was no "mirror sync completed" message when we implemented the scheduling. I don't recall if there is one yet.
openQA pulls the images to test directly from where the compose is initially spat out by Pungi, under https://kojipkgs.fedoraproject.org/compose/ . Most tests that use repos are also configured to use the repositories from the compose tree rather than the default mirrormanager repos. However, tests of the 'default install path' don't do this, as it would invalidate the test, so those tests usually wind up using the repo from the previous compose, which can cause results to 'lag' a bit. Triggering the tests after the mirror sync is complete would fix that issue, if we ever decided to do it. :P
Hope that helps! Please feel free to ping me any time if I can help with anything else.
Hi Adam,
Are release validation test results forwarded to the ResultsDB repository? If so, after this storage, if the compose passes the tests, the synchronization with the mirrors starts. Who is responsible for initiating the synchronization?
-- Best regards, Beatriz
Santa Catarina State University - UDESC
Em qua., 16 de jun. de 2021 às 16:33, Adam Williamson < adamwill@fedoraproject.org> escreveu:
On Wed, 2021-06-16 at 15:03 -0300, Beatriz Michelson Reichert wrote:
Hi, I'm Beatriz and I'm a student at the Santa Catarina State University.
Hi Beatriz! Glad to help out.
Currently, I'm studying the Fedora Release Life Cycle, and would like to know if anyone could help me with some questions about this subject:
- Is the Fedora CI infrastructure (
https://docs.fedoraproject.org/en-US/ci/) in production, i.e., is it being used to build/test official packages/images? Or is it only experimental?
It is in production. "Fedora CI" per se, though, isn't directly involved in compose testing/validation at present; it only runs tests at the level of individual package builds. These are reported and used for gating via Bodhi as part of the workflow of individual packages/updates being tagged stable, rather than the level of an entire compose being run from the stable package set.
- When considering the release validation tests. I understand that
each
day, a full compose of the tree is attempted and successful composes
are
synced to the /fedora/linux/development/ directory on the mirrors.
Right
after this process, each successful compose is tested by openQA, that
is,
the release validation tests are run. Am I right, or the validation
tests
are run before the synchronization with mirrors?
The openQA validation tests run before the mirror sync is complete; they're triggered by the "compose finished" message. This is partly to get them done as quickly as possible and partly because, IIRC anyway, there was no "mirror sync completed" message when we implemented the scheduling. I don't recall if there is one yet.
openQA pulls the images to test directly from where the compose is initially spat out by Pungi, under https://kojipkgs.fedoraproject.org/compose/ . Most tests that use repos are also configured to use the repositories from the compose tree rather than the default mirrormanager repos. However, tests of the 'default install path' don't do this, as it would invalidate the test, so those tests usually wind up using the repo from the previous compose, which can cause results to 'lag' a bit. Triggering the tests after the mirror sync is complete would fix that issue, if we ever decided to do it. :P
Hope that helps! Please feel free to ping me any time if I can help with anything else. -- Adam Williamson Fedora QA IRC: adamw | Twitter: adamw_ha https://www.happyassassin.net
test mailing list -- test@lists.fedoraproject.org To unsubscribe send an email to test-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/test@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
On Thu, 2021-06-17 at 11:49 -0300, Beatriz Michelson Reichert wrote:
Hi Adam,
Are release validation test results forwarded to the ResultsDB repository?
Yes. fedora_openqa has a fedora-messaging consumer which listens out for "test complete" messages and submits results to resultsdb.
If so, after this storage, if the compose passes the tests, the synchronization with the mirrors starts. Who is responsible for initiating the synchronization?
This is not actually how it works yet for automated nightly composes of Rawhide and Branched (when Branched exists). In those cases, the tests are currently 'advisory'. If the compose succeeds - i.e. if all 'fatal' compose tasks complete - it gets synced out to become the 'current' content of the release repository; the test results do not factor in. The synchronization is part of release engineering's overall compose process. I used to know where the actual script was, but either it moved or I forgot! Mohan Boddu could tell you.
We (Fedora) have had a plan approximately forever to make the nightly composes sync only if a given subset of the openQA tests pass, but it's never been fully implemented. We (QA) did implement all the bits that are our responsibility: the greenwave policy that was supposed to be used for this exists - https://pagure.io/fedora-infra/ansible/blob/main/f/roles/openshift-apps/gree... - and we even set up the daily "compose check reports" to log whether the compose passed that check or not, so we could see how often it passes and fails. But for whatever reason, things never progressed beyond that point. The next step would be for releng to adjust the compose process to actually wait after the compose is complete and only proceed to the sync step if the greenwave policy is satisfied.
For the process of doing stable releases, this all happens manually in a loop between releng and us, basically. We (usually I) request a candidate compose via a Pagure ticket. releng manually triggers the compose. It is synced from the initial output location under https://kojipkgs.fedoraproject.org/compose/ to the stage location - https://dl.fedoraproject.org/pub/alt/stage/ - by releng (not sure if this is manual or automated); that location is mirrored but only lightly.
The automated tests run automatically as soon as the compose is complete, the 'relvalconsumer' message consumer will automatically create a validation event in the wiki and announce it, and then humans run the manual tests. A candidate compose would only be 'synced' further than the stage location if it were actually released as the Beta or Final release, which happens after we do a go/no-go meeting and sign off on the release. That sync process is also handled manually, I believe (using a script).
HTH!
Hi Adam, thank you for replying, it helped a lot!
I have some more questions about the composes and Koji, and would like to know if you could help me with the questions about this subject too:
1. What repository does Koji forward the build of a package and the RPMs to? 2. The nightly compose process receives some package builds as input. What repository are these builds in? Is it the same repository as the previous question? 3. The candidate compose process receives the candidate packages (stable packages) as input. What repository are these candidate packages in?
-- Best regards, Beatriz
Santa Catarina State University - UDESC
Em sex., 18 de jun. de 2021 às 15:01, Adam Williamson < adamwill@fedoraproject.org> escreveu:
On Thu, 2021-06-17 at 11:49 -0300, Beatriz Michelson Reichert wrote:
Hi Adam,
Are release validation test results forwarded to the ResultsDB
repository?
Yes. fedora_openqa has a fedora-messaging consumer which listens out for "test complete" messages and submits results to resultsdb.
If so, after this storage, if the compose passes the tests, the synchronization with the mirrors starts. Who is responsible for
initiating
the synchronization?
This is not actually how it works yet for automated nightly composes of Rawhide and Branched (when Branched exists). In those cases, the tests are currently 'advisory'. If the compose succeeds - i.e. if all 'fatal' compose tasks complete - it gets synced out to become the 'current' content of the release repository; the test results do not factor in. The synchronization is part of release engineering's overall compose process. I used to know where the actual script was, but either it moved or I forgot! Mohan Boddu could tell you.
We (Fedora) have had a plan approximately forever to make the nightly composes sync only if a given subset of the openQA tests pass, but it's never been fully implemented. We (QA) did implement all the bits that are our responsibility: the greenwave policy that was supposed to be used for this exists -
https://pagure.io/fedora-infra/ansible/blob/main/f/roles/openshift-apps/gree...
- and we even set up the daily "compose check reports" to log whether
the compose passed that check or not, so we could see how often it passes and fails. But for whatever reason, things never progressed beyond that point. The next step would be for releng to adjust the compose process to actually wait after the compose is complete and only proceed to the sync step if the greenwave policy is satisfied.
For the process of doing stable releases, this all happens manually in a loop between releng and us, basically. We (usually I) request a candidate compose via a Pagure ticket. releng manually triggers the compose. It is synced from the initial output location under https://kojipkgs.fedoraproject.org/compose/ to the stage location
- https://dl.fedoraproject.org/pub/alt/stage/ - by releng (not sure if
this is manual or automated); that location is mirrored but only lightly.
The automated tests run automatically as soon as the compose is complete, the 'relvalconsumer' message consumer will automatically create a validation event in the wiki and announce it, and then humans run the manual tests. A candidate compose would only be 'synced' further than the stage location if it were actually released as the Beta or Final release, which happens after we do a go/no-go meeting and sign off on the release. That sync process is also handled manually, I believe (using a script).
HTH!
Adam Williamson Fedora QA IRC: adamw | Twitter: adamw_ha https://www.happyassassin.net
test mailing list -- test@lists.fedoraproject.org To unsubscribe send an email to test-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/test@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
On Sat, 2021-06-19 at 13:53 -0300, Beatriz Michelson Reichert wrote:
Hi Adam, thank you for replying, it helped a lot!
I have some more questions about the composes and Koji, and would like to know if you could help me with the questions about this subject too:
- What repository does Koji forward the build of a package and the RPMs
to? 2. The nightly compose process receives some package builds as input. What repository are these builds in? Is it the same repository as the previous question? 3. The candidate compose process receives the candidate packages (stable packages) as input. What repository are these candidate packages in?
Hi Beatriz! So these questions are sort of...backwards, in a sense. The compose processes are what *produce* the repositories. Koji doesn't exactly build packages and then put them in a repository.
Well, technically there is a repo metadata 'view' of each Koji tag. You can find those at kojipkgs.fedoraproject.org/repos/ - each tag has a directory there, and you can find repodata under each directory which gives you a repo 'view' of the latest builds for that tag. But those are not things that are really intended to be consumed by Fedora users or testers in most typical workflows.
I think the best way to think of it in terms of how most processes work is that Koji builds packages and then they just sort of...sit in Koji to be used as inputs to other things. A "compose" is the mechanism by which we take builds from Koji and produce the things that Fedora users 'consume', including images and the formal public-facing repositories.
So, just running a build in Koji does not necessarily result in it appearing in any conventionally-public-facing repository. For Rawhide there are some convenience mechanisms hooked up so when a packager does a build the "usual" way, it automatically gets tagged for Rawhide and included in the next compose, but even that can be avoided if desired. For a build to appear in a public repo, some kind of compose step has to happen in between.
To answer q2 directly - the nightly compose process for Rawhide and Branched uses the packages tagged with the stable tag for the release in question. So currently Rawhide composes use all packages tagged 'f35'. When F35 branches, Branched composes will use all packages tagged 'f35' and Rawhide composes will use all packages tagged 'f36'.
The composes that are run regularly to update the stable release update repositories use the '-updates' tags - so when we run a compose to update the F34 stable updates repo, for instance, it uses all packages tagged 'f34-updates'. (There are also composes run to update the updates-testing repositories for stable releases and Branched; these use the 'fXX-updates-testing' tags, not surprisingly). When a stable release update is "pushed stable" via Bodhi, what actually *happens* that makes it ultimately show up in the public stable updates repo is that it has the 'fXX-updates' tag applied to it in Koji, so the next stable update compose run pulls it in. (For Branched and Rawhide, when an update is 'pushed stable' via Bodhi, it has the fXX' tag applied, so the next nightly compose pulls it in).
Note that your q2 and q3 are phrased slightly wrong, btw. It's nightly composes that use "[all] stable packages" as input. Those composes are automated and use *only* the packages tagged 'f35' or 'f36' or whatever. Candidate composes can potentially pull in additional packages. The difference between a candidate compose and a nightly compose is some slightly different naming, configuration and metadata produced by running the compose tool differently, *plus* the potential to manually include packages that have not yet been 'pushed stable', which is done by releng by hand. Here's an example from the last release:
https://pagure.io/releng/issue/10093#comment-729024
that was the candidate compose request I filed that resulted in F34 Final RC2 being built, which is ultimately what was released as F34 Final. As you can see, I listed 9 updates to be included in the compose which were not 'stable' at the time. The automated nightly compose run on the same day would not have included those updates. In order to include them, releng would have tagged them into a side repository that is included in the candidate compose configuration before running the compose. I actually forget what that repo is called :P Again, mboddu could tell you. All of the above is my best recollection/understanding, btw - if Mohan or Kevin Fenzi tell you something different, they're probably right.
Hi Adam,
Thank you so much for replying, it helped a lot!
-- Best regards, Beatriz
Santa Catarina State University - UDESC
Em dom., 20 de jun. de 2021 às 03:45, Adam Williamson < adamwill@fedoraproject.org> escreveu:
On Sat, 2021-06-19 at 13:53 -0300, Beatriz Michelson Reichert wrote:
Hi Adam, thank you for replying, it helped a lot!
I have some more questions about the composes and Koji, and would like to know if you could help me with the questions about this subject too:
- What repository does Koji forward the build of a package and the
RPMs
to? 2. The nightly compose process receives some package builds as input. What repository are these builds in? Is it the same repository as the previous question? 3. The candidate compose process receives the candidate packages
(stable
packages) as input. What repository are these candidate packages in?
Hi Beatriz! So these questions are sort of...backwards, in a sense. The compose processes are what *produce* the repositories. Koji doesn't exactly build packages and then put them in a repository.
Well, technically there is a repo metadata 'view' of each Koji tag. You can find those at kojipkgs.fedoraproject.org/repos/ - each tag has a directory there, and you can find repodata under each directory which gives you a repo 'view' of the latest builds for that tag. But those are not things that are really intended to be consumed by Fedora users or testers in most typical workflows.
I think the best way to think of it in terms of how most processes work is that Koji builds packages and then they just sort of...sit in Koji to be used as inputs to other things. A "compose" is the mechanism by which we take builds from Koji and produce the things that Fedora users 'consume', including images and the formal public-facing repositories.
So, just running a build in Koji does not necessarily result in it appearing in any conventionally-public-facing repository. For Rawhide there are some convenience mechanisms hooked up so when a packager does a build the "usual" way, it automatically gets tagged for Rawhide and included in the next compose, but even that can be avoided if desired. For a build to appear in a public repo, some kind of compose step has to happen in between.
To answer q2 directly - the nightly compose process for Rawhide and Branched uses the packages tagged with the stable tag for the release in question. So currently Rawhide composes use all packages tagged 'f35'. When F35 branches, Branched composes will use all packages tagged 'f35' and Rawhide composes will use all packages tagged 'f36'.
The composes that are run regularly to update the stable release update repositories use the '-updates' tags - so when we run a compose to update the F34 stable updates repo, for instance, it uses all packages tagged 'f34-updates'. (There are also composes run to update the updates-testing repositories for stable releases and Branched; these use the 'fXX-updates-testing' tags, not surprisingly). When a stable release update is "pushed stable" via Bodhi, what actually *happens* that makes it ultimately show up in the public stable updates repo is that it has the 'fXX-updates' tag applied to it in Koji, so the next stable update compose run pulls it in. (For Branched and Rawhide, when an update is 'pushed stable' via Bodhi, it has the fXX' tag applied, so the next nightly compose pulls it in).
Note that your q2 and q3 are phrased slightly wrong, btw. It's nightly composes that use "[all] stable packages" as input. Those composes are automated and use *only* the packages tagged 'f35' or 'f36' or whatever. Candidate composes can potentially pull in additional packages. The difference between a candidate compose and a nightly compose is some slightly different naming, configuration and metadata produced by running the compose tool differently, *plus* the potential to manually include packages that have not yet been 'pushed stable', which is done by releng by hand. Here's an example from the last release:
https://pagure.io/releng/issue/10093#comment-729024
that was the candidate compose request I filed that resulted in F34 Final RC2 being built, which is ultimately what was released as F34 Final. As you can see, I listed 9 updates to be included in the compose which were not 'stable' at the time. The automated nightly compose run on the same day would not have included those updates. In order to include them, releng would have tagged them into a side repository that is included in the candidate compose configuration before running the compose. I actually forget what that repo is called :P Again, mboddu could tell you. All of the above is my best recollection/understanding, btw - if Mohan or Kevin Fenzi tell you something different, they're probably right. -- Adam Williamson Fedora QA IRC: adamw | Twitter: adamw_ha https://www.happyassassin.net
test mailing list -- test@lists.fedoraproject.org To unsubscribe send an email to test-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/test@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
On Sun, 2021-06-20 at 15:22 -0300, Beatriz Michelson Reichert wrote:
Hi Adam,
Thank you so much for replying, it helped a lot!
No problem! I'm aware this is all a bit complicated and sometimes I struggle to actually write it all out clearly even if I know (or...think I "know" :P) how it works in my head. So please let me know if anything's not clear or seems confusing, and it'd definitely be a good idea to check with Mohan and/or Kevin for their perspective too, as they're the ones actually doing/overseeing the releng side of things - it may look different to them than it does to me.
Unfortunately right now a lot of the release cycle isn't "happening" because we're in the sort of lull period after a release and before the next branch point, so you can't see a lot of this stuff in action. 35 is scheduled to branch on August 10th enter Beta freeze on August 24th, and you'd be able to watch and see how things really happen at that point. Not sure how that works with your schedule.
On Mon, Jun 21, 2021 at 12:13 AM Adam Williamson adamwill@fedoraproject.org wrote:
On Sun, 2021-06-20 at 15:22 -0300, Beatriz Michelson Reichert wrote:
Hi Adam,
Thank you so much for replying, it helped a lot!
No problem! I'm aware this is all a bit complicated and sometimes I struggle to actually write it all out clearly even if I know (or...think I "know" :P) how it works in my head. So please let me know if anything's not clear or seems confusing, and it'd definitely be a good idea to check with Mohan and/or Kevin for their perspective too, as they're the ones actually doing/overseeing the releng side of things
- it may look different to them than it does to me.
Unfortunately right now a lot of the release cycle isn't "happening" because we're in the sort of lull period after a release and before the next branch point, so you can't see a lot of this stuff in action.
+1 and Beatriz, if you are going to publish the study, do share it here!
35 is scheduled to branch on August 10th enter Beta freeze on August 24th, and you'd be able to watch and see how things really happen at that point. Not sure how that works with your schedule. -- Adam Williamson Fedora QA IRC: adamw | Twitter: adamw_ha https://www.happyassassin.net
test mailing list -- test@lists.fedoraproject.org To unsubscribe send an email to test-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/test@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
-
Adam Williamson -
Beatriz Michelson Reichert -
Sudhir Dharanendraiah