12:33 p.m.
https://fedoraproject.org/wiki/Changes/GatingRawhidePackages
Note that this was previously submitted as "Gating Rawhide - Single
package updates":
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.o...
== Summary ==
We want to gate packages on test results before they can land in
rawhide. This will reduce the amount of broken dependency,
uninstallable packages and broken composes leading to a more stable
rawhide as well as less work on the infrastructure and rel-eng teams
to keep composes working.
== Owner ==
* Name: [[User:pingou|Pierre-Yves Chibon]]
* Email: pingou - pingoured.fr
People who are/will be involved in this:
* Coordinator: [[User:pingou|Pierre-Yves Chibon]]
* Bodhi: [[User:bowlofeggs|Randy Barlow]], [[User:abompard|Aurélien
Bompard]] and [[User:cverna|Clément Verna]]
* koji: [[User:mizdebsk|Mikolaj Izdebski]] and [[User:mikem|Mike McLean]]
* fedpkg/rpkg: [[User:cqi|Chenxiong ]] and [[User:lsedlar|Lubomír Sedlář]]
* Fedora CI: [[User:dperpeet|Dominik Perpeet]]
== Detailed Description ==
Rawhide is an unique place in the Fedora eco-system. It is the only
place where large changes and rebuilds can be done. In stable Fedora
releases, the rebuilds of large sets of packages is discouraged by the
packaging guidelines, so the ratio of number of updates in bodhi which
a single build vs multiple builds (95% to 5%) is not reflective of the
reality of the rawhide ecosystem.
The basic principal of this proposal is to provide an environment in
which packages can be built and tested without affecting other
packages.
When considering gating rawhide package updates on test results, we
need to consider two workflows: single build updates, multi-builds
updates. For single build, the easiest solution is to provide a
dedicated koji tag in which these packages are built and where they
will wait for their tests to pass before they can enter the buildroot.
For multi-builds, the solution will be to rely on side-tags in koji.
These side-tags are basically tags created by the packager and
available to them to do their work, in this case rebuild all the
packages desired. The side-tag can then sent to the test system as
being one unit of change.
To use a, now well-known, analogy, a single-build update is like
sending a commit to a mailing list, it waits there to be reviewed and
tested before being merged into the main repository. Multi-build
updates are like pull-requests, they can contain one or more builds
and which are reviewed and tested all together as one change before
being merged.
Within this proposal, we aim at building the infrastructure allowing
to gate packages in rawhide. The goal is for packages to go through
bodhi, be tested and if the tests pass, land in the rawhide buildroot
as they do today. In the simplest case, the packager workflow will not
be affected by this proposal, less simple situation will require
adjustements to the packager workflow that we would like to try to be
as minimal as possible.
We do not aim at making any tests mandatory as part of these
proposals. Once packages gating is in place it will be up to the
community and likely FESCo to decide if they would like to enforce
some rules on all packages and which ones.
Note that this proposal completes previous initiative such as
[[Changes/NoMoreAlpha]].
=== Workflows ===
<gallery>
Single_package_GatingRawhide_bodhi.png|Single package update with gating
GatingRawhide_MultiplePkgs_bodhi.png|Multi-packages update with gating
</gallery>
== Benefit to Fedora ==
* As more packagers opt-into gating and add tests to their packages,
rawhide becomes more stable
* More stable rawhide will lead to easier composes and a smoother
release process
* Ability to use chain-build in rawhide and stable releases
== Scope ==
The scope of this work is adding a mechanism to gate builds/updates
before they enter the rawhide buildroot (and thus become accessible to
others) so broken packages are kept out and cannot affect other
packages or the compose until they are fixed by their maintainers.
'''The CI system, the tests and the decision on which tests are used
to gate upon are out of scope for the present document.'''
* Proposal owners: pingou will be coordinating the work among the
different stack holders
* Other developers:
** Bodhi developers: Implement the needed changes
** koji developers: Implement the needed changes
** fedpkg/rpkg developers: Implement the needed changes
** infrastructure: deploy the new version of bodhi and koji
* Release engineering: [https://pagure.io/releng/issue/8183 #8183]
* Policies and guidelines: Once the entire system is in place, FESCo
may want to set a policy on distribution-wide gating (ie: tests that
would be enforced for all packages in the distributions). This is
however out of scope of this proposal.
* Trademark approval: N/A (not needed for this Change)
=== Application impacted ===
==== Bodhi ====
* A message-bus listener will automatically create a bodhi update for
each package built in the rawhide-gated tag.
* A message-bus listener will automatically push bodhi updates created
for rawhide that have passed CI (the decision being announced by
Greenwave).
* A message-bus listener will automatically comment on bodhi update
about update in greenwave's decision
** This will notify users about the test results and the corresponding
gating status.
* bodhi will support creating a new update from a side-tag instead of
a list of builds.
==== Koji ====
* A koji plugin will allow packagers to create new side-tags
* How side-tags are created will be optimized so they do not take
hours to be available to the user requesting them
==== fedpkg/rpkg ====
* The command: `fedpkg side-tag create` will be added to create a new
side-tag in koji for this user using the new plugin
* The command: `fedpkg side-tag merge` will be added to create a bodhi
update corresponding from this side-tag
** Alternatively: Expand the existing `fedpkg update` command to add
support for creating a new update from a side-tag
==== Greenwave / WaiverDB ====
Nothing should change for these tools but we will have to confirm this
and test in staging that they behave as expected.
==== CI system ====
Nothing should change for the CI system but we will have to confirm
this and test in staging that they behave as expected.
== Upgrade/compatibility impact ==
N/A
== How To Test ==
=== Single build ===
==== With opting in into gating ====
# Add tests to a package you maintain in Fedora using the
[https://docs.fedoraproject.org/en-US/ci/quick-start-guide/
ci/quick-start-guide] documentation (see the full
[https://docs.fedoraproject.org/en-US/ci/ CI documentation] for more
information).
# Update your package in rawhide
# Ensure that the bodhi update is automatically created
# Check that the tests have properly ran (see the Tests tab in the bodhi update)
# Ensure that the bodhi update went through to rawhide once the tests passed
If one or more tests failed and you wish to waive them
# Waive the failing tests using `bodhi updates waive <id>` see `bodhi
updates waive --help` for more information
==== Without opting in into gating ====
# Ensure your package does not have a `gating.yaml` in its dist-git repository
# Build your package in rawhide as you do today
# Ensure that the bodhi update is automatically created
# Ensure that the bodhi update goes through to rawhide automatically
=== Multi-builds ===
# Add tests to a package you maintain in Fedora using the
[https://docs.fedoraproject.org/en-US/ci/quick-start-guide/
ci/quick-start-guide] documentation (see the full
[https://docs.fedoraproject.org/en-US/ci/ CI documentation] for more
information).
# Request a new side-tag in koji for your rebuilds
# Update your packages in this side-tag
# Request for this side-tag to be merged either using fedpkg or
bodhi's UI (create a new update from a side-tag)
# Check that the tests have properly ran (see the Tests tab in the bodhi update)
# Ensure that the bodhi update went through to rawhide once the tests passed
# Check that the side-tag was correctly cleaned afterward
==== Without opting in into gating ====
# Ensure all your packages do not have a `gating.yaml` in its dist-git
repository
# Build your packages in rawhide as you do today
# Ensure that the bodhi updates are automatically created
# Ensure that the bodhi updates go through to rawhide automatically
== User Experience ==
=== Musts ===
Here below are the list of requirements for this proposal:
* Packagers must be notified when the update corresponding to their
build(s) passes or fails CI
* Packagers must be able to see what is blocking an update (ie missing
requirements/test results)
* Packagers must be able to re-trigger a test run (via either an UI or a CLI)
* Packagers must be able to waive false-negative (via either an UI or a CLI)
=== Nice to have ===
Here below is a list of ideas that would make the user experience more
friendly but will not be part of the initial release:
* Packagers would like to follow the progress of the tests
** This could be implemented by notifying the packagers when a test
run starts and provide an URL where they could see and follow them
* Packagers would like to be able to re-trigger a test run from an UI and a CLI
* Packagers would like to be able to waive false-negative from an UI and a CLI
* Packagers would like the ability to transform a bodhi update
automatically created for a single build into a side-tag-specifi bodhi
update
** This would allow the workflow where a packager makes a build, which
fails CI because another package needs to be rebuilt as well. The
packager can then convert the bodhi update into a side-tag based
update, this would create the corresponding side-tag and move the
existing build into it, allowing the packager to do the missing
rebuilds before telling bodhi to process the update. This is entirely
doable by hand, which is why this feature while convenient is
considered a nice to have
* Packagers would like fedpkg chain-build to create the corresponding
side-tag automatically
=== Single build update ===
fedpkg clone rpms/foobar
cd foobar/
vim foobar.spec
git commit -am "Git commit message"
fedpkg build
If the CI tests pass, the package will land in rawhide, if they fail the user
will be able to go to bodhi to see what failed and why as well as waiving the
failed tests if needed.
=== Multi builds with gating ===
fedpkg clone rpms/foobar
cd foobar/
fedpkg side-tag create foo1.2
vim foobar.spec
git commit -am "Update of foobar to update foo to 1.2"
fedpkg push
fedpkg build --target=user_foo1.2
cd ..
fedpkg clone rpms/foobaz
cd foobaz
vim foobaz.spec
git commit -am "Update of foobaz to update foo to 1.2"
fedpkg push
fedpkg build --target=user_foo1.2
cd ..
fedpkg clone rpms/foo
cd foo
vim foo.spec
git commit -am "Update foo to 1.2"
fedpkg push
fedpkg build --target=user_foo1.2
fedpkg side-tag merge foo1.2
== Dependencies ==
* Bodhi
* koji
* fedpkg
== Contingency Plan ==
We can keep doing rawhide as we are doing it today.
== Documentation ==
To be written/updated
List of documentation pages to update:
* https://fedoraproject.org/wiki/Updates_Policy
* https://fedoraproject.org/wiki/Package_maintenance_guide
* https://fedoraproject.org/wiki/Package_update_HOWTO
* https://docs.fedoraproject.org/en-US/packaging-guidelines/
--
Ben Cotton
Fedora Program Manager
TZ=America/Indiana/Indianapolis
Pronouns: he/him