3:19 p.m.
https://fedoraproject.org/wiki/Changes/GatingRawhideSinglePackageUpdates
== Summary ==
We want to gate packages on test results before they can land in
rawhide. This will reduce the amount of broken dependency,
uninstallable packages and broken composes leading to a more stable
rawhide as well as less work on the infrastructure and rel-eng teams
to keep composes working.
This project will be split in two phases, at first only single package
updates will be supported, in a second stage, we will add support for
multi-packages updates.
This proposal is about the phase 1 of this project.
== Owner ==
* Name: [[User:pingou|Pierre-Yves Chibon]]
* Email: pingou - pingoured.fr
People who are/will be involved in this:
* Coordinator: [[User:pingou|Pierre-Yves Chibon]]
* Bodhi: [[User:bowlofeggs|Randy Barlow]] and [[User:abompard|Aurélien Bompard]]
* Fedora CI: [[User:dperpeet|Dominik Perpeet]]
== Detailed Description ==
Querying the Bodhi database, it was revealed that 95% of all our
updates involve a single package.
To be exhaustive, here are the exact number found by Randy:
Of all time:
Single build updates: 123,657 (94.1%)
Multi build updates: 7,766 ( 5.9%)
Total: 131,423
Per release:
Fedora 29:
Single: 4,675 (93.7%)
Multi: 316 ( 6.3%)
Fedora 28:
Single: 9,153 (94.5%)
Multi: 536 ( 5.5%)
EPEL 7:
Single: 12,664 (94.0%)
Multi: 814 ( 6.0%)
When considering gating rawhide package updates on test results, we
need to consider two workflows: single package updates, multi-package
updates. This proposal only considers the single package updates
workflow. [[Changes/GatingRawhideMultiPackagesUpdates|Another
proposal]] will be submitted in the future to support the multi
package updates workflow.
At first we want to build the system allowing to gating rawhide,
packagers will '''opt-in into gating'''
[https://docs.pagure.org/greenwave/package-specific-policies.html
using greenwave's opt-in gating mechanism] as they want. We will also
offer a way to '''opt-out''' so packages depending on other
packages
can still be built without issues.
We do not aim at making any tests mandatory as part of these
proposals. Once the two proposals have been deployed it will be up to
the community and likely FESCo to decide if they would like to enforce
some rules on all packages and which ones.
Note that this proposal completes previous initiative such as
[[Changes/NoMoreAlpha]]
=== Workflows ===
<gallery>
Single_package_GatingRawhide_bodhi.png|Single package update with gating
</gallery>
== Benefit to Fedora ==
* As more packagers opt-into gating and add tests to their packages,
rawhide becomes more stable
* More stable rawhide will lead to easier composes and a smoother
release process
* Ability to use chain-build in rawhide and stable releases
== Scope ==
The scope of this work is adding a mechanism to gate single package
updates before they enter the rawhide buildroot (and thus become
accessible to others) so broken packages are kept out and cannot
affect other packages or the compose until they are fixed by their
maintainers.
'''The CI system, the tests and the decision on which tests are used
to gate upon are out of scope for the present document.'''
* Proposal owners: pingou will be coordinating the work among the
different stack holder
* Other developers:
** Bodhi developers: Implement the needed changes
** infrastructure: deploy the new version of bodhi and the new koji plugin
* Release engineering: https://pagure.io/releng/issue/8183
* Policies and guidelines: Once the entire system is in place, FESCo
may want to set a policy on distribution-wide gating (ie: tests that
would be enforced for all packages in the distributions). This is
however out of scope of this proposal.
* Trademark approval: N/A (not needed for this Change)
=== Application impacted ===
==== Bodhi ====
* Single package update
** We will need to write a message-bus listener that will
automatically create a bodhi update for each package built in the
rawhide-gated tag.
** We will need to write a message-bus listener to automatically push
bodhi updates created for rawhide that have passed CI (the decision
being announced by Greenwave).
* Bodhi will comment on the updates when greenwave's decision about an
update is known
** This will notify users about the test results and the corresponding
gating status.
==== Greenwave / WaiverDB ====
Nothing should change for these tools but we will have to confirm this
and test in staging that they behave as expected.
==== CI system ====
Nothing should change for the CI system but we will have to confirm
this and test in staging that they behave as expected.
== Upgrade/compatibility impact ==
N/A
== How To Test ==
# Add tests to a package you maintain in Fedora using the [[
CI/Quick_Start_Guide ]] documentation (see the full [[CI|CI
documentation ]] for more information).
# Update your package in rawhide
# Ensure that the bodhi update is automatically created
# Check that the tests have properly ran (see the Tests tab in the bodhi update)
# Ensure that the bodhi update went through to rawhide once the tests passed
== User Experience ==
=== Single package update ===
fedpkg clone rpms/foobar
cd foobar/
vim foobar.spec
git commit -am "Git commit message"
fedpkg build
If the CI tests pass, the package will land in rawhide, if they fail the user
will be able to go to bodhi to see what failed and why as well as waiving the
failed tests if needed.
=== Multi package update with single package update gating ===
fedpkg clone rpms/foobar
cd foobar/
vim foobar.spec
git commit -am "Update of foobar to update foo to 1.2"
fedpkg push
fedpkg build --target=rawhide-candidate
cd ..
fedpkg clone rpms/foobaz
cd foobaz
vim foobaz.spec
git commit -am "Update of foobaz to update foo to 1.2"
fedpkg push
fedpkg build --target=rawhide-candidate
cd ..
fedpkg clone rpms/foo
cd foo
vim foo.spec
git commit -am "Update foo to 1.2"
fedpkg push
fedpkg build --target=rawhide-candidate
Users are practically by-passing the gating process by specifying a
specific build target.
== Dependencies ==
* Bodhi
** bus listener to automatically create updates when a build happens in rawhide
** bus listener to automatically push updates who passed gating
according to greenwave
** comment on updates about gating status change/announce to notify
packagers about it
== Contingency Plan ==
We can keep doing rawhide as we are doing it today.
== Documentation ==
To be written/updated
List of documentation pages to update:
* https://fedoraproject.org/wiki/Updates_Policy
* https://fedoraproject.org/wiki/Package_maintenance_guide
* https://fedoraproject.org/wiki/Package_update_HOWTO
* https://docs.fedoraproject.org/en-US/packaging-guidelines/
--
Ben Cotton
Fedora Program Manager
TZ=America/Indiana/Indianapolis
8:57 p.m.
On 03/01/2019 01:19 PM, Ben Cotton wrote:
https://fedoraproject.org/wiki/Changes/GatingRawhideSinglePackageUpdates
== Summary ==
We want to gate packages on test results before they can land in
rawhide. This will reduce the amount of broken dependency,
uninstallable packages and broken composes leading to a more stable
rawhide as well as less work on the infrastructure and rel-eng teams
to keep composes working.
Does the gating prevent packages from being tagged at all so that they
won't even end up in the buildroot, or does it just gate packages from
going into a compose?
<one more comment below>
This project will be split in two phases, at first only single
package
updates will be supported, in a second stage, we will add support for
multi-packages updates.
This proposal is about the phase 1 of this project.
== Owner ==
* Name: [[User:pingou|Pierre-Yves Chibon]]
* Email: pingou - pingoured.fr
People who are/will be involved in this:
* Coordinator: [[User:pingou|Pierre-Yves Chibon]]
* Bodhi: [[User:bowlofeggs|Randy Barlow]] and [[User:abompard|Aurélien Bompard]]
* Fedora CI: [[User:dperpeet|Dominik Perpeet]]
== Detailed Description ==
Querying the Bodhi database, it was revealed that 95% of all our
updates involve a single package.
To be exhaustive, here are the exact number found by Randy:
Of all time:
Single build updates: 123,657 (94.1%)
Multi build updates: 7,766 ( 5.9%)
Total: 131,423
Per release:
Fedora 29:
Single: 4,675 (93.7%)
Multi: 316 ( 6.3%)
Fedora 28:
Single: 9,153 (94.5%)
Multi: 536 ( 5.5%)
EPEL 7:
Single: 12,664 (94.0%)
Multi: 814 ( 6.0%)
When considering gating rawhide package updates on test results, we
need to consider two workflows: single package updates, multi-package
updates. This proposal only considers the single package updates
workflow. [[Changes/GatingRawhideMultiPackagesUpdates|Another
proposal]] will be submitted in the future to support the multi
package updates workflow.
At first we want to build the system allowing to gating rawhide,
packagers will '''opt-in into gating'''
[https://docs.pagure.org/greenwave/package-specific-policies.html
using greenwave's opt-in gating mechanism] as they want. We will also
offer a way to '''opt-out''' so packages depending on other
packages
can still be built without issues.
We do not aim at making any tests mandatory as part of these
proposals. Once the two proposals have been deployed it will be up to
the community and likely FESCo to decide if they would like to enforce
some rules on all packages and which ones.
Note that this proposal completes previous initiative such as
[[Changes/NoMoreAlpha]]
=== Workflows ===
<gallery>
Single_package_GatingRawhide_bodhi.png|Single package update with gating
</gallery>
== Benefit to Fedora ==
* As more packagers opt-into gating and add tests to their packages,
rawhide becomes more stable
* More stable rawhide will lead to easier composes and a smoother
release process
* Ability to use chain-build in rawhide and stable releases
== Scope ==
The scope of this work is adding a mechanism to gate single package
updates before they enter the rawhide buildroot (and thus become
accessible to others) so broken packages are kept out and cannot
affect other packages or the compose until they are fixed by their
maintainers.
'''The CI system, the tests and the decision on which tests are used
to gate upon are out of scope for the present document.'''
* Proposal owners: pingou will be coordinating the work among the
different stack holder
* Other developers:
** Bodhi developers: Implement the needed changes
** infrastructure: deploy the new version of bodhi and the new koji plugin
* Release engineering: https://pagure.io/releng/issue/8183
* Policies and guidelines: Once the entire system is in place, FESCo
may want to set a policy on distribution-wide gating (ie: tests that
would be enforced for all packages in the distributions). This is
however out of scope of this proposal.
* Trademark approval: N/A (not needed for this Change)
=== Application impacted ===
==== Bodhi ====
* Single package update
** We will need to write a message-bus listener that will
automatically create a bodhi update for each package built in the
rawhide-gated tag.
** We will need to write a message-bus listener to automatically push
bodhi updates created for rawhide that have passed CI (the decision
being announced by Greenwave).
* Bodhi will comment on the updates when greenwave's decision about an
update is known
** This will notify users about the test results and the corresponding
gating status.
==== Greenwave / WaiverDB ====
Nothing should change for these tools but we will have to confirm this
and test in staging that they behave as expected.
==== CI system ====
Nothing should change for the CI system but we will have to confirm
this and test in staging that they behave as expected.
I think this issue might have to be fixed first, otherwise we would end
up with a lot of false negatives:
https://pagure.io/fedora-ci/general/issue/20
-Tom
== Upgrade/compatibility impact ==
N/A
== How To Test ==
# Add tests to a package you maintain in Fedora using the [[
CI/Quick_Start_Guide ]] documentation (see the full [[CI|CI
documentation ]] for more information).
# Update your package in rawhide
# Ensure that the bodhi update is automatically created
# Check that the tests have properly ran (see the Tests tab in the bodhi update)
# Ensure that the bodhi update went through to rawhide once the tests passed
== User Experience ==
=== Single package update ===
fedpkg clone rpms/foobar
cd foobar/
vim foobar.spec
git commit -am "Git commit message"
fedpkg build
If the CI tests pass, the package will land in rawhide, if they fail the user
will be able to go to bodhi to see what failed and why as well as waiving the
failed tests if needed.
=== Multi package update with single package update gating ===
fedpkg clone rpms/foobar
cd foobar/
vim foobar.spec
git commit -am "Update of foobar to update foo to 1.2"
fedpkg push
fedpkg build --target=rawhide-candidate
cd ..
fedpkg clone rpms/foobaz
cd foobaz
vim foobaz.spec
git commit -am "Update of foobaz to update foo to 1.2"
fedpkg push
fedpkg build --target=rawhide-candidate
cd ..
fedpkg clone rpms/foo
cd foo
vim foo.spec
git commit -am "Update foo to 1.2"
fedpkg push
fedpkg build --target=rawhide-candidate
Users are practically by-passing the gating process by specifying a
specific build target.
== Dependencies ==
* Bodhi
** bus listener to automatically create updates when a build happens in rawhide
** bus listener to automatically push updates who passed gating
according to greenwave
** comment on updates about gating status change/announce to notify
packagers about it
== Contingency Plan ==
We can keep doing rawhide as we are doing it today.
== Documentation ==
To be written/updated
List of documentation pages to update:
* https://fedoraproject.org/wiki/Updates_Policy
* https://fedoraproject.org/wiki/Package_maintenance_guide
* https://fedoraproject.org/wiki/Package_update_HOWTO
* https://docs.fedoraproject.org/en-US/packaging-guidelines/
3:02 a.m.
On Fri, Mar 01, 2019 at 06:57:48PM -0800, Tom Stellard wrote:
On 03/01/2019 01:19 PM, Ben Cotton wrote:
> https://fedoraproject.org/wiki/Changes/GatingRawhideSinglePackageUpdates
>
> == Summary ==
> We want to gate packages on test results before they can land in
> rawhide. This will reduce the amount of broken dependency,
> uninstallable packages and broken composes leading to a more stable
> rawhide as well as less work on the infrastructure and rel-eng teams
> to keep composes working.
>
Does the gating prevent packages from being tagged at all so that they
won't even end up in the buildroot, or does it just gate packages from
going into a compose?
It gates the package from entering the buildroot, which will impact the packages
going into a compose, but as a side effect.
[...]
> ==== CI system ====
> Nothing should change for the CI system but we will have to confirm
> this and test in staging that they behave as expected.
>
I think this issue might have to be fixed first, otherwise we would end
up with a lot of false negatives:
https://pagure.io/fedora-ci/general/issue/20
Thanks for your feedback, I'll follow up on this ticket to see if we can get it
sorted out.
Pierre
8:24 a.m.
On 3/2/19 5:02 PM, Pierre-Yves Chibon wrote:
On Fri, Mar 01, 2019 at 06:57:48PM -0800, Tom Stellard wrote:
> On 03/01/2019 01:19 PM, Ben Cotton wrote:
>> https://fedoraproject.org/wiki/Changes/GatingRawhideSinglePackageUpdates
>>
>> == Summary ==
>> We want to gate packages on test results before they can land in
>> rawhide. This will reduce the amount of broken dependency,
>> uninstallable packages and broken composes leading to a more stable
>> rawhide as well as less work on the infrastructure and rel-eng teams
>> to keep composes working.
>>
>
> Does the gating prevent packages from being tagged at all so that they
> won't even end up in the buildroot, or does it just gate packages from
> going into a compose?
It gates the package from entering the buildroot, which will impact the packages
going into a compose, but as a side effect.
Given it blocks packages from entering buildroot, I wonder if it is a
good idea to start gating whole Rawhide lifetime, I mean, from the
starting of a ready-to-release release branched out of Rawhide?
My case is, we have a set of packages to update each release. They
cannot do in parallel - they depend on one another. Currently we only
update them in Rawhide, because each package can get into buildroot
shortly after we build it, and we don't need to file a
buildroot-override. Once even packages cannot get into Rawhide
automatically (for example, I need to click a "waive test result" or
something), this is more like a burden.
As for the Single build updates vs multi build updates ratio, I don't
quite understand what the number is from - does it comes out of Bodhi?
If it means the updates in Bodhi, I want to mention that, in my case, I
never want to update multi build updates in a stable (or post-freeze)
release. Thus I seldom file multi build updates in bodhi. Especially we
don't need to file Bodhi to get packages into Rawhide at this point,
this maybe misleading in deciding to enable gating for Rawhide.
As a summarize, I think it is a good idea to have some sort of gating.
But I think we need to think carefully if we do really need to gate Rawhide.
[...]
>> ==== CI system ====
>> Nothing should change for the CI system but we will have to confirm
>> this and test in staging that they behave as expected.
>>
> I think this issue might have to be fixed first, otherwise we would end
> up with a lot of false negatives:
> https://pagure.io/fedora-ci/general/issue/20
Thanks for your feedback, I'll follow up on this ticket to see if we can get it
sorted out.
Pierre
_______________________________________________
devel mailing list -- devel(a)lists.fedoraproject.org
To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
--
Zamir SUN
GPG : 1D86 6D4A 49CE 4BBD 72CF FCF5 D856 6E11 F2A0 525E
Want to know more about Fedora?
Visit https://fedoraproject.org/wiki/
Ready to contribute? See https://whatcanidoforfedora.org/
想了解更多中文资讯，访问 https://zh.fedoracommunity.org/
9:06 a.m.
On Thu, Mar 07, 2019 at 10:24:00PM +0800, Zamir SUN wrote:
On 3/2/19 5:02 PM, Pierre-Yves Chibon wrote:
> On Fri, Mar 01, 2019 at 06:57:48PM -0800, Tom Stellard wrote:
>> On 03/01/2019 01:19 PM, Ben Cotton wrote:
>>> https://fedoraproject.org/wiki/Changes/GatingRawhideSinglePackageUpdates
>>>
>>> == Summary ==
>>> We want to gate packages on test results before they can land in
>>> rawhide. This will reduce the amount of broken dependency,
>>> uninstallable packages and broken composes leading to a more stable
>>> rawhide as well as less work on the infrastructure and rel-eng teams
>>> to keep composes working.
>>>
>>
>> Does the gating prevent packages from being tagged at all so that they
>> won't even end up in the buildroot, or does it just gate packages from
>> going into a compose?
>
> It gates the package from entering the buildroot, which will impact the packages
> going into a compose, but as a side effect.
>
Given it blocks packages from entering buildroot, I wonder if it is a
good idea to start gating whole Rawhide lifetime, I mean, from the
starting of a ready-to-release release branched out of Rawhide?
My case is, we have a set of packages to update each release. They
cannot do in parallel - they depend on one another. Currently we only
update them in Rawhide, because each package can get into buildroot
shortly after we build it, and we don't need to file a
buildroot-override. Once even packages cannot get into Rawhide
automatically (for example, I need to click a "waive test result" or
something), this is more like a burden.
So what you are describing is basically the case of multi-packages update. You
want to ship as one packages that depend on each other and should be ship as one
unit.
The current approach specifically does not support this use case.
There are two ways to go about this:
- Do not opt-in into the gating (ie: do not add a gating.yaml in your dist-git
repo)
- Do opt-in but when you need something faster, opt-out, by simply building the
package in rawhide-candidate (or its equivalent) directly.
As for the Single build updates vs multi build updates ratio, I
don't
quite understand what the number is from - does it comes out of Bodhi?
They do yes and indeed as you're mentioning they are informative but likely not
fully representative of rawhide.
If it means the updates in Bodhi, I want to mention that, in my case,
I
never want to update multi build updates in a stable (or post-freeze)
release. Thus I seldom file multi build updates in bodhi. Especially we
don't need to file Bodhi to get packages into Rawhide at this point,
this maybe misleading in deciding to enable gating for Rawhide.
The bodhi updates will be filled automatically and if tests pass (or if there
are no gating.yaml) the build will land in rawhide, automatically as well, just
as it does today.
The point of bodhi is to give us a way to see in which state is a package or why
a package that was built did not land in the rawhide buildroot in a place that
is accessible to everyone in the community.
As a summarize, I think it is a good idea to have some sort of
gating.
But I think we need to think carefully if we do really need to gate Rawhide.
I don't think there are so many ways to stabilize rawhide so that compose (for
example) pass more often than they fail (there hasn't been a successful rawhide
compose in a week). CI and gating is one of them and one that is pretty standard
in our industry.
Pierre
9:21 a.m.
On Thu, Mar 7, 2019 at 10:07 AM Pierre-Yves Chibon <pingou(a)pingoured.fr> wrote:
On Thu, Mar 07, 2019 at 10:24:00PM +0800, Zamir SUN wrote:
>
>
> On 3/2/19 5:02 PM, Pierre-Yves Chibon wrote:
> > On Fri, Mar 01, 2019 at 06:57:48PM -0800, Tom Stellard wrote:
> >> On 03/01/2019 01:19 PM, Ben Cotton wrote:
> >>>
https://fedoraproject.org/wiki/Changes/GatingRawhideSinglePackageUpdates
> >>>
> >>> == Summary ==
> >>> We want to gate packages on test results before they can land in
> >>> rawhide. This will reduce the amount of broken dependency,
> >>> uninstallable packages and broken composes leading to a more stable
> >>> rawhide as well as less work on the infrastructure and rel-eng teams
> >>> to keep composes working.
> >>>
> >>
> >> Does the gating prevent packages from being tagged at all so that they
> >> won't even end up in the buildroot, or does it just gate packages from
> >> going into a compose?
> >
> > It gates the package from entering the buildroot, which will impact the
packages
> > going into a compose, but as a side effect.
> >
>
> Given it blocks packages from entering buildroot, I wonder if it is a
> good idea to start gating whole Rawhide lifetime, I mean, from the
> starting of a ready-to-release release branched out of Rawhide?
>
> My case is, we have a set of packages to update each release. They
> cannot do in parallel - they depend on one another. Currently we only
> update them in Rawhide, because each package can get into buildroot
> shortly after we build it, and we don't need to file a
> buildroot-override. Once even packages cannot get into Rawhide
> automatically (for example, I need to click a "waive test result" or
> something), this is more like a burden.
So what you are describing is basically the case of multi-packages update. You
want to ship as one packages that depend on each other and should be ship as one
unit.
The current approach specifically does not support this use case.
There are two ways to go about this:
- Do not opt-in into the gating (ie: do not add a gating.yaml in your dist-git
repo)
- Do opt-in but when you need something faster, opt-out, by simply building the
package in rawhide-candidate (or its equivalent) directly.
> As for the Single build updates vs multi build updates ratio, I don't
> quite understand what the number is from - does it comes out of Bodhi?
They do yes and indeed as you're mentioning they are informative but likely not
fully representative of rawhide.
> If it means the updates in Bodhi, I want to mention that, in my case, I
> never want to update multi build updates in a stable (or post-freeze)
> release. Thus I seldom file multi build updates in bodhi. Especially we
> don't need to file Bodhi to get packages into Rawhide at this point,
> this maybe misleading in deciding to enable gating for Rawhide.
The bodhi updates will be filled automatically and if tests pass (or if there
are no gating.yaml) the build will land in rawhide, automatically as well, just
as it does today.
The point of bodhi is to give us a way to see in which state is a package or why
a package that was built did not land in the rawhide buildroot in a place that
is accessible to everyone in the community.
> As a summarize, I think it is a good idea to have some sort of gating.
> But I think we need to think carefully if we do really need to gate Rawhide.
I don't think there are so many ways to stabilize rawhide so that compose (for
example) pass more often than they fail (there hasn't been a successful rawhide
compose in a week). CI and gating is one of them and one that is pretty standard
in our industry.
I hate to be the downer here, but I'm generally opposed to this idea
as it stands.
My ability to develop against Rawhide is principally centered around
the fact I can push packages and have them depend on each other.
Today, there is no freely accessible way for people to atomically
merge changes into the distribution. There's no way for people to
freely create spaces to do a bunch of work on a bunch of packages at
once and then merge it.
We kind of fake this by having to ask releng to grant unto a packager
or packagers a side-tag, which work is done there, and then it's
merged back into the distribution. But that model has only worked so
far because we have nothing like this in place for Rawhide, so we can
push all the stuff there, make sure it works, and then merge it into
branches. This is how KDE updates are done, for example. And
occasionally, the GNOME desktop is updated in the same manner.
The fatal misunderstanding here is that people think there's a
distinction between "single" and "multi" updates in Rawhide, when in
fact that doesn't exist. This proposal is doomed to failure because it
assumes stable updates workflows translate to development workflows.
And they don't. The reason why single updates are more common is
because people are ever-so-slightly more cautious, and huge rebuilds
or revdep rebuilds don't happen in stable releases. It doesn't help
that the workflow in Fedora makes multi-package updates rather
painful, but I don't think anyone cares enough to fix that, either...
And I still believe Bodhi is a bad place to do this, because you've
still allowed the package to succeed building into the distribution,
but I think I won't win on this point.
--
真実はいつも一つ！/ Always, there's only one truth!
1:09 a.m.
On Fri, 2019-03-01 at 16:19 -0500, Ben Cotton wrote:
https://fedoraproject.org/wiki/Changes/GatingRawhideSinglePackageUpdates
== Summary ==
We want to gate packages on test results before they can land in
rawhide. This will reduce the amount of broken dependency,
uninstallable packages and broken composes leading to a more stable
rawhide as well as less work on the infrastructure and rel-eng teams
to keep composes working.
This project will be split in two phases, at first only single package
updates will be supported, in a second stage, we will add support for
multi-packages updates.
This proposal is about the phase 1 of this project.
== Owner ==
* Name: [[User:pingou|Pierre-Yves Chibon]]
* Email: pingou - pingoured.fr
People who are/will be involved in this:
* Coordinator: [[User:pingou|Pierre-Yves Chibon]]
* Bodhi: [[User:bowlofeggs|Randy Barlow]] and [[User:abompard|Aurélien Bompard]]
* Fedora CI: [[User:dperpeet|Dominik Perpeet]]
It might be a good idea to have a QA contact here, in case people
choose to block on tests maintained by the QA team (Taskotron or openQA
tests).
== Detailed Description ==
Querying the Bodhi database, it was revealed that 95% of all our
updates involve a single package.
To be exhaustive, here are the exact number found by Randy:
Of all time:
Single build updates: 123,657 (94.1%)
Multi build updates: 7,766 ( 5.9%)
Total: 131,423
Per release:
Fedora 29:
Single: 4,675 (93.7%)
Multi: 316 ( 6.3%)
Fedora 28:
Single: 9,153 (94.5%)
Multi: 536 ( 5.5%)
EPEL 7:
Single: 12,664 (94.0%)
Multi: 814 ( 6.0%)
I'd suggest the implication that single-package updates are "the norm"
is slightly problematic, for two reasons:
1) Rawhide is very different from stable releases, and even from
Branched. Major changes like API breaks are not meant to be sent to
stable releases at all, by policy. So I don't think you can necessarily
rely very strongly on data regarding updates in Branched and stable
releases to draw conclusions about the likely nature of changes in
Rawhide.
2) This does not consider the not uncommon case that updates *should
have been* multi-package updates, but they were done wrong.
When considering gating rawhide package updates on test results, we
need to consider two workflows: single package updates, multi-package
updates. This proposal only considers the single package updates
workflow. [[Changes/GatingRawhideMultiPackagesUpdates|Another
proposal]] will be submitted in the future to support the multi
package updates workflow.
At first we want to build the system allowing to gating rawhide,
packagers will '''opt-in into gating'''
[https://docs.pagure.org/greenwave/package-specific-policies.html
using greenwave's opt-in gating mechanism] as they want. We will also
offer a way to '''opt-out''' so packages depending on other
packages
can still be built without issues.
We do not aim at making any tests mandatory as part of these
proposals. Once the two proposals have been deployed it will be up to
the community and likely FESCo to decide if they would like to enforce
some rules on all packages and which ones.
Note that this proposal completes previous initiative such as
[[Changes/NoMoreAlpha]]
That Change envisaged *both* package build gating *and* compose gating,
so this alone does not really complete it, for the record.
==== Bodhi ====
* Single package update
** We will need to write a message-bus listener that will
automatically create a bodhi update for each package built in the
rawhide-gated tag.
Does this mean that the plan has changed once again and it will no
longer bypass Bodhi and use side tags and integration with fedpkg?
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
3:12 a.m.
On Fri, Mar 01, 2019 at 11:09:48PM -0800, Adam Williamson wrote:
On Fri, 2019-03-01 at 16:19 -0500, Ben Cotton wrote:
> https://fedoraproject.org/wiki/Changes/GatingRawhideSinglePackageUpdates
>
> == Summary ==
> We want to gate packages on test results before they can land in
> rawhide. This will reduce the amount of broken dependency,
> uninstallable packages and broken composes leading to a more stable
> rawhide as well as less work on the infrastructure and rel-eng teams
> to keep composes working.
>
> This project will be split in two phases, at first only single package
> updates will be supported, in a second stage, we will add support for
> multi-packages updates.
>
> This proposal is about the phase 1 of this project.
>
> == Owner ==
> * Name: [[User:pingou|Pierre-Yves Chibon]]
> * Email: pingou - pingoured.fr
>
> People who are/will be involved in this:
> * Coordinator: [[User:pingou|Pierre-Yves Chibon]]
> * Bodhi: [[User:bowlofeggs|Randy Barlow]] and [[User:abompard|Aurélien Bompard]]
> * Fedora CI: [[User:dperpeet|Dominik Perpeet]]
It might be a good idea to have a QA contact here, in case people
choose to block on tests maintained by the QA team (Taskotron or openQA
tests).
Yes and no, yes I'm all in favor of having a contact person from QA to help
volunteer to pick up the right tests for their package, but no as which tests
are used to gate is out of the scope of the proposal itself.
I'd be in favor of adding something like:
"""
If you want to volunteer to opt-in into this workflow and need help figuring out
which tests would be interesting to run in your package, contact foo at
<email/list> or open a ticket at <tracker>
"""
Would that make sense?
If so, where should we recommend packagers to get help?
> == Detailed Description ==
>
> Querying the Bodhi database, it was revealed that 95% of all our
> updates involve a single package.
>
> To be exhaustive, here are the exact number found by Randy:
>
> Of all time:
>
> Single build updates: 123,657 (94.1%)
> Multi build updates: 7,766 ( 5.9%)
>
> Total: 131,423
>
> Per release:
>
> Fedora 29:
>
> Single: 4,675 (93.7%)
> Multi: 316 ( 6.3%)
>
> Fedora 28:
>
> Single: 9,153 (94.5%)
> Multi: 536 ( 5.5%)
>
> EPEL 7:
>
> Single: 12,664 (94.0%)
> Multi: 814 ( 6.0%)
I'd suggest the implication that single-package updates are "the norm"
is slightly problematic, for two reasons:
1) Rawhide is very different from stable releases, and even from
Branched. Major changes like API breaks are not meant to be sent to
stable releases at all, by policy. So I don't think you can necessarily
rely very strongly on data regarding updates in Branched and stable
releases to draw conclusions about the likely nature of changes in
Rawhide.
This is fair, but plays on both side, we're also supposed to send more updates
to rawhide than to stable releases. So there will be more multi-packages updates
in rawhide but also more single-packages updates.
While we may not be at 95 to 5 in rawhide, it is still expected that single
package will be much higher.
2) This does not consider the not uncommon case that updates *should
have been* multi-package updates, but they were done wrong.
That is true, I believe that by only onboarding packagers who want to (opt-in)
and providing a way to by-pass (opt-out) we should be able to cope with this.
[...]
> * Single package update
> ** We will need to write a message-bus listener that will
> automatically create a bodhi update for each package built in the
> rawhide-gated tag.
Does this mean that the plan has changed once again and it will no
longer bypass Bodhi and use side tags and integration with fedpkg?
We have considered a number of options, one of them was to no involve bodhi but
it had a number of disadvantage for Fedora for a similar level of work (amount
and complexity) which made us choose to keep bodhi involved.
Using side-tags and changes to fedpkg will be required for gating multi-packages
updates, they are not required for single package updates (there is a link in
this proposal to the follow-up proposal for multi packages updates, do note that
this is still a draft and thus may change when we get to this workflow).
Pierre
5:14 a.m.
On Sat, Mar 02, 2019 at 10:12:00AM +0100, Pierre-Yves Chibon wrote:
On Fri, Mar 01, 2019 at 11:09:48PM -0800, Adam Williamson wrote:
[...]
> > * Single package update
> > ** We will need to write a message-bus listener that will
> > automatically create a bodhi update for each package built in the
> > rawhide-gated tag.
>
> Does this mean that the plan has changed once again and it will no
> longer bypass Bodhi and use side tags and integration with fedpkg?
We have considered a number of options, one of them was to no involve bodhi but
it had a number of disadvantage for Fedora for a similar level of work (amount
and complexity) which made us choose to keep bodhi involved.
I have a document with the different approaches considered with their pros and
cons, I'll put it on the wiki shortly and send the link here and add it to the
proposal as well. It should give some context about why this design was made
over other alternatives.
Pierre
9:21 a.m.
On Sat, 2019-03-02 at 10:12 +0100, Pierre-Yves Chibon wrote:
On Fri, Mar 01, 2019 at 11:09:48PM -0800, Adam Williamson wrote:
> On Fri, 2019-03-01 at 16:19 -0500, Ben Cotton wrote:
> > https://fedoraproject.org/wiki/Changes/GatingRawhideSinglePackageUpdates
> >
> > == Summary ==
> > We want to gate packages on test results before they can land in
> > rawhide. This will reduce the amount of broken dependency,
> > uninstallable packages and broken composes leading to a more stable
> > rawhide as well as less work on the infrastructure and rel-eng teams
> > to keep composes working.
> >
> > This project will be split in two phases, at first only single package
> > updates will be supported, in a second stage, we will add support for
> > multi-packages updates.
> >
> > This proposal is about the phase 1 of this project.
> >
> > == Owner ==
> > * Name: [[User:pingou|Pierre-Yves Chibon]]
> > * Email: pingou - pingoured.fr
> >
> > People who are/will be involved in this:
> > * Coordinator: [[User:pingou|Pierre-Yves Chibon]]
> > * Bodhi: [[User:bowlofeggs|Randy Barlow]] and [[User:abompard|Aurélien
Bompard]]
> > * Fedora CI: [[User:dperpeet|Dominik Perpeet]]
>
> It might be a good idea to have a QA contact here, in case people
> choose to block on tests maintained by the QA team (Taskotron or openQA
> tests).
Yes and no, yes I'm all in favor of having a contact person from QA to help
volunteer to pick up the right tests for their package, but no as which tests
are used to gate is out of the scope of the proposal itself.
I'd be in favor of adding something like:
"""
If you want to volunteer to opt-in into this workflow and need help figuring out
which tests would be interesting to run in your package, contact foo at
<email/list> or open a ticket at <tracker>
"""
Would that make sense?
If so, where should we recommend packagers to get help?
It's more a case of: if people choose to gate on QA-maintained tests
and then there turns out to be some sort of problem involving tests not
running or flakes or whatever, it is going to be *perceived to be* a
part of this Change even if you're trying to keep it out of scope.
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
2:40 a.m.
Dne 02. 03. 19 v 8:09 Adam Williamson napsal(a):
On Fri, 2019-03-01 at 16:19 -0500, Ben Cotton wrote:
> https://fedoraproject.org/wiki/Changes/GatingRawhideSinglePackageUpdates
>
> == Summary ==
> We want to gate packages on test results before they can land in
> rawhide. This will reduce the amount of broken dependency,
> uninstallable packages and broken composes leading to a more stable
> rawhide as well as less work on the infrastructure and rel-eng teams
> to keep composes working.
>
> This project will be split in two phases, at first only single package
> updates will be supported, in a second stage, we will add support for
> multi-packages updates.
>
> This proposal is about the phase 1 of this project.
>
> == Owner ==
> * Name: [[User:pingou|Pierre-Yves Chibon]]
> * Email: pingou - pingoured.fr
>
> People who are/will be involved in this:
> * Coordinator: [[User:pingou|Pierre-Yves Chibon]]
> * Bodhi: [[User:bowlofeggs|Randy Barlow]] and [[User:abompard|Aurélien Bompard]]
> * Fedora CI: [[User:dperpeet|Dominik Perpeet]]
It might be a good idea to have a QA contact here, in case people
choose to block on tests maintained by the QA team (Taskotron or openQA
tests).
> == Detailed Description ==
>
> Querying the Bodhi database, it was revealed that 95% of all our
> updates involve a single package.
>
> To be exhaustive, here are the exact number found by Randy:
>
> Of all time:
>
> Single build updates: 123,657 (94.1%)
> Multi build updates: 7,766 ( 5.9%)
>
> Total: 131,423
>
> Per release:
>
> Fedora 29:
>
> Single: 4,675 (93.7%)
> Multi: 316 ( 6.3%)
>
> Fedora 28:
>
> Single: 9,153 (94.5%)
> Multi: 536 ( 5.5%)
>
> EPEL 7:
>
> Single: 12,664 (94.0%)
> Multi: 814 ( 6.0%)
I'd suggest the implication that single-package updates are "the norm"
is slightly problematic, for two reasons:
1) Rawhide is very different from stable releases, and even from
Branched. Major changes like API breaks are not meant to be sent to
stable releases at all, by policy. So I don't think you can necessarily
rely very strongly on data regarding updates in Branched and stable
releases to draw conclusions about the likely nature of changes in
Rawhide.
2) This does not consider the not uncommon case that updates *should
have been* multi-package updates, but they were done wrong.
These were my first thoughts as well.
Also, how are the mass rebuilds envisioned? I can't imagine that Ruby
rebuild will be held from entering rawhide due to some broken
dependency. Not sure how you want distinguish the package, which is
typically "single" from "multi" package.
Vít
2:47 a.m.
On Mon, Mar 04, 2019 at 09:40:32AM +0100, Vít Ondruch wrote:
Dne 02. 03. 19 v 8:09 Adam Williamson napsal(a):
> On Fri, 2019-03-01 at 16:19 -0500, Ben Cotton wrote:
>> https://fedoraproject.org/wiki/Changes/GatingRawhideSinglePackageUpdates
>>
>> == Summary ==
>> We want to gate packages on test results before they can land in
>> rawhide. This will reduce the amount of broken dependency,
>> uninstallable packages and broken composes leading to a more stable
>> rawhide as well as less work on the infrastructure and rel-eng teams
>> to keep composes working.
>>
>> This project will be split in two phases, at first only single package
>> updates will be supported, in a second stage, we will add support for
>> multi-packages updates.
>>
>> This proposal is about the phase 1 of this project.
>>
>> == Owner ==
>> * Name: [[User:pingou|Pierre-Yves Chibon]]
>> * Email: pingou - pingoured.fr
>>
>> People who are/will be involved in this:
>> * Coordinator: [[User:pingou|Pierre-Yves Chibon]]
>> * Bodhi: [[User:bowlofeggs|Randy Barlow]] and [[User:abompard|Aurélien
Bompard]]
>> * Fedora CI: [[User:dperpeet|Dominik Perpeet]]
> It might be a good idea to have a QA contact here, in case people
> choose to block on tests maintained by the QA team (Taskotron or openQA
> tests).
>
>> == Detailed Description ==
>>
>> Querying the Bodhi database, it was revealed that 95% of all our
>> updates involve a single package.
>>
>> To be exhaustive, here are the exact number found by Randy:
>>
>> Of all time:
>>
>> Single build updates: 123,657 (94.1%)
>> Multi build updates: 7,766 ( 5.9%)
>>
>> Total: 131,423
>>
>> Per release:
>>
>> Fedora 29:
>>
>> Single: 4,675 (93.7%)
>> Multi: 316 ( 6.3%)
>>
>> Fedora 28:
>>
>> Single: 9,153 (94.5%)
>> Multi: 536 ( 5.5%)
>>
>> EPEL 7:
>>
>> Single: 12,664 (94.0%)
>> Multi: 814 ( 6.0%)
> I'd suggest the implication that single-package updates are "the
norm"
> is slightly problematic, for two reasons:
>
> 1) Rawhide is very different from stable releases, and even from
> Branched. Major changes like API breaks are not meant to be sent to
> stable releases at all, by policy. So I don't think you can necessarily
> rely very strongly on data regarding updates in Branched and stable
> releases to draw conclusions about the likely nature of changes in
> Rawhide.
>
> 2) This does not consider the not uncommon case that updates *should
> have been* multi-package updates, but they were done wrong.
These were my first thoughts as well.
Also, how are the mass rebuilds envisioned?
Just as anyone would: by opting-out of gating.
Maybe that term is overloaded. The way I see it there are two ways to opt-out:
- remove the gating.yml file in the package's git repo, not scalable especially
for mass-rebuild
- build in the koji candidate tag directly, thus by-passing the testing tag
where tests happen. This would be the approach releng would do for
mass-rebuild.
I can't imagine that Ruby
rebuild will be held from entering rawhide due to some broken
dependency. Not sure how you want distinguish the package, which is
typically "single" from "multi" package.
This is a decision left in the hands of the packagers, I'm pretty sure they know
more their packages than I do and thus I'm in no position to make this decision.
Does this help?
Pierre
12:11 p.m.
On 3/4/19 12:47 AM, Pierre-Yves Chibon wrote:
On Mon, Mar 04, 2019 at 09:40:32AM +0100, Vít Ondruch wrote:
> Also, how are the mass rebuilds envisioned?
Just as anyone would: by opting-out of gating.
Maybe that term is overloaded. The way I see it there are two ways to opt-out:
- remove the gating.yml file in the package's git repo, not scalable especially
for mass-rebuild
- build in the koji candidate tag directly, thus by-passing the testing tag
where tests happen. This would be the approach releng would do for
mass-rebuild.
Well, to be clear, mass rebuilds already use a side tag and then are
merged in. So they would just be merged into the candidate tag or whatever.
> I can't imagine that Ruby
> rebuild will be held from entering rawhide due to some broken
> dependency. Not sure how you want distinguish the package, which is
> typically "single" from "multi" package.
This is a decision left in the hands of the packagers, I'm pretty sure they know
more their packages than I do and thus I'm in no position to make this decision.
I think perhaps it would be useful to easily show waivers?
like a weekly waiver report or something so we could tell what packages
and maintainers are waiving results?
kevin
1:57 p.m.
On Mon, Mar 04, 2019 at 10:11:27AM -0800, Kevin Fenzi wrote:
On 3/4/19 12:47 AM, Pierre-Yves Chibon wrote:
> On Mon, Mar 04, 2019 at 09:40:32AM +0100, Vít Ondruch wrote:
>> Also, how are the mass rebuilds envisioned?
>
> Just as anyone would: by opting-out of gating.
> Maybe that term is overloaded. The way I see it there are two ways to opt-out:
> - remove the gating.yml file in the package's git repo, not scalable especially
> for mass-rebuild
> - build in the koji candidate tag directly, thus by-passing the testing tag
> where tests happen. This would be the approach releng would do for
> mass-rebuild.
Well, to be clear, mass rebuilds already use a side tag and then are
merged in. So they would just be merged into the candidate tag or whatever.
>
>> I can't imagine that Ruby
>> rebuild will be held from entering rawhide due to some broken
>> dependency. Not sure how you want distinguish the package, which is
>> typically "single" from "multi" package.
>
> This is a decision left in the hands of the packagers, I'm pretty sure they
know
> more their packages than I do and thus I'm in no position to make this
decision.
I think perhaps it would be useful to easily show waivers?
like a weekly waiver report or something so we could tell what packages
and maintainers are waiving results?
We will definitely want to look into waivers at one point (both which tests as
well as which maintainers), I am not sure we need it from the get go though :)
Pierre
2:28 a.m.
On Sat, Mar 02, 2019 at 03:05:56PM -0600, Ken Dreyer wrote:
On Fri, Mar 1, 2019, 3:20 PM Ben Cotton
<[1]bcotton(a)redhat.com> wrote:
** infrastructure: deploy the new version of bodhi and the new koji
plugin
What's the new Koji plugin?
Good catch, this is a left over from when I started this page and we thought we
may do single and multi packages updates in one go. The koji plugin is meant for
the multi-package workflow, I've edited this line.
Thanks!
Pierre
3:36 p.m.
On 3/1/19 1:19 PM, Ben Cotton wrote:
...snip...
# Add tests to a package you maintain in Fedora using the [[
CI/Quick_Start_Guide ]] documentation (see the full [[CI|CI
documentation ]] for more information).
# Update your package in rawhide
# Ensure that the bodhi update is automatically created
# Check that the tests have properly ran (see the Tests tab in the bodhi update)
# Ensure that the bodhi update went through to rawhide once the tests passed
Might also test the 'opt out case' (ie, waiving or opting out
successfully lands the build in rawhide with failing tests?)
=== Multi package update with single package update gating ===
fedpkg clone rpms/foobar
cd foobar/
vim foobar.spec
git commit -am "Update of foobar to update foo to 1.2"
fedpkg push
fedpkg build --target=rawhide-candidate
cd ..
fedpkg clone rpms/foobaz
cd foobaz
vim foobaz.spec
git commit -am "Update of foobaz to update foo to 1.2"
fedpkg push
fedpkg build --target=rawhide-candidate
cd ..
fedpkg clone rpms/foo
cd foo
vim foo.spec
git commit -am "Update foo to 1.2"
fedpkg push
fedpkg build --target=rawhide-candidate
Users are practically by-passing the gating process by specifying a
specific build target.
Do note:
We removed the rawhide target in koji (
https://pagure.io/releng/issue/7664 ). I have put in a symlink so it
still works (without the repo regenning, pointing to f31-build). So, we
might want to name the target here the number? Just thought I would
mention it, we can sort it out when we need to.
Other than those nit-picks, I think it looks great. ;)
Thanks for leading this and I hope we have it in place soon...
kevin
2:37 a.m.
On Sat, Mar 02, 2019 at 01:36:10PM -0800, Kevin Fenzi wrote:
On 3/1/19 1:19 PM, Ben Cotton wrote:
> https://fedoraproject.org/wiki/Changes/GatingRawhideSinglePackageUpdates
>
...snip...
> # Add tests to a package you maintain in Fedora using the [[
> CI/Quick_Start_Guide ]] documentation (see the full [[CI|CI
> documentation ]] for more information).
> # Update your package in rawhide
> # Ensure that the bodhi update is automatically created
> # Check that the tests have properly ran (see the Tests tab in the bodhi update)
> # Ensure that the bodhi update went through to rawhide once the tests passed
Might also test the 'opt out case' (ie, waiving or opting out
successfully lands the build in rawhide with failing tests?)
I have added a sentence about how to waive failing tests and a section about how
to test without opting in into gating.
I'll add a section about how to opt-out but before I do that I'd like to clear
out something, so it may take me a day or two before I can write this section.
> === Multi package update with single package update gating ===
>
> fedpkg clone rpms/foobar
> cd foobar/
> vim foobar.spec
> git commit -am "Update of foobar to update foo to 1.2"
> fedpkg push
> fedpkg build --target=rawhide-candidate
> cd ..
> fedpkg clone rpms/foobaz
> cd foobaz
> vim foobaz.spec
> git commit -am "Update of foobaz to update foo to 1.2"
> fedpkg push
> fedpkg build --target=rawhide-candidate
> cd ..
> fedpkg clone rpms/foo
> cd foo
> vim foo.spec
> git commit -am "Update foo to 1.2"
> fedpkg push
> fedpkg build --target=rawhide-candidate
>
> Users are practically by-passing the gating process by specifying a
> specific build target.
Do note:
We removed the rawhide target in koji (
https://pagure.io/releng/issue/7664 ). I have put in a symlink so it
still works (without the repo regenning, pointing to f31-build). So, we
might want to name the target here the number? Just thought I would
mention it, we can sort it out when we need to.
I've kept "rawhide" and "rawhide-candidate" for now to be
consistent in the
document (and because rawhide should remain a self-descriptive for a while), but
I'm fine changing this if needed.
Other than those nit-picks, I think it looks great. ;)
Thanks :)
Thanks for leading this and I hope we have it in place soon...
We should have a target date sometime this week :)
Pierre
4:45 a.m.
On 01. 03. 19 22:19, Ben Cotton wrote:
'''The CI system, the tests and the decision on which
tests are used
to gate upon are out of scope for the present document.'''
This is both good (specifying explicitly what is this change about and what it
is not about) and bad...
Since the CI system is not part of this change, we cannot say this change is not
complete if the CI system is not complete. So later we say we have rawhide
gating and we'll all be \o/ \o/ \o/ yet nobody will care that the CI is
unfortunately:
* not locally reproducible https://pagure.io/fedora-ci/general/issue/4
* only working on on x86_64 https://pagure.io/fedora-ci/general/issue/16
* unreadable https://pagure.io/fedora-ci/general/issue/2
* unreliable (I file 1.75 issues per month on average)
* understaffed (my personal observation)
* tedious to start using (we focus on standards instead of UX)
* untested (the (sometimes fatal) issues I face go unnoticed until I'm hit)
My concern is that the CI experience still feels a bit rough and I'd rather see
us focusing on making it better. This can of course be done in parallel with
this change, yet I feel that we are building this on water.
Note that I don't blame the CI people, they are very helpful and great to work
with, they just don't have time/resources/people.
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
5:34 a.m.
On Sun, Mar 03, 2019 at 11:45:31AM +0100, Miro Hrončok wrote:
On 01. 03. 19 22:19, Ben Cotton wrote:
> '''The CI system, the tests and the decision on which tests are used
> to gate upon are out of scope for the present document.'''
This is both good (specifying explicitly what is this change about and what
it is not about) and bad...
Since the CI system is not part of this change, we cannot say this change is
not complete if the CI system is not complete. So later we say we have
rawhide gating and we'll all be \o/ \o/ \o/ yet nobody will care that the CI
is unfortunately:
* not locally reproducible https://pagure.io/fedora-ci/general/issue/4
* only working on on x86_64 https://pagure.io/fedora-ci/general/issue/16
* unreadable https://pagure.io/fedora-ci/general/issue/2
* unreliable (I file 1.75 issues per month on average)
* understaffed (my personal observation)
* tedious to start using (we focus on standards instead of UX)
* untested (the (sometimes fatal) issues I face go unnoticed until I'm hit)
My concern is that the CI experience still feels a bit rough and I'd rather
see us focusing on making it better. This can of course be done in parallel
with this change, yet I feel that we are building this on water.
Note that I don't blame the CI people, they are very helpful and great to
work with, they just don't have time/resources/people.
I think you are raising very good points and that they should be looked into,
however, in the design of gating rawhide we tried to be test system agnostic,
so, as Adam already pointed out, you could gate on results any of our test
systems (we currently have three: the CI pipeline, taskotron, OpenQA) and
possibly other in the future.
I believe Dominik is working on a new CI initiative at the council level and
I'll let him reply if these (or some of these) concerns can be covered by this
new initiative.
Thanks again,
Pierre
2:54 p.m.
On 04. 03. 19 12:34, Pierre-Yves Chibon wrote:
On Sun, Mar 03, 2019 at 11:45:31AM +0100, Miro Hrončok wrote:
> On 01. 03. 19 22:19, Ben Cotton wrote:
>> '''The CI system, the tests and the decision on which tests are used
>> to gate upon are out of scope for the present document.'''
>
> This is both good (specifying explicitly what is this change about and what
> it is not about) and bad...
>
> Since the CI system is not part of this change, we cannot say this change is
> not complete if the CI system is not complete. So later we say we have
> rawhide gating and we'll all be \o/ \o/ \o/ yet nobody will care that the CI
> is unfortunately:
>
> * not locally reproducible https://pagure.io/fedora-ci/general/issue/4
> * only working on on x86_64 https://pagure.io/fedora-ci/general/issue/16
> * unreadable https://pagure.io/fedora-ci/general/issue/2
> * unreliable (I file 1.75 issues per month on average)
> * understaffed (my personal observation)
> * tedious to start using (we focus on standards instead of UX)
> * untested (the (sometimes fatal) issues I face go unnoticed until I'm hit)
>
> My concern is that the CI experience still feels a bit rough and I'd rather
> see us focusing on making it better. This can of course be done in parallel
> with this change, yet I feel that we are building this on water.
>
> Note that I don't blame the CI people, they are very helpful and great to
> work with, they just don't have time/resources/people.
I think you are raising very good points and that they should be looked into,
however, in the design of gating rawhide we tried to be test system agnostic,
so, as Adam already pointed out, you could gate on results any of our test
systems (we currently have three: the CI pipeline, taskotron, OpenQA) and
possibly other in the future.
Taskotron is planing to migrate to the CI pipeline.
https://pagure.io/fedora-ci/general/issue/36
So that leaves 2 things:
* Fedora Ci with the above problems
* OpenQA
I honestly don't except many packagers writing their own OpenQA tests.
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
3:31 p.m.
On 04. 03. 19 12:34, Pierre-Yves Chibon wrote:
On Sun, Mar 03, 2019 at 11:45:31AM +0100, Miro Hrončok wrote:
> On 01. 03. 19 22:19, Ben Cotton wrote:
>> '''The CI system, the tests and the decision on which tests are used
>> to gate upon are out of scope for the present document.'''
>
> This is both good (specifying explicitly what is this change about and what
> it is not about) and bad...
>
> Since the CI system is not part of this change, we cannot say this change is
> not complete if the CI system is not complete. So later we say we have
> rawhide gating and we'll all be \o/ \o/ \o/ yet nobody will care that the CI
> is unfortunately:
>
> * not locally reproducible https://pagure.io/fedora-ci/general/issue/4
> * only working on on x86_64 https://pagure.io/fedora-ci/general/issue/16
> * unreadable https://pagure.io/fedora-ci/general/issue/2
> * unreliable (I file 1.75 issues per month on average)
> * understaffed (my personal observation)
> * tedious to start using (we focus on standards instead of UX)
> * untested (the (sometimes fatal) issues I face go unnoticed until I'm hit)
>
> My concern is that the CI experience still feels a bit rough and I'd rather
> see us focusing on making it better. This can of course be done in parallel
> with this change, yet I feel that we are building this on water.
>
> Note that I don't blame the CI people, they are very helpful and great to
> work with, they just don't have time/resources/people.
I think you are raising very good points and that they should be looked into,
however, in the design of gating rawhide we tried to be test system agnostic,
so, as Adam already pointed out, you could gate on results any of our test
systems (we currently have three: the CI pipeline, taskotron, OpenQA) and
possibly other in the future.
I believe Dominik is working on a new CI initiative at the council level and
I'll let him reply if these (or some of these) concerns can be covered by this
new initiative.
No word for almost a month. I've opened 2 more issues that I think must be
solved before the gating is considered:
CI errors are undecipherable:
https://pagure.io/fedora-ci/general/issue/43
CI errors happen far to often:
https://pagure.io/fedora-ci/general/issue/44
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
4:30 p.m.
For the record, the Change proposal I sent earlier without a title in
the subject is a follow-up to this proposal:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.o...
--
Ben Cotton
Fedora Program Manager
TZ=America/Indiana/Indianapolis
Pronouns: he/him
1:44 a.m.
On 01. 03. 19 22:19, Ben Cotton wrote:
https://fedoraproject.org/wiki/Changes/GatingRawhideSinglePackageUpdates
== Summary ==
We want to gate packages on test results before they can land in
rawhide. This will reduce the amount of broken dependency,
uninstallable packages and broken composes leading to a more stable
rawhide as well as less work on the infrastructure and rel-eng teams
to keep composes working.
This project will be split in two phases, at first only single package
updates will be supported, in a second stage, we will add support for
multi-packages updates.
This proposal is about the phase 1 of this project.
I was thinking. Does this change retiring packages at all? E.g. do we "push"
removals trough bodhi, or not?
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
1851
days inactive
1882
days old
23 comments
10 participants
participants (10)
-
Adam Williamson -
Ben Cotton -
Ken Dreyer -
Kevin Fenzi -
Miro Hrončok -
Neal Gompa -
Pierre-Yves Chibon -
Tom Stellard -
Vít Ondruch -
Zamir SUN