Hi!
Current Fedora CI documentation suggests: "The tests to be executed are stored in the dist-git repositories. The tests are stored or wrapped along-side the spec files..." [0] While working on porting tests to Fedora CI, several times we've noted concerns from developers/maintainers about placing test code directly into dist-git repositories. A common question is how to efficiently maintain tests & minimize test code duplication. There are some nice real-life examples available:
Examples
There are several shells which implement the POSIX specification: bash, ksh, mksh, zsh, dash. All of them share a significant amount of test coverage and it does not make sense to commit & maintain identical tests in five different repositories (+ possible branches). See the pull request [1] for a bit of context.
Another example is Ruby: With about 80 packages related to Ruby on Rails it would be useful and efficient to have a single place for integration tests which verify that the framework is correctly working after updating any of these packages. Conversely, maintaining those tests in 80 repos would be a tedious task.
Proposal: Share!
So this is where the idea of shared test repository comes from. In general, tests define how the software works and the basic functionality of many packages doesn’t change that often. We try hard to keep the backward compatibility where possible. Thus it seems natural that, for such components, tests guarding the spec could change at a slower pace than the distribution branches.
Main goals
* Package source and tests are linked in a discoverable and unambiguous way (dist-git) * Prevent test code duplication (minimize test maintenance) * Catch incompatibilities early
How it could work? A new namespace "tests" would be created in Fedora as suggested in [2], test code would reside there and the tests in the rpms dist git could directly reference a specific version of the tests to be run, e.g. with a pre_task:
pre_tasks: - name: Fetch the tests git: repo: https://src.fedoraproject.org/tests/shell.git dest: tests version: master
Also, standard-test-roles [3] should be enhanced to make it really easy to fetch tests from a remote repo. Ansible git module is quite simple but with standard-test-roles it could be even shorter. The tests.yml file could look like this (we could support fetching from multiple repositories):
- role: standard-test-beakerlib tags: - classic - container - atomic tests: - smoke repo: - https://src.fedoraproject.org/tests/shell.git version: master
A consistent location for shared tests is a good source of examples and helps drive automation in the future, e.g. trigger CI jobs on test changes.
For stable packages providing specification-like functionality a single master branch could be used, for those components which change more, dist-git tests.yml could pick a proper branch or even a specific commit (if having concerns about updated tests breaking the continuous integration).
This seems to give a nice flexibility while at the same time minimizing test code duplication. Referencing specific tests from rpms dist git ties tests and code together very well. Storing the test code in the tests namespace is only one option, any git repo accessible from the CI pipeline would suffice.
Just One Place?
We could possibly go one more step further: Shall we aim at having just a single test code for testing both Fedora and Red Hat Enterprise Linux? At least for those cases where this is possible: There might be some licensing or compatibility issues. For example, speaking about Beaker: restraint harness can also be instructed to fetch tests directly from git so the identical test code could perhaps be used for running tests in Beaker as well.
A recent experience of a libselinux pull request [4] shows how cumbersome it can be to keep both upstream and downstream version of the test code in sync. Wouldn't it be great to have everything at a single place and keep the test code up-to-date only there? Instead of doing several rounds of "pull request - give feedback - update internal tests - update the pull request - find another issue - give feedback..." with many unnecessary steps and reminders just directly collaborating on a single test repository?
When downstream testing is just a branch of upstream testing, we can use our standard open source workflows. Just as in upstream, there would likely be a downstream shared git repository that branches from the upstream one.
Bonus
As a bonus we would keep the dist-git for such repositories a bit more consistent as a place for metadata only: Build metadata (spec file = how to build the package) and test metadata (tests.yml = how to test the package).
Plus we would get a more streamlined test maintenance as QE and volunteers willing to help with improving the test coverage could be granted direct access to individual package tests repo. Not every QE is a Fedora packager so contributing to dist-git is not that straightforward.
Notes
There are cons of this approach as well. It isn’t possible to have new feature code and tests submitted in a single commit or triggering test execution for every change in tests (CI for tests). There is a need for handling relevance of individual tests for different environments. But the maintenance simplification seems to be worth it, based on experience of the BaseOS QE team which has been maintaining thousands of tests for many years and using them for testing large number of supported versions of Red Hat Enterprise Linux.
The overall benefits also depend on which test types we are speaking about. Tests should always be as far upstream as possible. Unit tests generally make more sense in upstream. For integration tests it might make more sense to live in the shared Fedora tests repository. Recommendations about which tests fit well into this "tests" namespace will evolve, but it will be up to each package to choose where to place its tests.
So?
What are your thoughts on this? Does it make sense to you?
psss...
[0] https://fedoraproject.org/wiki/CI#Tests_locations [1] https://src.fedoraproject.org/rpms/bash/pull-request/1 [2] https://pagure.io/fedora-infrastructure/issue/6478 [3] https://pagure.io/standard-test-roles/ [4] https://src.fedoraproject.org/rpms/libselinux/pull-request/1
Thanks, makes sense to me. The tests/ namespace is much better approach than some per-package, non-standard place (on github, let's say) to share the test code.
Ideally, the test automation should (except for explicit tests references in `pre_tasks:`) download implicitly `tests/FOO` for `FOO` component, when the `tests/FOO` is existing. Also `fedpkg clone-tests <COMPONENT>` would be convenient thing..
Pavel
On Mon, 20 Nov 2017 16:30:22 +0100 Petr Splichal psplicha@redhat.com wrote:
Hi!
Current Fedora CI documentation suggests: "The tests to be executed are stored in the dist-git repositories. The tests are stored or wrapped along-side the spec files..." [0] While working on porting tests to Fedora CI, several times we've noted concerns from developers/maintainers about placing test code directly into dist-git repositories. A common question is how to efficiently maintain tests & minimize test code duplication. There are some nice real-life examples available:
Examples
There are several shells which implement the POSIX specification: bash, ksh, mksh, zsh, dash. All of them share a significant amount of test coverage and it does not make sense to commit & maintain identical tests in five different repositories (+ possible branches). See the pull request [1] for a bit of context.
Another example is Ruby: With about 80 packages related to Ruby on Rails it would be useful and efficient to have a single place for integration tests which verify that the framework is correctly working after updating any of these packages. Conversely, maintaining those tests in 80 repos would be a tedious task.
Proposal: Share!
So this is where the idea of shared test repository comes from. In general, tests define how the software works and the basic functionality of many packages doesn’t change that often. We try hard to keep the backward compatibility where possible. Thus it seems natural that, for such components, tests guarding the spec could change at a slower pace than the distribution branches.
Main goals
- Package source and tests are linked in a discoverable and unambiguous way (dist-git)
- Prevent test code duplication (minimize test maintenance)
- Catch incompatibilities early
How are git repos in different namespaces of a larger repo ecosystem easier to link together than just urls that point to github or pagure.io ?
There has been some talk about pulling in tests that live in upstream repos which makes me think github. If this is true and we're going to be pulling other things in from github/pagure/gitlab/wherever, why not just have these shared repos in pagure.io?
How it could work? A new namespace "tests" would be created in Fedora as suggested in [2], test code would reside there and the tests in the rpms dist git could directly reference a specific version of the tests to be run, e.g. with a pre_task:
Is the Fedora dist-git instance set up to enable ACL lists made up of users who are not already packagers and who don't otherwise have write access to the corresponding package repo? If not, how much extra work are we talking about? Are there available cycles to get the work done in short order?
As I understand it, RHEL QE has a setup with a "tests" namespace which stores all the tests for a package whose name matches the name of that repo from the tests namespace. If we use "tests" as the namespace here, I fear that we will be adding to confusion from the people who are used to the RHEL version of a "tests" namespace. We would also not realistically be able to replicate a similar setup in Fedora. There seemed to be interest in something like that when the issue came up on devel@ a few years ago which also makes me think it'd be unwise to close this door unless there is no other reasonable choice. If we do go the dist-git namespace route, have there been any alternative names proposed?
pre_tasks: - name: Fetch the tests git: repo: https://src.fedoraproject.org/tests/shell.git dest: tests version: masterAlso, standard-test-roles [3] should be enhanced to make it really easy to fetch tests from a remote repo. Ansible git module is quite simple but with standard-test-roles it could be even shorter. The tests.yml file could look like this (we could support fetching from multiple repositories):
- role: standard-test-beakerlib tags: - classic - container - atomic tests: - smoke repo: - https://src.fedoraproject.org/tests/shell.git version: masterA consistent location for shared tests is a good source of examples and helps drive automation in the future, e.g. trigger CI jobs on test changes.
For stable packages providing specification-like functionality a single master branch could be used, for those components which change more, dist-git tests.yml could pick a proper branch or even a specific commit (if having concerns about updated tests breaking the continuous integration).
This seems to give a nice flexibility while at the same time minimizing test code duplication. Referencing specific tests from rpms dist git ties tests and code together very well. Storing the test code in the tests namespace is only one option, any git repo accessible from the CI pipeline would suffice.
I don't really have a problem with the idea of having a shared repository in the interest of decreasing duplication. That being said, I do have some questions about some more of the details behind how this would work.
While this isn't the most common case, it is possible to have a situation where a package changes and the tests need to change at the same time. It's not necessarily a fatal problem but it brings up some questions:
- For the PR testing case, how will the shared tests be included in that PR set for testing purposes?
- Will there be some delay in test/build kickoff on dist-git repo change to allow for tests to be pushed around the same time that there are code changes? Otherwise are we planning to expect some transitory failures if both the tests and packages need to change at the same time? Are there plans to verify that things are properly re-run if those transitory errors happen?
Just One Place?
We could possibly go one more step further: Shall we aim at having just a single test code for testing both Fedora and Red Hat Enterprise Linux? At least for those cases where this is possible: There might be some licensing or compatibility issues. For example, speaking about Beaker: restraint harness can also be instructed to fetch tests directly from git so the identical test code could perhaps be used for running tests in Beaker as well.
A recent experience of a libselinux pull request [4] shows how cumbersome it can be to keep both upstream and downstream version of the test code in sync. Wouldn't it be great to have everything at a single place and keep the test code up-to-date only there? Instead of doing several rounds of "pull request - give feedback - update internal tests - update the pull request - find another issue - give feedback..." with many unnecessary steps and reminders just directly collaborating on a single test repository?
When downstream testing is just a branch of upstream testing, we can use our standard open source workflows. Just as in upstream, there would likely be a downstream shared git repository that branches from the upstream one.
Bonus
As a bonus we would keep the dist-git for such repositories a bit more consistent as a place for metadata only: Build metadata (spec file = how to build the package) and test metadata (tests.yml = how to test the package).
Plus we would get a more streamlined test maintenance as QE and volunteers willing to help with improving the test coverage could be granted direct access to individual package tests repo. Not every QE is a Fedora packager so contributing to dist-git is not that straightforward.
The question of how to deal with non-packager write access for tests has been brought up a couple of times but I'm not sure it's been dealt with well yet.
I don't really like the idea of tests with arbitrary ACLs. As we work toward a vision of having CI gate all changes to packages before those changes are even accepted, changes to the tests associated with a package start approaching the importance of the non-test content for the same package.
If someone tosses an unproven test into a shared repo before going home for the weekend - there is a chance that could cause all sorts of problems from keeping a package's changes out of stable to breaking the build of a release-blocking image.
I realize that there are plans to handle situations like this with automatic reversion but to the best of my knowledge, that system doesn't exist yet (an admin could back stuff out before the automation exists, too). If we demand that packagers go through a vetting and mentoring process even if they're a co-maintainer, why shouldn't we have a similar requirement for the folks who can change the tests which are responsible for gating the bits we release?
Notes
There are cons of this approach as well. It isn’t possible to have new feature code and tests submitted in a single commit or triggering test execution for every change in tests (CI for tests). There is a need for handling relevance of individual tests for different environments. But the maintenance simplification seems to be worth it, based on experience of the BaseOS QE team which has been maintaining thousands of tests for many years and using them for testing large number of supported versions of Red Hat Enterprise Linux.
The overall benefits also depend on which test types we are speaking about. Tests should always be as far upstream as possible. Unit tests generally make more sense in upstream. For integration tests it might make more sense to live in the shared Fedora tests repository. Recommendations about which tests fit well into this "tests" namespace will evolve, but it will be up to each package to choose where to place its tests.
So?
What are your thoughts on this? Does it make sense to you?
It makes sense, just some concerns about the specifics of this proposal. Thanks for putting so much time into this.
Tim
psss...
[0] https://fedoraproject.org/wiki/CI#Tests_locations [1] https://src.fedoraproject.org/rpms/bash/pull-request/1 [2] https://pagure.io/fedora-infrastructure/issue/6478 [3] https://pagure.io/standard-test-roles/ [4] https://src.fedoraproject.org/rpms/libselinux/pull-request/1 _______________________________________________ CI mailing list -- ci@lists.fedoraproject.org To unsubscribe send an email to ci-leave@lists.fedoraproject.org
I will try to address a few of the points.
On 01/30/2018 04:33 PM, Tim Flink wrote:
How are git repos in different namespaces of a larger repo ecosystem easier to link together than just urls that point to github or pagure.io ?
There has been some talk about pulling in tests that live in upstream repos which makes me think github. If this is true and we're going to be pulling other things in from github/pagure/gitlab/wherever, why not just have these shared repos in pagure.io?
There is no technical reason currently, other than the convenience of grouping. One of the possible options with grouped repos is the ease to add something in the future, such as modifying access control or even providing optimized Pagure workflows for common use cases.
Is the Fedora dist-git instance set up to enable ACL lists made up of users who are not already packagers and who don't otherwise have write access to the corresponding package repo? If not, how much extra work are we talking about? Are there available cycles to get the work done in short order?
Currently only fedora "packagers" have access to dist-git itself for actions such as cloning. There is a workaround to open pull requests from other git repos: https://fedoraproject.org/wiki/CI/Tests#Creating_pull_request
As I understand it, RHEL QE has a setup with a "tests" namespace which stores all the tests for a package whose name matches the name of that repo from the tests namespace. If we use "tests" as the namespace here, I fear that we will be adding to confusion from the people who are used to the RHEL version of a "tests" namespace.
I believe all tests should be linked from the package's dist-git - wherever that path may lead (in the dist-git itself or another repo, even upstream).
- For the PR testing case, how will the shared tests be included in that PR set for testing purposes?
All relevant tests should be linked to the repo where they will gate changes (e.g. changing shell tests will gate on those passing on bash, ksh, et cetera).
- Will there be some delay in test/build kickoff on dist-git repo change to allow for tests to be pushed around the same time that there are code changes? Otherwise are we planning to expect some transitory failures if both the tests and packages need to change at the same time? Are there plans to verify that things are properly re-run if those transitory errors happen?
The best way here is to reference the exact version of the tests you want, by git repo and revision. Then you don't depend on chance and you can safely change the tests first, then the dist-git repo.
I realize that there are plans to handle situations like this with automatic reversion but to the best of my knowledge, that system doesn't exist yet (an admin could back stuff out before the automation exists, too). If we demand that packagers go through a vetting and mentoring process even if they're a co-maintainer, why shouldn't we have a similar requirement for the folks who can change the tests which are responsible for gating the bits we release?
The starting point of test execution is the dist-git repo itself, which underlies the known restrictions. It is up to the packager to decide which other repo to link to, if at all. The tests could feasibly be in the dist-git repo itself or any other repo the packager trusts (and that is publicly accessible). If we should limit that choice somehow, we need to discuss it.
It makes sense, just some concerns about the specifics of this proposal. Thanks for putting so much time into this.
Tim
Thank you for your feedback!
On Tue, 30 Jan 2018 18:32:02 +0100 'Dominik Perpeet' dperpeet@redhat.com wrote:
I will try to address a few of the points.
On 01/30/2018 04:33 PM, Tim Flink wrote:
How are git repos in different namespaces of a larger repo ecosystem easier to link together than just urls that point to github or pagure.io ?
There has been some talk about pulling in tests that live in upstream repos which makes me think github. If this is true and we're going to be pulling other things in from github/pagure/gitlab/wherever, why not just have these shared repos in pagure.io?
There is no technical reason currently, other than the convenience of grouping. One of the possible options with grouped repos is the ease to add something in the future, such as modifying access control or even providing optimized Pagure workflows for common use cases.
This didn't make a lot of sense to me until we spoke in person earlier today.
The bit I was missing was that there's a chance that Pagure could be enhanced to have a workflow which covers multiple repos with the same PR which would solve some of the concerns I have.
Is the Fedora dist-git instance set up to enable ACL lists made up of users who are not already packagers and who don't otherwise have write access to the corresponding package repo? If not, how much extra work are we talking about? Are there available cycles to get the work done in short order?
Currently only fedora "packagers" have access to dist-git itself for actions such as cloning. There is a workaround to open pull requests from other git repos: https://fedoraproject.org/wiki/CI/Tests#Creating_pull_request
I thought that PRs were the main solution to allowing contributions from people who aren't already packagers. Has this changed or is the implementation of non-packager PRs still pending?
As I understand it, RHEL QE has a setup with a "tests" namespace which stores all the tests for a package whose name matches the name of that repo from the tests namespace. If we use "tests" as the namespace here, I fear that we will be adding to confusion from the people who are used to the RHEL version of a "tests" namespace.
I believe all tests should be linked from the package's dist-git - wherever that path may lead (in the dist-git itself or another repo, even upstream).
I don't understand how this relates. There is a similarly named system/setup used by many of the same folks who will be contributing tests and working on tests in this new CI paradigm. I'm concerned that the similar name and function will lead to confusion and a steeper learning curve. Specifically, I'm talking about the tests/ namespace used inside Red Hat by QE folk to store tests and test-related data outside of the package's repo.
- For the PR testing case, how will the shared tests be included in that PR set for testing purposes?
All relevant tests should be linked to the repo where they will gate changes (e.g. changing shell tests will gate on those passing on bash, ksh, et cetera).
That's fair enough but my question was more around situations where you need both changes to land at the same time so that the test doesn't fail. Having a modified workflow in Pagure to support multiple repos in the same PR is one solution to one problem - what about tests which are stored in upstream projects or just outside of dist-git? Is it going to be possible to change both the tests and the packages at the same time without seeing what are effectively false positives from the various CI systems? If so, what is the planned workflow for dealing with simultaneous changes?
- Will there be some delay in test/build kickoff on dist-git repo change to allow for tests to be pushed around the same time that there are code changes? Otherwise are we planning to expect some transitory failures if both the tests and packages need to
change at the same time? Are there plans to verify that things are properly re-run if those transitory errors happen?
The best way here is to reference the exact version of the tests you want, by git repo and revision. Then you don't depend on chance and you can safely change the tests first, then the dist-git repo.
Does that mean that test runs won't be triggered on test change? Unless I'm missing something, how will we be dealing with the eventual issues where the git commit is out of date and propagates incorrect data through the pipeline?
For example, say that the tests for the package foo point to more tests in a foo-tests.git repo. If I change those tests and don't change the package - a triggered test run would likely indicate that my change continues to pass without issue.
I realize that there are plans to handle situations like this with automatic reversion but to the best of my knowledge, that system doesn't exist yet (an admin could back stuff out before the automation exists, too). If we demand that packagers go through a vetting and mentoring process even if they're a co-maintainer, why shouldn't we have a similar requirement for the folks who can change the tests which are responsible for gating the bits we release?
The starting point of test execution is the dist-git repo itself, which underlies the known restrictions. It is up to the packager to decide which other repo to link to, if at all. The tests could feasibly be in the dist-git repo itself or any other repo the packager trusts (and that is publicly accessible). If we should limit that choice somehow, we need to discuss it.
I'm not sure I understand your response here. Does that mean that contributions from non-packagers is not deemed important enough to worry about at this time? If so, how do we plan to get ported tests from downstream RHEL into Fedora if QEs are currently doing much of the porting work?
It makes sense, just some concerns about the specifics of this proposal. Thanks for putting so much time into this.
Tim
Thank you for your feedback!
Not a problem. Thanks for your quick response.
Tim
On Wed, Jan 31, 2018 at 02:21:45PM +0100, Tim Flink wrote:
On Tue, 30 Jan 2018 18:32:02 +0100 'Dominik Perpeet' dperpeet@redhat.com wrote:
I will try to address a few of the points.
On 01/30/2018 04:33 PM, Tim Flink wrote:
How are git repos in different namespaces of a larger repo ecosystem easier to link together than just urls that point to github or pagure.io ?
There has been some talk about pulling in tests that live in upstream repos which makes me think github. If this is true and we're going to be pulling other things in from github/pagure/gitlab/wherever, why not just have these shared repos in pagure.io?
There is no technical reason currently, other than the convenience of grouping. One of the possible options with grouped repos is the ease to add something in the future, such as modifying access control or even providing optimized Pagure workflows for common use cases.
This didn't make a lot of sense to me until we spoke in person earlier today.
The bit I was missing was that there's a chance that Pagure could be enhanced to have a workflow which covers multiple repos with the same PR which would solve some of the concerns I have.
I think the idea is to find a way to link multiple PRs together (something as simple as using: Requires <url to PR> in the commit message for example), rather than figuring out how to do 1 PR against multiple repos (which in the case of dist-git practically do not overlap in anyway).
Is the Fedora dist-git instance set up to enable ACL lists made up of users who are not already packagers and who don't otherwise have write access to the corresponding package repo? If not, how much extra work are we talking about? Are there available cycles to get the work done in short order?
Currently only fedora "packagers" have access to dist-git itself for actions such as cloning. There is a workaround to open pull requests from other git repos: https://fedoraproject.org/wiki/CI/Tests#Creating_pull_request
I thought that PRs were the main solution to allowing contributions from people who aren't already packagers. Has this changed or is the implementation of non-packager PRs still pending?
The situation is the latter. There is a will to allow PRs from non-packagers and the infrastructure team discussed it last week again, but this hasn't been implemented yet.
Pierre
Thanks for the review and feedback, Tim. Dominik has already addressed most of the questions. Adding a couple of thoughts from my point of view.
On 30 January 2018 at 18:32, Dominik Perpeet dperpeet@redhat.com wrote:
I will try to address a few of the points.
On 01/30/2018 04:33 PM, Tim Flink wrote:
How are git repos in different namespaces of a larger repo ecosystem easier to link together than just urls that point to github or pagure.io ?
There has been some talk about pulling in tests that live in upstream repos which makes me think github. If this is true and we're going to be pulling other things in from github/pagure/gitlab/wherever, why not just have these shared repos in pagure.io?
There is no technical reason currently, other than the convenience of grouping. One of the possible options with grouped repos is the ease to add something in the future, such as modifying access control or even providing optimized Pagure workflows for common use cases.
Grouping similar content such as tests at the same place rather than scattering around various locations sounds natural to me. Having a decent test coverage at one place can serve as a nice source of examples for inspiration and allows to develop tooling which can enhance the process. For example, upon detecting test update execute it against all supported environment combinations.
Is the Fedora dist-git instance set up to enable ACL lists made up of users who are not already packagers and who don't otherwise have write access to the corresponding package repo? If not, how much extra work are we talking about? Are there available cycles to get the work done in short order?
Currently only fedora "packagers" have access to dist-git itself for actions such as cloning. There is a workaround to open pull requests from other git repos: https://fedoraproject.org/wiki/CI/Tests#Creating_pull_request
The tests namespace should allow access not only for Fedora packagers but for anyone willing to contribute. This should also resolve the problem that many QE guys are not in the packagers group and thus cannot commit directly. With this approach qe/dev collaboration on the test coverage should be easier.
As I understand it, RHEL QE has a setup with a "tests" namespace which stores all the tests for a package whose name matches the name of that repo from the tests namespace. If we use "tests" as the namespace here, I fear that we will be adding to confusion from the people who are used to the RHEL version of a "tests" namespace.
I believe all tests should be linked from the package's dist-git
- wherever that path may lead (in the dist-git itself or another
repo, even upstream).
We would also not realistically be able to replicate a similar setup in Fedora. There seemed to be interest in something like that when the issue came up on devel@ a few years ago which also makes me think it'd be unwise to close this door unless there is no other reasonable choice. If we do go the dist-git namespace route, have there been any alternative names proposed?
I see your point about possible confusion. On the other hand, the test namespace basically is an open-source sister of the internal one: It shares the same motivation, which is to have shared tests guarding the feature specification hosted outside / independently of the package dist-git branches.
However, we should not expect that it will be mirrorring exactly in the same way. There will be differences. IMO it does not make sense to have a separate git repository created for each and every package. For example, it makes much more sense to have a single ruby repository for integration tests of many ruby-* packages.
- For the PR testing case, how will the shared tests be
included in that PR set for testing purposes?
All relevant tests should be linked to the repo where they will gate changes (e.g. changing shell tests will gate on those passing on bash, ksh, et cetera).
- Will there be some delay in test/build kickoff on dist-git
repo change to allow for tests to be pushed around the same time that there are code changes? Otherwise are we planning to expect some transitory failures if both the tests and packages need to change at the same time? Are there plans to verify that things are properly re-run if those transitory errors happen?
The best way here is to reference the exact version of the tests you want, by git repo and revision. Then you don't depend on chance and you can safely change the tests first, then the dist-git repo.
I expect the scenario described above suits more to upstream unit tests (adding feature with test in one commit). For OS CI I think we should be more focused on stable features (tests guiding the specification), basic functionality and integration testing rather than checking every little feature. Thus I do not expect so many collisions of this type.
But as Dominik already mentioned, in cases where it makes sense it's possible to fetch tests from a specific branch or even commit to prevent unwanted failures. Another way is to only add the new test to the dist-git tests.yaml file only when it's ready for testing. Or use an appropriate test metadata to select only relevant tests from the whole test set.
The question of how to deal with non-packager write access for tests has been brought up a couple of times but I'm not sure it's been dealt with well yet.
I don't really like the idea of tests with arbitrary ACLs. As we work toward a vision of having CI gate all changes to packages before those changes are even accepted, changes to the tests associated with a package start approaching the importance of the non-test content for the same package.
If someone tosses an unproven test into a shared repo before going home for the weekend - there is a chance that could cause all sorts of problems from keeping a package's changes out of stable to breaking the build of a release-blocking image.
The ACL for a test repository should not be arbitrary. By default I expect package maintainer and qe should be granted write access but otherwise collaboration on the tests should work in identical way as with other open-source projects: A new contributor can start with proposing pull requests and later can be granted write access when trust is established between them and repo maintainer after good experience / number of successful merges.
I realize that there are plans to handle situations like this with automatic reversion but to the best of my knowledge, that system doesn't exist yet (an admin could back stuff out before the automation exists, too). If we demand that packagers go through a vetting and mentoring process even if they're a co-maintainer, why shouldn't we have a similar requirement for the folks who can change the tests which are responsible for gating the bits we release?
The starting point of test execution is the dist-git repo itself, which underlies the known restrictions. It is up to the packager to decide which other repo to link to, if at all. The tests could feasibly be in the dist-git repo itself or any other repo the packager trusts (and that is publicly accessible). If we should limit that choice somehow, we need to discuss it.
It perfectly makes sense (especially when tests are getting more importance because of gating) to ensure that anybody who has write access to the tests repo is aware of all important aspects of the process. Keeping a good/clear/concise documentation up-to-date should be one of the essential steps here.
It makes sense, just some concerns about the specifics of this proposal. Thanks for putting so much time into this.
Tim
Thank you for a nice bunch of good points! :)
psss...
-
'Dominik Perpeet' -
Pavel Raiskup -
Petr Splichal -
Petr Šplíchal
-
Pierre-Yves Chibon -
Tim Flink