Share Test Code

[Petr Splichal]

Hi!

Current Fedora CI documentation suggests: "The tests to be
executed are stored in the dist-git repositories. The tests are
stored or wrapped along-side the spec files..." [0] While working
on porting tests to Fedora CI, several times we've noted concerns
from developers/maintainers about placing test code directly into
dist-git repositories. A common question is how to efficiently
maintain tests & minimize test code duplication. There are some
nice real-life examples available:

Examples

There are several shells which implement the POSIX specification:
bash, ksh, mksh, zsh, dash. All of them share a significant amount
of test coverage and it does not make sense to commit & maintain
identical tests in five different repositories (+ possible
branches). See the pull request [1] for a bit of context.

Another example is Ruby: With about 80 packages related to Ruby on
Rails it would be useful and efficient to have a single place for
integration tests which verify that the framework is correctly
working after updating any of these packages. Conversely,
maintaining those tests in 80 repos would be a tedious task.

Proposal: Share!

So this is where the idea of shared test repository comes from.
In general, tests define how the software works and the basic
functionality of many packages doesn’t change that often. We try
hard to keep the backward compatibility where possible. Thus it
seems natural that, for such components, tests guarding the spec
could change at a slower pace than the distribution branches.

Main goals

 * Package source and tests are linked in a discoverable and
   unambiguous way (dist-git)
 * Prevent test code duplication (minimize test maintenance)
 * Catch incompatibilities early

How it could work? A new namespace "tests" would be created in
Fedora as suggested in [2], test code would reside there and the
tests in the rpms dist git could directly reference a specific
version of the tests to be run, e.g. with a pre_task:

    pre_tasks:
    - name: Fetch the tests
      git:
        repo: https://src.fedoraproject.org/tests/shell.git
        dest: tests
        version: master

Also, standard-test-roles [3] should be enhanced to make it really
easy to fetch tests from a remote repo. Ansible git module is
quite simple but with standard-test-roles it could be even
shorter. The tests.yml file could look like this (we could support
fetching from multiple repositories):

    - role: standard-test-beakerlib
      tags:
      - classic
      - container
      - atomic
      tests:
      - smoke
      repo:
      - https://src.fedoraproject.org/tests/shell.git
      version: master

A consistent location for shared tests is a good source of
examples and helps drive automation in the future, e.g. trigger CI
jobs on test changes.

For stable packages providing specification-like functionality a
single master branch could be used, for those components which
change more, dist-git tests.yml could pick a proper branch or even
a specific commit (if having concerns about updated tests breaking
the continuous integration).

This seems to give a nice flexibility while at the same time
minimizing test code duplication. Referencing specific tests from
rpms dist git ties tests and code together very well. Storing the
test code in the tests namespace is only one option, any git repo
accessible from the CI pipeline would suffice.

Just One Place?

We could possibly go one more step further: Shall we aim at having
just a single test code for testing both Fedora and Red Hat
Enterprise Linux? At least for those cases where this is possible:
There might be some licensing or compatibility issues. For
example, speaking about Beaker: restraint harness can also be
instructed to fetch tests directly from git so the identical test
code could perhaps be used for running tests in Beaker as well.

A recent experience of a libselinux pull request [4] shows how
cumbersome it can be to keep both upstream and downstream version
of the test code in sync. Wouldn't it be great to have everything
at a single place and keep the test code up-to-date only there?
Instead of doing several rounds of "pull request - give feedback -
update internal tests - update the pull request - find another
issue - give feedback..." with many unnecessary steps and
reminders just directly collaborating on a single test repository?

When downstream testing is just a branch of upstream testing, we
can use our standard open source workflows. Just as in upstream,
there would likely be a downstream shared git repository that
branches from the upstream one.

Bonus

As a bonus we would keep the dist-git for such repositories a bit
more consistent as a place for metadata only: Build metadata (spec
file = how to build the package) and test metadata (tests.yml =
how to test the package).

Plus we would get a more streamlined test maintenance as QE and
volunteers willing to help with improving the test coverage could
be granted direct access to individual package tests repo. Not
every QE is a Fedora packager so contributing to dist-git is not
that straightforward.

Notes

There are cons of this approach as well. It isn’t possible to have
new feature code and tests submitted in a single commit or
triggering test execution for every change in tests (CI for
tests). There is a need for handling relevance of individual tests
for different environments. But the maintenance simplification
seems to be worth it, based on experience of the BaseOS QE team
which has been maintaining thousands of tests for many years and
using them for testing large number of supported versions of Red
Hat Enterprise Linux.

The overall benefits also depend on which test types we are
speaking about. Tests should always be as far upstream as
possible. Unit tests generally make more sense in upstream. For
integration tests it might make more sense to live in the shared
Fedora tests repository. Recommendations about which tests fit
well into this "tests" namespace will evolve, but it will be up to
each package to choose where to place its tests.

So?

What are your thoughts on this? Does it make sense to you?

psss...

[0] https://fedoraproject.org/wiki/CI#Tests_locations
[1] https://src.fedoraproject.org/rpms/bash/pull-request/1
[2] https://pagure.io/fedora-infrastructure/issue/6478
[3] https://pagure.io/standard-test-roles/
[4] https://src.fedoraproject.org/rpms/libselinux/pull-request/1

[Pavel Raiskup]

Thanks, makes sense to me.  The tests/ namespace is much better approach than
some per-package, non-standard place (on github, let's say) to share the test
code.

Ideally, the test automation should (except for explicit tests references in
`pre_tasks:`) download implicitly `tests/FOO` for `FOO` component, when
the `tests/FOO` is existing.  Also `fedpkg clone-tests <COMPONENT>` would
be convenient thing..

Pavel

[Tim Flink]

On Mon, 20 Nov 2017 16:30:22 +0100
Petr Splichal <psplicha(a)redhat.com&gt; wrote:

>  Hi!
>  
>  Current Fedora CI documentation suggests: "The tests to be
>  executed are stored in the dist-git repositories. The tests are
>  stored or wrapped along-side the spec files..." [0] While working
>  on porting tests to Fedora CI, several times we've noted concerns
>  from developers/maintainers about placing test code directly into
>  dist-git repositories. A common question is how to efficiently
>  maintain tests & minimize test code duplication. There are some
>  nice real-life examples available:
>  
>  Examples
>  
>  There are several shells which implement the POSIX specification:
>  bash, ksh, mksh, zsh, dash. All of them share a significant amount
>  of test coverage and it does not make sense to commit & maintain
>  identical tests in five different repositories (+ possible
>  branches). See the pull request [1] for a bit of context.
>  
>  Another example is Ruby: With about 80 packages related to Ruby on
>  Rails it would be useful and efficient to have a single place for
>  integration tests which verify that the framework is correctly
>  working after updating any of these packages. Conversely,
>  maintaining those tests in 80 repos would be a tedious task.
>  
>  Proposal: Share!
>  
>  So this is where the idea of shared test repository comes from.
>  In general, tests define how the software works and the basic
>  functionality of many packages doesn’t change that often. We try
>  hard to keep the backward compatibility where possible. Thus it
>  seems natural that, for such components, tests guarding the spec
>  could change at a slower pace than the distribution branches.
>  
>  Main goals
>  
>   * Package source and tests are linked in a discoverable and
>     unambiguous way (dist-git)
>   * Prevent test code duplication (minimize test maintenance)
>   * Catch incompatibilities early 

How are git repos in different namespaces of a larger repo ecosystem
easier to link together than just urls that point to github or
pagure.io ?

There has been some talk about pulling in tests that live in upstream
repos which makes me think github. If this is true and we're going to
be pulling other things in from github/pagure/gitlab/wherever, why not
just have these shared repos in pagure.io?


>  How it could work? A new namespace "tests" would be created
> in
>  Fedora as suggested in [2], test code would reside there and the
>  tests in the rpms dist git could directly reference a specific
>  version of the tests to be run, e.g. with a pre_task: 

Is the Fedora dist-git instance set up to enable ACL lists made up of
users who are not already packagers and who don't otherwise have write
access to the corresponding package repo? If not, how much extra work
are we talking about? Are there available cycles to get the work done
in short order?

As I understand it, RHEL QE has a setup with a "tests" namespace which
stores all the tests for a package whose name matches the name of that
repo from the tests namespace. If we use "tests" as the namespace here,
I fear that we will be adding to confusion from the people who are used
to the RHEL version of a "tests" namespace. We would also not
realistically be able to replicate a similar setup in Fedora. There
seemed to be interest in something like that when the issue came up on
devel@ a few years ago which also makes me think it'd be unwise to
close this door unless there is no other reasonable choice. If we do go
the dist-git namespace route, have there been any alternative names
proposed?


>      pre_tasks:
>      - name: Fetch the tests
>        git:
>          repo: https://src.fedoraproject.org/tests/shell.git
>          dest: tests
>          version: master
>  
>  Also, standard-test-roles [3] should be enhanced to make it really
>  easy to fetch tests from a remote repo. Ansible git module is
>  quite simple but with standard-test-roles it could be even
>  shorter. The tests.yml file could look like this (we could support
>  fetching from multiple repositories):
>  
>      - role: standard-test-beakerlib
>        tags:
>        - classic
>        - container
>        - atomic
>        tests:
>        - smoke
>        repo:
>        - https://src.fedoraproject.org/tests/shell.git
>        version: master
>  
>  A consistent location for shared tests is a good source of
>  examples and helps drive automation in the future, e.g. trigger CI
>  jobs on test changes.
>  
>  For stable packages providing specification-like functionality a
>  single master branch could be used, for those components which
>  change more, dist-git tests.yml could pick a proper branch or even
>  a specific commit (if having concerns about updated tests breaking
>  the continuous integration).
>  
>  This seems to give a nice flexibility while at the same time
>  minimizing test code duplication. Referencing specific tests from
>  rpms dist git ties tests and code together very well. Storing the
>  test code in the tests namespace is only one option, any git repo
>  accessible from the CI pipeline would suffice. 
I don't really have a problem with the idea of having a shared
repository in the interest of decreasing duplication. That being said,
I do have some questions about some more of the details behind how this
would work.

While this isn't the most common case, it is possible to have a
situation where a package changes and the tests need to change at the
same time. It's not necessarily a fatal problem but it brings up some
questions:

 - For the PR testing case, how will the shared tests be included in
   that PR set for testing purposes?

 - Will there be some delay in test/build kickoff on dist-git repo
   change to allow for tests to be pushed around the same time that
   there are code changes? Otherwise are we planning to expect some
   transitory failures if both the tests and packages need to change at
   the same time? Are there plans to verify that things are properly
   re-run if those transitory errors happen?


>  Just One Place?
>  
>  We could possibly go one more step further: Shall we aim at having
>  just a single test code for testing both Fedora and Red Hat
>  Enterprise Linux? At least for those cases where this is possible:
>  There might be some licensing or compatibility issues. For
>  example, speaking about Beaker: restraint harness can also be
>  instructed to fetch tests directly from git so the identical test
>  code could perhaps be used for running tests in Beaker as well.
>  
>  A recent experience of a libselinux pull request [4] shows how
>  cumbersome it can be to keep both upstream and downstream version
>  of the test code in sync. Wouldn't it be great to have everything
>  at a single place and keep the test code up-to-date only there?
>  Instead of doing several rounds of "pull request - give feedback -
>  update internal tests - update the pull request - find another
>  issue - give feedback..." with many unnecessary steps and
>  reminders just directly collaborating on a single test repository?
>  
>  When downstream testing is just a branch of upstream testing, we
>  can use our standard open source workflows. Just as in upstream,
>  there would likely be a downstream shared git repository that
>  branches from the upstream one.
>  
>  Bonus
>  
>  As a bonus we would keep the dist-git for such repositories a bit
>  more consistent as a place for metadata only: Build metadata (spec
>  file = how to build the package) and test metadata (tests.yml =
>  how to test the package).
>  
>  Plus we would get a more streamlined test maintenance as QE and
>  volunteers willing to help with improving the test coverage could
>  be granted direct access to individual package tests repo. Not
>  every QE is a Fedora packager so contributing to dist-git is not
>  that straightforward. 

The question of how to deal with non-packager write access for tests has
been brought up a couple of times but I'm not sure it's been dealt with
well yet.

I don't really like the idea of tests with arbitrary ACLs. As we work
toward a vision of having CI gate all changes to packages before those
changes are even accepted, changes to the tests associated with a
package start approaching the importance of the non-test content for
the same package.

If someone tosses an unproven test into a shared repo before going
home for the weekend - there is a chance that could cause all
sorts of problems from keeping a package's changes out of stable
to breaking the build of a release-blocking image.

I realize that there are plans to handle situations like this with
automatic reversion but to the best of my knowledge, that system
doesn't exist yet (an admin could back stuff out before the automation
exists, too). If we demand that packagers go through a vetting and
mentoring process even if they're a co-maintainer, why shouldn't we have
a similar requirement for the folks who can change the tests which are
responsible for gating the bits we release?

>  Notes
>  
>  There are cons of this approach as well. It isn’t possible to have
>  new feature code and tests submitted in a single commit or
>  triggering test execution for every change in tests (CI for
>  tests). There is a need for handling relevance of individual tests
>  for different environments. But the maintenance simplification
>  seems to be worth it, based on experience of the BaseOS QE team
>  which has been maintaining thousands of tests for many years and
>  using them for testing large number of supported versions of Red
>  Hat Enterprise Linux.
>  
>  The overall benefits also depend on which test types we are
>  speaking about. Tests should always be as far upstream as
>  possible. Unit tests generally make more sense in upstream. For
>  integration tests it might make more sense to live in the shared
>  Fedora tests repository. Recommendations about which tests fit
>  well into this "tests" namespace will evolve, but it will be up to
>  each package to choose where to place its tests.
>  
>  So?
>  
>  What are your thoughts on this? Does it make sense to you? 
It makes sense, just some concerns about the specifics of this
proposal. Thanks for putting so much time into this.

Tim

>  psss...
>  
>  [0] https://fedoraproject.org/wiki/CI#Tests_locations
>  [1] https://src.fedoraproject.org/rpms/bash/pull-request/1
>  [2] https://pagure.io/fedora-infrastructure/issue/6478
>  [3] https://pagure.io/standard-test-roles/
>  [4] https://src.fedoraproject.org/rpms/libselinux/pull-request/1
>  _______________________________________________
>  CI mailing list -- ci(a)lists.fedoraproject.org
>  To unsubscribe send an email to ci-leave(a)lists.fedoraproject.org

['Dominik Perpeet']

I will try to address a few of the points.

On 01/30/2018 04:33 PM, Tim Flink wrote:
>  How are git repos in different namespaces of a larger repo ecosystem
>  easier to link together than just urls that point to github or
>  pagure.io ?
>
>  There has been some talk about pulling in tests that live in upstream
>  repos which makes me think github. If this is true and we're going to
>  be pulling other things in from github/pagure/gitlab/wherever, why not
>  just have these shared repos in pagure.io? 
There is no technical reason currently,
other than the convenience of
grouping. One of the possible options with grouped repos is the ease to
add something in the future, such as modifying access control or even
providing optimized Pagure workflows for common use cases.
>  Is the Fedora dist-git instance set up to enable ACL lists made up of
>  users who are not already packagers and who don't otherwise have write
>  access to the corresponding package repo? If not, how much extra work
>  are we talking about? Are there available cycles to get the work done
>  in short order? 
Currently only fedora "packagers" have access to dist-git
itself for
actions such as cloning. There is a workaround to open pull requests
from other git repos:
https://fedoraproject.org/wiki/CI/Tests#Creating_pull_request

>  As I understand it, RHEL QE has a setup with a "tests" namespace which
>  stores all the tests for a package whose name matches the name of that
>  repo from the tests namespace. If we use "tests" as the namespace here,
>  I fear that we will be adding to confusion from the people who are used
>  to the RHEL version of a "tests" namespace.  
I believe all tests should
be linked from the package's dist-git -
wherever that path may lead (in the dist-git itself or another repo,
even upstream).
>   - For the PR testing case, how will the shared tests be included in
>     that PR set for testing purposes? 
All relevant tests should be linked to the
repo where they will gate
changes (e.g. changing shell tests will gate on those passing on bash,
ksh, et cetera).
>   - Will there be some delay in test/build kickoff on dist-git repo
>     change to allow for tests to be pushed around the same time that
>     there are code changes? Otherwise are we planning to expect some
>     transitory failures if both the tests and packages need to change at
>     the same time? Are there plans to verify that things are properly
>     re-run if those transitory errors happen? 
The best way here is to reference the
exact version of the tests you
want, by git repo and revision. Then you don't depend on chance and you
can safely change the tests first, then the dist-git repo.
>  I realize that there are plans to handle situations like this with
>  automatic reversion but to the best of my knowledge, that system
>  doesn't exist yet (an admin could back stuff out before the automation
>  exists, too). If we demand that packagers go through a vetting and
>  mentoring process even if they're a co-maintainer, why shouldn't we have
>  a similar requirement for the folks who can change the tests which are
>  responsible for gating the bits we release? 
The starting point of test execution is
the dist-git repo itself, which
underlies the known restrictions. It is up to the packager to decide
which other repo to link to, if at all. The tests could feasibly be in
the dist-git repo itself or any other repo the packager trusts (and that
is publicly accessible). If we should limit that choice somehow, we need
to discuss it.
>  It makes sense, just some concerns about the specifics of this
>  proposal. Thanks for putting so much time into this.
>
>  Tim
>  
Thank you for your feedback!

[Tim Flink]

On Tue, 30 Jan 2018 18:32:02 +0100
'Dominik Perpeet' <dperpeet(a)redhat.com&gt; wrote:

>  I will try to address a few of the points.
>  
>  On 01/30/2018 04:33 PM, Tim Flink wrote:
>  > How are git repos in different namespaces of a larger repo ecosystem
>  > easier to link together than just urls that point to github or
>  > pagure.io ?
>  >
>  > There has been some talk about pulling in tests that live in
>  > upstream repos which makes me think github. If this is true and
>  > we're going to be pulling other things in from
>  > github/pagure/gitlab/wherever, why not just have these shared repos
>  > in pagure.io?  
>  There is no technical reason currently, other than the convenience of
>  grouping. One of the possible options with grouped repos is the ease
>  to add something in the future, such as modifying access control or
>  even providing optimized Pagure workflows for common use cases. 
This didn't make a lot of sense to me until we spoke in person earlier
today.

The bit I was missing was that there's a chance that Pagure could be
enhanced to have a workflow which covers multiple repos with the same
PR which would solve some of the concerns I have.


>  > Is the Fedora dist-git instance set up to enable ACL lists made
> up
>  > of users who are not already packagers and who don't otherwise have
>  > write access to the corresponding package repo? If not, how much
>  > extra work are we talking about? Are there available cycles to get
>  > the work done in short order?  
>  Currently only fedora "packagers" have access to dist-git itself for
>  actions such as cloning. There is a workaround to open pull requests
>  from other git repos:
>  https://fedoraproject.org/wiki/CI/Tests#Creating_pull_request 
I thought that PRs were the main solution to allowing contributions
from people who aren't already packagers. Has this changed or is the
implementation of non-packager PRs still pending?


>  > As I understand it, RHEL QE has a setup with a "tests"
> namespace
>  > which stores all the tests for a package whose name matches the
>  > name of that repo from the tests namespace. If we use "tests" as
>  > the namespace here, I fear that we will be adding to confusion from
>  > the people who are used to the RHEL version of a "tests"
>  > namespace.   
>  I believe all tests should be linked from the package's dist-git -
>  wherever that path may lead (in the dist-git itself or another repo,
>  even upstream). 
I don't understand how this relates. There is a similarly named
system/setup used by many of the same folks who will be contributing
tests and working on tests in this new CI paradigm. I'm concerned that
the similar name and function will lead to confusion and a steeper
learning curve. Specifically, I'm talking about the tests/ namespace
used inside Red Hat by QE folk to store tests and test-related data
outside of the package's repo.


>  >  - For the PR testing case, how will the shared tests be
> included in
>  >    that PR set for testing purposes?  
>  All relevant tests should be linked to the repo where they will gate
>  changes (e.g. changing shell tests will gate on those passing on bash,
>  ksh, et cetera). 
That's fair enough but my question was more around situations where you
need both changes to land at the same time so that the test doesn't
fail. Having a modified workflow in Pagure to support multiple repos in
the same PR is one solution to one problem - what about tests which are
stored in upstream projects or just outside of dist-git? Is it going to
be possible to change both the tests and the packages at the same time
without seeing what are effectively false positives from the various CI
systems? If so, what is the planned workflow for dealing with
simultaneous changes?


>  >  - Will there be some delay in test/build kickoff on dist-git
> repo
>  >    change to allow for tests to be pushed around the same time that
>  >    there are code changes? Otherwise are we planning to expect some
>  >    transitory failures if both the tests and packages need to
>  > change at the same time? Are there plans to verify that things are
>  > properly re-run if those transitory errors happen?  
>  The best way here is to reference the exact version of the tests you
>  want, by git repo and revision. Then you don't depend on chance and
>  you can safely change the tests first, then the dist-git repo. 
Does that mean that test runs won't be triggered on test change? Unless
I'm missing something, how will we be dealing with the eventual issues
where the git commit is out of date and propagates incorrect data
through the pipeline?

For example, say that the tests for the package foo point to more tests
in a foo-tests.git repo. If I change those tests and don't change the
package - a triggered test run would likely indicate that my change
continues to pass without issue.


>  > I realize that there are plans to handle situations like this
> with
>  > automatic reversion but to the best of my knowledge, that system
>  > doesn't exist yet (an admin could back stuff out before the
>  > automation exists, too). If we demand that packagers go through a
>  > vetting and mentoring process even if they're a co-maintainer, why
>  > shouldn't we have a similar requirement for the folks who can
>  > change the tests which are responsible for gating the bits we
>  > release?  
>  The starting point of test execution is the dist-git repo itself,
>  which underlies the known restrictions. It is up to the packager to
>  decide which other repo to link to, if at all. The tests could
>  feasibly be in the dist-git repo itself or any other repo the
>  packager trusts (and that is publicly accessible). If we should limit
>  that choice somehow, we need to discuss it. 
I'm not sure I understand your response here. Does that mean that
contributions from non-packagers is not deemed important enough to
worry about at this time? If so, how do we plan to get ported tests
from downstream RHEL into Fedora if QEs are currently doing much of
the porting work?

>  > It makes sense, just some concerns about the specifics of this
>  > proposal. Thanks for putting so much time into this.
>  >
>  > Tim
>  >  
>  Thank you for your feedback! 
Not a problem. Thanks for your quick response.

Tim

[Pierre-Yves Chibon]

On Wed, Jan 31, 2018 at 02:21:45PM +0100, Tim Flink wrote:
>  On Tue, 30 Jan 2018 18:32:02 +0100
>  'Dominik Perpeet' <dperpeet(a)redhat.com&gt; wrote:
>  
>  > I will try to address a few of the points.
>  > 
>  > On 01/30/2018 04:33 PM, Tim Flink wrote:
>  > > How are git repos in different namespaces of a larger repo ecosystem
>  > > easier to link together than just urls that point to github or
>  > > pagure.io ?
>  > >
>  > > There has been some talk about pulling in tests that live in
>  > > upstream repos which makes me think github. If this is true and
>  > > we're going to be pulling other things in from
>  > > github/pagure/gitlab/wherever, why not just have these shared repos
>  > > in pagure.io?  
>  > There is no technical reason currently, other than the convenience of
>  > grouping. One of the possible options with grouped repos is the ease
>  > to add something in the future, such as modifying access control or
>  > even providing optimized Pagure workflows for common use cases.
>  
>  This didn't make a lot of sense to me until we spoke in person earlier
>  today.
>  
>  The bit I was missing was that there's a chance that Pagure could be
>  enhanced to have a workflow which covers multiple repos with the same
>  PR which would solve some of the concerns I have. 
I think the idea is to find a way to link multiple PRs together (something as
simple as using: Requires <url to PR> in the commit message for example), rather
than figuring out how to do 1 PR against multiple repos (which in the case of
dist-git practically do not overlap in anyway).
 
>  > > Is the Fedora dist-git instance set up to enable ACL lists
> made up
>  > > of users who are not already packagers and who don't otherwise have
>  > > write access to the corresponding package repo? If not, how much
>  > > extra work are we talking about? Are there available cycles to get
>  > > the work done in short order?  
>  > Currently only fedora "packagers" have access to dist-git itself for
>  > actions such as cloning. There is a workaround to open pull requests
>  > from other git repos:
>  > https://fedoraproject.org/wiki/CI/Tests#Creating_pull_request
>  
>  I thought that PRs were the main solution to allowing contributions
>  from people who aren't already packagers. Has this changed or is the
>  implementation of non-packager PRs still pending? 
The situation is the latter. There is a will to allow PRs from non-packagers and
the infrastructure team discussed it last week again, but this hasn't been
implemented yet.


Pierre

[Petr Šplíchal]

Thanks for the review and feedback, Tim. Dominik has already
addressed most of the questions. Adding a couple of thoughts
from my point of view.

On 30 January 2018 at 18:32, Dominik Perpeet <dperpeet(a)redhat.com&gt; wrote:
>  I will try to address a few of the points.
>
>  On 01/30/2018 04:33 PM, Tim Flink wrote:
>  > How are git repos in different namespaces of a larger repo
>  > ecosystem easier to link together than just urls that point to
>  > github or pagure.io ?
>  >
>  > There has been some talk about pulling in tests that live in
>  > upstream repos which makes me think github. If this is true and
>  > we're going to be pulling other things in from
>  > github/pagure/gitlab/wherever, why not just have these shared
>  > repos in pagure.io?
>
>  There is no technical reason currently, other than the
>  convenience of grouping. One of the possible options with
>  grouped repos is the ease to add something in the future, such
>  as modifying access control or even providing optimized Pagure
>  workflows for common use cases. 
Grouping similar content such as tests at the same place rather
than scattering around various locations sounds natural to me.
Having a decent test coverage at one place can serve as a nice
source of examples for inspiration and allows to develop tooling
which can enhance the process. For example, upon detecting test
update execute it against all supported environment combinations.

>  > Is the Fedora dist-git instance set up to enable ACL lists made
>  > up of users who are not already packagers and who don't
>  > otherwise have write access to the corresponding package repo?
>  > If not, how much extra work are we talking about? Are there
>  > available cycles to get the work done in short order?
>
>  Currently only fedora "packagers" have access to dist-git itself
>  for actions such as cloning. There is a workaround to open pull
>  requests from other git repos:
>  https://fedoraproject.org/wiki/CI/Tests#Creating_pull_request 
The tests namespace should allow access not only for Fedora
packagers but for anyone willing to contribute. This should also
resolve the problem that many QE guys are not in the packagers
group and thus cannot commit directly. With this approach qe/dev
collaboration on the test coverage should be easier.

>  > As I understand it, RHEL QE has a setup with a
> "tests"
>  > namespace which stores all the tests for a package whose name
>  > matches the name of that repo from the tests namespace. If we
>  > use "tests" as the namespace here, I fear that we will be
>  > adding to confusion from the people who are used to the RHEL
>  > version of a "tests" namespace.
>
>  I believe all tests should be linked from the package's dist-git
>  - wherever that path may lead (in the dist-git itself or another
>  repo, even upstream).
>
>  > We would also not realistically be able to replicate a similar
>  > setup in Fedora.  There seemed to be interest in something like
>  > that when the issue came up on devel@ a few years ago which
>  > also makes me think it'd be unwise to close this door unless
>  > there is no other reasonable choice. If we do go the dist-git
>  > namespace route, have there been any alternative names
>  > proposed? 
I see your point about possible confusion. On the other hand,
the test namespace basically is an open-source sister of the
internal one: It shares the same motivation, which is to have
shared tests guarding the feature specification hosted outside /
independently of the package dist-git branches.

However, we should not expect that it will be mirrorring exactly
in the same way. There will be differences. IMO it does not make
sense to have a separate git repository created for each and every
package. For example, it makes much more sense to have a single
ruby repository for integration tests of many ruby-* packages.

>  >  - For the PR testing case, how will the shared tests be
>  >  included in that PR set for testing purposes?
>
>  All relevant tests should be linked to the repo where they will
>  gate changes (e.g. changing shell tests will gate on those
>  passing on bash, ksh, et cetera).
>
>  >  - Will there be some delay in test/build kickoff on dist-git
>  >  repo change to allow for tests to be pushed around the same
>  >  time that there are code changes? Otherwise are we planning to
>  >  expect some transitory failures if both the tests and packages
>  >  need to change at the same time? Are there plans to verify
>  >  that things are properly re-run if those transitory errors
>  >  happen?
>
>  The best way here is to reference the exact version of the tests
>  you want, by git repo and revision. Then you don't depend on
>  chance and you can safely change the tests first, then the
>  dist-git repo. 
I expect the scenario described above suits more to upstream unit
tests (adding feature with test in one commit). For OS CI I think
we should be more focused on stable features (tests guiding the
specification), basic functionality and integration testing rather
than checking every little feature. Thus I do not expect so many
collisions of this type.

But as Dominik already mentioned, in cases where it makes sense
it's possible to fetch tests from a specific branch or even commit
to prevent unwanted failures. Another way is to only add the new
test to the dist-git tests.yaml file only when it's ready for
testing. Or use an appropriate test metadata to select only
relevant tests from the whole test set.

>  > The question of how to deal with non-packager write access for
>  > tests has been brought up a couple of times but I'm not sure
>  > it's been dealt with well yet.
>  >
>  > I don't really like the idea of tests with arbitrary ACLs. As
>  > we work toward a vision of having CI gate all changes to
>  > packages before those changes are even accepted, changes to the
>  > tests associated with a package start approaching the
>  > importance of the non-test content for the same package.
>  >
>  > If someone tosses an unproven test into a shared repo before
>  > going home for the weekend - there is a chance that could cause
>  > all sorts of problems from keeping a package's changes out of
>  > stable to breaking the build of a release-blocking image. 
The ACL for a test repository should not be arbitrary. By default
I expect package maintainer and qe should be granted write access
but otherwise collaboration on the tests should work in identical
way as with other open-source projects: A new contributor can
start with proposing pull requests and later can be granted write
access when trust is established between them and repo maintainer
after good experience / number of successful merges.

>  > I realize that there are plans to handle situations like this
>  > with automatic reversion but to the best of my knowledge, that
>  > system doesn't exist yet (an admin could back stuff out before
>  > the automation exists, too). If we demand that packagers go
>  > through a vetting and mentoring process even if they're a
>  > co-maintainer, why shouldn't we have a similar requirement for
>  > the folks who can change the tests which are responsible for
>  > gating the bits we release?
>
>  The starting point of test execution is the dist-git repo
>  itself, which underlies the known restrictions. It is up to the
>  packager to decide which other repo to link to, if at all. The
>  tests could feasibly be in the dist-git repo itself or any other
>  repo the packager trusts (and that is publicly accessible). If
>  we should limit that choice somehow, we need to discuss it. 
It perfectly makes sense (especially when tests are getting more
importance because of gating) to ensure that anybody who has write
access to the tests repo is aware of all important aspects of the
process. Keeping a good/clear/concise documentation up-to-date
should be one of the essential steps here.

>  > It makes sense, just some concerns about the specifics of this
>  > proposal. Thanks for putting so much time into this.
>  >
>  > Tim 
Thank you for a nice bunch of good points! :)

psss...