How to opt-in to gating by the CI pipeline

[Stef Walter]

I believe we can have packages opt in on the CI pipeline like this.
Pingou, Ari and I discussed this a bit:

 * A package with a tests/ directory according to the spec will
   have its tests invoked according to the spec. If those tests fail
   the package may be gated.

In other words the presence of tests in dist-git will be treated as an
acknowledgement from the dist-git repo that it is okay to gate the
package based on those tests.

As posted here last week, certain tests are appropriate for certain test
contexts. Certain tests are appropriate for containers, other for Atomic
Host systems, and others for classic installed systems.

At the current time the CI pipeline [0] only tests packages and tests
marked for Atomic Host, but in the future that scope can grow. This
initial scope would obviously also apply to gating.

Does that make sense? Any objections?

Cheers,

Stef

[Matthew Miller]

On Thu, Aug 03, 2017 at 04:14:21PM +0200, Stef Walter wrote:
>  I believe we can have packages opt in on the CI pipeline like this.
>  Pingou, Ari and I discussed this a bit:
>   * A package with a tests/ directory according to the spec will
>     have its tests invoked according to the spec. If those tests fail
>     the package may be gated.
>  In other words the presence of tests in dist-git will be treated as an
>  acknowledgement from the dist-git repo that it is okay to gate the
>  package based on those tests. 
Do we want to allow for the possibility of "advisory" tests, or is it
always all or nothing? You have more experience than me in doing this
on a real project; I can imagine wanting to create tests without making
them blocking, and then enable them as gating when you have confidence.
But maybe it really is better to force the issue.

>  As posted here last week, certain tests are appropriate for certain
> test
>  contexts. Certain tests are appropriate for containers, other for Atomic
>  Host systems, and others for classic installed systems. 
So... how does this work? What things are allowed to gate changes
globally? How does this intersect with Modulairty and Arbitrary
Branching?


>  At the current time the CI pipeline [0] only tests packages and
> tests
>  marked for Atomic Host, but in the future that scope can grow. This
>  initial scope would obviously also apply to gating. 

Are the *packages* marked or the *tests* marked?

-- 
Matthew Miller
<mattdm(a)fedoraproject.org&gt;
Fedora Project Leader

[Stef Walter]

On 03.08.2017 16:22, Matthew Miller wrote:
>  On Thu, Aug 03, 2017 at 04:14:21PM +0200, Stef Walter wrote:
> > I believe we can have packages opt in on the CI pipeline like this.
> > Pingou, Ari and I discussed this a bit:
> >  * A package with a tests/ directory according to the spec will
> >    have its tests invoked according to the spec. If those tests fail
> >    the package may be gated.
> > In other words the presence of tests in dist-git will be treated as an
> > acknowledgement from the dist-git repo that it is okay to gate the
> > package based on those tests.
>  
>  Do we want to allow for the possibility of "advisory" tests, or is it
>  always all or nothing? You have more experience than me in doing this
>  on a real project; I can imagine wanting to create tests without making
>  them blocking, and then enable them as gating when you have confidence.
>  But maybe it really is better to force the issue. 
While tests are in development they can be "advisory". Packagers can
place tests in a dist-git repository and not tag them for a specific
test context or target module, like 'atomic', 'container',
'classic'.

This will allow the packager to invoke the tests locally, and coordinate
work on them with others, without having them gate any particular
deliverable.

https://fedoraproject.org/wiki/CI/Tests

and

https://fedoraproject.org/wiki/Changes/InvokingTests

At the current time there is no plan for a pipeline to invoke such
tests, but in the future it may be possible to add such a feature ...
particularly if we get to the point where tests results are displayed in
a pull-request type interface.

> > As posted here last week, certain tests are appropriate for
> certain test
> > contexts. Certain tests are appropriate for containers, other for Atomic
> > Host systems, and others for classic installed systems.
>  
>  So... how does this work? What things are allowed to gate changes
>  globally? How does this intersect with Modulairty and Arbitrary
>  Branching? 
In the end we should be gating inclusion in modules. A test and package
that works in one module may be completely broken elsewhere. As we
proceed through the carry on effects of Modularity "globally" starts to
become a non-sequitor.

There are currently discussions in the Atomic Host WG about defining
Atomic Host as a module.

Fedora will take time to grow to become fully modular. The CI system
will have to make pragmatic choices until we get there.

Until that time, I believe that a test tagged 'atomic' would gate the
package "globally" if it fails. The packager/maintainer can either fix
the test or untag 'atomic' on the test if this proves problematic.

> > At the current time the CI pipeline [0] only tests packages and
> tests
> > marked for Atomic Host, but in the future that scope can grow. This
> > initial scope would obviously also apply to gating.
>  
>  
>  Are the *packages* marked or the *tests* marked? 
I should have said "only tests packages included in and tests marked for
Atomic Host". This is the initial scope, because we have to start
somewhere, rather than boil the ocean.

Individual tests are tagged for 'atomic' (and other tags).

The tagged tests live in dist-git repositories of packages and modules,
and so one could say the packages are tagged by the transitive property.

Cheers,

Stef

[Pierre-Yves Chibon]

On Thu, Aug 03, 2017 at 04:14:21PM +0200, Stef Walter wrote:
>  I believe we can have packages opt in on the CI pipeline like this.
>  Pingou, Ari and I discussed this a bit:
>  
>   * A package with a tests/ directory according to the spec will
>     have its tests invoked according to the spec. If those tests fail
>     the package may be gated.
>  
>  In other words the presence of tests in dist-git will be treated as an
>  acknowledgement from the dist-git repo that it is okay to gate the
>  package based on those tests. 
I have added a blob on the wiki page to officially document this behaviour:
https://fedoraproject.org/wiki/CI#Opting-in


Pierre