Trying to get some tests working

[Igor Gnatenko]

I'm trying to have shared tests between all 400+ rust packages.

I have few problems though:

* There is no package NEVRA in test environment so the shared test doesn't
know what to test - it is necessary to know which test to run
* No dist-git sources of a package - while trying to workaround previous
problem I have found that dist-git doesn't get into the environment so I
can't do something like dnf builddep

Please help me ;)

P.S> I'm using standard-test-basic role.

Thanks in advance!
-- 

-Igor Gnatenko

[Andrei Stepanov]

On Thu, Jul 19, 2018 at 9:08 PM Igor Gnatenko <
ignatenkobrain(a)fedoraproject.org&gt; wrote:

>  I'm trying to have shared tests between all 400+ rust packages.
>
>  I have few problems though:
>
>  * There is no package NEVRA in test environment so the shared test doesn't
>  know what to test - it is necessary to know which test to run.
>  
We trigger tests for dist-git.
I think about dist-git as a SRPM.
SRPM can produce many RPMs.
CI pipeline: install all RPMS from Koji build and run tests from /tests for
all installed RPMs.
The same approach is for manual testing.
Conclusion: test-environment-system has installed RPMS.
Tests from /tests dir check test-environment-system as a whole.

https://fedoraproject.org/wiki/CI/Standard_Test_Interface doesn't define
NEVRA.
Reason: STI tests are for all test-environment-system (all installed
packages).
/tests directory from dist-git knows that packages goes from dist-git/pkg
/tests/tests.yml is written for each SRPM package.
/tests/tests.yml knows that it will run against: SRPM + distro version.
/tests/tests.yml for Fedora 27 and for Fedora 28 can are different files.
If you have common tests, you have different /tests/tests.yml where you can
specify exactly that this tests are for this SRPM and for Branch 28 (for
example).



>  * No dist-git sources of a package - while trying to workaround
> previous
>  problem I have found that dist-git doesn't get into the environment so I
>  can't do something like dnf builddep
>
>  Please help me ;)
>  

We should decide: if tests can exist independently without dist-git.
For now, we presume they can.
This is current state.
If yes. There is nothing to do.

If no. We should tight together dist-git branch + /tests as one entity.
And copy /tests to test-environment system together with dist-git.

 If we decide to change this, than we need to make correspondent changes to
STI.
https://fedoraproject.org/wiki/CI/Standard_Test_Interface



>  P.S> I'm using standard-test-basic role.
>
>  Thanks in advance!
>  --
>
>  -Igor Gnatenko
>  _______________________________________________
>  CI mailing list -- ci(a)lists.fedoraproject.org
>  To unsubscribe send an email to ci-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
>
> https://lists.fedoraproject.org/archives/list/[e-mail]/...
>

[Igor Gnatenko]

On Fri, Jul 20, 2018 at 1:44 PM Andrei Stepanov <astepano(a)redhat.com&gt; wrote:

>  On Thu, Jul 19, 2018 at 9:08 PM Igor Gnatenko <
>  ignatenkobrain(a)fedoraproject.org&gt; wrote:
>
> > I'm trying to have shared tests between all 400+ rust packages.
> >  
> > I have few problems though:
> >  
> > * There is no package NEVRA in test environment so the
> shared test
> > doesn't know what to test - it is necessary to know which test to run.
> >  
>  We trigger tests for dist-git.
>  I think about dist-git as a SRPM.
>  SRPM can produce many RPMs.
>  CI pipeline: install all RPMS from Koji build and run tests from /tests
>  for all installed RPMs.
>  The same approach is for manual testing.
>  Conclusion: test-environment-system has installed RPMS.
>  
Just want to mention that this won't work if your subpackages conflict
(e.g. curl and curl-minimal.

Tests from /tests dir check test-environment-system as a whole.
>  
> https://fedoraproject.org/wiki/CI/Standard_Test_Interface doesn't define
> NEVRA.
> Reason: STI tests are for all test-environment-system (all installed
> packages).
> /tests directory from dist-git knows that packages goes from dist-git/pkg
> /tests/tests.yml is written for each SRPM package.
> /tests/tests.yml knows that it will run against: SRPM + distro version.
> /tests/tests.yml for Fedora 27 and for Fedora 28 can are different files.
> If you have common tests, you have different /tests/tests.yml where you
> can specify exactly that this tests are for this SRPM and for Branch 28
> (for example).
>  
 This is wwhat I have done so far, however test runner has all this
information, why not to simply pass it to the tests rather than asking
every maintainer to do that manually.


> * No dist-git sources of a package - while trying to workaround previous
>> problem I have found that dist-git doesn't get into the environment so I
>> can't do something like dnf builddep
>
>  
>> Please help me ;)
>
>  
>  
>  
> We
should decide: if tests can exist independently without dist-git.
> For now, we presume they can.
> This is current state.
> If yes. There is nothing to do.
>  
Why would they do?


> If no. We should tight together dist-git branch + /tests as one entity.
> And copy /tests to test-environment system together with dist-git.
>  
>  If we decide to change this, than we need to make
correspondent changes
> to STI.
> https://fedoraproject.org/wiki/CI/Standard_Test_Interface
>  
>  
>  
>
>  
>> P.S> I'm using standard-test-basic role.
>
>  
>> Thanks in advance!
>> --
>
>  
>> -Igor Gnatenko
>
>  
> _______________________________________________
>> CI mailing list -- ci(a)lists.fedoraproject.org
>> To unsubscribe send an email to ci-leave(a)lists.fedoraproject.org
>> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
>> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>> List Archives:
>>
https://lists.fedoraproject.org/archives/list/[e-mail]/...
>
>  
> _______________________________________________
> CI mailing list -- ci(a)lists.fedoraproject.org
> To unsubscribe send an email to ci-leave(a)lists.fedoraproject.org
> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
>
https://lists.fedoraproject.org/archives/list/[e-mail]/...
>  
-- 

-Igor Gnatenko

[Andrei Stepanov]

On Fri, Jul 20, 2018 at 1:54 PM Igor Gnatenko <
ignatenkobrain(a)fedoraproject.org&gt; wrote:

>  On Fri, Jul 20, 2018 at 1:44 PM Andrei Stepanov
> <astepano(a)redhat.com&gt;
>  wrote:
>
> > On Thu, Jul 19, 2018 at 9:08 PM Igor Gnatenko <
> > ignatenkobrain(a)fedoraproject.org&gt; wrote:
> >
> >> I'm trying to have shared tests between all 400+ rust packages.
> >>
> >> I have few problems though:
> >>
> >> * There is no package NEVRA in test environment so the shared test
> >> doesn't know what to test - it is necessary to know which test to run.
> >>
> >
> > We trigger tests for dist-git.
> > I think about dist-git as a SRPM.
> > SRPM can produce many RPMs.
> > CI pipeline: install all RPMS from Koji build and run tests from /tests
> > for all installed RPMs.
> > The same approach is for manual testing.
> > Conclusion: test-environment-system has installed RPMS.
> >
>
>  Just want to mention that this won't work if your subpackages conflict
>  (e.g. curl and curl-minimal.
>  
Right, and I would ask contra@ team or other guys for clarification, how CI
pipeline does care about above moment?


>  Tests from /tests dir check test-environment-system as a whole.
> >
>
> > https://fedoraproject.org/wiki/CI/Standard_Test_Interface doesn't define
> > NEVRA.
> > Reason: STI tests are for all test-environment-system (all installed
> > packages).
> > /tests directory from dist-git knows that packages goes from dist-git/pkg
> > /tests/tests.yml is written for each SRPM package.
> > /tests/tests.yml knows that it will run against: SRPM + distro version.
> > /tests/tests.yml for Fedora 27 and for Fedora 28 can are different files.
> > If you have common tests, you have different /tests/tests.yml where you
> > can specify exactly that this tests are for this SRPM and for Branch 28
> > (for example).
> >
>
>   This is wwhat I have done so far, however test runner has all this
>  information, why not to simply pass it to the tests rather than asking
>  every maintainer to do that manually.
>
>
> > * No dist-git sources of a package - while trying to workaround previous
> >> problem I have found that dist-git doesn't get into the environment so I
> >> can't do something like dnf builddep
> >>
> >> Please help me ;)
> >>
> >
> >
> > We should decide: if tests can exist independently without dist-git.
> > For now, we presume they can.
> > This is current state.
> > If yes. There is nothing to do.
> >
>
>  Why would they do?
>
>  
If you think that:

"parent dir for /tests must be contents from dist-git branch" , then:
please file a issue at https://pagure.io/standard-test-roles/issues
That I can announce discussion about this changes to STI to our team.
Please remember, this change should be consistent for all dist-git repos.




>  If no. We should tight together dist-git branch + /tests as one
> entity.
> > And copy /tests to test-environment system together with dist-git.
> >
> >  If we decide to change this, than we need to make correspondent changes
> > to STI.
> > https://fedoraproject.org/wiki/CI/Standard_Test_Interface
> >
> >
> >
> >>
> >> P.S> I'm using standard-test-basic role.
> >>
> >> Thanks in advance!
> >> --
> >>
> >> -Igor Gnatenko
> >>
> > _______________________________________________
> >> CI mailing list -- ci(a)lists.fedoraproject.org
> >> To unsubscribe send an email to ci-leave(a)lists.fedoraproject.org
> >> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> >> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> >> List Archives:
> >>
> https://lists.fedoraproject.org/archives/list/[e-mail]/...
> >>
> > _______________________________________________
> > CI mailing list -- ci(a)lists.fedoraproject.org
> > To unsubscribe send an email to ci-leave(a)lists.fedoraproject.org
> > Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives:
> >
> https://lists.fedoraproject.org/archives/list/[e-mail]/...
> >
>  --
>
>  -Igor Gnatenko
>  _______________________________________________
>  CI mailing list -- ci(a)lists.fedoraproject.org
>  To unsubscribe send an email to ci-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
>
> https://lists.fedoraproject.org/archives/list/[e-mail]/...
>

[Johnny Bieren]

On Fri, Jul 20, 2018 at 8:05 AM, Andrei Stepanov <astepano(a)redhat.com&gt;
wrote:

>  On Fri, Jul 20, 2018 at 1:54 PM Igor Gnatenko <
>  ignatenkobrain(a)fedoraproject.org&gt; wrote:
>
> > On Fri, Jul 20, 2018 at 1:44 PM Andrei Stepanov <astepano(a)redhat.com&gt;
> > wrote:
> >
> >> On Thu, Jul 19, 2018 at 9:08 PM Igor Gnatenko <
> >> ignatenkobrain(a)fedoraproject.org&gt; wrote:
> >>
> >>> I'm trying to have shared tests between all 400+ rust packages.
> >>>
> >>> I have few problems though:
> >>>
> >>> * There is no package NEVRA in test environment so the shared test
> >>> doesn't know what to test - it is necessary to know which test to run.
> >>>
> >>
> >> We trigger tests for dist-git.
> >> I think about dist-git as a SRPM.
> >> SRPM can produce many RPMs.
> >> CI pipeline: install all RPMS from Koji build and run tests from /tests
> >> for all installed RPMs.
> >> The same approach is for manual testing.
> >> Conclusion: test-environment-system has installed RPMS.
> >>
> >
> > Just want to mention that this won't work if your subpackages conflict
> > (e.g. curl and curl-minimal.
> >
>
>  Right, and I would ask contra@ team or other guys for clarification, how
>  CI pipeline does care about above moment?
>  
I don't think there is a solution for this currently. We do not have a
static blacklist going and I, at least, am not aware of some other way to
dynamically choose which rpms from the koji task to install (right now, it
just repoqueries the repo created by all the artifact rpms and tries
installing them).

> >
> > Tests from /tests dir check test-environment-system as a whole.
> >>
> >
> >> https://fedoraproject.org/wiki/CI/Standard_Test_Interface doesn't
> >> define NEVRA.
> >> Reason: STI tests are for all test-environment-system (all installed
> >> packages).
> >> /tests directory from dist-git knows that packages goes from dist-git/pkg
> >> /tests/tests.yml is written for each SRPM package.
> >> /tests/tests.yml knows that it will run against: SRPM + distro version.
> >> /tests/tests.yml for Fedora 27 and for Fedora 28 can are different files.
> >> If you have common tests, you have different /tests/tests.yml where you
> >> can specify exactly that this tests are for this SRPM and for Branch 28
> >> (for example).
> >>
> >
> >  This is wwhat I have done so far, however test runner has all this
> > information, why not to simply pass it to the tests rather than asking
> > every maintainer to do that manually.
> >
> >
> >> * No dist-git sources of a package - while trying to workaround previous
> >>> problem I have found that dist-git doesn't get into the environment so I
> >>> can't do something like dnf builddep
> >>>
> >>> Please help me ;)
> >>>
> >>
> >>
> >> We should decide: if tests can exist independently without dist-git.
> >> For now, we presume they can.
> >> This is current state.
> >> If yes. There is nothing to do.
> >>
> >
> > Why would they do?
> >
> >
>
>  If you think that:
>
>  "parent dir for /tests must be contents from dist-git branch" , then:
>  please file a issue at https://pagure.io/standard-test-roles/issues
>  That I can announce discussion about this changes to STI to our team.
>  Please remember, this change should be consistent for all dist-git repos.
>
>
>
>
> > If no. We should tight together dist-git branch + /tests as one entity.
> >> And copy /tests to test-environment system together with dist-git.
> >>
> >>  If we decide to change this, than we need to make correspondent changes
> >> to STI.
> >> https://fedoraproject.org/wiki/CI/Standard_Test_Interface
> >>
> >>
> >>
> >>>
> >>> P.S> I'm using standard-test-basic role.
> >>>
> >>> Thanks in advance!
> >>> --
> >>>
> >>> -Igor Gnatenko
> >>>
> >> _______________________________________________
> >>> CI mailing list -- ci(a)lists.fedoraproject.org
> >>> To unsubscribe send an email to ci-leave(a)lists.fedoraproject.org
> >>> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> >>> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> >>> List Archives: https://lists.fedoraproject.org/archives/list/ci@lists.
> >>> fedoraproject.org/message/ESOHHMZ37AZIVVOPAWQNTRBXDEYKWAF6/
> >>>
> >> _______________________________________________
> >> CI mailing list -- ci(a)lists.fedoraproject.org
> >> To unsubscribe send an email to ci-leave(a)lists.fedoraproject.org
> >> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> >> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> >> List Archives: https://lists.fedoraproject.org/archives/list/ci@lists.
> >> fedoraproject.org/message/45YUVGGKW3BSFRAFPRGAZ2PKPTFHXLTP/
> >>
> > --
> >
> > -Igor Gnatenko
> > _______________________________________________
> > CI mailing list -- ci(a)lists.fedoraproject.org
> > To unsubscribe send an email to ci-leave(a)lists.fedoraproject.org
> > Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives: https://lists.fedoraproject.org/archives/list/ci@lists.
> > fedoraproject.org/message/QCO7WV2N5LHUBPN62PZMEGBJL2SKUI33/
> >
>
>  _______________________________________________
>  CI mailing list -- ci(a)lists.fedoraproject.org
>  To unsubscribe send an email to ci-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives: https://lists.fedoraproject.org/archives/list/ci@lists.
>  fedoraproject.org/message/EWUIYSJKUZ6DPX3YGJ2ZEXUZFOEFC3PU/
>
>