Fedora CI update 2020-03-11

[Aleksandra Fedorova]

Hi, all.

Here is the summary of CI-related work happening in Fedora.

If you have questions or topics to discuss you can also join Fedora CI
SIG bi-weekly meeting. Next session is today in #fedora-ci IRC channel
at 15:30 UTC

https://apps.fedoraproject.org/calendar/SIGs/#m9618

========================================================

### Dist-git tests support multipackage updates

You can define package tests in dist-git via STR format

https://docs.fedoraproject.org/en-US/ci/standard-test-roles/

Note that dist-git/STR tests are different from running %check tests
in the rpm building phase. STR tests are executed in a clean virtual
machine with proper setup of repositories for the latest Fedora
Rawhide packages. This environment is better suited for integration
tests, which need to test the installed package, not the sources of
it.

Dist-git tests are fully compatible with the dynamic sidetag approach:
if you create a dynamic sidetag for the multi-package update, test
environment will be created with the content of the entire sidetag,
not an individual package.

Status: Ready to Use
Contact: Bruno Goncalves (bgoncalv) and #fedora-ci IRC channel.

### New test: rpminspect

There is a new rpminspect test running in Fedora Rawhide gating
enabled by default for all packages in a non-blocking mode.

More details:
https://github.com/rpminspect/rpminspect

Status: Ready to Use
Contact: David Cantrell (dcantrell)

### Tests for pull-requests via Zuul

Zuul team has enabled Zuul CI engine to test Pagure pull requests.

You can opt-in to Zuul CI per package.

On every pull-request Zuul will
* run a scratch build
* run rpminspect on that build
* run dist-git test defined in STR format(if available)
* provide comment in the pull-request
* wait for you to put an manual approval on the PR
* merge the PR

* you can also get Zuul to handle merge events, so that it will
automatically build the regular koji build, after the merge.

Zuul now has support for EPEL8 branches.

More details:
https://fedoraproject.org/wiki/Zuul-based-ci

Status: Ready to use
Contact: Fabien Boucher (fbo)

### Koschei as a gating test

With sidetag gating feature enabled it is now possible to run Koschei
for each dynamic sidetag and make it a part of the gating process.

We do have Koschei deployed on Fedora infrastructure. There is
on-going work by Mikolaj Izdebski to get it integrated with the Fedora
Messaging, so that sidetags are submitted in Koschei when created.

Status: research and prototyping
Contact: Serhii Turivnyi (sturivny)

### Infra change: new Jenkins master

New Jenkins master to run generic tests and inherit Taskotron pipelines.

Our current Jenkins master, which is used for dist-git tests, was not
updated for some time and it is by design bundled to the pipeline it
runs. So adding new pipelines and separating pipelines from the
Jenkins master configuration is problematic.

The goal here is to have a Jenkins master setup which is easy to
update. It will have a set of plugins pre-installed and configured for
Fedora infrastructure endpoints, but jobs configuration will be
applied to it independently.

More details:
Current work is done on a Communishift project. Code will be available
soon at https://github.com/fedora-ci

Status: WIP
Contact: Jim Bair (jbair)

### Infra change: common repository and common format for generic tests

We are refactoring the Groovy pipeline library so it is better suited
to run generic tests.

One of the goals is that generic tests are all run in the same way,
and you don’t need to add a lot of new Groovy code to add a certain
bash script as a generic test.

We’d like people to be able to contribute new generic tests without
prior knowledge of the Jenkins internal setup.

Current focus is rpmdeplint and rpminspect pipelines.

More details:
https://github.com/fedora-ci

Status: WIP
Contact: Michal Srb (msrb)

### Infra change: ODCS composes

We are updating ODCS staging infrastructure to the latest ODCS
release. Once the Fedora instractructure freeze is over, we will also
update the ODCS production instance. This work is preparation for
possible further use of ODCS to generate composes used by Fedora CI as
well as main Fedora composes.

Status: WIP
Contact: Jan Kaluza (jkaluza)

### Infra change: Testing Farm Service

Testing Farm Team is working on open-sourcing parts of the RH internal
CI infrastructure as a service, which will be used by Fedora CI's
general tests and functional tests pipeline. The main input of the
service will be test definitions in the TMT/FMF format.

TMT documentation:
https://tmt.readthedocs.io/en/latest/
(recently added testcloud + podman provisioner)

Code is hosted at GitLab:
https://gitlab.com/testing-farm/

Status: WIP (preview May 2020, GA August 2020)
Contact: Miroslav Vadkerti (mvadkert)

### CI and Gating documentation

There is a repository of CI documentation

https://pagure.io/fedora-ci/docs/

The docs are published at

https://docs.fedoraproject.org/en-US/ci/

Then there is another repository with docs on Rawhide Gating:

https://pagure.io/cpe/rawhide-gating-docs/

And result is available at:

https://docs.fedoraproject.org/en-US/rawhide-gating/

There are some rather deep or generic items there, which are not
always suitable for newcomers and are not easy to consume.

What I think we need is a smaller scale how-to’s answering specific
questions and implementing specific use-cases, which hook the CI and
gating into the packager workflow.

If you have experience with sidetag gating or CI in Fedora and figured
out the way how _you_ work with it, please share.

You can drop me a mail or write a draft page and send a pull request
to one of the repositories. We will figure out later in which section
to land it.

Status: Needs help
Contact: Aleksandra Fedorova (bookwar)

### Testing of GitHub PRs via Packit / Testing Farm on Fedora/CentOS

Packit service makes it possible to test copr builds built from GitHub
PRs on all Fedora released (including Rawhide), CentOS 6/7 and CentOS
stream via Testing Farm. Note that the Testing Farm deployed for
Packit is different from the one we are open sourcing, and once that
is ready i will replace this one.

Documentation:
https://packit.dev/testing-farm/

Status: In production (since August 2019)
Contact: Miroslav Vadkerti (mvadkert)

========================================================

For any of those topics you can contact Fedora CI SIG at #fedora-ci IRC channel.

-- 
Aleksandra Fedorova
bookwar

[Sudhir D]

On 3/11/20 7:14 PM, Aleksandra Fedorova wrote:
>  Hi, all.
>
>  Here is the summary of CI-related work happening in Fedora.
>
>  If you have questions or topics to discuss you can also join Fedora CI
>  SIG bi-weekly meeting. Next session is today in #fedora-ci IRC channel
>  at 15:30 UTC
>
>  https://apps.fedoraproject.org/calendar/SIGs/#m9618
>
>  ========================================================
>
>  ### Dist-git tests support multipackage updates
>
>  You can define package tests in dist-git via STR format
>
>  https://docs.fedoraproject.org/en-US/ci/standard-test-roles/
>
>  Note that dist-git/STR tests are different from running %check tests
>  in the rpm building phase. STR tests are executed in a clean virtual
>  machine with proper setup of repositories for the latest Fedora
>  Rawhide packages. This environment is better suited for integration
>  tests, which need to test the installed package, not the sources of
>  it.
>
>  Dist-git tests are fully compatible with the dynamic sidetag approach:
>  if you create a dynamic sidetag for the multi-package update, test
>  environment will be created with the content of the entire sidetag,
>  not an individual package.
>
>  Status: Ready to Use
>  Contact: Bruno Goncalves (bgoncalv) and #fedora-ci IRC channel.
>
>  ### New test: rpminspect
>
>  There is a new rpminspect test running in Fedora Rawhide gating
>  enabled by default for all packages in a non-blocking mode.
>
>  More details:
>  https://github.com/rpminspect/rpminspect
>
>  Status: Ready to Use
>  Contact: David Cantrell (dcantrell) 

Thanks for sending out all this information, Aleksandra :)

I also want to call out Tim Flink (irc: tflink) from Fedora QA team who 
worked extensively on getting rpminspect to run in a way that its 
results could show up in bodhi.


>  ### Tests for pull-requests via Zuul
>
>  Zuul team has enabled Zuul CI engine to test Pagure pull requests.
>
>  You can opt-in to Zuul CI per package.
>
>  On every pull-request Zuul will
>  * run a scratch build
>  * run rpminspect on that build
>  * run dist-git test defined in STR format(if available)
>  * provide comment in the pull-request
>  * wait for you to put an manual approval on the PR
>  * merge the PR
>
>  * you can also get Zuul to handle merge events, so that it will
>  automatically build the regular koji build, after the merge.
>
>  Zuul now has support for EPEL8 branches.
>
>  More details:
>  https://fedoraproject.org/wiki/Zuul-based-ci
>
>  Status: Ready to use
>  Contact: Fabien Boucher (fbo)
>
>  ### Koschei as a gating test
>
>  With sidetag gating feature enabled it is now possible to run Koschei
>  for each dynamic sidetag and make it a part of the gating process.
>
>  We do have Koschei deployed on Fedora infrastructure. There is
>  on-going work by Mikolaj Izdebski to get it integrated with the Fedora
>  Messaging, so that sidetags are submitted in Koschei when created.
>
>  Status: research and prototyping
>  Contact: Serhii Turivnyi (sturivny)
>
>  ### Infra change: new Jenkins master
>
>  New Jenkins master to run generic tests and inherit Taskotron pipelines.
>
>  Our current Jenkins master, which is used for dist-git tests, was not
>  updated for some time and it is by design bundled to the pipeline it
>  runs. So adding new pipelines and separating pipelines from the
>  Jenkins master configuration is problematic.
>
>  The goal here is to have a Jenkins master setup which is easy to
>  update. It will have a set of plugins pre-installed and configured for
>  Fedora infrastructure endpoints, but jobs configuration will be
>  applied to it independently.
>
>  More details:
>  Current work is done on a Communishift project. Code will be available
>  soon at https://github.com/fedora-ci
>
>  Status: WIP
>  Contact: Jim Bair (jbair)
>
>  ### Infra change: common repository and common format for generic tests
>
>  We are refactoring the Groovy pipeline library so it is better suited
>  to run generic tests.
>
>  One of the goals is that generic tests are all run in the same way,
>  and you don’t need to add a lot of new Groovy code to add a certain
>  bash script as a generic test.
>
>  We’d like people to be able to contribute new generic tests without
>  prior knowledge of the Jenkins internal setup.
>
>  Current focus is rpmdeplint and rpminspect pipelines.
>
>  More details:
>  https://github.com/fedora-ci
>
>  Status: WIP
>  Contact: Michal Srb (msrb)
>
>  ### Infra change: ODCS composes
>
>  We are updating ODCS staging infrastructure to the latest ODCS
>  release. Once the Fedora instractructure freeze is over, we will also
>  update the ODCS production instance. This work is preparation for
>  possible further use of ODCS to generate composes used by Fedora CI as
>  well as main Fedora composes.
>
>  Status: WIP
>  Contact: Jan Kaluza (jkaluza)
>
>  ### Infra change: Testing Farm Service
>
>  Testing Farm Team is working on open-sourcing parts of the RH internal
>  CI infrastructure as a service, which will be used by Fedora CI's
>  general tests and functional tests pipeline. The main input of the
>  service will be test definitions in the TMT/FMF format.
>
>  TMT documentation:
>  https://tmt.readthedocs.io/en/latest/
>  (recently added testcloud + podman provisioner)
>
>  Code is hosted at GitLab:
>  https://gitlab.com/testing-farm/
>
>  Status: WIP (preview May 2020, GA August 2020)
>  Contact: Miroslav Vadkerti (mvadkert)
>
>  ### CI and Gating documentation
>
>  There is a repository of CI documentation
>
>  https://pagure.io/fedora-ci/docs/
>
>  The docs are published at
>
>  https://docs.fedoraproject.org/en-US/ci/
>
>  Then there is another repository with docs on Rawhide Gating:
>
>  https://pagure.io/cpe/rawhide-gating-docs/
>
>  And result is available at:
>
>  https://docs.fedoraproject.org/en-US/rawhide-gating/
>
>  There are some rather deep or generic items there, which are not
>  always suitable for newcomers and are not easy to consume.
>
>  What I think we need is a smaller scale how-to’s answering specific
>  questions and implementing specific use-cases, which hook the CI and
>  gating into the packager workflow.
>
>  If you have experience with sidetag gating or CI in Fedora and figured
>  out the way how _you_ work with it, please share.
>
>  You can drop me a mail or write a draft page and send a pull request
>  to one of the repositories. We will figure out later in which section
>  to land it.
>
>  Status: Needs help
>  Contact: Aleksandra Fedorova (bookwar)
>
>  ### Testing of GitHub PRs via Packit / Testing Farm on Fedora/CentOS
>
>  Packit service makes it possible to test copr builds built from GitHub
>  PRs on all Fedora released (including Rawhide), CentOS 6/7 and CentOS
>  stream via Testing Farm. Note that the Testing Farm deployed for
>  Packit is different from the one we are open sourcing, and once that
>  is ready i will replace this one.
>
>  Documentation:
>  https://packit.dev/testing-farm/
>
>  Status: In production (since August 2019)
>  Contact: Miroslav Vadkerti (mvadkert)
>
>  ========================================================
>
>  For any of those topics you can contact Fedora CI SIG at #fedora-ci IRC channel.
>

[Aleksandra Fedorova]

Hi,

On Wed, Mar 11, 2020 at 6:14 PM Sudhir D <sdharane(a)redhat.com&gt; wrote:
>  On 3/11/20 7:14 PM, Aleksandra Fedorova wrote:
>  > Hi, all.
>  >
>  > Here is the summary of CI-related work happening in Fedora.
>  >
>  > If you have questions or topics to discuss you can also join Fedora CI
>  > SIG bi-weekly meeting. Next session is today in #fedora-ci IRC channel
>  > at 15:30 UTC
>  >
>  > https://apps.fedoraproject.org/calendar/SIGs/#m9618
>  >
>  > ========================================================
>  >
>  > ### Dist-git tests support multipackage updates
>  >
>  > You can define package tests in dist-git via STR format
>  >
>  > https://docs.fedoraproject.org/en-US/ci/standard-test-roles/
>  >
>  > Note that dist-git/STR tests are different from running %check tests
>  > in the rpm building phase. STR tests are executed in a clean virtual
>  > machine with proper setup of repositories for the latest Fedora
>  > Rawhide packages. This environment is better suited for integration
>  > tests, which need to test the installed package, not the sources of
>  > it.
>  >
>  > Dist-git tests are fully compatible with the dynamic sidetag approach:
>  > if you create a dynamic sidetag for the multi-package update, test
>  > environment will be created with the content of the entire sidetag,
>  > not an individual package.
>  >
>  > Status: Ready to Use
>  > Contact: Bruno Goncalves (bgoncalv) and #fedora-ci IRC channel.
>  >
>  > ### New test: rpminspect
>  >
>  > There is a new rpminspect test running in Fedora Rawhide gating
>  > enabled by default for all packages in a non-blocking mode.
>  >
>  > More details:
>  > https://github.com/rpminspect/rpminspect
>  >
>  > Status: Ready to Use
>  > Contact: David Cantrell (dcantrell)
>
>
>  Thanks for sending out all this information, Aleksandra :)
>
>  I also want to call out Tim Flink (irc: tflink) from Fedora QA team who
>  worked extensively on getting rpminspect to run in a way that its
>  results could show up in bodhi. 
I somehow took it for granted that everyone knows Tim is there.

Thanks for clearing that out.

>  >
>  > ### Tests for pull-requests via Zuul
>  >
>  > Zuul team has enabled Zuul CI engine to test Pagure pull requests.
>  >
>  > You can opt-in to Zuul CI per package.
>  >
>  > On every pull-request Zuul will
>  > * run a scratch build
>  > * run rpminspect on that build
>  > * run dist-git test defined in STR format(if available)
>  > * provide comment in the pull-request
>  > * wait for you to put an manual approval on the PR
>  > * merge the PR
>  >
>  > * you can also get Zuul to handle merge events, so that it will
>  > automatically build the regular koji build, after the merge.
>  >
>  > Zuul now has support for EPEL8 branches.
>  >
>  > More details:
>  > https://fedoraproject.org/wiki/Zuul-based-ci
>  >
>  > Status: Ready to use
>  > Contact: Fabien Boucher (fbo)
>  >
>  > ### Koschei as a gating test
>  >
>  > With sidetag gating feature enabled it is now possible to run Koschei
>  > for each dynamic sidetag and make it a part of the gating process.
>  >
>  > We do have Koschei deployed on Fedora infrastructure. There is
>  > on-going work by Mikolaj Izdebski to get it integrated with the Fedora
>  > Messaging, so that sidetags are submitted in Koschei when created.
>  >
>  > Status: research and prototyping
>  > Contact: Serhii Turivnyi (sturivny)
>  >
>  > ### Infra change: new Jenkins master
>  >
>  > New Jenkins master to run generic tests and inherit Taskotron pipelines.
>  >
>  > Our current Jenkins master, which is used for dist-git tests, was not
>  > updated for some time and it is by design bundled to the pipeline it
>  > runs. So adding new pipelines and separating pipelines from the
>  > Jenkins master configuration is problematic.
>  >
>  > The goal here is to have a Jenkins master setup which is easy to
>  > update. It will have a set of plugins pre-installed and configured for
>  > Fedora infrastructure endpoints, but jobs configuration will be
>  > applied to it independently.
>  >
>  > More details:
>  > Current work is done on a Communishift project. Code will be available
>  > soon at https://github.com/fedora-ci
>  >
>  > Status: WIP
>  > Contact: Jim Bair (jbair)
>  >
>  > ### Infra change: common repository and common format for generic tests
>  >
>  > We are refactoring the Groovy pipeline library so it is better suited
>  > to run generic tests.
>  >
>  > One of the goals is that generic tests are all run in the same way,
>  > and you don’t need to add a lot of new Groovy code to add a certain
>  > bash script as a generic test.
>  >
>  > We’d like people to be able to contribute new generic tests without
>  > prior knowledge of the Jenkins internal setup.
>  >
>  > Current focus is rpmdeplint and rpminspect pipelines.
>  >
>  > More details:
>  > https://github.com/fedora-ci
>  >
>  > Status: WIP
>  > Contact: Michal Srb (msrb)
>  >
>  > ### Infra change: ODCS composes
>  >
>  > We are updating ODCS staging infrastructure to the latest ODCS
>  > release. Once the Fedora instractructure freeze is over, we will also
>  > update the ODCS production instance. This work is preparation for
>  > possible further use of ODCS to generate composes used by Fedora CI as
>  > well as main Fedora composes.
>  >
>  > Status: WIP
>  > Contact: Jan Kaluza (jkaluza)
>  >
>  > ### Infra change: Testing Farm Service
>  >
>  > Testing Farm Team is working on open-sourcing parts of the RH internal
>  > CI infrastructure as a service, which will be used by Fedora CI's
>  > general tests and functional tests pipeline. The main input of the
>  > service will be test definitions in the TMT/FMF format.
>  >
>  > TMT documentation:
>  > https://tmt.readthedocs.io/en/latest/
>  > (recently added testcloud + podman provisioner)
>  >
>  > Code is hosted at GitLab:
>  > https://gitlab.com/testing-farm/
>  >
>  > Status: WIP (preview May 2020, GA August 2020)
>  > Contact: Miroslav Vadkerti (mvadkert)
>  >
>  > ### CI and Gating documentation
>  >
>  > There is a repository of CI documentation
>  >
>  > https://pagure.io/fedora-ci/docs/
>  >
>  > The docs are published at
>  >
>  > https://docs.fedoraproject.org/en-US/ci/
>  >
>  > Then there is another repository with docs on Rawhide Gating:
>  >
>  > https://pagure.io/cpe/rawhide-gating-docs/
>  >
>  > And result is available at:
>  >
>  > https://docs.fedoraproject.org/en-US/rawhide-gating/
>  >
>  > There are some rather deep or generic items there, which are not
>  > always suitable for newcomers and are not easy to consume.
>  >
>  > What I think we need is a smaller scale how-to’s answering specific
>  > questions and implementing specific use-cases, which hook the CI and
>  > gating into the packager workflow.
>  >
>  > If you have experience with sidetag gating or CI in Fedora and figured
>  > out the way how _you_ work with it, please share.
>  >
>  > You can drop me a mail or write a draft page and send a pull request
>  > to one of the repositories. We will figure out later in which section
>  > to land it.
>  >
>  > Status: Needs help
>  > Contact: Aleksandra Fedorova (bookwar)
>  >
>  > ### Testing of GitHub PRs via Packit / Testing Farm on Fedora/CentOS
>  >
>  > Packit service makes it possible to test copr builds built from GitHub
>  > PRs on all Fedora released (including Rawhide), CentOS 6/7 and CentOS
>  > stream via Testing Farm. Note that the Testing Farm deployed for
>  > Packit is different from the one we are open sourcing, and once that
>  > is ready i will replace this one.
>  >
>  > Documentation:
>  > https://packit.dev/testing-farm/
>  >
>  > Status: In production (since August 2019)
>  > Contact: Miroslav Vadkerti (mvadkert)
>  >
>  > ========================================================
>  >
>  > For any of those topics you can contact Fedora CI SIG at #fedora-ci IRC channel.
>  >
>  

-- 
-- 
Aleksandra Fedorova
bookwar

[Zbigniew Jędrzejewski-Szmek]

Hi,
thanks for the update, it seems nice progress is being made.

>  ### Dist-git tests support multipackage updates
>  
>  You can define package tests in dist-git via STR format
>  
>  https://docs.fedoraproject.org/en-US/ci/standard-test-roles/ 
The docs have a "Quick Start Guide", but it describes a relatively
complex case — the bash package which shares tests with other packages
using a separate repo, which obviously adds another layer of complexity.
(The repo we are looking at has a playbook which just clones another
repo with another playbook, which then runs the playbooks from
standard-test-roles. Wow.)

Simply following the procedure as described does not work, the command
fails with "This command has to be run under the root user.". I guess
'sudo ...' might be the answer, but it's not obvious, because of the layers
of indirection. (Should ansible's "become" be used instead?)

The docs also talks about many different kinds of tests... For example
something using docker. In F32, docker doesn't even run out of the
box, so it probably shouldn't be mentioned in any kind of beginner
docs.

In the "Examples" section, there's an example for "did" package.
I copy-pasted that, and after running it (under sudo), the command
does not fail, but it seems the tests failed based on the text output.
This made me realize that the docs don't describe how to check if
the tests pass. (I hope reading the transcript is not the answer.)

There's a mention somewhere that this destructively modifies the host
machine, and using a VM is suggested. But not in the "Quick Start
Guide". I'd expect this to be prominently mentioned at the top.

After looking at the docs as they are, this all seems soooo complex.
Maybe this complexity is unavoidable, but I would love to see an
introductory howto, from beginning to the checking of the test
results and hooking up to bodhi gating, for a _simple_ case, in
particular no separate tests repo cloned over the network from a
server.

Zbyszek

[Miro Hrončok]

On 12. 03. 20 10:44, Zbigniew Jędrzejewski-Szmek wrote:
>  There's a mention somewhere that this destructively modifies the
> host
>  machine, and using a VM is suggested. But not in the "Quick Start
>  Guide". I'd expect this to be prominently mentioned at the top. 
This together with:

>  Simply following the procedure as described does not work, the
> command
>  fails with "This command has to be run under the root user.". 
Makes running he tests extremely painful locally. I've opened a ticket for this:

https://pagure.io/fedora-ci/general/issue/4

(Opened 2 years ago according to Pagure.)

I even voted against gating change at FESCo because I consider the 
(non)usability of the CI a blocker. My vote against was not enough -- gating is 
not coupled with the CI and an argument was made that enabling the gating will 
add more users of the CI and it will get better. It didn't.

-- 
Miro Hrončok
--
Phone: [phone]
IRC: mhroncok

[Aleksandra Fedorova]

Hi,

On Thu, Mar 12, 2020 at 10:46 AM Zbigniew Jędrzejewski-Szmek
<zbyszek(a)in.waw.pl&gt; wrote:
>  Hi,
>  thanks for the update, it seems nice progress is being made.
>
>  > ### Dist-git tests support multipackage updates
>  >
>  > You can define package tests in dist-git via STR format
>  >
>  > https://docs.fedoraproject.org/en-US/ci/standard-test-roles/
>
>  The docs have a "Quick Start Guide", but it describes a relatively
>  complex case — the bash package which shares tests with other packages
>  using a separate repo, which obviously adds another layer of complexity.
>  (The repo we are looking at has a playbook which just clones another
>  repo with another playbook, which then runs the playbooks from
>  standard-test-roles. Wow.)
>
>  Simply following the procedure as described does not work, the command
>  fails with "This command has to be run under the root user.". I guess
>  'sudo ...' might be the answer, but it's not obvious, because of the layers
>  of indirection. (Should ansible's "become" be used instead?)
>
>  The docs also talks about many different kinds of tests... For example
>  something using docker. In F32, docker doesn't even run out of the
>  box, so it probably shouldn't be mentioned in any kind of beginner
>  docs.
>
>  In the "Examples" section, there's an example for "did" package.
>  I copy-pasted that, and after running it (under sudo), the command
>  does not fail, but it seems the tests failed based on the text output.
>  This made me realize that the docs don't describe how to check if
>  the tests pass. (I hope reading the transcript is not the answer.)
>
>  There's a mention somewhere that this destructively modifies the host
>  machine, and using a VM is suggested. But not in the "Quick Start
>  Guide". I'd expect this to be prominently mentioned at the top.
>
>  After looking at the docs as they are, this all seems soooo complex.
>  Maybe this complexity is unavoidable, but I would love to see an
>  introductory howto, from beginning to the checking of the test
>  results and hooking up to bodhi gating, for a _simple_ case, in
>  particular no separate tests repo cloned over the network from a
>  server. 
Thanks for the feedback. I will revise a Quick Start doc and make updates to it.

And I have just pushed the doc I had in drafts for too long, please take a look

https://docs.fedoraproject.org/en-US/ci/how-to-add-dist-git-test/

Dist-git tests are actually easy, I agree we made them look
unnecessarily complex.

>  Zbyszek
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail] 
-- 
Aleksandra Fedorova
bookwar

[Zbigniew Jędrzejewski-Szmek]

On Thu, Mar 12, 2020 at 03:30:18PM +0100, Aleksandra Fedorova wrote:
>  Hi,
>  
>  On Thu, Mar 12, 2020 at 10:46 AM Zbigniew Jędrzejewski-Szmek
>  <zbyszek(a)in.waw.pl&gt; wrote:
>  >
>  > Hi,
>  > thanks for the update, it seems nice progress is being made.
>  >
>  > > ### Dist-git tests support multipackage updates
>  > >
>  > > You can define package tests in dist-git via STR format
>  > >
>  > > https://docs.fedoraproject.org/en-US/ci/standard-test-roles/
>  >
>  > The docs have a "Quick Start Guide", but it describes a relatively
>  > complex case — the bash package which shares tests with other packages
>  > using a separate repo, which obviously adds another layer of complexity.
>  > (The repo we are looking at has a playbook which just clones another
>  > repo with another playbook, which then runs the playbooks from
>  > standard-test-roles. Wow.)
>  >
>  > Simply following the procedure as described does not work, the command
>  > fails with "This command has to be run under the root user.". I guess
>  > 'sudo ...' might be the answer, but it's not obvious, because of the
> layers
>  > of indirection. (Should ansible's "become" be used instead?)
>  >
>  > The docs also talks about many different kinds of tests... For example
>  > something using docker. In F32, docker doesn't even run out of the
>  > box, so it probably shouldn't be mentioned in any kind of beginner
>  > docs.
>  >
>  > In the "Examples" section, there's an example for "did"
> package.
>  > I copy-pasted that, and after running it (under sudo), the command
>  > does not fail, but it seems the tests failed based on the text output.
>  > This made me realize that the docs don't describe how to check if
>  > the tests pass. (I hope reading the transcript is not the answer.)
>  >
>  > There's a mention somewhere that this destructively modifies the host
>  > machine, and using a VM is suggested. But not in the "Quick Start
>  > Guide". I'd expect this to be prominently mentioned at the top.
>  >
>  > After looking at the docs as they are, this all seems soooo complex.
>  > Maybe this complexity is unavoidable, but I would love to see an
>  > introductory howto, from beginning to the checking of the test
>  > results and hooking up to bodhi gating, for a _simple_ case, in
>  > particular no separate tests repo cloned over the network from a
>  > server.
>  
>  Thanks for the feedback. I will revise a Quick Start doc and make updates to it.
>  
>  And I have just pushed the doc I had in drafts for too long, please take a look
>  
>  https://docs.fedoraproject.org/en-US/ci/how-to-add-dist-git-test/ 
Thanks, that looks very promising.

>  Dist-git tests are actually easy, I agree we made them look
>  unnecessarily complex. 
Zbyszek

[Jan Pazdziora]

On Wed, Mar 11, 2020 at 02:44:14PM +0100, Aleksandra Fedorova wrote:
>  
>  ### New test: rpminspect
>  
>  There is a new rpminspect test running in Fedora Rawhide gating
>  enabled by default for all packages in a non-blocking mode.
>  
>  More details:
>  https://github.com/rpminspect/rpminspect
>  
>  Status: Ready to Use
>  Contact: David Cantrell (dcantrell) 
Hello,

I've checked random no-noarch rawhide build in Koji

	https://koji.fedoraproject.org/koji/buildinfo?buildID=1478656

found its update in Bodhi

	https://bodhi.fedoraproject.org/updates/FEDORA-2020-ccc90fa0aa

but I don't see rpminspect among its tests. Is that expected?

Incidently I see fedora-ci.koji-build.rpminspect.static-analysis listed
in my 12 days old errata

	https://bodhi.fedoraproject.org/updates/FEDORA-2020-ed976e62df

but looking at the log

	http://fedora-build-checks.apps.ci.centos.org/blue/rest/organizations/jen...

it's not quite clear with what options the check was run, or indeed,
where the run-rpminspect.sh came from.

Is there a plan to make understanding what is being run and how easier,
for example by linking the CI configs from koji and bodhi, so that
the maintainer has a clear path for finding the information?

>  ### Tests for pull-requests via Zuul
>  
>  Zuul team has enabled Zuul CI engine to test Pagure pull requests. 
This is about pull requests in the upstream projects, not about
dist-git / src.fedoraproject.org pull requests, right?

-- 
Jan Pazdziora
Product Owner, Platform Security Readiness, Red Hat

[Fabien Boucher]

Hi,

>  > ### Tests for pull-requests via Zuul
>  >
>  > Zuul team has enabled Zuul CI engine to test Pagure pull requests.
>
>  This is about pull requests in the upstream projects, not about
>  dist-git / src.fedoraproject.org pull requests, right?
>  
Both pagure.io and src.fedoraproject.org are supported.
We also provide a set of jobs for dist-git.
You will find more info in the wiki page the project:
https://fedoraproject.org/wiki/Zuul-based-ci
Here is a PR on a dist-git project tested by Zuul:
https://src.fedoraproject.org/rpms/python3/pull-request/179

Fabien

[Jan Pazdziora]

On Wed, Mar 18, 2020 at 03:26:14PM +0100, Fabien Boucher wrote:
>  
>  Both pagure.io and src.fedoraproject.org are supported.
>  We also provide a set of jobs for dist-git.
>  You will find more info in the wiki page the project:
>  https://fedoraproject.org/wiki/Zuul-based-ci
>  Here is a PR on a dist-git project tested by Zuul:
>  https://src.fedoraproject.org/rpms/python3/pull-request/179 
In dist-git, how does Zuul relate to simple-koji-ci?

The pull request in question had run a scratch build

	https://koji.fedoraproject.org/koji/taskinfo?taskID=42447845

done in koji via simple-koji-ci, but on the Zuul build page

	https://fedora.softwarefactory-project.io/zuul/buildset/58266604a5ca4f829...

I also see rpm-scratch-build with link pointing to

	https://fedora.softwarefactory-project.io/zuul/build/37899d1668eb4716bce3...

which has x86_64 files under Logs > buildset ... but thore rpms seem
to have different Build Host than those in koji.

Isn't work being duplicated here?

-- 
Jan Pazdziora
Product Owner, Platform Security Readiness, Red Hat

[Fabien Boucher]

On Wed, Mar 18, 2020 at 3:54 PM Jan Pazdziora <jpazdziora(a)redhat.com&gt; wrote:
>  On Wed, Mar 18, 2020 at 03:26:14PM +0100, Fabien Boucher wrote:
>  >
>  > Both pagure.io and src.fedoraproject.org are supported.
>  > We also provide a set of jobs for dist-git.
>  > You will find more info in the wiki page the project:
>  > https://fedoraproject.org/wiki/Zuul-based-ci
>  > Here is a PR on a dist-git project tested by Zuul:
>  > https://src.fedoraproject.org/rpms/python3/pull-request/179
>
>  In dist-git, how does Zuul relate to simple-koji-ci? 
The Zuul CI instance is run by the Software Factory project, where we
provide CI for
RDO and some other projects as you can see here
https://softwarefactory-project.io/zuul/tenants.
Zuul CI (https://zuul-ci.org/) and simple-koji-ci
(https://pagure.io/fedora-ci/simple-koji-ci)
are two different projects.

>  The pull request in question had run a scratch build
>
>          https://koji.fedoraproject.org/koji/taskinfo?taskID=42447845
>
>  done in koji via simple-koji-ci, but on the Zuul build page
>
>         
> https://fedora.softwarefactory-project.io/zuul/buildset/58266604a5ca4f829...
>
>  I also see rpm-scratch-build with link pointing to
>
>         
> https://fedora.softwarefactory-project.io/zuul/build/37899d1668eb4716bce3...
>
>  which has x86_64 files under Logs > buildset ... but thore rpms seem
>  to have different Build Host than those in koji. 
rpm-scratch-build builds on koji. The related scratch build is
https://koji.fedoraproject.org/koji/taskinfo?taskID=42447907
This link appears in the rpm-scratch-build's console log
here
https://fedora.softwarefactory-project.io/zuul/build/37899d1668eb4716bce3...
buildvm-26.phx2.fedoraproject.org is a koji builder
rpm -qpi python3-3.7.7-1.fc30.x86_64.rpm | grep 'Build Host'
Build Host  : buildvm-26.phx2.fedoraproject.org

>  Isn't work being duplicated here? 
Yes the scratch build part can be seen as duplication between Zuul and
simple-koji-ci for
this project's PR. TBH, I don't know how project maintainers can
activate/deactivate simple-koji-ci.
The Zuul CI runs a scratch build then runs some linter (rpmlint,
rpminspect) jobs + the embedded
tests.yml if exists against built rpms.

[Jan Pazdziora]

On Wed, Mar 18, 2020 at 02:03:21PM +0100, Jan Pazdziora wrote:
>  On Wed, Mar 11, 2020 at 02:44:14PM +0100, Aleksandra Fedorova wrote:
>  > 
>  > ### New test: rpminspect
>  > 
>  > There is a new rpminspect test running in Fedora Rawhide gating
>  > enabled by default for all packages in a non-blocking mode.
>  > 
>  > More details:
>  > https://github.com/rpminspect/rpminspect
>  > 
>  > Status: Ready to Use
>  > Contact: David Cantrell (dcantrell)
>  
>  Hello,
>  
>  I've checked random no-noarch rawhide build in Koji
>  
>  	https://koji.fedoraproject.org/koji/buildinfo?buildID=1478656
>  
>  found its update in Bodhi
>  
>  	https://bodhi.fedoraproject.org/updates/FEDORA-2020-ccc90fa0aa
>  
>  but I don't see rpminspect among its tests. Is that expected? 
Could someone with more insight into the CI setup please check why
rpminspect was not run?

-- 
Jan Pazdziora
Product Owner, Platform Security Readiness, Red Hat

[Tim Flink]

On Mon, 23 Mar 2020 16:27:32 +0100
Jan Pazdziora <jpazdziora(a)redhat.com&gt; wrote:

>  On Wed, Mar 18, 2020 at 02:03:21PM +0100, Jan Pazdziora wrote:
>  > On Wed, Mar 11, 2020 at 02:44:14PM +0100, Aleksandra Fedorova
>  > wrote:  
>  > > 
>  > > ### New test: rpminspect
>  > > 
>  > > There is a new rpminspect test running in Fedora Rawhide gating
>  > > enabled by default for all packages in a non-blocking mode.
>  > > 
>  > > More details:
>  > > https://github.com/rpminspect/rpminspect
>  > > 
>  > > Status: Ready to Use
>  > > Contact: David Cantrell (dcantrell)  
>  > 
>  > Hello,
>  > 
>  > I've checked random no-noarch rawhide build in Koji
>  > 
>  > 	https://koji.fedoraproject.org/koji/buildinfo?buildID=1478656
>  > 
>  > found its update in Bodhi
>  > 
>  > 	https://bodhi.fedoraproject.org/updates/FEDORA-2020-ccc90fa0aa
>  > 
>  > but I don't see rpminspect among its tests. Is that expected?  
>  
>  Could someone with more insight into the CI setup please check why
>  rpminspect was not run?
>   
Sorry, I managed to miss your previous email. The jenkins instance
running rpminspect managed to crap itself last week. I didn't think
there were still missing runs but I'll look into it and make sure that
the missing ones are re-scheduled.

Thanks for bringing this to my attention.

Tim