Experiments with Zuul CI

[Fabien Boucher]

Hello,

I'm writing to this list to share about a proof of concept me and my team
are setting up. The goal of the POC is to show that Zuul, OpenStack's code
gating system [1], can be used to automate RPM-related jobs for Fedora's CI.

We've managed to implement the following workflows:

* Run jobs when PRs are opened/changed on a Pagure instance. Recently Zuul
added support for Pagure events so we were able to set up some packaging
jobs on Zuul triggered by PRs on src.fedoraproject.org. There are three
jobs attached:
** The first job runs a scratch build on Koji and shares artifacts (rpms)
with two child jobs.
** The first child job runs a rpm lint.
** The second child job runs the functional tests included in the package.

Here [2] you can see a PR where those Zuul jobs ran. More details can be
found here [3].

* Run jobs when PRs are opened/changed that depend on others PRs. A
packager is able to tell Zuul that a PR depends on others PRs by using the
"Depends-On:" keyword in the PR's initial message. The artifacts from
dependent PRs are available on a PR's job's workspace as well; in [4] the
python-mock RPM from the dependent PR is used when building python-redis.

* Run jobs when a package is built on Koji. The idea is to listen to the
fedmsg queue, waiting for the topic "buildsys.build.state.change", run the
configured Zuul jobs, and finally publish the jobs' results on fedmsg. At
the moment we are able to run a linter job each time a package is built,
but we don't publish the results on fedmsg. Here is the list of the most
recent linter job results [5]. More details can be found here [6].

We started that POC because we think Zuul offers a lot of useful features
[1] like handling PR dependencies that could be of interest to the Fedora
project. The POC has shown some working workflows but now we need insights
from people who know what the needs of Fedora's CI are. Let us know your
thoughts, and we will be happy to extend the POC and onboard anybody wanting
to experiment with Zuul in that context.

Regards,
Fabien Boucher


[1] https://fedoraproject.org/wiki/Zuul-based-ci#What_is_Zuul
[2] https://src.fedoraproject.org/rpms/python-gear/pull-request/5
[3] https://fedoraproject.org/wiki/Zuul-based-ci#Pagure_PR_tests_via_Zuul
[4] https://stg.pagure.io/python-redis-distgit/pull-request/1
[5]
https://fedora.softwarefactory-project.io/zuul/t/fedora-staging/builds?jo...
[6]
https://fedoraproject.org/wiki/Zuul-based-ci#Buildsys_build_validation_vi...

[Miroslav Vadkerti]

Hi,

This is awesome! Looking forward meeting you on Flock to discuss where this
is going.

On Mon, Jul 1, 2019 at 2:53 PM Fabien Boucher <fboucher(a)redhat.com&gt; wrote:

>  Hello,
>
>  I'm writing to this list to share about a proof of concept me and my team
>  are setting up. The goal of the POC is to show that Zuul, OpenStack's
>  code gating system [1], can be used to automate RPM-related jobs for
>  Fedora's CI.
>
>  We've managed to implement the following workflows:
>
>  * Run jobs when PRs are opened/changed on a Pagure instance. Recently Zuul
>  added support for Pagure events so we were able to set up some packaging
>  jobs on Zuul triggered by PRs on src.fedoraproject.org. There are three
>  jobs attached:
>  ** The first job runs a scratch build on Koji and shares artifacts (rpms)
>  with two child jobs.
>  
Is this really build on Koji, or you build it in mock on your nodes?

https://fedora.softwarefactory-project.io/logs/5/5/616ac07a47e38483d6bbf6...



>  ** The first child job runs a rpm lint.
>  ** The second child job runs the functional tests included in the package.
>
>  Here [2] you can see a PR where those Zuul jobs ran. More details can be
>  found here [3].
>
>  * Run jobs when PRs are opened/changed that depend on others PRs. A
>  packager is able to tell Zuul that a PR depends on others PRs by using
>  the "Depends-On:" keyword in the PR's initial message. The artifacts from
>  dependent PRs are available on a PR's job's workspace as well; in [4] the
>  python-mock RPM from the dependent PR is used when building python-redis.
>  
Very cool feature! I am glad to see Zuul support for Pagure.


>  * Run jobs when a package is built on Koji. The idea is to listen to the
>  fedmsg queue, waiting for the topic "buildsys.build.state.change", run
>  the configured Zuul jobs, and finally publish the jobs' results on
>  fedmsg. At the moment we are able to run a linter job each time a package
>  is built, but we don't publish the results on fedmsg. Here is the list of the
>  most recent linter job results [5]. More details can be found here [6].
>  
Zuul-gateway seems to be quite a workaround, is the long term plan to
change to support event based triggers "directly" ?


>  We started that POC because we think Zuul offers a lot of useful
> features
>  [1] like handling PR dependencies that could be of interest to the Fedora
>  project. The POC has shown some working workflows but now we need
>  insights from people who know what the needs of Fedora's CI are. Let us
>  know your thoughts, and we will be happy to extend the POC and onboard anybody
>  wanting to experiment with Zuul in that context.
>  
My Testing Farm team would like to experiment using Zuul for our workflows.

Does Nodepool support prioritization of different drivers? We would like to
offload some of our workloads to AWS in case of spikes, but would like to
use it only if non-paid resources (Openstack, Openshift) had been exhausted.

How is the driver selected for a job? Our testing jobs describe an
environment which we would like the CI system to translate to a specific
driver, according to an predefined priority.

How hard would it be to support non-Ansible based jobs please?

Thanks,
/M



>  Regards,
>  Fabien Boucher
>
>
>  [1] https://fedoraproject.org/wiki/Zuul-based-ci#What_is_Zuul
>  [2] https://src.fedoraproject.org/rpms/python Fabien Boucher
>  -gear/pull-request/5
>  <https://src.fedoraproject.org/rpms/python-gear/pull-request/5>
>  [3] https://fedoraproject.org/wiki/Zuul-based-ci#Pagure_PR_tests_via_Zuul
>  [4] https://stg.pagure.io/python-redis-distgit/pull-request/1
>  [5]
>  https://fedora.softwarefactory-project.io/zuul/t/fedora-staging/builds?jo...
>  [6]
>  https://fedoraproject.org/wiki/Zuul-based-ci#Buildsys_build_validation_vi...
>  _______________________________________________
>  CI mailing list -- ci(a)lists.fedoraproject.org
>  To unsubscribe send an email to ci-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct:
>  https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
>  https://lists.fedoraproject.org/archives/list/[e-mail]
>  

-- 
Miroslav Vadkerti :: Principal QE :: Testing Farm / BaseOS QE / OSCI
IRC mvadkert #tft #osci #qe :: GPG 0x7B5B2E95
TPB-C 2C215 :: Mobile [phone]
Red Hat Czech s.r.o, Purkyňova 115, 612 00, Brno, CR

[Fabien Boucher]

Hi,

On Tue, Jul 2, 2019 at 1:21 AM Miroslav Vadkerti <mvadkert(a)redhat.com&gt;
wrote:

>  Hi,
>
>  This is awesome! Looking forward meeting you on Flock to discuss where
>  this is going.
>
>  
Sure.


>  On Mon, Jul 1, 2019 at 2:53 PM Fabien Boucher
> <fboucher(a)redhat.com&gt; wrote:
>
> > Hello,
> >
> > I'm writing to this list to share about a proof of concept me and my team
> > are setting up. The goal of the POC is to show that Zuul, OpenStack's
> > code gating system [1], can be used to automate RPM-related jobs for
> > Fedora's CI.
> >
> > We've managed to implement the following workflows:
> >
> > * Run jobs when PRs are opened/changed on a Pagure instance. Recently
> > Zuul added support for Pagure events so we were able to set up some
> > packaging jobs on Zuul triggered by PRs on src.fedoraproject.org. There
> > are three jobs attached:
> > ** The first job runs a scratch build on Koji and shares artifacts
> > (rpms) with two child jobs.
> >
>
>  Is this really build on Koji, or you build it in mock on your nodes?
>
>
>
> https://fedora.softwarefactory-project.io/logs/5/5/616ac07a47e38483d6bbf6...
>
>
>  

The SRPM is built on the job's node in a mock then the built SRPM is passed
to koji. See below:
https://fedora.softwarefactory-project.io/logs/5/5/616ac07a47e38483d6bbf6...

The ARA report is great to get an overview of the job flow:
https://fedora.softwarefactory-project.io/logs/5/5/616ac07a47e38483d6bbf6...



>  ** The first child job runs a rpm lint.
> > ** The second child job runs the functional tests included in the package
> > .
> >
> > Here [2] you can see a PR where those Zuul jobs ran. More details can be
> > found here [3].
> >
> > * Run jobs when PRs are opened/changed that depend on others PRs. A
> > packager is able to tell Zuul that a PR depends on others PRs by using
> > the "Depends-On:" keyword in the PR's initial message. The artifacts
> > from dependent PRs are available on a PR's job's workspace as well; in [4]
> > the python-mock RPM from the dependent PR is used when building
> > python-redis.
> >
>
>  Very cool feature! I am glad to see Zuul support for Pagure.
>
>
> >
> > * Run jobs when a package is built on Koji. The idea is to listen to the
> > fedmsg queue, waiting for the topic "buildsys.build.state.change", run
> > the configured Zuul jobs, and finally publish the jobs' results on
> > fedmsg. At the moment we are able to run a linter job each time a
> > package is built, but we don't publish the results on fedmsg. Here is
> > the list of the most recent linter job results [5]. More details can be
> > found here [6].
> >
>
>  Zuul-gateway seems to be quite a workaround, is the long term plan to
>  change to support event based triggers "directly" ?
>  

Yes it is a workaround. We have started a discussion about an AMQP trigger
with Zuul's maintainers
http://lists.zuul-ci.org/pipermail/zuul-discuss/2019-May/000929.html. But
no real plan for the moment.



> > We started that POC because we think Zuul offers a lot of useful features
> > [1] like handling PR dependencies that could be of interest to the
> > Fedora project. The POC has shown some working workflows but now we need
> > insights from people who know what the needs of Fedora's CI are. Let us
> > know your thoughts, and we will be happy to extend the POC and onboard anybody
> > wanting to experiment with Zuul in that context.
> >
>
>  My Testing Farm team would like to experiment using Zuul for our workflows.
>
>  
Sure, let's us know how we could help.



>  Does Nodepool support prioritization of different drivers? We would
> like
>  to offload some of our workloads to AWS in case of spikes, but would like
>  to use it only if non-paid resources (Openstack, Openshift) had been
>  exhausted.
>
>  
AFAIK not out of the box. But with an external process you might be able to
detect your are out of quota on the OpenStack provider and push in the
Nodepool providers configuration the setting AWS:max-server value from 0 to
something else. Then Nodepool will start to use the AWS provider. When
spikes stop you revert the max-server setting to 0 then Nodepool will stop
spawning nodes on AWS.



>  How is the driver selected for a job? Our testing jobs describe an
>  environment which we would like the CI system to translate to a specific
>  driver, according to an predefined priority.
>  

A job definition in Zuul needs to specify a nodeset. A nodeset is a
collection of nodes for the job's Ansible inventory. Each node must match a
Nodepool node's label. In other words in Nodepool you define the nodes
details (provider type (openstack/aws/...), base image, VM flavor) and set
labels to nodes. In Zuul, via a nodeset, you define the available nodes
(using the Nodepool labels) for the job's inventory. For instance you could
have the same label accross multiple providers. However there is not any
priority system in that mechanics.
https://zuul-ci.org/docs/zuul/user/config.html#nodeset



>  How hard would it be to support non-Ansible based jobs please?
>
>  
Difficult to respond w/o more details, Zuul only understand a pre-defined
job format https://zuul-ci.org/docs/zuul/user/config.html#job so a
migration tool would need to be written.



>  Thanks,
>  /M
>
>
>
> >
> > Regards,
> > Fabien Boucher
> >
> >
> > [1] https://fedoraproject.org/wiki/Zuul-based-ci#What_is_Zuul
> > [2] https://src.fedoraproject.org/rpms/python Fabien Boucher
> > -gear/pull-request/5
> > <https://src.fedoraproject.org/rpms/python-gear/pull-request/5>
> > [3] https://fedoraproject.org/wiki/Zuul-based-ci#Pagure_PR_tests_via_Zuul
> > [4] https://stg.pagure.io/python-redis-distgit/pull-request/1
> > [5]
> >
> https://fedora.softwarefactory-project.io/zuul/t/fedora-staging/builds?jo...
> > [6]
> >
> https://fedoraproject.org/wiki/Zuul-based-ci#Buildsys_build_validation_vi...
> > _______________________________________________
> > CI mailing list -- ci(a)lists.fedoraproject.org
> > To unsubscribe send an email to ci-leave(a)lists.fedoraproject.org
> > Fedora Code of Conduct:
> > https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives:
> > https://lists.fedoraproject.org/archives/list/[e-mail]
> >
>
>
>  --
>  Miroslav Vadkerti :: Principal QE :: Testing Farm / BaseOS QE / OSCI
>  IRC mvadkert #tft #osci #qe :: GPG 0x7B5B2E95
>  TPB-C 2C215 :: Mobile [phone]
>  Red Hat Czech s.r.o, Purkyňova 115, 612 00, Brno, CR
>
>
>  _______________________________________________
>  CI mailing list -- ci(a)lists.fedoraproject.org
>  To unsubscribe send an email to ci-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct:
>  https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
>  https://lists.fedoraproject.org/archives/list/[e-mail]
>

[Fabien Boucher]

Hi,

On Tue, Jul 2, 2019 at 1:21 AM Miroslav Vadkerti <mvadkert(a)redhat.com&gt;
wrote:

>  Hi,
>
>  This is awesome! Looking forward meeting you on Flock to discuss where
>  this is going.
>
>  
Sure.


>  * Run jobs when PRs are opened/changed on a Pagure instance. Recently Zuul
>  added support for Pagure events so we were able to set up some packaging
>  jobs on Zuul triggered by PRs on src.fedoraproject.org. There are three
>  jobs attached:
>  ** The first job runs a scratch build on Koji and shares artifacts (rpms)
>  with two child jobs.
>
>  Is this really build on Koji, or you build it in mock on your nodes?
>
>
>
> https://fedora.softwarefactory-project.io/logs/5/5/616ac07a47e38483d6bbf6...
>
>  
The SRPM is built on the job's node in a mock then the built SRPM is passed
to koji. See below:
https://fedora.softwarefactory-project.io/logs/5/5/616ac07a47e38483d6bbf6...

The ARA report is great to get an overview of the job flow:
https://fedora.softwarefactory-project.io/logs/5/5/616ac07a47e38483d6bbf6...



>  Zuul-gateway seems to be quite a workaround, is the long term plan to
>  change to support event based triggers "directly" ?
>
>  
Yes it is a workaround. We have started a discussion about an AMQP trigger
with Zuul's maintainers
http://lists.zuul-ci.org/pipermail/zuul-discuss/2019-May/000929.html. But
no real plan for the moment.


>  My Testing Farm team would like to experiment using Zuul for our
> workflows.
>
>  
Sure, let's us know how we could help.


>  Does Nodepool support prioritization of different drivers? We would
> like
>  to offload some of our workloads to AWS in case of spikes, but would like
>  to use it only if non-paid resources (Openstack, Openshift) had been
>  exhausted.
>  

AFAIK not out of the box. But with an external process you might be able to
detect your are out of quota on the OpenStack provider and push in the
Nodepool providers configuration the setting AWS:max-server value from 0 to
something else. Then Nodepool will start to use the AWS provider. When
spikes stop you revert the max-server setting to 0 then Nodepool will stop
spawning nodes on AWS.



>  How is the driver selected for a job? Our testing jobs describe an
>  environment which we would like the CI system to translate to a specific
>  driver, according to an predefined priority.
>
>  
A job definition in Zuul needs to specify a nodeset. A nodeset is a
collection of nodes for the job's Ansible inventory. Each node must match a
Nodepool node's label. In other words in Nodepool you define the nodes
details (provider type (openstack/aws/...), base image, VM flavor) and set
labels to nodes. In Zuul, via a nodeset, you define the available nodes
(using the Nodepool labels) for the job's inventory. For instance you could
have the same label accross multiple providers. However there is not any
priority system in that mechanics.
https://zuul-ci.org/docs/zuul/user/config.html#nodeset



>  How hard would it be to support non-Ansible based jobs please?
>
>  
Difficult to respond w/o more details, Zuul only understand a pre-defined
job format https://zuul-ci.org/docs/zuul/user/config.html#job so a
migration tool would need to be written.

[Miroslav Vadkerti]

Hi,

On Tue, Jul 2, 2019 at 2:26 PM Fabien Boucher <fboucher(a)redhat.com&gt; wrote:

>  Hi,
>
>  On Tue, Jul 2, 2019 at 1:21 AM Miroslav Vadkerti <mvadkert(a)redhat.com&gt;
>  wrote:
>
> > Hi,
> >
> > This is awesome! Looking forward meeting you on Flock to discuss where
> > this is going.
> >
> >
>
>  Sure.
>
>
> >
> >
> > * Run jobs when PRs are opened/changed on a Pagure instance. Recently
> > Zuul added support for Pagure events so we were able to set up some
> > packaging jobs on Zuul triggered by PRs on src.fedoraproject.org. There
> > are three jobs attached:
> > ** The first job runs a scratch build on Koji and shares artifacts
> > (rpms) with two child jobs.
> >
> > Is this really build on Koji, or you build it in mock on your nodes?
> >
> >
> >
> https://fedora.softwarefactory-project.io/logs/5/5/616ac07a47e38483d6bbf6...
> >
>
> >
> >
>
>  The SRPM is built on the job's node in a mock then the built SRPM is
>  passed to koji. See below:
>
>
> https://fedora.softwarefactory-project.io/logs/5/5/616ac07a47e38483d6bbf6...
>
>  The ARA report is great to get an overview of the job flow:
>
>
> https://fedora.softwarefactory-project.io/logs/5/5/616ac07a47e38483d6bbf6...
>  
Oh right :) thanks! make sense ...

> >
> > Zuul-gateway seems to be quite a workaround, is the long term plan to
> > change to support event based triggers "directly" ?
> >
> >
>
>  Yes it is a workaround. We have started a discussion about an AMQP trigger
>  with Zuul's maintainers
>  http://lists.zuul-ci.org/pipermail/zuul-discuss/2019-May/000929.html. But
>  no real plan for the moment.
>  
Thanks for the pointers ...


> > My Testing Farm team would like to experiment using Zuul for our
> > workflows.
> >
> >
>  Sure, let's us know how we could help.
>  
Thanks, will leave that for discussion for Flock.



> > Does Nodepool support prioritization of different drivers? We would like
> > to offload some of our workloads to AWS in case of spikes, but would like
> > to use it only if non-paid resources (Openstack, Openshift) had been
> > exhausted.
> >
>
>
>  AFAIK not out of the box. But with an external process you might be able
>  to detect your are out of quota on the OpenStack provider and push in the
>  Nodepool providers configuration the setting AWS:max-server value from 0 to
>  something else. Then Nodepool will start to use the AWS provider. When
>  spikes stop you revert the max-server setting to 0 then Nodepool will stop
>  spawning nodes on AWS.
>  
Agreed, thanks for the pointers!


>  How is the driver selected for a job? Our testing jobs describe an
> > environment which we would like the CI system to translate to a specific
> > driver, according to an predefined priority.
> >
>
>  A job definition in Zuul needs to specify a nodeset. A nodeset is a
>  collection of nodes for the job's Ansible inventory. Each node must match a
>  Nodepool node's label. In other words in Nodepool you define the nodes
>  details (provider type (openstack/aws/...), base image, VM flavor) and set
>  labels to nodes. In Zuul, via a nodeset, you define the available nodes
>  (using the Nodepool labels) for the job's inventory. For instance you could
>  have the same label accross multiple providers. However there is not any
>  priority system in that mechanics.
>  https://zuul-ci.org/docs/zuul/user/config.html#nodeset
>  
Thanks for the explanation!


> > How hard would it be to support non-Ansible based jobs please?
> >
>
>  Difficult to respond w/o more details, Zuul only understand a pre-defined
>  job format https://zuul-ci.org/docs/zuul/user/config.html#job so a
>  migration tool would need to be written.
>  
Agreed, James replied to what I wanted to know.


Best regards,
/M



>  _______________________________________________
>  CI mailing list -- ci(a)lists.fedoraproject.org
>  To unsubscribe send an email to ci-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct:
>  https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
>  https://lists.fedoraproject.org/archives/list/[e-mail]
>  

-- 
Miroslav Vadkerti :: Principal QE :: Testing Farm / OSCI / BaseOS QE
IRC mvadkert #tft #baseosci #osci :: GPG 0x7B5B2E95
TPB-C 2C215 :: Mobile [phone]
Red Hat Czech s.r.o, Purkyňova 115, 612 00, Brno, CR

[James E. Blair]

>  How hard would it be to support non-Ansible based jobs please?
It is simple to tell Ansible to run something else.  For instance, in
some systems I am responsible for, we use Ansible to orchestrate Puppet
runs and simple shell scripts.  The puppet/shell bits do all the work,
Ansible just causes them to execute on remote hosts.

In fact, there is already a job in Zuul's standard library to run a test
command (like "run-tests.sh"), so you don't even have to write the
Ansible to do that:

  https://zuul-ci.org/docs/zuul-jobs/general-jobs.html#job-run-test-command

We chose Ansible to be the execution engine that Zuul uses. because we
didn't want to invent yet another remote orchestration system.  Zuul
supports running a single job in complex multi-node simulated
environments, and Ansible is an excellent multi-node orchestration
execution engine.  But it's also easy to escape if your workload doesn't
benefit from it.

-Jim

[Miroslav Vadkerti]

Hi,

On Tue, Jul 2, 2019 at 5:01 PM James E. Blair <corvus(a)inaugust.com&gt; wrote:

>  > How hard would it be to support non-Ansible based jobs please?
>
>  It is simple to tell Ansible to run something else.  For instance, in
>  some systems I am responsible for, we use Ansible to orchestrate Puppet
>  runs and simple shell scripts.  The puppet/shell bits do all the work,
>  Ansible just causes them to execute on remote hosts. 

>  In fact, there is already a job in Zuul's standard library to run
> a test
>  command (like "run-tests.sh"), so you don't even have to write the
>  Ansible to do that:
>
>    https://zuul-ci.org/docs/zuul-jobs/general-jobs.html#j
>  <https://zuul-ci.org/docs/zuul-jobs/general-jobs.html#job-run-test-command...
>  ob-run-test-command
>  <https://zuul-ci.org/docs/zuul-jobs/general-jobs.html#job-run-test-command... 

>  We chose Ansible to be the execution engine that Zuul uses. because
> we
>  didn't want to invent yet another remote orchestration system.  Zuul
>  supports running a single job in complex multi-node simulated
>  environments, and Ansible is an excellent multi-node orchestration
>  execution engine.  But it's also easy to escape if your workload doesn't
>  benefit from it.
>  
Thanks for the explanation James!

Best regards,
/M



>  -Jim
>  _______________________________________________
>  CI mailing list -- ci(a)lists.fedoraproject.org
>  To unsubscribe send an email to ci-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct:
>  https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
>  https://lists.fedoraproject.org/archives/list/[e-mail]
>  

-- 
Miroslav Vadkerti :: Principal QE :: Testing Farm / OSCI / BaseOS QE
IRC mvadkert #tft #baseosci #osci :: GPG 0x7B5B2E95
TPB-C 2C215 :: Mobile [phone]
Red Hat Czech s.r.o, Purkyňova 115, 612 00, Brno, CR