Syncing Up on Generic Checks

[Tim Flink]

I brought this up in the last ci meeting but as I'm learning more, I
think this topic could use a broader discussion to be more sure that I
and maybe others aren't misunderstanding what the broader plan is.

Below, I've written up a summary of my understanding of what's going on
around generic checks. I figure that's the fastest way to see what I'm
missing and doubles as a decent summary of current work beyond what
bookwar sent out earlier this week.

I should have been more clear on what I was planning to do so that
there aren't overlaps and I'm looking to start correcting that now.
Please let me know if I've misunderstood any of the bits below.

Thanks,

Tim


----------
rpmdeplint
----------

I know that msrb is working on a pipeline to get rpmdeplint running for
Fedora builds. That work is being tracked in taiga [1] and for the
moment, lives in github [2].

Once that prototype is complete, there will be more discussion on where
rpmdeplint will be run going forwards. I don't know if the prototype is
deployed anywhere at the moment.

[1] https://teams.fedoraproject.org/project/ci/us/71
[2] https://github.com/fedora-ci/rawhide-rpmdeplint-pipeline


------------------
pipeline libraries
------------------

msrb is also working on a new (?) pipeline library for the use of more
generic, lint-ish checks going forward [3]. I am unclear on how or if
there will be any relationship with the existing pipeline libraries or
what pipelines/checks will be using the new library going forward.

[3] https://github.com/fedora-ci/jenkins-pipeline-library


-------
Jenkins
-------

Outside the pipelines which are used for running package-specific CI
tests, I am aware of at least two Jenkins masters which are being used
for something related to CI.

rpminspect production running in the CentOS CI cluster [5]
the thing jbair is working on in Fedora communishift [no link]
msrb may have an instance for rpmdeplint

I also have a second project in the CentOS CI openshift cluster which I
was planning to use as a staging instance for the rpminspect pipeline
and the other generic checks that will go away when Taskotron goes EOL
at the end of April.

I'm unclear on if there are plans around combining all these Jenkins
masters or what those plans might be. I'm hesitant to continue work on
a staging instance if it's not going to live very long.

As far as I know, the docker images used by these various efforts are
all different to varying degrees. It sounds like there could be plans
to do some unification based on a ticket in taiga [4] but I'm not clear
on those plans

[4] https://teams.fedoraproject.org/project/ci/us/70

----------------------------------
My Previous Plans for Jenkins Work
----------------------------------

I had been planning the following work on the rpminspect setup. It
turns out that some of the work is already being done by others and I'm
unclear on the rest.

 - Move rpmdeplint from Taskotron to Jenkins until testing farm is
   running it so that we don't have gaps in coverage after Taskotron
   goes EOL.
   * AFAIK, msrb is working on this and I will not continue the work I
     had been doing

 - Complete the transition of the generic task setup to JCasC
   * jbair is working on this for a different project now? I'm unclear
     on what's going on here

 - Set up a custom tag for the Jenkins master so that we have more
   control over when upgrades happen
   * also improve the image tagging that is currently used so that
     stuff can be tested before it shows up in production

 - Set up a staging instance of the generic master which is currently
   running rpminspect. This would enable better testing of:
   * changes to how rpminspect and other generic checks are run without
     significant risk to production results
   * new versions of the actual tools like rpminspect or rpmdeplint
   * This would also enable better plugin upgrade testing and new
     Jenkins master version testing

I say previous plans because I'm not very clear on what the larger plans
are going forward. As far as I know, all generic checks will be handled
by Testing Farm starting later this summer. To me, this means that most
of what I've outlined above is not worth working on and going beyond
"good enough for now" is not going to be worth the effort for things
which are not directly contributing to Testing Farm's efforts.

[Michal Srb]

Hi Tim, folks...

On Thu, Mar 12, 2020 at 6:39 PM Tim Flink <tflink(a)redhat.com&gt; wrote:

>  I brought this up in the last ci meeting but as I'm learning
> more, I
>  think this topic could use a broader discussion to be more sure that I
>  and maybe others aren't misunderstanding what the broader plan is.
>
>  Below, I've written up a summary of my understanding of what's going on
>  around generic checks. I figure that's the fastest way to see what I'm
>  missing and doubles as a decent summary of current work beyond what
>  bookwar sent out earlier this week.
>
>  I should have been more clear on what I was planning to do so that
>  there aren't overlaps and I'm looking to start correcting that now.
>  Please let me know if I've misunderstood any of the bits below.
>
>  Thanks,
>
>  Tim
>
>
>  ----------
>  rpmdeplint
>  ----------
>
>  I know that msrb is working on a pipeline to get rpmdeplint running for
>  Fedora builds. That work is being tracked in taiga [1] and for the
>  moment, lives in github [2].
>
>  Once that prototype is complete, there will be more discussion on where
>  rpmdeplint will be run going forwards. I don't know if the prototype is
>  deployed anywhere at the moment.
>  
I am running experiments on a non-public Jenkins instance right now, but
the plan is to move it someplace visible once we have such a "throwaway"
instance for experiments. That could be Jim's Jenkins master.


>  [1] https://teams.fedoraproject.org/project/ci/us/71
>  [2] https://github.com/fedora-ci/rawhide-rpmdeplint-pipeline
>
>
>  ------------------
>  pipeline libraries
>  ------------------
>
>  msrb is also working on a new (?) pipeline library for the use of more
>  generic, lint-ish checks going forward [3]. I am unclear on how or if
>  there will be any relationship with the existing pipeline libraries or
>  what pipelines/checks will be using the new library going forward.
>
>  [3] https://github.com/fedora-ci/jenkins-pipeline-library
>
>  
The goal is to have a small opinionated library that would cover the needs
of (hopefully most) generic pipelines. Like for example talking to the
Testing Farm and retrieving results. Introducing new generic pipelines in
future should be relatively simple task and all those new pipelines could
then benefit from the work done in the library. The goal is not to replace
existing libraries, or to force anybody to use it :)


>  -------
>  Jenkins
>  -------
>
>  Outside the pipelines which are used for running package-specific CI
>  tests, I am aware of at least two Jenkins masters which are being used
>  for something related to CI.
>
>  rpminspect production running in the CentOS CI cluster [5]
>  the thing jbair is working on in Fedora communishift [no link]
>  msrb may have an instance for rpmdeplint
>  
I definitely don't want to maintain my own Jenkins just for rpmdeplint.


>  I also have a second project in the CentOS CI openshift cluster which
> I
>  was planning to use as a staging instance for the rpminspect pipeline
>  and the other generic checks that will go away when Taskotron goes EOL
>  at the end of April.
>
>  I'm unclear on if there are plans around combining all these Jenkins
>  masters or what those plans might be. I'm hesitant to continue work on
>  a staging instance if it's not going to live very long.
>
>  As far as I know, the docker images used by these various efforts are
>  all different to varying degrees. It sounds like there could be plans
>  to do some unification based on a ticket in taiga [4] but I'm not clear
>  on those plans
>
>  
AFAIK, no such unification plans at the moment (although worth discussing
IMO), just brainstorming where (to which container registry) to push such
images. I don't know if there is a registry in Fedora that we could use. In
the meantime, quay.io seems to be a good choice.


>  [4] https://teams.fedoraproject.org/project/ci/us/70
>
>  ----------------------------------
>  My Previous Plans for Jenkins Work
>  ----------------------------------
>
>  I had been planning the following work on the rpminspect setup. It
>  turns out that some of the work is already being done by others and I'm
>  unclear on the rest.
>
>   - Move rpmdeplint from Taskotron to Jenkins until testing farm is
>     running it so that we don't have gaps in coverage after Taskotron
>     goes EOL.
>     * AFAIK, msrb is working on this and I will not continue the work I
>       had been doing
>
>   - Complete the transition of the generic task setup to JCasC
>     * jbair is working on this for a different project now? I'm unclear
>       on what's going on here
>
>   - Set up a custom tag for the Jenkins master so that we have more
>     control over when upgrades happen
>     * also improve the image tagging that is currently used so that
>       stuff can be tested before it shows up in production
>
>   - Set up a staging instance of the generic master which is currently
>     running rpminspect. This would enable better testing of:
>     * changes to how rpminspect and other generic checks are run without
>       significant risk to production results
>     * new versions of the actual tools like rpminspect or rpmdeplint
>     * This would also enable better plugin upgrade testing and new
>       Jenkins master version testing
>
>  I say previous plans because I'm not very clear on what the larger plans
>  are going forward. As far as I know, all generic checks will be handled
>  by Testing Farm starting later this summer. To me, this means that most
>  of what I've outlined above is not worth working on and going beyond
>  "good enough for now" is not going to be worth the effort for things
>  which are not directly contributing to Testing Farm's efforts.
>  _______________________________________________
>  CI mailing list -- ci(a)lists.fedoraproject.org
>  To unsubscribe send an email to ci-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct:
>  https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
>  https://lists.fedoraproject.org/archives/list/[e-mail]
>

[Pierre-Yves Chibon]

On Fri, Mar 13, 2020 at 02:05:26PM +0100, Michal Srb wrote:
>       I also have a second project in the CentOS CI openshift cluster
> which I
>       was planning to use as a staging instance for the rpminspect pipeline
>       and the other generic checks that will go away when Taskotron goes EOL
>       at the end of April.
>  
>       I'm unclear on if there are plans around combining all these Jenkins
>       masters or what those plans might be. I'm hesitant to continue work on
>       a staging instance if it's not going to live very long.
>  
>       As far as I know, the docker images used by these various efforts are
>       all different to varying degrees. It sounds like there could be plans
>       to do some unification based on a ticket in taiga [4] but I'm not clear
>       on those plans
>  
>     AFAIK, no such unification plans at the moment (although worth discussing
>     IMO), just brainstorming where (to which container registry) to push such
>     images. I don't know if there is a registry in Fedora that we could use.
>     In the meantime, quay.io seems to be a good choice. 
Technically Fedora has one, but the idea is to deprecate it in the future, once
quay supports multi-arch and flatpaks.
If you are fine with these restrictions (no multi-arch and no flatpaks), I think
it's fine to stick to quay.


Pierre

[Pierre-Yves Chibon]

On Mon, Mar 16, 2020 at 10:24:58AM +0100, Pierre-Yves Chibon wrote:
>  On Fri, Mar 13, 2020 at 02:05:26PM +0100, Michal Srb wrote:
>  >      I also have a second project in the CentOS CI openshift cluster which I
>  >      was planning to use as a staging instance for the rpminspect pipeline
>  >      and the other generic checks that will go away when Taskotron goes EOL
>  >      at the end of April.
>  > 
>  >      I'm unclear on if there are plans around combining all these Jenkins
>  >      masters or what those plans might be. I'm hesitant to continue work on
>  >      a staging instance if it's not going to live very long.
>  > 
>  >      As far as I know, the docker images used by these various efforts are
>  >      all different to varying degrees. It sounds like there could be plans
>  >      to do some unification based on a ticket in taiga [4] but I'm not clear
>  >      on those plans
>  > 
>  >    AFAIK, no such unification plans at the moment (although worth discussing
>  >    IMO), just brainstorming where (to which container registry) to push such
>  >    images. I don't know if there is a registry in Fedora that we could use.
>  >    In the meantime, quay.io seems to be a good choice.
>  
>  Technically Fedora has one, but the idea is to deprecate it in the future, once
>  quay supports multi-arch and flatpaks.
>  If you are fine with these restrictions (no multi-arch and no flatpaks), I think
>  it's fine to stick to quay. 
Note: I'm being told that there is a Fedora org on quay and we could create you
a team under that org if you wish.


Pierre

[Jan Pazdziora]

On Thu, Mar 12, 2020 at 11:38:56AM -0600, Tim Flink wrote:
>  I brought this up in the last ci meeting but as I'm learning
> more, I
>  think this topic could use a broader discussion to be more sure that I
>  and maybe others aren't misunderstanding what the broader plan is.
>  
>  Below, I've written up a summary of my understanding of what's going on
>  around generic checks. I figure that's the fastest way to see what I'm 
Are generic checks the Automated Tests in bodhi that are run even
if the packages does not configure any tests/tests.yml? If yes,
could the Automated Test Results page show (links to) some description
of what the test is supposed to be testing, as well as links to
sources and configuration of the tests?

When I look at the Automated Test Results page of random Fedora update,
for example

	https://bodhi.fedoraproject.org/updates/FEDORA-2020-a7ca62459a

I see a bunch of dist.rpm* tests ... but clicking for example the
dist.rpmgrill.buld-log or dist.rpmgrill.setxid lines brings me in
both cases to

	https://taskotron.fedoraproject.org/artifacts/all/56cc7402-316c-11ea-8436...

where I see

      {
         "module" : "Setxid",
         "order" : 91,
         "results" : [],
         "run_time" : 0,
         "status" : "completed"
      },
      {
         "module" : "BuildLog",
         "order" : 92,
         "results" : [],
         "run_time" : 0,
         "status" : "completed"
      },

but with no indication what that test actually did and tested.

-- 
Jan Pazdziora
Product Owner, Platform Security Readiness, Red Hat