Re: rpminspect Jenkins is effectively down

[Tim Flink]

That took a bit longer than I had hoped but as of the time I'm sending
out this email, the Jenkins instance running rpminspect is back up and
is in the process of slowly working through its backlog.

There are some rpminspect results that were lost and aren't currently
part of the backlog. My next work item is to get all of those
re-scheduled.

Unfortunately, I still don't know what caused the failure that required
a rebuild of the Jenkins master. I've reworked a few things and put some
job throttling in place to limit the number of concurrent jobs on the
Jenkins instance. Hopefully, that will keep the master from getting
overwhelmed and going down again. I'm also going to be putting more
monitoring in place soon - thanks to the Testing Farm folks for letting
us use their prometheus instance.

Please let me know if you notice anything not working. I'll send out
another email once the lost runs have been re-scheduled.

Tim

On Mon, 23 Mar 2020 10:29:28 -0600
Tim Flink <tflink(a)redhat.com&gt; wrote:

>  I figured I would start a new thread instead of continuing on with
> the
>  old one.
>  
>  The Jenkins instance running rpminspect has crapped itself for the
>  second time in less than a week and it's currently not processing the
>  rpminspect runs it should be.
>  
>  There are also reports of missing rpminspect runs which likely stem
>  from the Jenkins issues.
>  
>  I'm working on it and will send updates as things are fixed and
>  skipped runs are re-scheduled.
>  
>  Tim