rpminspect Jenkins is effectively down
by Tim Flink
I figured I would start a new thread instead of continuing on with the
old one.
The Jenkins instance running rpminspect has crapped itself for the
second time in less than a week and it's currently not processing the
rpminspect runs it should be.
There are also reports of missing rpminspect runs which likely stem
from the Jenkins issues.
I'm working on it and will send updates as things are fixed and skipped
runs are re-scheduled.
Tim
4 years
sidetags in Fedora (for CI purposes)
by Michal Srb
Hello,
I am trying to understand sidetags in Fedora.
$ koji list-tags | grep side
...
f33-build-side-20525
...
The command above lists tags that look like could be sidetags, so I guess
they are already in use. I am trying to understand who creates and when?
Are they created automatically for example for each bodhi update, or how
does it work?
Thanks!
Michal
4 years
Jenkins unable to read config.xml
by Tim Flink
I'm having some fun with the Jenkins instance that's supposed to be running rpminspect and hope that someone here might have some suggestions.
The instance is hosted in the CentOS CI OpenShift cluster and the configuration is here:
https://github.com/tflink/fedora-ci-generic-checks/
At this point, every Jenkins master pod that attempts to start up gets an error like the following:
2020-03-23 21:21:31.509+0000 [id=28] INFO jenkins.InitReactorRunner$1#onAttained: Augmented all extensions
2020-03-23 21:21:31.535+0000 [id=28] SEVERE jenkins.InitReactorRunner$1#onTaskFailed: Failed Loading global config
java.lang.ArrayIndexOutOfBoundsException: Index -1 out of bounds for length 16
at com.thoughtworks.xstream.core.util.FastStack.pop(FastStack.java:42)
at com.thoughtworks.xstream.io.xml.AbstractPullReader.move(AbstractPullReader.java:125)
at com.thoughtworks.xstream.io.xml.AbstractPullReader.moveDown(AbstractPullReader.java:103)
at com.thoughtworks.xstream.io.xml.XppReader.<init>(XppReader.java:63)
at com.thoughtworks.xstream.io.xml.AbstractXppDriver.createReader(AbstractXppDriver.java:54)
at com.thoughtworks.xstream.io.xml.AbstractXppDriver.createReader(AbstractXppDriver.java:65)
at hudson.XmlFile.unmarshal(XmlFile.java:180)
Caused: java.io.IOException: Unable to read /var/lib/jenkins/config.xml
at hudson.XmlFile.unmarshal(XmlFile.java:183)
at hudson.XmlFile.unmarshal(XmlFile.java:163)
at jenkins.model.Jenkins.loadConfig(Jenkins.java:3081)
at jenkins.model.Jenkins.access$1200(Jenkins.java:315)
at jenkins.model.Jenkins$13.run(Jenkins.java:3182)
at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169)
at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:296)
at jenkins.model.Jenkins$5.runTask(Jenkins.java:1121)
at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:214)
at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
2020-03-23 21:21:31.536+0000 [id=21] SEVERE hudson.util.BootFailure#publish: Failed to initialize Jenkins
java.lang.ArrayIndexOutOfBoundsException: Index -1 out of bounds for length 16
at com.thoughtworks.xstream.core.util.FastStack.pop(FastStack.java:42)
at com.thoughtworks.xstream.io.xml.AbstractPullReader.move(AbstractPullReader.java:125)
at com.thoughtworks.xstream.io.xml.AbstractPullReader.moveDown(AbstractPullReader.java:103)
at com.thoughtworks.xstream.io.xml.XppReader.<init>(XppReader.java:63)
at com.thoughtworks.xstream.io.xml.AbstractXppDriver.createReader(AbstractXppDriver.java:54)
at com.thoughtworks.xstream.io.xml.AbstractXppDriver.createReader(AbstractXppDriver.java:65)
at hudson.XmlFile.unmarshal(XmlFile.java:180)
Caused: java.io.IOException: Unable to read /var/lib/jenkins/config.xml
at hudson.XmlFile.unmarshal(XmlFile.java:183)
at hudson.XmlFile.unmarshal(XmlFile.java:163)
at jenkins.model.Jenkins.loadConfig(Jenkins.java:3081)
at jenkins.model.Jenkins.access$1200(Jenkins.java:315)
at jenkins.model.Jenkins$13.run(Jenkins.java:3182)
at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169)
at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:296)
at jenkins.model.Jenkins$5.runTask(Jenkins.java:1121)
at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:214)
at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused: org.jvnet.hudson.reactor.ReactorException
at org.jvnet.hudson.reactor.Reactor.execute(Reactor.java:282)
at jenkins.InitReactorRunner.run(InitReactorRunner.java:48)
at jenkins.model.Jenkins.executeReactor(Jenkins.java:1154)
at jenkins.model.Jenkins.<init>(Jenkins.java:965)
at hudson.model.Hudson.<init>(Hudson.java:85)
at hudson.model.Hudson.<init>(Hudson.java:81)
at hudson.WebAppMain$3.run(WebAppMain.java:233)
Caused: hudson.util.HudsonFailedToLoad
at hudson.WebAppMain$3.run(WebAppMain.java:250)
2020-03-23 21:21:31.550+0000 [id=21] INFO jenkins.model.Jenkins#cleanUp: Stopping Jenkins
2020-03-23 21:21:31.564+0000 [id=27] WARNING hudson.model.UpdateCenter#save: Failed to save /var/lib/jenkins/hudson.model.UpdateCenter.xml
java.nio.channels.ClosedByInterruptException
at java.base/java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:199)
at java.base/sun.nio.ch.FileChannelImpl.endBlocking(FileChannelImpl.java:162)
at java.base/sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:285)
at hudson.util.FileChannelWriter.write(FileChannelWriter.java:73)
at java.base/java.io.Writer.write(Writer.java:290)
at hudson.util.AtomicFileWriter.write(AtomicFileWriter.java:162)
at java.base/java.io.Writer.write(Writer.java:249)
at hudson.XmlFile.write(XmlFile.java:191)
at hudson.model.UpdateCenter.save(UpdateCenter.java:893)
at hudson.util.PersistedList.onModified(PersistedList.java:173)
at hudson.util.PersistedList.replaceBy(PersistedList.java:85)
at hudson.model.UpdateCenter.load(UpdateCenter.java:907)
at hudson.model.UpdateCenter.init(UpdateCenter.java:2465)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at hudson.init.TaskMethodFinder.invoke(TaskMethodFinder.java:104)
at hudson.init.TaskMethodFinder$TaskImpl.run(TaskMethodFinder.java:175)
at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:296)
at jenkins.model.Jenkins$5.runTask(Jenkins.java:1121)
at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:214)
at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
2020-03-23 21:21:31.714+0000 [id=21] INFO jenkins.model.Jenkins$18#onAttained: Started termination
Has anyone seen this before? Any suggestions on things I could try? At this point, the only idea I have to fix the thing is to delete the persistent storage and rebuild the thing. If I do that, all the result links in rpminspect results would be broken and the logs from those runs lost.
Thanks,
Tim
4 years, 1 month
Fedora CI update 2020-03-11
by Aleksandra Fedorova
Hi, all.
Here is the summary of CI-related work happening in Fedora.
If you have questions or topics to discuss you can also join Fedora CI
SIG bi-weekly meeting. Next session is today in #fedora-ci IRC channel
at 15:30 UTC
https://apps.fedoraproject.org/calendar/SIGs/#m9618
========================================================
### Dist-git tests support multipackage updates
You can define package tests in dist-git via STR format
https://docs.fedoraproject.org/en-US/ci/standard-test-roles/
Note that dist-git/STR tests are different from running %check tests
in the rpm building phase. STR tests are executed in a clean virtual
machine with proper setup of repositories for the latest Fedora
Rawhide packages. This environment is better suited for integration
tests, which need to test the installed package, not the sources of
it.
Dist-git tests are fully compatible with the dynamic sidetag approach:
if you create a dynamic sidetag for the multi-package update, test
environment will be created with the content of the entire sidetag,
not an individual package.
Status: Ready to Use
Contact: Bruno Goncalves (bgoncalv) and #fedora-ci IRC channel.
### New test: rpminspect
There is a new rpminspect test running in Fedora Rawhide gating
enabled by default for all packages in a non-blocking mode.
More details:
https://github.com/rpminspect/rpminspect
Status: Ready to Use
Contact: David Cantrell (dcantrell)
### Tests for pull-requests via Zuul
Zuul team has enabled Zuul CI engine to test Pagure pull requests.
You can opt-in to Zuul CI per package.
On every pull-request Zuul will
* run a scratch build
* run rpminspect on that build
* run dist-git test defined in STR format(if available)
* provide comment in the pull-request
* wait for you to put an manual approval on the PR
* merge the PR
* you can also get Zuul to handle merge events, so that it will
automatically build the regular koji build, after the merge.
Zuul now has support for EPEL8 branches.
More details:
https://fedoraproject.org/wiki/Zuul-based-ci
Status: Ready to use
Contact: Fabien Boucher (fbo)
### Koschei as a gating test
With sidetag gating feature enabled it is now possible to run Koschei
for each dynamic sidetag and make it a part of the gating process.
We do have Koschei deployed on Fedora infrastructure. There is
on-going work by Mikolaj Izdebski to get it integrated with the Fedora
Messaging, so that sidetags are submitted in Koschei when created.
Status: research and prototyping
Contact: Serhii Turivnyi (sturivny)
### Infra change: new Jenkins master
New Jenkins master to run generic tests and inherit Taskotron pipelines.
Our current Jenkins master, which is used for dist-git tests, was not
updated for some time and it is by design bundled to the pipeline it
runs. So adding new pipelines and separating pipelines from the
Jenkins master configuration is problematic.
The goal here is to have a Jenkins master setup which is easy to
update. It will have a set of plugins pre-installed and configured for
Fedora infrastructure endpoints, but jobs configuration will be
applied to it independently.
More details:
Current work is done on a Communishift project. Code will be available
soon at https://github.com/fedora-ci
Status: WIP
Contact: Jim Bair (jbair)
### Infra change: common repository and common format for generic tests
We are refactoring the Groovy pipeline library so it is better suited
to run generic tests.
One of the goals is that generic tests are all run in the same way,
and you don’t need to add a lot of new Groovy code to add a certain
bash script as a generic test.
We’d like people to be able to contribute new generic tests without
prior knowledge of the Jenkins internal setup.
Current focus is rpmdeplint and rpminspect pipelines.
More details:
https://github.com/fedora-ci
Status: WIP
Contact: Michal Srb (msrb)
### Infra change: ODCS composes
We are updating ODCS staging infrastructure to the latest ODCS
release. Once the Fedora instractructure freeze is over, we will also
update the ODCS production instance. This work is preparation for
possible further use of ODCS to generate composes used by Fedora CI as
well as main Fedora composes.
Status: WIP
Contact: Jan Kaluza (jkaluza)
### Infra change: Testing Farm Service
Testing Farm Team is working on open-sourcing parts of the RH internal
CI infrastructure as a service, which will be used by Fedora CI's
general tests and functional tests pipeline. The main input of the
service will be test definitions in the TMT/FMF format.
TMT documentation:
https://tmt.readthedocs.io/en/latest/
(recently added testcloud + podman provisioner)
Code is hosted at GitLab:
https://gitlab.com/testing-farm/
Status: WIP (preview May 2020, GA August 2020)
Contact: Miroslav Vadkerti (mvadkert)
### CI and Gating documentation
There is a repository of CI documentation
https://pagure.io/fedora-ci/docs/
The docs are published at
https://docs.fedoraproject.org/en-US/ci/
Then there is another repository with docs on Rawhide Gating:
https://pagure.io/cpe/rawhide-gating-docs/
And result is available at:
https://docs.fedoraproject.org/en-US/rawhide-gating/
There are some rather deep or generic items there, which are not
always suitable for newcomers and are not easy to consume.
What I think we need is a smaller scale how-to’s answering specific
questions and implementing specific use-cases, which hook the CI and
gating into the packager workflow.
If you have experience with sidetag gating or CI in Fedora and figured
out the way how _you_ work with it, please share.
You can drop me a mail or write a draft page and send a pull request
to one of the repositories. We will figure out later in which section
to land it.
Status: Needs help
Contact: Aleksandra Fedorova (bookwar)
### Testing of GitHub PRs via Packit / Testing Farm on Fedora/CentOS
Packit service makes it possible to test copr builds built from GitHub
PRs on all Fedora released (including Rawhide), CentOS 6/7 and CentOS
stream via Testing Farm. Note that the Testing Farm deployed for
Packit is different from the one we are open sourcing, and once that
is ready i will replace this one.
Documentation:
https://packit.dev/testing-farm/
Status: In production (since August 2019)
Contact: Miroslav Vadkerti (mvadkert)
========================================================
For any of those topics you can contact Fedora CI SIG at #fedora-ci IRC channel.
--
Aleksandra Fedorova
bookwar
4 years, 1 month
Syncing Up on Generic Checks
by Tim Flink
I brought this up in the last ci meeting but as I'm learning more, I
think this topic could use a broader discussion to be more sure that I
and maybe others aren't misunderstanding what the broader plan is.
Below, I've written up a summary of my understanding of what's going on
around generic checks. I figure that's the fastest way to see what I'm
missing and doubles as a decent summary of current work beyond what
bookwar sent out earlier this week.
I should have been more clear on what I was planning to do so that
there aren't overlaps and I'm looking to start correcting that now.
Please let me know if I've misunderstood any of the bits below.
Thanks,
Tim
----------
rpmdeplint
----------
I know that msrb is working on a pipeline to get rpmdeplint running for
Fedora builds. That work is being tracked in taiga [1] and for the
moment, lives in github [2].
Once that prototype is complete, there will be more discussion on where
rpmdeplint will be run going forwards. I don't know if the prototype is
deployed anywhere at the moment.
[1] https://teams.fedoraproject.org/project/ci/us/71
[2] https://github.com/fedora-ci/rawhide-rpmdeplint-pipeline
------------------
pipeline libraries
------------------
msrb is also working on a new (?) pipeline library for the use of more
generic, lint-ish checks going forward [3]. I am unclear on how or if
there will be any relationship with the existing pipeline libraries or
what pipelines/checks will be using the new library going forward.
[3] https://github.com/fedora-ci/jenkins-pipeline-library
-------
Jenkins
-------
Outside the pipelines which are used for running package-specific CI
tests, I am aware of at least two Jenkins masters which are being used
for something related to CI.
rpminspect production running in the CentOS CI cluster [5]
the thing jbair is working on in Fedora communishift [no link]
msrb may have an instance for rpmdeplint
I also have a second project in the CentOS CI openshift cluster which I
was planning to use as a staging instance for the rpminspect pipeline
and the other generic checks that will go away when Taskotron goes EOL
at the end of April.
I'm unclear on if there are plans around combining all these Jenkins
masters or what those plans might be. I'm hesitant to continue work on
a staging instance if it's not going to live very long.
As far as I know, the docker images used by these various efforts are
all different to varying degrees. It sounds like there could be plans
to do some unification based on a ticket in taiga [4] but I'm not clear
on those plans
[4] https://teams.fedoraproject.org/project/ci/us/70
----------------------------------
My Previous Plans for Jenkins Work
----------------------------------
I had been planning the following work on the rpminspect setup. It
turns out that some of the work is already being done by others and I'm
unclear on the rest.
- Move rpmdeplint from Taskotron to Jenkins until testing farm is
running it so that we don't have gaps in coverage after Taskotron
goes EOL.
* AFAIK, msrb is working on this and I will not continue the work I
had been doing
- Complete the transition of the generic task setup to JCasC
* jbair is working on this for a different project now? I'm unclear
on what's going on here
- Set up a custom tag for the Jenkins master so that we have more
control over when upgrades happen
* also improve the image tagging that is currently used so that
stuff can be tested before it shows up in production
- Set up a staging instance of the generic master which is currently
running rpminspect. This would enable better testing of:
* changes to how rpminspect and other generic checks are run without
significant risk to production results
* new versions of the actual tools like rpminspect or rpmdeplint
* This would also enable better plugin upgrade testing and new
Jenkins master version testing
I say previous plans because I'm not very clear on what the larger plans
are going forward. As far as I know, all generic checks will be handled
by Testing Farm starting later this summer. To me, this means that most
of what I've outlined above is not worth working on and going beyond
"good enough for now" is not going to be worth the effort for things
which are not directly contributing to Testing Farm's efforts.
4 years, 1 month
