- CI - Fedora mailing-lists

Failing jobs for Rawhide (and F39?) in Fedora CI due to use of old mock configs
by Adam Williamson 13 Aug '23

13 Aug '23

I've seen several folks report that jobs (usually scratch build tests) are failing in Fedora CI, e.g. this one: https://osci-jenkins-1.ci.fedoraproject.org/job/fedora-ci/job/dist-git-buil… the failures all seem to have a common cause. You'll note these lines: DEBUG: The GPG keys listed for the "fedora" repository are already installed but they are not correct for this package. DEBUG: Check that the correct key URLs are configured for this repository.. Failing package is: curl-8.2.1-2.fc40.x86_64 DEBUG: GPG Keys are configured as: file:///usr/share/distribution-gpg-keys/fedora/RPM-GPG-KEY-fedora-39-primary, file:///usr/share/distribution-gpg-keys/fedora/RPM-GPG-KEY-fedora-39-primary, file:///usr/share/distribution-gpg-keys/fedora/RPM-GPG-KEY-fedora-38-primary DEBUG: Public key for fedora-gpg-keys-40-0.1.noarch.rpm is not installed. Failing package is: fedora-gpg-keys-40-0.1.noarch ...for several packages. If you look carefully, you'll see it's using the wrong GPG keys: it's using the keys for F38 and F39, trying to validate a package from Rawhide. Obviously that will fail. This is happening because CI is running EL 8, and the EPEL 8 mock/mock- core-configs update for F39 branching is not yet stable: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2023-a5c155a6c0 the current stable mock-core-configs for EPEL 8 still thinks Rawhide is F39, so that's why it tries to use the F39 and F38 GPG keys to validate Rawhide packages. It needs the newer version of mock-core-configs that knows Rawhide is now F40. I've tried pinging CI folks about this to see if they can pull mock in from updates-testing, but so far no response. I guess it would help if folks running EPEL 8 can test mock, confirm it works, and up-karma the update. -- Adam Williamson (he/him/his) Fedora QA Fedora Chat: @adamwill:fedora.im | Mastodon: @adamw@fosstodon.org https://www.happyassassin.net

2 1

Re: bodhi and testing farm disagrees on test result
by Adam Williamson 04 Aug '23

04 Aug '23

On Fri, 2023-08-04 at 15:40 +0200, Dan Horák wrote: > Hi, > > seems there is an issue with the test results presented in bodhi, please > see https://bodhi.fedoraproject.org/updates/FEDORA-2023-94f22746e1 > > bodhi thinks fedora-ci.koji-build.rpmdeplint.functional failed (it's > "red"), but when I click on the test in bodhi testing farm say "passed". > > What can be wrong? Well, Bodhi just gets the result from resultsdb. And the result there is definitely "failed": https://resultsdb.fedoraproject.org/results/41380875 Fedora CI files that result via ci-resultsdb-listener: https://pagure.io/ci-resultsdb-listener I think the message that triggered that result was this one: https://apps.fedoraproject.org/datagrepper/v2/id?id=0bb07f00-a232-48e9-a64f… which is actually an 'error' message, not a 'failed' message. resultsdb does - as of fairly recently - have an ERROR outcome, at least in the Fedora deployment, but it did not have one for a long time; ci- resultsdb-listener specifically uses the FAILED outcome for .error results, which we could perhaps improve now ERROR is available. The message doesn't tell us exactly what the error was, just "Infrastructure Failure". But this is basically what happened - some kind of system error occurred during/after the test run which somehow caused CI to publish a message indicating the test errored out, even though (as you saw) the logs indicate it passed. From there the details of how we report CI results to resultsdb turned it into a "failed" result. CCing the ci@ list - someone from there should be able to look more closely into what went on here and if anything can be improved (beyond making ci-resultsdb-listener use the ERROR outcome, now we have it). Practically speaking, we should be able to just re-run the tests and it should come out good. I'll bonk that button now. -- Adam Williamson (he/him/his) Fedora QA Fedora Chat: @adamwill:fedora.im | Mastodon: @adamw@fosstodon.org https://www.happyassassin.net

1 1

Asking questions about adding tests to fedora
by Ondrej Mejzlik 31 Jul '23

31 Jul '23

Hello, In the past few months we have been creating test repositories in Fedora tests git for components used in rhel. https://src.fedoraproject.org/projects/tests/%2A These repositories contain tests from RedHat which we decided can be public. With the intention of allowing public contributors to add tests of their own or review our test and then use these tests in public centOS testing. We would like to know when a maintainer of a package wants to add tests to Fedora in this way, is this ci(a)lists.fedoraproject.org the correct place to ask for help with the process or issues when something does not work? Would it be a problem for you if people ask questions about writing tests or test organization? Do we need our own separate mailing group for this purpose? Thank you. Ondrej Mejzlik

2 1

Re: Scheduled Maintenance Notice - Temporary Downtime of https://osci-jenkins-1.ci.fedoraproject.org/
by Miroslav Vadkerti 25 Jul '23

25 Jul '23

Hi, Just a heads-up there is an outage of Fedora CI from last week, sorry for the trouble. They hope to resurrect the instance until tmrw. Best regards, /M On Mon, Jul 24, 2023 at 12:33 AM Andrei Stepanov <astepano(a)redhat.com> wrote: > Hey there, > > Just a quick heads up! The https://osci-jenkins-1.ci.fedoraproject.org/ > server is currently under maintenance. We're updating the k8s EKS and other > dependent services to fresh versions. Don't worry, we'll be back up and > running by Jul 25. > During this maintenance period, the server will be temporarily > unavailable. We apologize for any inconvenience this may cause. > Thanks for your patience! > > Best regards. > -- Miroslav Vadkerti :: Senior Principal QE :: Testing Farm / Linux QE IRC mvadkert #tft #tmt #osci :: Mobile +420 773 944 252 Remote Czech Republic :: Red Hat Czech s.r.o

2 1

[release] Testing Farm 2023-05.1 with tmt-1.23
by Miroslav Vadkerti 23 Jun '23

23 Jun '23

Hi, Testing Farm release 2023-05.1 <https://docs.testing-farm.io/general/0.1/releases/2023-05.1.html> is deployed 🎉 (TFT-1871 <https://issues.redhat.com/browse/TFT-1871>). *🎮 Upgrades* - tmt <https://tmt.readthedocs.io/en/stable/> updated from 1.22 to 1.24 <https://github.com/teemtee/tmt/releases/tag/1.24.0>, see also release notes for 1.23 <https://github.com/teemtee/tmt/releases/tag/1.23.0> - Artemis upgraded to v0.0.57 <https://gitlab.com/testing-farm/artemis/-/tags/v0.0.57>, improving hardware requirements <https://docs.testing-farm.io/general/0.1/test-request.html#hardware> *🎯 Highlights* - Testing Farm CLI <https://docs.testing-farm.io/general/0.1/cli.html> reserve command available, for easy provisioning of a single machine from Testing Farm. (TFT-2073 <https://issues.redhat.com/browse/TFT-2073>) (DevConf mini-demo <https://youtu.be/nS0Z1OilOas?t=18474>) - Non-RHEL teams are onboarding to Testing Farm, see for example strizmi-kafk2a-operator <https://github.com/strimzi/strimzi-kafka-operator/blob/main/systemtest/tmt/…>. (fedoramagazine blog post <https://fedoramagazine.org/how-to-use-testing-farm-outside-of-rhel/>) - Support for customizing global timeout for a request. Default is 12 hours. (TFT-1763 <https://issues.redhat.com/browse/TFT-1763>) - Support for private GIT repositories, the username and password can be passed directly in the git url and are properly treated as secrets. ( TFT-1656 <https://issues.redhat.com/browse/TFT-1656>) - Support for tests filtering via the API request. (TFT-1993 <https://issues.redhat.com/browse/TFT-1993>) (docs <https://docs.testing-farm.io/general/0.1/test-request.html#tmt>) - Support for installing only certain packages from a repository. ( TFT-2070 <https://issues.redhat.com/browse/TFT-2070>) - Support for merge request refs for GitLab and GitHub. (TFT-1578 <https://issues.redhat.com/browse/TFT-1578>) - Metrics for average queue time available for Red Hat and Public Ranch available. (TFT-1764 <https://issues.redhat.com/browse/TFT-1764>) (link <http://metrics.osci.redhat.com/d/NnHWU1dnz/testing-farm?orgId=1&refresh=1m> ) - Kickstart support for Beaker provisioner, on par with kickstart support in the Beaker XML. (TFT-1395 <https://issues.redhat.com/browse/TFT-1395>) (thanks to Martin Litwora / sst_upgrades) - Support for different location of tmt metadata root in the git repositories. (TFT-1706 <https://issues.redhat.com/browse/TFT-1706>) - Artemis can now detect spot instances pre-maturely terminated. ( TFT-1888 <https://issues.redhat.com/browse/TFT-1888>) - TESTING_FARM_REQUEST_ID environment variable exposed to tests. ( TFT-2044 <https://issues.redhat.com/browse/TFT-2044>) (Thanks to Lukas Kotek / KVM QE) - Support for tuning Beaker watchdog timeout in Artemis. (TFT-1949 <https://issues.redhat.com/browse/TFT-1949>) (Thanks to Martin Litwora / sst_upgrades) - Testing Farm CLI run command available, to easily run small shell snippets in Testing Farm. (TFT-2019 <https://issues.redhat.com/browse/TFT-2019>) *🐞 Bugfixes* - Excluding of packages for repository and copr build artifacts properly works now. (TFT-977 <https://issues.redhat.com/browse/TFT-977>) - Compose and arch are not part of the plan name anymore in the JUnit XML. Oculus Improved to show compose and arch below the plan name. ( TFT-1810 <https://issues.redhat.com/browse/TFT-1810>) - Multiple fixes for dnf5 which landed in Fedora Rawhide. Zuul PR testing disabled due to dnf5/issues/549 <https://github.com/rpm-software-management/dnf5/issues/549>. - Non-printable characters in results-junit.xml are now properly handled (TFT-2039 <https://issues.redhat.com/browse/TFT-2039>). - Plans with non-standard characters supported. (TFT-1881 <https://issues.redhat.com/browse/TFT-1881>) - Test machine's hostname, FQDN and reverse DNS records are now properly resolvable. (TFT-1987 <https://issues.redhat.com/browse/TFT-1987>) - Improved stability of aarch64 in AWS by using non-spot instances. ( TFT-2080 <https://issues.redhat.com/browse/TFT-2080>) *🔍 Statistics* - Error rate ~ 3.69% in May <http://metrics.osci.redhat.com/d/NnHWU1dnz/testing-farm?viewPanel=12&orgId=…> - ~ 56k testing requests in May <http://metrics.osci.redhat.com/d/NnHWU1dnz/testing-farm?viewPanel=8&orgId=1…> - Average queued time ~24 second in Public <http://metrics.osci.redhat.com/d/NnHWU1dnz/testing-farm?viewPanel=26&orgId=…>, ~5 minutes in Red Hat <http://metrics.osci.redhat.com/d/NnHWU1dnz/testing-farm?viewPanel=27&orgId=…> in th last month <https://issues.redhat.com/browse/TFT-1761> <https://issues.redhat.com/browse/TFT-1761>On behalf of the Testing Farm Team, Miroslav Vadkerti -- Miroslav Vadkerti :: Senior Principal QE :: Testing Farm / Linux QE IRC mvadkert #tft #tmt #osci :: Mobile +420 773 944 252 Remote Czech Republic :: Red Hat Czech s.r.o

1 1

Fedora-Rawhide testing blocked in Testing Farm
by Miroslav Vadkerti 22 Jun '23

22 Jun '23

Hi, Fedora Rawhide testing blocked on dnf5 update, which broke Testing Farm. We are trying to hotpatch the problems to bring back testing against Rawhide. ETA 2 hours. For updates watch: https://status.testing-farm.io/issues/2023-06-21-rawhide-outage/ Best regards, /M -- Miroslav Vadkerti :: Senior Principal QE :: Testing Farm / Linux QE IRC mvadkert #tft #tmt #osci :: Mobile +420 773 944 252 Remote Czech Republic :: Red Hat Czech s.r.o

1 2

Please help me: STI tests with standard-test-source succeed without actually running
by Miro Hrončok 06 Jun '23

06 Jun '23

Hello folks. In https://gitlab.com/redhat/centos-stream/rpms/pyproject-rpm-macros/-/merge_r… I discovered a problem with the STI tests. Before adding the standard-test-source import_role, the STI test would run for approximately *20 minutes*, succeed and the testing farm artifacts page would list all the tests (green). After adding the standard-test-source thing, the STI tests run for approximately *5 minutes*, succeed and the testing farm artifacts page lists only one item with a cryptic name (such as git-refs-merge-requests-18-headaan6h3u3/tests/tests.yml). I believe the STI tests have actually not run at all and the job reported success. I've applied my change in isolation at https://src.fedoraproject.org/rpms/pyproject-rpm-macros/pull-request/400 -- it behaves the same (both on zuul's rpm-sti-test and Fedora CI - dist-git test. The added construct is copied from the python-tox package, where it worked 2 months ago, see e.g. the zuul results of https://src.fedoraproject.org/rpms/python-tox/pull-request/124 However now it misbehaves there as well, see the results of https://src.fedoraproject.org/rpms/python-tox/pull-request/126 Specifically, compare: https://artifacts.dev.testing-farm.io/b02f4e61-5517-4796-91d4-feaad434ce41/ ok https://artifacts.dev.testing-farm.io/30af69cb-eff7-4f6d-804a-d5533685f3e9/ bad What is going on here? I've been debugging this for ~2 hours and I am totally lost. The nice green SUCCESS thing now scares me because it it is meaningless :( -- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok

3 5

Fedora CI / Zuul based CI for Fedora blocked on Testing Farm outage
by Miroslav Vadkerti 18 Apr '23

18 Apr '23

Hi, We are experiencing an unexpected outage in Testing Farm: https://status.testing-farm.io/issues/2023-04-18-public-ranch-outage/ All testing will be blocked until we resolve it. Watch the status page for updates. Best regards, /M -- Miroslav Vadkerti :: Senior Principal QE :: Testing Farm / Linux QE IRC mvadkert #tft #tmt #osci :: Mobile +420 773 944 252 Remote Czech Republic :: Red Hat Czech s.r.o

1 1

Queued / running test status visible in Bodhi
by Adam Williamson 05 Apr '23

05 Apr '23

Hi folks! I already mentioned this in a devel@ thread, but Miro suggested pointing it out here too to make sure folks are aware. With Bodhi 7.1.1 deployed into production recently, queued/running tests are now shown as such in Bodhi. Previously, tests that were required for gating but which had not yet finished were shown as "missing" - the gating status referred to them as such, and they got a row on the Automated Results tab which had a red background and wasn't clickable (as we had no idea where a link should go). Now, so long as the test system reported a "result" with outcome QUEUED or RUNNING, the test will show up as such in Bodhi. The gating status summary now refers to running tests as such, and they (even non-gating ones) get a row on the Automated Results tab with a *blue* background that is clickable. (It should also have an hourglass icon, but I messed up the icon names, so that doesn't work in 7.1.1; when https://github.com/fedora-infra/bodhi/pull/5187 is merged and reaches deployment, that will be fixed). Hopefully this will make things less confusing for folks. I believe Fedora CI reports the correct "results" for all test executions. openQA reports QUEUED in *most* cases; it does not yet report RUNNING (so openQA tests will show as queued until they're finished). It doesn't report QUEUED when a failed test is auto- restarted; I'm going to fix that upstream, I just didn't get the roundtuits yet. When that happens, the gating status will be failed and the test will show as failed in Bodhi until the rerun completes. See attached screenshot for an example of how it looks right now. -- Adam Williamson (he/him/his) Fedora QA Fedora Chat: @adamwill:fedora.im | Mastodon: @adamw@fosstodon.org https://www.happyassassin.net

2 1

Software Factory 3.8 upgrade fedora.softwarefactory-project.io: 2023-02-08 and 2023-02-09 from 14:00 to 18:00 UTC
by Nicolas Hicher 09 Feb '23

09 Feb '23

Hello folks, We plan to perform an upgrade of Software Factory deployments with version 3.8. We will upgrade softwarefactory-project.io on 2023-02-08 from 14:00 to 18:00 UTC and plan to upgrade fedora.softwarefactory-project.io on 2023-02-09 from 14:00 to 18:00 UTC For zuul services are hosted on softwarefactory-project.io, fedora.softwarefactory-project.io will feature Zuul [1] and Nodepool [2] last versions Wednesday. The affected services will be: - Zuul CI not running jobs for pagure.io or src.fedoraproject.org. Regards, Nicolas, on behalf of the Software Factory Operation Team [1] https://zuul-ci.org/docs/zuul/latest/releasenotes.html#relnotes-8-1-0 [2] https://zuul-ci.org/docs/nodepool/latest/releasenotes.html#relnotes-8-1-0

1 2

2024

2023

2022

2021

2020

2019

2018

2017

CI