June 2023 - test - Fedora Mailing-Lists

Re: Thoughts welcome: interface between automated test gating and the "critical path"

by Adam Williamson

On Fri, 2022-09-02 at 08:37 +0000, Zbigniew Jędrzejewski-Szmek wrote: > > > > Now, because I glued openQA to the critpath because it was handy, there > > are two sets of consequences to a package being in critical path: > > > > 1. Tighter Bodhi requirements > > 2. openQA tests are run, and results gate the update (except Rawhide) > > > > So, one of the implicit questions here is, is it OK to keep twinning > > these two sets of consequences, or should we split them up? Splitting > > them up kinda implies answer 2) from my original mail: "Keep the > > current "critical path" concept but define a broader group > > of "gated packages" somewhere". Because we would then need some new > > concept that isn't "critical path". As I said, that's more *work* - > > it'd require us to write new code in several places[0]. Even if we > > decide it'd be nice to do this, is it nice *enough* to be worth doing > > that work? > > I'd still vote for keeping a single critpath list and using it as > "the list of packages that require extra care and testing". > > As you describe, the original meaning of critpath has shifted, but > it's because the way we do updates and QA has also shifted. Doing > gating tests for a package seems much more useful than just keeping > it longer in 'updates-testing' in hope that somebody discovers an > important regresion in the second week. Well, there's a caveat there - openQA doesn't test everything. On the whole we cover quite a lot with the set of tests that gets run on updates, but there's certainly lots of potential for there to be important bugs it misses, that a human tester might catch. So I think there is still a case for the higher karma requirements too. > > So yeah, I don't think it makes sense to do the extra work to split > the concepts. Also because we have way too many concepts and processes > in Fedora already. On the whole, though, I agree with you. I just don't trust my own opinion because it's obviously biased by what's convenient for me. :D > > If we don't think it's worth doing that work, then we're kinda stuck > > with openQA glomming onto the critpath definition to decide which > > updates to test and gate, because I don't have any other current viable > > choices for that, really. And we'd have to figure out a critpath > > definition that's as viable as possible for both purposes. > > > > > > BTW, one other thought I've had in relation to all this is that we > > could enhance the current critpath definition somewhat. Right now, it's > > built out of package groups in comps which are kinda topic-separated: > > there's a critpath-kde, a critpath-gnome, a critpath-server, and so on. > > But the generated critical path package list is a monolith: it doesn't > > distinguish between a package that's on the GNOME critpath and a > > package that's on the KDE critpath, you just get a big list of all > > critpath packages. It might be nice if we actually did distinguish > > between those - the critpath definition could keep track of which > > critpath topic(s) a package is included in, and Bodhi could display > > that information in the web UI and provide it via the API. That way > > manual testers could get a bit more info on why a package is critpath > > and what areas to test, and openQA could potentially target its test > > runs to conserve resources a bit, though this might require a bit more > > coding work on the gating stuff now I think about it. > > That sounds useful. We only need a volunteer to figure out the details > and do the work ;) I actually did a huge rewrite of the thing that generates the critpath data this week, and it probably wouldn't be tooooo much work, honestly. The most annoying bit would be the Bodhi frontend stuff, but that's because I'm bad at frontend dev in general. :P But yeah, this is definitely off in sky-castle land. I'll add it to my ever-growing list of sky-castle projects to do when I get a couple of years of spare time... -- Adam Williamson Fedora QA IRC: adamw | Twitter: adamw_ha https://www.happyassassin.net

1 month, 2 weeks

3
4
0 / 0

new criterion proposal: toolbox functionality

by Sumantro Mukherjee

Related to a recently proposed blocker [0], I propose a new release criterion which would cover that case. There is a push from the Dekstop team to have toolbox as a default app in the installer iso as well. Here it is: ~~~~~~~~~~~~ Toolbox aka Toolbx should be usable. Toolbox should be able to list, create and provide the desired containerized environment to the user without root privileges. Toolbox images should be generally functional. Footnote - "What does this cover?": Toolbox's general functionality as covered by this test cases https://fedoraproject.org/wiki/QA:Testcase_toolbox ~~~~~~~~~~~~~ I think this criterion could target F39 Beta Release Criteria or Basic Release Criteria [2]. Please tell me your thoughts, thanks! [0] https://bugzilla.redhat.com/show_bug.cgi?id=2183038 [1] https://fedoraproject.org/wiki/Basic_Release_Criteria -- //sumantro Fedora QE TRIED AND PERSONALLY TESTED, ERGO TRUSTED

3 months, 1 week

6
25
0 / 0

Self-introduction: (Ahmed Samir)

by Ahmed Samir

Hi , I'm very glad to join at Fedora QA Team. I have working a lot with Fedora and Red Hat Linux since 20 years ago. So, let's time to contribute with the project. I'm studying now python Development. So, let me introduce myself. My name is Ahmed Samir, I'm 48 years old and live at Cairo Egypt. My first contact with Linux was approximately 20 years ago and since that year I have been working only with Linux and found Fedora is the best distro So, i hope for accepted me at Fedora Project. Bye

3 months, 1 week

1
0
0 / 0

PSA: Rawhide - what to do if switch to dnf5 fails because dnf is protected

by Adam Williamson

I've heard a couple of folks report that they tried updating a Rawhide system or container and it failed with: Problem: The operation would result in removing the following protected packages: dnf The problem seems to be caused by doing the upgrade with an older dnf installed. dnf-4.15.1-1.fc39 was tagged into Rawhide on May 18th, and it dropped the protection of dnf (and yum). So there was a month or so where the 'current' dnf package you got on Rawhide update dropped the protection of dnf itself, and if you updated to that version before the dnf5-by-default update arrived this week, your upgrade to dnf5 should go fine. However, if you *didn't* update your system to dnf-4.15.1-1.fc39, you'll encounter this problem when you try to update now, because the older dnf build you have installed still considers itself protected. You have a couple of options if you're stuck in this situation: either update to 4.15.1 first (by grabbing the packages from https://koji.fedoraproject.org/koji/buildinfo?buildID=2202454 and updating them), or rename or edit the file /etc/dnf/protected.d/dnf.conf so dnf is no longer protected. After doing that, the upgrade should work correctly. It's also been reported that using the pre-switchover dnf5 to do the switchover upgrade (the one that makes dnf5 the default and removes dnf) may not work. If it doesn't, then just use dnf to run that upgrade instead. Sorry for any trouble caused by this! -- Adam Williamson (he/him/his) Fedora QA Fedora Chat: @adamwill:fedora.im | Mastodon: @adamw(a)fosstodon.org https://www.happyassassin.net

3 months, 1 week

1
0
0 / 0

Re: Rawhide update gating on openQA tests will be enabled Wednesday

by Adam Williamson

On Mon, 2023-06-19 at 18:28 +0200, Adam Williamson wrote: > Hey folks! So, as per https://pagure.io/fesco/issue/3011 , we plan to > enable gating of Rawhide updates on openQA test results on Wednesday. This is now enabled! Please let me and/or releng know if it causes any terrible consequences, and in the worst case, we'll turn it off again. I did notice while doing final checks that we still have a bit of a result display inconsistency. If you check https://bodhi.fedoraproject.org/updates/FEDORA-2023-3c2931ff23 - the bad sqlite update - you will see that Bodhi/Greenwave still think several tests are "running". They are actually cancelled. This is because they're jobs that depend on other tests that failed; when that happens, openQA cancels the dependent jobs, but does not publish any event about it in its internal event messaging system. Our system for publishing Fedora messages and reporting results to resultsdb triggers off those internal events, so because we don't get an internal openQA "job_done" or "job_cancel" message, we never send any Fedora messages or file a failed/cancelled result to ResultsDB, so Greenwave and hence Bodhi don't know the job has been cancelled. They'll just think it's 'queued' forever. I'm working on a fix for this upstream now. The issue doesn't have any particularly terrible consequences - any time we land in this situation, we'll have the record that the *parent* job failed, so the update will be gated as it should be - but it does look a bit confusing in Bodhi. -- Adam Williamson (he/him/his) Fedora QA Fedora Chat: @adamwill:fedora.im | Mastodon: @adamw(a)fosstodon.org https://www.happyassassin.net

3 months, 2 weeks

2
1
0 / 0

DNF5 testing ~ Unknown argument "--skip-broken"

by Ian Laurie

The man page for DNF5 in Rawhide shows --skip-broken to be a legal generic option, however it does not show up in a "dnf --help" and in practice its use results in this error: Unknown argument "--skip-broken" for command "dnf5". Add "--help" for more information about the arguments. Is this option being added or is this a bug? Ian -- Ian Laurie FAS: nixuser | IRC: nixuser TZ: Australia/Sydney

3 months, 2 weeks

3
5
0 / 0

2023-06-12 @ 15:00 UTC - Fedora QA Meeting

by Adam Williamson

# Fedora Quality Assurance Meeting # Date: 2023-06-12 # Time: 15:00 UTC (https://fedoraproject.org/wiki/Infrastructure/UTCHowto) # Location: #fedora-meeting on irc.libera.chat Greetings testers! It's meeting time again! If anyone has any other items for the agenda, please reply to this email and suggest them! Thanks. == Proposed Agenda Topics == 1. Previous meeting follow-up 2. Fedora 39 status 3. Rawhide update test gating 4. Toolbox criterion proposal 5. Test Day / community event status 6. Open floor -- Adam Williamson (he/him/his) Fedora QA Fedora Chat: @adamwill:fedora.im | Mastodon: @adamw(a)fosstodon.org https://www.happyassassin.net

3 months, 3 weeks

1
0
0 / 0

Plan / proposal: enable openQA update testing and potentially gating on Rawhide updates

by Adam Williamson

Hi folks! We've had openQA testing of updates for stable and branched releases, and gating based on those tests, enabled for a while now. I believe this is going quite well, and I think we addressed the issues reported when we first enabled gating - Bodhi's gating status updates work more smoothly now, and openQA respects Bodhi's "re-run tests" button so failed tests can be re-triggered. A few weeks ago, I enabled testing of Rawhide updates in the openQA lab/stg instance. This was to see how smoothly the tests run, how often we run into unexpected failures or problems, and whether the hardware resources we have are sufficient for the extra load. So far this has been going more smoothly than I anticipated, if anything. The workers seem to keep up with the test load, even though one out of three worker systems for the stg instance is currently out of commission (we're using it to investigate a bug). We do get occasional failures which seem to be related to Rawhide kernel slowness (e.g. operations timing out that usually don't otherwise time out), but on the whole, the level of false failures is (I would say) acceptably low, enough that my current regime of checking the test results daily and restarting failed ones that don't seem to indicate a real bug should be sufficient. So, I'd like to propose that we enable Rawhide update testing on the production openQA instance also. This would cause results to appear on the Automated Tests tab in Bodhi, but they would be only informational (and unless the update was gated by a CI test, or somehow otherwise configured not to be pushed automatically, updates would continue to be pushed 'stable' almost immediately on creation, regardless of the openQA results). More significantly, I'd also propose that we turn on gating on openQA results for Rawhide updates. This would mean Rawhide updates would be held from going 'stable' (and included in the next compose) until the gating openQA tests had run and passed. We may want to do this a bit after turning on the tests; perhaps Fedora 37 branch point would be a natural time to do it. Currently this would usually mean a wait from update submission to 'stable push' (which really means that the build goes into the buildroot, and will go into the next Rawhide compose when it happens) of somewhere between 45 minutes and a couple of hours. It would also mean that if Rawhide updates for inter-dependent packages are not correctly grouped, the dependent update(s) will fail testing and be gated until the update they depend on has passed testing and been pushed. The tests for the dependent update(s) would then need to be re- run, either by someone hitting the button in Bodhi or an openQA admin noticing and restarting them, before the dependent update(s) could be pushed. In the worst case, if updated packages A and B both need the other to work correctly but the updates are submitted separately, both updates may fail tests and be blocked. This could only be resolved by waiving the failures, or replacing the separate updates with an update containing both packages. All of those considerations are already true for stable and branched releases, but people are probably more used to grouping updates for stable and branched than doing it for Rawhide, and the typical flow of going from a build to an update provides more opportunity to create grouped updates for branched/stable. For Rawhide the easiest way to do it if you need to do it is to do the builds in a side tag and use Bodhi's ability to create updates from a side tag. As with branched/stable, only critical path updates would have the tests run and be gated on the results. Non-critpath updates would be unaffected. (There's a small allowlist of non-critpath packages for which the tests are also run, but they are not currently gated on the results). I think doing this could really help us keep Rawhide solid and avoid introducing major compose-breaking bugs, at minimal cost. But it's a significant change and I wanted to see what folks think. In particular, if you find the existing gating of updates for stable/branched releases to cause problems in any way, I'd love to hear about it. Thanks folks! -- Adam Williamson Fedora QA IRC: adamw | Twitter: adamw_ha https://www.happyassassin.net

3 months, 3 weeks

3
4
0 / 0

introduction

by fedtest＠post.carik.tech

hello all, I'm very glad to apply to volunteer at the Fedora Test Team. I first installed Red Hat in 1998 and have been an exclusive Linux user ever since, now my everyday linux distros are both fedora and arch (I like having options) but predominantly fedora. it's well past time I contributed something back to the project other than ad-hoc testing such as 6.3.1 recently I am retired 20 years, live in Ireland and have time to spare I'm known as ffyon or grumpy or both FAS - grumpyffyon irc.libera.chat - ffyon | grumpyffyon email - fedtest at post.carik.tech cheers

4 months

2
1
0 / 0

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

test June 2023