reversible dual-boot test station
by Chris Murphy
Hi,
I mentioned this in a QA meeting, and have given it enough testing that I think it's broadly usable. If desired it can be copied out of my user account and put up somewhere where QA folks will see it and can modify it as issues or improvements are discovered.
What is it? The idea is to produce a system that can confidently be used for baremetal testing, without risking the primary operating system. While VM's are a great way to test, it's also a really idealized environment that tends to not expose an assortment of bugs that affect particular hardware. But then quite a lot of folks reasonably don't want to upgrade their daily use hardware early on, because they don't want to always have to debug things, or have to figure out how to undo the upgrade if it really goes badly.
Therefore, I present a dual-boot setup offering:
* no re-partitioning;
* no installation step, instead system upgrade is used;
* reversibility, or undoability, i.e. with just a few steps you can delete the "test OS".
https://fedoraproject.org/wiki/User:Chrismurphy/Draft/dualboot_teststation
--
Chris Murphy
3 months
Plan / proposal: enable openQA update testing and potentially
gating on Rawhide updates
by Adam Williamson
Hi folks!
We've had openQA testing of updates for stable and branched releases,
and gating based on those tests, enabled for a while now. I believe
this is going quite well, and I think we addressed the issues reported
when we first enabled gating - Bodhi's gating status updates work more
smoothly now, and openQA respects Bodhi's "re-run tests" button so
failed tests can be re-triggered.
A few weeks ago, I enabled testing of Rawhide updates in the openQA
lab/stg instance. This was to see how smoothly the tests run, how often
we run into unexpected failures or problems, and whether the hardware
resources we have are sufficient for the extra load.
So far this has been going more smoothly than I anticipated, if
anything. The workers seem to keep up with the test load, even though
one out of three worker systems for the stg instance is currently out
of commission (we're using it to investigate a bug). We do get
occasional failures which seem to be related to Rawhide kernel slowness
(e.g. operations timing out that usually don't otherwise time out), but
on the whole, the level of false failures is (I would say) acceptably
low, enough that my current regime of checking the test results daily
and restarting failed ones that don't seem to indicate a real bug
should be sufficient.
So, I'd like to propose that we enable Rawhide update testing on the
production openQA instance also. This would cause results to appear on
the Automated Tests tab in Bodhi, but they would be only informational
(and unless the update was gated by a CI test, or somehow otherwise
configured not to be pushed automatically, updates would continue to be
pushed 'stable' almost immediately on creation, regardless of the
openQA results).
More significantly, I'd also propose that we turn on gating on openQA
results for Rawhide updates. This would mean Rawhide updates would be
held from going 'stable' (and included in the next compose) until the
gating openQA tests had run and passed. We may want to do this a bit
after turning on the tests; perhaps Fedora 37 branch point would be a
natural time to do it.
Currently this would usually mean a wait from update submission to
'stable push' (which really means that the build goes into the
buildroot, and will go into the next Rawhide compose when it happens)
of somewhere between 45 minutes and a couple of hours. It would also
mean that if Rawhide updates for inter-dependent packages are not
correctly grouped, the dependent update(s) will fail testing and be
gated until the update they depend on has passed testing and been
pushed. The tests for the dependent update(s) would then need to be re-
run, either by someone hitting the button in Bodhi or an openQA admin
noticing and restarting them, before the dependent update(s) could be
pushed.
In the worst case, if updated packages A and B both need the other to
work correctly but the updates are submitted separately, both updates
may fail tests and be blocked. This could only be resolved by waiving
the failures, or replacing the separate updates with an update
containing both packages.
All of those considerations are already true for stable and branched
releases, but people are probably more used to grouping updates for
stable and branched than doing it for Rawhide, and the typical flow of
going from a build to an update provides more opportunity to create
grouped updates for branched/stable. For Rawhide the easiest way to do
it if you need to do it is to do the builds in a side tag and use
Bodhi's ability to create updates from a side tag.
As with branched/stable, only critical path updates would have the
tests run and be gated on the results. Non-critpath updates would be
unaffected. (There's a small allowlist of non-critpath packages for
which the tests are also run, but they are not currently gated on the
results).
I think doing this could really help us keep Rawhide solid and avoid
introducing major compose-breaking bugs, at minimal cost. But it's a
significant change and I wanted to see what folks think. In particular,
if you find the existing gating of updates for stable/branched releases
to cause problems in any way, I'd love to hear about it.
Thanks folks!
--
Adam Williamson
Fedora QA
IRC: adamw | Twitter: adamw_ha
https://www.happyassassin.net
11 months, 2 weeks
F37 seems to require a reboot after updates of firmware packages in
order for firefox to run.
by stan
I'm running an actual install of F37. I have had this experience twice
now. I run the updates in a virtual console using dnf, they
complete successfully, they have a firmware package among the updates.
I then start X to run LXDE, and it starts without complaint. But when
I try to run either nightly from /usr/local or firefox from /usr/bin in
a terminal, they start and just hang. There are no messages in the
journal or in the terminal, and when I look at them using ps they are
in interruptable sleep.
In both cases, I killed them, and exited X, restarted X, and tried to
run them again. No change. But, when I reboot the computer,
everything starts working again the way I would expect; firefox starts.
Is this a change in F37 that requires a reboot after the install of
firmware packages? I know that Gnome, and I think KDE, now require a
reboot after every update. Has it moved to other desktops? Can I
turn it off, if it has? Is there a command I can run from the command
line to do the equivalent of a reboot? Or is it all just coincidence,
and there is another reason?
1 year, 8 months
F37 Beta blocker status email
by Ben Cotton
With F37 now branched from Rawhide, it's time to pick up our weekly
blocker status emails!
Action summary
====================
Accepted blockers
-----------------
1. kde-settings — KDE needs to pick up F37 backgrounds — NEW
ACTION: Maintainers to make new kde-settings build
Proposed blockers
-----------------
1. anaconda — Anaconda crashed on signal 11 - keyboard layout
encryption password prompt — ON_QA
ACTION: QA, reporter to verify anaconda-37.12-1
NEEDINFO: doug.hs
2. anaconda — Anaconda will not start F37 Raw — NEW
ACTION: Reporter to provide requested log files
NEEDINFO: aalmeleh.whatever.0101
3. dnf — "dnf update" runs out of memory on swapless 0.5 GiB RAM machine — NEW
ACTION: Manitainers to investigate short-term dnf improvements
NEEDINFO: jmracek
4. dracut — Media check fails with checkisomd5 "service failed because
the control process exited with error code" — POST
ACTION: Maintainers to build with upstream PR
5. polkit — Switching users in Gnome results in polkitd errors that
prevent powering off the computer. — ASSIGNED
ACTION: Maintainers to diagnose issue
6. sddm — Logout from KDE session on Rawhide goes to console, not SDDM — NEW
ACTION: Maintainers to diagnose issue
Bug-by-bug detail
=============
Accepted blockers
-----------------
1. kde-settings — https://bugzilla.redhat.com/show_bug.cgi?id=2117706 — NEW
KDE needs to pick up F37 backgrounds
New desktop backgrounds require a new KDE settings update.
Proposed blockers
-----------------
1. anaconda — https://bugzilla.redhat.com/show_bug.cgi?id=2070823 — ON_QA
Anaconda crashed on signal 11 - keyboard layout encryption password prompt
Clicking the keyboard layout button in the LUKS decryption popup
crashes anaconda. anaconda-37.12-1 contains a candidate fix.
2. anaconda — https://bugzilla.redhat.com/show_bug.cgi?id=2101229 — NEW
Anaconda will not start F37 Raw
Neither clicking the desktop icon nor the taskbar icon will launch
anaconda. openQA is not experiencing this. Using basic graphics mode
results in the expected behavior.
3. dnf — https://bugzilla.redhat.com/show_bug.cgi?id=1907030 — NEW
"dnf update" runs out of memory on swapless 0.5 GiB RAM machine
dnf is killed by oom-kill on machines with 512 MB of RAM. Adding a
swap disk or disabling the updates repo works around this issue.
Recently, FCOS has seen this behavior with 1 GB VMs. The issue appears
to be the size of the metadata for the repository. The short term
fixes look like they may need to be made on the repo side, not in dnf.
4. dracut — https://bugzilla.redhat.com/show_bug.cgi?id=2107858 — POST
Media check fails with checkisomd5 "service failed because the control
process exited with error code"
Media check fails on certain hardware. Skipping the media check
results in a successful boot. Upstream PR #1882 contains a candidate
fix: https://github.com/dracutdevs/dracut/pull/1882
5. polkit — https://bugzilla.redhat.com/show_bug.cgi?id=2109145 — ASSIGNED
Switching users in Gnome results in polkitd errors that prevent
powering off the computer.
Under certain conditions (including but not limited to several user
switches), the power off button disappears from the GNOME menu due to
a polkitd error. `systemctl poweroff -i` as an unprivileged user does
not work. It may be an issue with using the duktape JS engine instead
of mozjs91. polkit-121-3 reverts to mozjs91, but is not a sufficient
fix.
6. sddm — https://bugzilla.redhat.com/show_bug.cgi?id=2110801 — NEW
Logout from KDE session on Rawhide goes to console, not SDDM
After logging out of KDE Plasma, you end up at a console on tty2. SDDM
had been running on tty1 but just shows a flashing cursor.
--
Ben Cotton
He / Him / His
Fedora Program Manager
Red Hat
TZ=America/Indiana/Indianapolis
1 year, 8 months
Request for testing: Fedora 37 pre-Beta validation tests
by Adam Williamson
Hey folks!
So we're in freeze for Fedora 37 Beta now, and the first go/no-go
meeting should be on September 8.
It would be really great if we can get the validation tests run now so
we can find any remaining blocker bugs in good time to get them fixed.
Right now the blocker list looks short, but there are definitely some
tests that have not been run.
You can use the testcase_stats view to find tests that need running:
https://openqa.fedoraproject.org/testcase_stats/37/
For each validation test set (Base, Desktop etc.) it shows when each
test was last performed. So you can easily look for Basic and Beta
tests that have not yet been run. We need to run all of these.
You can enter results using `relval report-results`, or edit the
summary results page at
https://fedoraproject.org/wiki/Test_Results:Current_Summary . That's a
redirect link which will always point to the validation results page
for the currently-nominated compose, which right now is 20220826.n.0.
Sumantro will be running a validation 'test week' starting on
Wednesday, so you can drop by the Fedora Test Day room on
chat.fedoraproject.org to hang out with other testers and get any help
you need in testing. See
https://lists.fedoraproject.org/archives/list/test-announce@lists.fedorap...
for that announcement.
Thanks folks!
--
Adam Williamson
Fedora QA
IRC: adamw | Twitter: adamw_ha
https://www.happyassassin.net
1 year, 8 months
Re: Thoughts welcome: interface between automated test gating and
the "critical path"
by Adam Williamson
On Wed, 2022-08-31 at 12:43 +0000, Zbigniew Jędrzejewski-Szmek wrote:
> On Tue, Aug 30, 2022 at 12:10:00PM -0700, Adam Williamson wrote:
> > On Tue, 2022-08-30 at 09:14 -0400, Ben Cotton wrote:
> > > From my perspective, anything that blocks the release is on the
> > > critical path. So any time there's a violation of the release criteria
> > > and the package is not on the critical path definition, that's a bug
> > > in the definition.
> > >
> > > I recognize that this is a somewhat naïve view. For one, it may
> > > broaden the definition beyond the current capacity of our test
> > > infrastructure. It also may broaden the definition beyond what
> > > maintainers are willing to put up with. These are both legitimate
> > > problems. But the closer we can get to this ideal state, the better.
> > >
> > > For anyone who is curious, I just searched for all accepted blockers
> > > in the "Fedora" product in Bugzilla. 327 components have been a
> > > blocker at least once. Some of those may no longer be blocking and
> > > others will be added over time as our criteria change. The full list
> > > with counts is at
> > > https://bcotton.fedorapeople.org/release-blocking-components.csv if
> > > you're interested.
> >
> > Honestly, something along these lines would be my preference too, I
> > just don't know if others would agree/support changing the critical
> > path definition to "all release-blocking functionality" rather than
> > "functionality needed to boot a basically-functional system".
>
> I'd support increasing the scope to cover more packages in this fashion.
>
> Being on critpath list is somewhat annoying because the bodhi update
> minimum times are twice as long. For many packages this is a not a problem,
> popular packages get enough karma within a day or two,
> but if we expanded the list a lot, it could prove annoying to those
> packagers. I think we could discuss lowering the minium update time
> if this turns out to be the case.
So, that's an interesting question to consider as well, of course:
what's the actual impact of critpath?
It'd be an interesting outcome if we broadened the critpath definition
but diluted the Bodhi requirements, because the original purpose of
critpath was more or less entirely the stricter Bodhi requirements.
Until openQA came along, that was all it really meant: updates to these
packages have stricter Bodhi requirements.
Now, because I glued openQA to the critpath because it was handy, there
are two sets of consequences to a package being in critical path:
1. Tighter Bodhi requirements
2. openQA tests are run, and results gate the update (except Rawhide)
So, one of the implicit questions here is, is it OK to keep twinning
these two sets of consequences, or should we split them up? Splitting
them up kinda implies answer 2) from my original mail: "Keep the
current "critical path" concept but define a broader group
of "gated packages" somewhere". Because we would then need some new
concept that isn't "critical path". As I said, that's more *work* -
it'd require us to write new code in several places[0]. Even if we
decide it'd be nice to do this, is it nice *enough* to be worth doing
that work?
If we don't think it's worth doing that work, then we're kinda stuck
with openQA glomming onto the critpath definition to decide which
updates to test and gate, because I don't have any other current viable
choices for that, really. And we'd have to figure out a critpath
definition that's as viable as possible for both purposes.
BTW, one other thought I've had in relation to all this is that we
could enhance the current critpath definition somewhat. Right now, it's
built out of package groups in comps which are kinda topic-separated:
there's a critpath-kde, a critpath-gnome, a critpath-server, and so on.
But the generated critical path package list is a monolith: it doesn't
distinguish between a package that's on the GNOME critpath and a
package that's on the KDE critpath, you just get a big list of all
critpath packages. It might be nice if we actually did distinguish
between those - the critpath definition could keep track of which
critpath topic(s) a package is included in, and Bodhi could display
that information in the web UI and provide it via the API. That way
manual testers could get a bit more info on why a package is critpath
and what areas to test, and openQA could potentially target its test
runs to conserve resources a bit, though this might require a bit more
coding work on the gating stuff now I think about it.
[0]: at least: releng critpath generation scripts, bodhi, the openQA
scheduler, greenwave policies, and possibly greenwave
--
Adam Williamson
Fedora QA
IRC: adamw | Twitter: adamw_ha
https://www.happyassassin.net
1 year, 8 months
Thoughts welcome: interface between automated test gating and the
"critical path"
by Adam Williamson
Hi folks!
I have one of those definitional quandaries and I figured I'd throw it
at the lists for some input.
Right now, we kind of take advantage of the "critical path" concept for
automated update testing and gating via openQA.
openQA does not test all updates, only critical path updates plus
updates containing any package on a short allowlist.
Bodhi and Greenwave do not gate all updates on the openQA tests since
not all updates are tested; again we use the critpath definition. If an
update is critical path, it gets gated on the openQA tests. If it
isn't, it doesn't.
If you've been paying attention, that means there's a bit of a hole:
the packages on the 'allowlist'. These are tested, but not gated.
What's on the allowlist? Basically, FreeIPA-related packages. We have a
good set of FreeIPA tests in openQA, and both I and the FreeIPA team
wanted relevant updates to be tested with them. But those packages are
not on the critical path. So I set up this 'allowlist' mechanism.
However, the lack of gating kinda sucks. I would much prefer if we
could gate these updates as well as just testing them.
The obvious thing to do is add the FreeIPA-related packages to the
critical path, but that really is not supported by the critical path
definition:
https://fedoraproject.org/wiki/Critical_path_package
which defines it as packages required to "perform the most fundamental
actions on a system", with a list of specific areas, none of which is
"run a domain server".
So...what to do?
I can think of I guess four options:
1. Broaden the definition of the "critical path" somehow. We could just
write in that it includes FreeIPA functionality, I guess, though that
seems special purpose. We could broaden it to include any functionality
covered by the release criteria, which would be quite a big increase
but seems like a reasonable and concise definition. Or we could write
in that it includes any functionality that is exercised by the gating
openQA tests, though that seems a bit arbitrary - there's no
particularly great organizing principle to the openQA tests we choose
to run on updates, if I'm honest, it's a sort of grab bag I came up
with.
2. Keep the current "critical path" concept but define a broader group
of "gated packages" somewhere (could be comps or somewhere else). This
would require engineering work - we'd have to touch probably the openQA
scheduler, Bodhi, and greenwave configs. It's also another maintenance
burden.
3. Add gating config to each allowlisted package repo's gating.yml one
by one. I don't really like this option. It'd be a chunk of work to do
initially, and multiples the maintenance required when the list of
tests to gate on changes for some reason.
4. Do nothing, just keep only gating things that are "critical path" on
the current definition. In a utopian future, we'd manage to deploy
openQA in the cloud or get a giant pile of super fast servers so we
could test and gate *every* update, and this wouldn't be an issue any
more.
What do folks think? Any preferences? Thanks!
--
Adam Williamson
Fedora QA
IRC: adamw | Twitter: adamw_ha
https://www.happyassassin.net
1 year, 8 months
Re: Thoughts welcome: interface between automated test gating and
the "critical path"
by Adam Williamson
On Tue, 2022-08-30 at 09:39 +0300, Alexander Bokovoy wrote:
> On ma, 29 elo 2022, Adam Williamson wrote:
> > On Tue, 2022-08-30 at 00:32 -0400, DJ Delorie wrote:
> > > It sounds to me like the problem is "how do we best use the available
> > > automated test resources?" so I'll answer accordingly. Ignore me if I
> > > misunderstood ;-)
> >
> > No, not really, sorry if I didn't explain clearly enough :D It's more
> > just a 'what's the best way to work what I want to do around the
> > existing definitions' question. What I want to do is have updates of
> > FreeIPA-related packages gated. The awkwardness is that right now the
> > definition of what gets gated is "critical path packages", and those
> > clearly don't meet the current definition of "critical path".
>
> Not all of them meet the current definition of 'critical path' but many
> do. In fact, almost all crypto-implementing packages should be on that
> list, if not already.
Sure, but freeipa itself, for instance, can't really be as things
stand. Nor could bind, I don't think. And several others.
>
> Ideally, use of gating tests should be defined in the component itself.
> Do we have a mechanism to trigger OpenQA runs from the gating.yaml in
> the specific component? If we do, then we really should move it there.
No. I don't think this is in-scope for gating.yaml. I *could* extend
the openQA scheduler to check some kind of per-repo config for every
package in the update, but there is nothing like this at present.
In any case, triggering the *tests* isn't the problem here; we already
do that, with the allowlist. It's doing the *gating* that's the
problem. As I said, we could add *that* to gating.yaml for each repo,
but I don't really love that idea, because you have to provide a full
list of what tests to gate on, and doing that for each repo and keeping
it updated seems like kind of a drag.
I guess in a sense it's appropriate for these packages, though, as you
could say we'd "really" want to gate on the FreeIPA-specific tests, or
at least just the server tests. But honestly, if an update to a
freeipa-related package causes GNOME to start crashing or something,
that's something I want to look into before the update goes stable,
even if it seems very weird.
>
> Another part of the problem is that most of these runs should probably
> be consulting, not gating in the sense that failing them should give you
> a clue that things might be wrong but there needs to be a human overview
> of the failures. It is what we have right now with OpenQA-reported
> failures for Bodhi updates in most cases, the test results aren't really
> preventing the submissions from the going forward unless a maintainer
> defined the update configuration to be so.
The problem cases here are what I was thinking about. They are:
Branched before Beta freeze, and Rawhide.
For those cases, updates are automatically created on `fedpkg build`
and immediately submitted for stable. If there is no applicable gating
policy they are immediately pushed stable. There is no time for anyone
to consult.
If you use a side tag there's no automatic submission, I don't think,
but people do not use side tags as a matter of course.
Right now we don't gate Rawhide updates, but I would like to and I am
actively working on it (this would prevent me having the kind of
nightmare week I had last week where I spent dozens of hours tracking
down things that had broken in Rawhide, getting them fixed, and re-
running tests).
We *do* currently gate Branched updates before Beta freeze, and I want
FreeIPA-related packages included in that for the next cycle. I would
have liked it if, for the F37 cycle, FreeIPA-related updates that were
submitted between Branch point and Beta freeze were gated. But they
weren't.
Practically speaking, the way we do things in Fedora, there is very
little difference between "gating" and "consulting"; because of the
waiver mechanism they're effectively identical. You can always issue
waivers to override the gating. So really, gating is consulting. You
can look at the failures, decide they're bogus, and hit the waiver
button, which will immediately make the update 'pass' gating. I do
encourage people not to abuse the waiver function - if the failure
seems like a flake please use the 're-run tests' button, if you really
think it's bogus please poke me or Lukas or someone as I'm probably
working right then on dealing with it - but it is right there.
>
> Ideally, we should have reverse dependency updates to trigger FreeIPA
> tests and give their results a visibility but don't block on failures.
> This would give an opportunnity to notice a failurer earlier but don't
> discourage maintainers from participating in a joint development.
We would like reverse dependency testing for all sorts of reasons, but
it's also all sorts of difficult (and, potentially, resource-
intensive). Of course, in openQA testing we do sort of have this
already. We run *all* tests on all critpath updates. So if an update to
a critpath component breaks FreeIPA, we find out about it immediately.
We don't have to wait for a FreeIPA-related update to show up. If the
dep isn't critpath, though, we don't test it.
> Perhaps, The Fedora Packager Dashboard could be extended to pick up
> results of tests relevant to your packages and display them together?
> This way FreeIPA maintainers can see an overview of all tests related to
> FreeIPA in one place and see if any help is needed to resolve failures.
This would be lovely, sure, but at this point we're quite far away from
the fairly concrete initial scope of my mail. I don't want to get
sidetracked into sky castle building where we file tickets and wait two
years for people to build stuff. I would like to gate updates to
FreeIPA-related packages *now*. I was all set to submit a pull request
to comps to add them to the server critpath definition, but then I
checked the official definition of "critical path" and noticed this
problem. That's where I'm coming from here: I want to make a concrete
change now, not design awesome speculative improvements on paper.
--
Adam Williamson
Fedora QA
IRC: adamw | Twitter: adamw_ha
https://www.happyassassin.net
1 year, 8 months
Thoughts welcome: interface between automated test gating and the
"critical path"
by pmkellly＠frontier.com
On 8/29/22 6:50 PM, Adam Williamson wrote:
> Hi folks!
>
> I have one of those definitional quandaries and I figured I'd throw it
> at the lists for some input.
I spent some years as a project manager and I have lots of experience
with schedules. The term Crital Path was formalized in the business
world when project managers learned to use Gantt Charts for their
scheduling. The Critical Path was fluid and changed as various task
dependancies were actually or potentially impacting the project
completion date. The tasks and resources on the critical path got
special attention to mitigate the delay. This usually led to changes in
the critical path or new a critical path would come up. Redefining the
term is is not a clean solution ot this problem.
First, I think the terminology needs to change. I propose having a
critical package list. The critical package list would be defined as
something along the lines: "These packages should be working in Fedora
as released as Beta or Final. Failing packages must have blocker bugs
and/or freeze exceptions." Notice the should gives room for making
exceptions as packages fail near release time.
After that all critical packages should be tested and evaluated with
openQA and Gating.
To adopt this approach their two tasks:
First decide what packages are on the Critical Package List. The reasons
will be many and I don't see a need to document the reasons. A small
group of people can make up the list and put it before the comunity for
agreement.
Second change the testing and gatiing so they work from the list
Have Great Day!
Pat tablepc
1 year, 8 months
