reversible dual-boot test station
by Chris Murphy
Hi,
I mentioned this in a QA meeting, and have given it enough testing that I think it's broadly usable. If desired it can be copied out of my user account and put up somewhere where QA folks will see it and can modify it as issues or improvements are discovered.
What is it? The idea is to produce a system that can confidently be used for baremetal testing, without risking the primary operating system. While VM's are a great way to test, it's also a really idealized environment that tends to not expose an assortment of bugs that affect particular hardware. But then quite a lot of folks reasonably don't want to upgrade their daily use hardware early on, because they don't want to always have to debug things, or have to figure out how to undo the upgrade if it really goes badly.
Therefore, I present a dual-boot setup offering:
* no re-partitioning;
* no installation step, instead system upgrade is used;
* reversibility, or undoability, i.e. with just a few steps you can delete the "test OS".
https://fedoraproject.org/wiki/User:Chrismurphy/Draft/dualboot_teststation
--
Chris Murphy
2 months
Re: Thoughts welcome: interface between automated test gating and
the "critical path"
by Adam Williamson
On Fri, 2022-09-02 at 08:37 +0000, Zbigniew Jędrzejewski-Szmek wrote:
> >
> > Now, because I glued openQA to the critpath because it was handy, there
> > are two sets of consequences to a package being in critical path:
> >
> > 1. Tighter Bodhi requirements
> > 2. openQA tests are run, and results gate the update (except Rawhide)
> >
> > So, one of the implicit questions here is, is it OK to keep twinning
> > these two sets of consequences, or should we split them up? Splitting
> > them up kinda implies answer 2) from my original mail: "Keep the
> > current "critical path" concept but define a broader group
> > of "gated packages" somewhere". Because we would then need some new
> > concept that isn't "critical path". As I said, that's more *work* -
> > it'd require us to write new code in several places[0]. Even if we
> > decide it'd be nice to do this, is it nice *enough* to be worth doing
> > that work?
>
> I'd still vote for keeping a single critpath list and using it as
> "the list of packages that require extra care and testing".
>
> As you describe, the original meaning of critpath has shifted, but
> it's because the way we do updates and QA has also shifted. Doing
> gating tests for a package seems much more useful than just keeping
> it longer in 'updates-testing' in hope that somebody discovers an
> important regresion in the second week.
Well, there's a caveat there - openQA doesn't test everything. On the
whole we cover quite a lot with the set of tests that gets run on
updates, but there's certainly lots of potential for there to be
important bugs it misses, that a human tester might catch. So I think
there is still a case for the higher karma requirements too.
>
> So yeah, I don't think it makes sense to do the extra work to split
> the concepts. Also because we have way too many concepts and processes
> in Fedora already.
On the whole, though, I agree with you. I just don't trust my own
opinion because it's obviously biased by what's convenient for me. :D
> > If we don't think it's worth doing that work, then we're kinda stuck
> > with openQA glomming onto the critpath definition to decide which
> > updates to test and gate, because I don't have any other current viable
> > choices for that, really. And we'd have to figure out a critpath
> > definition that's as viable as possible for both purposes.
> >
> >
> > BTW, one other thought I've had in relation to all this is that we
> > could enhance the current critpath definition somewhat. Right now, it's
> > built out of package groups in comps which are kinda topic-separated:
> > there's a critpath-kde, a critpath-gnome, a critpath-server, and so on.
> > But the generated critical path package list is a monolith: it doesn't
> > distinguish between a package that's on the GNOME critpath and a
> > package that's on the KDE critpath, you just get a big list of all
> > critpath packages. It might be nice if we actually did distinguish
> > between those - the critpath definition could keep track of which
> > critpath topic(s) a package is included in, and Bodhi could display
> > that information in the web UI and provide it via the API. That way
> > manual testers could get a bit more info on why a package is critpath
> > and what areas to test, and openQA could potentially target its test
> > runs to conserve resources a bit, though this might require a bit more
> > coding work on the gating stuff now I think about it.
>
> That sounds useful. We only need a volunteer to figure out the details
> and do the work ;)
I actually did a huge rewrite of the thing that generates the critpath
data this week, and it probably wouldn't be tooooo much work, honestly.
The most annoying bit would be the Bodhi frontend stuff, but that's
because I'm bad at frontend dev in general. :P But yeah, this is
definitely off in sky-castle land. I'll add it to my ever-growing list
of sky-castle projects to do when I get a couple of years of spare
time...
--
Adam Williamson
Fedora QA
IRC: adamw | Twitter: adamw_ha
https://www.happyassassin.net
8 months, 2 weeks
Plan / proposal: enable openQA update testing and potentially
gating on Rawhide updates
by Adam Williamson
Hi folks!
We've had openQA testing of updates for stable and branched releases,
and gating based on those tests, enabled for a while now. I believe
this is going quite well, and I think we addressed the issues reported
when we first enabled gating - Bodhi's gating status updates work more
smoothly now, and openQA respects Bodhi's "re-run tests" button so
failed tests can be re-triggered.
A few weeks ago, I enabled testing of Rawhide updates in the openQA
lab/stg instance. This was to see how smoothly the tests run, how often
we run into unexpected failures or problems, and whether the hardware
resources we have are sufficient for the extra load.
So far this has been going more smoothly than I anticipated, if
anything. The workers seem to keep up with the test load, even though
one out of three worker systems for the stg instance is currently out
of commission (we're using it to investigate a bug). We do get
occasional failures which seem to be related to Rawhide kernel slowness
(e.g. operations timing out that usually don't otherwise time out), but
on the whole, the level of false failures is (I would say) acceptably
low, enough that my current regime of checking the test results daily
and restarting failed ones that don't seem to indicate a real bug
should be sufficient.
So, I'd like to propose that we enable Rawhide update testing on the
production openQA instance also. This would cause results to appear on
the Automated Tests tab in Bodhi, but they would be only informational
(and unless the update was gated by a CI test, or somehow otherwise
configured not to be pushed automatically, updates would continue to be
pushed 'stable' almost immediately on creation, regardless of the
openQA results).
More significantly, I'd also propose that we turn on gating on openQA
results for Rawhide updates. This would mean Rawhide updates would be
held from going 'stable' (and included in the next compose) until the
gating openQA tests had run and passed. We may want to do this a bit
after turning on the tests; perhaps Fedora 37 branch point would be a
natural time to do it.
Currently this would usually mean a wait from update submission to
'stable push' (which really means that the build goes into the
buildroot, and will go into the next Rawhide compose when it happens)
of somewhere between 45 minutes and a couple of hours. It would also
mean that if Rawhide updates for inter-dependent packages are not
correctly grouped, the dependent update(s) will fail testing and be
gated until the update they depend on has passed testing and been
pushed. The tests for the dependent update(s) would then need to be re-
run, either by someone hitting the button in Bodhi or an openQA admin
noticing and restarting them, before the dependent update(s) could be
pushed.
In the worst case, if updated packages A and B both need the other to
work correctly but the updates are submitted separately, both updates
may fail tests and be blocked. This could only be resolved by waiving
the failures, or replacing the separate updates with an update
containing both packages.
All of those considerations are already true for stable and branched
releases, but people are probably more used to grouping updates for
stable and branched than doing it for Rawhide, and the typical flow of
going from a build to an update provides more opportunity to create
grouped updates for branched/stable. For Rawhide the easiest way to do
it if you need to do it is to do the builds in a side tag and use
Bodhi's ability to create updates from a side tag.
As with branched/stable, only critical path updates would have the
tests run and be gated on the results. Non-critpath updates would be
unaffected. (There's a small allowlist of non-critpath packages for
which the tests are also run, but they are not currently gated on the
results).
I think doing this could really help us keep Rawhide solid and avoid
introducing major compose-breaking bugs, at minimal cost. But it's a
significant change and I wanted to see what folks think. In particular,
if you find the existing gating of updates for stable/branched releases
to cause problems in any way, I'd love to hear about it.
Thanks folks!
--
Adam Williamson
Fedora QA
IRC: adamw | Twitter: adamw_ha
https://www.happyassassin.net
10 months, 3 weeks
Fedora Linux 38 blocker status summary
by Ben Cotton
We've branched F38 from Rawhide, so it's time to start everyone's
favorite Friday email from Ben! The F38 Beta freeze begins on 21
February. The current target release date is the early target date
(2023-03-14).
Action summary
====================
Accepted blockers
-----------------
1–3. distribution — {Workstation,Everything,Server} boot x86_64 image
exceeds maximum size — ASSIGNED
ACTION: Relevant Teams to reduce image size or increase the limit
Proposed blockers
-----------------
1. anaconda — installer fails to boot in text mode or rescue mode with
systemd 253 — MODIFIED
ACTION: Maintainers to build an update that includes upstream PR
2. grub2 — ext4 filessystem support missing — NEW
ACTION: Maintainer to diagnose issue
NEEDINFO: ausil
3. kwin — kwin_wayland often crashed when used as the sddm Wayland
compositor and logging out of Plasma resulting in a black screen — NEW
ACTION: Maintainer to diagnose issue
4. mesa — KDE crashes on start with mesa-23.0.0~rc3-3.fc38 — ASSIGNED
ACTION: Maintainer to diagnose issue
Bug-by-bug detail
=============
Accepted blockers
-----------------
1–3. distribution — {Workstation,Everything,Server} boot x86_64 image
exceeds maximum size — ASSIGNED
https://bugzilla.redhat.com/show_bug.cgi?id=2149246
https://bugzilla.redhat.com/show_bug.cgi?id=2151495
https://bugzilla.redhat.com/show_bug.cgi?id=2151497
Image sizes exceed the specified limits. The choices are to either
shrink the image by removing packages or to riase the limits.
Proposed blockers
-----------------
1. anaconda — https://bugzilla.redhat.com/show_bug.cgi?id=2165433 — MODIFIED
installer fails to boot in text mode or rescue mode with systemd 253
anaconda is writing systemd units to an unexpected area. Upstream PR
4534 contains a candidate fix:
https://github.com/rhinstaller/anaconda/pull/4534
2. grub2 — https://bugzilla.redhat.com/show_bug.cgi?id=2166412 — NEW
ext4 filessystem support missing
Between grub2-2.06-76 and grub2-2.06-78, grub2 apparently lost the
ability to detect ext4 filesystems.
3. kwin — https://bugzilla.redhat.com/show_bug.cgi?id=2168034 — NEW
kwin_wayland often crashed when used as the sddm Wayland compositor
and logging out of Plasma resulting in a black screen
Logging out of Plasma sometimes triggers a kwin crash. This may be
limited to running in a virtual machine.
4. mesa — https://bugzilla.redhat.com/show_bug.cgi?id=2164667 — ASSIGNED
KDE crashes on start with mesa-23.0.0~rc3-3.fc38
A recent mesa update caused both GNOME and KDE to crash on start.
Update FEDORA-2023-40b973fa06 fixed GNOME, but KDE is still seeing
this issue. The root cause is still uncertain.
--
Ben Cotton
He / Him / His
Fedora Program Manager
Red Hat
TZ=America/Indiana/Indianapolis
1 year
Unable to install locally built rpms
by Ralf Corsépius
Hi,
on f38, I am unable to install any locally built package (signed with a
local key, I have been using for many years):
# rpm -U xpetri-0.4.8-0.fc38.x86_64.rpm
error: xpetri-0.4.8-0.fc38.x86_64.rpm: Header V4 RSA/SHA256 Signature,
key ID a6b9312e: BAD
error: xpetri-0.4.8-0.fc38.x86_64.rpm cannot be installed
# rpm -qip xpetri-0.4.8-0.fc38.x86_64.rpm
error: xpetri-0.4.8-0.fc38.x86_64.rpm: Header V4 RSA/SHA256 Signature,
key ID a6b9312e: BAD
error: xpetri-0.4.8-0.fc38.x86_64.rpm: not an rpm package (or package
manifest)
# dnf install xpetri-0.4.8-0.fc38.x86_64.rpm
Last metadata expiration check: 1:30:47 ago on Tue 28 Feb 2023 06:25:45
AM CET.
Dependencies resolved.
...
0.4.8-0.fc38 @commandline
...
Installing dependencies:
...
Downloading Packages:
(1/6): XXX.rpm ...
Total
1.2 MB/s |
130 kB 00:00
...
Problem opening package XXX.rpm
...
The downloaded packages were saved in cache until the next successful
transaction.
You can remove cached packages by executing 'dnf clean packages'.
Error: GPG check FAILED
Worse, after trying forcefully to install packages using rpm -U --nogpg
this happens:
# rpm -qa
gpg-pubkey-d651ff2e-5dadbbc1
gpg-pubkey-8ff214b4-3afa5d46
gpg-pubkey-a6b9312e-5227e975
gpg-pubkey-94843c65-5dadbc64
error: rpmdbNextIterator: skipping h# 5
Header V4 DSA/SHA1 Signature, key ID 8ff214b4: BAD
Header SHA256 digest: OK
Header SHA1 digest: OK
error: rpmdbNextIterator: skipping h# 6
Header V3 RSA/SHA1 Signature, key ID d651ff2e: BAD
Header SHA256 digest: OK
Header SHA1 digest: OK
gpg-pubkey-5323552a-6112bcdc
...
=> nogpg is not ignored, as it is supposed to be.
What are people supposed to do?
Ralf
1 year, 1 month
Xfce ~ Lost all menu icons in Rawhide and Beta Branch
by Ian Laurie
In Rawhide (fc39) and the beta branch (fc38), with all updates applied,
I'm seeing all menu icons are gone in Xfce... both the system menu and
right-click context menus.
Seems the Settings Tab in Settings->Appearance now has all options
turned off, including "Show images in menus". Updates did this.
Ticking this option brings sanity back. Was this really intentional or
is it a bug? I think this deviation from traditional defaults is going
to cause beginners a lot of grief.
The Xfce updates in the total update payload were as follows:
xfce4-about-4.18.2-1.fc38.x86_64
xfce4-notifyd-0.8.0-1.fc38.x86_64
xfce4-panel-4.18.2-1.fc38.x86_64
xfce4-power-manager-4.18.1-1.fc38.x86_64
xfce4-session-4.18.1-1.fc38.x86_64
xfce4-settings-4.18.2-1.fc38.x86_64
I'm guessing it was xfce4-settings that did it? If this was intentional
I think it was a really bad idea.
Ian
--
Ian Laurie
FAS: nixuser | IRC: nixuser
TZ: Australia/Sydney
1 year, 1 month
Joining Fedora QA
by Mark Rosenbaum
Hello,
I am looking to join Fedora QA and help mostly with triage/bug reproducibility. I am currently a contributor to a multitude of open source projects including the Chromium Project where I hold bug editing permissions(mark.rosenbaum(a)chromium.org).
I look forward to working with you,
Mark Rosenbaum
1 year, 1 month
Fedora 38: GDM not recognizing Wayland, only X11
by Scott Beamer
Greetings,
I did a fresh install of Fedora 38 on 19 February from the latest Live
images (both Workstation and KDE Spin).
Workstation install lists no option for either X11 or Wayland. Default
GNOME selection lands you in a GNOME session under X11.
I then installed the "KDE Plasma Workspaces" package group.
GDM then listed Plasma as an option, but only under X11.
I then switched from GDM to SDDM and SDDM shows options for both KDE and
GNOME under either X11 or Wayland.
If I select GNOME (Wayland) in SDDM and log in, I'm indeed logged into a
GNOME session under Wayland.
I did the reverse in the KDE Spin. SDDM recognized both Wayland and
X11, switching to GDM only gave me X11 as an option under either GNOME
or KDE.
Has anyone else seen this?
Scott
1 year, 2 months
Re: Test upgrades from F37 to F38 - it will take you just a minute
by Luna Jernberg
Did this 16th February and it worked then, other then a few packages
missing in F38 that was in F37 then mostly Google Fonts and RPM Fusion
stuff
On 2/22/23, Miroslav Suchý <msuchy(a)redhat.com> wrote:
> Do you want to make Fedora 38 better? Please spend 1 minute of your time and
> try to run:
>
> # Run this only if you use default Fedora modules
> # next time you run any DNF command default modules will be enabled again
> sudo dnf module reset '*'
>
> dnf --releasever=38 --setopt=module_platform_id=platform:f38 \
> --enablerepo=updates-testing \
> $(rpm -q fedora-repos-modular >/dev/null && echo
> --enablerepo=updates-testing-modular) \
> --assumeno distro-sync
>
>
> This command does not replace `dnf system-upgrade`, but it will reveals
> potential problems.
>
> You may also run `dnf upgrade` before running this command.
>
>
> The `--assumeno` will just test the transaction, but does not make the
> actual upgrade.
>
>
> In case you hit dependency issues, please report it against the appropriate
> package.
>
> Or against fedora-obsolete-packages if that package should be removed in
> Fedora 38. Please check existing reports against
>
> fedora-obsolete-packages first:
>
> https://red.ht/2kuBDPu
>
> and also there is already bunch of "Fails to install" (F38FailsToInstall)
> reports:
>
> https://bugzilla.redhat.com/show_bug.cgi?id=F38FailsToInstall
>
> Thank you
>
> Miroslav
>
1 year, 2 months
