How good is the real quality and reliability of our deliverables?

[Peter Boy]

We are officially distributing a Fedora Server Edition disk image for ARM, typically used
for SBCs. I just tried to install this image on a Radxa Rock Pi 4 and failed miserably.
That SBC device is part of the installation procedure provided by the ARM team. With the
F34 image I ended up with a black screen, no display, no chance to get the device working
in the normal way (I got it to work in tricky detours). With F35 Beta I got a working
display, but no network connection. I haven't found a solution for that yet.

And then I remembered F33, where it was discovered by chance (!) at the last minute, that
in the x86_64 version Cockpit did not work properly (and probably also in the aarch64
default install version, we didn't and probably couldn’t even check that afterwards).
We have not solved the underlying issue for this until today, although the factual reason
has been clear for a long time (issue #32). 

Similar problems had emerged after the release with the switch to systemd-resolved, where
"surprisingly" problems with some server configurations were encountered,
especially with elaborate use of KVM virtualization on a server.



These events are clear violations of our release criteria. In principle, a release must be
at least successfully installable and basically working.



For me, this brings up a few questions:

(1)
Did any of us actually have Fedora Beta installed on a ARM device (SBC or other) using the
image file and on which model? 
Or are we just flying blind with that deliverable?

(2)
What about the two aarch64 install iso files that target SBSA hardware? Does anyone have
such a beast at hand and tested the isos? Or is that (also) flying blind?

(3)
And what about x86_64 installation? Most of us will be in possession of this hardware. But
what about tests? When I started a discussion about testing a year ago, Adam reassured us,
release building and testing is heavily automated these days. No need to worry too much
about testing it manually ourselves. But did anyone of us at least worry a bit?


I would like to discuss how we can gain some improvement.

Regarding arch architecture, could some kind of cooperation with arm group be beneficial?
And what can we offer? Is it testing? Is it documentation? Or something else? 
If we want to take SBC devices seriously, perhaps we could identify a few reference
systems where we can actually ensure, i.e. test, whether a successful installation and
basic functionality is met. If we don't want that, we should not distribute the image
file under the Fedora Server Edition branding.

And in terms of our "main" architecture, we should agree on what QA actually
expects in terms of non-automated testing and how we want to tackle that in the next
round. Based on Adam's comments, we have relied heavily on automation for the last few
releases. So it's not clear to me what, for example, the page 
https://fedoraproject.org/wiki/Test_Results:Fedora_35_RC_1.2_Server means in detail in
terms of work.





Happy Sunday
Peter