We are officially distributing a Fedora Server Edition disk image for ARM, typically used for SBCs. I just tried to install this image on a Radxa Rock Pi 4 and failed miserably. That SBC device is part of the installation procedure provided by the ARM team. With the F34 image I ended up with a black screen, no display, no chance to get the device working in the normal way (I got it to work in tricky detours). With F35 Beta I got a working display, but no network connection. I haven't found a solution for that yet.
And then I remembered F33, where it was discovered by chance (!) at the last minute, that in the x86_64 version Cockpit did not work properly (and probably also in the aarch64 default install version, we didn't and probably couldn’t even check that afterwards). We have not solved the underlying issue for this until today, although the factual reason has been clear for a long time (issue #32).
Similar problems had emerged after the release with the switch to systemd-resolved, where "surprisingly" problems with some server configurations were encountered, especially with elaborate use of KVM virtualization on a server.
These events are clear violations of our release criteria. In principle, a release must be at least successfully installable and basically working.
For me, this brings up a few questions:
(1) Did any of us actually have Fedora Beta installed on a ARM device (SBC or other) using the image file and on which model? Or are we just flying blind with that deliverable?
(2) What about the two aarch64 install iso files that target SBSA hardware? Does anyone have such a beast at hand and tested the isos? Or is that (also) flying blind?
(3) And what about x86_64 installation? Most of us will be in possession of this hardware. But what about tests? When I started a discussion about testing a year ago, Adam reassured us, release building and testing is heavily automated these days. No need to worry too much about testing it manually ourselves. But did anyone of us at least worry a bit?
I would like to discuss how we can gain some improvement.
Regarding arch architecture, could some kind of cooperation with arm group be beneficial? And what can we offer? Is it testing? Is it documentation? Or something else? If we want to take SBC devices seriously, perhaps we could identify a few reference systems where we can actually ensure, i.e. test, whether a successful installation and basic functionality is met. If we don't want that, we should not distribute the image file under the Fedora Server Edition branding.
And in terms of our "main" architecture, we should agree on what QA actually expects in terms of non-automated testing and how we want to tackle that in the next round. Based on Adam's comments, we have relied heavily on automation for the last few releases. So it's not clear to me what, for example, the page https://fedoraproject.org/wiki/Test_Results:Fedora_35_RC_1.2_Server means in detail in terms of work.
Happy Sunday Peter
Dear Peter Boy,
I didn't yet test the server variant of the x86_64 architecture of Fedora 35 RC 1.2 manually; however I have tested some of the base images for Fedora workstation 35 RC 1.1 for the MATE spin. All of which passed. Also I tested the "Everything netinst" for Fedora Workstation 35 RC 1.2 that passed as well. Majority of the tests are ran by Coconut, one of the bots. Mostly through openQA. If required I can test the server server variant of the x86_64 architecture of Fedora 35 RC 1.2 manually. I do think that manual testing is better as bots are able to sometimes become faulty.
On Sun, 31 Oct 2021 at 11:50, Peter Boy pboy@uni-bremen.de wrote:
We are officially distributing a Fedora Server Edition disk image for ARM, typically used for SBCs. I just tried to install this image on a Radxa Rock Pi 4 and failed miserably. That SBC device is part of the installation procedure provided by the ARM team. With the F34 image I ended up with a black screen, no display, no chance to get the device working in the normal way (I got it to work in tricky detours). With F35 Beta I got a working display, but no network connection. I haven't found a solution for that yet.
And then I remembered F33, where it was discovered by chance (!) at the last minute, that in the x86_64 version Cockpit did not work properly (and probably also in the aarch64 default install version, we didn't and probably couldn’t even check that afterwards). We have not solved the underlying issue for this until today, although the factual reason has been clear for a long time (issue #32).
Similar problems had emerged after the release with the switch to systemd-resolved, where "surprisingly" problems with some server configurations were encountered, especially with elaborate use of KVM virtualization on a server.
These events are clear violations of our release criteria. In principle, a release must be at least successfully installable and basically working.
For me, this brings up a few questions:
(1) Did any of us actually have Fedora Beta installed on a ARM device (SBC or other) using the image file and on which model? Or are we just flying blind with that deliverable?
(2) What about the two aarch64 install iso files that target SBSA hardware? Does anyone have such a beast at hand and tested the isos? Or is that (also) flying blind?
(3) And what about x86_64 installation? Most of us will be in possession of this hardware. But what about tests? When I started a discussion about testing a year ago, Adam reassured us, release building and testing is heavily automated these days. No need to worry too much about testing it manually ourselves. But did anyone of us at least worry a bit?
I would like to discuss how we can gain some improvement.
Regarding arch architecture, could some kind of cooperation with arm group be beneficial? And what can we offer? Is it testing? Is it documentation? Or something else? If we want to take SBC devices seriously, perhaps we could identify a few reference systems where we can actually ensure, i.e. test, whether a successful installation and basic functionality is met. If we don't want that, we should not distribute the image file under the Fedora Server Edition branding.
And in terms of our "main" architecture, we should agree on what QA actually expects in terms of non-automated testing and how we want to tackle that in the next round. Based on Adam's comments, we have relied heavily on automation for the last few releases. So it's not clear to me what, for example, the page https://fedoraproject.org/wiki/Test_Results:Fedora_35_RC_1.2_Server means in detail in terms of work.
Happy Sunday Peter
server mailing list -- server@lists.fedoraproject.org To unsubscribe send an email to server-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/server@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Dear Ahmed
Am 31.10.2021 um 13:39 schrieb Ahmed Almeleh aalmeleh.whatever.0101@gmail.com:
I didn't yet test the server variant of the x86_64 architecture of Fedora 35 RC 1.2 manually; however I have tested some of the base images for Fedora workstation 35 RC 1.1 for the MATE spin. All of which passed. Also I tested the "Everything netinst" for Fedora Workstation 35 RC 1.2 that passed as well. Majority of the tests are ran by Coconut, one of the bots. Mostly through openQA. If required I can test the server server variant of the x86_64 architecture of Fedora 35 RC 1.2 manually. I do think that manual testing is better as bots are able to sometimes become faulty.
many many thanks for your offering. I'm afraid it's too late now for testing the installation. F35 is go and Tuesday is release time. Now one can only hope. But it would be very helpful if we could count on your help for rawhide / F36.
I really don’t know what is best to do. I’m wondering (and worrying) how these issues came about. In fact, I don't even know what is tested per bot in detail. With workstations, there are likely to be hardware and driver peculiarities (different graphics adapters, proprietary wifi, etc.). With headless servers, many things do not even arise. How does the workstation WG organize all of that?
Thanks Peter
mostly through written tickets on bugzilla and fedora blocker bugs. Then we QA test different bugs to see if we get the same results.
On Sun, 31 Oct 2021 at 14:35, Peter Boy pboy@uni-bremen.de wrote:
Dear Ahmed
Am 31.10.2021 um 13:39 schrieb Ahmed Almeleh <
aalmeleh.whatever.0101@gmail.com>:
I didn't yet test the server variant of the x86_64 architecture of
Fedora 35 RC 1.2 manually; however I have tested some of the base images for Fedora workstation 35 RC 1.1 for the MATE spin. All of which passed. Also I tested the "Everything netinst" for Fedora Workstation 35 RC 1.2 that passed as well. Majority of the tests are ran by Coconut, one of the bots. Mostly through openQA. If required I can test the server server variant of the x86_64 architecture of Fedora 35 RC 1.2 manually. I do think that manual testing is better as bots are able to sometimes become faulty.
many many thanks for your offering. I'm afraid it's too late now for testing the installation. F35 is go and Tuesday is release time. Now one can only hope. But it would be very helpful if we could count on your help for rawhide / F36.
I really don’t know what is best to do. I’m wondering (and worrying) how these issues came about. In fact, I don't even know what is tested per bot in detail. With workstations, there are likely to be hardware and driver peculiarities (different graphics adapters, proprietary wifi, etc.). With headless servers, many things do not even arise. How does the workstation WG organize all of that?
Thanks Peter _______________________________________________ server mailing list -- server@lists.fedoraproject.org To unsubscribe send an email to server-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/server@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Am 31.10.2021 um 16:03 schrieb Ahmed Almeleh aalmeleh.whatever.0101@gmail.com:
mostly through written tickets on bugzilla and fedora blocker bugs. Then we QA test different bugs to see if we get the same results.
there were none regarding server, as far as I know.
Let's wait and see what other members come up with.
On Sun, 31 Oct 2021 at 14:35, Peter Boy pboy@uni-bremen.de wrote: Dear Ahmed
Am 31.10.2021 um 13:39 schrieb Ahmed Almeleh aalmeleh.whatever.0101@gmail.com:
I didn't yet test the server variant of the x86_64 architecture of Fedora 35 RC 1.2 manually; however I have tested some of the base images for Fedora workstation 35 RC 1.1 for the MATE spin. All of which passed. Also I tested the "Everything netinst" for Fedora Workstation 35 RC 1.2 that passed as well. Majority of the tests are ran by Coconut, one of the bots. Mostly through openQA. If required I can test the server server variant of the x86_64 architecture of Fedora 35 RC 1.2 manually. I do think that manual testing is better as bots are able to sometimes become faulty.
many many thanks for your offering. I'm afraid it's too late now for testing the installation. F35 is go and Tuesday is release time. Now one can only hope. But it would be very helpful if we could count on your help for rawhide / F36.
I really don’t know what is best to do. I’m wondering (and worrying) how these issues came about. In fact, I don't even know what is tested per bot in detail. With workstations, there are likely to be hardware and driver peculiarities (different graphics adapters, proprietary wifi, etc.). With headless servers, many things do not even arise. How does the workstation WG organize all of that?
Thanks Peter
Not sure if it counts as "test": I usually get impatient if it looks like a new release is soon to come (around first beta), so I upgrade to the new version (x86_64), so this might be some kind of upgrade-path test, but its in no way structured in any way.
I think automated testing can catch most of the basic bugs, which are not related to specific hardware problems - a lot of bugs seem to be caught by the automatic installation test.
And manual testing seems to be a lot of work, with less reward. It looks to me that bugs not detected by automatic testing have some specific requirement on hardware/non-standard installation/network environment/… which is likely to be found during normal use, so I'd just recommend to run beta releases if possible, that way releases are better tested and the main release might contain less bugs.
All the best, Astra
On Sun, Oct 31, 2021 at 04:56:12PM +0100, Peter Boy wrote:
Am 31.10.2021 um 16:03 schrieb Ahmed Almeleh aalmeleh.whatever.0101@gmail.com:
mostly through written tickets on bugzilla and fedora blocker bugs. Then we QA test different bugs to see if we get the same results.
there were none regarding server, as far as I know.
Let's wait and see what other members come up with.
On Sun, 31 Oct 2021 at 14:35, Peter Boy pboy@uni-bremen.de wrote: Dear Ahmed
Am 31.10.2021 um 13:39 schrieb Ahmed Almeleh aalmeleh.whatever.0101@gmail.com:
I didn't yet test the server variant of the x86_64 architecture of Fedora 35 RC 1.2 manually; however I have tested some of the base images for Fedora workstation 35 RC 1.1 for the MATE spin. All of which passed. Also I tested the "Everything netinst" for Fedora Workstation 35 RC 1.2 that passed as well. Majority of the tests are ran by Coconut, one of the bots. Mostly through openQA. If required I can test the server server variant of the x86_64 architecture of Fedora 35 RC 1.2 manually. I do think that manual testing is better as bots are able to sometimes become faulty.
many many thanks for your offering. I'm afraid it's too late now for testing the installation. F35 is go and Tuesday is release time. Now one can only hope. But it would be very helpful if we could count on your help for rawhide / F36.
I really don’t know what is best to do. I’m wondering (and worrying) how these issues came about. In fact, I don't even know what is tested per bot in detail. With workstations, there are likely to be hardware and driver peculiarities (different graphics adapters, proprietary wifi, etc.). With headless servers, many things do not even arise. How does the workstation WG organize all of that?
Thanks Peter
server mailing list -- server@lists.fedoraproject.org To unsubscribe send an email to server-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/server@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
On Sun, 31 Oct 2021 12:50:19 +0100, you wrote:
We are officially distributing a Fedora Server Edition disk image for ARM, typically used for SBCs. I just tried to install this image on a Radxa Rock Pi 4 and failed miserably. That SBC device is part of the installation procedure provided by the ARM team. With the F34 image I ended up with a black screen, no display, no chance to get the device working in the normal way (I got it to work in tricky detours). With F35 Beta I got a working display, but no network connection. I haven't found a solution for that yet.
Is the Radxa Rock Pi 4 supported by Fedora? I don't see it listed on the Fedora ARM wiki page.
The lack of standard support in the Linux kernel for a lot of the cheap ARM boards means options for running Fedora are limited.
Am 31.10.2021 um 18:11 schrieb Gerald Henriksen ghenriks@gmail.com:
Is the Radxa Rock Pi 4 supported by Fedora? I don't see it listed on the Fedora ARM wiki page.
Yes it is (it is ‚workable‘, but not ’supported’ in the terms of arm sig - ‚supported' are none but a very few boards). It's a Rock board and falls in the list into the category 'RockChips based devices'. The board is part of the uboot-images-armv8 package, too. And with the current kernel version, the PCIe interface is workable. So you get a nice and compact capable Fedora Server for special purposes (e.g. system monitoring via nagios or its alternatives).
The lack of standard support in the Linux kernel for a lot of the cheap ARM boards means options for running Fedora are limited.
That’s true. Unfortunately, most of the current powerful devices (and even Raspberry) use proprietary kernel extensions. But the arm crew is doing an excellent job to develop free replacements. And the board is not sooo cheap, by the way. It’s about the same as Pine64 RockPro.
* Peter Boy [31/10/2021 12:50] :
(1) Did any of us actually have Fedora Beta installed on a ARM device (SBC or other) using the image file and on which model? Or are we just flying blind with that deliverable?
I've obtained several SBCs (RPi, O-Droid C2, Khadus Vim) over the years, hoping to run Fedora Server on at least one of them but support for Fedora has either been spotty or not-existant.
If someone can point me towards a supported board, that would be great.
(3) And what about x86_64 installation? Most of us will be in possession of this hardware. But what about tests? When I started a discussion about testing a year ago, Adam reassured us, release building and testing is heavily automated these days. No need to worry too much about testing it manually ourselves. But did anyone of us at least worry a bit?
Watching from afar, I was under the impression that Fedora QA covers all our testing needs. If that isn't the case, I believe we need to host a Test Day during future release cycles.
Emmanuel
On Mon, Nov 1, 2021, 19:14 Emmanuel Seyman emmanuel@seyman.fr wrote:
- Peter Boy [31/10/2021 12:50] :
(1) Did any of us actually have Fedora Beta installed on a ARM device (SBC or other) using the image file and on which model? Or are we just flying blind with that deliverable?
I've obtained several SBCs (RPi, O-Droid C2, Khadus Vim) over the years, hoping to run Fedora Server on at least one of them but support for Fedora has either been spotty or not-existant.
If someone can point me towards a supported board, that would be great.
SBC support is almost always out of tree with vendor BCP releases.
(3) And what about x86_64 installation? Most of us will be in possession of this hardware. But what about tests? When I started a discussion about testing a year ago, Adam reassured us, release building and testing is heavily automated these days. No need to worry too much about testing it manually ourselves. But did anyone of us at least worry a bit?
Watching from afar, I was under the impression that Fedora QA covers all our testing needs. If that isn't the case, I believe we need to host a Test Day during future release cycles.
It's a community effort, pitch in by all means.
Fedora QA does yeomans work, but every use case or hardware is not covered.
Emmanuel _______________________________________________ server mailing list -- server@lists.fedoraproject.org To unsubscribe send an email to server-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/server@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
On Sun, 31 Oct 2021 at 07:50, Peter Boy pboy@uni-bremen.de wrote:
We are officially distributing a Fedora Server Edition disk image for ARM, typically used for SBCs. I just tried to install this image on a Radxa Rock Pi 4 and failed miserably. That SBC device is part of the installation procedure provided by the ARM team. With the F34 image I ended up with a black screen, no display, no chance to get the device working in the normal way (I got it to work in tricky detours). With F35 Beta I got a working display, but no network connection. I haven't found a solution for that yet.
And then I remembered F33, where it was discovered by chance (!) at the last minute, that in the x86_64 version Cockpit did not work properly (and probably also in the aarch64 default install version, we didn't and probably couldn’t even check that afterwards). We have not solved the underlying issue for this until today, although the factual reason has been clear for a long time (issue #32).
Similar problems had emerged after the release with the switch to systemd-resolved, where "surprisingly" problems with some server configurations were encountered, especially with elaborate use of KVM virtualization on a server.
These events are clear violations of our release criteria. In principle, a release must be at least successfully installable and basically working.
For me, this brings up a few questions:
(1) Did any of us actually have Fedora Beta installed on a ARM device (SBC or other) using the image file and on which model? Or are we just flying blind with that deliverable?
(2) What about the two aarch64 install iso files that target SBSA hardware? Does anyone have such a beast at hand and tested the isos? Or is that (also) flying blind?
(3) And what about x86_64 installation? Most of us will be in possession of this hardware. But what about tests? When I started a discussion about testing a year ago, Adam reassured us, release building and testing is heavily automated these days. No need to worry too much about testing it manually ourselves. But did anyone of us at least worry a bit?
I would like to discuss how we can gain some improvement.
Regarding arch architecture, could some kind of cooperation with arm group be beneficial? And what can we offer? Is it testing? Is it documentation? Or something else? If we want to take SBC devices seriously, perhaps we could identify a few reference systems where we can actually ensure, i.e. test, whether a successful installation and basic functionality is met. If we don't want that, we should not distribute the image file under the Fedora Server Edition branding.
The issue is that SBC devices vary a lot even when they are the same 'brand'. Many of these boards are made in small shipments and so you find that StonePi-X board 1 uses a slightly different chipset than StonePi-X board 4309. This makes bug checking on them really hard because 'it works for me' but 'may not work for you'. Larger build boards like the raspberry-pi4 have different issues that the broadcom chipset is not fully open and so a lot of reverse engineering is needed. Thus they usually all require various post-install steps to make the system 'fully functional'. We can probably at best say we support 'ARM system-ready(TM)' systems without post-install but would need to focus with the Fedora ARM group on outlining all the steps for other boards.
And in terms of our "main" architecture, we should agree on what QA actually expects in terms of non-automated testing and how we want to tackle that in the next round. Based on Adam's comments, we have relied heavily on automation for the last few releases. So it's not clear to me what, for example, the page https://fedoraproject.org/wiki/Test_Results:Fedora_35_RC_1.2_Server means in detail in terms of work.
Basically the way to supplement automated testing is to call specific "Test Days" where a specific matrix for testing a specific tool or set of architectures is done.
Happy Sunday Peter
server mailing list -- server@lists.fedoraproject.org To unsubscribe send an email to server-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/server@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
(3) And what about x86_64 installation? Most of us will be in possession of this hardware. But what about tests? When I started a discussion about testing a year ago, Adam reassured us, release building and testing is heavily automated these days. No need to worry too much about testing it manually ourselves. But did anyone of us at least worry a bit?
I ran some of the validation tests for Fedora Server on the Beta and RC candidates (in particular, the Cockpit and Active Directory domain-joining tests).
I'll try to summerize our discussion so far.
(1) Many of us check / try out betas and RCs according to their respective relevancies. So we don't completly "fly blind" or rely exclusively on automated testing (to my relief). For me its virtualisation, PostgreSQL, Web service; for Stephen it may be Cockpit, Active Diretory. I suppose everybody has some areas specifically interested in. Can we gather our special interests and systematize it a little bit and make sure that key areas are not just automatically tested?
(2) Additionally, we should have a look on our technical specifications and our release critera (Stephen Gallagher proposed that some times ago) and check automatic testing is OK and where manual testing makes sense, along with a general revisit as Stephen proposed.
(3) Regarding ARM SBCs several of us have some experiences with it and own one or the other device. I suggest we gather our knowledge and create a set of reference devices that we know basically work with Server (i.e. installable, bootable, network connection, storage, text terminal / keyboard). IoT provides such a list (in a very short form, a bit more detailed information would be desirable).
(4) There seems to be agreement that manual testing is useful / necessary for specific more complex issues, possibly specific for individual changes in a release. Examples would be the change to systemd-resolved or the modularization of libvirt, which led to 'surprises'. We should plan as a routine, around the time the first beta is completed, to go through the list of release changes and check for potential issues and manual testing requirement (and determine who does what).
QUESTION: Do we want to use our next IRC meeting to start a discussion about our technical specifications and release criteria? (Feedback greatly appreciated)
For some topics worthwhile discussing see mail from Stephen cf. https://lists.fedoraproject.org/archives/list/server@lists.fedoraproject.org...
Am 02.11.2021 um 15:33 schrieb Stephen Gallagher sgallagh@redhat.com:
(3) And what about x86_64 installation? Most of us will be in possession of this hardware. But what about tests? When I started a discussion about testing a year ago, Adam reassured us, release building and testing is heavily automated these days. No need to worry too much about testing it manually ourselves. But did anyone of us at least worry a bit?
I ran some of the validation tests for Fedora Server on the Beta and RC candidates (in particular, the Cockpit and Active Directory domain-joining tests).
server@lists.fedoraproject.org