Cuirrent state of F39 Fedora Server Edition for ARM SBCs / RaspberryPi

[Peter Boy]

As with the latest release candidate (rc 1.5) from today we have still an issue with the
ARM SBC / RPi image. 

* The RPi image needs a nomodeset fix on the arm-imange-installer comandline, otherwise it
boots with a black screen

* Afterwards you can’t manage the storage because LVM is broken. 

* All my Rockchip based devices (RockPro64, Rock Pi 4, Roc-3399-pc) can’t boot at all,
because the broken LVM prevents the U-Boot files to get copied onto the SD card. The same
will be true for all other devices besides  RPi, too, that need U-Boot files to be able to
boot


See details in the bug reports:
https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=2246871 
https://bugzilla.redhat.com/show_bug.cgi?id=2244305 



There are ways around that

* You can fix that by manually deleting the content of /etc/lvm/devices, which is
currently not documented anywhere and probably not a location, an admin should mess with.
An RPi would then work as far as I know. For all other devices this doesn’t work without
additional steps. 

* You get the other devices booting which have a workable *and* compatible SPI U-Boot
installed and can substitute the U-Boot files otherwise be copied to the SD card by
arm-image-installer. And then use the before mentioned fix.

* Fedora provides for some devices a SPI module which should work (I didn’t test that).
But it is a real challenge to install those the Fedora way
(https://nullr0ute.com/2021/05/fedora-on-the-pinebook-pro/), and a real adventure if you
have a non-US keyboard.

* Some devices are supported by non-Fedora software, e.g. tow-boot (https://tow-boot.org
<https://tow-boot.org/>). You can install the SPI very comfortable and the current
version can also boot the Fedora image without Fedora U-Boot files.

In the latter 3 cases you would additionally have to fix the LVM image.




The question is, how to proceed with that situation.

a) I suggested to make the bug a release blocker.

b) Others objected, because there is a workaround available.


In case of b)

Do we really want to publish a broken image along with the other media as a regular
release?

Alternatively, we might

- Publish it separately well below our regular media as kind of "technical sneak
preview“

- Skip F39 for ARM SBC and continue to distribute the F38 image

- rely to third party software, e.g. tow-boot for devices it supports

- Other ideas?


We should discuss this today at the meeting.











--
Peter Boy
https://fedoraproject.org/wiki/User:Pboy
PBoy(a)fedoraproject.org

Timezone: CET (UTC+1) / CEST (UTC+2)

Fedora Server Edition Working Group member
Fedora Docs team contributor and board member
Java developer and enthusiast

[Adam Williamson]

On Wed, 2023-11-01 at 13:33 +0100, Peter Boy wrote:
>  As with the latest release candidate (rc 1.5) from today we have
> still an issue with the ARM SBC / RPi image. 
>  
>  * The RPi image needs a nomodeset fix on the arm-imange-installer comandline, otherwise
> it boots with a black screen
>  
>  * Afterwards you can’t manage the storage because LVM is broken. 
>  
>  * All my Rockchip based devices (RockPro64, Rock Pi 4, Roc-3399-pc) can’t boot at all,
> because the broken LVM prevents the U-Boot files to get copied onto the SD card. The same
> will be true for all other devices besides  RPi, too, that need U-Boot files to be able to
> boot 
Other testing does not confirm this. Both Peter Robinson and Paul
Whalen say they have booted Server images successfully on non-Pi
hardware. So has Geoff -
https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=2246871#c34 .

We are trying to figure out exactly what problem(s) you are seeing, but
in general it does not seem to be accurate to say that Server will not
boot on non-Pi hardware.
-- 
Adam Williamson (he/him/his)
Fedora QA
Fedora Chat: @adamwill:fedora.im | Mastodon: @adamw(a)fosstodon.org
https://www.happyassassin.net

[Peter Boy]

>  Am 02.11.2023 um 07:26 schrieb Adam Williamson
> <adamwill(a)fedoraproject.org&gt;:
>  
>  On Wed, 2023-11-01 at 13:33 +0100, Peter Boy wrote:
> > As with the latest release candidate (rc 1.5) from today we have still an issue with
> the ARM SBC / RPi image. 
> > 
> > * The RPi image needs a nomodeset fix on the arm-imange-installer comandline,
> otherwise it boots with a black screen
> > 
> > * Afterwards you can’t manage the storage because LVM is broken. 
> > 
> > * All my Rockchip based devices (RockPro64, Rock Pi 4, Roc-3399-pc) can’t boot at
> all, because the broken LVM prevents the U-Boot files to get copied onto the SD card. The
> same will be true for all other devices besides  RPi, too, that need U-Boot files to be
> able to boot
>  
>  Other testing does not confirm this. Both Peter Robinson and Paul
>  Whalen say they have booted Server images successfully on non-Pi
>  hardware. So has Geoff -
>  https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=2246871#c34 . 
Well, we have 4 non-PRi devices where we know, that an installation using the Fedora
arm-image-installer tool fails. And we can exactly name these devices: Radxa Rock-Pi 4a,
Radxa Rock-Pi 4b+, Pine64 RockPro64, LibreComputer Roc-3399-pc and the installation
environment (Server) and installation procedure (Server installation guide).

Moreover, we exactly know, *why* they fail. It was Paul who found the reason. The 1.5
image (as well as the previous ones) contain a file /etc/lvm/devices/system.devices  with
a wrong specification of the LVM volume inside the rc 1.5 disk image. 

And we exactly know how to reproduce the bug. The installation/test environment is Fedora
server rc 1.5

There are some devices, which don’t fail. We don’t know, if this is due to some unknown
technical differences in the devices or due to differences in the test/istallation
environment. Unfortunately, we cannot perform cross-testing.   May be, the tool, that the
installation script uses, acts differently if you use it on a Fedora Server system with
active LVM or on Fedora Workstation without LVM. We don’t know at the moment. 

But we know, if we distribute rc 1.5 as is, it will definitely fail for an unknown number
of users!

And, by the way, we know, or at least we have a strong hypothesis based on the known
technical properties, how to fix it for those unknown number of users: create an image
without that wrong content in /etc/lvm/devices.


>  We are trying to figure out exactly what problem(s) you are seeing,
As I understand the discussion, we are non trying anymore but want to release
immediately.

>  but
>  in general it does not seem to be accurate to say that Server will not
>  boot on non-Pi hardware. 
Sorry, that is the optimistic and wishful thinking driven view of the situation. In fact,
we don’t know if it „in general“ fails or if it „in general" succeeds. If if fails
for non-rpi devices, you have no way to delete that file at all, at least not with Fedora
provided tools. There is no way to install it with common Fedora installation and
administration tools.

I think it is understandable not to delay the release any further. The SBC devices are
probably not relevant enough for Server. 

But to intentionally distribute an installation image that we know will cause for an
unknown number of users, and especially Fedora Server users, an unrecoverable abort of the
installation, and then to knowingly respond with misleading "common bugs“ (delete
that mentioned file), that is most certainly wrong as well.  And especially it is bad
marketing for Fedora Server. Fedora (Server) becomes the laughing stock of the community.


So let’s release Fedora 39 as it is, but without the Server installation media for ARM
SBC. Let’s continue to offer F38 for download. Or, if that is absolutely not possible, let
the Server ARM SBC installation media completely off, at least from the Server download
page. That’s not desirable, but still better than providing failing installation media.



--
Peter Boy
https://fedoraproject.org/wiki/User:Pboy
PBoy(a)fedoraproject.org

Timezone: CET (UTC+1) / CEST (UTC+2)

Fedora Server Edition Working Group member
Fedora Docs team contributor and board member
Java developer and enthusiast

[Peter Boy]

Some additional information:
============================

While writing the email below I got the idea to modify the installation environment. So I
booted a Workstation live CD on my test box, and surprise, surprise the
arm-image-installer worked and created a bootable mSD card for all my SBCs. 

It is ironic that you cannot install a server edition using a server edition. But is the
smallest problem, I guess. At least we have a serious solution for the failing
arm-image-installer.


But, the software is rather fragile. The first try worked, but the rock-pi-4 booted into a
black screen. In a second try to add the nomodeset flag, arm-image-installer failed again
with the LVM issue. Before a third try I zeroed the first 2 GiBs of the mSD card and it
worked again, although complaining again about some LVM issues. 


All SBCs ran into the /etc/lvm/devices issue. I analyze that issue now further.






>  Am 02.11.2023 um 10:10 schrieb Peter Boy <pboy(a)uni-bremen.de&gt;:
>  
>  
>  
> > Am 02.11.2023 um 07:26 schrieb Adam Williamson <adamwill(a)fedoraproject.org&gt;:
> > 
> > On Wed, 2023-11-01 at 13:33 +0100, Peter Boy wrote:
> >> As with the latest release candidate (rc 1.5) from today we have still an issue
> with the ARM SBC / RPi image. 
> >> 
> >> * The RPi image needs a nomodeset fix on the arm-imange-installer comandline,
> otherwise it boots with a black screen
> >> 
> >> * Afterwards you can’t manage the storage because LVM is broken. 
> >> 
> >> * All my Rockchip based devices (RockPro64, Rock Pi 4, Roc-3399-pc) can’t boot at
> all, because the broken LVM prevents the U-Boot files to get copied onto the SD card. The
> same will be true for all other devices besides  RPi, too, that need U-Boot files to be
> able to boot
> > 
> > Other testing does not confirm this. Both Peter Robinson and Paul
> > Whalen say they have booted Server images successfully on non-Pi
> > hardware. So has Geoff -
> > https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=2246871#c34 .
>  
>  Well, we have 4 non-PRi devices where we know, that an installation using the Fedora
> arm-image-installer tool fails. And we can exactly name these devices: Radxa Rock-Pi 4a,
> Radxa Rock-Pi 4b+, Pine64 RockPro64, LibreComputer Roc-3399-pc and the installation
> environment (Server) and installation procedure (Server installation guide).
>  
>  Moreover, we exactly know, *why* they fail. It was Paul who found the reason. The 1.5
> image (as well as the previous ones) contain a file /etc/lvm/devices/system.devices  with
> a wrong specification of the LVM volume inside the rc 1.5 disk image. 
>  
>  And we exactly know how to reproduce the bug. The installation/test environment is Fedora
> server rc 1.5
>  
>  There are some devices, which don’t fail. We don’t know, if this is due to some unknown
> technical differences in the devices or due to differences in the test/istallation
> environment. Unfortunately, we cannot perform cross-testing.   May be, the tool, that the
> installation script uses, acts differently if you use it on a Fedora Server system with
> active LVM or on Fedora Workstation without LVM. We don’t know at the moment. 
>  
>  But we know, if we distribute rc 1.5 as is, it will definitely fail for an unknown number
> of users!
>  
>  And, by the way, we know, or at least we have a strong hypothesis based on the known
> technical properties, how to fix it for those unknown number of users: create an image
> without that wrong content in /etc/lvm/devices.
>  
>  
> > We are trying to figure out exactly what problem(s) you are seeing,
>  
>  As I understand the discussion, we are non trying anymore but want to release
> immediately.
>  
> > but
> > in general it does not seem to be accurate to say that Server will not
> > boot on non-Pi hardware.
>  
>  Sorry, that is the optimistic and wishful thinking driven view of the situation. In fact,
> we don’t know if it „in general“ fails or if it „in general" succeeds. If if fails
> for non-rpi devices, you have no way to delete that file at all, at least not with Fedora
> provided tools. There is no way to install it with common Fedora installation and
> administration tools.
>  
>  I think it is understandable not to delay the release any further. The SBC devices are
> probably not relevant enough for Server. 
>  
>  But to intentionally distribute an installation image that we know will cause for an
> unknown number of users, and especially Fedora Server users, an unrecoverable abort of the
> installation, and then to knowingly respond with misleading "common bugs“ (delete
> that mentioned file), that is most certainly wrong as well.  And especially it is bad
> marketing for Fedora Server. Fedora (Server) becomes the laughing stock of the community.
>
>  
>  So let’s release Fedora 39 as it is, but without the Server installation media for ARM
> SBC. Let’s continue to offer F38 for download. Or, if that is absolutely not possible, let
> the Server ARM SBC installation media completely off, at least from the Server download
> page. That’s not desirable, but still better than providing failing installation media.
>  
>  
>  
>  --
>  Peter Boy
>  https://fedoraproject.org/wiki/User:Pboy
>  PBoy(a)fedoraproject.org
>  
>  Timezone: CET (UTC+1) / CEST (UTC+2)
>  
>  Fedora Server Edition Working Group member
>  Fedora Docs team contributor and board member
>  Java developer and enthusiast
>  
>  
>  
>  _______________________________________________
>  server mailing list -- server(a)lists.fedoraproject.org
>  To unsubscribe send an email to server-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]
>  Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue

--
Peter Boy
https://fedoraproject.org/wiki/User:Pboy
PBoy(a)fedoraproject.org

Timezone: CET (UTC+1) / CEST (UTC+2)

Fedora Server Edition Working Group member
Fedora Docs team contributor and board member
Java developer and enthusiast

[Adam Williamson]

On Thu, 2023-11-02 at 10:10 +0100, Peter Boy wrote:
>  
>  > Am 02.11.2023 um 07:26 schrieb Adam Williamson <adamwill(a)fedoraproject.org&gt;:
>  > 
>  > On Wed, 2023-11-01 at 13:33 +0100, Peter Boy wrote:
>  > > As with the latest release candidate (rc 1.5) from today we have still an issue
> with the ARM SBC / RPi image. 
>  > > 
>  > > * The RPi image needs a nomodeset fix on the arm-imange-installer comandline,
> otherwise it boots with a black screen
>  > > 
>  > > * Afterwards you can’t manage the storage because LVM is broken. 
>  > > 
>  > > * All my Rockchip based devices (RockPro64, Rock Pi 4, Roc-3399-pc) can’t boot
> at all, because the broken LVM prevents the U-Boot files to get copied onto the SD card.
> The same will be true for all other devices besides  RPi, too, that need U-Boot files to
> be able to boot
>  > 
>  > Other testing does not confirm this. Both Peter Robinson and Paul
>  > Whalen say they have booted Server images successfully on non-Pi
>  > hardware. So has Geoff -
>  > https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=2246871#c34 .
>  
>  Well, we have 4 non-PRi devices where we know, that an installation using the Fedora
> arm-image-installer tool fails. And we can exactly name these devices: Radxa Rock-Pi 4a,
> Radxa Rock-Pi 4b+, Pine64 RockPro64, LibreComputer Roc-3399-pc and the installation
> environment (Server) and installation procedure (Server installation guide).
>  
>  Moreover, we exactly know, *why* they fail. It was Paul who found the reason. The 1.5
> image (as well as the previous ones) contain a file /etc/lvm/devices/system.devices  with
> a wrong specification of the LVM volume inside the rc 1.5 disk image.  
Yes, you have outlined that theory several times, but Peter and Paul
just don't agree with it, and testing so for does not back it up. IIRC,
Paul tested on one of the devices you claimed would be affected, and it
boots fine for him.

They all agree there is an issue with the LVM device configuration
file, but they do *not* agree that it prevents devices from booting.
(For one thing, that's an obvious chicken-and-egg problem with that
theory: in order to access a file under /etc you need to have already
successfully identified and mounted the root filesystem, at which
point, why would the system not boot?)
>  
>  And we exactly know how to reproduce the bug. The installation/test environment is Fedora
> server rc 1.5 
We don't. As I said, three people have *tried* to reproduce it, and
have *not* reproduced it as you describe. Their systems boot fine. You
can't say we "exactly know how to reproduce the bug" until someone
other than yourself has actually reproduced it.

-- 
Adam Williamson (he/him/his)
Fedora QA
Fedora Chat: @adamwill:fedora.im | Mastodon: @adamw(a)fosstodon.org
https://www.happyassassin.net