As with the latest release candidate (rc 1.5) from today we have still an issue with the ARM SBC / RPi image.
* The RPi image needs a nomodeset fix on the arm-imange-installer comandline, otherwise it boots with a black screen
* Afterwards you can’t manage the storage because LVM is broken.
* All my Rockchip based devices (RockPro64, Rock Pi 4, Roc-3399-pc) can’t boot at all, because the broken LVM prevents the U-Boot files to get copied onto the SD card. The same will be true for all other devices besides RPi, too, that need U-Boot files to be able to boot
See details in the bug reports: https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=2246871 https://bugzilla.redhat.com/show_bug.cgi?id=2244305
There are ways around that
* You can fix that by manually deleting the content of /etc/lvm/devices, which is currently not documented anywhere and probably not a location, an admin should mess with. An RPi would then work as far as I know. For all other devices this doesn’t work without additional steps.
* You get the other devices booting which have a workable *and* compatible SPI U-Boot installed and can substitute the U-Boot files otherwise be copied to the SD card by arm-image-installer. And then use the before mentioned fix.
* Fedora provides for some devices a SPI module which should work (I didn’t test that). But it is a real challenge to install those the Fedora way (https://nullr0ute.com/2021/05/fedora-on-the-pinebook-pro/), and a real adventure if you have a non-US keyboard.
* Some devices are supported by non-Fedora software, e.g. tow-boot (https://tow-boot.org https://tow-boot.org/). You can install the SPI very comfortable and the current version can also boot the Fedora image without Fedora U-Boot files.
In the latter 3 cases you would additionally have to fix the LVM image.
The question is, how to proceed with that situation.
a) I suggested to make the bug a release blocker.
b) Others objected, because there is a workaround available.
In case of b)
Do we really want to publish a broken image along with the other media as a regular release?
Alternatively, we might
- Publish it separately well below our regular media as kind of "technical sneak preview“
- Skip F39 for ARM SBC and continue to distribute the F38 image
- rely to third party software, e.g. tow-boot for devices it supports
- Other ideas?
We should discuss this today at the meeting.
-- Peter Boy https://fedoraproject.org/wiki/User:Pboy PBoy@fedoraproject.org
Timezone: CET (UTC+1) / CEST (UTC+2)
Fedora Server Edition Working Group member Fedora Docs team contributor and board member Java developer and enthusiast
On Wed, 2023-11-01 at 13:33 +0100, Peter Boy wrote:
As with the latest release candidate (rc 1.5) from today we have still an issue with the ARM SBC / RPi image.
The RPi image needs a nomodeset fix on the arm-imange-installer comandline, otherwise it boots with a black screen
Afterwards you can’t manage the storage because LVM is broken.
All my Rockchip based devices (RockPro64, Rock Pi 4, Roc-3399-pc) can’t boot at all, because the broken LVM prevents the U-Boot files to get copied onto the SD card. The same will be true for all other devices besides RPi, too, that need U-Boot files to be able to boot
Other testing does not confirm this. Both Peter Robinson and Paul Whalen say they have booted Server images successfully on non-Pi hardware. So has Geoff - https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=2246871#c34 .
We are trying to figure out exactly what problem(s) you are seeing, but in general it does not seem to be accurate to say that Server will not boot on non-Pi hardware.
Am 02.11.2023 um 07:26 schrieb Adam Williamson adamwill@fedoraproject.org:
On Wed, 2023-11-01 at 13:33 +0100, Peter Boy wrote:
As with the latest release candidate (rc 1.5) from today we have still an issue with the ARM SBC / RPi image.
The RPi image needs a nomodeset fix on the arm-imange-installer comandline, otherwise it boots with a black screen
Afterwards you can’t manage the storage because LVM is broken.
All my Rockchip based devices (RockPro64, Rock Pi 4, Roc-3399-pc) can’t boot at all, because the broken LVM prevents the U-Boot files to get copied onto the SD card. The same will be true for all other devices besides RPi, too, that need U-Boot files to be able to boot
Other testing does not confirm this. Both Peter Robinson and Paul Whalen say they have booted Server images successfully on non-Pi hardware. So has Geoff - https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=2246871#c34 .
Well, we have 4 non-PRi devices where we know, that an installation using the Fedora arm-image-installer tool fails. And we can exactly name these devices: Radxa Rock-Pi 4a, Radxa Rock-Pi 4b+, Pine64 RockPro64, LibreComputer Roc-3399-pc and the installation environment (Server) and installation procedure (Server installation guide).
Moreover, we exactly know, *why* they fail. It was Paul who found the reason. The 1.5 image (as well as the previous ones) contain a file /etc/lvm/devices/system.devices with a wrong specification of the LVM volume inside the rc 1.5 disk image.
And we exactly know how to reproduce the bug. The installation/test environment is Fedora server rc 1.5
There are some devices, which don’t fail. We don’t know, if this is due to some unknown technical differences in the devices or due to differences in the test/istallation environment. Unfortunately, we cannot perform cross-testing. May be, the tool, that the installation script uses, acts differently if you use it on a Fedora Server system with active LVM or on Fedora Workstation without LVM. We don’t know at the moment.
But we know, if we distribute rc 1.5 as is, it will definitely fail for an unknown number of users!
And, by the way, we know, or at least we have a strong hypothesis based on the known technical properties, how to fix it for those unknown number of users: create an image without that wrong content in /etc/lvm/devices.
We are trying to figure out exactly what problem(s) you are seeing,
As I understand the discussion, we are non trying anymore but want to release immediately.
but in general it does not seem to be accurate to say that Server will not boot on non-Pi hardware.
Sorry, that is the optimistic and wishful thinking driven view of the situation. In fact, we don’t know if it „in general“ fails or if it „in general" succeeds. If if fails for non-rpi devices, you have no way to delete that file at all, at least not with Fedora provided tools. There is no way to install it with common Fedora installation and administration tools.
I think it is understandable not to delay the release any further. The SBC devices are probably not relevant enough for Server.
But to intentionally distribute an installation image that we know will cause for an unknown number of users, and especially Fedora Server users, an unrecoverable abort of the installation, and then to knowingly respond with misleading "common bugs“ (delete that mentioned file), that is most certainly wrong as well. And especially it is bad marketing for Fedora Server. Fedora (Server) becomes the laughing stock of the community.
So let’s release Fedora 39 as it is, but without the Server installation media for ARM SBC. Let’s continue to offer F38 for download. Or, if that is absolutely not possible, let the Server ARM SBC installation media completely off, at least from the Server download page. That’s not desirable, but still better than providing failing installation media.
-- Peter Boy https://fedoraproject.org/wiki/User:Pboy PBoy@fedoraproject.org
Timezone: CET (UTC+1) / CEST (UTC+2)
Fedora Server Edition Working Group member Fedora Docs team contributor and board member Java developer and enthusiast
Some additional information: ============================
While writing the email below I got the idea to modify the installation environment. So I booted a Workstation live CD on my test box, and surprise, surprise the arm-image-installer worked and created a bootable mSD card for all my SBCs.
It is ironic that you cannot install a server edition using a server edition. But is the smallest problem, I guess. At least we have a serious solution for the failing arm-image-installer.
But, the software is rather fragile. The first try worked, but the rock-pi-4 booted into a black screen. In a second try to add the nomodeset flag, arm-image-installer failed again with the LVM issue. Before a third try I zeroed the first 2 GiBs of the mSD card and it worked again, although complaining again about some LVM issues.
All SBCs ran into the /etc/lvm/devices issue. I analyze that issue now further.
Am 02.11.2023 um 10:10 schrieb Peter Boy pboy@uni-bremen.de:
Am 02.11.2023 um 07:26 schrieb Adam Williamson adamwill@fedoraproject.org:
On Wed, 2023-11-01 at 13:33 +0100, Peter Boy wrote:
As with the latest release candidate (rc 1.5) from today we have still an issue with the ARM SBC / RPi image.
The RPi image needs a nomodeset fix on the arm-imange-installer comandline, otherwise it boots with a black screen
Afterwards you can’t manage the storage because LVM is broken.
All my Rockchip based devices (RockPro64, Rock Pi 4, Roc-3399-pc) can’t boot at all, because the broken LVM prevents the U-Boot files to get copied onto the SD card. The same will be true for all other devices besides RPi, too, that need U-Boot files to be able to boot
Other testing does not confirm this. Both Peter Robinson and Paul Whalen say they have booted Server images successfully on non-Pi hardware. So has Geoff - https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=2246871#c34 .
Well, we have 4 non-PRi devices where we know, that an installation using the Fedora arm-image-installer tool fails. And we can exactly name these devices: Radxa Rock-Pi 4a, Radxa Rock-Pi 4b+, Pine64 RockPro64, LibreComputer Roc-3399-pc and the installation environment (Server) and installation procedure (Server installation guide).
Moreover, we exactly know, *why* they fail. It was Paul who found the reason. The 1.5 image (as well as the previous ones) contain a file /etc/lvm/devices/system.devices with a wrong specification of the LVM volume inside the rc 1.5 disk image.
And we exactly know how to reproduce the bug. The installation/test environment is Fedora server rc 1.5
There are some devices, which don’t fail. We don’t know, if this is due to some unknown technical differences in the devices or due to differences in the test/istallation environment. Unfortunately, we cannot perform cross-testing. May be, the tool, that the installation script uses, acts differently if you use it on a Fedora Server system with active LVM or on Fedora Workstation without LVM. We don’t know at the moment.
But we know, if we distribute rc 1.5 as is, it will definitely fail for an unknown number of users!
And, by the way, we know, or at least we have a strong hypothesis based on the known technical properties, how to fix it for those unknown number of users: create an image without that wrong content in /etc/lvm/devices.
We are trying to figure out exactly what problem(s) you are seeing,
As I understand the discussion, we are non trying anymore but want to release immediately.
but in general it does not seem to be accurate to say that Server will not boot on non-Pi hardware.
Sorry, that is the optimistic and wishful thinking driven view of the situation. In fact, we don’t know if it „in general“ fails or if it „in general" succeeds. If if fails for non-rpi devices, you have no way to delete that file at all, at least not with Fedora provided tools. There is no way to install it with common Fedora installation and administration tools.
I think it is understandable not to delay the release any further. The SBC devices are probably not relevant enough for Server.
But to intentionally distribute an installation image that we know will cause for an unknown number of users, and especially Fedora Server users, an unrecoverable abort of the installation, and then to knowingly respond with misleading "common bugs“ (delete that mentioned file), that is most certainly wrong as well. And especially it is bad marketing for Fedora Server. Fedora (Server) becomes the laughing stock of the community.
So let’s release Fedora 39 as it is, but without the Server installation media for ARM SBC. Let’s continue to offer F38 for download. Or, if that is absolutely not possible, let the Server ARM SBC installation media completely off, at least from the Server download page. That’s not desirable, but still better than providing failing installation media.
-- Peter Boy https://fedoraproject.org/wiki/User:Pboy PBoy@fedoraproject.org
Timezone: CET (UTC+1) / CEST (UTC+2)
Fedora Server Edition Working Group member Fedora Docs team contributor and board member Java developer and enthusiast
server mailing list -- server@lists.fedoraproject.org To unsubscribe send an email to server-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/server@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
-- Peter Boy https://fedoraproject.org/wiki/User:Pboy PBoy@fedoraproject.org
Timezone: CET (UTC+1) / CEST (UTC+2)
Fedora Server Edition Working Group member Fedora Docs team contributor and board member Java developer and enthusiast
On Thu, 2023-11-02 at 10:10 +0100, Peter Boy wrote:
Am 02.11.2023 um 07:26 schrieb Adam Williamson adamwill@fedoraproject.org:
On Wed, 2023-11-01 at 13:33 +0100, Peter Boy wrote:
As with the latest release candidate (rc 1.5) from today we have still an issue with the ARM SBC / RPi image.
The RPi image needs a nomodeset fix on the arm-imange-installer comandline, otherwise it boots with a black screen
Afterwards you can’t manage the storage because LVM is broken.
All my Rockchip based devices (RockPro64, Rock Pi 4, Roc-3399-pc) can’t boot at all, because the broken LVM prevents the U-Boot files to get copied onto the SD card. The same will be true for all other devices besides RPi, too, that need U-Boot files to be able to boot
Other testing does not confirm this. Both Peter Robinson and Paul Whalen say they have booted Server images successfully on non-Pi hardware. So has Geoff - https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=2246871#c34 .
Well, we have 4 non-PRi devices where we know, that an installation using the Fedora arm-image-installer tool fails. And we can exactly name these devices: Radxa Rock-Pi 4a, Radxa Rock-Pi 4b+, Pine64 RockPro64, LibreComputer Roc-3399-pc and the installation environment (Server) and installation procedure (Server installation guide).
Moreover, we exactly know, *why* they fail. It was Paul who found the reason. The 1.5 image (as well as the previous ones) contain a file /etc/lvm/devices/system.devices with a wrong specification of the LVM volume inside the rc 1.5 disk image.
Yes, you have outlined that theory several times, but Peter and Paul just don't agree with it, and testing so for does not back it up. IIRC, Paul tested on one of the devices you claimed would be affected, and it boots fine for him.
They all agree there is an issue with the LVM device configuration file, but they do *not* agree that it prevents devices from booting. (For one thing, that's an obvious chicken-and-egg problem with that theory: in order to access a file under /etc you need to have already successfully identified and mounted the root filesystem, at which point, why would the system not boot?)
And we exactly know how to reproduce the bug. The installation/test environment is Fedora server rc 1.5
We don't. As I said, three people have *tried* to reproduce it, and have *not* reproduced it as you describe. Their systems boot fine. You can't say we "exactly know how to reproduce the bug" until someone other than yourself has actually reproduced it.
server@lists.fedoraproject.org
-
Adam Williamson -
Peter Boy