Fwd: [Call for Action] Help Testing F39 ARM

[Peter Robinson]

For arm testers too

---------- Forwarded message ---------
From: Geoffrey Marr 
Date: Thu, Oct 26, 2023 at 5:05 AM
Subject: [Call for Action] Help Testing F39 ARM
To: Fedora discussions about the Internet of Things

Hi IoT Testers,

Tomorrow will be the Fedora 39 Go/No-Go (26 October 2023). Currently,
there are several Raspberry Pi-related blockers that will be up for
consideration at the meeting. There is a plan to attempt to get
sufficient testing on an RC that was spun up several hours ago, and
then vote to ship this as F39 with a notice that RPi is not supported
at launch and that users should not upgrade from F38 until the issue
is fixed. Because of the widespread use of RPi hardware in IoT and ARM
testing however, and these bugs affecting the ability of users to test
with the RPi, the F39 ARM version specifically is lacking in test
results.

I am reaching out to the IoT list in the hope that some of you might
have ARM hardware *other than* RPi that could be used to help fill in
the test matrix [0]. This could be the Jetson Nano, Odroid XU4, or any
other ARM board that can boot F39 ARM and help with testing. You can
find a list of other supported hardware here: [1].

This approach is a bit unorthodox, but the commonalities between
Fedora IoT and Fedora ARM are such that some of the IoT users might
have ARM hardware laying around that could be used for testing. Your
test results will help us ship F39 without slipping again!

Thanks everyone!

Geoff Marr
IRC: coremodule

[0] https://fedoraproject.org/wiki/Test_Results:Fedora_39_RC_1.2_Summary
[1] https://fedoraproject.org/wiki/Architectures/ARM#Supported_Hardware_and_Devi...
_______________________________________________
IoT mailing list -- [e-mail]
To unsubscribe send an email to [e-mail]
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/[e-mail]
Do not reply to spam, report it:
https://pagure.io/fedora-infrastructure/new_issue

[Call for Action] Help Testing F39 ARM

[Peter Boy]

> Am 26.10.2023 um 13:36 schrieb Peter Robinson :
>
> For arm testers too

I just tested RC 1.2 with the server image.  It’s rather broken and I don’t think we should release it.

1. (minor)

With arm-installer you get an error message:
 To: /dev/mmcblk0 ....
dd: warning: partial read (8192 bytes); suggest iflag=fullblock

If you add this (w and w/o leading —-) it doesn’t start and you get a usage message listing all available parameters and that iflag isn’t among them.

(I didn’t add it to the dd command line, most users wouldn’t do that, I suppose)

2. (major)

At the end of arm-installer you get a message:

= Writing image complete!
mount: /tmp/root: unknown filesystem type 'LVM2_member'.
       dmesg(1) may have more information after failed mount system call.
sed: /tmp/fw/EFI/fedora/grubenv kann nicht gelesen werden: Datei oder Verzeichnis nicht gefunden
= No U-Boot files found for rock-pi-4-rk3399.

= Installation Complete! Insert into the rock-pi-4-rk3399 and boot.

(a)
You can’t create a usable image without U-Boot files, of course and the final message is badly misleading.

(b)
We had this (or a similar looking) error briefly in F38 as well. The U-Boot files  are perfectly where they belong. The reason was that several lines before copying the U-Boot files, the paths were changed specifically for IOT, but these changes were then used for all other editions, too, when copying the U-Boot files, instead of only for IoT.

So, it’s quite easy to fix, I suppose. But we need another RC, I think.

I got that for all of my SBCs besides RPi4:

= Writing image complete!
mount: /tmp/root: unknown filesystem type 'LVM2_member'.
       dmesg(1) may have more information after failed mount system call.
sed: /tmp/fw/EFI/fedora/grubenv kann nicht gelesen werden: Datei oder Verzeichnis nicht gefunden
= Raspberry Pi 4 Uboot is already in place, no changes needed.

= Installation Complete! Insert into the rpi4 and boot.

But it didn’t boot either.

--
Peter Boy
https://fedoraproject.org/wiki/User:Pboy
[e-mail]

Timezone: CET (UTC+1) / CEST (UTC+2)

Fedora Server Edition Working Group member
Fedora Docs team contributor and board member
Java developer and enthusiast

[Call for Action] Help Testing F39 ARM

[Frantisek Zatloukal]

With rpi4, I didn't hit any issues:

*$ sudo fedora-arm-image-installer
--image=Fedora-Workstation-39-1.2.aarch64.raw.xz --norootpass
--media=/dev/sdaPlace your finger on the fingerprint readerFailed to match
fingerprintPlace your finger on the fingerprint
reader====================================================== Selected
Image:                                 =
Fedora-Workstation-39-1.2.aarch64.raw.xz= Selected Media : /dev/sda= Root
Password will be
removed.=====================================================
******************************************************************************************************************
WARNING! ALL DATA WILL BE DESTROYED
******************************************************************************************************************
 Type
'YES' to proceed, anything else to exit now  = Proceed? yesGPT PMBR size
mismatch (4156811 != 250085375) will be corrected by write.The backup GPT
table is corrupt, but the primary appears OK, so that will be used.The
backup GPT table is not on the end of the device.= Writing: =
Fedora-Workstation-39-1.2.aarch64.raw.xz = To: /dev/sda ....dd: warning:
partial read (65536 bytes); suggest iflag=fullblock17014218752 bytes (17
GB, 16 GiB) copied, 213 s, 79.9 MB/s17179869184 bytes (17 GB, 16 GiB)
copied, 213.812 s, 80.4 MB/s0+619058 records in0+619058 records
out17179869184 bytes (17 GB, 16 GiB) copied, 213.818 s, 80.3 MB/s= Writing
image complete!= No U-boot will be written.= Removing the root password.=
Installation Complete! Insert into the board and boot.*

And rpi4 boots just fine from the created mSD with 39 1.2 .

On Thu, Oct 26, 2023 at 2:51 PM Peter Boy  wrote:
> > Am 26.10.2023 um 13:36 schrieb Peter Robinson :
> >
> > For arm testers too
>
> I just tested RC 1.2 with the server image.  It’s rather broken and I
> don’t think we should release it.
> (minor)
>
> With arm-installer you get an error message:
>  To: /dev/mmcblk0 ....
> dd: warning: partial read (8192 bytes); suggest iflag=fullblock
>
> If you add this (w and w/o leading —-) it doesn’t start and you get a
> usage message listing all available parameters and that iflag isn’t among
> them.
>
> (I didn’t add it to the dd command line, most users wouldn’t do that, I
> suppose)
> (major)
>
> At the end of arm-installer you get a message:
>
> = Writing image complete!
> mount: /tmp/root: unknown filesystem type 'LVM2_member'.
>        dmesg(1) may have more information after failed mount system call.
> sed: /tmp/fw/EFI/fedora/grubenv kann nicht gelesen werden: Datei oder
> Verzeichnis nicht gefunden
> = No U-Boot files found for rock-pi-4-rk3399.
>
> = Installation Complete! Insert into the rock-pi-4-rk3399 and boot.
>
> (a)
> You can’t create a usable image without U-Boot files, of course and the
> final message is badly misleading.
>
> (b)
> We had this (or a similar looking) error briefly in F38 as well. The
> U-Boot files  are perfectly where they belong. The reason was that several
> lines before copying the U-Boot files, the paths were changed specifically
> for IOT, but these changes were then used for all other editions, too, when
> copying the U-Boot files, instead of only for IoT.
>
> So, it’s quite easy to fix, I suppose. But we need another RC, I think.
>
> I got that for all of my SBCs besides RPi4:
>
> = Writing image complete!
> mount: /tmp/root: unknown filesystem type 'LVM2_member'.
>        dmesg(1) may have more information after failed mount system call.
> sed: /tmp/fw/EFI/fedora/grubenv kann nicht gelesen werden: Datei oder
> Verzeichnis nicht gefunden
> = Raspberry Pi 4 Uboot is already in place, no changes needed.
>
> = Installation Complete! Insert into the rpi4 and boot.
>
> But it didn’t boot either.
>
> --
> Peter Boy
> https://fedoraproject.org/wiki/User:Pboy
> [e-mail]
>
> Timezone: CET (UTC+1) / CEST (UTC+2)
>
> Fedora Server Edition Working Group member
> Fedora Docs team contributor and board member
> Java developer and enthusiast
>
> arm mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]
> Do not reply to spam, report it:
> https://pagure.io/fedora-infrastructure/new_issue

[Call for Action] Help Testing F39 ARM

[Peter Boy]

> Am 26.10.2023 um 15:30 schrieb Frantisek Zatloukal :
>
> With rpi4, I didn't hit any issues:
>
> $ sudo fedora-arm-image-installer --image=Fedora-Workstation-39-1.2.aarch64.raw.xz --norootpass --media=/dev/sda
> Place your finger on the fingerprint reader
> Failed to match fingerprint
> Place your finger on the fingerprint reader
>
> =====================================================
> = Selected Image:                                 
> = Fedora-Workstation-39-1.2.aarch64.raw.xz
> = Selected Media : /dev/sda
> = Root Password will be removed.
> =====================================================
>
> ******** WARNING! ALL DATA WILL BE DESTROYED ********
>
> Type 'YES' to proceed, anything else to exit now
>
> = Proceed? yes
> GPT PMBR size mismatch (4156811 != 250085375) will be corrected by write.
> The backup GPT table is corrupt, but the primary appears OK, so that will be used.
> The backup GPT table is not on the end of the device.
> = Writing: 
> = Fedora-Workstation-39-1.2.aarch64.raw.xz 
> = To: /dev/sda ....
> dd: warning: partial read (65536 bytes); suggest iflag=fullblock
> 17014218752 bytes (17 GB, 16 GiB) copied, 213 s, 79.9 MB/s17179869184 bytes (17 GB, 16 GiB) copied, 213.812 s, 80.4 MB/s
>
> 0+619058 records in
> 0+619058 records out
> 17179869184 bytes (17 GB, 16 GiB) copied, 213.818 s, 80.3 MB/s
> = Writing image complete!
> = No U-boot will be written.
> = Removing the root password.
>
> = Installation Complete! Insert into the board and boot.

Well, if you look at the source, you’ll see the issue is in line 543:
> = Writing image complete!
> mount: /tmp/root: unknown filesystem type 'LVM2_member'.
>        dmesg(1) may have more information after failed mount system call.
> sed: /tmp/fw/EFI/fedora/grubenv kann nicht gelesen werden: Datei oder Verzeichnis nicht gefunden
> DEBUT info:
> /tmp/root/usr/share/uboot/rock-pi-4-rk3399
> = No U-Boot files found for rock-pi-4-rk3399.
>
> = Installation Complete! Insert into the rock-pi-4-rk3399 and boot.

The PREFIX is "/tmp/root“ but should be empty.

It is set in line 531 for IoT device and in line 533 adjusted for BTRFS (=Workstation). And for all others it is set in line with 535 to a value, which is wrong at least for Server and I guess all non-OStree and non-BTRFS editions.

And you didn’t set an U-Boot target explicitly, I did. Maybe, there is yet another issue in the code.

I checked again the rpi4. The device does boot and the display works up to the EFI screen and turns black shortly thereafter. But I can log in via SSH.

--
Peter Boy
https://fedoraproject.org/wiki/User:Pboy
[e-mail]

Timezone: CET (UTC+1) / CEST (UTC+2)

Fedora Server Edition Working Group member
Fedora Docs team contributor and board member
Java developer and enthusiast

[Call for Action] Help Testing F39 ARM

[Adam Williamson]

On Thu, 2023-10-26 at 16:50 +0200, Peter Boy wrote:
> > Am 26.10.2023 um 15:30 schrieb Frantisek Zatloukal :
> >
> > With rpi4, I didn't hit any issues:
> >
> > $ sudo fedora-arm-image-installer --image=Fedora-Workstation-39-1.2.aarch64.raw.xz --norootpass --media=/dev/sda
> > Place your finger on the fingerprint reader
> > Failed to match fingerprint
> > Place your finger on the fingerprint reader
> >
> > =====================================================
> > = Selected Image:                                 
> > = Fedora-Workstation-39-1.2.aarch64.raw.xz
> > = Selected Media : /dev/sda
> > = Root Password will be removed.
> > =====================================================
> >
> > ******** WARNING! ALL DATA WILL BE DESTROYED ********
> >
> > Type 'YES' to proceed, anything else to exit now
> >
> > = Proceed? yes
> > GPT PMBR size mismatch (4156811 != 250085375) will be corrected by write.
> > The backup GPT table is corrupt, but the primary appears OK, so that will be used.
> > The backup GPT table is not on the end of the device.
> > = Writing: 
> > = Fedora-Workstation-39-1.2.aarch64.raw.xz 
> > = To: /dev/sda ....
> > dd: warning: partial read (65536 bytes); suggest iflag=fullblock
> > 17014218752 bytes (17 GB, 16 GiB) copied, 213 s, 79.9 MB/s17179869184 bytes (17 GB, 16 GiB) copied, 213.812 s, 80.4 MB/s
> >
> > 0+619058 records in
> > 0+619058 records out
> > 17179869184 bytes (17 GB, 16 GiB) copied, 213.818 s, 80.3 MB/s
> > = Writing image complete!
> > = No U-boot will be written.
> > = Removing the root password.
> >
> > = Installation Complete! Insert into the board and boot.
>
> Well, if you look at the source, you’ll see the issue is in line 543:
> > = Writing image complete!
> > mount: /tmp/root: unknown filesystem type 'LVM2_member'.
> >        dmesg(1) may have more information after failed mount system call.
> > sed: /tmp/fw/EFI/fedora/grubenv kann nicht gelesen werden: Datei oder Verzeichnis nicht gefunden
> > DEBUT info:
> > /tmp/root/usr/share/uboot/rock-pi-4-rk3399
> > = No U-Boot files found for rock-pi-4-rk3399.
> >
> > = Installation Complete! Insert into the rock-pi-4-rk3399 and boot.
>
> The PREFIX is "/tmp/root“ but should be empty.
>
> It is set in line 531 for IoT device and in line 533 adjusted for BTRFS (=Workstation). And for all others it is set in line with 535 to a value, which is wrong at least for Server and I guess all non-OStree and non-BTRFS editions.

Looking at the script, I don't think your diagnosis is quite right,
here.

What the script is trying to do is mount the target root filesystem to
/tmp/root , then do stuff to it. The prefix definitely *should* be
/tmp/root, otherwise it will be doing stuff to *the host system*, which
is not what we want.

Looking at your output, I think the problem in your case is rather that
mounting the target root filesystem on /tmp/root did not work. That's
what `mount: /tmp/root: unknown filesystem type 'LVM2_member'.` tells
us.

I suspect the problem may be here, in fact:
https://pagure.io/arm-image-installer/blob/main/f/arm-image-installer#_477
if you compare it to this later similar line:
https://pagure.io/arm-image-installer/blob/main/f/arm-image-installer#_511
it reads to me like the later block is the 'normal'/original path for
mounting /tmp/root, and the earlier block was kinda added later to
handle mounting earlier and resizing the filesystem in certain cases.
But it looks like on the earlier path we may try to mount the wrong
thing - $ROOTPART instead of /dev/$LVM_NAME/root .

This exact codepath is only hit when there is an LVM LV on the media
and the filesystem is xfs, which should be exactly the case for the
server image, I guess. (I am confused as to how the detection for both
those things somehow *works* in this case, but oh well).

I have quite a lot of other questions about how this script is written,
but, well...I think that's the problem :)