Error during Fedora 30 kickstart on Raspberry Pi 3 B+

[Felix Geschwindner]

Hey Guys,

I've been using Fedora on Raspberry Pis for a while now and I love it. The only problem I consistently face is the process of reinstalling (due to broken SD, sudo SNAFU etc.):

1. Put SD card in secondary machine
2. DD the Fedora aarch64 image onto SD card
3. Grow partition
4. Put SD card back
5. Boot
6. Go through initial setup
7. Update all packages
8. Continue with regular config and host management

Since I've used PXE in x86_64 environments for years, I started looking into doing the same for the RPI. The idea is to condense the process into a single step:

1. Either insert (new) empty SD card or destroy the partition table on the active SD card and reboot

In theory this should work because by default the RPI 3 B+ boots from SD card first and on failure moves on to PXE which then loads the PXE components from TFTP and starts the kickstart process. That should give (with the proper kickstart instructions) a completely headless and unattended installation environment for RPI.

This has been done for preseeding raspbian here: https://github.com/debian-pi/raspbian-ua-netinst and I'd love to do this for Fedora as well.

I brought up my PXE environment and the RPI is reliably pulling everything from initial bootcode.bin all the way to U-Boot and I end up with a U-Boot shell prompt.
Then loading grubaa64.efi through TFTP brings me to the GRUB shell which pulls the vmlinuz and initrd.img devices and starts the kernel.

This is where I'm stuck. When the system is loading the stage2 install.img it looks like it's running out of space on the tmpfs and I'm getting a "No space left on device" error and kernel panic (not sure yet if they are related).

I'm not very skilled in dissecting initrd images so I'd appreciate any pointers. Is there a kernel or anaconda parameter to increase the tmpfs size? Maybe the 1G of memory is just not big enough to hold the ~470MB install.img AND have enough left to finish the kickstart. Would it be feasible to reduce install.img?

[   47.515164] usbcore: registered new interface driver lan78xx
[   49.874865] ------------[ cut here ]------------
[   49.892272] irq 79 handler irq_default_primary_handler+0x0/0x20 enabled interrupts
[   49.912598] WARNING: CPU: 0 PID: 0 at kernel/irq/handle.c:152 __handle_irq_event_percpu+0x1fc/0x228
[   49.934342] Modules linked in: microchip lan78xx hid_ite snd_pcm snd_timer snd soundcore drm_kms_helper mmc_block syscopyarea sysfillrect sysimgblt fb_sys_fops drm dwc2 sdhci_iproc sdhci_pltfm crct10dif_ce udc_core sdhci gpio_raspberrypi_exp pwm_bcm2835 i2c_bcm2835 bcm2835 bcm2835_dma phy_generic sunrpc lrw dm_crypt dm_round_robin dm_multipath linear raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor xor_neon raid6_pq libcrc32c raid1 raid0 squashfs zstd_decompress cramfs be2iscsi bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi
[   50.057520] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.0.9-301.fc30.aarch64 #1
[   50.078134] Hardware name: raspberrypi rpi/rpi, BIOS 2019.04 04/18/2019
[   50.098047] pstate: 60400005 (nZCv daif +PAN -UAO)
[   50.116053] pc : __handle_irq_event_percpu+0x1fc/0x228
[   50.134467] lr : __handle_irq_event_percpu+0x1fc/0x228
[   50.152808] sp : ffff000010003ce0
[   50.168935] x29: ffff000010003ce0 x28: ffff80002aa46a00
[   50.186785] x27: ffff000011184018 x26: ffff000011184018
[   50.204289] x25: ffff00001157ef60 x24: ffff000010003d6c
[   50.221399] x23: 0000000000000000 x22: 000000000000004f
[   50.238168] x21: ffff80002aa46a28 x20: 0000000000000002
[   50.254596] x19: ffff80002370d700 x18: 0000000000000000
[   50.270753] x17: 0000000000000000 x16: 0000000000000000
[   50.286512] x15: 0000000000000010 x14: 0000000000000001
[   50.301835] x13: 0000000000001280 x12: ffff000010bf0db8
[   50.316804] x11: 0000000000000000 x10: ffff800037ada0b0
[   50.331455] x9 : 00000000ffffffff x8 : 0000000000000152
[   50.346058] x7 : ffff0000176d1820 x6 : 0000000000000001
[   50.360583] x5 : 0000000000000000 x4 : 0000000000000001
[   50.374766] x3 : 0000000000000000 x2 : 0000000000000040
[   50.388589] x1 : 0000000000000004 x0 : 0000000000000046
[   50.402281] Call trace:
[   50.412701]  __handle_irq_event_percpu+0x1fc/0x228
[   50.425284]  handle_irq_event_percpu+0x28/0x70
[   50.437447]  handle_irq_event+0x50/0xa8
[   50.448636]  handle_simple_irq+0xac/0xf8
[   50.459564]  generic_handle_irq+0x34/0x50
[   50.470387]  intr_complete+0x8c/0x178 [lan78xx]
[   50.481577]  __usb_hcd_giveback_urb+0x74/0x140
[   50.492398]  usb_giveback_urb_bh+0xd0/0x150
[   50.502669]  tasklet_action_common.isra.0+0x88/0xf0
[   50.513687]  tasklet_hi_action+0x2c/0x38
[   50.523602]  __do_softirq+0x128/0x31c
[   50.533054]  irq_exit+0xc0/0xe0
[   50.541612]  __handle_domain_irq+0x70/0xc0
[   50.550842]  bcm2836_arm_irqchip_handle_irq+0x74/0xd8
[   50.560783]  el1_irq+0xe8/0x1c0
[   50.568434]  arch_cpu_idle+0x34/0x1b8
[   50.576319]  do_idle+0x1fc/0x248
[   50.583591]  cpu_startup_entry+0x2c/0x30
[   50.591483]  rest_init+0xc0/0xcc
[   50.598557]  arch_call_rest_init+0x14/0x1c
[   50.606494]  start_kernel+0x46c/0x480
[   50.614059] ---[ end trace b9c80dddae7c6c2b ]---
[   50.628673] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[   54.412911] dracut-initqueue[880]: Warning: can't find installer main image path in .treeinfo
[   54.483584] dracut-initqueue[880]:   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
[   54.484525] dracut-initqueue[880]:                                  Dload  Upload   Total   Spent    Left  Speed
 38  482M   38  187M    0     0  38.1M      0  0:00:12  0:00:04  0:00:08 38.2M
[   59.408000] dracut-initqueue[880]: curl: (23) Failed writing body (2946 != 5792)
[   60.172589] loop: module loaded
[   60.334211] dracut-initqueue[880]: mount: /run/initramfs/squashfs: wrong fs t[   60.342641] dracut: FATAL: Failed to find a root filesystem in /tmp/curl_fetch_url1/install.img.
ype, bad option, bad superblock on /dev/loop0, missing codepage or helpe[   60.342703] dracut: Refusing to continue
r program, or other error.
[   60.337120] dracut-initqueue[880]: /lib/dracut-lib.sh: line 465: echo: write error: No space left on device
[   60.367607] dracut-initqueue[880]: /lib/dracut-lib.sh: line 466: echo: write error: No space left on device
[   60.425966] dracut-initqueue[880]: /lib/anaconda-lib.sh: line 151: printf: write error: No space left on device

[Felix Geschwindner]

I tried unpacking the initrd.img into a NFS directory on another machine and added "root=/dev/nfs nfsroot=172.16.16.12:/opt/rootfs,vers=3,tcp" to the kernel parameters (I tried UDP and vers=4 as well) but now I'm getting NFS access errors:

[    3.984691] VFS: Cannot open root device "nfs" or unknown-block(0,255): error -6
[    4.000335] Please append a correct "root=" boot option; here are the available partitions:
[    4.017435] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,255)

Ugh - any ideas?

[Paul Whalen]

Hi Felix,

----- Original Message -----
> Hey Guys,
>
> I've been using Fedora on Raspberry Pis for a while now and I love it. The
> only problem I consistently face is the process of reinstalling (due to
> broken SD, sudo SNAFU etc.):
> Put SD card in secondary machine
> DD the Fedora aarch64 image onto SD card
> Grow partition
> Put SD card back
> Boot
> Go through initial setup
> Update all packages
> Continue with regular config and host management
>
> Since I've used PXE in x86_64 environments for years, I started looking into
> doing the same for the RPI. The idea is to condense the process into a
> single step:
> Either insert (new) empty SD card or destroy the partition table on the
>
> active SD card and reboot
>
> In theory this should work because by default the RPI 3 B+ boots from SD card
> first and on failure moves on to PXE which then loads the PXE components
> from TFTP and starts the kickstart process. That should give (with the
> proper kickstart instructions) a completely headless and unattended
> installation environment for RPI.
>
> This has been done for preseeding raspbian here:
> https://github.com/debian-pi/raspbian-ua-netinst and I'd love to do this for
> Fedora as well.
>
> I brought up my PXE environment and the RPI is reliably pulling everything
> from initial bootcode.bin all the way to U-Boot and I end up with a U-Boot
> shell prompt.
> Then loading grubaa64.efi through TFTP brings me to the GRUB shell which
> pulls the vmlinuz and initrd.img devices and starts the kernel.
>
> This is where I'm stuck. When the system is loading the stage2 install.img it
> looks like it's running out of space on the tmpfs and I'm getting a "No
> space left on device" error and kernel panic (not sure yet if they are
> related).
>
> I'm not very skilled in dissecting initrd images so I'd appreciate any
> pointers. Is there a kernel or anaconda parameter to increase the tmpfs
> size? Maybe the 1G of memory is just not big enough to hold the ~470MB
> install.img AND have enough left to finish the kickstart. Would it be
> feasible to reduce install.img?

A network installs need about 2G ram unfortunately. You can use NFS to
provide the initramfs using 'inst.stage2=nfs://192.168.X.X:/path/to/install.img'.

Paul
> [   47.515164] usbcore: registered new interface driver lan78xx
> [   49.874865] ------------[ cut here ]------------
> [   49.892272] irq 79 handler irq_default_primary_handler+0x0/0x20 enabled
> interrupts
> [   49.912598] WARNING: CPU: 0 PID: 0 at kernel/irq/handle.c:152
> __handle_irq_event_percpu+0x1fc/0x228
> [   49.934342] Modules linked in: microchip lan78xx hid_ite snd_pcm snd_timer
> snd soundcore drm_kms_helper mmc_block syscopyarea sysfillrect sysimgblt
> fb_sys_fops drm dwc2 sdhci_iproc sdhci_pltfm crct10dif_ce udc_core sdhci
> gpio_raspberrypi_exp pwm_bcm2835 i2c_bcm2835 bcm2835 bcm2835_dma phy_generic
> sunrpc lrw dm_crypt dm_round_robin dm_multipath linear raid10 raid456
> async_raid6_recov async_memcpy async_pq async_xor async_tx xor xor_neon
> raid6_pq libcrc32c raid1 raid0 squashfs zstd_decompress cramfs be2iscsi
> bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx
> iscsi_boot_sysfs iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi
> [   50.057520] CPU: 0 PID: 0 Comm: swapper/0 Not tainted
> 5.0.9-301.fc30.aarch64 #1
> [   50.078134] Hardware name: raspberrypi rpi/rpi, BIOS 2019.04 04/18/2019
> [   50.098047] pstate: 60400005 (nZCv daif +PAN -UAO)
> [   50.116053] pc : __handle_irq_event_percpu+0x1fc/0x228
> [   50.134467] lr : __handle_irq_event_percpu+0x1fc/0x228
> [   50.152808] sp : ffff000010003ce0
> [   50.168935] x29: ffff000010003ce0 x28: ffff80002aa46a00
> [   50.186785] x27: ffff000011184018 x26: ffff000011184018
> [   50.204289] x25: ffff00001157ef60 x24: ffff000010003d6c
> [   50.221399] x23: 0000000000000000 x22: 000000000000004f
> [   50.238168] x21: ffff80002aa46a28 x20: 0000000000000002
> [   50.254596] x19: ffff80002370d700 x18: 0000000000000000
> [   50.270753] x17: 0000000000000000 x16: 0000000000000000
> [   50.286512] x15: 0000000000000010 x14: 0000000000000001
> [   50.301835] x13: 0000000000001280 x12: ffff000010bf0db8
> [   50.316804] x11: 0000000000000000 x10: ffff800037ada0b0
> [   50.331455] x9 : 00000000ffffffff x8 : 0000000000000152
> [   50.346058] x7 : ffff0000176d1820 x6 : 0000000000000001
> [   50.360583] x5 : 0000000000000000 x4 : 0000000000000001
> [   50.374766] x3 : 0000000000000000 x2 : 0000000000000040
> [   50.388589] x1 : 0000000000000004 x0 : 0000000000000046
> [   50.402281] Call trace:
> [   50.412701]  __handle_irq_event_percpu+0x1fc/0x228
> [   50.425284]  handle_irq_event_percpu+0x28/0x70
> [   50.437447]  handle_irq_event+0x50/0xa8
> [   50.448636]  handle_simple_irq+0xac/0xf8
> [   50.459564]  generic_handle_irq+0x34/0x50
> [   50.470387]  intr_complete+0x8c/0x178 [lan78xx]
> [   50.481577]  __usb_hcd_giveback_urb+0x74/0x140
> [   50.492398]  usb_giveback_urb_bh+0xd0/0x150
> [   50.502669]  tasklet_action_common.isra.0+0x88/0xf0
> [   50.513687]  tasklet_hi_action+0x2c/0x38
> [   50.523602]  __do_softirq+0x128/0x31c
> [   50.533054]  irq_exit+0xc0/0xe0
> [   50.541612]  __handle_domain_irq+0x70/0xc0
> [   50.550842]  bcm2836_arm_irqchip_handle_irq+0x74/0xd8
> [   50.560783]  el1_irq+0xe8/0x1c0
> [   50.568434]  arch_cpu_idle+0x34/0x1b8
> [   50.576319]  do_idle+0x1fc/0x248
> [   50.583591]  cpu_startup_entry+0x2c/0x30
> [   50.591483]  rest_init+0xc0/0xcc
> [   50.598557]  arch_call_rest_init+0x14/0x1c
> [   50.606494]  start_kernel+0x46c/0x480
> [   50.614059] ---[ end trace b9c80dddae7c6c2b ]---
> [   50.628673] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
> [   54.412911] dracut-initqueue[880]: Warning: can't find installer main
> image path in .treeinfo
> [   54.483584] dracut-initqueue[880]:   % Total    % Received % Xferd
> Average Speed   Time    Time     Time  Current
> [   54.484525] dracut-initqueue[880]:                                  Dload
> Upload   Total   Spent    Left  Speed
>  38  482M   38  187M    0     0  38.1M      0  0:00:12  0:00:04  0:00:08
>  38.2M
> [   59.408000] dracut-initqueue[880]: curl: (23) Failed writing body (2946 !=
> 5792)
> [   60.172589] loop: module loaded
> [   60.334211] dracut-initqueue[880]: mount: /run/initramfs/squashfs: wrong
> fs t[   60.342641] dracut: FATAL: Failed to find a root filesystem in
> /tmp/curl_fetch_url1/install.img.
> ype, bad option, bad superblock on /dev/loop0, missing codepage or helpe[
> 60.342703] dracut: Refusing to continue
> r program, or other error.
> [   60.337120] dracut-initqueue[880]: /lib/dracut-lib.sh: line 465: echo:
> write error: No space left on device
> [   60.367607] dracut-initqueue[880]: /lib/dracut-lib.sh: line 466: echo:
> write error: No space left on device
> [   60.425966] dracut-initqueue[880]: /lib/anaconda-lib.sh: line 151: printf:
> write error: No space left on device
> _______________________________________________
> arm mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]

[Felix Geschwindner]

> A network installs need about 2G ram unfortunately. You can use NFS to
> provide the initramfs using 'inst.stage2=nfs://192.168.X.X:/path/to/install.img'.

Thanks for the hint Paul.
I tried with the file name appended as you suggested as well as just the directory. When using the file name in the mount path, it complains that it can't mount due to not being a directory and when I remove the filename it mounts but says it can't find any suitable images. But I can see the install.img in the mount path:

[   51.219503] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[   55.251825] FS-Cache: Loaded
[   55.427167] FS-Cache: Netfs 'nfs' registered for caching
[   55.523210] dracut-initqueue[881]: /lib/anaconda-lib.sh: line 71: shift: 2: shift count out of range
[   55.560399] dracut-initqueue[881]: Warning: no suitable images
[  226.962375] dracut-initqueue[881]: Warning: dracut-initqueue timeout - starting timeout scripts

dracut:/# mount
172.16.16.12:/opt/rootfs/ on /run/install/repo type nfs (ro,relatime,vers=3,rsize=262144,wsize=262144,namlen=255,hard,nolock,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=172.16.16.12,mountvers=3,mountport=20048,mountproto=udp,local_lock=all,addr=172.16.16.12)

dracut:/# ls -l /run/install/repo
total 478920
-rw-r--r-- 1 root root 490414080 Jun  3  2019 install.img

[Paul Whalen]

Hi Felix,

----- Original Message -----
> > A network installs need about 2G ram unfortunately. You can use NFS to
> > provide the initramfs using
> > 'inst.stage2=nfs://192.168.X.X:/path/to/install.img'.
>
> Thanks for the hint Paul.
> I tried with the file name appended as you suggested as well as just the
> directory. When using the file name in the mount path, it complains that it
> can't mount due to not being a directory and when I remove the filename it
> mounts but says it can't find any suitable images. But I can see the

Right, as you note you need to serve a directory and within that a directory
called 'images' with the install.img in there (to match the installation tree).
> install.img in the mount path:
>
> [   51.219503] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
> [   55.251825] FS-Cache: Loaded
> [   55.427167] FS-Cache: Netfs 'nfs' registered for caching
> [   55.523210] dracut-initqueue[881]: /lib/anaconda-lib.sh: line 71: shift:
> 2: shift count out of range
> [   55.560399] dracut-initqueue[881]: Warning: no suitable images
> [  226.962375] dracut-initqueue[881]: Warning: dracut-initqueue timeout -
> starting timeout scripts
>
> dracut:/# mount
> 172.16.16.12:/opt/rootfs/ on /run/install/repo type nfs
> (ro,relatime,vers=3,rsize=262144,wsize=262144,namlen=255,hard,nolock,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=172.16.16.12,mountvers=3,mountport=20048,mountproto=udp,local_lock=all,addr=172.16.16.12)
>
> dracut:/# ls -l /run/install/repo
> total 478920
> -rw-r--r-- 1 root root 490414080 Jun  3  2019 install.img
> _______________________________________________
> arm mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]

[Felix Geschwindner]

That did the trick! Thanks, Paul

The kickstart installation is humming along nicely until the bootloader is being installed. It errors out and I have to manually select "yes" to continue. The system is still installed fine and starts without problems but it's not unattended:

21:33:35,996 DBG bootloader.installation: Configuring the boot loader.
21:33:36,003 DBG bootloader.installation: Collecting the OS images for: 5.1.5-300.fc30.aarch64
21:33:36,006 DBG bootloader.base: new default image: <pyanaconda.bootloader.image.LinuxBootLoaderImage object at 0xffffa2472eb8>
21:33:36,010 DBG bootloader.installation: Writing to /etc/sysconfig/kernel.
21:33:36,083 DBG bootloader.installation: Installing the boot loader.
21:33:36,085 INF bootloader.installation: boot loader stage1 target device is mmcblk0p1
21:33:36,086 INF bootloader.installation: boot loader stage2 target device is mmcblk0p2
21:33:36,754 INF bootloader.grub2: bootloader.py: used boot args:
21:33:51,449 ERR bootloader.installation: bootloader.write failed: Failed to set new efi boot target. This is most likely a kernel or firmware bug.

In my kickstart I'm using this to configure the bootloader:

bootloader --location=mbr --timeout=1 --boot-drive=mmcblk0

which is exactly the same that is in the KS file that can be found in the official AARCH64 image.

My uneducated guess is that GRUB has some sort of beef with the lack of a EFI ROM so it can't register a boot entry. Can that be disabled somehow? Haven't found an option for that in the man pages.
I tried  --location=none which clears the error but it also doesn't generate the grub.cfg so maybe I need to call grub2-mkconfig in my %post section to compensate for that. Any other ideas?