Kickstart Fedora 30 (aarch64) on Raspberry Pi 3 B+

[Felix Geschwindner]

Hi Fedora community,

Fedora on RPI is AMAZING and I'm working on a project to automate provisioning for a Raspberry Pi cluster. This would add a ton of CI capabilities and convenience. It would also take the edge of the nuisance that RPIs eat SD cards like I eat Oreos.
Right now the process is pretty manual:

1. Put SD card in secondary machine
2. DD Fedora's minimal aarch64 image onto SD card
3. Grow partition
4. Put SD card back
5. Boot
6. Go through initial setup
7. Update all packages
8. Continue with regular config and host management

This requires being physically in front of the RPIs (and provisioning machine) and involves a lot of hand holding.

I'd like to make this process as headless as possible so that re-kicking a RPI is as easy as "setting a flag", rebooting and waiting for the RPI to come back up freshly installed.
Right now I have the following path in mind how this could potentially work:

The network interface of the RPI3B+ is capable of PXE booting, yay! So I'm thinking something along the lines of:

1. DHCP
2. Load files through TFTP (bootcode.bin -> start_x.elf -> DTB(?) -> u-boot.bin/IPXE -> kernel -> initramfs)
3. Boot kernel/initramfs which fires up Anaconda
4. Install Fedora normally
5. Use %post to copy device tree files to /boot and do other things necessary to make the RPI boot

If this worked, provisioning would be as easy as:

(0. Create DHCP reservation)
1. SD card into RPI
2. Turn on RPI
3. Login via SSH/VNC/etc.

I managed to boot U-Boot, chain load iPXE (probably not necessary since U-Boot is capable enough) but when pulling the regular aarch64 vmlinuz/initrd.gz pxeboot images it just errors out with an error saying "Invalid application". My knowledge about the kernel/initramfs chain gets blurry here which is why all tries in altering the initrd.gz image have failed gloriously.

If we can make this possible I'd be happy to turn it into a Wiki page so that others can extend their RPI environment with unattended provisioning.

Happy memorial to everyone celebrating and thank you for your interest!
Vascko

[Samuel Sieb]

On 5/27/19 10:13 PM, Felix Geschwindner via users wrote:
> I managed to boot U-Boot, chain load iPXE (probably not necessary since U-Boot is capable enough) but when pulling the regular aarch64 vmlinuz/initrd.gz pxeboot images it just errors out with an error saying "Invalid application". My knowledge about the kernel/initramfs chain gets blurry here which is why all tries in altering the initrd.gz image have failed gloriously.

I don't know much about this, but "Invalid application" suggests that 
iPXE doesn't like the kernel image you're trying to start.  Unless it's 
uboot not liking your iPXE.

[Felix Geschwindner]

For the sake of completion I also tried cutting out iPXE and loading the kernel/initramfs directly through u-boot which also didn't get me far. As soon as I'm in front of the system again I'll grab the correct messages I'm seeing. I also just got my UART cable which will give me more debug insight

When booting a RPI that is running a replayed RAW aarch64 image, it is using the packages from the native aarch64 repo here: https://dl.fedoraproject.org/pub/fedora/linux/releases/30/Everything/aarch64 including kernel and initramfs which is what I'm experimenting with too. So I don't think there is something wrong with the images I'm using. Even the device tree is loaded from the first partition of the SD card (or TFTP) and is not part of the kernel or initramfs as can be seen when examining /boot on a RPI running the raw image.

I even unpacked both the PXE and the installed initramfs images and they don't seem to be that different (I could be missing something of course) other than that the PXE image is solely meant to load install.img and bootstrap Anaconda.

I'm guessing it has something to do with the kernel and initramfs not being loaded into the right memory addresses

[James Szinger]

On Mon, May 27, 2019 at 11:13 PM Felix Geschwindner via users
 wrote:
> Fedora on RPI is AMAZING and I'm working on a project to automate provisioning for a Raspberry Pi cluster. This would add a ton of CI capabilities and convenience. It would also take the edge of the nuisance that RPIs eat SD cards like I eat Oreos.
> Right now the process is pretty manual:
> Put SD card in secondary machine
> DD Fedora's minimal aarch64 image onto SD card
> Grow partition
> Put SD card back
> Boot
> Go through initial setup
> Update all packages
> Continue with regular config and host management
>
> This requires being physically in front of the RPIs (and provisioning machine) and involves a lot of hand holding.

I can't help with the PXE boot, but perhaps the Fedora Arm list might
help

I have managed to use virt-install with a kickstart file to setup an
aarch64 virtual-machine on an x86_64 host.  I think if you copy the
magic bits from the official Fedora arm kickstarts, you end up with an
image that can be copied to an SD card.  Perhaps the following might
work:

1. Write a kickstart file setup the way you want.
2. Run virt-install with this kickstart.  Make the disk image big
enough that you don't have to resize.
2a. Periodically refresh the master image.
3. Clone the VM and run virt-sysprep to do the initial setup.
4. dd the image to the SD card.
5. put the SD card back and boot.
6. Continue with regular config and host management.

Pretty much everything except handling the SD cards can be scripted.

Jim

[Felix Geschwindner]

Thanks Jim, I'll start a thread on the ARM list and reference this discussion.

I played around a bit with both virt_install and lorax and they just don't feel as smooth and clean as the traditional PXE way