RPI4 8G - usb ssd boot Fedora Server with UEFI for Pi

[Zach Villers]

I recently got Fedora 33 Server booting on my RPI 4 - 8G using the RPI4 UEFI firmware. The usb3 adapter I used worked fine with Raspbian, but with Fedora 33, there was an issue. I've worked around this issue and another involving F33 complaining about the lack of sd card. These are my brief notes, hopefully they help someone. If a longer write up or a bug report would be helpful, I'm happy to do so. Also looking for feedback if I could have done anything better/different.

Hardware:
RPI 4B - 8G model
128G "Inland" brand ssd - OEM Phison - Model 'SATA AAD'
Orico Brand SATA to USB3 external enclosure - model 'ORICO 2139U3' 
  - uses JMicron Technology Corp. JMS578 chip - Sata 6Gb/s

UEFI Firmware: https://github.com/pftf/RPi4/releases/tag/v1.21
After Fedora install;
- switched advanced settings in UEFI from 'ahci' to 'devicetree'
- removed 3G ram limit

### Problems

1. Pi treats usb drive as uas device causing extremely slow read/write - ext4 fs would not mount rw
      '''
          [ 1247.365069] usb 2-1: reset SuperSpeed Gen 1 USB device number 3 using xhci_hcd
          [ 1247.387663] scsi host0: uas_eh_device_reset_handler success
          [ 1279.648929] sd 0:0:0:0: [sda] tag#3 uas_eh_abort_handler 0 uas-tag 4 inflight: CMD IN
          [ 1279.648937] sd 0:0:0:0: [sda] tag#3 CDB: opcode=0x28 28 00 03 fa 64 50 00 01 00 00
          [ 1279.649085] sd 0:0:0:0: [sda] tag#1 uas_eh_abort_handler 0 uas-tag 2 inflight: CMD IN
          [ 1279.649091] sd 0:0:0:0: [sda] tag#1 CDB: opcode=0x28 28 00 00 00 10 00 00 04 00 00
          [ 1279.649553] sd 0:0:0:0: [sda] tag#0 uas_eh_abort_handler 0 uas-tag 1 inflight: CMD IN
          [ 1279.649559] sd 0:0:0:0: [sda] tag#0 CDB: opcode=0x28 28 00 00 00 0c 00 00 04 00 00
          [ 1286.049165] sd 0:0:0:0: tag#4 uas_eh_abort_handler 0 uas-tag 5 inflight: CMD
          [ 1286.049173] sd 0:0:0:0: tag#4 CDB: opcode=0x0 00 00 00 00 00 00
          [ 1286.065838] scsi host0: uas_eh_device_reset_handler start
      '''
2. Pi is missing sdhc card ( mostly annoying, could've just turned off kernel message echo to console? )
      '''
    mmc1: Timeout waiting for hardware cmd interrupt.
    mmc1: sdhci: =========== SDHCI REGISTER DUMP ===========
    mmc1: sdhci: blah
      '''
    		
### Solutions
1. uas device
   - add usb quirk to kernel args in /etc/default/grub
       '''
    	GRUB_CMDLINE_LINUX="usb-storage.quirks=152d:0578:u"
       '''
     - add quirk to config.txt (is this necessary?)
     - rebuild grub with grub2-mkconfig
     - dracut -f (not sure if I needed to do this?)
2. sdhc card
   - block(black)list sdhc drivers
    - /etc/modprobe.d/no_sdhc.conf
       '''
    	blacklist sdhci
    	blacklist sdhci_platform
    	blacklist sdhci_iproc
    	install sdhci /bin/false
    	install sdhci_platform /bin/false
    	install sdhci_iproc /bin/false
       '''
    - From [finishing steps for rhel 8](https://access.redhat.com/solutions/41278)
       - added 'sdhci.blacklist=1 rd.driver.blacklist=sdhci' to the above GRUB_CMDLINE_LINUX
       - rebuilt grub again
       - (make a copy of and) rebuilt initramfs
              'dracut --omit-drivers sdhci -f'

### Things I didn't do
1. fool around with kdump
2. Edit the installer kernel CMDLINE to blacklist the sdhci module and add the usb quirks

### Other Helpful notes
Article - https://fwmotion.com/blog/operating-systems/2020-09-04-installing-fedora-ser...

[Matthias Runge]

On 03/01/2021 19:46, Zach Villers wrote:
> I recently got Fedora 33 Server booting on my RPI 4 - 8G using the RPI4 UEFI firmware. The usb3 adapter I used worked fine with Raspbian, but with Fedora 33, there was an issue. I've worked around this issue and another involving F33 complaining about the lack of sd card. These are my brief notes, hopefully they help someone. If a longer write up or a bug report would be helpful, I'm happy to do so. Also looking for feedback if I could have done anything better/different.
>
> Hardware:
> RPI 4B - 8G model
> 128G "Inland" brand ssd - OEM Phison - Model 'SATA AAD'
> Orico Brand SATA to USB3 external enclosure - model 'ORICO 2139U3'
> uses JMicron Technology Corp. JMS578 chip - Sata 6Gb/s
>
> UEFI Firmware: https://github.com/pftf/RPi4/releases/tag/v1.21
> After Fedora install;
> switched advanced settings in UEFI from 'ahci' to 'devicetree'
> removed 3G ram limit

Hi Zach,

thank you for sending this. By coincidence, I got my couple of RPi4 
working over Christmas as well.
There seem to be around a couple of buggy/misbehaving USB adapters. 
Personally, I found Jason Chambers notes very helpful[1], and I didn't 
have to use any quirks. The earlier RP4 I have seems to have issues with 
rebooting though, and I didn't get to the bottom of this (yet).

[1] 
https://jamesachambers.com/raspberry-pi-4-usb-boot-config-guide-for-ssd-flas...
> ### Other Helpful notes
> Article - https://fwmotion.com/blog/operating-systems/2020-09-04-installing-fedora-ser...

Yes! This blog post was very helpful for me, kudos to Robert here.

Matthias

[Zach Villers]

Thanks Matthias. I should have mentioned the article on jamesachambers.com as well. Ah! now having re-read, I see that the enclosure I'm using is on the 'naughty' list now. I could've sworn they were on the ok list when I bought them. Oh well, it just goes to show that you can either pay attention, pay for books, or pay with your time later on.

Cheers.

[Peter Robinson]

Hi Zach,
> I recently got Fedora 33 Server booting on my RPI 4 - 8G using the RPI4 UEFI firmware. The usb3 adapter I used worked fine with Raspbian, but with Fedora 33, there was an issue. I've worked around this issue and another involving F33 complaining about the lack of sd card. These are my brief notes, hopefully they help someone. If a longer write up or a bug report would be helpful, I'm happy to do so. Also looking for feedback if I could have done anything better/different.

Thanks for the outline here. I think the write up is fine here, I've
added a few notes below. Let me know what bits we can improve in
Fedora.
> Hardware:
> RPI 4B - 8G model
> 128G "Inland" brand ssd - OEM Phison - Model 'SATA AAD'
> Orico Brand SATA to USB3 external enclosure - model 'ORICO 2139U3'
> uses JMicron Technology Corp. JMS578 chip - Sata 6Gb/s
>
> UEFI Firmware: https://github.com/pftf/RPi4/releases/tag/v1.21
> After Fedora install;
> switched advanced settings in UEFI from 'ahci' to 'devicetree'
> removed 3G ram limit
>
> ### Problems
> Pi treats usb drive as uas device causing extremely slow read/write - ext4 fs would not mount rw
>    '''
>        [ 1247.365069] usb 2-1: reset SuperSpeed Gen 1 USB device number 3 using xhci_hcd
>        [ 1247.387663] scsi host0: uas_eh_device_reset_handler success
>        [ 1279.648929] sd 0:0:0:0: [sda] tag#3 uas_eh_abort_handler 0 uas-tag 4 inflight: CMD IN
>        [ 1279.648937] sd 0:0:0:0: [sda] tag#3 CDB: opcode=0x28 28 00 03 fa 64 50 00 01 00 00
>        [ 1279.649085] sd 0:0:0:0: [sda] tag#1 uas_eh_abort_handler 0 uas-tag 2 inflight: CMD IN
>        [ 1279.649091] sd 0:0:0:0: [sda] tag#1 CDB: opcode=0x28 28 00 00 00 10 00 00 04 00 00
>        [ 1279.649553] sd 0:0:0:0: [sda] tag#0 uas_eh_abort_handler 0 uas-tag 1 inflight: CMD IN
>        [ 1279.649559] sd 0:0:0:0: [sda] tag#0 CDB: opcode=0x28 28 00 00 00 0c 00 00 04 00 00
>        [ 1286.049165] sd 0:0:0:0: tag#4 uas_eh_abort_handler 0 uas-tag 5 inflight: CMD
>        [ 1286.049173] sd 0:0:0:0: tag#4 CDB: opcode=0x0 00 00 00 00 00 00
>        [ 1286.065838] scsi host0: uas_eh_device_reset_handler start
>    ''
> Pi is missing sdhc card ( mostly annoying, could've just turned off kernel message echo to console? )
>    '''
>      mmc1: Timeout waiting for hardware cmd interrupt.
>      mmc1: sdhci: =========== SDHCI REGISTER DUMP ===========
>      mmc1: sdhci: blah
>    '''
>
> ### Solutions
> uas device
> add usb quirk to kernel args in /etc/default/grub
>       '''
>            GRUB_CMDLINE_LINUX="usb-storage.quirks=152d:0578:u"
>       '''

Not sure why this is needed as the quirk for this device has had a
quirk upstream in Linux for 3 years. I wonder if it needs extra quirks
enabled.

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?i...
>      - add quirk to config.txt (is this necessary?)

I don't think so, not sure what you added but we don't use any of the
kernel command line bits from there.
>      - rebuild grub with grub2-mkconfig
>      - dracut -f (not sure if I needed to do this?)

Don't think you shoul there.
> sdhc card
> block(black)list sdhc drivers
> /etc/modprobe.d/no_sdhc.conf
>  '''
>       blacklist sdhci
>       blacklist sdhci_platform
>       blacklist sdhci_iproc
>       install sdhci /bin/false
>       install sdhci_platform /bin/false
>       install sdhci_iproc /bin/false
>  '''
> From [finishing steps for rhel 8](https://access.redhat.com/solutions/41278)
> added 'sdhci.blacklist=1 rd.driver.blacklist=sdhci' to the above GRUB_CMDLINE_LINUX
> rebuilt grub again
> (make a copy of and) rebuilt initramfs
>      'dracut --omit-drivers sdhci -f'

Is this just for the annoying message above? Does it stop the boot if
you don't do this? Ultimately it shouldn't error, but more info would
be useful here/
> ### Things I didn't do
> fool around with kdump
> Edit the installer kernel CMDLINE to blacklist the sdhci module and add the usb quirks
>
> ### Other Helpful notes
> Article - https://fwmotion.com/blog/operating-systems/2020-09-04-installing-fedora-ser...
>
> --
>   Zach Villers
>   [e-mail]
> _______________________________________________
> arm mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedoraproject.org/archives/list/[e-mail]

[Zach Villers]

> Not sure why this is needed as the quirk for this device has had a
> quirk upstream in Linux for 3 years. I wonder if it needs extra quirks
> enabled.
>
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit...

I was just barely able to complete the install within anaconda without the quirk. The install took an inordinately long time. When I booted into the new install, I was not able to log in. Several services had failed to start (sssd and NetworkManager among them if I recall).

After I added the quirk to grub, I was able to log in normally. I'm happy to wipe the install and recreate the problem so I can capture logs from installation if you would like? Is it possible that the patch you linked initially allowed the devices with that JMicron chip to work, but the RPI4 just wasn't able to deal with the repeated complaining?
> Is this just for the annoying message above? Does it stop the boot if
> you don't do this? Ultimately it shouldn't error, but more info would
> be useful here/

Yes - the sdhc error was repeating so often to console I could barely keep up with what I was typing.

Overall, I'm very happy with the install now. Super fast, cockpit is nice and responsive. Looking forward to using it! Thanks for all the great work y'all do.

[Daniel Riek]

Hi,

thanks for sharing Zach.

I have been fairly successful getting F33 work with the current stock RPi4
firmware on the 8GB model by borrowing these
two lines from the raspbian config.txt:
> dtoverlay=vc4-fkms-v3d
> max_framebuffers=2

I still owe Peter to figure out which of them.

The one issue I have is that on occasion it drops into uboot cli on reboot.
I have connected a serial console to diagnose that.

It has been running stable but with no load. Will hopefully get around to
put some load on it later this week.

Regards,

Daniel

On Sun, Jan 3, 2021 at 1:47 PM Zach Villers  wrote:
> I recently got Fedora 33 Server booting on my RPI 4 - 8G using the RPI4
> UEFI firmware. The usb3 adapter I used worked fine with Raspbian, but with
> Fedora 33, there was an issue. I've worked around this issue and another
> involving F33 complaining about the lack of sd card. These are my brief
> notes, hopefully they help someone. If a longer write up or a bug report
> would be helpful, I'm happy to do so. Also looking for feedback if I could
> have done anything better/different.
>
> Hardware:
> RPI 4B - 8G model
> 128G "Inland" brand ssd - OEM Phison - Model 'SATA AAD'
> Orico Brand SATA to USB3 external enclosure - model 'ORICO 2139U3'
> uses JMicron Technology Corp. JMS578 chip - Sata 6Gb/s
>
> UEFI Firmware: https://github.com/pftf/RPi4/releases/tag/v1.21
> After Fedora install;
> switched advanced settings in UEFI from 'ahci' to 'devicetree'
> removed 3G ram limit
>
> ### Problems
> Pi treats usb drive as uas device causing extremely slow read/write -
>
> ext4 fs would not mount rw
>       '''
>           [ 1247.365069] usb 2-1: reset SuperSpeed Gen 1 USB device number
> 3 using xhci_hcd
>           [ 1247.387663] scsi host0: uas_eh_device_reset_handler success
>           [ 1279.648929] sd 0:0:0:0: [sda] tag#3 uas_eh_abort_handler 0
> uas-tag 4 inflight: CMD IN
>           [ 1279.648937] sd 0:0:0:0: [sda] tag#3 CDB: opcode=0x28 28 00 03
> fa 64 50 00 01 00 00
>           [ 1279.649085] sd 0:0:0:0: [sda] tag#1 uas_eh_abort_handler 0
> uas-tag 2 inflight: CMD IN
>           [ 1279.649091] sd 0:0:0:0: [sda] tag#1 CDB: opcode=0x28 28 00 00
> 00 10 00 00 04 00 00
>           [ 1279.649553] sd 0:0:0:0: [sda] tag#0 uas_eh_abort_handler 0
> uas-tag 1 inflight: CMD IN
>           [ 1279.649559] sd 0:0:0:0: [sda] tag#0 CDB: opcode=0x28 28 00 00
> 00 0c 00 00 04 00 00
>           [ 1286.049165] sd 0:0:0:0: tag#4 uas_eh_abort_handler 0 uas-tag
> 5 inflight: CMD
>           [ 1286.049173] sd 0:0:0:0: tag#4 CDB: opcode=0x0 00 00 00 00 00
> 00
>           [ 1286.065838] scsi host0: uas_eh_device_reset_handler start
>       '''
> 2. Pi is missing sdhc card ( mostly annoying, could've just turned off
> kernel message echo to console? )
>       '''
>         mmc1: Timeout waiting for hardware cmd interrupt.
>         mmc1: sdhci: =========== SDHCI REGISTER DUMP ===========
>         mmc1: sdhci: blah
>       '''
>
> ### Solutions
> uas device
> add usb quirk to kernel args in /etc/default/grub
>       '''
>            GRUB_CMDLINE_LINUX="usb-storage.quirks=152d:0578:u"
>       '''
>     - add quirk to config.txt (is this necessary?)
>     - rebuild grub with grub2-mkconfig
>     - dracut -f (not sure if I needed to do this?)
> sdhc card
> block(black)list sdhc drivers
> /etc/modprobe.d/no_sdhc.conf
>  '''
>       blacklist sdhci
>       blacklist sdhci_platform
>       blacklist sdhci_iproc
>       install sdhci /bin/false
>       install sdhci_platform /bin/false
>       install sdhci_iproc /bin/false
>  '''
> From [finishing steps for rhel 8](
>
> https://access.redhat.com/solutions/41278)
>        - added 'sdhci.blacklist=1 rd.driver.blacklist=sdhci' to the above
> GRUB_CMDLINE_LINUX
>        - rebuilt grub again
>        - (make a copy of and) rebuilt initramfs
>               'dracut --omit-drivers sdhci -f'
>
> ### Things I didn't do
> fool around with kdump
> Edit the installer kernel CMDLINE to blacklist the sdhci module and add
>
> the usb quirks
>
> ### Other Helpful notes
> Article -
> https://fwmotion.com/blog/operating-systems/2020-09-04-installing-fedora-ser...
>
> --
>   Zach Villers
>   [e-mail]
> _______________________________________________
> arm mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]

[Jeremy Linton]

Hi,

On 1/3/21 12:46 PM, Zach Villers wrote:
> I recently got Fedora 33 Server booting on my RPI 4 - 8G using the RPI4 UEFI firmware. The usb3 adapter I used worked fine with Raspbian, but with Fedora 33, there was an issue. I've worked around this issue and another involving F33 complaining about the lack of sd card. These are my brief notes, hopefully they help someone. If a longer write up or a bug report would be helpful, I'm happy to do so. Also looking for feedback if I could have done anything better/different.
>
> Hardware:
> RPI 4B - 8G model
> 128G "Inland" brand ssd - OEM Phison - Model 'SATA AAD'
> Orico Brand SATA to USB3 external enclosure - model 'ORICO 2139U3'
> uses JMicron Technology Corp. JMS578 chip - Sata 6Gb/s
>
> UEFI Firmware: https://github.com/pftf/RPi4/releases/tag/v1.21
> After Fedora install;
> switched advanced settings in UEFI from 'ahci' to 'devicetree'
> removed 3G ram limit

So, I think you meant you switched from acpi to devicetree. This doesn't 
really buy you much at the moment unless you also change the sd config 
to emmc2. That is because linux doesn't know how to change the 
undocumented SD routing, so the sd card is routed to the PIO arasan, and 
the wifi is routed to the emmc2 controller. Hence the piles of messages 
your seeing below. Of course if you change the emmc2 setting you will be 
stuck with the firmware settings until you upgrade firmware versions.

There are patches to fix this: 
https://edk2.groups.io/g/devel/topic/patch_v4_0_7_rpi4_enable/79453713?p=,,,...

as well as the ACPI patches for the SD/wifi. So when all this is done, 
the recommended setting is going to be acpi/emmc2/no 3G limit/PCIe conduit.
> ### Problems
> Pi treats usb drive as uas device causing extremely slow read/write - ext4 fs would not mount rw
>     '''
>         [ 1247.365069] usb 2-1: reset SuperSpeed Gen 1 USB device number 3 using xhci_hcd
>         [ 1247.387663] scsi host0: uas_eh_device_reset_handler success
>         [ 1279.648929] sd 0:0:0:0: [sda] tag#3 uas_eh_abort_handler 0 uas-tag 4 inflight: CMD IN
>         [ 1279.648937] sd 0:0:0:0: [sda] tag#3 CDB: opcode=0x28 28 00 03 fa 64 50 00 01 00 00
>         [ 1279.649085] sd 0:0:0:0: [sda] tag#1 uas_eh_abort_handler 0 uas-tag 2 inflight: CMD IN
>         [ 1279.649091] sd 0:0:0:0: [sda] tag#1 CDB: opcode=0x28 28 00 00 00 10 00 00 04 00 00
>         [ 1279.649553] sd 0:0:0:0: [sda] tag#0 uas_eh_abort_handler 0 uas-tag 1 inflight: CMD IN
>         [ 1279.649559] sd 0:0:0:0: [sda] tag#0 CDB: opcode=0x28 28 00 00 00 0c 00 00 04 00 00
>         [ 1286.049165] sd 0:0:0:0: tag#4 uas_eh_abort_handler 0 uas-tag 5 inflight: CMD
>         [ 1286.049173] sd 0:0:0:0: tag#4 CDB: opcode=0x0 00 00 00 00 00 00
>         [ 1286.065838] scsi host0: uas_eh_device_reset_handler start
>     '''
> Pi is missing sdhc card ( mostly annoying, could've just turned off kernel message echo to console? )
>     '''
> mmc1: Timeout waiting for hardware cmd interrupt.
> mmc1: sdhci: =========== SDHCI REGISTER DUMP ===========
> mmc1: sdhci: blah
>     '''
>
> ### Solutions
> uas device
> add usb quirk to kernel args in /etc/default/grub
> '''
> GRUB_CMDLINE_LINUX="usb-storage.quirks=152d:0578:u"
>    '''
> add quirk to config.txt (is this necessary?)
> rebuild grub with grub2-mkconfig
> dracut -f (not sure if I needed to do this?)
> sdhc card
> block(black)list sdhc drivers
> /etc/modprobe.d/no_sdhc.conf
>  '''
> blacklist sdhci
>  blacklist sdhci_platform
>  blacklist sdhci_iproc
>  install sdhci /bin/false
>  install sdhci_platform /bin/false
>  install sdhci_iproc /bin/false
>    '''
> From [finishing steps for rhel 8](https://access.redhat.com/solutions/41278)
> added 'sdhci.blacklist=1 rd.driver.blacklist=sdhci' to the above GRUB_CMDLINE_LINUX
> rebuilt grub again
> (make a copy of and) rebuilt initramfs
>      'dracut --omit-drivers sdhci -f'
>
> ### Things I didn't do
> fool around with kdump
> Edit the installer kernel CMDLINE to blacklist the sdhci module and add the usb quirks
>
> ### Other Helpful notes
> Article - https://fwmotion.com/blog/operating-systems/2020-09-04-installing-fedora-ser...