Change in kernel PPS GPIO handling?

[Chris Adams]

I have an RPi4 running Fedora 35.  It hadn't been updated in a while, so
I applied updates today.  When I boot to the kernel 5.16.11-200, I lose
the PPS device from my GPS hat.  Boot back to 5.14.16-301.fc35 and it
works.

I'm booting with EFI firmware, and with a device tree config.txt that
has "dtoverlay=pps-gpio,gpiopin=4" in it.  I removed the /boot/dtb
symlink and set /etc/u-boot.conf to not re-add it.

When I boot 5.16, I see /proc/device-tree, and it has
/proc/device-tree/pps@4 in it (and the contents look correct), but
loading the pps-gpio kernel module just gives:

pps-gpio: probe of pps@4 failed with error -22

-- 
Chris Adams <linux(a)cmadams.net&gt;

[Peter Robinson]

>  I have an RPi4 running Fedora 35.  It hadn't been updated in a
> while, so
>  I applied updates today.  When I boot to the kernel 5.16.11-200, I lose
>  the PPS device from my GPS hat.  Boot back to 5.14.16-301.fc35 and it
>  works.
>
>  I'm booting with EFI firmware, and with a device tree config.txt that 
All our firmware are EFI, that's the only way we support booting, is
it the default U-Boot based one or are you using the edk2 one?

>  has "dtoverlay=pps-gpio,gpiopin=4" in it.  I removed the
> /boot/dtb
>  symlink and set /etc/u-boot.conf to not re-add it.
>
>  When I boot 5.16, I see /proc/device-tree, and it has
>  /proc/device-tree/pps@4 in it (and the contents look correct), but
>  loading the pps-gpio kernel module just gives:
>
>  pps-gpio: probe of pps@4 failed with error -22 
Any chance you can tell us which kernel it started with, 5.14.x to
5.16.x is a big window to debug and looking at kernel logs for
drivers/pps there's been no changes in that space in the 5.15+ kernels
at all.

I wonder if it's a change/regression in the firmware overlay <->
kernel interface.

[Stefan Wahren]

Hi Chris,

Am 02.03.22 um 10:51 schrieb Peter Robinson:
> > I have an RPi4 running Fedora 35.  It hadn't been updated in
> a while, so
> > I applied updates today.  When I boot to the kernel 5.16.11-200, I lose
> > the PPS device from my GPS hat.  Boot back to 5.14.16-301.fc35 and it
> > works.
> >
> > I'm booting with EFI firmware, and with a device tree config.txt that
>  All our firmware are EFI, that's the only way we support booting, is
>  it the default U-Boot based one or are you using the edk2 one?
>
> > has "dtoverlay=pps-gpio,gpiopin=4" in it.  I removed the /boot/dtb
> > symlink and set /etc/u-boot.conf to not re-add it.
> >
> > When I boot 5.16, I see /proc/device-tree, and it has
> > /proc/device-tree/pps@4 in it (and the contents look correct), but
> > loading the pps-gpio kernel module just gives:
> >
> > pps-gpio: probe of pps@4 failed with error -22
>  Any chance you can tell us which kernel it started with, 5.14.x to
>  5.16.x is a big window to debug and looking at kernel logs for
>  drivers/pps there's been no changes in that space in the 5.15+ kernels
>  at all.
>
>  I wonder if it's a change/regression in the firmware overlay <->
>  kernel interface. 
do you use the DTB file from 5.14.x or 5.16x?

I guess this is related to this commit:

https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?...

Unfortunately this is a change which requires this DTS fix:

https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?...

So DTB and kernel must "match".

> _______________________________________________
> arm mailing list -- arm(a)lists.fedoraproject.org
> To unsubscribe send an email to arm-leave(a)lists.fedoraproject.org
> Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
https://lists.fedoraproject.org/archives/list/[e-mail]
> Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure

[Peter Robinson]

Hi Stefan,

>  Am 02.03.22 um 10:51 schrieb Peter Robinson:
>  >> I have an RPi4 running Fedora 35.  It hadn't been updated in a while, so
>  >> I applied updates today.  When I boot to the kernel 5.16.11-200, I lose
>  >> the PPS device from my GPS hat.  Boot back to 5.14.16-301.fc35 and it
>  >> works.
>  >>
>  >> I'm booting with EFI firmware, and with a device tree config.txt that
>  > All our firmware are EFI, that's the only way we support booting, is
>  > it the default U-Boot based one or are you using the edk2 one?
>  >
>  >> has "dtoverlay=pps-gpio,gpiopin=4" in it.  I removed the /boot/dtb
>  >> symlink and set /etc/u-boot.conf to not re-add it.
>  >>
>  >> When I boot 5.16, I see /proc/device-tree, and it has
>  >> /proc/device-tree/pps@4 in it (and the contents look correct), but
>  >> loading the pps-gpio kernel module just gives:
>  >>
>  >> pps-gpio: probe of pps@4 failed with error -22
>  > Any chance you can tell us which kernel it started with, 5.14.x to
>  > 5.16.x is a big window to debug and looking at kernel logs for
>  > drivers/pps there's been no changes in that space in the 5.15+ kernels
>  > at all.
>  >
>  > I wonder if it's a change/regression in the firmware overlay <->
>  > kernel interface.
>
>  do you use the DTB file from 5.14.x or 5.16x? 
I'm guessing this is fixed in the RPi firmware DTs with their rebase to 5.15.x?

I've not pushed newer firmware/DT to stable Fedora of late as we've
had a bunch of regressions on various RPi devices, currently upstream
firmware on Fedora doesn't boot on any 32 bit RPi device for example.

> I guess this is related to this commit:
>
>
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?...
>
> Unfortunately this is a change which requires this DTS fix:
>
>
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?...
>
> So DTB and kernel must "match".
>
> > _______________________________________________
> > arm mailing list -- arm(a)lists.fedoraproject.org
> > To unsubscribe send an email to arm-leave(a)lists.fedoraproject.org
> > Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives:
https://lists.fedoraproject.org/archives/list/[e-mail]
> > Do not reply to spam on the list, report it:
https://pagure.io/fedora-infrastructure

[Chris Adams]

Once upon a time, Stefan Wahren <stefan.wahren(a)i2se.com&gt; said:
>  do you use the DTB file from 5.14.x or 5.16x? 
I'm booting with EFI firmware from https://github.com/pftf/RPi4 and
using the DTB file from that I guess.

The reason I'm booting EFI is:

- the Adafruit GPS hat connects to the serial lines, and the standard
  firmware would see input and stop boot

- it looked like I could change that behavior if I booted from an SD
  card by saving a variable file there

- but if I put in an SD card, Fedora wouldn't boot, just got a loop of
  timeouts

So I found that I could put the EFI firmware (and Fedora) on a USB
drive.  And then it looked like I could only use a config.txt to enable
the PPS if I didn't use the kernel dtb.

So, what's my best path forward?  Can I copy the bcm2711-rpi-4-b.dtb
file from the dtb-5.16.11-200.fc35.aarch64 directory to the efi
directory (and then just remember to swap back to the EFI-provided file
if I boot an older kernel)?

Sorry, this is my only Pi, so still learning about this side of things.
:)

-- 
Chris Adams <linux(a)cmadams.net&gt;

[Stefan Wahren]

Hi Peter,

Am 02.03.22 um 14:37 schrieb Peter Robinson:
>  Hi Stefan,
>
> > Am 02.03.22 um 10:51 schrieb Peter Robinson:
> >>> I have an RPi4 running Fedora 35.  It hadn't been updated in a while, so
> >>> I applied updates today.  When I boot to the kernel 5.16.11-200, I lose
> >>> the PPS device from my GPS hat.  Boot back to 5.14.16-301.fc35 and it
> >>> works.
> >>>
> >>> I'm booting with EFI firmware, and with a device tree config.txt that
> >> All our firmware are EFI, that's the only way we support booting, is
> >> it the default U-Boot based one or are you using the edk2 one?
> >>
> >>> has "dtoverlay=pps-gpio,gpiopin=4" in it.  I removed the /boot/dtb
> >>> symlink and set /etc/u-boot.conf to not re-add it.
> >>>
> >>> When I boot 5.16, I see /proc/device-tree, and it has
> >>> /proc/device-tree/pps@4 in it (and the contents look correct), but
> >>> loading the pps-gpio kernel module just gives:
> >>>
> >>> pps-gpio: probe of pps@4 failed with error -22
> >> Any chance you can tell us which kernel it started with, 5.14.x to
> >> 5.16.x is a big window to debug and looking at kernel logs for
> >> drivers/pps there's been no changes in that space in the 5.15+ kernels
> >> at all.
> >>
> >> I wonder if it's a change/regression in the firmware overlay <->
> >> kernel interface.
> > do you use the DTB file from 5.14.x or 5.16x?
>  I'm guessing this is fixed in the RPi firmware DTs with their rebase to 5.15.x?
>
>  I've not pushed newer firmware/DT to stable Fedora of late as we've
>  had a bunch of regressions on various RPi devices, currently upstream
>  firmware on Fedora doesn't boot on any 32 bit RPi device for example. 
Please
never ever use the DTS from the Raspberry Pi tree in combination
with the mainline kernel. Please use the DTS from mainline tree. Any
other combination is not tested and has unexpected results.
>
>> I guess this is related to this commit:
>>
>>
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?...
>>
>> Unfortunately this is a change which requires this DTS fix:
>>
>>
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?...
>>
>> So DTB and kernel must "match".
>>
>>> _______________________________________________
>>> arm mailing list -- arm(a)lists.fedoraproject.org
>>> To unsubscribe send an email to arm-leave(a)lists.fedoraproject.org
>>> Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>>> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>>> List Archives:
https://lists.fedoraproject.org/archives/list/[e-mail]
>>> Do not reply to spam on the list, report it:
https://pagure.io/fedora-infrastructure

[Peter Robinson]

>  >>> I wonder if it's a change/regression in the firmware
> overlay <->
>  >>> kernel interface.
>  >> do you use the DTB file from 5.14.x or 5.16x?
>  > I'm guessing this is fixed in the RPi firmware DTs with their rebase to 5.15.x?
>  >
>  > I've not pushed newer firmware/DT to stable Fedora of late as we've
>  > had a bunch of regressions on various RPi devices, currently upstream
>  > firmware on Fedora doesn't boot on any 32 bit RPi device for example.
>  Please never ever use the DTS from the Raspberry Pi tree in combination
>  with the mainline kernel. Please use the DTS from mainline tree. Any
>  other combination is not tested and has unexpected results. 
In the case of arm32 it's all using upstream kernel/DT with RPi
firmware it still has unexpected results. Either way it's a game of
whack-a-mole, we also have cases where the Firmware DT works and the
upstream kernel DT does not [1] but I've not had time to debug that
one yet.

[1] https://bugzilla.redhat.com/show_bug.cgi?id=2011423

[Stefan Wahren]

Hi Peter,

Am 02.03.22 um 15:19 schrieb Peter Robinson:
> >>>> I wonder if it's a change/regression in the
> firmware overlay <->
> >>>> kernel interface.
> >>> do you use the DTB file from 5.14.x or 5.16x?
> >> I'm guessing this is fixed in the RPi firmware DTs with their rebase to
> 5.15.x?
> >>
> >> I've not pushed newer firmware/DT to stable Fedora of late as we've
> >> had a bunch of regressions on various RPi devices, currently upstream
> >> firmware on Fedora doesn't boot on any 32 bit RPi device for example.
> > Please never ever use the DTS from the Raspberry Pi tree in combination
> > with the mainline kernel. Please use the DTS from mainline tree. Any
> > other combination is not tested and has unexpected results.
>  In the case of arm32 it's all using upstream kernel/DT with RPi
>  firmware it still has unexpected results. Either way it's a game of
>  whack-a-mole, we also have cases where the Firmware DT works and the
>  upstream kernel DT does not [1] but I've not had time to debug that
>  one yet. 
i saw these "invalid bus width" report, but was never able to reproduce
it with my setup (upstream kernel + upstream DTS + RPI OS rootfs for 32
and 64 bit w/o u-boot or EFI). Are you able to reproduce it with any
BCM2711 platform?

Looks like the EFI leaves the SDHCI part in a unexpected state and
kernel doesn't properly handover.

Is there a good starting point which image i should try first?

Nevertheless these many possible sources of DTBs are a nightmare. At
least i plan to submit a patch to print a kernel warning in case of
missing gpio-ranges. So we can identify this issue faster.

>  [1] https://bugzilla.redhat.com/show_bug.cgi?id=2011423

[Chris Adams]

Once upon a time, Stefan Wahren <stefan.wahren(a)i2se.com&gt; said:
>  Hi Chris,
>  
>  Am 02.03.22 um 10:51 schrieb Peter Robinson:
>  >> I have an RPi4 running Fedora 35.  It hadn't been updated in a while, so
>  >> I applied updates today.  When I boot to the kernel 5.16.11-200, I lose
>  >> the PPS device from my GPS hat.  Boot back to 5.14.16-301.fc35 and it
>  >> works.
>  >>
>  >> I'm booting with EFI firmware, and with a device tree config.txt that
>  > All our firmware are EFI, that's the only way we support booting, is
>  > it the default U-Boot based one or are you using the edk2 one?
>  >
>  >> has "dtoverlay=pps-gpio,gpiopin=4" in it.  I removed the /boot/dtb
>  >> symlink and set /etc/u-boot.conf to not re-add it.
>  >>
>  >> When I boot 5.16, I see /proc/device-tree, and it has
>  >> /proc/device-tree/pps@4 in it (and the contents look correct), but
>  >> loading the pps-gpio kernel module just gives:
>  >>
>  >> pps-gpio: probe of pps@4 failed with error -22
>  > Any chance you can tell us which kernel it started with, 5.14.x to
>  > 5.16.x is a big window to debug and looking at kernel logs for
>  > drivers/pps there's been no changes in that space in the 5.15+ kernels
>  > at all.
>  >
>  > I wonder if it's a change/regression in the firmware overlay <->
>  > kernel interface.
>  
>  do you use the DTB file from 5.14.x or 5.16x?
>  
>  I guess this is related to this commit:
>  
>
> https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?...
>  
>  Unfortunately this is a change which requires this DTS fix:
>  
>
> https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?...
>  
>  So DTB and kernel must "match". 
Okay, so what's the best way to do that?

I have EFI firmware from https://github.com/pftf/RPi4 in /boot/efi, and
the /boot/dtb symlink removed.  The bcm2711-rpi-4-b.dtb file in
/boot/efi is from the RPi4_UEFI_Firmware_v1.32.zip file.  I have a
config.txt in /boot/efi that references overlays in /boot/efi/overlays
from RPM bcm283x-overlays (like pps-gpio)..

I'm not quite sure how I need to fit all this together.
-- 
Chris Adams <linux(a)cmadams.net&gt;

[Jeffrey Walton]

On Wed, Mar 2, 2022 at 11:45 AM Stefan Wahren <stefan.wahren(a)i2se.com&gt; wrote:
> ...
>  Nevertheless these many possible sources of DTBs are a nightmare. At
>  least i plan to submit a patch to print a kernel warning in case of
>  missing gpio-ranges. So we can identify this issue faster. 
A root cause analysis may reveal that the DTBs need a unique name and
version (and possibly vendor) to ensure an expected DTB(s) is used
with a particular kernel. It will probably help put an end to the
madness.

I don't know enough about the architecture so I can't say.

Jeff

[John Florian]

I was pointed here from https://bugzilla.redhat.com/show_bug.cgi?id=2060490 and this does
look similiar to my issue though I'm doing nothing special with DTs.  Plain old
vanilla Fedora was working for me perfectly up through 5.15.11-200.fc35.aarch64 and the
problems (for me) began with 5.15.12-200.fc35.aarch64.

Please let me know if there's anything I can provide or try.

[Chris Adams]

I poked at this some more tonight, starting with a fresh SD card install
of F35.  This is in a Pi 4B Rev 1.4 with 4G RAM, and starting with the
initial F35 kernel, 5.14.10-300.fc35.aarch64.

I can boot that kernel from a card in the on-board SD slot, but then it
gets stuck trying to read from that card, with looping errors like:

   mmc0: error -22 whilst initialising SD card
   mmc0: invalid bus width

So I moved the card to a USB adapter.  This is a problem for me, because
then I can't set the uboot environment variable to disable stdin from
the serial port (so my GPS HAT input on the serial port stops boot);
that's why I was using the EDK2 EFI firmware, but for these tests, I
stuck with the Fedora-provided uboot firmware and just typed "boot"
every time.

When I remove the /boot/dtb symlink and edit /boot/efi/config.txt to
comment out the "dtoverlay=upstream-pi4" lien and add the following:

   dtoverlay=disable-bt
   dtoverlay=disable-wifi
   dtoverlay=pps-gpio,gpiopin=4

I get my PPS and ttyACM0 devices.  Yay, desired outcome.

Next, I updated the kernel (to 5.16.11-200.fc35) and uboot-images-armv8
and ran rpi-uboot-update.  After reboot, I don't get /dev/pps0, and
there's the error:

   pps-gpio: probe of pps@4 failed with error -22

So, from earlier responses, it seems there's some issue with the
different device trees, included with the kernel vs. not?  If I use the
kernel-included device tree (I guess by not removing /boot/dtb), how do
I get the overlays so I can use my GPS hat?

Interesting, when I boot 5.16.11-200.fc35 with the /boot/dtb symlink, I
lose HDMI signal a little after the kernel loads (I've removed "rhgb
quiet" from the boot arg).  I can see some kernel messages, but then I
lose HDMI.  SSHing in, I see in dmesg:

   [drm] Initialized vc4 0.0.0 20140616 for gpu on minor 0
   Unable to handle kernel read from unreadable memory at virtual address
0000000000000348

and so on with a kernel Oops dump.  So it seems the kernel-included
device tree has its own problems on a Pi 4.

-- 
Chris Adams <linux(a)cmadams.net&gt;

[Stefan Wahren]

Am 03.03.22 um 02:29 schrieb Chris Adams:
>  Once upon a time, Stefan Wahren <stefan.wahren(a)i2se.com&gt; said:
> > Hi Chris,
> >
> > Am 02.03.22 um 10:51 schrieb Peter Robinson:
> >>> I have an RPi4 running Fedora 35.  It hadn't been updated in a while, so
> >>> I applied updates today.  When I boot to the kernel 5.16.11-200, I lose
> >>> the PPS device from my GPS hat.  Boot back to 5.14.16-301.fc35 and it
> >>> works.
> >>>
> >>> I'm booting with EFI firmware, and with a device tree config.txt that
> >> All our firmware are EFI, that's the only way we support booting, is
> >> it the default U-Boot based one or are you using the edk2 one?
> >>
> >>> has "dtoverlay=pps-gpio,gpiopin=4" in it.  I removed the /boot/dtb
> >>> symlink and set /etc/u-boot.conf to not re-add it.
> >>>
> >>> When I boot 5.16, I see /proc/device-tree, and it has
> >>> /proc/device-tree/pps@4 in it (and the contents look correct), but
> >>> loading the pps-gpio kernel module just gives:
> >>>
> >>> pps-gpio: probe of pps@4 failed with error -22
> >> Any chance you can tell us which kernel it started with, 5.14.x to
> >> 5.16.x is a big window to debug and looking at kernel logs for
> >> drivers/pps there's been no changes in that space in the 5.15+ kernels
> >> at all.
> >>
> >> I wonder if it's a change/regression in the firmware overlay <->
> >> kernel interface.
> > do you use the DTB file from 5.14.x or 5.16x?
> >
> > I guess this is related to this commit:
> >
> >
> https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?...
> >
> > Unfortunately this is a change which requires this DTS fix:
> >
> >
> https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?...
> >
> > So DTB and kernel must "match".
>  Okay, so what's the best way to do that?
>
>  I have EFI firmware from https://github.com/pftf/RPi4 in /boot/efi, and
>  the /boot/dtb symlink removed.  The bcm2711-rpi-4-b.dtb file in
>  /boot/efi is from the RPi4_UEFI_Firmware_v1.32.zip file.  I have a
>  config.txt in /boot/efi that references overlays in /boot/efi/overlays
>  from RPM bcm283x-overlays (like pps-gpio)..
>
>  I'm not quite sure how I need to fit all this together. 
Let me recap the situation. mainline and rpi vendor tree are independent
in development. Synchronization (incl. device tree) happens in both
directions.

U-Boot bootloader decided to keep an own copy of the mainline DT files
(which is currently based on an older 5.15 version which lacks the
gpio-ranges property). According to this statement [1] the EDK2 UEFI
decided to use the rpi vendor tree and don't care about the mainline DT
files.

I'm sorry but as a spare-time kernel developer, i don't have to time to
fight against all this mess.

I hope i will have some time for debugging in the near future ...

[1] - https://github.com/pftf/RPi4/issues/193

[Stefan Wahren]

Am 04.03.22 um 21:50 schrieb Stefan Wahren:
>  Am 03.03.22 um 02:29 schrieb Chris Adams:
> > Once upon a time, Stefan Wahren <stefan.wahren(a)i2se.com&gt; said:
> >> Hi Chris,
> >>
> >> Am 02.03.22 um 10:51 schrieb Peter Robinson:
> >>>> I have an RPi4 running Fedora 35.  It hadn't been updated in a while,
> so
> >>>> I applied updates today.  When I boot to the kernel 5.16.11-200, I lose
> >>>> the PPS device from my GPS hat.  Boot back to 5.14.16-301.fc35 and it
> >>>> works.
> >>>>
> >>>> I'm booting with EFI firmware, and with a device tree config.txt
> that
> >>> All our firmware are EFI, that's the only way we support booting, is
> >>> it the default U-Boot based one or are you using the edk2 one?
> >>>
> >>>> has "dtoverlay=pps-gpio,gpiopin=4" in it.  I removed the
> /boot/dtb
> >>>> symlink and set /etc/u-boot.conf to not re-add it.
> >>>>
> >>>> When I boot 5.16, I see /proc/device-tree, and it has
> >>>> /proc/device-tree/pps@4 in it (and the contents look correct), but
> >>>> loading the pps-gpio kernel module just gives:
> >>>>
> >>>> pps-gpio: probe of pps@4 failed with error -22
> >>> Any chance you can tell us which kernel it started with, 5.14.x to
> >>> 5.16.x is a big window to debug and looking at kernel logs for
> >>> drivers/pps there's been no changes in that space in the 5.15+ kernels
> >>> at all.
> >>>
> >>> I wonder if it's a change/regression in the firmware overlay <->
> >>> kernel interface.
> >> do you use the DTB file from 5.14.x or 5.16x?
> >>
> >> I guess this is related to this commit:
> >>
> >>
> https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?...
> >>
> >> Unfortunately this is a change which requires this DTS fix:
> >>
> >>
> https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?...
> >>
> >> So DTB and kernel must "match".
> > Okay, so what's the best way to do that?
> >
> > I have EFI firmware from https://github.com/pftf/RPi4 in /boot/efi, and
> > the /boot/dtb symlink removed.  The bcm2711-rpi-4-b.dtb file in
> > /boot/efi is from the RPi4_UEFI_Firmware_v1.32.zip file.  I have a
> > config.txt in /boot/efi that references overlays in /boot/efi/overlays
> > from RPM bcm283x-overlays (like pps-gpio)..
> >
> > I'm not quite sure how I need to fit all this together.
>  Let me recap the situation. mainline and rpi vendor tree are independent
>  in development. Synchronization (incl. device tree) happens in both
>  directions.
>
>  U-Boot bootloader decided to keep an own copy of the mainline DT files
>  (which is currently based on an older 5.15 version which lacks the
>  gpio-ranges property). According to this statement [1] the EDK2 UEFI
>  decided to use the rpi vendor tree and don't care about the mainline DT
>  files.
>
>  I'm sorry but as a spare-time kernel developer, i don't have to time to
>  fight against all this mess.
>
>  I hope i will have some time for debugging in the near future ...
>
>  [1] - https://github.com/pftf/RPi4/issues/193
>  
Today i tried Fedora 35 Minimal for my first time, here are my test results:

Raspberry Pi 400 32 bit => bus width issue
Raspberry Pi 400 64 bit => hangs while show graphics
Raspberry Pi 4 B 4 GB 32 bit => kernel crash
Raspberry Pi 4 B 4 GB 64 bit => boot into setup
Raspberry Pi 4 B 8 GB 32 bit => black screen, fails to start kernel?
Raspberry Pi 4 B 8 GB 64 bit => boot into setup

At least the 32 bit issues on Raspberry Pi 4 are expected since the
kernel config doesn't have ARM_LPAE enabled.

[Stefan Wahren]

Am 05.03.22 um 12:34 schrieb Stefan Wahren:
>  Am 04.03.22 um 21:50 schrieb Stefan Wahren:
> > Am 03.03.22 um 02:29 schrieb Chris Adams:
> >> Once upon a time, Stefan Wahren <stefan.wahren(a)i2se.com&gt; said:
> >>> Hi Chris,
> >>>
> >>> Am 02.03.22 um 10:51 schrieb Peter Robinson:
> >>>>> I have an RPi4 running Fedora 35.  It hadn't been updated in a
> while, so
> >>>>> I applied updates today.  When I boot to the kernel 5.16.11-200, I
> lose
> >>>>> the PPS device from my GPS hat.  Boot back to 5.14.16-301.fc35 and
> it
> >>>>> works.
> >>>>>
> >>>>> I'm booting with EFI firmware, and with a device tree config.txt
> that
> >>>> All our firmware are EFI, that's the only way we support booting, is
> >>>> it the default U-Boot based one or are you using the edk2 one?
> >>>>
> >>>>> has "dtoverlay=pps-gpio,gpiopin=4" in it.  I removed the
> /boot/dtb
> >>>>> symlink and set /etc/u-boot.conf to not re-add it.
> >>>>>
> >>>>> When I boot 5.16, I see /proc/device-tree, and it has
> >>>>> /proc/device-tree/pps@4 in it (and the contents look correct), but
> >>>>> loading the pps-gpio kernel module just gives:
> >>>>>
> >>>>> pps-gpio: probe of pps@4 failed with error -22
> >>>> Any chance you can tell us which kernel it started with, 5.14.x to
> >>>> 5.16.x is a big window to debug and looking at kernel logs for
> >>>> drivers/pps there's been no changes in that space in the 5.15+
> kernels
> >>>> at all.
> >>>>
> >>>> I wonder if it's a change/regression in the firmware overlay
> <->
> >>>> kernel interface.
> >>> do you use the DTB file from 5.14.x or 5.16x?
> >>>
> >>> I guess this is related to this commit:
> >>>
> >>>
> https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?...
> >>>
> >>> Unfortunately this is a change which requires this DTS fix:
> >>>
> >>>
> https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?...
> >>>
> >>> So DTB and kernel must "match".
> >> Okay, so what's the best way to do that?
> >>
> >> I have EFI firmware from https://github.com/pftf/RPi4 in /boot/efi, and
> >> the /boot/dtb symlink removed.  The bcm2711-rpi-4-b.dtb file in
> >> /boot/efi is from the RPi4_UEFI_Firmware_v1.32.zip file.  I have a
> >> config.txt in /boot/efi that references overlays in /boot/efi/overlays
> >> from RPM bcm283x-overlays (like pps-gpio)..
> >>
> >> I'm not quite sure how I need to fit all this together.
> > Let me recap the situation. mainline and rpi vendor tree are independent
> > in development. Synchronization (incl. device tree) happens in both
> > directions.
> >
> > U-Boot bootloader decided to keep an own copy of the mainline DT files
> > (which is currently based on an older 5.15 version which lacks the
> > gpio-ranges property). According to this statement [1] the EDK2 UEFI
> > decided to use the rpi vendor tree and don't care about the mainline DT
> > files.
> >
> > I'm sorry but as a spare-time kernel developer, i don't have to time to
> > fight against all this mess.
> >
> > I hope i will have some time for debugging in the near future ...
> >
> > [1] - https://github.com/pftf/RPi4/issues/193
> >
>  Today i tried Fedora 35 Minimal for my first time, here are my test results:
>
>  Raspberry Pi 400 32 bit => bus width issue
>  Raspberry Pi 400 64 bit => hangs while show graphics
>  Raspberry Pi 4 B 4 GB 32 bit => kernel crash
>  Raspberry Pi 4 B 4 GB 64 bit => boot into setup
>  Raspberry Pi 4 B 8 GB 32 bit => black screen, fails to start kernel?
>  Raspberry Pi 4 B 8 GB 64 bit => boot into setup
>
>  At least the 32 bit issues on Raspberry Pi 4 are expected since the
>  kernel config doesn't have ARM_LPAE enabled.
>  
Okay, here is the explanation for the different behavior on Raspberry Pi
400 and Raspberry Pi 4 B. The Raspberry Pi 400 has a newer BCM2711 SoC
(Stepping C0), which have less DMA restrictions for the emmc2 interface
(responsible for SD card access). For the Raspberry Pi 4 B there are
older boards which have Stepping B0 and all the new boards should have
Stepping C0 [1].

Unfortunately there is no 100% reliable way to detect the stepping from
the kernel side. So currently the Raspberry Pi firmware patches the
dma-ranges in the firmware DT [2]. So in case U-Boot [3] or another
bootloader ignores this firmware DT and read a fresh DTB the right
dma-ranges get lost. Finally this results in unexpected behavior as soon
the emmc2 switches to DMA mode [4].

Best regards

[1] -
https://www.jeffgeerling.com/blog/2021/raspberry-pi-4-model-bs-arriving-n...

[2] -
https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/commi...

[3] - https://forums.raspberrypi.com/viewtopic.php?t=319125

[4] - https://bugzilla.redhat.com/show_bug.cgi?id=2011423

Change in kernel PPS GPIO handling and Pi 3B+

[David W. Legg]

Just to supplement Stephan Wahren's observations about Pi 4s:-

I have tried the latest beta F36 with my Pi 3B+, and the GPIOs do not 
seem to work (yet):

i.e. Fedora-Minimal-36_Beta-1.4.aarch64.raw.xz

The command, gpioset `gpiofind GPIO27`=0, seems to do nothing, as does 
'=1' instead of the '=0'.

The latest kernel I have where GPIOs and the above gpioset command work 
is the F35 5.14.10-300 aarch64 one. Later F35 kernels do not work (yet).

Using the 5.14.10-300 F35 kernel, the command, 'gpioset `gpiofind 
GPIO27`=0', raises the voltage on the GPIO pin to 3.3V, rather than 
lowering it as I would have expected.

Hope this helps some-one somewhere ...

:D


On 05/03/2022 11:34, Stefan Wahren wrote:
> Today i tried Fedora 35 Minimal for my first time, here are my test results:
>
> Raspberry Pi 400 32 bit => bus width issue
> Raspberry Pi 400 64 bit => hangs while show graphics
> Raspberry Pi 4 B 4 GB 32 bit => kernel crash
> Raspberry Pi 4 B 4 GB 64 bit => boot into setup
> Raspberry Pi 4 B 8 GB 32 bit => black screen, fails to start kernel?
> Raspberry Pi 4 B 8 GB 64 bit => boot into setup
>
> At least the 32 bit issues on Raspberry Pi 4 are expected since the
> kernel config doesn't have ARM_LPAE enabled.

[Stefan Wahren]

Am 11.03.22 um 17:37 schrieb Stefan Wahren:
>  Am 05.03.22 um 12:34 schrieb Stefan Wahren:
> > Am 04.03.22 um 21:50 schrieb Stefan Wahren:
> >> Am 03.03.22 um 02:29 schrieb Chris Adams:
> >>> Once upon a time, Stefan Wahren <stefan.wahren(a)i2se.com&gt; said:
> >>>> Hi Chris,
> >>>>
> >>>> Am 02.03.22 um 10:51 schrieb Peter Robinson:
> >>>>>> I have an RPi4 running Fedora 35.  It hadn't been updated in
> a while, so
> >>>>>> I applied updates today.  When I boot to the kernel 5.16.11-200,
> I lose
> >>>>>> the PPS device from my GPS hat.  Boot back to 5.14.16-301.fc35
> and it
> >>>>>> works.
> >>>>>>
> >>>>>> I'm booting with EFI firmware, and with a device tree
> config.txt that
> >>>>> All our firmware are EFI, that's the only way we support booting,
> is
> >>>>> it the default U-Boot based one or are you using the edk2 one?
> >>>>>
> >>>>>> has "dtoverlay=pps-gpio,gpiopin=4" in it.  I removed
> the /boot/dtb
> >>>>>> symlink and set /etc/u-boot.conf to not re-add it.
> >>>>>>
> >>>>>> When I boot 5.16, I see /proc/device-tree, and it has
> >>>>>> /proc/device-tree/pps@4 in it (and the contents look correct),
> but
> >>>>>> loading the pps-gpio kernel module just gives:
> >>>>>>
> >>>>>> pps-gpio: probe of pps@4 failed with error -22
> >>>>> Any chance you can tell us which kernel it started with, 5.14.x to
> >>>>> 5.16.x is a big window to debug and looking at kernel logs for
> >>>>> drivers/pps there's been no changes in that space in the 5.15+
> kernels
> >>>>> at all.
> >>>>>
> >>>>> I wonder if it's a change/regression in the firmware overlay
> <->
> >>>>> kernel interface.
> >>>> do you use the DTB file from 5.14.x or 5.16x?
> >>>>
> >>>> I guess this is related to this commit:
> >>>>
> >>>>
> https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?...
> >>>>
> >>>> Unfortunately this is a change which requires this DTS fix:
> >>>>
> >>>>
> https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?...
> >>>>
> >>>> So DTB and kernel must "match".
> >>> Okay, so what's the best way to do that?
> >>>
> >>> I have EFI firmware from https://github.com/pftf/RPi4 in /boot/efi, and
> >>> the /boot/dtb symlink removed.  The bcm2711-rpi-4-b.dtb file in
> >>> /boot/efi is from the RPi4_UEFI_Firmware_v1.32.zip file.  I have a
> >>> config.txt in /boot/efi that references overlays in /boot/efi/overlays
> >>> from RPM bcm283x-overlays (like pps-gpio)..
> >>>
> >>> I'm not quite sure how I need to fit all this together.
> >> Let me recap the situation. mainline and rpi vendor tree are independent
> >> in development. Synchronization (incl. device tree) happens in both
> >> directions.
> >>
> >> U-Boot bootloader decided to keep an own copy of the mainline DT files
> >> (which is currently based on an older 5.15 version which lacks the
> >> gpio-ranges property). According to this statement [1] the EDK2 UEFI
> >> decided to use the rpi vendor tree and don't care about the mainline DT
> >> files.
> >>
> >> I'm sorry but as a spare-time kernel developer, i don't have to time to
> >> fight against all this mess.
> >>
> >> I hope i will have some time for debugging in the near future ...
> >>
> >> [1] - https://github.com/pftf/RPi4/issues/193
> >>
> > Today i tried Fedora 35 Minimal for my first time, here are my test results:
> >
> > Raspberry Pi 400 32 bit => bus width issue
> > Raspberry Pi 400 64 bit => hangs while show graphics
> > Raspberry Pi 4 B 4 GB 32 bit => kernel crash
> > Raspberry Pi 4 B 4 GB 64 bit => boot into setup
> > Raspberry Pi 4 B 8 GB 32 bit => black screen, fails to start kernel?
> > Raspberry Pi 4 B 8 GB 64 bit => boot into setup
> >
> > At least the 32 bit issues on Raspberry Pi 4 are expected since the
> > kernel config doesn't have ARM_LPAE enabled.
> >
>  Okay, here is the explanation for the different behavior on Raspberry Pi
>  400 and Raspberry Pi 4 B. The Raspberry Pi 400 has a newer BCM2711 SoC
>  (Stepping C0), which have less DMA restrictions for the emmc2 interface
>  (responsible for SD card access). For the Raspberry Pi 4 B there are
>  older boards which have Stepping B0 and all the new boards should have
>  Stepping C0 [1].
>
>  Unfortunately there is no 100% reliable way to detect the stepping from
>  the kernel side. So currently the Raspberry Pi firmware patches the
>  dma-ranges in the firmware DT [2]. So in case U-Boot [3] or another
>  bootloader ignores this firmware DT and read a fresh DTB the right
>  dma-ranges get lost. Finally this results in unexpected behavior as soon
>  the emmc2 switches to DMA mode [4].
>  
Okay, at least i found a fix [1] for the 64 bit boot issues (Original 
Fedora 35, Linux 5.14) with RPi 400/RPi 4 Stepping C0. This requires the 
DTB files to be updated and U-Boot to choose between the B0 and the C0 
variant of the Rpi 4 DTB file.

Unfortunately this doesn't fix the SD card issues on 32 bit.

Best regards

[1] - 
https://github.com/lategoodbye/rpi-zero/commit/d3b2bcf1ca8c0d21625fbf11d0...

[Peter Robinson]

Hi Stefan,

>  >> At least the 32 bit issues on Raspberry Pi 4 are expected
> since the
>  >> kernel config doesn't have ARM_LPAE enabled.
>  >>
>  > Okay, here is the explanation for the different behavior on Raspberry Pi
>  > 400 and Raspberry Pi 4 B. The Raspberry Pi 400 has a newer BCM2711 SoC
>  > (Stepping C0), which have less DMA restrictions for the emmc2 interface
>  > (responsible for SD card access). For the Raspberry Pi 4 B there are
>  > older boards which have Stepping B0 and all the new boards should have
>  > Stepping C0 [1].
>  >
>  > Unfortunately there is no 100% reliable way to detect the stepping from
>  > the kernel side. So currently the Raspberry Pi firmware patches the
>  > dma-ranges in the firmware DT [2]. So in case U-Boot [3] or another
>  > bootloader ignores this firmware DT and read a fresh DTB the right
>  > dma-ranges get lost. Finally this results in unexpected behavior as soon
>  > the emmc2 switches to DMA mode [4].
>  >
>  Okay, at least i found a fix [1] for the 64 bit boot issues (Original
>  Fedora 35, Linux 5.14) with RPi 400/RPi 4 Stepping C0. This requires the
>  DTB files to be updated and U-Boot to choose between the B0 and the C0
>  variant of the Rpi 4 DTB file. 
Thanks for the update on this, great news. In the case of the RPi-400
are they all the C0 stepping? I don't see the 400 in your patch.

Is there a documented way of detecting the stepping that can be used
in U-Boot I can use to create a patch there, or did you have a test
patch for this?

Do you plan to send this patch upstream, if so feel free to add me to
the cc and in the mean time I'll test it.

>  Unfortunately this doesn't fix the SD card issues on 32 bit.
That's less of a problem, Fedora 36 will be the last version of Fedora
to support 32 bit and I've always suggested people run 64 bit Fedora
on their RPi4 and most people do tend to do that.

Thanks,
Peter

[Stefan Wahren]

Hi Peter,

Am 31.03.22 um 15:19 schrieb Peter Robinson:
>  Hi Stefan,
>
> >>> At least the 32 bit issues on Raspberry Pi 4 are expected since the
> >>> kernel config doesn't have ARM_LPAE enabled.
> >>>
> >> Okay, here is the explanation for the different behavior on Raspberry Pi
> >> 400 and Raspberry Pi 4 B. The Raspberry Pi 400 has a newer BCM2711 SoC
> >> (Stepping C0), which have less DMA restrictions for the emmc2 interface
> >> (responsible for SD card access). For the Raspberry Pi 4 B there are
> >> older boards which have Stepping B0 and all the new boards should have
> >> Stepping C0 [1].
> >>
> >> Unfortunately there is no 100% reliable way to detect the stepping from
> >> the kernel side. So currently the Raspberry Pi firmware patches the
> >> dma-ranges in the firmware DT [2]. So in case U-Boot [3] or another
> >> bootloader ignores this firmware DT and read a fresh DTB the right
> >> dma-ranges get lost. Finally this results in unexpected behavior as soon
> >> the emmc2 switches to DMA mode [4].
> >>
> > Okay, at least i found a fix [1] for the 64 bit boot issues (Original
> > Fedora 35, Linux 5.14) with RPi 400/RPi 4 Stepping C0. This requires the
> > DTB files to be updated and U-Boot to choose between the B0 and the C0
> > variant of the Rpi 4 DTB file.
>  Thanks for the update on this, great news. In the case of the RPi-400
>  are they all the C0 stepping? I don't see the 400 in your patch. 
Yes, please look at [1]

The patch changes makes bcm2711.dtsi default to C0. So all including RPi 
4 boards (RPi 4, CM4, RPi 400) will be affected and all new boards don't 
need to care about it.

>  Is there a documented way of detecting the stepping that can be used
>  in U-Boot I can use to create a patch there, or did you have a test
>  patch for this? 
I searched in the Raspberry Pi forum and there was the statement
that 
there is no direct (official) way to detect the SoC stepping. A 
possibility might be to detect the DMA range provided by the firmware 
FDT (U-Boot has dev_get_dma_range() for this) and based on the value 
decide which DTB should be loaded (bcm2711-rpi-4-b.dtb for C0 and 
bcm2711-rpi-4-b-b0.dtb for B0).
>  Do you plan to send this patch upstream, if so feel free to add me
> to
>  the cc and in the mean time I'll test it. 
Yes. I didn't send it yet, because the merge window is currently open 
and i want to rebase it on 5.18-rc1. The patch itself is a little bit 
controversial (fix which introduce new file and change behavior).

Best regards

> > Unfortunately this doesn't fix the SD card issues on 32 bit.
>  That's less of a problem, Fedora 36 will be the last version of Fedora
>  to support 32 bit and I've always suggested people run 64 bit Fedora
>  on their RPi4 and most people do tend to do that.
>
>  Thanks,
>  Peter 
[1] - https://github.com/raspberrypi/firmware/issues/1704

[Stefan Wahren]

Sorry, missed the link

Am 31.03.22 um 16:08 schrieb Stefan Wahren:
>  Hi Peter,
>
>  Am 31.03.22 um 15:19 schrieb Peter Robinson:
> > Hi Stefan,
> >
> > Thanks for the update on this, great news. In the case of the RPi-400
> > are they all the C0 stepping? I don't see the 400 in your patch.
>
>  Yes, please look at [1]
>
>  
https://github.com/raspberrypi/firmware/issues/1704