Fedora 27 on Banana Pi M64

[Jan Pazdziora]

Hello,

I've got Banana Pi M64 on which I am able to run

	2017-08-14-ubuntu-16.04-mate-desktop-beta-aarch64-bpi-m64-sd-emmc.img

from http://www.banana-pi.org/m64-download.html reasonably well,
except for failing WiFi connections to hidden SSID networks and
for kodi being unbearably slow, even the UI and mouse movements.

So seeing Ubuntu MATE not failing completely, I thought I'd give
Fedora a try. I've used

	Fedora-Minimal-27-1.6.aarch64.raw.xz

Upon boot I see U-Boot (?) output and EFI lines and then row of Tuxes
and initial boot messages but then my TV goes blank. At no point during
the boot (like grub interface) do I have USB (mouse, keyboard) working.

When I add the dtb symlink to workaround

	https://bugzilla.redhat.com/show_bug.cgi?id=1528575

I see wired networking and DHCP working. Upgrading kernel seems to break
the wired networking

	https://bugzilla.redhat.com/show_bug.cgi?id=1528593

but with all the other packages from updates upgraded, the board still
works including wired networking and DHCP.

The initial-setup.service seems to be looping during the boot and it
seems to also affect dnf operations so I've disabled it.

Is there any opinion about how well can Banana Pi M64 be supported
by Fedora at this point, given that the board seems fine under Ubuntu
MATE but that might include some non-OSS bits that we might not want
in Fedora.

Specifically missing at this point seems to be USB support, console
after boot, and WiFi networking. Is there anything which might be useful
to try?

-- 
Jan Pazdziora
Senior Principal Software Engineer, OpenShift Security Team, Red Hat

[Peter Robinson]

On Fri, Dec 22, 2017 at 12:37 PM, Jan Pazdziora <jpazdziora(a)redhat.com&gt; wrote:
>  Hello,
>
>  I've got Banana Pi M64 on which I am able to run
>
>          2017-08-14-ubuntu-16.04-mate-desktop-beta-aarch64-bpi-m64-sd-emmc.img
>
>  from http://www.banana-pi.org/m64-download.html reasonably well,
>  except for failing WiFi connections to hidden SSID networks and
>  for kodi being unbearably slow, even the UI and mouse movements.
>
>  So seeing Ubuntu MATE not failing completely, I thought I'd give 
Well I suspect the Ubuntu images uses a vendor kernel and proprietary
display drivers, neither of which we will support in Fedora for
obvious reasons.

>  Fedora a try. I've used
>
>          Fedora-Minimal-27-1.6.aarch64.raw.xz
>
>  Upon boot I see U-Boot (?) output and EFI lines and then row of Tuxes
>  and initial boot messages but then my TV goes blank. At no point during
>  the boot (like grub interface) do I have USB (mouse, keyboard) working. 
There is no display support [1] for the A64 SoC as yet, you'll need to
use a serial console.

There might be simple FB support landing in the 4.16 kernel so if that
support does land there will be basic text output support in Fedora
28.

[1] https://nullr0ute.com/2017/11/overview-of-aarch64-sbc-support-in-fedora-27/

>  When I add the dtb symlink to workaround
>
>          https://bugzilla.redhat.com/show_bug.cgi?id=1528575
>
>  I see wired networking and DHCP working. Upgrading kernel seems to break
>  the wired networking
>
>          https://bugzilla.redhat.com/show_bug.cgi?id=1528593 
You should add the ARMTracker tracking bug so we'e aware of ARM
related kernel bugs.

I'm aware of the breakage but I took some PTO and given the NIC
drivers for the 64 bit AllWinner stuff was pulled back I've not had
time to fix it. I'm sorry (I'm really not) but you can either stick on
the 4.13 kernels until I get time to fix it or go to the 4.15rc6
kernel [2] which is known to work.

[2] https://koji.fedoraproject.org/koji/buildinfo?buildID=1012407

>  but with all the other packages from updates upgraded, the board
> still
>  works including wired networking and DHCP.
>
>  The initial-setup.service seems to be looping during the boot and it
>  seems to also affect dnf operations so I've disabled it. 
If it's run properly from the serial console on the first boot it will
disable itself.

>  Is there any opinion about how well can Banana Pi M64 be supported
>  by Fedora at this point, given that the board seems fine under Ubuntu
>  MATE but that might include some non-OSS bits that we might not want
>  in Fedora. 
As I explain in the blog post referenced above there is no display
support for any of the AllWinner 64 bit devices.... and even when that
lands it'll be average support because it's a MALI chipset and won't
be accelerated. This is the case on all Allwinner devices

>  Specifically missing at this point seems to be USB support, console
>  after boot, and WiFi networking. Is there anything which might be useful 
Have you looked at the upstream mailing lists to see if there's kernel
patches? As far as I can see, at least on linus's master, usb should
work (it definitely does on the various Pine64 variants) and wifi
might work if you have the appropriate brcm*.txt text file in
/lib/firmware/brcm/ (like what's needed on the RPi3). A grep through
dmesg for brcm should show more details there. I've not checked
anything older than linus's current head WRT to either of those.

[Jan Pazdziora]

On Mon, Jan 08, 2018 at 08:17:17AM +0000, Peter Robinson wrote:
>  On Fri, Dec 22, 2017 at 12:37 PM, Jan Pazdziora
> <jpazdziora(a)redhat.com&gt; wrote:
>  >
>  > Hello,
>  >
>  > I've got Banana Pi M64 on which I am able to run
>  >
>  >         2017-08-14-ubuntu-16.04-mate-desktop-beta-aarch64-bpi-m64-sd-emmc.img
>  >
>  > from http://www.banana-pi.org/m64-download.html reasonably well,
>  > except for failing WiFi connections to hidden SSID networks and
>  > for kodi being unbearably slow, even the UI and mouse movements.
>  >
>  > So seeing Ubuntu MATE not failing completely, I thought I'd give
>  
>  Well I suspect the Ubuntu images uses a vendor kernel and proprietary
>  display drivers, neither of which we will support in Fedora for
>  obvious reasons. 
Do you think that

	https://github.com/yuq/linux-lima
	
might be something to watch? Or

	https://free-electrons.com/blog/mali-opengl-support-on-allwinner-platform...

?

>  > Upon boot I see U-Boot (?) output and EFI lines and then row of
> Tuxes
>  > and initial boot messages but then my TV goes blank. At no point during
>  > the boot (like grub interface) do I have USB (mouse, keyboard) working.
>  
>  There is no display support [1] for the A64 SoC as yet, you'll need to
>  use a serial console.
>  
>  There might be simple FB support landing in the 4.16 kernel so if that
>  support does land there will be basic text output support in Fedora
>  28. 
OK. Out of curiosity, the boot messages that I can see at the early
boot stage of Fedora 27 get output via what mechanism?

>  > I see wired networking and DHCP working. Upgrading kernel seems
> to break
>  > the wired networking
>  >
>  >         https://bugzilla.redhat.com/show_bug.cgi?id=1528593
>  
>  You should add the ARMTracker tracking bug so we'e aware of ARM
>  related kernel bugs. 
WILCO (will try to not forget ;-) next time.

>  I'm aware of the breakage but I took some PTO and given the NIC
>  drivers for the 64 bit AllWinner stuff was pulled back I've not had
>  time to fix it. I'm sorry (I'm really not) but you can either stick on
>  the 4.13 kernels until I get time to fix it or go to the 4.15rc6
>  kernel [2] which is known to work. 
I'll try it when I get some cycles to focus on it again, thanks for
the hint.

>  > but with all the other packages from updates upgraded, the board
> still
>  > works including wired networking and DHCP.
>  >
>  > The initial-setup.service seems to be looping during the boot and it
>  > seems to also affect dnf operations so I've disabled it.
>  
>  If it's run properly from the serial console on the first boot it will
>  disable itself. 
Does it make sense to track the fact that on console-less machine, we
might want it to fail more gracefully?

>  > Specifically missing at this point seems to be USB support,
> console
>  > after boot, and WiFi networking. Is there anything which might be useful
>  
>  Have you looked at the upstream mailing lists to see if there's kernel
>  patches? As far as I can see, at least on linus's master, usb should
>  work (it definitely does on the various Pine64 variants) and wifi
>  might work if you have the appropriate brcm*.txt text file in
>  /lib/firmware/brcm/ (like what's needed on the RPi3). A grep through
>  dmesg for brcm should show more details there. I've not checked
>  anything older than linus's current head WRT to either of those. 
I'll put this to the list of things to check.

Thank you for you replies!

-- 
Jan Pazdziora
Senior Principal Software Engineer, OpenShift Security Team, Red Hat

[Peter Robinson]

On Fri, Jan 12, 2018 at 9:15 AM, Jan Pazdziora <jpazdziora(a)redhat.com&gt; wrote:
>  On Mon, Jan 08, 2018 at 08:17:17AM +0000, Peter Robinson wrote:
> > On Fri, Dec 22, 2017 at 12:37 PM, Jan Pazdziora <jpazdziora(a)redhat.com&gt; wrote:
> > >
> > > Hello,
> > >
> > > I've got Banana Pi M64 on which I am able to run
> > >
> > >         2017-08-14-ubuntu-16.04-mate-desktop-beta-aarch64-bpi-m64-sd-emmc.img
> > >
> > > from http://www.banana-pi.org/m64-download.html reasonably well,
> > > except for failing WiFi connections to hidden SSID networks and
> > > for kodi being unbearably slow, even the UI and mouse movements.
> > >
> > > So seeing Ubuntu MATE not failing completely, I thought I'd give
> >
> > Well I suspect the Ubuntu images uses a vendor kernel and proprietary
> > display drivers, neither of which we will support in Fedora for
> > obvious reasons.
>
>  Do you think that
>
>          https://github.com/yuq/linux-lima 
Yes, I'm watching it already. Time will tell, there's been a lot of
false starts and it took etnaviv years to be usable. It's also only of
use for the 400 series as the newer GPUs are completely different.

>  might be something to watch? Or
>
>         
> https://free-electrons.com/blog/mali-opengl-support-on-allwinner-platform...
Someone might want to look at it, will probably need work around
libglvnd to co-exist with mesa similar to that of the nvidia binary. I
also hear it'll never support Wayland. If someone was interested in
getting it working and into rpmfusion I'm sure a lot of people would
appreciate it but personally I have no interest (or time) to do
anything around this.

> > > Upon boot I see U-Boot (?) output and EFI lines and then row
> of Tuxes
> > > and initial boot messages but then my TV goes blank. At no point during
> > > the boot (like grub interface) do I have USB (mouse, keyboard) working.
> >
> > There is no display support [1] for the A64 SoC as yet, you'll need to
> > use a serial console.
> >
> > There might be simple FB support landing in the 4.16 kernel so if that
> > support does land there will be basic text output support in Fedora
> > 28.
>
>  OK. Out of curiosity, the boot messages that I can see at the early
>  boot stage of Fedora 27 get output via what mechanism? 
Probably the simple framebuffer that's in u-boot until the kernel
re-initialises some bit of hardware but that's a guess, I personally
have never bothered to plug my Pine64 devices into a HDMI port and run
them using serial console.

> > > I see wired networking and DHCP working. Upgrading kernel
> seems to break
> > > the wired networking
> > >
> > >         https://bugzilla.redhat.com/show_bug.cgi?id=1528593
> >
> > You should add the ARMTracker tracking bug so we'e aware of ARM
> > related kernel bugs.
>
>  WILCO (will try to not forget ;-) next time.
>
> > I'm aware of the breakage but I took some PTO and given the NIC
> > drivers for the 64 bit AllWinner stuff was pulled back I've not had
> > time to fix it. I'm sorry (I'm really not) but you can either stick on
> > the 4.13 kernels until I get time to fix it or go to the 4.15rc6
> > kernel [2] which is known to work.
>
>  I'll try it when I get some cycles to focus on it again, thanks for
>  the hint.
>
> > > but with all the other packages from updates upgraded, the board still
> > > works including wired networking and DHCP.
> > >
> > > The initial-setup.service seems to be looping during the boot and it
> > > seems to also affect dnf operations so I've disabled it.
> >
> > If it's run properly from the serial console on the first boot it will
> > disable itself.
>
>  Does it make sense to track the fact that on console-less machine, we
>  might want it to fail more gracefully? 
Maybe, someone needs to have the time to implement it or at least
engage with the initial-setup developers and convince them it's worthy
of their time. At least for the vast majoriity of the newer allwinner
devices this will hopefully not be a problem with F-28/4.16 as we
should have simple console output so they'll be able to run text based
there.

> > > Specifically missing at this point seems to be USB support,
> console
> > > after boot, and WiFi networking. Is there anything which might be useful
> >
> > Have you looked at the upstream mailing lists to see if there's kernel
> > patches? As far as I can see, at least on linus's master, usb should
> > work (it definitely does on the various Pine64 variants) and wifi
> > might work if you have the appropriate brcm*.txt text file in
> > /lib/firmware/brcm/ (like what's needed on the RPi3). A grep through
> > dmesg for brcm should show more details there. I've not checked
> > anything older than linus's current head WRT to either of those.
>
>  I'll put this to the list of things to check. 
Reply here when you have more details.

Thanks,
Peter

[Jan Pazdziora]

On Mon, Jan 08, 2018 at 08:17:17AM +0000, Peter Robinson wrote:
>  >
>  > I see wired networking and DHCP working. Upgrading kernel seems to break
>  > the wired networking
>  >
>  >         https://bugzilla.redhat.com/show_bug.cgi?id=1528593
>  
>  You should add the ARMTracker tracking bug so we'e aware of ARM
>  related kernel bugs.
>  
>  I'm aware of the breakage but I took some PTO and given the NIC
>  drivers for the 64 bit AllWinner stuff was pulled back I've not had
>  time to fix it. I'm sorry (I'm really not) but you can either stick on
>  the 4.13 kernels until I get time to fix it or go to the 4.15rc6
>  kernel [2] which is known to work.
>  
>  [2] https://koji.fedoraproject.org/koji/buildinfo?buildID=1012407 
I've upgraded to kernel-4.15.3-300.fc27.aarch64 which landed in Fedora
27 repos but the wired networking / DHCP still do not work with that
kernel.

I don't have serial console to do interactive troubleshooting so
I don't really know what is happening there ... checking the journal
after I've rebooted back to 4.13.9-300.fc27.aarch64, there does not
even seem to be a record of boot into that 4.15.3, so the problem
likely isn't the networking at all -- the kernel simply does not boot.

Should I try to go to rawhide kernels instead?

-- 
Jan Pazdziora
Senior Principal Software Engineer, OpenShift Security Team, Red Hat

[Jan Pazdziora]

On Fri, Feb 23, 2018 at 12:26:15PM +0100, Jan Pazdziora wrote:
>  
>  I don't have serial console to do interactive troubleshooting so
>  I don't really know what is happening there ... checking the journal
>  after I've rebooted back to 4.13.9-300.fc27.aarch64, there does not
>  even seem to be a record of boot into that 4.15.3, so the problem
>  likely isn't the networking at all -- the kernel simply does not boot.
>  
>  Should I try to go to rawhide kernels instead? 
I tried

	4.16.0-0.rc2.git1.1.fc28.aarch64

from koji without much success.

On Mon, Jan 08, 2018 at 08:17:17AM +0000, Peter Robinson wrote:
>  
>  There might be simple FB support landing in the 4.16 kernel so if that
>  support does land there will be basic text output support in Fedora
>  28. 
Does the 4.16.0-0.rc2.git1.1.fc28.aarch64 kernel have the FB
support? The (HDMI) screen goes blank after the initial set of
boot messages but maybe I just need to configure something
differently to use the framebuffer?

-- 
Jan Pazdziora
Senior Principal Software Engineer, OpenShift Security Team, Red Hat

[Peter Robinson]

On Fri, Feb 23, 2018 at 11:26 AM, Jan Pazdziora <jpazdziora(a)redhat.com&gt; wrote:
>  On Mon, Jan 08, 2018 at 08:17:17AM +0000, Peter Robinson wrote:
> > >
> > > I see wired networking and DHCP working. Upgrading kernel seems to break
> > > the wired networking
> > >
> > >         https://bugzilla.redhat.com/show_bug.cgi?id=1528593
> >
> > You should add the ARMTracker tracking bug so we'e aware of ARM
> > related kernel bugs.
> >
> > I'm aware of the breakage but I took some PTO and given the NIC
> > drivers for the 64 bit AllWinner stuff was pulled back I've not had
> > time to fix it. I'm sorry (I'm really not) but you can either stick on
> > the 4.13 kernels until I get time to fix it or go to the 4.15rc6
> > kernel [2] which is known to work.
> >
> > [2] https://koji.fedoraproject.org/koji/buildinfo?buildID=1012407
>
>  I've upgraded to kernel-4.15.3-300.fc27.aarch64 which landed in Fedora
>  27 repos but the wired networking / DHCP still do not work with that
>  kernel.
>
>  I don't have serial console to do interactive troubleshooting so
>  I don't really know what is happening there ... checking the journal
>  after I've rebooted back to 4.13.9-300.fc27.aarch64, there does not
>  even seem to be a record of boot into that 4.15.3, so the problem
>  likely isn't the networking at all -- the kernel simply does not boot. 
So 4.15.x has been solid for me on my two Pine64s which is the same
SoC without the network issues I saw with 4.14.

I can't even begin to debug it further, I'm sorry but if you're flying
blind without a console I have zero ability to use telepathy to try
and magically guess at the issues.

[Peter Robinson]

On Fri, Feb 23, 2018 at 11:43 AM, Jan Pazdziora <jpazdziora(a)redhat.com&gt; wrote:
>  On Fri, Feb 23, 2018 at 12:26:15PM +0100, Jan Pazdziora wrote:
> >
> > I don't have serial console to do interactive troubleshooting so
> > I don't really know what is happening there ... checking the journal
> > after I've rebooted back to 4.13.9-300.fc27.aarch64, there does not
> > even seem to be a record of boot into that 4.15.3, so the problem
> > likely isn't the networking at all -- the kernel simply does not boot.
> >
> > Should I try to go to rawhide kernels instead?
>
>  I tried
>
>          4.16.0-0.rc2.git1.1.fc28.aarch64
>
>  from koji without much success.
>
>  On Mon, Jan 08, 2018 at 08:17:17AM +0000, Peter Robinson wrote:
> >
> > There might be simple FB support landing in the 4.16 kernel so if that
> > support does land there will be basic text output support in Fedora
> > 28.
>
>  Does the 4.16.0-0.rc2.git1.1.fc28.aarch64 kernel have the FB
>  support? The (HDMI) screen goes blank after the initial set of
>  boot messages but maybe I just need to configure something
>  differently to use the framebuffer? 
No, it's not upstream yet, I've not had the time to look to see if
it's something sane we could pull in, it would need to be scheduled to
land in 4.17 because I don't want to have to keep rebasing patches.
When I get a few spare hours to review and test locally I'll decide.

[Jan Pazdziora]

On Fri, Feb 23, 2018 at 12:19:43PM +0000, Peter Robinson wrote:
>  >
>  > I've upgraded to kernel-4.15.3-300.fc27.aarch64 which landed in Fedora
>  > 27 repos but the wired networking / DHCP still do not work with that
>  > kernel.
>  >
>  > I don't have serial console to do interactive troubleshooting so
>  > I don't really know what is happening there ... checking the journal
>  > after I've rebooted back to 4.13.9-300.fc27.aarch64, there does not
>  > even seem to be a record of boot into that 4.15.3, so the problem
>  > likely isn't the networking at all -- the kernel simply does not boot.
>  
>  So 4.15.x has been solid for me on my two Pine64s which is the same
>  SoC without the network issues I saw with 4.14.
>  
>  I can't even begin to debug it further, I'm sorry but if you're flying
>  blind without a console I have zero ability to use telepathy to try
>  and magically guess at the issues. 
I understand.

To investigate some more, I flashed
Fedora-Minimal-Rawhide-20180220.n.0.aarch64.raw.xz with
kernel-4.16.0-0.rc2.git0.1.fc28.aarch64 and it boots and DHCPs fine.

I then flashed back Fedora-Minimal-27-1.6.aarch64.raw.xz and then
installed kernel{,-core,-modules}-4.16.0-0.rc2.git0.1.fc28.aarch64.rpm
(and of course updated /boot/dtb). After reboot, things do not work.

So then I've replaced the
/boot/initramfs-4.16.0-0.rc2.git0.1.fc28.aarch64.img which was
generated during the package installation (file size 18841270) with
the file from the Fedora-Minimal-Rawhide-20180220.n.0.aarch64.raw.xz
install (file size 52326700).

I've rebooted and things work.

So the problem seems to be the initramfs produced during package
installation, rather than the kernel per se.

In your tests with Pine64, do you do kernel rpm installations?

-- 
Jan Pazdziora
Senior Principal Software Engineer, OpenShift Security Team, Red Hat

[Jan Pazdziora]

On Mon, Jan 08, 2018 at 08:17:17AM +0000, Peter Robinson wrote:
>  
>  work (it definitely does on the various Pine64 variants) and wifi
>  might work if you have the appropriate brcm*.txt text file in
>  /lib/firmware/brcm/ (like what's needed on the RPi3). A grep through
>  dmesg for brcm should show more details there. I've not checked
>  anything older than linus's current head WRT to either of those. 
I confirm that with kernel-4.16.0-0.rc2.git0.1.fc28.aarch64, running

	curl
http://jwrdegoede.danny.cz/brcm-firmware/brcmfmac43430a0-sdio.bin.7.10.68... -o
/lib/firmware/brcm/brcmfmac43430a0-sdio.bin 
	curl http://jwrdegoede.danny.cz/brcm-firmware/brcmfmac43430a0-sdio.txt.jumper-...
-o /lib/firmware/brcm/brcmfmac43430a0-sdio.txt 

gives me working WiFi. I was pointed to these URLs from

	https://patchwork.kernel.org/patch/9791523/

-- 
Jan Pazdziora
Senior Principal Software Engineer, OpenShift Security Team, Red Hat