Starting a riscv64 VM from an ArchLinux x86_64 host

[Andrej Podzimek]

Hi Fedora developers!

I'm trying to start a Fedora riscv64 VM on an ArchLinux x86_64 host based on this wiki
page: https://fedoraproject.org/wiki/Architectures/RISC-V/Installing I have a reasonably
recent qemu and virt-manager:

```
qemu 6.0.0-3
qemu-arch-extra 6.0.0-3
libvirt 1:7.3.0-1
virt-manager 3.2.0-1
```

All virt-builder steps work perfectly fine and produce an image.

However, I cannot start an instance. I tried the image prepared with virt-builder, the
manually downloaded image, the nightly image, direct kernel loading (configured in libvirt
/ virt-manager), but none of that worked.

For the nightly instances in particular, the wiki says one should extract
"firmware" from /opensbi/... in the image. However, there is _no_ such directory
in the image (and also no file called .elf). This information may be outdated.

When using the downloaded .elf file (as described in the "Download using
virt-builder" section), I get this from qemu:

```
qemu-system-riscv64: Some ROM regions are overlapping
These ROM regions might have been loaded by direct user request or by default.
They could be BIOS/firmware images, a guest kernel, initrd or some other file loaded into
guest memory.
Check whether you intended to load all this guest code, and whether it has been built to
load to the correct addresses.

The following two regions overlap (in the memory address space):
   /usr/bin/../share/qemu/opensbi-riscv64-generic-fw_dynamic.bin (addresses
0x0000000080000000 - 0x0000000080012558)
  
VirtualMachineMedia/Fedora-Developer-Rawhide-20200108.n.0-fw_payload-uboot-qemu-virt-smode.elf
ELF program header segment 0 (addresses 0x0000000080000000 - 0x000000008000d0e0)
```

The qemu command line was:

```
qemu-system-riscv64 \
     -nographic \
     -machine virt \
     -smp 8 \
     -m 32G \
     -kernel
VirtualMachineMedia/Fedora-Developer-Rawhide-20200108.n.0-fw_payload-uboot-qemu-virt-smode.elf
\
     -object rng-random,filename=/dev/urandom,id=rng0 \
     -device virtio-rng-device,rng=rng0 \
     -device virtio-blk-device,drive=hd0\
     -drive
file=VirtualMachineMedia/Fedora-Developer-Rawhide-20210421.n.0-sda.raw,format=raw,id=hd0
\
     -device virtio-net-device,netdev=usernet \
     -netdev user,id=usernet,hostfwd=tcp::10000-:22
```

Idea No. 1: Drop the "-kernel ..." argument. In that case I get an OpenSBI
splash that looks like this: https://pastebin.com/5QKJqkRg Sadly, the VM is frozen and
nothing else happens. There is no network communication either.

It looks like this^^^ configuration might start something, but fails to load a kernel
properly (or the like). The nightly image appears to contain multiple bootloaders, but the
wiki doesn't say how to run them.

Idea No. 2: Delete /usr/share/qemu/opensbi-riscv64-generic-fw_dynamic.bin to see if the
conflict goes away. (For the record, the file is owned by qemu-arch-extra.) Well, qemu
then says 'qemu-system-riscv64: Unable to load the RISC-V firmware
"opensbi-riscv64-generic-fw_dynamic.bin"'.

Idea No. 3: Extract the kernel (vmlinuz-5.10.6-200.0.riscv64.fc33.riscv64) and initramdisk
(initramfs-5.10.6-200.0.riscv64.fc33.riscv64.img) from the nightly image and load them
"directly" (using virt-manager) like this: https://pastebin.com/36RVR75r

This^^^ failed the same way as all virt-manager experiments (with nightly / manually
downloaded / built with virt-builder) images: A black console with a blinking cursor and
nothing happens. No activity on the virtual bridge either.

Interestingly, each time qemu starts and freezes, host CPU utilization is about 200%
(i.e., 2 cores worth of load, not 1 core, not 8 cores etc.). (The host machine is a Ryzen
3950X with 32 CPUs altogether and 128 GB RAM.)

What am I doing wrong (apart from not using Fedora as a host)?
Why am I getting a weird conflict with a system built-in OpenSBI? Could the built-in
firmware be incompatible with the Fedora image?
Are there any up-to-date instructions on how to get the recent nightly images (without the
/opensbi directory) working?
How can I debug why qemu / OpenSBI freezes? Does OpenSBI need a bit of configuration to
boot the image?

I'll retry this on a Fedora VM if need be, but would also like to understand what the
issue is on Arch. Also, I wanted to double-check that the riscv64 port actually works on
current systems...

Just about any hint would help a lot. :-)

Cheers!
Andrej

[Richard W.M. Jones]

On Tue, Jul 06, 2021 at 03:15:00AM +0200, Andrej Podzimek via devel wrote:
>  Hi Fedora developers!
>  
>  I'm trying to start a Fedora riscv64 VM on an ArchLinux x86_64 host based on this
> wiki page: https://fedoraproject.org/wiki/Architectures/RISC-V/Installing I have a
> reasonably recent qemu and virt-manager: 
I'm not sure, but you might want to chat with david in #fedora-riscv
on Libera.

Also it may work better using libvirt to manage the guest.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
libguestfs lets you edit virtual machines.  Supports shell scripting,
bindings from many languages.  http://libguestfs.org

[Andrej Podzimek]

> > Hi Fedora developers! 
> > 
> > I'm trying to start a Fedora riscv64 VM on an ArchLinux
> x86_64 host based on this wiki page:
> https://fedoraproject.org/wiki/Architectures/RISC-V/Installing I have a reasonably recent
> qemu and virt-manager: 
>  
>  I'm not sure, but you might want to chat with david in
> #fedora-riscv 
>  on Libera. 
>  
>  Also it may work better using libvirt to manage the guest. 


Alright, I'll try to post into #fedora-riscv then.



For the record, I've re-done all my experiments on a Fedora machine today (an
all-default Fedora 34 VM installed from the KDE spin live image). (For riscv emulation it
hopefully shouldn't matter if the host is a VM or not. The machine can do nested
virtualization, but this is irrelevant for emulation.)



I tried both raw qemu-system-riscv64 and libvirt, but sadly the results were exactly the
same as before, both the address conflict and the qemu / OpenSBI freeze with the
"conflicting" firmware image flag removed. Whatever the problem is, the
ArchLinux host was likely not the cause. The raw qemu at least shows an OpenSBI splash
text before freezing. With libvirt the console connection is hopelessly frozen and nothing
appears.



Andrej

[Takayuki Nagata]

Hi,

Maybe, you need to add the "-bios none" option. I booted the image
with the following options on Fedora 33.

~~~
$ qemu-system-riscv64 -bios none -nographic -machine virt -smp 1 -m 1G \
-kernel Fedora-Minimal-Rawhide-20200108.n.0-fw_payload-uboot-qemu-virt-smode.elf
\
-device virtio-blk-device,drive=hd0 \
-drive file=Fedora-Minimal-Rawhide-20200108.n.0-sda.raw,format=raw,id=hd0 \
-device virtio-net-device,netdev=usernet \
-netdev user,id=usernet,hostfwd=tcp::10000-:22
~~~

Takayuki Nagata

2021年7月6日(火) 10:16 Andrej Podzimek via devel <devel(a)lists.fedoraproject.org&gt;:
>  Hi Fedora developers!
>
>  I'm trying to start a Fedora riscv64 VM on an ArchLinux x86_64 host based on this
> wiki page: https://fedoraproject.org/wiki/Architectures/RISC-V/Installing I have a
> reasonably recent qemu and virt-manager:
>
>  ```
>  qemu 6.0.0-3
>  qemu-arch-extra 6.0.0-3
>  libvirt 1:7.3.0-1
>  virt-manager 3.2.0-1
>  ```
>
>  All virt-builder steps work perfectly fine and produce an image.
>
>  However, I cannot start an instance. I tried the image prepared with virt-builder, the
> manually downloaded image, the nightly image, direct kernel loading (configured in libvirt
> / virt-manager), but none of that worked.
>
>  For the nightly instances in particular, the wiki says one should extract
> "firmware" from /opensbi/... in the image. However, there is _no_ such directory
> in the image (and also no file called .elf). This information may be outdated.
>
>  When using the downloaded .elf file (as described in the "Download using
> virt-builder" section), I get this from qemu:
>
>  ```
>  qemu-system-riscv64: Some ROM regions are overlapping
>  These ROM regions might have been loaded by direct user request or by default.
>  They could be BIOS/firmware images, a guest kernel, initrd or some other file loaded into
> guest memory.
>  Check whether you intended to load all this guest code, and whether it has been built to
> load to the correct addresses.
>
>  The following two regions overlap (in the memory address space):
>     /usr/bin/../share/qemu/opensbi-riscv64-generic-fw_dynamic.bin (addresses
> 0x0000000080000000 - 0x0000000080012558)
>    
> VirtualMachineMedia/Fedora-Developer-Rawhide-20200108.n.0-fw_payload-uboot-qemu-virt-smode.elf
> ELF program header segment 0 (addresses 0x0000000080000000 - 0x000000008000d0e0)
>  ```
>
>  The qemu command line was:
>
>  ```
>  qemu-system-riscv64 \
>       -nographic \
>       -machine virt \
>       -smp 8 \
>       -m 32G \
>       -kernel
> VirtualMachineMedia/Fedora-Developer-Rawhide-20200108.n.0-fw_payload-uboot-qemu-virt-smode.elf
> \
>       -object rng-random,filename=/dev/urandom,id=rng0 \
>       -device virtio-rng-device,rng=rng0 \
>       -device virtio-blk-device,drive=hd0\
>       -drive
> file=VirtualMachineMedia/Fedora-Developer-Rawhide-20210421.n.0-sda.raw,format=raw,id=hd0
> \
>       -device virtio-net-device,netdev=usernet \
>       -netdev user,id=usernet,hostfwd=tcp::10000-:22
>  ```
>
>  Idea No. 1: Drop the "-kernel ..." argument. In that case I get an OpenSBI
> splash that looks like this: https://pastebin.com/5QKJqkRg Sadly, the VM is frozen and
> nothing else happens. There is no network communication either.
>
>  It looks like this^^^ configuration might start something, but fails to load a kernel
> properly (or the like). The nightly image appears to contain multiple bootloaders, but the
> wiki doesn't say how to run them.
>
>  Idea No. 2: Delete /usr/share/qemu/opensbi-riscv64-generic-fw_dynamic.bin to see if the
> conflict goes away. (For the record, the file is owned by qemu-arch-extra.) Well, qemu
> then says 'qemu-system-riscv64: Unable to load the RISC-V firmware
> "opensbi-riscv64-generic-fw_dynamic.bin"'.
>
>  Idea No. 3: Extract the kernel (vmlinuz-5.10.6-200.0.riscv64.fc33.riscv64) and
> initramdisk (initramfs-5.10.6-200.0.riscv64.fc33.riscv64.img) from the nightly image and
> load them "directly" (using virt-manager) like this:
> https://pastebin.com/36RVR75r
>
>  This^^^ failed the same way as all virt-manager experiments (with nightly / manually
> downloaded / built with virt-builder) images: A black console with a blinking cursor and
> nothing happens. No activity on the virtual bridge either.
>
>  Interestingly, each time qemu starts and freezes, host CPU utilization is about 200%
> (i.e., 2 cores worth of load, not 1 core, not 8 cores etc.). (The host machine is a Ryzen
> 3950X with 32 CPUs altogether and 128 GB RAM.)
>
>  What am I doing wrong (apart from not using Fedora as a host)?
>  Why am I getting a weird conflict with a system built-in OpenSBI? Could the built-in
> firmware be incompatible with the Fedora image?
>  Are there any up-to-date instructions on how to get the recent nightly images (without
> the /opensbi directory) working?
>  How can I debug why qemu / OpenSBI freezes? Does OpenSBI need a bit of configuration to
> boot the image?
>
>  I'll retry this on a Fedora VM if need be, but would also like to understand what the
> issue is on Arch. Also, I wanted to double-check that the riscv64 port actually works on
> current systems...
>
>  Just about any hint would help a lot. :-)
>
>  Cheers!
>  Andrej
>
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]
>  Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure

[Steven A. Falco]

I too was having the error about overlapping regions.  Adding "-bios none"
solved the problem.  Thanks!

I wanted to use virt-install to set up my VM, and it turns out I could get the same effect
by adding --qemu-commandline='-bios none' to that command line:

virt-install --qemu-commandline='-bios none'  --name fedora-riscv   \
   --arch riscv64   --machine virt   --vcpus 8   --memory 2048   \
   --os-variant fedora30 --boot
kernel=Fedora-Developer-Rawhide-20191123.n.0-fw_payload-uboot-qemu-virt-smode.elf \
   --import --disk path=Fedora-Developer-Rawhide-20191123.n.0-sda.raw \
   --network network=default,model=virtio   --rng device=/dev/urandom,model=virtio   \
   --channel name=org.qemu.guest_agent.0   --graphics none

	Steve

On 7/6/21 8:23 PM, Takayuki Nagata wrote:
>  Hi,
>  
>  Maybe, you need to add the "-bios none" option. I booted the image
>  with the following options on Fedora 33.
>  
>  ~~~
>  $ qemu-system-riscv64 -bios none -nographic -machine virt -smp 1 -m 1G \
>  -kernel Fedora-Minimal-Rawhide-20200108.n.0-fw_payload-uboot-qemu-virt-smode.elf
>  \
>  -device virtio-blk-device,drive=hd0 \
>  -drive file=Fedora-Minimal-Rawhide-20200108.n.0-sda.raw,format=raw,id=hd0 \
>  -device virtio-net-device,netdev=usernet \
>  -netdev user,id=usernet,hostfwd=tcp::10000-:22
>  ~~~
>  
>  Takayuki Nagata
>  
>  2021年7月6日(火) 10:16 Andrej Podzimek via devel <devel(a)lists.fedoraproject.org&gt;:
> >
> > Hi Fedora developers!
> >
> > I'm trying to start a Fedora riscv64 VM on an ArchLinux x86_64 host based on this
> wiki page: https://fedoraproject.org/wiki/Architectures/RISC-V/Installing I have a
> reasonably recent qemu and virt-manager:
> >
> > ```
> > qemu 6.0.0-3
> > qemu-arch-extra 6.0.0-3
> > libvirt 1:7.3.0-1
> > virt-manager 3.2.0-1
> > ```
> >
> > All virt-builder steps work perfectly fine and produce an image.
> >
> > However, I cannot start an instance. I tried the image prepared with virt-builder,
> the manually downloaded image, the nightly image, direct kernel loading (configured in
> libvirt / virt-manager), but none of that worked.
> >
> > For the nightly instances in particular, the wiki says one should extract
> "firmware" from /opensbi/... in the image. However, there is _no_ such directory
> in the image (and also no file called .elf). This information may be outdated.
> >
> > When using the downloaded .elf file (as described in the "Download using
> virt-builder" section), I get this from qemu:
> >
> > ```
> > qemu-system-riscv64: Some ROM regions are overlapping
> > These ROM regions might have been loaded by direct user request or by default.
> > They could be BIOS/firmware images, a guest kernel, initrd or some other file loaded
> into guest memory.
> > Check whether you intended to load all this guest code, and whether it has been built
> to load to the correct addresses.
> >
> > The following two regions overlap (in the memory address space):
> >     /usr/bin/../share/qemu/opensbi-riscv64-generic-fw_dynamic.bin (addresses
> 0x0000000080000000 - 0x0000000080012558)
> >    
> VirtualMachineMedia/Fedora-Developer-Rawhide-20200108.n.0-fw_payload-uboot-qemu-virt-smode.elf
> ELF program header segment 0 (addresses 0x0000000080000000 - 0x000000008000d0e0)
> > ```
> >
> > The qemu command line was:
> >
> > ```
> > qemu-system-riscv64 \
> >       -nographic \
> >       -machine virt \
> >       -smp 8 \
> >       -m 32G \
> >       -kernel
> VirtualMachineMedia/Fedora-Developer-Rawhide-20200108.n.0-fw_payload-uboot-qemu-virt-smode.elf
> \
> >       -object rng-random,filename=/dev/urandom,id=rng0 \
> >       -device virtio-rng-device,rng=rng0 \
> >       -device virtio-blk-device,drive=hd0\
> >       -drive
> file=VirtualMachineMedia/Fedora-Developer-Rawhide-20210421.n.0-sda.raw,format=raw,id=hd0
> \
> >       -device virtio-net-device,netdev=usernet \
> >       -netdev user,id=usernet,hostfwd=tcp::10000-:22
> > ```
> >
> > Idea No. 1: Drop the "-kernel ..." argument. In that case I get an OpenSBI
> splash that looks like this: https://pastebin.com/5QKJqkRg Sadly, the VM is frozen and
> nothing else happens. There is no network communication either.
> >
> > It looks like this^^^ configuration might start something, but fails to load a kernel
> properly (or the like). The nightly image appears to contain multiple bootloaders, but the
> wiki doesn't say how to run them.
> >
> > Idea No. 2: Delete /usr/share/qemu/opensbi-riscv64-generic-fw_dynamic.bin to see if
> the conflict goes away. (For the record, the file is owned by qemu-arch-extra.) Well, qemu
> then says 'qemu-system-riscv64: Unable to load the RISC-V firmware
> "opensbi-riscv64-generic-fw_dynamic.bin"'.
> >
> > Idea No. 3: Extract the kernel (vmlinuz-5.10.6-200.0.riscv64.fc33.riscv64) and
> initramdisk (initramfs-5.10.6-200.0.riscv64.fc33.riscv64.img) from the nightly image and
> load them "directly" (using virt-manager) like this:
> https://pastebin.com/36RVR75r
> >
> > This^^^ failed the same way as all virt-manager experiments (with nightly / manually
> downloaded / built with virt-builder) images: A black console with a blinking cursor and
> nothing happens. No activity on the virtual bridge either.
> >
> > Interestingly, each time qemu starts and freezes, host CPU utilization is about 200%
> (i.e., 2 cores worth of load, not 1 core, not 8 cores etc.). (The host machine is a Ryzen
> 3950X with 32 CPUs altogether and 128 GB RAM.)
> >
> > What am I doing wrong (apart from not using Fedora as a host)?
> > Why am I getting a weird conflict with a system built-in OpenSBI? Could the built-in
> firmware be incompatible with the Fedora image?
> > Are there any up-to-date instructions on how to get the recent nightly images
> (without the /opensbi directory) working?
> > How can I debug why qemu / OpenSBI freezes? Does OpenSBI need a bit of configuration
> to boot the image?
> >
> > I'll retry this on a Fedora VM if need be, but would also like to understand what
> the issue is on Arch. Also, I wanted to double-check that the riscv64 port actually works
> on current systems...
> >
> > Just about any hint would help a lot. :-)
> >
> > Cheers!
> > Andrej
> >
> > _______________________________________________
> > devel mailing list -- devel(a)lists.fedoraproject.org
> > To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
> > Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]
> > Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]
>  Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
>