On Wed, Jun 29, 2016 at 6:03 AM, Daniel P. Berrange <berrange(a)redhat.com> wrote:
For those who aren't familiar, QEMU actually provides two
different sets of emulators
- system emulators - they emulate a full virtual machine and thus run
a full guest OS.
- user emulators - they emulate the Linux userspace ABI letting you
run non-native arch executables directly.
The user emulators are what I'm concerned with in this mail, so ignore
the system emulators.
Currently all the user emulators are provided in the "qemu-user" RPM
which also includes files in /usr/lib/binfmt.d to register each emulator
binary as a binary format handler for its respective architecture.
This is ok if you have a non-native arch binary that's statically linked
and you just want to run it from context of your main OS root filesystem.
Running dynamic linked binaries won't fly because if say running an arm
binary on x86_64 host, it'll look for /lib/libc.so and find the i386 one,
instead of the arm one. You can't set LD_LIBRARY_PATH to override this
as the env var will apply to both qemu-arm (an x86_64 binary) and the
binary it is trying to run (an arm binary).
More typical though is that you have a directory containing an fullish
install tree of a non-native architecture and you just want to chroot
into that. When doing such a chroot, the qemu-$ARCH emulator must be
present inside the chroot too. ie the x86_64 build of /usr/bin/qemu-arm
must be present inside at /my/chroot/for/fedora-arm/usr/bin/qemu-arm.
So again you have the potential problem of clashing libc.so in /usr/lib
It is a shame Fedora doesn't have full multi-arch support, instead of
merely multi-lib to avoid these clashing lib dirs across architecture
The recommended way to deal with this for the qemu user emulator binaries
to be statically linked, so when copied inside the non-native arch chroot,
they never need to resolve any native arch libraries. Fedora's qemu user
binaries are all dynamic linked right now.
jfyi, nothing really to contribute on how to do this, but this has
been a long-standing gripe of mine about fedora, and somehow packaging
qemu-static is something I'm a really big fan of.
(in a previous life, using sbox2 to "natively" build arm stuff on x86
was quite useful..)
> Debian handles this by having several packages 
> - qemu-user - the dynamic linked qemu user binaries
> - qemu-binfmt - binfmt rules registering the dynamic linked binaries
> - qemu-user-static - the static linked qemu user binaries *and* binfmt
> rules to register them. The static binaries all
> have -static suffix on their name
> NB, this means qemu-binfmt and qemu-user-static are mutually exclusive
> since they both provide the same binfmt files. You can however have both
> qemu-user and qemu-user-static installed as their binary names won't
> clash, and in this case the static ones will be registered as binfmts
> This nice thing about this multiple package approach is that when you
> copied the x86_64 build of the "qemu-arm-static" binary into your arm
> chroot, you still then have the possibility of installing the arm build
> of the "qemu-arm" binary inside that chroot without filename clash.
> An alternative simpler approach would be to just have one package,
> qemu-user, which contains the static binaries and never ship any
> dynamic linked qemu user binaries. This is slightly more restrictive
> though, as explained in the previous paragraph, so I'd like to avoid
> doing that.
> I'd like to make using non-native arch chroots simple with Fedora without
> people needing to manually build their own static QEMU binaries, or download
> static binaries provided by another distro. So I'm suggesting to make a
> change to Fedora qemu packages to essentially copy the way Debian has done
> things. Specifically I will
> - Pull the binfmt registration files out of qemu-user and into a
> new qemu-binfmt package which depends on qemu-user.
> - Add static builds of qemu user emulators to a new qemu-user-static
> package, along with binfmt registration files
> The static build of QEMU user emulators is moderately light on
> dependancies, only requiring glib2-static, pcre-static, zlib-static
> and glibc-static packages.
> The change to introduce a qemu-binfmt package has small upgrade
> implications since anyone with qemu-user installed today, will loose
> the binary format rules unless they manually install qemu-binfmt. I
> think the number of people affected is probably quite small, and some
> of them may well wish to use qemu-user-static instead anyway.
> Obviously this would only be done in rawhide, not any existing stable
> releases of Fedora.
> Nothing will change about the rest of QEMU packaging - ie all system
> emulators will continue to use dynamic linking
>  https://wiki.debian.org/QemuUserEmulation
> |: http://berrange.com
> |: http://libvirt.org
> |: http://autobuild.org
> |: http://entangle-photo.org
> devel mailing list