multi-CPU optimization inside a distribution

[Dan Horák]

Hi all,

we were talking about this item for some time, so let's start a thread
for it to have the discussion and hopefully also a solution documented.
This is meant as discussion initiation based on the situation in
Fedora on POWER. I would like to ask the bigger experts than me to fill
the missing details and options.

Currently we set a minimal CPU level for an architecture (or use the
toolchain default) on the distribution level
(/usr/lib/rpm/redhat/rpmpc owned by redhat-rpm-config [1]). It allows
the distro to run on such CPU and any newer evolution of it (omitting
any kernel or hw issues), but it also means it doesn't generally take
advantage of features and improvements in the newer CPUs.

For ppc64 (the big endian POWER) the base is set by the toolchain
default which is Power4/ppc970. When Power7 came we were asked what are
the options to take the advantage of these CPUs, 3 generations newer
than the base. The solution was the introduction of ppc64p7 subarch into
the packaging and release engineering tools. But it showed more as a
hack than a solution, touching rpm, yum, koji, .... The list of packages
is manually maintained, requires manual updates to the buildsystem for
new releases, seems new packaging tools (dnf) don't understand it
correctly. Is there a way to make the subarch mechanism generic and
integrated with the other tools? So we could have ppc64p8 and ppc64p9
inside Fedora for POWER ...

Now I'm getting to an area where others are the experts :-) Glibc
allows to build and install multiple per-cpu optimized runtimes that
are selected based on the CPU. There is the IFUNC mechanism and
function multi-versioning in GCC (user-friendly IFUNC) [2] to allow
multiple implementations inside one library/binary.

Some packages do the CPU detection during runtime and select the
optimized functions themselves.

There is also an option to introduce a "tertiary architecture" and
rebuild everything for the new CPU, but keeping the rpm arch the same.
But it has its own costs too.

What do you think? Are there any recommendations for both the distro
and package maintainers and upstream developers? I suppose even primary
architectures are facing the same problem.

[1] http://pkgs.fedoraproject.org/cgit/rpms/redhat-rpm-config.git/tree/rpmrc
[2] https://lwn.net/Articles/691932/


		Dan

[Jan Kratochvil]

On Thu, 30 Jun 2016 12:12:37 +0200, Dan Horák wrote:
>  For ppc64 (the big endian POWER) the base is set by the toolchain
>  default which is Power4/ppc970. When Power7 came we were asked what are
>  the options to take the advantage of these CPUs, 3 generations newer
>  than the base. The solution was the introduction of ppc64p7 subarch 
But now there is ppc64le for >=Power8.  So <=Power7 can use ppc64 but newer
> =Power8 hardware does use ppc64le which is already Power8 optimized.
So isn't this problem (somehow temporarily) solved for now?

Or are really the Power7 (and not yet Power8) machines the real problem?


Jan

[Josh Boyer]

On Thu, Jun 30, 2016 at 9:53 AM, Jan Kratochvil
<jan.kratochvil(a)redhat.com&gt; wrote:
>  On Thu, 30 Jun 2016 12:12:37 +0200, Dan Horák wrote:
> > For ppc64 (the big endian POWER) the base is set by the toolchain
> > default which is Power4/ppc970. When Power7 came we were asked what are
> > the options to take the advantage of these CPUs, 3 generations newer
> > than the base. The solution was the introduction of ppc64p7 subarch
>
>  But now there is ppc64le for >=Power8.  So <=Power7 can use ppc64 but newer
> >=Power8 hardware does use ppc64le which is already Power8 optimized.
>  So isn't this problem (somehow temporarily) solved for now? 
POWER8 can run in either endian mode.  So no, the problem is not solved.

>  Or are really the Power7 (and not yet Power8) machines the real
> problem? 
Realistically, it isn't about either specifically.  Each iteration of
POWER tends to require tuning specifically for that generation if you
want to get the most performance out of your software.  That tuning is
incompatible with older generations of hardware (endian completely
aside).  So as Dan alludes to, this is fairly cumbersome.  To make it
widely useable, you default to the oldest generation you want to
support.

x86 theoretically has similar problems, but we tend to just not care
as much about tuning at the distro level.  The applications can tune
however they'd like, but given that most datacenters are not
homogeneous among x86 machines, it often means people cannot tune for
one specific x86 generation because they have to run on a wider
variety.

Contrast that with POWER, where people using that tend to buy one or
two very large systems to consolidate their workload on.  They can and
often do want the best performance they can eek out since they're
consolidating, which means they often want the distro components tuned
as well, particularly system libraries.

Going forward, I'd seriously look at dropping anything older than
POWER7 support.  I know we all dearly love our Mac G5 machines
(ppc970), but those are getting very very old at this point.  POWER5
was a non-event for Fedora, POWER6 was a weird generation even for
IBM.  POWER7 and POWER8 are where most of the effort is, and that will
only continue as next generation POWER chips come out.  Given that
Rackspace and others have actually started using the openPOWER stuff
with POWER8, it probably makes sense to focus on where the users are
headed and not where they were 10 years ago.

josh

[Jan Kratochvil]

On Thu, 30 Jun 2016 16:13:59 +0200, Josh Boyer wrote:
>  On Thu, Jun 30, 2016 at 9:53 AM, Jan Kratochvil
> <jan.kratochvil(a)redhat.com&gt; wrote:
>  > But now there is ppc64le for >=Power8.  So <=Power7 can use ppc64 but newer
>  >>=Power8 hardware does use ppc64le which is already Power8 optimized.
>  > So isn't this problem (somehow temporarily) solved for now?
>  
>  POWER8 can run in either endian mode.  So no, the problem is not solved. 
End-users do not even know what is endianity.  And all application developers
need to write software BE/LE-independently (such as using ntohl()/htonl()
etc.) so they also do not care whether the host is BE or LE.  Only <1% of
developers for high performance core or very low-end libraries (like glibc or
glib) need to care about BE/LE.

So why to run Power8 in BE?  Power is switching to LE AFAIK which makes it
also better compatible with buggy software ignoring BE/LE difference which is
(as all the software) targeting primarily x86* (=LE).


>  Realistically, it isn't about either specifically.  Each
> iteration of
>  POWER tends to require tuning specifically for that generation if you
>  want to get the most performance out of your software. 
Latest CPU features get utilized mostly only in inner loops of high
performance code which should use IFUNC.  So building the whole distro for the
latest CPU brings only incompatibility with no real performance advantage.


At least this is my expectation, the OP misses any real benchmark numbers
whether the distro build for the latest CPU generation is worth it.
IIRC for x86* it was not worth it.


Jan

[Peter Robinson]

On Thu, Jun 30, 2016 at 5:01 PM, Jan Kratochvil
<jan.kratochvil(a)redhat.com&gt; wrote:
>  On Thu, 30 Jun 2016 16:13:59 +0200, Josh Boyer wrote:
> > On Thu, Jun 30, 2016 at 9:53 AM, Jan Kratochvil <jan.kratochvil(a)redhat.com&gt;
> wrote:
> > > But now there is ppc64le for >=Power8.  So <=Power7 can use ppc64 but
> newer
> > >>=Power8 hardware does use ppc64le which is already Power8 optimized.
> > > So isn't this problem (somehow temporarily) solved for now?
> >
> > POWER8 can run in either endian mode.  So no, the problem is not solved.
>
>  End-users do not even know what is endianity.  And all application developers
>  need to write software BE/LE-independently (such as using ntohl()/htonl()
>  etc.) so they also do not care whether the host is BE or LE.  Only <1% of
>  developers for high performance core or very low-end libraries (like glibc or
>  glib) need to care about BE/LE.
>
>  So why to run Power8 in BE?  Power is switching to LE AFAIK which makes it
>  also better compatible with buggy software ignoring BE/LE difference which is
>  (as all the software) targeting primarily x86* (=LE). 
Data conversion is expensive, slow and problematic so once it's in one
format it tends to stay that way. There's also legacy/proprietary
tools that run as big endian on the Power64 platform. For a new
deployment there's no reason not to do it as LE from the outset, and
migration from other LE platforms makes sense but if you're in a BE
world (AIX, Solaris, Z-series and friends) it makes sense to run in
BE.

There are some applications where the data comes in in BE format and
it's costly to deal with that in real time, especially in high
throughput scenarios.

> > Realistically, it isn't about either specifically.  Each
> iteration of
> > POWER tends to require tuning specifically for that generation if you
> > want to get the most performance out of your software.
>
>  Latest CPU features get utilized mostly only in inner loops of high
>  performance code which should use IFUNC.  So building the whole distro for the
>  latest CPU brings only incompatibility with no real performance advantage. 
We're looking at IFUNC as one option, Function multi-versioning in GCC
6 [1] is presumably also another option, although wouldn't be useful
for already released enterprise releases, although obviously
irrelevant for the Fedora conversation.

>  At least this is my expectation, the OP misses any real benchmark
> numbers
>  whether the distro build for the latest CPU generation is worth it.
>  IIRC for x86* it was not worth it. 

[1] https://lwn.net/Articles/691932/

[Josh Boyer]

On Thu, Jun 30, 2016 at 12:01 PM, Jan Kratochvil
<jan.kratochvil(a)redhat.com&gt; wrote:
>  On Thu, 30 Jun 2016 16:13:59 +0200, Josh Boyer wrote:
> > On Thu, Jun 30, 2016 at 9:53 AM, Jan Kratochvil <jan.kratochvil(a)redhat.com&gt;
> wrote:
> > > But now there is ppc64le for >=Power8.  So <=Power7 can use ppc64 but
> newer
> > >>=Power8 hardware does use ppc64le which is already Power8 optimized.
> > > So isn't this problem (somehow temporarily) solved for now?
> >
> > POWER8 can run in either endian mode.  So no, the problem is not solved.
>
>  End-users do not even know what is endianity.  And all application developers
>  need to write software BE/LE-independently (such as using ntohl()/htonl()
>  etc.) so they also do not care whether the host is BE or LE.  Only <1% of
>  developers for high performance core or very low-end libraries (like glibc or
>  glib) need to care about BE/LE.
>
>  So why to run Power8 in BE?  Power is switching to LE AFAIK which makes it
>  also better compatible with buggy software ignoring BE/LE difference which is
>  (as all the software) targeting primarily x86* (=LE). 
It isn't switching.  It's adding.  I'd maybe agree that LE will become
more prevalent over time, particularly as it is used in cloud
workloads.  However, BE mode isn't going away anytime soon.

> > Realistically, it isn't about either specifically.  Each
> iteration of
> > POWER tends to require tuning specifically for that generation if you
> > want to get the most performance out of your software.
>
>  Latest CPU features get utilized mostly only in inner loops of high
>  performance code which should use IFUNC.  So building the whole distro for the
>  latest CPU brings only incompatibility with no real performance advantage.
>
>
>  At least this is my expectation, the OP misses any real benchmark numbers
>  whether the distro build for the latest CPU generation is worth it.
>  IIRC for x86* it was not worth it. 
You might not be wrong, but you're dismissing the market dynamics of
POWER that I mentioned.  Those same dynamics lead to the distribution
components for ppc64 being built as 64-bit binaries, even though at
the time 32-bit binaries for 99% of packages were actually more than
sufficient.

Comparing performance numbers within POWER is fine, but applying the
"this is how we do it in x86" logic isn't really suitable.  They are
different markets.  I believe the markets will perhaps merge over
time, but they clearly haven't yet today.

josh

[Gerald Henriksen]

On Thu, 30 Jun 2016 18:01:38 +0200, you wrote:

> > Realistically, it isn't about either specifically.  Each
> iteration of
> > POWER tends to require tuning specifically for that generation if you
> > want to get the most performance out of your software.
>
> Latest CPU features get utilized mostly only in inner loops of high
> performance code which should use IFUNC.  So building the whole distro for the
> latest CPU brings only incompatibility with no real performance advantage. 
When you spend the type of money that a POWER type system costs, you
expect to get every bit of benefit you can out of that system, and
that means everything from the OS on up will be expected to be
compiled for the version of POWER you are running.

[Peter Robinson]

On Thu, Jun 30, 2016 at 2:53 PM, Jan Kratochvil
<jan.kratochvil(a)redhat.com&gt; wrote:
>  On Thu, 30 Jun 2016 12:12:37 +0200, Dan Horák wrote:
> > For ppc64 (the big endian POWER) the base is set by the toolchain
> > default which is Power4/ppc970. When Power7 came we were asked what are
> > the options to take the advantage of these CPUs, 3 generations newer
> > than the base. The solution was the introduction of ppc64p7 subarch
>
>  But now there is ppc64le for >=Power8.  So <=Power7 can use ppc64 but newer
> >=Power8 hardware does use ppc64le which is already Power8 optimized.
>  So isn't this problem (somehow temporarily) solved for now?
>
>  Or are really the Power7 (and not yet Power8) machines the real problem? 
No your confusing endian with CPU features. We support both Big Endian
(ppc64) and Little Endian (ppc64le) options for the Power64
architecture. Both can run on Power8 hardware, in fact the builder VMs
run like with with both BE and LE VMs running on the same physical
Power8 host side by. There's shortly going to also be Power9 [1] with
options that need optimisation to make the most of new architectural
enhancements which will be primarily focused towards ppc64le (but will
also work on ppc64) but we don't want to exclude Power8 from ppc64le.

Peter

[1]
http://www.nextplatform.com/2016/04/18/power9-will-bring-competition-data...

[Richard W.M. Jones]

On Thu, Jun 30, 2016 at 12:12:37PM +0200, Dan Horák wrote:
>  For ppc64 (the big endian POWER) the base is set by the toolchain
>  default which is Power4/ppc970. When Power7 came we were asked what are
>  the options to take the advantage of these CPUs, 3 generations newer
>  than the base. The solution was the introduction of ppc64p7 subarch into
>  the packaging and release engineering tools. But it showed more as a
>  hack than a solution, touching rpm, yum, koji, .... The list of packages
>  is manually maintained, requires manual updates to the buildsystem for
>  new releases, seems new packaging tools (dnf) don't understand it
>  correctly. Is there a way to make the subarch mechanism generic and
>  integrated with the other tools? So we could have ppc64p8 and ppc64p9
>  inside Fedora for POWER ... 
I will just snipe here that the situation would be better if POWER > 4
hardware was generally available to buy.

>  Now I'm getting to an area where others are the experts :-)
> Glibc
>  allows to build and install multiple per-cpu optimized runtimes that
>  are selected based on the CPU. There is the IFUNC mechanism and
>  function multi-versioning in GCC (user-friendly IFUNC) [2] to allow
>  multiple implementations inside one library/binary. 
Be careful with using IFUNC.  It's a very desirable feature,
implemented in a somewhat ill-defined way:

https://bugzilla.redhat.com/show_bug.cgi?id=1312462#c22
https://bugzilla.redhat.com/show_bug.cgi?id=1312462#c27
  etc.

Have you looked at what Debian does with the /lib/<arch> directories?
I think we should follow Debian if appropriate.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
libguestfs lets you edit virtual machines.  Supports shell scripting,
bindings from many languages.  http://libguestfs.org

[Solomon Peachy]

On Thu, Jun 30, 2016 at 08:04:57PM +0100, Richard W.M. Jones wrote:
>  I will just snipe here that the situation would be better if POWER
> > 4
>  hardware was generally available to buy. 
The upcoming Raptor Talos POWER8 board [1] looks promising, but at $3700 
a pop just for the motherboard and an entry-level [1] CPU, it's out of 
reach of many otherwise-interested developers.  Assuming it goes into 
production at all..

  [1] https://www.raptorengineering.com/TALOS/
  [2] Insofar as an 8-core/64-thread CPU can be called entry-level

 - Solomon
-- 
Solomon Peachy        		       pizza at shaftnet dot org
Delray Beach, FL                          ^^ (email/xmpp) ^^
Quidquid latine dictum sit, altum viditur.

[Dan Horák]

On Thu, 30 Jun 2016 15:59:38 -0400
Solomon Peachy <pizza(a)shaftnet.org&gt; wrote:

>  On Thu, Jun 30, 2016 at 08:04:57PM +0100, Richard W.M. Jones wrote:
>  > I will just snipe here that the situation would be better if POWER
>  > > 4 hardware was generally available to buy.
>  
>  The upcoming Raptor Talos POWER8 board [1] looks promising, but at
>  $3700 a pop just for the motherboard and an entry-level [1] CPU, it's
>  out of reach of many otherwise-interested developers.  Assuming it
>  goes into production at all.. 
unfortunately it does not in foreseeable future (based on official
update on their IRC channel). It was my hope to get newer workstation
class HW and be able to lift the ppc64 (BE) support to Power7 and up
(or so). With the OpenPOWER ecosystem there is still chance someone
else will try it.


		Dan

[Gerald Henriksen]

On Thu, 30 Jun 2016 22:07:24 +0200, you wrote:

> On Thu, 30 Jun 2016 15:59:38 -0400
> Solomon Peachy <pizza(a)shaftnet.org&gt; wrote:
>
> > On Thu, Jun 30, 2016 at 08:04:57PM +0100, Richard W.M. Jones wrote:
> > > I will just snipe here that the situation would be better if POWER
> > > > 4 hardware was generally available to buy.
> > 
> > The upcoming Raptor Talos POWER8 board [1] looks promising, but at
> > $3700 a pop just for the motherboard and an entry-level [1] CPU, it's
> > out of reach of many otherwise-interested developers.  Assuming it
> > goes into production at all..
>
> unfortunately it does not in foreseeable future (based on official
> update on their IRC channel). It was my hope to get newer workstation
> class HW and be able to lift the ppc64 (BE) support to Power7 and up
> (or so). With the OpenPOWER ecosystem there is still chance someone
> else will try it. 
I wouldn't count on it.

To get developers they need to get a CPU/motherboard combination out
for under $1000, likely closer to $500, as these will be purchases as
a secondary system to "play with" because there is no direct way to
make money as a POWER developer (this also applies to Aarch64, which
is also looking rather absent).

At least for POWER8, this would likely mean someone from OpenPOWER
would need to fund the development of a affordable POWER8 chip with no
expectation of directly recouping that cost, which seems unlikely.

Which is a shame, because with more developers/users there might be
more volunteers for Fedora such that a specific POWER8 spin would
become viable, helping to solve the current problem.

[Bruno Wolff III]

On Fri, Jul 01, 2016 at 00:34:02 -0400,
  Gerald Henriksen <ghenriks(a)gmail.com&gt; wrote:
> To get developers they need to get a CPU/motherboard combination out
> for under $1000, likely closer to $500, as these will be purchases as
> a secondary system to "play with" because there is no direct way to
> make money as a POWER developer (this also applies to Aarch64, which
> is also looking rather absent). 
There is another reason to get the power8 Talos board and that is that you 
get source code for the firmware and there is a reduced chance of a 
deliberate backdoor and bugs that are discovered will br fixable. You don't 
get that with recent Intel or AMD systems.

This doesn't solve problems with firmware in storage systems. They also 
weren't able to include complete documentation related to some parts used 
because of NDAs.

This is still a significant change, and if people want to see hardware 
makers move in this direction voting with your wallet helps.

If I could afford it, I'd pay a significant premium to get a Talos board. 
But as it is, I'm going to be stuck using second hand, pretty old computers 
for the foreseeable future.

[Solomon Peachy]

On Fri, Jul 01, 2016 at 11:35:27AM -0500, Bruno Wolff III wrote:
>  If I could afford it, I'd pay a significant premium to get a
> Talos board.
>  But as it is, I'm going to be stuck using second hand, pretty old computers
>  for the foreseeable future. 
And even though "significant premium" is a polite understatement in this 
case, the Talos was the cheapest [proposed] POWER8 solution by a 
substantial margin.

I've also been quite disappointed in the AArch64 offerings so far.

With very few exceptions (eg Gigabyte's mostly-unobtainable MP30-AR0 or 
some locked-down appliance-type servers) they've all been ultra-dense 
high-end boxes (dozens of cores per U) priced into the stratosphere or 
glorified cell phone devkits (eg Tegra-based stuff) that don't sport 
enough memory (or other expandability) to run a modern desktop OS.

/rant

 - Solomon
-- 
Solomon Peachy        		       pizza at shaftnet dot org
Delray Beach, FL                          ^^ (email/xmpp) ^^
Quidquid latine dictum sit, altum viditur.

[Josh Boyer]

On Fri, Jul 1, 2016 at 1:42 PM, Solomon Peachy <pizza(a)shaftnet.org&gt; wrote:
>  On Fri, Jul 01, 2016 at 11:35:27AM -0500, Bruno Wolff III wrote:
> > If I could afford it, I'd pay a significant premium to get a Talos board.
> > But as it is, I'm going to be stuck using second hand, pretty old computers
> > for the foreseeable future.
>
>  And even though "significant premium" is a polite understatement in this
>  case, the Talos was the cheapest [proposed] POWER8 solution by a
>  substantial margin.
>
>  I've also been quite disappointed in the AArch64 offerings so far.
>
>  With very few exceptions (eg Gigabyte's mostly-unobtainable MP30-AR0 or
>  some locked-down appliance-type servers) they've all been ultra-dense
>  high-end boxes (dozens of cores per U) priced into the stratosphere or
>  glorified cell phone devkits (eg Tegra-based stuff) that don't sport
>  enough memory (or other expandability) to run a modern desktop OS. 
So the thing is, I'm genuinely curious why you would want to run a
desktop OS on either architecture.

From a development perspective, they likely are not going to be wide
enough to attract a ton of manufacturer attention in the
laptop/desktop form factors.  They're a niche market at best, likely
to be filled by Chromebooks in the aarch64 case.  As a developer,
you'd be targeting ChromeOS or Android in that case, not plain Linux.
Certainly not Windows.  All of that seems rather speculative for
making it an attractive development target for a desktop OS and
applications that run on one.

From a manufacturer perspective, you aren't going to make
laptop/desktop boards.  The margins on that would be extremely small,
or your product would be so expensive nobody would buy it.  What
you're going to do is focus on things where you can get
bang-for-the-buck.  Either a consolidation play in the datacenter or
in a massive scale-out workload that's energy efficient.  In either
case, you make the big expensive machines or you make smaller boards
that can expand out in a rack.  That leads to a datacenter/cloud
market.

Cloud instances are relatively cheap, and can be paid for by the hour.
I would think we'd want to focus our efforts around good Cloud and
Server Editions for these architectures.  Doing so attracts both
manufacturers and developers at the same time.

Sure, it would be neat to have a desktop solution and I'm not saying
we shouldn't try there.  I simply think that unlike x86_64, it is
likely the least important thing to focus on because the hardware
won't really be suitable and the developer market is going to be
mostly focused elsewhere for these architectures.

josh

[Solomon Peachy]

On Fri, Jul 01, 2016 at 02:08:48PM -0400, Josh Boyer wrote:
>  So the thing is, I'm genuinely curious why you would want to run
> a
>  desktop OS on either architecture. 
For the same reason Fedora puts out releases targeting those 
architechures?  :) 

I'm not wedded to any particular hardware architechure.  The only 
proprietary software on my personal systems are their BIOSes and other 
low-level firmware.  I suspect that, in the Fedora world, I'm far from 
unique.

 - Solomon
-- 
Solomon Peachy        		       pizza at shaftnet dot org
Delray Beach, FL                          ^^ (email/xmpp) ^^
Quidquid latine dictum sit, altum viditur.

[Richard W.M. Jones]

On Fri, Jul 01, 2016 at 01:42:40PM -0400, Solomon Peachy wrote:
>  With very few exceptions (eg Gigabyte's mostly-unobtainable
> MP30-AR0 or 
>  some locked-down appliance-type servers) they've all been ultra-dense 
>  high-end boxes (dozens of cores per U) priced into the stratosphere or 
>  glorified cell phone devkits (eg Tegra-based stuff) that don't sport 
>  enough memory (or other expandability) to run a modern desktop OS. 
While I generally agree with what you're saying, I just want to point
out a few systems that are available now.

The Gigabyte MP30-AR0 can be purchased, eg from
https://www.xcase.co.uk/server-motherboards/gigabyte-mp30-ar0-with-applie...

It's basically an X-Gene 1 which is 3+ years old at this point (made
on an even older process) so don't expect dazzling performance, but
this is still a reliable workhorse for development.  It boots the
upstream kernel, takes up to 128 GB of RAM, and has UEFI
(unfortunately not out of the box).  I have one, and I can stuff a
dozen virtual machines, mixed armv7 & aarch64, without it breaking
into a sweat.

Just appeared a few days ago is this one:
http://www.cnx-software.com/2016/06/26/599-softiron-overdrive-1000-server...
Quad core AMD Opteron (codename Seattle, Cortex A57).  A bit cheaper
than the Gigabyte.  UEFI out of the box and comes with SUSE installed.

Finally the LeMaker Cello (also quad core AMD Seattle) is maybe/
possibly/probably going to ship this month, at least according to an
email that they sent out a couple of days ago.  If and when I ever get
it I'll write about it on my blog, https://rwmj.wordpress.com/

I think the aarch64 situation - while hardly ideal - is much better
than POWER.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine.  Supports Linux and Windows.
http://people.redhat.com/~rjones/virt-df/