Soliciting for input on ARM

[Jon Masters]

Hi everyone,

As you probably know, we had a FESCo meeting the other day, where we
raised our "work in progress" proposal for Fedora ARM Primary Arch
status in F18. This is a work in progress, and isn't final yet. We meant
to ping you about now, but probably should have done so even before
posting that draft so you don't feel left out :) Note that we know
Fedora 18 is an optimistic goal, but this is Fedora, and there will of
course be a Fedora 19, and 20, and so on if it takes that.

Why are we doing this? We think it's consistent with Fedora's mission
statement. After all, Fedora is all about driving technology, and if you
want to see where the buzz and excitement is, any IT magazine will tell
you that ARM is the "next big thing". We think it already is pretty big,
and it's growing. In a few years, it's going to be just amazing. We can
get there as a Secondary if we have to, but we think it's time to start
looking at changing that. Unlike other secondaries, ARM is growing
rapidly, and it's cheap. You can buy an ARM board for less than an order
of magnitude cheaper than a cheap Intel box.

All this awesomeness does mean you get some compromises. For example,
typical Fedora ARM users don't run Anaconda today, they dd a pre-built
image onto an SD Card and boot that (and we're working on getting these
built using the standard "appliance" like Fedora tooling). In the coming
months though, we'll see ARM servers with PXE and TFTP. In the future,
you'll have ARM servers booting using UEFI and smelling increasingly
like x86 over time. But it's not just server. The Raspberry Pi is going
to be - *is* - blowing everyone away. It reminds me of being a kid and
getting a BBC Master/B computer and being able to write programs and
understand how the computer actually worked. As Eben says, he's trying
to rekindle that kind of feeling kids used to have. We have made good
choices to keep armv5 support around and that means many Pi users will
be running Fedora by *default*. In the future, the next OLPC device (as
you know) will join the Pi in being ARM based.

Anyway. Here is the proposal we're working on:

https://fedoraproject.org/wiki/Features/FedoraARM

I want to get some specific input on a few questions if I may, and then
also ask you that you please let me/us know of any concerns/input:

0). Hardware. We are making certain plans for ARM hardware to be made
available for Fedora developers, in addition to the existing FAS-based
approach we have in Seneca today. What developer hardware would you
consider that you would want/need in order to be able to support ARM as
a target in Fedora? Do we need to buy each of you an ARM board? :)

1). Kernel sub-packages. 32-bit ARM is a bit of a zoo, and everyone
knows it. It's being worked on by Deepak's ARM kernel team at Linaro,
but well, nobody is kidding around in thinking we're at a single zimage
stage quite yet. That being the case, and for general sanity and
supportability, we do not intend to target kernels for every ARM
platform in existence. Instead, we propose a small number of kernels
(e.g. for omap, tegra, versatile, pi, and Calxeda highbank) as our upper
limit, all derived from pure-upstream sources. We currently build a
default of the versatile kernel (for Versatile Express, the ARM built
development platform that qemu also provides a simulation thereof) as
our fallback because it represents real hardware, and is qemu-able.

We want to be able to build a number of official ARM kernels (which will
shrink over time), and for those weird and whacky things that nobody is
even remotely calling mainline Fedora ARM, we're sure third parties will
provide independent non-Fedora bits for the adventurous. Anyway, back to
our official Fedora plan. What we don't want to do is disrupt your
workflows by taking massively longer to build a kernel. Right now, Koji
tells me an x86 build takes as little as 12 minutes, whereas an ARM
build (on non-Enterprise hardware, the kind that will be replaced with
real early 32-bit ARM server builders as part of this) takes 4 hours
today and will hopefully fall to 1.5, but still won't be instantaneous.
And that won't give us all the variants either. So we have some
possibilities:

	- We build every ARM kernel every time
	- We build e.g. versatile in general and periodically build the other
kernels by use of a SPEC file macro knob

I think the second is probably a non-starter for you. So assuming, the
first is the preferred option, then my question becomes:

	- How long is acceptable for a kernel build to take?

Now, a trivial SPEC file or general non-arch kernel bug is likely to
fail on x86 well before it fails on ARM. That will of course take care
of many generic build issues that will fail a parent Koji build quickly.
And we're looking to base the official Fedora ARM builder hardware in
Phoenix on the first generation 32-bit servers, and these will have up
to 288 cores in 2U. Several of those should provide enough builders that
we don't get to a point of not accepting additional builds. Still,
that's great, but pipelining doesn't really solve the fundamental speed
issue. So, another option is that we modify Koji to submit sub-package
tasks across multiple builders. i.e. all of the ARM subpackages (and
this would happen for x86 variants too) would get submitted to builders
at the same time, rather than linearly. It's a lot of work, but it's
doable. Especially if it's the only option. It comes down to how long
you guys think is the longest you are willing to wait for an overall
all-arch build of the kernel to take in Koji.

To be clear, nothing in here is concerned with 64-bit yet. Those crazy
core counts I'm talking about in servers we'll use for building are all
happening this year, in the next 6 months. They'll be individually
coherent quad core Cortex-A9-like systems, like your Android tablet on
steroids, but not (yet) taking on x86. We don't ever expect ARM to take
on Intel head-to-head for performance, they're in it for density and
power saving, amongst other things. Next year, we'll see A15 based
servers. And the following year or thereabouts, I suspect we'll see the
first ARMv8 (AArch64) servers in testing. At that point, you're looking
at individual coherence domains of 8 cores+ (and it's a lot +), at a few
GHz, and so on. So we'll get you higher performing builders :)

2). Impact. What does making ARM a primary arch mean to you? Not what do
we think it means, but what do you think it means to you in terms of:

	- How will this positively or negatively impact your role?
	- What level of disruption are you willing to accept?
	- Are you willing to make any changes to workflow?

We don't intend for this to be a trainwreck. If it's not ready, it won't
be PA, period. But we want to know how flexible you guys are willing to
be as we figure this out. If you want to wait until we have total
parity, single zimage, and we're just like x86, that is good to know
(and discuss) right now :)

I'm sure there's other stuff we can discuss. So let's do that :) If we
have useful input on IRC, I'll followup to this thread, but I like the
idea of doing it by email to keep a record of the discussion.

Thanks for reading - whew! I am...verbose.

Jon.