Soliciting for input on ARM

[Josh Boyer]

On Wed, Mar 21, 2012 at 10:13:45PM -0400, Jon Masters wrote:

Just a note here: These are pretty much just my thoughts with a kernel
maintainer hat on.  I'm not speaking for the whole team.

> Anyway. Here is the proposal we're working on:
> 
> https://fedoraproject.org/wiki/Features/FedoraARM
> 
> I want to get some specific input on a few questions if I may, and then
> also ask you that you please let me/us know of any concerns/input:
> 
> 0). Hardware. We are making certain plans for ARM hardware to be made
> available for Fedora developers, in addition to the existing FAS-based
> approach we have in Seneca today. What developer hardware would you
> consider that you would want/need in order to be able to support ARM as
> a target in Fedora? Do we need to buy each of you an ARM board? :)

I think that depends on a number of things.  I would have put this
question last if I were making the list to be honest.  However I do
appreciate that you're actively thinking about how to increase the ARM
developer community ;)
 
> 1). Kernel sub-packages. 32-bit ARM is a bit of a zoo, and everyone
> knows it. It's being worked on by Deepak's ARM kernel team at Linaro,
> but well, nobody is kidding around in thinking we're at a single zimage
> stage quite yet. That being the case, and for general sanity and
> supportability, we do not intend to target kernels for every ARM
> platform in existence. Instead, we propose a small number of kernels
> (e.g. for omap, tegra, versatile, pi, and Calxeda highbank) as our upper
> limit, all derived from pure-upstream sources. We currently build a
> default of the versatile kernel (for Versatile Express, the ARM built
> development platform that qemu also provides a simulation thereof) as
> our fallback because it represents real hardware, and is qemu-able.
> 
> We want to be able to build a number of official ARM kernels (which will
> shrink over time), and for those weird and whacky things that nobody is
> even remotely calling mainline Fedora ARM, we're sure third parties will
> provide independent non-Fedora bits for the adventurous. Anyway, back to
> our official Fedora plan. What we don't want to do is disrupt your
> workflows by taking massively longer to build a kernel. Right now, Koji
> tells me an x86 build takes as little as 12 minutes, whereas an ARM
> build (on non-Enterprise hardware, the kind that will be replaced with
> real early 32-bit ARM server builders as part of this) takes 4 hours
> today and will hopefully fall to 1.5, but still won't be instantaneous.
> And that won't give us all the variants either. So we have some
> possibilities:
> 
> 	- We build every ARM kernel every time
> 	- We build e.g. versatile in general and periodically build the other
> kernels by use of a SPEC file macro knob
> 
> I think the second is probably a non-starter for you. So assuming, the
> first is the preferred option, then my question becomes:

The second option isn't necessarily a non-starter.  It depends on how
it's done.  If the macro was in place to enable additional board kernels
and was toggled ON from Beta->release, then it might be acceptable.  You
would leave it off during the ramp up to Beta to limit impact perhaps.
Not great, but not entirely unresonable and somewhat along the lines of
how we handle the debug options for the kernel during development.

My biggest requirement in this regard is that whatever ARM kernels are
built are done from a single SRPM in a single koji build invocation.  If
more kernels are needed, that's a new SRPM and new koji build.  I don't
want to see someone trying to rebuild an SRPM already built in koji to
enable more kernels and shove those into the repo from some side
location.

> 	- How long is acceptable for a kernel build to take?

I'm not sure there's a hard and fast timeframe.  Even the x86 builds
vary for things like building release vs. debug kernels.  Pretty sure
we'd know when we see something that would be bad though.  For example:

http://arm.koji.fedoraproject.org/koji/buildinfo?buildID=58088
http://arm.koji.fedoraproject.org/koji/buildinfo?buildID=56442

Those are probably way too long.  This one:

http://arm.koji.fedoraproject.org/koji/buildinfo?buildID=59980

seems totally acceptable to me, but looking at what it produced I would
gather it's mostly useless for the ARM project (perhaps just the qemu
thing?).  I'm not sure building only the qemu kernel is a great way to
go either, both from a ARM _and_ kernel perspective.  It might be better
to build that + 1 board from each armv5tel and armv7hl.  I know that
gets into "which do we build" but at least it allows people to test on
their hardware.  I looked, but didn't seem to find a build configured
like that.  It would be interesting to see what the time looks like
there.

> Now, a trivial SPEC file or general non-arch kernel bug is likely to
> fail on x86 well before it fails on ARM. That will of course take care
> of many generic build issues that will fail a parent Koji build quickly.

Yes.  However, waiting for a day for your build to complete only to have
it be canceled because some ARM variant died is one of the larger
concerns.

The time issue mostly becomes important when it starts negatively
impacting our ability to get fixes out, particularly security fixes.  If
we have to wait 8 hours to get a build for a CVE fix, that effectively
means we're not going to get it out until the following day in a lot of
cases.

The other time sensitive issue is during a rebase of the kernel in
rawhide.  Not sensitive in the same way as above, but along the lines of
"oh hell.  ARM doesn't build in the latest merge window kernel I'm
trying to build and it took 12 hours to find that out."  That kind of
thing is going to seriously wear on the patience of whoever happens to
be working on rawhide that month.  I realize there's no great solution
to that, but it is a concern.

> issue. So, another option is that we modify Koji to submit sub-package
> tasks across multiple builders. i.e. all of the ARM subpackages (and
> this would happen for x86 variants too) would get submitted to builders
> at the same time, rather than linearly. It's a lot of work, but it's
> doable. Especially if it's the only option. It comes down to how long
> you guys think is the longest you are willing to wait for an overall
> all-arch build of the kernel to take in Koji.

That sounds like you'd need to both modify Koji to handle that, and
rework package spec files to somehow tell Koji "hey, we're going to be X
variants from this SRPM".  I'm not sure the kernel.spec lends itself to
easily farming out builds to separate machines at the moment, since it
just loops through all variants it needs to build and does them in a
serial fashion (e.g. i686, i686-debug, i686-PAE, i686-PAEdebug).

That option is interesting, but I don't consider it feasible for f18 at
all and I would even be surprised if koji was able to handle it for f19.

> 2). Impact. What does making ARM a primary arch mean to you? Not what do
> we think it means, but what do you think it means to you in terms of:
> 
> 	- How will this positively or negatively impact your role?

Positive: Broader Fedora dominance, I guess.  Maybe some new learning.
Negative: More bugs, more platforms to support, more time spent.

That might be pretty black and white, but at least I came up with _some_
positives.  Honestly, if ARM is handled well upstream and by the ARM
team, it might not impact us more than waiting a bit for builds.  If it
_isn't_ handled well upstream it can quickly become a nightmare.

> 	- What level of disruption are you willing to accept?

Can you elaborate on disruption?  We already spend time mucking around
with ARM configs during rebase (not intelligently mind you), and I don't
think we grumbled too much about it.  Aside from more bugs, is there
something else you were thinking?

> 	- Are you willing to make any changes to workflow?

I think we're pretty open minded.  We might not be as gung-ho about
something as the owners, but if the changes aren't unreasonable we tend
to evaluate them fairly.  It's going to depend on what's proposed.

I'll counter with a question to you.  Who (as in name(s)) is going to be
the main ARM kernel person?  My ARM knowledge is limited at best, I
don't think Dave has much at all, and I'm not sure about Justin but I'd
be surprised if he was a closet ARM expert.  Do you have people in mind
to handle the HW specific issues that pop up so we can assign bugs to
them?

> We don't intend for this to be a trainwreck. If it's not ready, it won't
> be PA, period. But we want to know how flexible you guys are willing to
> be as we figure this out. If you want to wait until we have total
> parity, single zimage, and we're just like x86, that is good to know
> (and discuss) right now :)

Personally, I don't think it has to be one vmlinux + 18 FDTs to be
suitable.  A smaller set of board specific kernels is probably doable,
depending on some of the factors mentioned above.  I see that as similar
to how we built e.g. ppc, ppc-smp, ppc64, ppc64-kdump.  Nobody
particularly _liked_ doing that but we lived.

josh