Phase out 32bit ppc due to bugfix and maintenance burden

[Phil Knirsch]

On 05/09/2014 12:15 PM, David Woodhouse wrote:
> On Fri, 2014-05-09 at 11:32 +0200, Phil Knirsch wrote:
>>
>> So for 1) this would effectively mean we'd have to look at every failed
>> build for 32bit ppc and put ExcludeArch: ppc in it. On top of that we'd
>> then need to additionally look for all components that require or
>> buildrequire recursively these packages and do the same for all those
>> packages, too. Just for the java stack thats several hundreds of
>> packages alone. Overall thats an enormous amount of effort and would
>> have to be constantly done for any new 32bit ppc package failing, so we
>> dropped this solution.
>
> Hm, does this really have to be that much work? If feasible, it would
> have been my preferred option. Although since I do almost nothing on PPC
> these days I appreciate how little my opinion counts :)
>
> Let's assume that you can write a script which, when run as a
> provenpackager, will check out a package from the git tree, add the
> ExcludeArch tag to it and commit it, and file a bug in bugzilla marked
> as blocking the 'PPC32 ExcludeArch tracker'. So the actual work involved
> in adding the ExcludeArch is basically limited to giving a *reason*.
>

Sure, but remember the chained dependencies we have in the build 
requires in Fedora. One of the recent efforts in the Base WG that me and 
Harald Hoyer were actually driving was investigate the inter build 
dependencies for a self hosting Fedora that includes gcc, rpm, yum and 
anaconda. We ended up needing over 1800 packages to be self hosting, but 
if we'd drop the self hosting requirement that dropped to less than 700. 
Which means the build requires pulls in more than 2x the amount of 
packages. And if you e.g. have to ExcludeArch eclipse (just as a typical 
example of a component that has proven to be quite difficult) you're 
then either left with unraveling the whole build requires chain of 
eclipse, java and all it's friends or excludearch the whole java world, 
which then in chain requires you to disable db4 and db, which is 
required by rpm. So you can't really disable the java stack but have to 
manually clean up the buildrequires and as you can imagine, doing that 
is a lot of work.

> It's also not that hard to use repoquery or similar tools to
> automatically run that script up the dependency chain, surely?

Jup, we've done exactly that for the Base WG effort. Harald produced a 
nice graph of the inter dependencies here:

http://www.harald-hoyer.de/2014/01/14/self-hosting-fedora-base/

So we know it's a mess, and running repoquery regularly only will 
confirm that, but without someone dedicated and having the spare cycles 
to actually do the work to unravel those, that in itself won't help a 
lot unfortunatly. :(

> If we're genuinely talking about an "enormous amount of effort" then we
> really can script it and make it easy. And "constantly done for any new
> 32bit ppc package failing" also surprised me. As I said, I've been
> paying little attention to PPC for a while, but we used to have fairly
> much 100% coverage... are things really falling apart that much, and at
> such a rate?

It's rather often core packages where we see build failures, be it java 
or toolchain related, or somewhere in the high level desktop space where 
things are quite dependent on another, too.

And it's often enough to having us spend a considerable amount of time 
having to investigate the failures, work with maintainers on potential 
fixes etc. And although most maintainers are really very helpful and 
forthcoming with finding fixes even for secondary architectures, 
occasionally we get a "don't have time" from someone (which is of course 
totally understandable and not meant as an accusation), which then means 
we'll have to find the fix ourselves or again, unravel dep build chains 
and would have to start ExludeArchs:

> If we really can't manage this, then I suppose I have no fundamental
> objection to just configuring the PPC koji to build only 64-bit from now
> on, and let someone set up a PPC32 koji instance if they want to. I
> think that's what you're proposing, right? All the RPM multilib bits
> ought to still work, if the packages come into existence again.
>

Exactly, maybe i should have been clearer about that. We're totally fine 
and would be happy if someone sets up a ppc32 koji and builds and 
composes away, it's just that we simply don't have the time to really do 
this anymore as a more official secondary arch for Fedora anymore.

> Configuring the PPC32 koji to just *accept* failures of the 32-bit
> builds, while still allowing PPC64 builds to complete successfully when
> that happens, would be nice. But that requires hacking on koji, doesn't
> it?
>

We thought about it, but the problem comes with the ppc and ppc64 bit 
architectures being multiarch. They are defined like that in rpm, yum 
and all the tooling we have for dealing with those architectures. If 
we'd only hack koji to accept builds if the 32bit fails but the 64bit 
succeeds that would break general multilib closure and David Aquilina 
said the side effects of different versions then could lead to another 
really ugly mess in the long run.

Hope that clarifies it a bit more in depth, and apologies for leaving 
out some of the alternatives that are of course absolutely viable (e.g. 
for someone to set up his own ppc32 bit koji and continuing to build 
there then).

Thanks & regards, Phil

-- 
Philipp Knirsch              | Tel.:  +49-711-96437-470
Manager Core Services        | Fax.:  +49-711-96437-111
Red Hat GmbH                 | Email: Phil Knirsch <pknirsch at redhat.com>
Wankelstrasse 5              | Web:   http://www.redhat.com/
D-70563 Stuttgart, Germany