Phase out 32bit ppc due to bugfix and maintenance burden

[Al Dunsmuir]

On Monday, May 12, 2014, 7:57:07 AM, Dan Horák wrote:
> On Fri, 9 May 2014 12:17:58 -0400
> Al Dunsmuir <al.dunsmuir at sympatico.ca> wrote:

>> On Friday, May 9, 2014, 7:18:13 AM, Phil Knirsch wrote:
>> > On 05/09/2014 12:15 PM, David Woodhouse wrote:
>> >> On Fri, 2014-05-09 at 11:32 +0200, Phil Knirsch wrote:
>> >>>
>> >>> So for 1) this would effectively mean we'd have to look at every
>> >>> failed build for 32bit ppc and put ExcludeArch: ppc in it. On top
>> >>> of that we'd then need to additionally look for all components
>> >>> that require or buildrequire recursively these packages and do
>> >>> the same for all those packages, too. Just for the java stack
>> >>> thats several hundreds of packages alone. Overall thats an
>> >>> enormous amount of effort and would have to be constantly done
>> >>> for any new 32bit ppc package failing, so we dropped this
>> >>> solution.
>> 
>> I  would suggest that supporting all desktops and the full package set
>> would indeed be an unrealistic goal. Not just for ppc-32, but also for
>> ppc-64.
>> 
>> One  of the limiting factors is that the graphics hardware simply does
>> not  have the capabilities in most cases. Perhaps Wayland can simplify
>> that.
>> 
>> From  my  point  of view, for ppc-32 I would prioritize those packages
>> required  to  operate  the system, and do development in a set of core
>> languages.  I  would tend to prioritize the basic languages as C, C++,
>> Python, and Perl.
>> 
>> I  would  tend  to  downgrade support for Java, Ruby and the others if
>> that was the cost of life or death of ppc32.

> That's not how the secondary arches are supposed to work. The build
> process is automated and the secondary arches follow all builds from
> primary with a short delay. And builds in secondaries are using the
> same package versions in the buildroots as were used on primary. Yes,
> there are exceptions, but the exceptions should be about using newer
> versions, when the exact one fails to build. And also with a different
> package set our product couldn't be called Fedora. So called remixes
> allow using different package sets.

Dan,

Indeed, "downgrade support" is the wrong terminology.

What  I  meant  was  if  we  want  a  working  ppc32, we have to first
concentrate  on the core components. We need the kernel, boot support,
lorax,  x11,  rpm,  core  languages,  and components required to build
those  components  to  be  fully  functional  before we can move on to
diagnosing and solving issues with in other components.

Phil  suggested  automatically  using  ExcludeArch to remove a failing
component  for  ppc32,  to prevent a working build for that package in
ppc64  from  also  being  treated as a failure. He needs to do this so
that the ppc64 team can deliver his package set for F21.

While that solves ppc64's immediate problem for that package, it means
that  rpm  dependencies  can  and  will  cause  immediate  ppc32 build
failures. We have to prioritize those failures for the core components
(so  the  ppc32  system  can  still  boot  and build new changes) over
failures for other components that are less critical.

As  usual  with  build failures, some issues will be readily solved by
the  ppc32  team,  while others will require assistance by the package
owner, or by other knowledgeable parties.

Phil wanted proven packagers involved to reduce the overhead in fixing
trivial  problems  across many packages (since they have acl access to
all packages, not just their own).

It will take a while (likely F22) before ppc32 is stable enough to let
the situation return to normal so that ppc32 builds are relevant.

>> >> Hm, does this really have to be that much work? If feasible, it
>> >> would have been my preferred option. Although since I do almost
>> >> nothing on PPC these days I appreciate how little my opinion
>> >> counts :)
>> >>
>> >> Let's assume that you can write a script which, when run as a
>> >> provenpackager, will check out a package from the git tree, add the
>> >> ExcludeArch tag to it and commit it, and file a bug in bugzilla
>> >> marked as blocking the 'PPC32 ExcludeArch tracker'. So the actual
>> >> work involved in adding the ExcludeArch is basically limited to
>> >> giving a *reason*.
>> >>
>> 
>> > Sure, but remember the chained dependencies we have in the build 
>> > requires in Fedora. One of the recent efforts in the Base WG that
>> > me and Harald Hoyer were actually driving was investigate the inter
>> > build dependencies for a self hosting Fedora that includes gcc,
>> > rpm, yum and anaconda. We ended up needing over 1800 packages to be
>> > self hosting, but if we'd drop the self hosting requirement that
>> > dropped to less than 700. Which means the build requires pulls in
>> > more than 2x the amount of packages. And if you e.g. have to
>> > ExcludeArch eclipse (just as a typical example of a component that
>> > has proven to be quite difficult) you're then either left with
>> > unraveling the whole build requires chain of eclipse, java and all
>> > it's friends or excludearch the whole java world, which then in
>> > chain requires you to disable db4 and db, which is required by rpm.
>> > So you can't really disable the java stack but have to manually
>> > clean up the buildrequires and as you can imagine, doing that is a
>> > lot of work.
>> 
>> >> It's also not that hard to use repoquery or similar tools to
>> >> automatically run that script up the dependency chain, surely?
>> 
>> > Jup, we've done exactly that for the Base WG effort. Harald
>> > produced a nice graph of the inter dependencies here:
>> 
>> > http://www.harald-hoyer.de/2014/01/14/self-hosting-fedora-base/
>> 
>> > So we know it's a mess, and running repoquery regularly only will 
>> > confirm that, but without someone dedicated and having the spare
>> > cycles to actually do the work to unravel those, that in itself
>> > won't help a lot unfortunatly. :(
>> 
>> >> If we're genuinely talking about an "enormous amount of effort"
>> >> then we really can script it and make it easy. And "constantly
>> >> done for any new 32bit ppc package failing" also surprised me. As
>> >> I said, I've been paying little attention to PPC for a while, but
>> >> we used to have fairly much 100% coverage... are things really
>> >> falling apart that much, and at such a rate?
>> 
>> > It's rather often core packages where we see build failures, be it
>> > java or toolchain related, or somewhere in the high level desktop
>> > space where things are quite dependent on another, too.
>> 
>> > And it's often enough to having us spend a considerable amount of
>> > time having to investigate the failures, work with maintainers on
>> > potential fixes etc. And although most maintainers are really very
>> > helpful and forthcoming with finding fixes even for secondary
>> > architectures, occasionally we get a "don't have time" from someone
>> > (which is of course totally understandable and not meant as an
>> > accusation), which then means we'll have to find the fix ourselves
>> > or again, unravel dep build chains and would have to start
>> > ExludeArchs:
>> 
>> >> If we really can't manage this, then I suppose I have no
>> >> fundamental objection to just configuring the PPC koji to build
>> >> only 64-bit from now on, and let someone set up a PPC32 koji
>> >> instance if they want to. I think that's what you're proposing,
>> >> right? All the RPM multilib bits ought to still work, if the
>> >> packages come into existence again.
>> 
>> > Exactly, maybe i should have been clearer about that. We're totally
>> > fine and would be happy if someone sets up a ppc32 koji and builds
>> > and composes away, it's just that we simply don't have the time to
>> > really do this anymore as a more official secondary arch for Fedora
>> > anymore.
>> 
>> >> Configuring the PPC32 koji to just *accept* failures of the 32-bit
>> >> builds, while still allowing PPC64 builds to complete successfully
>> >> when that happens, would be nice. But that requires hacking on
>> >> koji, doesn't it?
>> 
>> > We thought about it, but the problem comes with the ppc and ppc64
>> > bit architectures being multiarch. They are defined like that in
>> > rpm, yum and all the tooling we have for dealing with those
>> > architectures. If we'd only hack koji to accept builds if the 32bit
>> > fails but the 64bit succeeds that would break general multilib
>> > closure and David Aquilina said the side effects of different
>> > versions then could lead to another really ugly mess in the long
>> > run.
>> 
>> The  more  you  separate  ppc  and  ppc64,  the  harder it would be to
>> maintain  multiarch  support. There are a certain set of core packages
>> that   require   actual  ppc  32-bit  hardware  for  development,  but
>> interested  parties could provide assistance on most of the user space
>> packages with a viable mulilib.
>> 
>> Reviving  support for 32-bit ppc installs (using a more modern package
>> set,  including grub2) is a core requirement. Likely the same on older
>> Mac  G5s.  If  multilib breaks, then we would have to also support ppc
>> installs  on  newer  ppc64  (Prep  and other) hardware and that may be
>> problematic in the general case.
>> 
>> > Hope that clarifies it a bit more in depth, and apologies for
>> > leaving out some of the alternatives that are of course absolutely
>> > viable (e.g. for someone to set up his own ppc32 bit koji and
>> > continuing to build there then).
>> 
>> If  that someone is not within the Fedora engineering group, you would
>> introduce  a whole new set of issues with coordination and security. I
>> don't think they would approve key signing with the Fedora keys unless
>> the final builds were on Fedora hosting.
>> 
>> > Thanks & regards, Phil
>> 
>> 
>> _______________________________________________
>> ppc mailing list
>> ppc at lists.fedoraproject.org
>> https://admin.fedoraproject.org/mailman/listinfo/ppc
> _______________________________________________
> ppc mailing list
> ppc at lists.fedoraproject.org
> https://admin.fedoraproject.org/mailman/listinfo/ppc