Patching aarch64 support into Fedora

[Brendan Conoboy]

Hi everybody,

Fedora 19 has many of the enablers for native aarch64 in core packages 
such as glibc and gcc.  Many more packages would compile if config.guess 
and config.sub recognized aarch64 as a valid architecture, but only the 
latest version of autoconf knows about aarch64.  At last week's 
fedora-arm meeting we talked about the viability of automatically 
patching such packages.  The outcome of that discussion was that we 
should first identify how many packages need such a patch, then decide 
what to do based on that number.

Red Hat's Al Stone has written a script which automatically generates 
patches for each package that needs it (And updates its spec file). 
After a complete run we can say that ~1850 packages need such a patch. 
The number may be larger, but it is definitely not smaller, as his only 
considers autoconf-using packages.  If another auto configuration system 
is in use by a number of packages it too many need updating (cmake?). 
Now that we have this number, what do we want to do?

I see a few options:

1. Do nothing, trip over this issue at least 1850 times during bootstrap.

2. Mail all package owners asking for action.

3. Proven packager commits the patches, package owners take them out 
once unnecessary.

4. Run autoconf during build, incurring wrath of any packager whose 
package isn't compatible with the latest autoconf.

5. Your much more sensible idea goes here.

What say you?

[Al Stone]

On 03/05/2013 11:44 AM, Brendan Conoboy wrote:
> [snip..]
>
> Red Hat's Al Stone has written a script which automatically generates
> patches for each package that needs it (And updates its spec file).
> After a complete run we can say that ~1850 packages need such a patch.
> The number may be larger, but it is definitely not smaller, as his only
> considers autoconf-using packages.  If another auto configuration system
> is in use by a number of packages it too many need updating (cmake?).

For the curious, the script used is in 
git://fedorapeople.org/~ahs3/patchify.git

[Peter Robinson]

On Tue, Mar 5, 2013 at 3:44 AM, Brendan Conoboy  wrote:
> Hi everybody,
>
> Fedora 19 has many of the enablers for native aarch64 in core packages such
> as glibc and gcc.  Many more packages would compile if config.guess and
> config.sub recognized aarch64 as a valid architecture, but only the latest
> version of autoconf knows about aarch64.  At last week's fedora-arm meeting
> we talked about the viability of automatically patching such packages.  The
> outcome of that discussion was that we should first identify how many
> packages need such a patch, then decide what to do based on that number.
>
> Red Hat's Al Stone has written a script which automatically generates
> patches for each package that needs it (And updates its spec file). After a
> complete run we can say that ~1850 packages need such a patch. The number
> may be larger, but it is definitely not smaller, as his only considers
> autoconf-using packages.  If another auto configuration system is in use by
> a number of packages it too many need updating (cmake?). Now that we have
> this number, what do we want to do?
>
> I see a few options:
>
> Do nothing, trip over this issue at least 1850 times during bootstrap.
>
> Mail all package owners asking for action.
>
> Proven packager commits the patches, package owners take them out once
>
> unnecessary.
> Run autoconf during build, incurring wrath of any packager whose package
>
> isn't compatible with the latest autoconf.
> Your much more sensible idea goes here.
>
> What say you?

Firstly where is the list of 1850 packages that are affected?

Also we do need to know whether the likes of cmake do need work or
what else might need work. There's 13K packages in Fedora, of that how
many are autoconf driven (so what sort of percentage is the 1850, it
might give us an idea of what has been updated).

I don't see 1 and 4 as an option. Option 2 should read "File a bug
against every affected package with it linked to a tracking bug asking
them for action"

I think a "5) file a rel-eng bug with the patches and script and ask
them to do it" which was I believe the plan prior to mass rebuild but
the script/list wasn't ready in time/

Ultimately we need to get this done and any analysis of other non
autoconf build platforms that might need the work too prior to the
branching which is scheduled to happen on March 12. That gives us a
week to get this sorted.

Peter

[Matthew Miller]

On Mon, Mar 04, 2013 at 07:44:16PM -0800, Brendan Conoboy wrote:
> Do nothing, trip over this issue at least 1850 times during bootstrap.
> Mail all package owners asking for action.

How about posting the list and the suggested action to correct it to
fedora-devel, as an initial pass?

[Dennis Gilmore]

On Mon, 04 Mar 2013 19:44:16 -0800
Brendan Conoboy  wrote:
> Hi everybody,
>
> Fedora 19 has many of the enablers for native aarch64 in core
> packages such as glibc and gcc.  Many more packages would compile if
> config.guess and config.sub recognized aarch64 as a valid
> architecture, but only the latest version of autoconf knows about
> aarch64.  At last week's fedora-arm meeting we talked about the
> viability of automatically patching such packages.  The outcome of
> that discussion was that we should first identify how many packages
> need such a patch, then decide what to do based on that number.
>
> Red Hat's Al Stone has written a script which automatically generates 
> patches for each package that needs it (And updates its spec file). 
> After a complete run we can say that ~1850 packages need such a
> patch. The number may be larger, but it is definitely not smaller, as
> his only considers autoconf-using packages.  If another auto
> configuration system is in use by a number of packages it too many
> need updating (cmake?). Now that we have this number, what do we want
> to do?
>
> I see a few options:
> Do nothing, trip over this issue at least 1850 times during
>
> bootstrap.
>
> Mail all package owners asking for action.
>
> Proven packager commits the patches, package owners take them out
>
> once unnecessary.
> Run autoconf during build, incurring wrath of any packager whose
>
> package isn't compatible with the latest autoconf.
> Your much more sensible idea goes here.
>
> What say you?

I say we need the list of packages effected. then we can evaluate how
critical it is that we take action.

Dennis

[Brendan Conoboy]

On 03/06/2013 06:37 AM, Dennis Gilmore wrote:
> I say we need the list of packages effected. then we can evaluate how
> critical it is that we take action.

Here is the initial list:

http://people.fedoraproject.org/~blc/fedora-arm/patchify/configure.pkgs

I think it may actually be much larger.  Al and I are going back and 
forth on fixing up patchify to identify more cases.

[Peter Robinson]

On Wed, Mar 6, 2013 at 6:11 PM, Brendan Conoboy  wrote:
> On 03/06/2013 06:37 AM, Dennis Gilmore wrote:
> > I say we need the list of packages effected. then we can evaluate how
> > critical it is that we take action.
>
> Here is the initial list:
>
> http://people.fedoraproject.org/~blc/fedora-arm/patchify/configure.pkgs
>
> I think it may actually be much larger.  Al and I are going back and forth
> on fixing up patchify to identify more cases.

Ultimately there's a number of things like alsa* globus* as well as a
lot of X related stuff where upstream should be contacted because they
do have regular releases so there's no reason why they shouldn't be
fixed by default from upstream.

Peter

[Mark Salter]

On Wed, 2013-03-06 at 10:11 -0800, Brendan Conoboy wrote:
> On 03/06/2013 06:37 AM, Dennis Gilmore wrote:
> > I say we need the list of packages effected. then we can evaluate how
> > critical it is that we take action.
>
> Here is the initial list:
>
> http://people.fedoraproject.org/~blc/fedora-arm/patchify/configure.pkgs
>
> I think it may actually be much larger.  Al and I are going back and 
> forth on fixing up patchify to identify more cases.

I notice that gcc is missing from the list. I had to patch the config
files in the isl subdir for the aarch64 build.

https://bugzilla.redhat.com/show_bug.cgi?id=917749

--Mark

[Dennis Gilmore]

On Mar 6, 2013, at 12:11 PM, Brendan Conoboy  wrote:
> On 03/06/2013 06:37 AM, Dennis Gilmore wrote:
> > I say we need the list of packages effected. then we can evaluate how
> > critical it is that we take action.
>
> Here is the initial list:
>
> http://people.fedoraproject.org/~blc/fedora-arm/patchify/configure.pkgs
>
> I think it may actually be much larger.  Al and I are going back and forth on fixing up patchify to identify more cases.

Talking with the X guys they run autoreconf in %build so all of their packages are false positives.

Dennis

[Brendan Conoboy]

On 03/06/2013 07:24 PM, Dennis Gilmore wrote:
> Talking with the X guys they run autoreconf in %build so all of their packages are false positives.

Good catch.  Al is updating patchify to also scan %build for such 
occurrences.  We'll have an updated list tomorrow.

[Al Stone]

On 03/06/2013 10:39 PM, Brendan Conoboy wrote:
> On 03/06/2013 07:24 PM, Dennis Gilmore wrote:
> > Talking with the X guys they run autoreconf in %build so all of their
> > packages are false positives.
>
> Good catch.  Al is updating patchify to also scan %build for such
> occurrences.  We'll have an updated list tomorrow.

Sorry for the delay....updated results can be found here:

http://fedorapeople.org/~ahs3/patchify/run3/patched.pkgs

That list is all the packages that use config.{guess,sub}
and have not already been patched upstream for aarch64 -- a
total of 1976 packages that I've been able to find.  Some
434 have already been patched.

Note that there may still be false positives; I did not
eliminate the packages that use some form of autoconf because
it's not clear one can assume that autoconf will actually be
invoked during the %build -- or at least I can't unless I'm
willing to do some extensive parsing of the spec file that I
have not done so far (it would probably easier to just build
the package).

The packages that use ./configure in some form in the spec
file (2623 of them) are listed here:

http://fedorapeople.org/~ahs3/patchify/run3/configure.pkgs

This lists 2623 packages.  In actuality, by searching for
the existence of config.{guess,sub} there are only 2459.
The remainder are the results of the regex not being able
to separate out, for example, './configure' in a comment
vs an actual use.

So, 2459 total packages using ./configure; patches were
created for 1976 of them, 434 were already patched, and
5 ran into errors while creating a patch.  I'm missing
44 somewhere still that I'll see if I can find.

I've also stashed copies of the log from running the script
(run3.log) and several other lists of the various packages
found (e.g., dead packages, packages not using config*, and
so on) in that same directory on fp.o.  Holler if there's
questions...