Bootstrapping new perl releases/architectures

[Paul Howarth]

Every twelve months or so there is a new major release of perl and we
do a mass rebuild of all perl packages. Last time, for the perl 5.14.x
release, an attempt was made to do a clean bootstrap of the Fedora perl
ecosystem, starting with the main perl package and then incrementally
building other modules on top of that, avoiding pulling in packages
built with the old perl 5.12.x release. However, this was a struggle
because of widespread circular build dependencies between packages. In
order to work around this, a %perl_bootstrap macro was defined, which
packages could check to alter their build and/or runtime dependencies so
as to break the cycles and allow a clean bootstrap. Unfortunately
though it all took too long to locate and fix the cycles and we ended
up pulling in old 5.12.x builds to get the last few hundred packages
built.

As it will soon be time to do a mass rebuild for perl 5.16.x, and
similar bootstrapping issues affect the secondary architectures when
trying to get their first Fedora build up, I decided to try to work
through all of these issues ahead of time, find and fix the dependency
cycles and work out an optimum build order for the whole Fedora perl
ecosystem. Here is what I came up with.

First of all, a high-level overview of my approach.

I split the Fedora perl ecosystem into two groups of packages, which
I've called "providers" and "consumers". The "provider" packages are
those that contain perl modules in the %perl_vendorlib/arch and
%perl_perllib/installarch directories, and those are the ones that I
have concentrated on as they all need to be rebuilt. The "consumer"
packages are those that have dependencies on the "provider" packages
but aren't actually "providers" themselves. I don't think they really
need rebuilding, though doing so should be trivially achieved in a
single pass once the "providers" have been done. The "consumers" do
need to be taken into account during the bootstrapping process through,
because they (e.g. automake/autoconf) may be needed to build some of
the "provider" packages.

I've taken as the start point of the process the point at which the
main perl package has been built and all of the packages in the mock
buildroot are installable. Thereafter, I've split the bootstrapping
process into a series of passes, where I first determine which of the
"consumer" packages have become available (a large proportion only
require Perl core modules and can installed as soon as the main perl
package has been built at the start of the process), and then which of
the "provider" packages can be built, which means that all of their
build requirements (with %perl_bootstrap assumed to be defined to 1)
are available, plus all of the dependencies of those
build-requirements. I have assumed that all of the dual-lived modules
are available from the start of the process, so there's no need to
wait, for instance, for perl-Module-Build to be built before
perl(Module::Build) is available for building against. All of the
packages built in any given pass should be buildable in parallel, and
will have at least one build dependency that was built in the previous
pass.

The result of this process is available at
http://www.city-fan.org/~paul/perl-boot/buildorder

The main bootstrap process takes 38 passes, with perl-Task-Catalyst the
last "provider" package built at the top of the dependency tree.
Following the bootstrap build, I feel there should be three further
sets of rebuilds with %perl_bootstrap undefined:

1. Packages that have different runtime dependencies when built with
%perl_bootstrap defined. There are mercifully few of these and they
need to be done first to get to a state where all of the packages in
the repo are available with the same dependency tree as if they'd all
been built without special bootstrapping treatment.

2. Packages that have different build requirements when built with
%perl_bootstrap defined. These will typically have run fewer tests in
%check than they would have done for a regular build, so rebuilding
them checks for FTBFS issues around the extra tests. It's also possible
that the change in the runtime dependency tree resulting from the
rebuilds in the previous pass might result in extra packages been
pulled into the build too, which could affect the build.

3. Packages that were not rebuilt in the second post-bootstrap pass but
whose buildroots would have had different packages as a result of the
changes in the runtime dependency tree from the first
post-bootstrap rebuild pass. For example, perl-Bio-SamTools would have
been built without perl-bioperl-run in its buildroot originally but a
rebuild would pull it in via the perl-bioperl<->perl-bioperl-run
circular dependency (I wouldn't be surprised actually if the bootstrap
build of perl-Bio-SamTools failed because of this but I haven't check
that yet). Again, this is a check for FTBFS issues.

If it's felt that the "consumer" packages should be rebuilt too, that
could be done at this stage.

I thought about describing my approach to working all of this out too
but I think this email is long enough already. I can discuss that in
follow-up messages if anyone is interested or would like to
try/modify/improve it themselves.

Paul.