[fedora-arm] Distributed CC

[Lennert Buytenhek]

Hi Rabeeh,

On Mon, Oct 01, 2007 at 01:47:14PM +0200, Rabeeh Khoury wrote:

> Lately I'v tried building boost package in two different methods while
> measuring the time it consumes.
> 
> The first using Orion-2 system and running completely native build using
> the FC-8 RPMs.
> The second is using Orion-2 that has distcc client that calls through
> fast Ethernet to a Core 2 duo x86 machine that has the FC-8
> cross-compiler environment with distcc daemon.

I have a mostly drop-in setup for this -- cross compiler RPMS at:
	http://www.wantstofly.org/~buytenh/cross/

and distcc (client and server) RPMS at:
	http://fedora-arm.wantstofly.org/add-ons/

The only configuration it needs is:

Server (x86) side:
- Edit /etc/sysconfig/distccd:
  - Set arm box IP range, and add --jobs 200 or so.
  - Set DISTCCD_PATH to /usr/armv5tel-redhat-linux-gnueabi/bin:$PATH

Client (arm box) side:
- Put IP of build machine into /etc/distcc/hosts.

You might also want to:
- Hardcode %{_smp_mflags} on the ARM side to '-j 4' or so, to use
  multiple gcc instances in parallel for the build (to overcome
  network latency.)

I've used these RPMS successfully to (pseudo-natively) compile glibc,
eglibc, gcc, etc.

I guess we should add this info to the wiki...


> The total build time using the distcc for boost was 5X faster (complete
> build with archiving).
> 
> The question is can we use this as an addition to Koji build system to
> make building from scratch faster?

Probably.  All it would need is:

- Create an ARM package that contains a suitable version of
  /etc/distcc/hosts (called, say, distcc-arm-stubs.)
- Create a new version of buildsys-build that pulls this package in.

This way, the distcc hooks get pulled into the build root
automatically.

Note that the distcc client package has /etc/profile.d/ scripts
that automatically add the distcc hook directory to the path if
/etc/distcc/hosts is present.


> I know about the efforts to make everything cross-build, but until that
> becomes ready the distributed build can be very attractive.

There are a couple more differences:
- Cross-building can (at least in theory! :) be used to bootstrap
  a distro, whereas the distcc method can not.
- Cross-building does not (at least in theory! :) need actual ARM
  hardware.

I realise that you can do the distcc trick with a qemu-system-arm
instance, so you don't really need actual physical ARM _hardware_,
but it's still somewhat cumbersome to set that up.


> Note that for the build I didn't need to change anything in RPM specs,
> or RPM macros and configuration; only installed distcc and changed the
> path.

ACK.

As a pure build speedup method, the distcc trick is fine (we have
been using this trick for the Debian big-endian ARM port for a couple
of years now, since the build hardware we have is pretty slow), but
there are couple of advantages of pure cross building that you can't
really duplicate with this method.

For building Fedora ARM packages, we use 14 build machines (2 real
ARM boxes and 12 qemu-system-arm instances), and the speedup from
distcc isn't really critical, so we don't use it.


cheers,
Lennert