changes in the infrastructure

[Dan Horák]

Hi all,

there are some changes that needs to be done on the Fedora/s390x
infrastructure for various reasons and we should prepare a plan how and
when to do them.

Tasks
-----
1. upgrade builders to something with kernel 2.6.32+ like EL-6 beta
Yesterday a new glibc was built in primary Fedora, it drops some
compatibility stuff for old kernels and now requires kernel 2.6.32+ on
the builders.

2. redesign resource allocation for the builders
The builders (and the squid cache) don't behave very well when under
full load. Timeouts when downloading the buildroots thru the cache are
quite rare (but they happen every day). Other observed behaviour looks
like a completely swapped out guest, it doesn't respond to ping, koji
doesn't update its status on the hub and it can take minutes, maybe even
tens of minutes, before the machine starts responding again.
Unfortunately often this behaviour means stuck builds and manually
restarting the builder daemon.
I think part of this issue could be solved on the z/VM side (size of
RAM, number of CPUs per builder), part could be tuning the koji
configuration (max jobs per builder, max load, parallel make, ...). Also
interesting would be to see performance/resource usage statistics from
z/VM.

3. rebuild the storage on the hub
The sub-optimal storage configuration is known for some time, but there
is still room on the disks for few months of work (my guess).

4. upgrade Koji to 1.4
Here we should be on par with the primary Fedora buildsystem.

Timing
------
1. ASAP, let's say till end of next week, because it blocks building
packages that were built after July 21 in primary koji
2. together with 1.
3. till end of August, earlier date means fewer data to sync
4. together with 3

Outages
-------
1. none, can be done one builder at a time
2. yes
3. yes
4. yes

Comments, ideas, corrections, etc. are welcome


Dan