builders of the future!!!!!

[seth vidal]

On Tue, 20 Mar 2012 21:38:13 -0500
Dennis Gilmore <dennis at ausil.us> wrote:

 
> Today there is not a way to do an anaconda install on any arm system.
> though hopefully we will have that for deployment.

I would hope so. :)

> probably we would be adding 100-300 systems. not only do we need to
> consider overloading of puppet, but also logging and monitoring. I
> guess its more how do we scale our infrastructure from at a guess ~100
> nodes today to 3 to 4 times that 

Centrally logging the builders is probably unnecessary. Especially if
we're bouncing them all the time.


> honestly we could do this instead of the monthly updates. just rebuild
> them instead

Sure - but I'm thinking of the emergency "oh look at that nightmare"
updates. 


> im ok with that, im pretty sure fas will scale to the extra boxes. do
> we drop monitoring of the builders? what about collectd etc. 

Collectd - off. We're not gaining much by having that punish the syslog
server.
We can monitor the builders w/o needing all of the copious info that
collectd provides.

fas I'm not very worried about - though I suspect a couple of things
will change w/how we get the dbs onto the hosts.
 
 
> main issue is that today we are not 100% sure of how we will install
> arm boxes. how do we deal with all the non puppet related systems?

I think, if the playbooks are working well, we can use ansible to do
this.

> also need to look into how we can better scale koji itself. when we
> go from 20 to 200+ builders we need to make sure that load doesn't
> cause koji to fall over.

okay - but I think that's more something for the kojidevs than fedora
infra?

 
 
> all the arm boxes will have management consoles. but today im not 100%
> sure how access to that would be. we would also need to deploy fedora
> for any arm based systems. things we need to reconsider also is
> networking today the storage network and the builder networks
> are /24's so we could use 253 nodes. i suspect we will go over that
> on the build network. we could not have the storage network on arm
> builders. it is really only needed for createrepo. but we may need to
> look at expanding kojipkgs to more nodes. or increase its network
> throughput with multiple bonded gig network ports. think mass rebuild
> and 100 or 200 buildroots initialising at once.  it will stress our
> resources on all levels. but the flexibility of so many nodes could
> allow us to deploy solid solutions to scale and show that fedora is
> still the leader in open infrastructure and sets industry best
> practices.

So one thing I'm not sure I understand - why would we need so many arm
builders? Is it b/c there are so many more arm archs so there will need
to be more pkgs built?


-sv