P2P Packaging/Koji Cloud

[Denis Arnaud]

Hello,

RedHat-hosted Koji servers offer an invaluable service by allowing all of
us, package maintainers, to build all of "our" Fedora packages. I guess
that that infrastructure is not cost-less for RedHat and and the quality of
service is great (for instance, the wait in the queues, before Koji
actually builds the packages submitted via the command-line client, is not
so long).

As Fedora is pretty advanced in the cloud/virtualisation arena, we could
imagine a "Koji Cloud", hosted on VMs offered by volunteers. For instance,
I could contribute a few VMs in Europe (hosted on http://www.ovh.co.uk/).
Our Cloud SIG (https://fedoraproject.org/wiki/Cloud_SIG) and/or Virt ML (
https://admin.fedoraproject.org/mailman/listinfo/virt and
https://fedoraproject.org/wiki/Getting_started_with_virtualization)/RedHat
ET (http://et.redhat.com/) colleagues could help designing and implementing
the following infrastructure:
 * VM template/images, ready to be started on the volunteer's servers
everywhere in the world, 24x7.
    - SSH public keys of Koji administrators would be part of the images,
so that they can have an easy access to them, just in case.
    - Those VMs would update themselves automatically.
    - The containers could be standardised as well. For instance,
ProxMox/OpenVZ or Fedora/CentOS with libvirt.
 * A directory (LDAP, or something less centralised, like the address book
of Skype, for instance), keeping track of all those VMs:
    - with the corresponding last known status;
    - with the VM configurations (Fedora/CentOS release, CPU, memory, disk
usage, etc);
    - with some rating corresponding to their quality of service (build
duration, reliability of the VM, MTBF, etc).
 * A dispatcher system:
    - which would route the Koji build requests to available VMs;
    - collect the outcome of the builds (logs, RPM packages, statistics,
QoS, etc) and store them in the current ("centralised") Koji infrastructure.

As I am not a specialist of all those technologies, I may have forgotten a
lot of things, but you get the idea.
Doesn't it sound great? Does it sound realisable? Am I crazy to dream to
such an infrastructure?

Cheers

Denis

[Josh Boyer]

On Wed, Dec 7, 2011 at 8:46 AM, Denis Arnaud
<denis.arnaud_fedora(a)m4x.org&gt; wrote:
>  Hello,
>
>  RedHat-hosted Koji servers offer an invaluable service by allowing all of
>  us, package maintainers, to build all of "our" Fedora packages. I guess that
>  that infrastructure is not cost-less for RedHat and and the quality of
>  service is great (for instance, the wait in the queues, before Koji actually
>  builds the packages submitted via the command-line client, is not so long).
>
>  As Fedora is pretty advanced in the cloud/virtualisation arena, we could
>  imagine a "Koji Cloud", hosted on VMs offered by volunteers. For instance, I
>  could contribute a few VMs in Europe (hosted on http://www.ovh.co.uk/). Our
>  Cloud SIG (https://fedoraproject.org/wiki/Cloud_SIG) and/or Virt ML
>
> (https://admin.fedoraproject.org/mailman/listinfo/virt and https://fedorap...
>  ET (http://et.redhat.com/) colleagues could help designing and implementing
>  the following infrastructure: 
I believe Seth Vidal looked at auto-starting koji builder instances in
the cloud a while ago.  Hopefully he'll chime in.

The one thing you don't address is security.  If we're going to push
stuff to a public cloud, how do you ensure that the builder VMs aren't
compromised?

josh

[Seth Vidal]

On Wed, 7 Dec 2011 14:46:18 +0100
Denis Arnaud <denis.arnaud_fedora(a)m4x.org&gt; wrote:

>  Hello,
>  
>  RedHat-hosted Koji servers offer an invaluable service by allowing
>  all of us, package maintainers, to build all of "our" Fedora
>  packages. I guess that that infrastructure is not cost-less for
>  RedHat and and the quality of service is great (for instance, the
>  wait in the queues, before Koji actually builds the packages
>  submitted via the command-line client, is not so long).
>  
>  As Fedora is pretty advanced in the cloud/virtualisation arena, we
>  could imagine a "Koji Cloud", hosted on VMs offered by volunteers.
>  For instance, I could contribute a few VMs in Europe (hosted on
>  http://www.ovh.co.uk/). Our Cloud SIG
>  (https://fedoraproject.org/wiki/Cloud_SIG) and/or Virt ML
>  ( https://admin.fedoraproject.org/mailman/listinfo/virt and
>  https://fedoraproject.org/wiki/Getting_started_with_virtualization)/RedHat
>  ET (http://et.redhat.com/) colleagues could help designing and
>  implementing the following infrastructure:
>   * VM template/images, ready to be started on the volunteer's servers
>  everywhere in the world, 24x7.
>      - SSH public keys of Koji administrators would be part of the
>  images, so that they can have an easy access to them, just in case.
>      - Those VMs would update themselves automatically.
>      - The containers could be standardised as well. For instance,
>  ProxMox/OpenVZ or Fedora/CentOS with libvirt.
>   * A directory (LDAP, or something less centralised, like the address
>  book of Skype, for instance), keeping track of all those VMs:
>      - with the corresponding last known status;
>      - with the VM configurations (Fedora/CentOS release, CPU, memory,
>  disk usage, etc);
>      - with some rating corresponding to their quality of service
>  (build duration, reliability of the VM, MTBF, etc).
>   * A dispatcher system:
>      - which would route the Koji build requests to available VMs;
>      - collect the outcome of the builds (logs, RPM packages,
>  statistics, QoS, etc) and store them in the current ("centralised")
>  Koji infrastructure.
>  
>  As I am not a specialist of all those technologies, I may have
>  forgotten a lot of things, but you get the idea.
>  Doesn't it sound great? Does it sound realisable? Am I crazy to dream
>  to such an infrastructure?
>   

I've looked into spawning virt instances to do building and it is
pretty doable. The problem with them being offered by volunteers is
trust:

1. how do we trust the initial installation hasn't been poisoned unless
we ship all the bits over ourselves.
2. how do we trust the in-flight build isn't molested
3. how do the people providing the trust insure against
tainted/dangerous builds doing $bad_things on their systems.

this is why I concluded that the idea of donated/volunteered VM was not
going to work - additionally b/c the bandwidth requirements are
non-trivial for many builds.

However, building on environments where we have a contractual (read:
financial) relationship will work better and where the remote end has
protected themselves against attacks from the VMs. I'm speaking of
cloud hosting providers like amazon ec2 and rackspace cloud servers.

I've worked on some code to spawn off an instance, submit jobs +
packages, build them (a chain-build so you don't have to keep
respawning them) then collect all the results back to your local
machine. It works - it requires setting up trusted images at those
cloud providers but that's not very hard to do and keep current. Right
now I'm porting the code to use a different cloud-communication API
than I was using before. 

The problems still persist with bandwidth consumption and to some
extent with trust but trust is mitigated b/c the relationship with
the provider is more standardized and less haphazard.

I have a couple of systems inside the red hat colo that I had planned
on reinstalling to f16 and setting up openstack on them to play with the
same idea but on a local cloud instance. 

Is all this inline with the problems you've thought about?

-sv

[Denis Arnaud]

2011/12/7 seth vidal <skvidal(a)fedoraproject.org
>  
>  I've looked into spawning virt instances to do building and it is
> pretty
>  doable. The problem with them being offered by volunteers is trust
>  [...]
>  
You are right. I had not thought at that... how naive of me :(

The volunteers/trustees would sign the builds with their own private keys,
for instance with their FAS keys. Then, we could have some
"trustworthiness" ratings for all the submitters, like we have today for
the packagers (new packager, proven-packager, sponsor). While the submitter
is still not trusted, the centralised Koji infrastructure can duplicate the
build, and check that it gives the same results. And even when the
submitter is trusted, some random duplicate builds can occur. If the
submitter taints the builds, it will be flagged as a potential "fraud". A
human being would have to have a look at it then.

Or, the VMs could do "scratch" builds (only). When those builds are
successful, the VMs then just act as a standard clients to the central Koji
servers, and the packages are re-built in that safe
infrastructure. Overall, the central Koji infrastructure would be
off-loaded from all the scratch builds, as well as from the failed builds.
Which is already not so bad, is it?


I've worked on some code to spawn off an instance, submit jobs + packages,
>  build them (a chain-build so you don't have to keep respawning
> them) then
>  collect all the results back to your local machine. It works - it requires
>  setting up trusted images at those cloud providers but that's not very hard
>  to do and keep current. Right now I'm porting the code to use a different
>  cloud-communication API than I was using before.
>  
That would be very cool. Do you intend to use DeltaCloud (
http://deltacloud.apache.org/), or something like that?



>  I have a couple of systems inside the red hat colo that I had planned
> on
>  reinstalling to f16 and setting up openstack on them to play with the same
>  idea but on a local cloud instance.
>  
For sure, I would like to set up something like that for my own usage.


Is all this inline with the problems you've thought about?
>  
Yes, that is fully in-line, and very interesting!

Denis

PS: why isn't there a virtualisation SIG? As there is already a mailing
list, it may be just a question of adding the corresponding Wiki page?

[Seth Vidal]

On Wed, 7 Dec 2011 18:31:27 +0100
Denis Arnaud <denis.arnaud_fedora(a)m4x.org&gt; wrote:

>  2011/12/7 seth vidal <skvidal(a)fedoraproject.org&gt;
>  
>  > I've looked into spawning virt instances to do building and it is
>  > pretty doable. The problem with them being offered by volunteers is
>  > trust [...]
>  >
>  
>  You are right. I had not thought at that... how naive of me :(
>  
>  The volunteers/trustees would sign the builds with their own private
>  keys, for instance with their FAS keys. Then, we could have some
>  "trustworthiness" ratings for all the submitters, like we have today
>  for the packagers (new packager, proven-packager, sponsor). While the
>  submitter is still not trusted, the centralised Koji infrastructure
>  can duplicate the build, and check that it gives the same results.
>  And even when the submitter is trusted, some random duplicate builds
>  can occur. If the submitter taints the builds, it will be flagged as
>  a potential "fraud". A human being would have to have a look at it
>  then.
>   

Well it's beyond that - we still can't trust the build host isn't
compromised in such a way to also add trojans to a mock chroot we then
build inside of. In short, if people are volunteering VMs for us - we
have to be incredibly suspicious of those VMs and from what they are
built.

If I were going to use random vm's I'd want to:
1. connect using ssh
2. push over my own rpm/python/etc binaries
3. checksum all the rest of the installed (and running) software
4. verify those checksums versus my known good set
5. THEN push over the pkgs to be built

If we run this on ec2 or cloudservers then we can build up our own
image and use that for the initial "trusted" environment. ec2 has the
benefit of letting you share an image with others and from there we can
build.

my goal is a very short stack of code such that anyone can run it to
build pkgs - if only b/c they don't want to run mock locally b/c of
limited local bandwidth or cpu time.

>  Or, the VMs could do "scratch" builds (only). When those
> builds are
>  successful, the VMs then just act as a standard clients to the
>  central Koji servers, and the packages are re-built in that safe
>  infrastructure. Overall, the central Koji infrastructure would be
>  off-loaded from all the scratch builds, as well as from the failed
>  builds. Which is already not so bad, is it? 

I think a reasonable goal is:
 - make it trivial to build pkgs in a remote cloud instance from
 arbitrary yum repository sources
 - implement the requirements to make this trustworthy enough for koji
   to do it, too.

>  
>  
>  That would be very cool. Do you intend to use DeltaCloud (
>  http://deltacloud.apache.org/), or something like that? 
I'm using libcloud, actually. I'm interested in pursuing this in
python, not ruby.

>  >
>  
>  Yes, that is fully in-line, and very interesting! 
 
>  PS: why isn't there a virtualisation SIG? As there is already a
>  mailing list, it may be just a question of adding the corresponding
>  Wiki page? 
I thought there was a cloud sig or something. To be honest the sigs are
mind-numbingly confusing to me as to 1. what they do and 2. which ones
actually still exist/do something. So I tend to not pay much attention
to them.

-sv

[Mo Morsi]

On 12/07/2011 01:25 PM, seth vidal wrote:
> >
> > That would be very cool. Do you intend to use DeltaCloud (
> > http://deltacloud.apache.org/), or something like that?
>  I'm using libcloud, actually. I'm interested in pursuing this in
>  python, not ruby.
>  

Deltacloud's primary interface is REST based so you can use it from any 
language. Additionally there are python specific bindings to the REST 
interface that are pretty simple to use.

https://github.com/mifo/deltacloud-python

In any case, another nifty idea would be to use Snap to take snapshots 
of the build environments on the cloud for local replication on 
different Koji systems and/or local vms:

https://github.com/movitto/snap

   -Mo

[Nicolas Mailhot]

Le mercredi 07 décembre 2011 à 10:36 -0500, seth vidal a écrit :

>  I've looked into spawning virt instances to do building and it
> is
>  pretty doable. The problem with them being offered by volunteers is
>  trust:
>  
>  1. how do we trust the initial installation hasn't been poisoned unless
>  we ship all the bits over ourselves.
>  2. how do we trust the in-flight build isn't molested
>  3. how do the people providing the trust insure against
>  tainted/dangerous builds doing $bad_things on their systems.
>  
>  this is why I concluded that the idea of donated/volunteered VM was not
>  going to work - additionally b/c the bandwidth requirements are
>  non-trivial for many builds. 
Concerning trust, the classic way it has been solved before (by seti…)
is to farm the same build to several independant nodes, cheksum results
and make sure they all agree

Of course that supposes builds are strictly reproductible (centos folks
would love this) and that makes the system a lot less efficient. But
then, trust has a price too


-- 
Nicolas Mailhot

[Denis Arnaud]

2011/12/7 Nicolas Mailhot <nicolas.mailhot(a)laposte.net&gt;

>  Concerning trust, the classic way it has been solved before (by
> seti…)
>  is to farm the same build to several independant nodes, cheksum results
>  and make sure they all agree
>  
Again, we could use that P2P build system just to alleviate the centralised
Koji servers from the scratch builds, on one hand, and from the failing
builds on the other hand. That way, the centralised Koji servers would need
no alteration.

As a side note, rather than using Snap (and Augeas, and...), we (in my
department) tend to prefer Chef (http://www.opscode.com/chef/), which has
got a broader scope, and allows much more complex configurations and
automation tasks.

Denis

[Mo Morsi]

On 12/07/2011 06:20 PM, Denis Arnaud wrote:
>  As a side note, rather than using Snap (and Augeas, and...), we (in my 
>  department) tend to prefer Chef (http://www.opscode.com/chef/), which 
>  has got a broader scope, and allows much more complex configurations 
>  and automation tasks.
>
>  Denis 
Chef, like puppet, is a great tool. But it is somewhat different than 
Snap, Augeas, etc.

With chef, puppet, etc. The end user sets up centralized recipes which 
describe what systems should look like, how they behave, etc.

With Snap, you have an existing system which you would like to use as 
the basis to replicate at some later point (either locally or elsewhere).

In any case, slightly off topic, but Snap is now in Fedora (well 
updates-testing, will be pushed to updates at the end of this week, 
sooner if I can get +1 karma). Plus alot more features have gone in / 
are being pushed, including much expanded documentation, a test harness, 
and the start of the ability to migrate snapshots between hosts (eg 
backup an Ubuntu system, restore on Fedora).

Stay tuned for more updates!
   -Mo