devel Digest, Vol 94, Issue 22

Thu Dec 8 00:53:19 UTC 2011

> Date: Thu, 08 Dec 2011 04:34:57 +0900
> From: 夜神 岩男 <supergiantpotato at yahoo.co.jp>
>
> Each time a build is made, the building system makes a hash of the set
> of RPMs in $wherever/mock/result/{foo,bar} and sends the completed data
> back to the Fedora build system.
> [...]
> This system depends chiefly on one thing: Having a boatload of
> contributors. I think we could easily expect 1,000+ active contributors,
> particularly if we make this a happy competition complete with a stats
> tracker the way the BOINC and SETI at Home trackers work. People who send
> more than 1 bad hash could be notified by their system that things
> aren't working out which could be an early warning in identifying
> whether the system is indeed being attacked (or the individual's system
> has been compromised). A throttle based on such indicators could let us
> know to halt the distributed build and switch back to "old koji".
>

I really like the idea of having a build infrastructure the same way
SETI at Home provided. It is maybe just a matter of generation (younger
developers do not even know what SETI is, I guess)?
It reminds me another related initiative, my trainee and I started two
years ago (http://www.corefarm.org:81/blog/), which gave birth to Corefarm
(computing grid for 3D image rendering). By the way, the model could be
similar to that of Corefarm: one version (http://www.corefarm.com/) using
some "safe" Amazon EC2 and/or RedHat/Fedora trusted clouds; another version
http://www.corefarm.org, community-based, for building personal scratch
builds.

1. Sorry, silly package name idea from the image of city
> construction "koji" + "cloud"
>

Skyline is just fine :)

> Date: Wed, 07 Dec 2011 15:02:42 -0500
> From: Przemek Klosowski <przemek.klosowski at nist.gov>
>
> I'd say we need to be paranoid on this one and consider a tainted
> kernel where your own binaries would report A-OK on a rigged gcc because
> kernel has a special case for it. Test builds and QA might be OK, but
> nothing that results in shipped bits.
>

It reminds me of the back-door introduced by Ken Thompson (
http://cm.bell-labs.com/who/ken/trust.html), and having lived a long way
before being eventually disclosed by its author.
Hmm... when we moreover add Adam's objection (that someone could tap into a
proven-packager's flow, and then tamper any package on the central Fedora
repository) on top of it, the security issue does not appear that much easy
to alleviate :(

> Date: Thu, 08 Dec 2011 05:35:02 +0900
> From: 夜神 岩男 <supergiantpotato at yahoo.co.jp>
>
> I think you are correct in essentially asking if this is a solution
> in search of a problem.
>

Let me try to state my use case.
1. I would like to be able to throw chain builds (for one of my upstream
projects, there are something like 15 components to be rebuilt one after
the other) and mass rebuilds (for instance, when Boost is upgraded, i.e.,
every Fedora release) on some dedicated buildroots and/or with some
specific tags. I know that Koji already allows to do that.
2. I would like to be able to throw those build requests from anywhere,
including behind corporate firewalls. And, here, Koji is not (yet?) able to
do that.

While it is very possible to do something like this, and the idea is
> exciting because it is something new, I've never heard of anyone kicking
> and screaming about a wait queue bottleneck or insufficient resources in
> the Fedora infrastructure.

Yes, security put aside, I believe that it would be very exciting, for a
lot of contributors, to participate in that way to Fedora. It would make
Fedora unique, as well, among all the Linux distributions. But Fedora does
not need to be unique, as it is already the best distribution, isn't it?

Date: Wed, 7 Dec 2011 16:15:56 -0500
> From: seth vidal <skvidal at fedoraproject.org>
>
> So I have two thoughts on this:
> 1. Scratch or personal chainbuilds could be built in ec2 or rax
> or anywhere w/o an issue
> 2. For our shipping pkgs, building them in the existing koji build-systems
> and/or in a dedicated private cloud instance makes sense - if only for
> resource allocation and control.
>

Again, security put aside, it sounds great.

It would be a very good start, already, to provide a simple way (i.e., a
package group like "fedora-cloud-packager"?) to set up, locally and/or on a
private cloud, a package build and repository system. All the components
are more or less already in Fedora today (Koji server, cloud software
stack); it is then a matter of gluing everything together in a consistent
manner, so that it becomes easy for any packager to set up and to use.

Date: Wed, 07 Dec 2011 13:25:28 -0800
> From: Adam Williamson <awilliam at redhat.com>
>
> I'm not sure we can treat scratch / personal builds with *quite* so
> much abandon. They're still valuable targets for anyone trying to
> compromise Fedora, after all.
> Who uses scratch builds the most? Well, probably Fedora packagers, right?
> And we probably wind up deploying them on our own systems after we build
> them. That's what scratch builds are _for_ - testing your stuff before
> pushing it out more widely.
>

While I tend to agree that this perspective is pretty scaring, I also
believe that 夜神 岩男's ideas could tame that threat, at least a little bit.
We could maybe have circles of trust/friends, much like PGP keys are
managed and trusted, and much like some P2P networks (e.g., OneSwarm (
http://www.oneswarm.org/) and RetroShare (http://retroshare.sourceforge.net/))
are secured.

> Date: Wed, 7 Dec 2011 12:45:59 -0900
> From: Jef Spaleta <jspaleta at gmail.com>
>
> I'd also like to see if we can use utility cloud resources to efficiently
> scale out a mass rebuild on demand and perform self-hosting test runs or
> failure to build runs on a regular scheduled basis. Something we can do in
> a well managed utility cloud I would think without slowing down day to day
>  packaging work.
>

For instance, we could build more frequently specific spins, and multiply
their number.

Regards

Denis
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.fedoraproject.org/pipermail/devel/attachments/20111208/0f84a032/attachment.html 