FESCo wants to ban direct stable pushes in Bodhi (urgent call for feedback)

[Josh Boyer]

On Wed, Mar 03, 2010 at 10:54:57AM +0100, Michael Schwendt wrote:
>On Tue, 02 Mar 2010 17:53:40 -0800, Jesse wrote:
>
>> On Wed, 2010-03-03 at 02:37 +0100, Kevin Kofler wrote:
>> > Jesse Keating wrote:
>> > > That's a fair point, but there are significantly fewer people around to
>> > > fix critical issues should they arise on a weekend, and after working 5
>> > > weekdays, some of us like taking the weekend off.
>> > 
>> > Well, I'm around on the weekends and the lack of update pushes for the whole 
>> > weekend has irked me more than once. My intention is not to force you or 
>> > Josh Boyer to work on weekends, but maybe we can find a new volunteer to do 
>> > weekend pushes (and only weekend pushes, so they wouldn't be doing Fedora 
>> > work the full week)? And ideally, update pushes should eventually be 
>> > automatic, just like the Rawhide composes.
>> > 
>> >         Kevin Kofler
>> > 
>> 
>> Except there aren't enough key people available on the weekend to clean
>> up the crap if something goes wrong.
>
>What sort of "crap"? And what precautions could be added to avoid
>producing such crap that requires someone to clean it up (manually)?

1) Packages need to be signed.  To do this, you need access to the signing keys.
This is a rather large hurdle to get over, but we're trying to make sure that
sigul lowers it a bit.  It's not quite ready for more use yet, as we're
currently hitting issues with it crashing under load.  This will be looked at
soon.

The end goal is probably to have koji sign the RPMs right after build and just
use a single "build" gpg-key to sign everything.  However I'm not sure how
close we are to that.

2) Bodhi failures.  These come in a variety of flavors.  The most common is that
it goes to mash an updates-testing repo and koji has "nicely" pruned the signed
copies of the RPMs and mash can't download them.  Fixing requires koji admin
access, again not something given out lightly.

We are taking some precautions on this by essentially re-writing the signed
copies for anything left in the various f1x-updates-testing tags on a daily
basis.  That works well enough for us to actually get about a push-per day done,
but it certainly has races.

Other failures of the more bizarre nature happen as well, such as koji tag moves
failing, or bodhi getting turned off in the middle of a push, or people editing
updates mid-push and bodhi freaking out about that.  These are more rare, but
do happen and often require lots of head scratching and admin-level access to
fix.  At times, a new bodhi needs to be rolled out to fix it and only one person
can do that right now.

josh