Need some updates push changes

[Josh Boyer]

Pushing updates is getting a bit untenable.  I start signing every morning and
there are usually hundreds of RPMs to sign for each repo we push for.  That
usually takes at a minimum 2 hours.  Given that the FAS timeout is 20min, I
have to restart the push and by then there are usually a number more updates
that have been added which causes me to have to sign more.  Rinse, repeat until
I can catch up.

Again, I do this every day, so it's not like I'm letting them sit for a while.
Last week it took two working days to get things signed.  There are various
reasons for this (pushes take a while with drpms, KDE introduces hundreds of
RPMs to sign per update, kernel and oo.org take a while to sign due to size,
etc).  Whatever the case may be, it's getting harder and harder to get updates
pushed on a regular basis.  If it takes me well into the afternoon to get stuff
signed, then the updates mashes start to overlap with the rawhide mashes and
that generally leads to huge mash times.

The idea I had for this a while ago is to have a -P --save option to bodhi,
combined with a -P --restore.  This would basically save off the json request
that is sent to bodhi in a file somewhere.  I could sign the RPMs included in
that specific push request without having to worry about the FAS timeout.  Then
I could use the --restore option to get that exact state back and send the
push through.  I think this would really help the turn around time.

However, there are a couple of complications.  The first is that I suck and
haven't found the time to actually look at implementing this.  The second is
that the existing non-TG2 bodhi is in a bit of a lame duck state.  The thrid
(and possibly most important) reason is that the --save/--restore wouldn't
take into account updates in that set being deleted, etc.

This got me thinking more about some of the various issues, and I really think
we need to be able to lock updates once they are in 'push' state so that they
aren't editable.  We've had a number of weird issues arise because people edit
updates that are in the middle of a push.  Locking the updates would also allow
a save/restore to work properly without too many other race condition type
issues.

On the signing front alone, there are a couple things we could do with some
additional bodhi/koji work.  The first is to have koji auto-sign everything.  I
think that is the best solution, but it's also the farthest off and I would
rather not wait for that.  Another idea is to have bodhi put packages in a
special tag when they are requested for push and remove them once the push is
complete.  E.g.

User A submits package for F12 updates-testing push.  Bodhi queues it up like
normal, and does the equivalent of 'koji tag-pkg f12-updates-testing-push'.
When the push is complete, it untags the packages from said tags.

Then I could actually run the sigul script on the tag instead of relying on
bodhi to get me a list of packages that need signing.  It would increase the
time I have for signing as well, since bodhi won't give me the list of packages
queued while a push is going on.

So those are my rambles.  I have no idea how feasible they are, but I don't see
the number of updates/newpackage requests we get dropping anytime soon.  In
the absence of updates sanity, I guess we have to deal with the crazy that
grows.

josh

(P.S.  Luke, totally unrelated to this but don't forget about mash in mock
chroot for bodhi TG2)