6:54 a.m.
Pushing updates is getting a bit untenable. I start signing every morning and
there are usually hundreds of RPMs to sign for each repo we push for. That
usually takes at a minimum 2 hours. Given that the FAS timeout is 20min, I
have to restart the push and by then there are usually a number more updates
that have been added which causes me to have to sign more. Rinse, repeat until
I can catch up.
Again, I do this every day, so it's not like I'm letting them sit for a while.
Last week it took two working days to get things signed. There are various
reasons for this (pushes take a while with drpms, KDE introduces hundreds of
RPMs to sign per update, kernel and oo.org take a while to sign due to size,
etc). Whatever the case may be, it's getting harder and harder to get updates
pushed on a regular basis. If it takes me well into the afternoon to get stuff
signed, then the updates mashes start to overlap with the rawhide mashes and
that generally leads to huge mash times.
The idea I had for this a while ago is to have a -P --save option to bodhi,
combined with a -P --restore. This would basically save off the json request
that is sent to bodhi in a file somewhere. I could sign the RPMs included in
that specific push request without having to worry about the FAS timeout. Then
I could use the --restore option to get that exact state back and send the
push through. I think this would really help the turn around time.
However, there are a couple of complications. The first is that I suck and
haven't found the time to actually look at implementing this. The second is
that the existing non-TG2 bodhi is in a bit of a lame duck state. The thrid
(and possibly most important) reason is that the --save/--restore wouldn't
take into account updates in that set being deleted, etc.
This got me thinking more about some of the various issues, and I really think
we need to be able to lock updates once they are in 'push' state so that they
aren't editable. We've had a number of weird issues arise because people edit
updates that are in the middle of a push. Locking the updates would also allow
a save/restore to work properly without too many other race condition type
issues.
On the signing front alone, there are a couple things we could do with some
additional bodhi/koji work. The first is to have koji auto-sign everything. I
think that is the best solution, but it's also the farthest off and I would
rather not wait for that. Another idea is to have bodhi put packages in a
special tag when they are requested for push and remove them once the push is
complete. E.g.
User A submits package for F12 updates-testing push. Bodhi queues it up like
normal, and does the equivalent of 'koji tag-pkg f12-updates-testing-push'.
When the push is complete, it untags the packages from said tags.
Then I could actually run the sigul script on the tag instead of relying on
bodhi to get me a list of packages that need signing. It would increase the
time I have for signing as well, since bodhi won't give me the list of packages
queued while a push is going on.
So those are my rambles. I have no idea how feasible they are, but I don't see
the number of updates/newpackage requests we get dropping anytime soon. In
the absence of updates sanity, I guess we have to deal with the crazy that
grows.
josh
(P.S. Luke, totally unrelated to this but don't forget about mash in mock
chroot for bodhi TG2)
10:53 a.m.
On Wed, Nov 25, 2009 at 07:54:31AM -0500, Josh Boyer wrote:
The idea I had for this a while ago is to have a -P --save option to
bodhi,
combined with a -P --restore. This would basically save off the json request
that is sent to bodhi in a file somewhere. I could sign the RPMs included in
that specific push request without having to worry about the FAS timeout. Then
I could use the --restore option to get that exact state back and send the
push through. I think this would really help the turn around time.
I like this idea, and it would be fairly simple to implement, since the
/admin/mash method accepts a list of updates.
However, there are a couple of complications. The first is that I
suck and
haven't found the time to actually look at implementing this. The second is
that the existing non-TG2 bodhi is in a bit of a lame duck state. The thrid
(and possibly most important) reason is that the --save/--restore wouldn't
take into account updates in that set being deleted, etc.
Yeah, we'd have to add logic in the AdminController to verify the state
of the updates to ensure that they haven't been deleted/obsoleted before
sending them off to the masher.
This got me thinking more about some of the various issues, and I
really think
we need to be able to lock updates once they are in 'push' state so that they
aren't editable. We've had a number of weird issues arise because people edit
updates that are in the middle of a push. Locking the updates would also allow
a save/restore to work properly without too many other race condition type
issues.
Yep, we definitely need some sort of push state within updates to avoid
race conditions. I have yet to tackle this problem with the bodhi v2.0
model so far, but it's something we can discuss plan at FUDCon.
For example, if an update is being "pushed", and a developer really
doesn't want it to go out, what should that workflow look like?
"Click here to notify releng?", "Unpush anyway" (and have the masher
check the state of the updates multiple times throughout the process to
weed out any unpushed/obsolete updates?), etc.
On the signing front alone, there are a couple things we could do
with some
additional bodhi/koji work. The first is to have koji auto-sign everything. I
think that is the best solution, but it's also the farthest off and I would
rather not wait for that. Another idea is to have bodhi put packages in a
special tag when they are requested for push and remove them once the push is
complete. E.g.
User A submits package for F12 updates-testing push. Bodhi queues it
up like
normal, and does the equivalent of 'koji tag-pkg f12-updates-testing-push'.
When the push is complete, it untags the packages from said tags.
The push tag is an interesting idea. This would also allow bodhi to
easily determine if an update is being pushed, w/o making any database
changes. I like it.
So those are my rambles. I have no idea how feasible they are, but I
don't see
the number of updates/newpackage requests we get dropping anytime soon. In
the absence of updates sanity, I guess we have to deal with the crazy that
grows.
Productive rambles, thank you for brain dumping. This is all important
stuff, and it all seems feasible. Are you going to be at FUDCon, Josh?
Either way, I definitely plan on putting some cycles into solving these
problems in the very near future. I'm focusing on Moksha/FComm stuff
for my FUDCon presentation now, but will be diving deep into Bodhi
during and post-FUDCon.
(P.S. Luke, totally unrelated to this but don't forget about
mash in mock
chroot for bodhi TG2)
Yes, it's on the plate :)
luke
7:05 p.m.
On Wed, Nov 25, 2009 at 11:53:52AM -0500, Luke Macken wrote:
> So those are my rambles. I have no idea how feasible they are,
but I don't see
> the number of updates/newpackage requests we get dropping anytime soon. In
> the absence of updates sanity, I guess we have to deal with the crazy that
> grows.
Productive rambles, thank you for brain dumping. This is all important
stuff, and it all seems feasible. Are you going to be at FUDCon, Josh?
Unfortunately, no. My family is going through some rough times and it's
better if I'm around during the crunch time between Thanksgiving and
Christmas for a variety of reasons.
Either way, I definitely plan on putting some cycles into solving
these
problems in the very near future. I'm focusing on Moksha/FComm stuff
for my FUDCon presentation now, but will be diving deep into Bodhi
during and post-FUDCon.
That's good to know. I do plan on reviewing whatever materials come
out of FUDCon and participating online if possible. I'm hoping to
find some time to actually look at hacking a bit too.
> (P.S. Luke, totally unrelated to this but don't forget about
mash in mock
> chroot for bodhi TG2)
Yes, it's on the plate :)
Good deal ;)
josh
10:56 a.m.
On Wed, 2009-11-25 at 07:54 -0500, Josh Boyer wrote:
On the signing front alone, there are a couple things we could do
with some
additional bodhi/koji work. The first is to have koji auto-sign everything. I
think that is the best solution, but it's also the farthest off and I would
rather not wait for that. Another idea is to have bodhi put packages in a
special tag when they are requested for push and remove them once the push is
complete. E.g.
User A submits package for F12 updates-testing push. Bodhi queues it up like
normal, and does the equivalent of 'koji tag-pkg f12-updates-testing-push'.
When the push is complete, it untags the packages from said tags.
Then I could actually run the sigul script on the tag instead of relying on
bodhi to get me a list of packages that need signing. It would increase the
time I have for signing as well, since bodhi won't give me the list of packages
queued while a push is going on.
We also need to get some mitr time to make sigul run multithreaded.
We're far far underutilizing the hardware we allocated for this system.
As to the above, why can't we just sign everything in the various
*-updates-candidate tags? You'll wind up signing more than what is
going to be pushed, but since you're doing it frequently it'll not
matter as much. Since we are using only one key for F11 and F12
updates, this would accomplish all the signing needed for those trees.
--
Jesse Keating
Fedora -- Freedom² is a feature!
identi.ca: http://identi.ca/jkeating
12:42 p.m.
On Wed, Nov 25, 2009 at 08:56:06AM -0800, Jesse Keating wrote:
On Wed, 2009-11-25 at 07:54 -0500, Josh Boyer wrote:
> On the signing front alone, there are a couple things we could do with some
> additional bodhi/koji work. The first is to have koji auto-sign everything. I
> think that is the best solution, but it's also the farthest off and I would
> rather not wait for that. Another idea is to have bodhi put packages in a
> special tag when they are requested for push and remove them once the push is
> complete. E.g.
>
> User A submits package for F12 updates-testing push. Bodhi queues it up like
> normal, and does the equivalent of 'koji tag-pkg f12-updates-testing-push'.
> When the push is complete, it untags the packages from said tags.
>
> Then I could actually run the sigul script on the tag instead of relying on
> bodhi to get me a list of packages that need signing. It would increase the
> time I have for signing as well, since bodhi won't give me the list of packages
> queued while a push is going on.
We also need to get some mitr time to make sigul run multithreaded.
We're far far underutilizing the hardware we allocated for this system.
As to the above, why can't we just sign everything in the various
*-updates-candidate tags? You'll wind up signing more than what is
going to be pushed, but since you're doing it frequently it'll not
matter as much. Since we are using only one key for F11 and F12
updates, this would accomplish all the signing needed for those trees.
Aside from the time and koji storage, I see no real issues. I was going
to try that next week. I'll let you know how it goes.
josh
12:43 p.m.
On Wed, 2009-11-25 at 13:42 -0500, Josh Boyer wrote:
On Wed, Nov 25, 2009 at 08:56:06AM -0800, Jesse Keating wrote:
>On Wed, 2009-11-25 at 07:54 -0500, Josh Boyer wrote:
>> On the signing front alone, there are a couple things we could do with some
>> additional bodhi/koji work. The first is to have koji auto-sign everything. I
>> think that is the best solution, but it's also the farthest off and I would
>> rather not wait for that. Another idea is to have bodhi put packages in a
>> special tag when they are requested for push and remove them once the push is
>> complete. E.g.
>>
>> User A submits package for F12 updates-testing push. Bodhi queues it up like
>> normal, and does the equivalent of 'koji tag-pkg
f12-updates-testing-push'.
>> When the push is complete, it untags the packages from said tags.
>>
>> Then I could actually run the sigul script on the tag instead of relying on
>> bodhi to get me a list of packages that need signing. It would increase the
>> time I have for signing as well, since bodhi won't give me the list of
packages
>> queued while a push is going on.
>
>We also need to get some mitr time to make sigul run multithreaded.
>We're far far underutilizing the hardware we allocated for this system.
>
>As to the above, why can't we just sign everything in the various
>*-updates-candidate tags? You'll wind up signing more than what is
>going to be pushed, but since you're doing it frequently it'll not
>matter as much. Since we are using only one key for F11 and F12
>updates, this would accomplish all the signing needed for those trees.
Aside from the time and koji storage, I see no real issues. I was going
to try that next week. I'll let you know how it goes.
josh
Koji storage would be a good test, as it's a small flavor of
"automatically sign everything" which is an approved proposal.
--
Jesse Keating
Fedora -- Freedom² is a feature!
identi.ca: http://identi.ca/jkeating
5263
days inactive
5265
days old
rel-eng@lists.fedoraproject.org
5 comments
3 participants
participants (3)
-
Jesse Keating -
Josh Boyer -
Luke Macken