Speedup the availability of updates (was: Re: Push scripts, mash) pushes in Bodhi (urgent call for feedback))

[Jesse Keating]

On Fri, 2010-03-05 at 13:49 +0100, Till Maas wrote:
> Hi,
> 
> I have some ideas to speedup the availability of updates. Are there any
> reasons except that the tools to do this do not exist yet, to switch to
> this? I created a wiki page for this:
> https://fedoraproject.org/wiki/User:Till/update_availability_speedup_ideas
> 
> The basic idea is to create new repos $repo-recent, that only includes
> metadata for the new rpms, but references to the same rpms on the
> filesystem that $repo uses.
> 

Giving feedback here but you can throw it in the wiki if you want.

I do like the idea of having the rpms in a single directory.  I had
toyed with the thought a while ago, just never got around to fleshing
out a proposal.  When I was thinking of the mechanics of it however I
was thinking of using hardlinks into the updates-testing and whatever
other directory.  This helps keep the packages with the repodata and
prevents mirroring issues if a mirror hosts updates in a different
location from development.  It would also help us more easily detect
when a package is no longer in use in any of the repos.

I'm not sure I like some of the other suggestions, repodata for -recent
or adding the rpms to the mirrors as soon as the bodhi ticket is filed.
I think those would be a bit more complicated to accomplish.

Here is a walk through of what I think we could easily accomplish:
  Bodhi pushes would put rpms in an updates-packages/ type directory,
hardlink specific things to updates/ and updates-testing/.  
  Rsync updates-packages, updates, updates-testing without --delete to
the master mirror.  
  Rsync updates and updates-testing with --delete to the master mirror.
  Nightly branched compose would rsync with --delete and --link-dest to
updates-packages
  Tail end of nightly branched compose would examine updates-packages
and look for items with no links and no pending bodhi update, or items
only linked to development/13/ and prune them.

I think this would ensure that the package stays on the filesystem at
all times as it moves from one repo into another, only deleting it once
it has gone into stable, or has been removed all together from bodhi.

This is pretty close what you have, but I think it's a bit more doable
from the infrastructure side.  Not something I would want to turn on now
though, not without a lot of testing first, there are already too many
moving parts right now trying to settle into a groove.

-- 
Jesse Keating
Fedora -- Freedom² is a feature!
identi.ca: http://identi.ca/jkeating
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: This is a digitally signed message part
Url : http://lists.fedoraproject.org/pipermail/devel/attachments/20100308/44099f98/attachment.bin