announcing repository updates over Fedora Message Bus

[Ralph Bean]

On Mon, Sep 10, 2012 at 08:56:26AM -0400, Kamil Paral wrote:
> I wonder, is it possible for Fedora Infrastructure team to implement announcements of new repository pushes over Fedora Message Bus[1]?
> 
> By new repository pushes I mean updates to 'fedora', 'updates' and 'updates-testing' repositories for all Fedora releases as well as updates to 'branched' and 'rawhide' trees.

Some of this should already be happening.  There is a
"org.fedoraproject.prod.bodhi.mashtask.sync.done" message that should
be getting emitted, but we don't have a way to check the history and
verify yet. https://bugzilla.redhat.com/show_bug.cgi?id=853252 should
take care of that.

"branched" and "rawhide" are another story.  We'd need to add
"fedmsg-logger" statements to some of the scripts in
http://git.fedorahosted.org/cgit/releng/tree/scripts, but I'm not sure
where yet.

> The benefits of this solution:
> 1. Mirrors that download from dl.fp.o could start syncing very soon after an update is pushed (after a random timeout to decrease the load).
> 2. Those mirrors wouldn't have to periodically query the server for updates.
> 3. The duration when those mirrors are outdated would be shorter.
> 
> I can just guess about other people, but in our office I believe the message bus would help a lot. Currently we query the master server every 4 hours for updates. That presents a lot of bandwidth/IO load on both sides. Using the message bug there would be no useless queries. Also it would help us stay up-to-date more often that we currently are (if 'branched' is updated an hour after our last query, there are 3 hours wasted until we run rsync again).
> 
> This solution could be extended in the future to all Fedora public mirrors, so that they could announce using our message bus when they have finished syncing and users that sync from that particular mirror would know when to start downloading from it. (E.g. if I use mirrors1.kernel.org to sync my private mirror, I would wait for their announcement on our message bus).
> 
> What do you think?

This sounds great for the future.  I've been working on docs this week
(not quite done yet).  I'll have docs out soon with examples on how to
do something like this.