6:58 p.m.
#6420: Hypothetical maximum speed mirroring
-----------------------------+------------------------
Reporter: tibbs | Owner: rel-eng@…
Type: task | Status: new
Milestone: Fedora 24 Alpha | Component: koji
Keywords: | Blocked By:
Blocking: |
-----------------------------+------------------------
Over the past few days I've put together something I cal quick-fedora-
mirror. It lives at https://pagure.io/quick-fedora-mirror and the
documentation there should describe how it works. But an executive
summary:
* There's a server-generated file list.
* It turns the "transferring file list" portion of a full rsync update of
fedora-buffet0 from eleven hours to a few seconds unless there has been a
large amount of updated content. In all cases except one (when the mirror
has no content at all) it is faster.
* I can poll every module in fedora-buffet0 every ten minutes and I don't
think the servers even notice. Each poll requires one rsync call to fetch
one file per rsync module, plus one to transfer everything. If there's no
changed content, the poll takes about six seconds.
* Yes, it correctly copies hardlinked content on the server as hardlinks
on the client. This works even between modules, assuming the client
actually mirrors those modules.
* It works for mirrors mirroring from other mirrors.
* It handles the bitflip and fetching files which are missing or deleted
on the client.
* The client needs only rsync and (currently) zsh. I believe it is
portable to bash4 without too much trouble. bash3 might be tough and
would need an expert. But the point is that clients don't need to have
any special software to do this.
I am looking at ways to leverage quick-fedora-mirror, fedmsg and perhaps
mirrormanager to maximize the speed at which we get content out to the
mirrors.
The first pass:
When an updated file list is generated on the master, emit a fedbus
message. I think this is easy to do. Tier 1 mirrors can just watch for
this message, sleep $((RANDOM*60/32768)) or whatever, and fire off a
transfer. They should all have, say, updated fedora updates content in
ten or fifteen minutes.
The second pass:
The server can tell mirrormanager that updated content is ready, and
mirrormanager can emit the message.
The third pass:
When a Tier 1 runs report-mirror, mirrormanager can emit a message to the
bus indicating the particular tier 1 is saying it now has updated content.
Tier 2 mirrors can listen for their upstream host (maybe even several of
them) and start their run. And so on.
I have a two-tiered system running right now. One mirror pulls from the
Fedora masters (by polling, currently) and then tickles two slave mirrors
to get the new content.
Questions:
* Does anyone see any flaws in this plan?
* Is this (i.e. fanning out new content quickly) something we would want?
* Is it reasonable for mirrormanager to do this? I know that currently
the master only informs it of new fedora/epel content, I don't know about
the other modules The master does know when they change (if only because a
cron job runs) and I don't think it would be hard to let mirrormanager
know. I don't know if mirrormanager can do anything with that
information.
* Is there any possibility of getting sufficient mirrors onboard to make
this useful? I tried to severely limit the dependencies on the client (to
a shell script, rsync, awk and the usual utilities). Even report_mirror
needs python.
--
Ticket URL: <https://fedorahosted.org/rel-eng/ticket/6420>
Fedora Release Engineering <http://fedorahosted.org/rel-eng>
Release Engineering for the Fedora Project