InstantMirror needs a rethink

[Arthur Pemberton]

On Jan 23, 2008 6:02 PM, Warren Togami <wtogami at redhat.com> wrote:
> Today InstantMirror is pretty useful for home and small office mirrors,
> but its limitations make it unsustainable without manual intervention of
> the sysadmin.
>
> I've been beginning to think that perhaps InstantMirror is heading down
> the wrong path and we seriously need to rethink it.  There are simply
> too many limitations of the current "stateless" operation of
> InstantMirror where it runs only on-demand as mod_python script:
>
> - Synchronization/locking of multiple connections downloading the same
> file is awkward and broken.
> - There is no good way to clean up aborted tmp files.
> - There is no good way to know what are old files that need pruning.
> - There is no good way of keeping track of the "Big Picture" of its own
> cache, "least recently used" knowing what files were unpopular locally
> and should be pruned.
>
> https://fedorahosted.org/InstantMirror/wiki/InstantMirrorDaemon
> We need a daemon to handle all this.  Perhaps the daemon could allow
> socket connections from a mod_python script for accesses.  Or perhaps it
> might be better for the daemon itself to handle serving connections.
>
> Stepping back, what we really need is:
> A reverse proxy caching server with all the logic of squid or varnish,
> except it stores its cache with file and directory names intact.
>
> How do we get there?
> 1) Write a new daemon from scratch?
> 2) Write a new backend storage engine for squid or varnish?  (Store
> files in target directory structure, store metadata elsewhere.)
> 3) ???
>
> Any thoughts?


Am about to investigate on-demand style mirroring as opposed to
rsync/ftp myself, and came across InstantMirror. What I have not yet
been able to determine is what is the benefit of InstantMirror over
just running squid and having yum always use said proxy?? (yum really
should be able to independently use an http proxy)

-- 
Fedora 7 : sipping some of that moonshine
( www.pembo13.com )