Re: InstantMirror Redesign / current best /simplest way to achieve

[chasd]

Warren Togami wrote:

>  Arthur Pemberton wrote:
> >
> > I wanted to try InstantMirror, but was in a rush. I just used a basic
> > squid setup.
> >
> > Are their any advantages of using it as a reverse mirror vs. regular
> > squid usage?
> >
>
>  Regular squid wont be able to effectively cache if MirrorManager is
>  telling you to use random mirrors.  If you use MirrorManager you could
>  receive the same local (reverse proxy) mirror as the first mirror  
>  every
>  time if yum is used from your network block.
>
>  refresh_pattern repodata/.*$    0       0%      0
>  refresh_pattern .*rpm$    0     0%      0
>
>  Also with any squid.conf you will need these lines in order to  
>  guarantee
>  that your repodata and RPMS stay consistent with your upstream source.
>  This is because proxies do not handle data changing without  
>  changing the
>  filename. 
Um, from reading the comments in the squid config file relating to
"refresh_pattern," don't those settings in effect negate caching,
since that will always return STALE ?  From the squid config file :

>  #	Basically a cached object is:
>  #
>  #		FRESH if expires < now, else STALE
>  #		STALE if age > max
>  #		FRESH if lm-factor < percent, else STALE
>  #		FRESH if age < min
>  #		else STALE 
Wouldn't you want squid to keep a file at least a couple of hours, if  
not a day ?

Settings I use to better cache OS updates using squid :

# Use more than the default 8 MB of RAM
cache_mem 128 MB
# Use 8 MB for the largest object in RAM cache instead of 8KB
maximum_object_size_in_memory 8192 KB

A large percentage of RPMs are less than 8 MB

# Use 30GB for the cache instead of the default 100 MB
cache_dir aufs /var/spool/squid 30000 48 512

If you have an update run that totals close to the default 100 MB,
and you have multiple clients set to update via a cron script,
the cache will thrash if you don't set that higher.

Those settings above are hardware dependent,
if you have more, let squid use more

I also use an ext2 partition for where the cache lives since it
seems the journaling of ext3 can impact squid performance,
particularly on older PPC hardware kjournald will keep rising to the  
first line
of "top" and squid can block on I / O.
If the fs gets mangled, only the squid cache is on it
so I can wipe it and re-init the cache

# Use maximum object size of 120 MB not 4 MB - gotta cache openoffice  
fer sure.
maximum_object_size 122880 KB

That said, I use InstantMirror with a custom repo file for our Fedora  
needs.


Charles Dostale

InstantMirror Redesign / current best /simplest way to achieve

[Warren Togami]

chasd wrote:
> > refresh_pattern repodata/.*$    0       0%      0
> > refresh_pattern .*rpm$    0     0%      0
> >
> > Also with any squid.conf you will need these lines in order to guarantee
> > that your repodata and RPMS stay consistent with your upstream source.
> > This is because proxies do not handle data changing without changing the
> > filename.
>  
>  Um, from reading the comments in the squid config file relating to
>  "refresh_pattern," don't those settings in effect negate caching,
>  since that will always return STALE ?  From the squid config file : 
This was already discussed earlier in this thread.  This doesn't expire 
the object from the cache entirely.  It does check the upstream source 
for changes but it doesn't need to download the entire file again if it 
did not change.

>  If you have an update run that totals close to the default 100 MB,
>  and you have multiple clients set to update via a cron script,
>  the cache will thrash if you don't set that higher. 
Right, you need to adjust the cache_dir size and maximum object size.  I 
personally use a maximum object size of like 4GB because my reverse 
proxy is used *only* for a Fedora mirror.

>  That said, I use InstantMirror with a custom repo file for our Fedora
>
>  needs. 
Note, InstantMirror behaves exactly as described above with the squid 
refresh_pattern rule.  InstantMirror checks the original source files 
for changes upon every client request.

Due to repodata/*, images/* and sometimes *.rpm files changing without 
their contents changing, running a caching proxy without such rules 
causes you to occasionally have metadata mismatch failures until the 
proxy server naturally refreshes itself sometime later.

Warren Togami
wtogami(a)redhat.com