8:41 p.m.
I just had an in-depth discussion with Henrik Nordström of the Squid
project about how HTTP mirrors and the yum tool itself could be improved
to safely handle proxy caches. He gave me lots of good advice about how
HTTP mirrors can be configured for cache safety, Squid can be configured
for yum metadata cache safety, and yum itself can be improved to be more
robust in dealing with proxy caches.
(It turns out that Henrik is an avid Fedora user, and I might have
convinced him to come onboard the Fedora Project to contribute another
useful tool and become co-maintainer of his own package. It would be an
honor to have him onboard as a Fedora Developer. =)
Yum and Proxy Caches: Current Dangers
=====================================
Users may be using proxy servers in 3 (or more) ways:
1) Many users today are behind a transparent proxy cache, either
instituted by their ISP, school, or business network.
2) Other users might have Internet access *only* through a proxy server.
3) Other users might be using a reverse proxy server on their local
network as a caching yum mirror.
There are two cases where yum has problems with proxy caches:
1) A RPM package changes content without changing filename. This
usually happens only in instances where a package was pushed unsigned
then was later signed. A simple workaround within yum is discussed
later in this mail.
2) yum currently has problems with proxy caches due to common cases
where metadata can become partially out of sync. This happens because
repomd.xml is grabbed often while other repodata files are grabbed less
often. repomd.xml is then checked for origin "freshness" more often.
When repodata changes on the origin, repomd.xml is refreshed on the
cache before other repodata files. yum clients seeing the new
repomd.xml but old primary.sqlite.bz2 error out.
Ideal Solution for #2 Partial Repodata Sync Problem
===================================================
Henrik highly suggests using versioned repodata files as the ideal
solution to this problem. This way caches can serve repodata without
fear of the sync problem, and also without querying the origin server
upon every client download. repomd.xml would contain changing filenames
perhaps with timestamp or something in their filenames.
i.e.
primary-1201140584.sqlite.bz2
This would be an elegant solution, but will it be possible for us to
migrate to because older clients wouldn't be able to handle it?
I'm guessing not, so here are other less efficient but workable solutions.
"Cache-Control: max-age=0"
==========================
This HTTP header directive can be either in the request or response.
This instructs the proxy cache server to always query the origin HTTP
server to check if the requested file has changed. It compares the
origin's reported Last-Modified or ETag to what Squid knows in its own
cache.
This means that each and every request for repodata/* files will trigger
a query to the origin server. This is a relatively quick operation and
an acceptable compromise if we cannot make repodata filenames versioned.
This HTTP directive can be done for repodata/* files at three levels:
1) Origin HTTP mirrors can be configured to serve "Cache-Control:
max-age=0" in HTTP headers whenever they serve repodata/* files. This
can become a standard recommendation for all Fedora mirrors. Does
anyone know how to configure Apache to do this?
2) Squid refresh_pattern can use a regex to override max-age=0 for
repodata/* files. I haven't figured out exactly what the syntax is for
this. Anybody know squid.conf?
3) yum can always include the HTTP directive in its request for
repodata/* files. Can we make this the default in future versions of
yum? I personally don't see a drawback (unless repodata becomes
versioned, then we don't want this.)
Yum and "X-Cache: HIT"
======================
If you use wget --server-response and a target file, you see the raw
HTTP headers of that request. If the file is already cached, you see a
HTTP header like below:
X-Cache: HIT from proxyserver.example.com
Proposal:
Improve yum with the following download logic:
IF (a downloaded repodata/* file doesn't match the repomd.xml checksum
OR a downloaded RPM doesn't match the expected checksum)
AND "X-Cache: HIT from" was in its HTTP header
THEN download it again with URLGrabber option: http_headers =
(('Pragma', 'no-cache')
This should solve the case where RPM files legitimately change contents
without changing filenames, like RPM signing. This also correctly does
NOT trigger additional downloads upon other errors like corrupted files.
Squid Configuration Suggestion
==============================
collapsed_forwarding
This option is not default, but pretty useful for proxy servers of our
type. This option makes multiple clients asking for the same file not
yet in the server's cache to wait on the same origin download connection
instead of spawning more downloads of the same thing.
Upstream Squid on Adapting Squid's Storage Engine
=================================================
<hno> Squid currently has an abstract index keyed on integers which
makes this hard to implement, but we are planning to break that out from
Squid allowing the cache to structure itself in whatever manner, with
one possible approach to use store the URLs as-is in the filesystem
(within certain bounds)
<hno> Apache do not have this same abstract internal layer, and writing
a mod_disk_cache replacement which keeps a mirror type file structure
should be pretty easy thing to do.
<adri> Its at least on my draft squid-2 roadmap for ~6 months from now
<adri> Since its going to be important for people running squid in
environments where high number sof lookups for large objects isn't
required, but they want to cache gigabytes/terabytes of $LARGE objects
<adri> without having huge amounts of RAM involved
So unfortunately squid currently cannot be adapted to be the perfect
InstantMirror, but we might be able to achieve it quickly by adapting
Apache's mod_disk_cache. Anybody touched this part of Apache before?
Warren Togami
wtogami(a)redhat.com
9:04 p.m.
On Wed, 23 Jan 2008 21:41:40 -0500
Warren Togami <wtogami(a)redhat.com> wrote:
i.e.
primary-1201140584.sqlite.bz2
This would be an elegant solution, but will it be possible for us to
migrate to because older clients wouldn't be able to handle it?
I'm guessing not, so here are other less efficient but workable
solutions.
Hrm, I would think that this would be possible for recentish clients
(RHEL5 and later?). repomd.xml lists out what the other files are,
and the checksums of them, so if the files are named slightly different
yum should handle it I would think.
In fact, I just tried this by hand. I edited repomd.xml and changed
'primary.sqlite.bz2' to 'primary-1234.sqlite.bz2' and then moved the
file on the file system. Yum client didn't even blink, it happily
downloaded the -1234 file.
So if this method is possible, does that change things for the better
for you? I think all it would require is slight modification to
createrepo so that it could timestamp the repo files.
--
Jesse Keating
Fedora -- All my bits are free, are yours?
10:14 p.m.
On Wed, 23 Jan 2008 22:04:51 -0500, Jesse Keating <jkeating(a)redhat.com>
wrote:
On Wed, 23 Jan 2008 21:41:40 -0500
Warren Togami <wtogami(a)redhat.com> wrote:
> i.e.
> primary-1201140584.sqlite.bz2
>
> This would be an elegant solution, but will it be possible for us to
> migrate to because older clients wouldn't be able to handle it?
>
> I'm guessing not, so here are other less efficient but workable
> solutions.
Hrm, I would think that this would be possible for recentish clients
(RHEL5 and later?). repomd.xml lists out what the other files are,
and the checksums of them, so if the files are named slightly different
yum should handle it I would think.
In fact, I just tried this by hand. I edited repomd.xml and changed
'primary.sqlite.bz2' to 'primary-1234.sqlite.bz2' and then moved the
file on the file system. Yum client didn't even blink, it happily
downloaded the -1234 file.
(away from my computer, can't test this until tomorrow)
Oh cool! Can somebody please test how far back in yum versions this works
and report back here?
So if this method is possible, does that change things for the better
for you? I think all it would require is slight modification to
createrepo so that it could timestamp the repo files.
We need to think a bit more and do some testing to be sure there are no
drawbacks in doing this.
If our findings are good, are we comfortable changing this for the Fedora 7
and Fedora 8 updates repos? This would instantly make reverse proxy
mirrors of Fedora less error prone and *WAY* easier to deploy.
(Let us please do this on static-repos as well, the problem is even worse
there.)
Warren Togami
wtogami(a)redhat.com
11:20 p.m.
On Wed, Jan 23, 2008 at 11:14:34PM -0500, Warren Togami wrote:
> In fact, I just tried this by hand. I edited repomd.xml and
changed
> 'primary.sqlite.bz2' to 'primary-1234.sqlite.bz2' and then moved
the
> file on the file system. Yum client didn't even blink, it happily
> downloaded the -1234 file.
(away from my computer, can't test this until tomorrow)
Oh cool! Can somebody please test how far back in yum versions this works
and report back here?
Works on FC5, yum-2.6.1-0.fc5, ftp and http baseurls. I renamed all
the xml files and tested yum install, yum search, yum info, yum
provides. No problems.
This change would help enormously with people stuck behind transparent
proxies, often without their knowledge.
2:22 a.m.
Le Jeu 24 janvier 2008 04:04, Jesse Keating a écrit :
On Wed, 23 Jan 2008 21:41:40 -0500
Warren Togami <wtogami(a)redhat.com> wrote:
In fact, I just tried this by hand. I edited repomd.xml and changed
'primary.sqlite.bz2' to 'primary-1234.sqlite.bz2' and then moved the
file on the file system. Yum client didn't even blink, it happily
downloaded the -1234 file.
So if this method is possible, does that change things for the better
for you?
I made the same analysis several months ago when I setup my own local
mod_proxy cache. I'm glad to see Warren is getting through better than
me at the time. Changing file contents while keeping the same filename
is the ultimate proxy sin.
--
Nicolas Mailhot
1:51 p.m.
On Thu, 2008-01-24 at 09:22 +0100, Nicolas Mailhot wrote:
Le Jeu 24 janvier 2008 04:04, Jesse Keating a écrit :
> On Wed, 23 Jan 2008 21:41:40 -0500
> Warren Togami <wtogami(a)redhat.com> wrote:
> In fact, I just tried this by hand. I edited repomd.xml and changed
> 'primary.sqlite.bz2' to 'primary-1234.sqlite.bz2' and then moved
the
> file on the file system. Yum client didn't even blink, it happily
> downloaded the -1234 file.
>
> So if this method is possible, does that change things for the better
> for you?
I made the same analysis several months ago when I setup my own local
mod_proxy cache. I'm glad to see Warren is getting through better than
me at the time. Changing file contents while keeping the same filename
is the ultimate proxy sin.
Wow, yeh that's one possibility, let's put that down here:
1. Warren managed to finally get through to the stupid yum developers by
having a flamewar on fedora-devel-list, they relented and have turned
the "do all the correct proxy stuff" option on for 3.2.9. Damn them,
infidels, if only they would have changed that option before!
Now we'll have a look around and come up with another possibility, just
as a control, and we'll put here:
2. Yum developers wanted to do transactions over the index and metadata,
to solve a whole bunch of problems and so said developers discussed the
details late-ish last year on #yum and yum-devel and even said it was
going to be fixed when Jesse opened a related bug[1] at the end of
November. It was then written and tested by all of said developers,
over the next 6 weeks or so[2]. Then it was also discussed generating
unique names for the metdata itself within the index, now that we could
see both the old and new index and so do the right thing fairly easily.
Again, more testing and discussion and some code. Then the release of
3.2.9.
Now maybe it's just me, but I have an idea why #1 might not appear to
work so well over the long term ... but feel free to continue using that
method, if you think that's better.
[1] 405201
[2] upstream git commit c3d6b04587499992a32caf36ff3c6f03f8ef2426
--
James Antill <james(a)fedoraproject.com>
2:13 p.m.
Le jeudi 24 janvier 2008 à 14:51 -0500, James Antill a écrit :
Now maybe it's just me, but I have an idea why #1 might not
appear to
work so well over the long term ...
I'd be impressed if the problem was first reported last november. Since
it was reported many times before, and your reaction is the blame the
messenger once again, I'm not.
--
Nicolas Mailhot
2:29 p.m.
On Thu, 2008-01-24 at 21:13 +0100, Nicolas Mailhot wrote:
Le jeudi 24 janvier 2008 à 14:51 -0500, James Antill a écrit :
> Now maybe it's just me, but I have an idea why #1 might not appear to
> work so well over the long term ...
I'd be impressed if the problem was first reported last november. Since
it was reported many times before, and your reaction is the blame the
messenger once again, I'm not.
I was _not_ saying you shouldn't report any problems you have, please
do so and also request any features that you want. Seth esp. will often
respond to a BZ within a frighteningly short amount of time.
Just that there might be a good way of going about something, and a not
so good way.
But no, we don't have infinite resources, and depending on what
you/whoever asked for it might well have been considered too hard or
lower priority than something else.
--
James Antill <james(a)fedoraproject.com>
Fedora
9:28 p.m.
Warren Togami wrote:
Yum and Proxy Caches: Current Dangers
=====================================
Users may be using proxy servers in 3 (or more) ways:
Is there a reasonable way to make yum automatically use files cached in
these local proxies when run by multiple users that share the proxy but
don't know about each other? Picking some random mirror for each will
defeat the caching since they will have different URLs to get the same file.
--
Les Mikesell
lesmikesell(a)gmail.com
10:45 p.m.
On Wed, 23 Jan 2008 21:28:01 -0600, Les Mikesell <lesmikesell(a)gmail.com>
wrote:
Warren Togami wrote:
> Yum and Proxy Caches: Current Dangers
> =====================================
> Users may be using proxy servers in 3 (or more) ways:
Is there a reasonable way to make yum automatically use files cached in
these local proxies when run by multiple users that share the proxy but
don't know about each other? Picking some random mirror for each will
defeat the caching since they will have different URLs to get the same
file.
http://fedoraproject.org/wiki/Infrastructure/Mirroring/SiteLocalMirrors
This page needs updating, but it sort of describes what you need to do.
Use your Fedora account to login to MirrorManager and add a private mirror
for yourself along with a site-local netblock in CIDR notation so the
mirror master knows to serve to you when yum clients come from your
network. After a few hours, mirror lists will serve your mirror first,
then random other mirrors in your country/region.
Example, I have InstantMirror running on my Buffalo Linkstation (hard drive
+ ethernet running embedded Linux) at home. I gave that box a private IP
address and pointed a sub-domain name at that (so I could use Apache
name-based VirtualHost for InstantMirror). I then added to MirrorManager
"Warren's Private Home Mirror" which only I can see when I login to the
management interface. In both cases it is REALLY handy when all of your
local clients automatically begin using your local mirror.
Oh, there might be one more complication. You may need to checkout the
sources for MirrorManager and run the script that reports to MirrorManager
that your mirror is populated. I think mdomsch mentioned that the checkbox
on the MirrorManager interface isn't yet working. Matt, what's the status
of this?
Hopefully that last wrinkle should be gone soon.
Warren Togami
wtogami(a)redhat.com
11:39 p.m.
Warren Togami wrote:
On Wed, 23 Jan 2008 21:28:01 -0600, Les Mikesell
<lesmikesell(a)gmail.com>
wrote:
> Warren Togami wrote:
>
>> Yum and Proxy Caches: Current Dangers
>> =====================================
>> Users may be using proxy servers in 3 (or more) ways:
> Is there a reasonable way to make yum automatically use files cached in
> these local proxies when run by multiple users that share the proxy but
> don't know about each other? Picking some random mirror for each will
> defeat the caching since they will have different URLs to get the same
> file.
http://fedoraproject.org/wiki/Infrastructure/Mirroring/SiteLocalMirrors
This page needs updating, but it sort of describes what you need to do.
Use your Fedora account to login to MirrorManager and add a private
mirror
for yourself along with a site-local netblock in CIDR notation so the
mirror master knows to serve to you when yum clients come from your
network. After a few hours, mirror lists will serve your mirror first,
then random other mirrors in your country/region.
Interesting, but it still requires custom setup for any distro/version
that the proxy admin would want to support. What I'd really like to
happen is for yum to just always prefer the same URL when working
through the same proxy so caching would work by default without needing
to be aware of the cache content. This would work automatically if the
target was a single site, RRDNS, or geo-ip managed DNS, but you probably
can't arrange that for all the repo mirrors. There has to be some clever
way to get the same effect even when using a mirrorlist - like making
sure the mirrorlist itself is cached and always picking the same entry
so any client will use the same URL that the mirrormanger gave to the
first one that made a request. Of course you'd need a reasonable retry
mechanism to pick something else if this choice fails but I'd guess it
would be a big win in bandwidth use and load on the mirrors if it worked
most of the time to take advantage of existing local caches with no
modifications.
--
Les Mikesell
lesmikesell(a)gmail.com
1:31 a.m.
Les Mikesell wrote:
Interesting, but it still requires custom setup for any distro/version
that the proxy admin would want to support. What I'd really like to
happen is for yum to just always prefer the same URL when working
through the same proxy so caching would work by default without needing
to be aware of the cache content. This would work automatically if the
target was a single site, RRDNS, or geo-ip managed DNS, but you probably
can't arrange that for all the repo mirrors. There has to be some clever
way to get the same effect even when using a mirrorlist - like making
sure the mirrorlist itself is cached and always picking the same entry
so any client will use the same URL that the mirrormanger gave to the
first one that made a request. Of course you'd need a reasonable retry
mechanism to pick something else if this choice fails but I'd guess it
would be a big win in bandwidth use and load on the mirrors if it worked
most of the time to take advantage of existing local caches with no
modifications.
I am NOT SURE about this, and it might be NOT SAFE to do (might make a
public mirror disappear from others?), but I suspect that adding your
site-local netblock to one of the mirrors might make that mirror always
appear first in your mirrorlist.
Anyone else know how MirrorManager handles this?
Currently only the mirror owner has access to add site-local netblocks
so this isn't a self-serve solution for users yet.
Warren Togami
wtogami(a)redhat.com
1:52 a.m.
Les Mikesell wrote:
Interesting, but it still requires custom setup for any distro/version
that the proxy admin would want to support. What I'd really like to
happen is for yum to just always prefer the same URL when working
through the same proxy so caching would work by default without needing
to be aware of the cache content. This would work automatically if the
target was a single site, RRDNS, or geo-ip managed DNS, but you probably
can't arrange that for all the repo mirrors. There has to be some clever
way to get the same effect even when using a mirrorlist - like making
sure the mirrorlist itself is cached and always picking the same entry
so any client will use the same URL that the mirrormanger gave to the
first one that made a request. Of course you'd need a reasonable retry
mechanism to pick something else if this choice fails but I'd guess it
would be a big win in bandwidth use and load on the mirrors if it worked
most of the time to take advantage of existing local caches with no
modifications.
I just thought of a simple but gross solution for you.
http://mirrors.fedoraproject.org/mirrorlist?repo=fedora-$releasever&a...
It sounds like you are using a transparent proxy. Just redirect
mirrors.fedoraproject.org to localhost at another port and serve files
so the mirrorlist URL's hand back a single mirror of your choosing.
Warren Togami
wtogami(a)redhat.com
5:32 a.m.
On Thu, Jan 24, 2008 at 02:52:37AM -0500, Warren Togami wrote:
It sounds like you are using a transparent proxy. Just redirect
mirrors.fedoraproject.org to localhost at another port and serve files
so the mirrorlist URL's hand back a single mirror of your choosing.
Actually thats another way to handle the caching problem with most proxies
and static content servers. If you get
http://....../....static.file?random-16-digits
apache and most web servers will discard the query bits and the cache won't
be able to decide if the content is the same ;)
6:50 a.m.
On Thu, 2008-01-24 at 06:32 -0500, Alan Cox wrote:
On Thu, Jan 24, 2008 at 02:52:37AM -0500, Warren Togami wrote:
> It sounds like you are using a transparent proxy. Just redirect
> mirrors.fedoraproject.org to localhost at another port and serve files
> so the mirrorlist URL's hand back a single mirror of your choosing.
Actually thats another way to handle the caching problem with most proxies
and static content servers. If you get
http://....../....static.file?random-16-digits
apache and most web servers will discard the query bits and the cache won't
be able to decide if the content is the same ;)
"most"? I'm guessing you don't mean that by number of web servers ...
maybe by "most commonly deployed". Others can/do 301 redirect you to the
actual URL, or 404, or...
To mis-quote Henry Spencer:
Thou shalt foreswear, renounce, and abjure the vile heresy which
claimeth that "All the world's Apache-httpd", and have no
commerce with the benighted heathens who cling to this barbarous
belief, that the days of thy program may be long even though the
days of thy current web-server be short.
--
James Antill <james.antill(a)redhat.com>
Red Hat
8:12 a.m.
James Antill wrote:
>> It sounds like you are using a transparent proxy. Just redirect
>> mirrors.fedoraproject.org to localhost at another port and serve files
>> so the mirrorlist URL's hand back a single mirror of your choosing.
> Actually thats another way to handle the caching problem with most proxies
> and static content servers. If you get
>
> http://....../....static.file?random-16-digits
>
> apache and most web servers will discard the query bits and the cache won't
> be able to decide if the content is the same ;)
"most"? I'm guessing you don't mean that by number of web servers ...
maybe by "most commonly deployed". Others can/do 301 redirect you to the
actual URL, or 404, or...
Will all of them work correctly if you send 'Pragma: no-cache' or http
1.1 Cache-control: headers?
--
Les Mikesell
lesmikesell(a)gmail.com
8:05 a.m.
Warren Togami wrote:
Les Mikesell wrote:
>
> Interesting, but it still requires custom setup for any distro/version
> that the proxy admin would want to support. What I'd really like to
> happen is for yum to just always prefer the same URL when working
> through the same proxy so caching would work by default without
> needing to be aware of the cache content. This would work
> automatically if the target was a single site, RRDNS, or geo-ip
> managed DNS, but you probably can't arrange that for all the repo
> mirrors. There has to be some clever way to get the same effect even
> when using a mirrorlist - like making sure the mirrorlist itself is
> cached and always picking the same entry so any client will use the
> same URL that the mirrormanger gave to the first one that made a
> request. Of course you'd need a reasonable retry mechanism to pick
> something else if this choice fails but I'd guess it would be a big
> win in bandwidth use and load on the mirrors if it worked most of the
> time to take advantage of existing local caches with no modifications.
>
I just thought of a simple but gross solution for you.
http://mirrors.fedoraproject.org/mirrorlist?repo=fedora-$releasever&a...
It sounds like you are using a transparent proxy. Just redirect
mirrors.fedoraproject.org to localhost at another port and serve files
so the mirrorlist URL's hand back a single mirror of your choosing.
I think you are missing my point, which is that it would be a huge win
if yum automatically used typical existing caching proxies with no extra
setup on anyone's part, so that any number of people behind them would
get the cached packages without knowing about each other or that they
need to do something special to defeat the random URLs. I used to run a
number of centos3 boxes in several locations and it always worked nicely
to just:
http_proxy=http://my_proxy.domain:port yum update
pointing at a local squid because the mirrors used RRDNS so the URLs
were the same among the machines - and this would have happened
automatically with a transparent proxy or on machines set to use a
proxy by default as they must be in many locations. Since yum started
randomizing the requests with a mirrorlist, updates are a lot slower.
Maybe yum needs to do some tricks with cache control headers or
appending random arguments to ensure the repo data is fresh, but there
has to be some way to make it re-use packages already downloaded in a
local proxy cache without any local changes. We have several locations
where everyone in a large building has to use the same proxy to get out,
but the people who would be installing/updating their own linux boxes
would not know what anyone else is doing or be likely to coordinate the
choice of a URL if they had to change anything - and I'd guess that's a
common situation.
--
Les Mikesell
lesmikesell(a)gmail.com
8:29 a.m.
On Thu, 2008-01-24 at 08:05 -0600, Les Mikesell wrote:
I think you are missing my point, which is that it would be a huge
win
if yum automatically used typical existing caching proxies with no extra
setup on anyone's part, so that any number of people behind them would
get the cached packages without knowing about each other or that they
need to do something special to defeat the random URLs.
HTTP doesn't define a way to do this, much like the Pragma header
suggestion is a pretty bad abuse of HTTP ... suddenly the origin server
going down means you can't get the data from the proxy.
Please don't assume half of a solution, that never works well. What you
_actually want_ is:
"On my group my machines X, try not to download data more than once."
...at the ISP level this is solved by getting your mirror into mirror
manager directly.
At the personal level this is likely better solved by having something
like "Have a zero-conf like service to share pacakges across the local
network". There has even been some work to do this.
In neither case is "work around HTTPs design in yum" a good solution,
IMNSHO.
--
James Antill <james.antill(a)redhat.com>
Red Hat
10:50 a.m.
James Antill wrote:
> I think you are missing my point, which is that it would be a
huge win
> if yum automatically used typical existing caching proxies with no extra
> setup on anyone's part, so that any number of people behind them would
> get the cached packages without knowing about each other or that they
> need to do something special to defeat the random URLs.
HTTP doesn't define a way to do this, much like the Pragma header
suggestion is a pretty bad abuse of HTTP ...
It's worked for years in browsers. If you have a stale copy in an
intermediate cache, a ctl-refresh pulls a fresh one.
suddenly the origin server
going down means you can't get the data from the proxy.
Only if the DNS for the origin server points to a single dead IP or the
client is too dumb to retry the alternates. Even IE can handle this, so
it can't be all that difficult... And yum could have some other retry
strategy too.
Please don't assume half of a solution, that never works well.
What you
_actually want_ is:
"On my group my machines X, try not to download data more than once."
Add in "by default, with no prearrangement other than configuring a
proxy cache to handle large files" and you are close. The other piece
is that the same solution should work for all duplicate http downloads.
...at the ISP level this is solved by getting your mirror into
mirror
manager directly.
Per-disto, per location setup just isn't going to happen.
At the personal level this is likely better solved by having
something
like "Have a zero-conf like service to share pacakges across the local
network". There has even been some work to do this.
What does 'local' mean in this context? Many of the machines sharing
local proxies would not be in broadcast or multicast range of each other.
In neither case is "work around HTTPs design in yum" a
good solution,
IMNSHO.
I'd rather call it "using existing infrastructure and protocols
intelligently" - instead of cluttering everyone's caches with randomized
URLs to get duplicate files.
--
Les Mikesell
lesmikesell(a)gmail.com
11:20 a.m.
On Thu, 2008-01-24 at 10:50 -0600, Les Mikesell wrote:
James Antill wrote:
>> I think you are missing my point, which is that it would be a huge win
>> if yum automatically used typical existing caching proxies with no extra
>> setup on anyone's part, so that any number of people behind them would
>> get the cached packages without knowing about each other or that they
>> need to do something special to defeat the random URLs.
>
> HTTP doesn't define a way to do this, much like the Pragma header
> suggestion is a pretty bad abuse of HTTP ...
It's worked for years in browsers. If you have a stale copy in an
intermediate cache, a ctl-refresh pulls a fresh one.
Sure, the _user_ has a single URL to work with and can make
semi-intelligent decisions with only themselves to blame. Not quite the
same as a program with multiple URLs to the same data.
Now if you wanted to add ETag support in various places, patches are
very likely to be accepted, then any program could make intelligent
decisions.
> In neither case is "work around HTTPs design in yum"
a good solution,
> IMNSHO.
I'd rather call it "using existing infrastructure and protocols
intelligently" - instead of cluttering everyone's caches with randomized
URLs to get duplicate files.
So you think the best thing to do is remove mirrorlist entirely, and
just rely on proxies ... you are obviously free to your opinion, and you
can do that today.
--
James Antill <james.antill(a)redhat.com>
Red Hat
12:07 p.m.
James Antill wrote:
>> In neither case is "work around HTTPs design in yum" a good solution,
>> IMNSHO.
> I'd rather call it "using existing infrastructure and protocols
> intelligently" - instead of cluttering everyone's caches with randomized
> URLs to get duplicate files.
So you think the best thing to do is remove mirrorlist entirely, and
just rely on proxies ...
No, I think the best thing would be a mechanism that would make any set
of yum's behind the same proxy use the cached mirrorlist and always pick
the same first choice from it, with some appropriate alternates to retry
only if the first choice doesn't respond. This doesn't 'rely' on the
proxy but it will reduce load on the mirrors by the number of machines
that happen to share a working proxy as well as speeding up all but the
first update.
you are obviously free to your opinion, and you
can do that today.
I can't do it usefully since I don't know what URL anyone else who is so
inclined might have chosen. Or, for that matter, who else behind the
same proxy might be using fedora.
--
Les Mikesell
lesmikesell(a)gmail.com
12:06 p.m.
Les Mikesell wrote:
I can't do it usefully since I don't know what URL anyone else who is so
inclined might have chosen. Or, for that matter, who else behind the
same proxy might be using fedora.
If you already run your own site's proxy, why not run another proxy but
this one reverse in order to act as your site's Fedora mirror? Then you
can solve your yum mirrorlist problem TODAY.
Warren Togami
wtogami(a)redhat.com
3:12 p.m.
Warren Togami <wtogami(a)redhat.com> writes:
If you already run your own site's proxy, why not run another
proxy
but this one reverse in order to act as your site's Fedora mirror?
Then you can solve your yum mirrorlist problem TODAY.
That doesn't solve the problem of getting the machines to pick that
mirror by default, without having to configure each one.
/Benny
3:11 p.m.
Benny Amorsen wrote:
Warren Togami <wtogami(a)redhat.com> writes:
> If you already run your own site's proxy, why not run another proxy
> but this one reverse in order to act as your site's Fedora mirror?
> Then you can solve your yum mirrorlist problem TODAY.
That doesn't solve the problem of getting the machines to pick that
mirror by default, without having to configure each one.
/Benny
*SMACK*
Warren
4:13 a.m.
tor 2008-01-24 klockan 09:29 -0500 skrev James Antill:
At the personal level this is likely better solved by having
something
like "Have a zero-conf like service to share pacakges across the local
network". There has even been some work to do this.
Wild idea:
A yum plugin that short-circuits any downloading where the expected
checksum of the data is known beforehand. It would first ask on the
local network for the file, providing:
* The baseurl or mirrorlist url.
* The path relative to the root of the repo.
* The name of the file.
* The expected checksum.
If no yum cache service is available on the network, no such service has
a suitable file or the request times out (a short timeout, 1 second
perhaps), the plugin would fall back to the baseurl or mirrors as usual.
It would check both the checksum of the file and the GPG key (unless
you've disabled that) so security-wise it shouldn't be much worse than
just downloading the file from somewhere via FTP/HTTP.
For the client, just install/enable the plugin and you're done. The
server part could just serve data out of /var/cache/yum if it's there
and the repo has been marked as public in the yum repo config. Maybe
just run it on all hosts as well, marking all fedora repos as public by
default?
/abo
8:30 a.m.
Les Mikesell wrote:
Warren Togami wrote:
> Les Mikesell wrote:
>>
>> Interesting, but it still requires custom setup for any
>> distro/version that the proxy admin would want to support. What I'd
>> really like to happen is for yum to just always prefer the same URL
>> when working through the same proxy so caching would work by default
>> without needing to be aware of the cache content. This would work
>> automatically if the target was a single site, RRDNS, or geo-ip
>> managed DNS, but you probably can't arrange that for all the repo
>> mirrors. There has to be some clever way to get the same effect even
>> when using a mirrorlist - like making sure the mirrorlist itself is
>> cached and always picking the same entry so any client will use the
>> same URL that the mirrormanger gave to the first one that made a
>> request. Of course you'd need a reasonable retry mechanism to pick
>> something else if this choice fails but I'd guess it would be a big
>> win in bandwidth use and load on the mirrors if it worked most of the
>> time to take advantage of existing local caches with no modifications.
>>
>
> I just thought of a simple but gross solution for you.
>
>
http://mirrors.fedoraproject.org/mirrorlist?repo=fedora-$releasever&a...
>
>
> It sounds like you are using a transparent proxy. Just redirect
> mirrors.fedoraproject.org to localhost at another port and serve files
> so the mirrorlist URL's hand back a single mirror of your choosing.
I think you are missing my point, which is that it would be a huge win
if yum automatically used typical existing caching proxies with no extra
setup on anyone's part, so that any number of people behind them would
get the cached packages without knowing about each other or that they
need to do something special to defeat the random URLs. I used to run a
number of centos3 boxes in several locations and it always worked nicely
to just:
http_proxy=http://my_proxy.domain:port yum update
pointing at a local squid because the mirrors used RRDNS so the URLs
were the same among the machines - and this would have happened
automatically with a transparent proxy or on machines set to use a
proxy by default as they must be in many locations. Since yum started
randomizing the requests with a mirrorlist, updates are a lot slower.
This has nothing to do with yum, it is the disto there decides how to
give access to mirrors, yum just does what the distro repo contig tell
it to do.
"Don't blame the postman for the bad news" :)
Maybe yum needs to do some tricks with cache control headers or
appending random arguments to ensure the repo data is fresh, but there
has to be some way to make it re-use packages already downloaded in a
local proxy cache without any local changes. We have several locations
where everyone in a large building has to use the same proxy to get out,
but the people who would be installing/updating their own linux boxes
would not know what anyone else is doing or be likely to coordinate the
choice of a URL if they had to change anything - and I'd guess that's a
common situation.
This thread is upside down IMHO.
The proxy don't do the right thing, so let us redesign yum, to fix the
problems with the proxy. I seems to be the wrong approach IMHO.
Tim "Proxies is nothing but trouble" :)
10:31 a.m.
Tim Lauridsen wrote:
This has nothing to do with yum, it is the disto there decides how to
give access to mirrors, yum just does what the distro repo contig tell
it to do.
Yes, as I said, the CentOS 3.x setup was just great where they used
RRDNS or a geo-ip enabled dns server to keep the URL's consistent.
However I don't think it is realistic to expect all mirrors to be able
to work that way - and if you don't have dynamic checking on the DNS
side (like an F5 Global Load Balancer) or clients that know to retry
multiple IPs in the returned list, you can end up stuck trying to access
a dead site.
"Don't blame the postman for the bad news" :)
Even a postman is going to have trouble when you randomize the addresses...
> Maybe yum needs to do some tricks with cache control headers or
> appending random arguments to ensure the repo data is fresh, but there
> has to be some way to make it re-use packages already downloaded in a
> local proxy cache without any local changes. We have several
> locations where everyone in a large building has to use the same proxy
> to get out, but the people who would be installing/updating their own
> linux boxes would not know what anyone else is doing or be likely to
> coordinate the choice of a URL if they had to change anything - and
> I'd guess that's a common situation.
This thread is upside down IMHO.
The proxy don't do the right thing, so let us redesign yum, to fix the
problems with the proxy. I seems to be the wrong approach IMHO.
Yum is the piece you can fix, so making it do something reasonable in
the presence of proxies sounds like not only the right approach but the
only one possible. Unless, of course, you'd like to re-invent caching
proxies to work the way you like and replace all of the existing ones.
Tim "Proxies is nothing but trouble" :)
Note that if yum acted intelligently with proxies, the whole concept of
mirror management could be replaced by appropriately placed squid (or
similar) proxies with no configuration/maintenance other than setting
them to cache large files and restricting public ones to certain
targets. Then if yum noticed that no local proxy was being used, it
could use some mirrorlist-like mechanism to find one nearby and use it
explicitly, eliminating any need for a reverse-proxy setup on the proxy
server side.
--
Les Mikesell
lesmikesell(a)gmail.com
11:21 a.m.
Les Mikesell wrote:
Even a postman is going to have trouble when you randomize the addresses...
The real solution that you need is within MirrorManager. Perhaps today
MirrorManager will not do what you need. Let us define the
functionality that you do need:
"Users should be able to define their own site-local netblock and
associate it with a preferred public mirror. That preferred public
mirror will then *always* show up as the first mirror, followed by
others in the region."
https://fedorahosted.org/mirrormanager
Want to help write this option into MirrorManager? The code is open.
> Tim "Proxies is nothing but trouble" :)
The yum developers have an erroneous assumption that only "broken"
proxies are at fault for yum problems. I have described at the
beginning of this thread that this assumption is false. *Any* proxy
server that does caching will occasionally cause the partial sync yum
failure, or resigned RPMS will cause a failure.
Fortunately, the yum developers were already thinking about using unique
(probably SHA1) names of repodata as of yum-3.2.9 for other reasons.
This should solve the common partial sync failure in Fedora 9+.
However, there remains no good solution for the resigned RPMS case other
than falling back to the next mirror. Fortunately this is a rare
problem with our stable releases, it mainly effects our development and
testers.
Note that if yum acted intelligently with proxies, the whole concept of
mirror management could be replaced by appropriately placed squid (or
similar) proxies with no configuration/maintenance other than setting
them to cache large files and restricting public ones to certain
targets. Then if yum noticed that no local proxy was being used, it
could use some mirrorlist-like mechanism to find one nearby and use it
explicitly, eliminating any need for a reverse-proxy setup on the proxy
server side.
I can't begin to respond to this because of how misguided it is.
Warren Togami
wtogami(a)redhat.com
12:49 p.m.
Warren Togami wrote:
> Even a postman is going to have trouble when you randomize the
> addresses...
>
The real solution that you need is within MirrorManager. Perhaps today
MirrorManager will not do what you need. Let us define the
functionality that you do need:
"Users should be able to define their own site-local netblock and
associate it with a preferred public mirror. That preferred public
mirror will then *always* show up as the first mirror, followed by
others in the region."
Anything that requires coordination among any or all of the proxy
manager or fedora users that happen to be behind the proxy just isn't
going to happen here and I'm having trouble imagining an organization
where it would. Can yum make this happen automatically, or without
intervention, can the mirrormanager list always return the same first
choice when requested from the same address? If that already happens,
then maybe things aren't as bad as I thought.
https://fedorahosted.org/mirrormanager
Want to help write this option into MirrorManager? The code is open.
>> Tim "Proxies is nothing but trouble" :)
The yum developers have an erroneous assumption that only "broken"
proxies are at fault for yum problems. I have described at the
beginning of this thread that this assumption is false. *Any* proxy
server that does caching will occasionally cause the partial sync yum
failure, or resigned RPMS will cause a failure.
I'm missing something here. A cache is only going to resend what it got
the first time. If the first user got the right thing, so will the
second. If the first user got something wrong, don't blame the cache
for it.
Fortunately, the yum developers were already thinking about using
unique
(probably SHA1) names of repodata as of yum-3.2.9 for other reasons.
This should solve the common partial sync failure in Fedora 9+.
This should be needed to fix the first user's problem as much as
subsequent ones.
> Note that if yum acted intelligently with proxies, the whole
concept
> of mirror management could be replaced by appropriately placed squid
> (or similar) proxies with no configuration/maintenance other than
> setting them to cache large files and restricting public ones to
> certain targets. Then if yum noticed that no local proxy was being
> used, it could use some mirrorlist-like mechanism to find one nearby
> and use it explicitly, eliminating any need for a reverse-proxy setup
> on the proxy server side.
>
I can't begin to respond to this because of how misguided it is.
I think anything that needs individual per-distro, per-version,
per-location management just to act like a working cache is even more
misguided, but maybe there some other approach that would work. Perhaps
you could ask the squid expert if a slightly more intelligent client
could eliminate that need and make cascaded proxies work better.
--
Les Mikesell
lesmikesell(a)gmail.com
12:52 p.m.
Les Mikesell wrote:
> The yum developers have an erroneous assumption that only
"broken"
> proxies are at fault for yum problems. I have described at the
> beginning of this thread that this assumption is false. *Any* proxy
> server that does caching will occasionally cause the partial sync yum
> failure, or resigned RPMS will cause a failure.
I'm missing something here. A cache is only going to resend what it got
the first time. If the first user got the right thing, so will the
second. If the first user got something wrong, don't blame the cache
for it.
> Fortunately, the yum developers were already thinking about using
> unique (probably SHA1) names of repodata as of yum-3.2.9 for other
> reasons. This should solve the common partial sync failure in Fedora 9+.
This should be needed to fix the first user's problem as much as
subsequent ones.
Read the original post of this thread again. You completely missed the
point of that post, which was NOT your problem at all.
Warren
1:23 p.m.
Le jeudi 24 janvier 2008 à 12:49 -0600, Les Mikesell a écrit :
I'm missing something here. A cache is only going to resend what
it got
the first time. If the first user got the right thing, so will the
second.
No it won't. Common failures are:
- T0: user A updates foo and in the course of doing it populates the
proxy cache with yum T0 metadata and foo rpm only.
- T1: upstream updates repo changing bar rpm version. Metadata is
regenerated with the same filenames as before. For the proxy its T0 copy
is still valid till the expiration delay it got in http headers the
first time.
- T2: user B wants to update bar. The proxy happily serves yum the
cached T0 metadata (since yum didn't request any special treatment for
it). It tells yum the T0 bar is available. Unfortunately it's not - user
A didn't download it so it's not in the cache, and upstream moved to a
new version
- T0: user A updates foo and in the course of doing it populates the
proxy cache with yum T0 metadata and foo rpm
- T1: upstream signs foo rpm and regenerates the repo. Metadata is
regenerated with the same filenames as before. For the proxy its T0 copy
is still valid till the expiration delay it got in http headers the
first time.
- T2: proxy cache is contended, it evicts foo rpm since it takes a lot
of room. T0 metadata is small so it's retained for now
- T3: user B wants to update foo. The proxy happily serves yum the
cached T0 metadata (since yum didn't request any special treatment for
it). It tells yum foo is available with T0 checksum. Unfortunately it's
not - T0 foo rpm was evicted from the cache, when user B requests the
same URL as user A it gets T1 signed foo with a new checksum.
(you can invert this case with a proxy which strategy is to keep big
files which are expensive to download, and drop small metadata)
In all those cases the proxy worked as designed and was fooled by
yum/createrepo using the same URLs for different objects.
A proxy will always serve you a set of files as they existed on the
original location at some time. But there is zero insurance they were
taken from this location at the same time, so you *must* take care new
content is created with new filenames, or your web server tells the
proxy when old data will be replaced by new one (via expires), or your
system is able to cope with a mix of files from different times.
--
Nicolas Mailhot
1:49 p.m.
Nicolas Mailhot wrote:
> I'm missing something here. A cache is only going to resend what it got
> the first time. If the first user got the right thing, so will the
> second.
No it won't. Common failures are:
- T0: user A updates foo and in the course of doing it populates the
proxy cache with yum T0 metadata and foo rpm only.
- T1: upstream updates repo changing bar rpm version. Metadata is
regenerated with the same filenames as before. For the proxy its T0 copy
is still valid till the expiration delay it got in http headers the
first time.
- T2: user B wants to update bar. The proxy happily serves yum the
cached T0 metadata (since yum didn't request any special treatment for
it). It tells yum the T0 bar is available. Unfortunately it's not - user
A didn't download it so it's not in the cache, and upstream moved to a
new version
Can't the T1->T2 transition happen while user A is in mid-update? I'm
fairly sure I've seen something like that without any proxies involved.
- T0: user A updates foo and in the course of doing it populates the
proxy cache with yum T0 metadata and foo rpm
- T1: upstream signs foo rpm and regenerates the repo. Metadata is
regenerated with the same filenames as before. For the proxy its T0 copy
is still valid till the expiration delay it got in http headers the
first time.
- T2: proxy cache is contended, it evicts foo rpm since it takes a lot
of room. T0 metadata is small so it's retained for now
- T3: user B wants to update foo. The proxy happily serves yum the
cached T0 metadata (since yum didn't request any special treatment for
it). It tells yum foo is available with T0 checksum. Unfortunately it's
not - T0 foo rpm was evicted from the cache, when user B requests the
same URL as user A it gets T1 signed foo with a new checksum.
(you can invert this case with a proxy which strategy is to keep big
files which are expensive to download, and drop small metadata)
In all those cases the proxy worked as designed and was fooled by
yum/createrepo using the same URLs for different objects.
Again, I think you'll still have the same issues if user A's update
spans any of these arbitrary Tx designations - at least the ones where
you've got metadata and the rpms are changed before you get them.
A proxy will always serve you a set of files as they existed on the
original location at some time. But there is zero insurance they were
taken from this location at the same time, so you *must* take care new
content is created with new filenames, or your web server tells the
proxy when old data will be replaced by new one (via expires), or your
system is able to cope with a mix of files from different times.
But you are doing things that aren't guaranteed to work in the first
place. All the cache does is expand the window for breakage a bit.
--
Les Mikesell
lesmikesell(a)gmal.com
2:04 p.m.
Le jeudi 24 janvier 2008 à 13:49 -0600, Les Mikesell a écrit :
Can't the T1->T2 transition happen while user A is in
mid-update? I'm
fairly sure I've seen something like that without any proxies involved.
...
Again, I think you'll still have the same issues if user A's
update
spans any of these arbitrary Tx designations - at least the ones where
you've got metadata and the rpms are changed before you get them.
...
But you are doing things that aren't guaranteed to work in the
first
place. All the cache does is expand the window for breakage a bit.
You're perfectly right. Smart web clients know the internet can shift
under their feet and refresh files in case of problems instead of
failing horribly on users (in the browser case you have a big refresh
button for this reason). Though smart web designers know client
refreshes are expensive and limit the refresh needs by avoiding to reuse
the same filenames for different stuff.
--
Nicolas Mailhot
2:28 p.m.
Nicolas Mailhot wrote:
> But you are doing things that aren't guaranteed to work in the first
> place. All the cache does is expand the window for breakage a bit.
You're perfectly right. Smart web clients know the internet can shift
under their feet and refresh files in case of problems instead of
failing horribly on users (in the browser case you have a big refresh
button for this reason). Though smart web designers know client
refreshes are expensive and limit the refresh needs by avoiding to reuse
the same filenames for different stuff.
Getting a bit off the original topic here, but the "other" thing I've
always wished yum could do is "repeatable" updates. That is, the
ability to update one machine, test some things, then update another and
get only the same set of changes even if some newer packages had been
subsequently added to the repos. Currently I believe the only way to do
this is to mirror the entire set of repos in each state that you might
want to re-use. Perhaps some transactioning info could fix both things
at once.
--
Les Mikesell
lesmikesell(a)gmail.com
2:30 p.m.
On Thu, 2008-01-24 at 14:28 -0600, Les Mikesell wrote:
Nicolas Mailhot wrote:
>
>> But you are doing things that aren't guaranteed to work in the first
>> place. All the cache does is expand the window for breakage a bit.
>
> You're perfectly right. Smart web clients know the internet can shift
> under their feet and refresh files in case of problems instead of
> failing horribly on users (in the browser case you have a big refresh
> button for this reason). Though smart web designers know client
> refreshes are expensive and limit the refresh needs by avoiding to reuse
> the same filenames for different stuff.
Getting a bit off the original topic here, but the "other" thing I've
always wished yum could do is "repeatable" updates. That is, the
ability to update one machine, test some things, then update another and
get only the same set of changes even if some newer packages had been
subsequently added to the repos. Currently I believe the only way to do
this is to mirror the entire set of repos in each state that you might
want to re-use. Perhaps some transactioning info could fix both things
at once.
If a newer set of pkgs has been added to the repo you can specify the
one you want to install using the complete ver-rel string.
If the older packages are no longer available there's not much yum can
do.
You can pass around sets of specific things to do from machine to
machine using a 'yum shell' script.
-sv
3:07 p.m.
seth vidal wrote:
If a newer set of pkgs has been added to the repo you can specify the
one you want to install using the complete ver-rel string.
I suppose there is some variation of rpm -qa .. that would build the
whole installed list to reproduce. Is that a reasonable thing to pass
to yum on the command line?
--
Les Mikesell
lesmikesell(a)gmail.com
3:16 p.m.
On Thu, 2008-01-24 at 15:07 -0600, Les Mikesell wrote:
seth vidal wrote:
>
> If a newer set of pkgs has been added to the repo you can specify the
> one you want to install using the complete ver-rel string.
I suppose there is some variation of rpm -qa .. that would build the
whole installed list to reproduce. Is that a reasonable thing to pass
to yum on the command line?
you can do that, yah. There are prettier ways - but that could work.
-sv
2:44 p.m.
On Thu, 2008-01-24 at 14:28 -0600, Les Mikesell wrote:
Getting a bit off the original topic here, but the "other"
thing I've
always wished yum could do is "repeatable" updates. That is, the
ability to update one machine, test some things, then update another and
get only the same set of changes even if some newer packages had been
subsequently added to the repos. Currently I believe the only way to do
this is to mirror the entire set of repos in each state that you might
want to re-use. Perhaps some transactioning info could fix both things
at once.
The data isn't there to make this possible, atm., in Fedora:
% sqlite3 /var/cache/yum/updates/primary.sqlite
SELECT name,count(*) AS num FROM packages GROUP BY name,arch HAVING
num > 1;
...and the primary reason for that is the the metadata download size
would explode, but we do have features like that we are looking at (for
situations where the data is there).
But again, asking on this mailing list is not the best method of
getting new features or bug fixes.
I'd also note that a bunch of the utilities in yum-utils make creating
complete local mirrors at a specific point in time, easier than you
might think.
--
James Antill <james.antill(a)redhat.com>
Red Hat
12:43 p.m.
Le jeudi 24 janvier 2008 à 15:30 +0100, Tim Lauridsen a écrit :
This thread is upside down IMHO.
The proxy don't do the right thing, so let us redesign yum, to fix the
problems with the proxy. I seems to be the wrong approach IMHO.
No, it's "yum metadata is proxy-unfriendly, someone finally acknowledged
this, let's fix yum and createrepo to play well with existing internet
infrastructure".
--
Nicolas Mailhot
9:33 a.m.
On Wed, 2008-01-23 at 23:45 -0500, Warren Togami wrote:
On Wed, 23 Jan 2008 21:28:01 -0600, Les Mikesell
<lesmikesell(a)gmail.com>
wrote:
> Warren Togami wrote:
>
>> Yum and Proxy Caches: Current Dangers
>> =====================================
>> Users may be using proxy servers in 3 (or more) ways:
>
> Is there a reasonable way to make yum automatically use files cached in
> these local proxies when run by multiple users that share the proxy but
> don't know about each other? Picking some random mirror for each will
> defeat the caching since they will have different URLs to get the same
> file.
http://fedoraproject.org/wiki/Infrastructure/Mirroring/SiteLocalMirrors
This page needs updating, but it sort of describes what you need to do.
Use your Fedora account to login to MirrorManager and add a private mirror
for yourself along with a site-local netblock in CIDR notation so the
mirror master knows to serve to you when yum clients come from your
network. After a few hours, mirror lists will serve your mirror first,
then random other mirrors in your country/region.
Example, I have InstantMirror running on my Buffalo Linkstation (hard drive
+ ethernet running embedded Linux) at home. I gave that box a private IP
address and pointed a sub-domain name at that (so I could use Apache
name-based VirtualHost for InstantMirror). I then added to MirrorManager
"Warren's Private Home Mirror" which only I can see when I login to the
management interface. In both cases it is REALLY handy when all of your
local clients automatically begin using your local mirror.
Oh, there might be one more complication. You may need to checkout the
sources for MirrorManager and run the script that reports to MirrorManager
that your mirror is populated. I think mdomsch mentioned that the checkbox
on the MirrorManager interface isn't yet working. Matt, what's the status
of this?
Hopefully that last wrinkle should be gone soon.
Warren Togami
wtogami(a)redhat.com
Slightly off topic but, could you elaborate on the MirrorManager setup
for this? I've got a local mirror at home and would like to set up
automatic usage from my local hosts on my private NATed network. Did you
set up a single host netblock for your public ip address? I have cable
internet and my public ip stays the same for several months at a time
but it does change. Is there any way to have a more dynamic set up, like
a netblock on a domain name so I only have to change my DNS and
mirrormanager picks that up?
Thanks,
tjb
--
=======================================================================
| Thomas Baker email: tjb(a)unh.edu |
| Systems Programmer |
| Research Computing Center voice: (603) 862-4490 |
| University of New Hampshire fax: (603) 862-1761 |
| 332 Morse Hall |
| Durham, NH 03824 USA http://wintermute.sr.unh.edu/~tjb |
=======================================================================
5:17 a.m.
On Wed, Jan 23, 2008 at 09:41:40PM -0500, Warren Togami wrote:
primary-1201140584.sqlite.bz2
This would be an elegant solution, but will it be possible for us to
migrate to because older clients wouldn't be able to handle it?
Yes - keep the "current" sqlite database in two places (or linked if the
server is set up to allow that). Old clients will fetch the db blindly new
ones will fetch the versioned ones
6:41 a.m.
On Wed, 2008-01-23 at 21:41 -0500, Warren Togami wrote:
I just had an in-depth discussion with Henrik Nordström of the Squid
project about how HTTP mirrors and the yum tool itself could be improved
to safely handle proxy caches. He gave me lots of good advice about how
HTTP mirrors can be configured for cache safety, Squid can be configured
for yum metadata cache safety, and yum itself can be improved to be more
robust in dealing with proxy caches.
(It turns out that Henrik is an avid Fedora user, and I might have
convinced him to come onboard the Fedora Project to contribute another
useful tool and become co-maintainer of his own package. It would be an
honor to have him onboard as a Fedora Developer. =)
You might have had a small discussion on #yum then, as any of the
regulars there know the answers to all of your questions.
Yum and Proxy Caches: Current Dangers
=====================================
Users may be using proxy servers in 3 (or more) ways:
1) Many users today are behind a transparent proxy cache, either
instituted by their ISP, school, or business network.
2) Other users might have Internet access *only* through a proxy server.
3) Other users might be using a reverse proxy server on their local
network as a caching yum mirror.
There are two cases where yum has problems with proxy caches:
1) A RPM package changes content without changing filename. This
usually happens only in instances where a package was pushed unsigned
then was later signed. A simple workaround within yum is discussed
later in this mail.
2) yum currently has problems with proxy caches due to common cases
where metadata can become partially out of sync. This happens because
repomd.xml is grabbed often while other repodata files are grabbed less
often. repomd.xml is then checked for origin "freshness" more often.
When repodata changes on the origin, repomd.xml is refreshed on the
cache before other repodata files. yum clients seeing the new
repomd.xml but old primary.sqlite.bz2 error out.
#2 is worked around as good as is possible, in the upcoming 3.2.9, in
that yum will basically create a transaction over the repomd.xml and the
metadata itself. If you use mdpolicy=group:all ... this will always
work, the downside is that you'll need to download all of the metadata
so the default is not that.
Ideal Solution for #2 Partial Repodata Sync Problem
===================================================
Henrik highly suggests using versioned repodata files as the ideal
solution to this problem. This way caches can serve repodata without
fear of the sync problem, and also without querying the origin server
upon every client download. repomd.xml would contain changing filenames
perhaps with timestamp or something in their filenames.
i.e.
primary-1201140584.sqlite.bz2
This would be an elegant solution, but will it be possible for us to
migrate to because older clients wouldn't be able to handle it?
I'm guessing not, so here are other less efficient but workable solutions.
We've discussed this and think this is probably the best solution, but:
1. Don't use timestamps, use the sha1 of the file, because then
multiple createrepo's runs will always create the same filenames.
2. This requires work inside yum as atm. yum doesn't do any cleanup on
it's metadata downloads so /var/cache/yum would grow without bound
(although "yum clean ..." will work).
...we can fix #2 for 3.2.9, so we could do this in Fedora 9 onwards.
"Cache-Control: max-age=0"
==========================
This HTTP header directive can be either in the request or response.
This instructs the proxy cache server to always query the origin HTTP
server to check if the requested file has changed. It compares the
origin's reported Last-Modified or ETag to what Squid knows in its own
cache.
This means that each and every request for repodata/* files will trigger
a query to the origin server. This is a relatively quick operation and
an acceptable compromise if we cannot make repodata filenames versioned.
This is a horrible hack, IMO, and I can pretty much guarantee that not
all of the mirrors will do this. If it was possible for us to control
all of the mirrors then we could just require them all to setup ETags
and use that ... but again, I think that's hoping for way too much.
[...]
Yum and "X-Cache: HIT"
======================
If you use wget --server-response and a target file, you see the raw
HTTP headers of that request. If the file is already cached, you see a
HTTP header like below:
X-Cache: HIT from proxyserver.example.com
Proposal:
Improve yum with the following download logic:
IF (a downloaded repodata/* file doesn't match the repomd.xml checksum
OR a downloaded RPM doesn't match the expected checksum)
AND "X-Cache: HIT from" was in its HTTP header
THEN download it again with URLGrabber option: http_headers =
(('Pragma', 'no-cache')
This should solve the case where RPM files legitimately change contents
without changing filenames, like RPM signing. This also correctly does
NOT trigger additional downloads upon other errors like corrupted files.
You'd have to do this change inside URLgrabber itself, as by the time
yum could react to it URLgrabber would already have decided to remove
that mirror from it's list and moved on.
--
James Antill <james.antill(a)redhat.com>
Red Hat
8:53 a.m.
On Wed, 2008-01-23 at 21:41 -0500, Warren Togami wrote:
I just had an in-depth discussion with Henrik Nordström of the Squid
project about how HTTP mirrors and the yum tool itself could be improved
to safely handle proxy caches. He gave me lots of good advice about how
HTTP mirrors can be configured for cache safety, Squid can be configured
for yum metadata cache safety, and yum itself can be improved to be more
robust in dealing with proxy caches.
"Cache-Control: max-age=0"
==========================
This HTTP header directive can be either in the request or response.
snip
3) yum can always include the HTTP directive in its request for
repodata/* files. Can we make this the default in future versions of
yum? I personally don't see a drawback (unless repodata becomes
versioned, then we don't want this.)
I recommend this strongly as an option (instead of 'always') but I'd
love to be able to set this network wide (yum.conf over NIS? just
kidding).
I currently have numerous problems with this because all my systems sit
behind a squid cache used only for caching yum packages, but we have in
internal repo here, and if I push changes to it 'too fast' I get in
trouble. I've created a shell script which 'invalidates' the squid
cache for repos found in /etc/yum.repos.d using 'wget --no-cache' .
Script attached for instructional purposes.
Another problem not mentioned is the non-stable nature of mirrorlist.
If you use one mirror on one update, and another on the next, the cache
cannot help.
This requires some non-trivial (in terms of setup cost) fiddling with
the yum config on every host to change the way mirror (or baseurl)
selection works, and is brittle w.r.t future change.
David
5934
days inactive
5937
days old
42 comments
15 participants
participants (15)
-
Alan Cox -
Alexander Boström -
Benny Amorsen -
Chuck Anderson -
David Mansfield -
James Antill -
James Antill -
Jesse Keating -
Les Mikesell -
Nicolas Mailhot -
Seth Vidal -
Thomas J. Baker -
Tim Lauridsen -
Warren Togami -
Warren Togami