Reository wisdom (not lookin' fer howtos)

[Brian Fahrlander]

    Well, my research into LDAP for authentication is showing GREAT
benefits over NIS, and I have a third machine (for the home) installing
upstairs as I type this. This one's delay due to a dirty CDROM, and I
have to go 'hit' it once in a while.  An FTP install would probably be
more accurate, if the bandwidth is available.

    I'm curious about repositories; since these could be massive groups
of PCs (say, hundreds or thousands) it should make sense to maintain a
separate repo (yum, presumably) on a server somewhere.

    My question is, with all the questions about repo stability and
such, what's the best way to go about it, strategically? I'm talking
about whether I should find a local college to sync with, get it
straight from fedora.us, or exactly what seems to go into making a nice,
stable mirror?

-- 
------------------------------------------------------------------------
Brian Fahrländer                  Christian, Conservative, and Technomad
Evansville, IN                                 http://www.fahrlander.net
ICQ 5119262
AIM: WheelDweller
------------------------------------------------------------------------

[ne...]

On Aug 17, 2004 at 08:14, Brian Fahrlander in a soothing rage wrote:
[...]
>     My question is, with all the questions about repo stability and
> such, what's the best way to go about it, strategically? I'm talking
> about whether I should find a local college to sync with, get it
> straight from fedora.us, or exactly what seems to go into making a nice,
> stable mirror? 
Do what I do. Look for something close to to you that is fast. Sync 
with that. Keep it secret.

N.Emile...
-- 
Registered Linux User # 125653 (http://counter.li.org)
Switch to: http://www.speakeasy.net/refer/190653
"Go to Heaven for the climate, Hell for the company."
-- Mark Twain
 10:57:23  up 50 days,  4:12,  3 users,  load average: 0.00, 0.00, 0.00

[Ed K.]

On Tue, 17 Aug 2004, Brian Fahrlander wrote:
>     Well, my research into LDAP for authentication is showing GREAT
>  benefits over NIS, and I have a third machine (for the home) installing
>  upstairs as I type this. This one's delay due to a dirty CDROM, and I
>  have to go 'hit' it once in a while.  An FTP install would probably be
>  more accurate, if the bandwidth is available.
>
>     I'm curious about repositories; since these could be massive groups
>  of PCs (say, hundreds or thousands) it should make sense to maintain a
>  separate repo (yum, presumably) on a server somewhere.
>
>     My question is, with all the questions about repo stability and
>  such, what's the best way to go about it, strategically? I'm talking
>  about whether I should find a local college to sync with, get it
>  straight from fedora.us, or exactly what seems to go into making a nice,
>  stable mirror?
>  
I know everyone wants to create their own local repository. But may I
suggest a different approach. I have many clients with many computers, I
wouldn't want to create a local repository for each client, so I created a
single repository that when used with a proxy server acts like a local
repository.

Have a look:
http://www.edebris.com/fedora.redhat/mirror/
http://www.edebris.com/fedora.us/mirror/

If you configure all of your computers to use the same proxy server, then
you will get the speed of a local yum repository, without configuring
anything but the proxy server you should have anyway.

Note, that this work only because of the proper use of the expires and
If-Modified-Since information sent with the files.

ed

[Brian Fahrlander]

On Tue, 2004-08-17 at 10:44, ed(a)hp.uab.edu wrote:

>  I know everyone wants to create their own local repository. But may
> I
>  suggest a different approach. I have many clients with many computers, I
>  wouldn't want to create a local repository for each client, so I created a
>  single repository that when used with a proxy server acts like a local
>  repository. 
    Well, this isn't for me...this is for large corporations where it
makes a lot of sense to not swamp the main mirrors during the overnight.

>  Have a look:
>  http://www.edebris.com/fedora.redhat/mirror/
>  http://www.edebris.com/fedora.us/mirror/ 
    Looks slick.

>  If you configure all of your computers to use the same proxy server,
> then
>  you will get the speed of a local yum repository, without configuring
>  anything but the proxy server you should have anyway.
>  
>  Note, that this work only because of the proper use of the expires and
>  If-Modified-Since information sent with the files. 
    I guess I'll just do whatever the main mirrors are doing; this isn't
a small project; and it needs to be done right.

-- 
------------------------------------------------------------------------
Brian Fahrländer                  Christian, Conservative, and Technomad
Evansville, IN                                 http://www.fahrlander.net
ICQ 5119262
AIM: WheelDweller
------------------------------------------------------------------------

[Rodolfo J. Paiz]

On Tue, 2004-08-17 at 13:22, Brian Fahrlander wrote:
>      I guess I'll just do whatever the main mirrors are doing;
> this isn't
>  a small project; and it needs to be done right. 
rsync is what many/most of the mirrors use. With good, creative use of
"exclude-from" and "include-from" you can sync almost anything or any
combination of things. It's not only what most of them use, it's easy as
pie, too.

To be honest with you the proxy server idea sounded complicated to me.
Just keep a local copy of the updates and/or release directories with
rsync and you're off to the races.

Oh, and one hint... for every release, I download *only* the ISO images.
I then unpack those and populate the proper directories (takes about
three commands and is dead easy) from the ISO files. Given 6 CD images,
saves me from downloading an additional 3.5GB (and putting that
additional load on someone's server, too).

I used to run a mirror and I kept the whole tree on my disk at home, so
I have about 60GB of the Red Hat FTP tree local. <grin> I can share my
rsync script and my include/exclude files, if you like. No secret.

Cheers,

-- 
Rodolfo J. Paiz <rpaiz(a)simpaticus.com&gt;
Simpaticus.com

[Brian Fahrlander]

On Wed, 2004-08-18 at 13:14, Rodolfo J. Paiz wrote:

[Useful stuff that you've seen, snipped]

>  I used to run a mirror and I kept the whole tree on my disk at home,
> so
>  I have about 60GB of the Red Hat FTP tree local. <grin> I can share my
>  rsync script and my include/exclude files, if you like. No secret. 
    Ya see? THAT's what I was looking for.  One of the guys doing this,
to have what I think they call a 'meta-discussion' of mirroring, not
greasy details of command lines, etc.  After all, this isn't an
alternative to Googling; it's about sharing wisdom...it just happens to
include some of the other a great deal.

    Thank you SO much for exactly what I needed.  When I'm ready to roll
this part out, I'll contact you again.

    Thanks!
-- 
------------------------------------------------------------------------
Brian Fahrländer                  Christian, Conservative, and Technomad
Evansville, IN                                 http://www.fahrlander.net
ICQ 5119262
AIM: WheelDweller
------------------------------------------------------------------------

[Thom Paine]

On Wed, 2004-08-18 at 14:14, Rodolfo J. Paiz wrote:

>  I used to run a mirror and I kept the whole tree on my disk at home,
> so
>  I have about 60GB of the Red Hat FTP tree local. <grin> I can share my
>  rsync script and my include/exclude files, if you like. No secret.
>   
I'd be interested in that.

Thanks.

-=/>Thom

[Rodolfo J. Paiz]

On Wed, 2004-08-18 at 13:30, Thom Paine wrote:
>  On Wed, 2004-08-18 at 14:14, Rodolfo J. Paiz wrote:
>  
>  > I used to run a mirror and I kept the whole tree on my disk at home, so
>  > I have about 60GB of the Red Hat FTP tree local. <grin> I can share my
>  > rsync script and my include/exclude files, if you like. No secret.
>  > 
>  
>  I'd be interested in that.
>   

Thom,

I've been offline (just too much work) for weeks and didn't see this
post. In order to avoid sending a few small attachments to the entire
list, would you be so kind as to send me a direct email so I can reply?
I'll be more than happy to include the files in the reply.

Cheers,

-- 
Rodolfo J. Paiz <rpaiz(a)simpaticus.com&gt;
Simpaticus.com

[Mike Ramirez]

On Tue, 2004-08-17 at 06:14, Brian Fahrlander wrote:
>      Well, my research into LDAP for authentication is showing GREAT
>  benefits over NIS, and I have a third machine (for the home) installing
>  upstairs as I type this. This one's delay due to a dirty CDROM, and I
>  have to go 'hit' it once in a while.  An FTP install would probably be
>  more accurate, if the bandwidth is available.
>  
>      I'm curious about repositories; since these could be massive groups
>  of PCs (say, hundreds or thousands) it should make sense to maintain a
>  separate repo (yum, presumably) on a server somewhere.
>  
>      My question is, with all the questions about repo stability and
>  such, what's the best way to go about it, strategically? I'm talking
>  about whether I should find a local college to sync with, get it
>  straight from fedora.us, or exactly what seems to go into making a nice,
>  stable mirror? 
Yam by Dag Wieers

http://dag.wieers.com/home-made/yam/


I have found this to be a great way to mirror multiple repositories.  It
creates yum and apt headers and the httpd.conf entry for you.  You can
specify which distro you want also architechusre types.  Right now it
works with fc2 i386 and x86_64 "right out of the box". Yam supports
remote installations.

[Brian Fahrlander]

On Thu, 2004-08-19 at 19:40, Mike Ramirez wrote:

>  Yam by Dag Wieers
>  
>  http://dag.wieers.com/home-made/yam/
>  
>  
>  I have found this to be a great way to mirror multiple repositories.  It
>  creates yum and apt headers and the httpd.conf entry for you.  You can
>  specify which distro you want also architechusre types.  Right now it
>  works with fc2 i386 and x86_64 "right out of the box". Yam supports
>  remote installations.   
    Sweet, thanks!
-- 
------------------------------------------------------------------------
Brian Fahrländer                  Christian, Conservative, and Technomad
Evansville, IN                                 http://www.fahrlander.net
ICQ 5119262
AIM: WheelDweller
------------------------------------------------------------------------