Setting up private mirror...

[Brad Knowles]

Folks,

For a project at my current customer (ihiji.com), we would like to set up a private mirror, but since our production servers are all "in the cloud", we don't fall into any of the obvious categories listed at http://fedoraproject.org/wiki/Infrastructure/Mirroring and https://fedorahosted.org/mirrormanager/.  We don't have our own ASN or a CIDR block assigned to us, we're using IP addresses that are pseudo-randomly assigned to us by our cloud provider, and we're sharing the NAT and other network facilities with all their other thousands and millions of customers.

I've looked at the page for IntelligentMirror (see https://fedorahosted.org/intelligentmirror/) and it doesn't seem to have been updated since 2008.  I like the idea, but it doesn't seem that this concept has gone anywhere in the last few years.

I understand that rsync is the preferred mirror method (as opposed to using reposync from yum-utils), but I was wondering if you had any other advice?

I've run large-scale mirrors before -- in the 1999-2000 or so range, I set up the largest mirror server in Belgium at Belgacom Skynet SA/NV (my employer at the time).  But that was the ancient days of setting up mirrors, where many sites used tools like wget, where most people were using protocols like FTP, and a mirror server that had 60GB of storage available was a truly humongous machine.

But time has moved on and I want to make sure that I'm running our mirrors according to the Best Current Practice.

Thanks!

[Matt Domsch]

How will  your clients know to use your mirror?  Editing /etc/yum.repos.d/* ?    By having an assigned netblock or ASN, you can avoid that pain, but if you can't use those, oh well, edit away.

The floating IP address won't matter - you'll list a DDNS name in your URLs.

rsync is definitely the right way - please pull from a listed public mirror, and you can exclude any directories (not files within a directory) you wish.  E.g.  if you don't need iso/, then exclude such.

You may wish to set up access control lists on your mirror server httpd to allow only your clients in.

--
Matt Domsch
Technology Strategist
Dell | Office of the CTO

-----Original Message-----
From: [e-mail] [mailto:[e-mail]] On Behalf Of Brad Knowles
Sent: Monday, October 31, 2011 1:57 PM
To: [e-mail]
Cc: Brad Knowles
Subject: Setting up private mirror...

Folks,

For a project at my current customer (ihiji.com), we would like to set up a private mirror, but since our production servers are all "in the cloud", we don't fall into any of the obvious categories listed at http://fedoraproject.org/wiki/Infrastructure/Mirroring and https://fedorahosted.org/mirrormanager/.  We don't have our own ASN or a CIDR block assigned to us, we're using IP addresses that are pseudo-randomly assigned to us by our cloud provider, and we're sharing the NAT and other network facilities with all their other thousands and millions of customers.

I've looked at the page for IntelligentMirror (see https://fedorahosted.org/intelligentmirror/) and it doesn't seem to have been updated since 2008.  I like the idea, but it doesn't seem that this concept has gone anywhere in the last few years.

I understand that rsync is the preferred mirror method (as opposed to using reposync from yum-utils), but I was wondering if you had any other advice?

I've run large-scale mirrors before -- in the 1999-2000 or so range, I set up the largest mirror server in Belgium at Belgacom Skynet SA/NV (my employer at the time).  But that was the ancient days of setting up mirrors, where many sites used tools like wget, where most people were using protocols like FTP, and a mirror server that had 60GB of storage available was a truly humongous machine.

But time has moved on and I want to make sure that I'm running our mirrors according to the Best Current Practice.

Thanks!

[Brad Knowles]

On Oct 31, 2011, at 2:16 PM,   wrote:
> How will  your clients know to use your mirror?  Editing /etc/yum.repos.d/* ?    By having an assigned netblock or ASN, you can avoid that pain, but if you can't use those, oh well, edit away.

I can edit the baseurl, but I was hoping that there was a cool trick I could play by editing the timedhosts.txt and mirrorlist.txt to direct yum to point to my local mirror as the "fastest", but still be able to fall over to public mirrors in case my private mirror happens to be hosed or someone happens to be looking for an architecture or release version that I don't have.

This will all be automated with Chef, so once I've figured out how to make the necessary changes, I won't have to keep updating the files each time.
> The floating IP address won't matter - you'll list a DDNS name in your URLs.

We've cooked up a DDNS solution (using the REST interface to our DDNS provider), so that shouldn't be a problem.
> rsync is definitely the right way - please pull from a listed public mirror, and you can exclude any directories (not files within a directory) you wish.  E.g.  if you don't need iso/, then exclude such.

Will do.
> You may wish to set up access control lists on your mirror server httpd to allow only your clients in.

That would be my next problem.  Thanks!
> --
> Matt Domsch
> Technology Strategist
> Dell | Office of the CTO

Ahh, interesting -- are you in Round Rock?  We're actually a small startup hosted by ATI at what used to be the MCC building (before UT Austin bought the building).  Of course, all our servers are in the Rackspace datacenter in Dallas, so our fastest mirror is usually lstn.net (~1-2ms away).

I don't suppose you know David Duncan, another fellow member of CT-LUG?

Thanks again!