Mirroring Fedora

[thomas cameron]

Howdy -

I mirror Fedora 10 to my home server, and it eats up a *lot* of space. 
It's not a big deal, I've got 500GiB drives in my file server but I'm 
interested in keeping it as small as possible.

Currently I use these commands in a script in cron.daily to mirror:

echo Fedora 10 Updates
echo
rsync -va --exclude=debug --numeric-ids --delete --delete-after \
rsync://mirrors.kernel.org/fedora/updates/10/i386 \
/var/www/html/pub/fedora/updates/10/i386/

echo Fedora 10 Updates
echo
until rsync -va --exclude=debug --numeric-ids --delete --delete-after \
rsync://mirrors.kernel.org/fedora/updates/10/x86_64/ \
/var/www/html/pub/fedora/updates/10/x86_64/

The thing is, F10 is taking up like 50GiB of disk space between the 
install media, Everything and updates.

I know that there are a ton of duplicate files between the i386 and the 
x86_64 directories.  Is there an easy way to do something like hard link 
duplicate files or maybe a better way to rsync so that duplicate files 
are considered?

Thanks!
Thomas

[Todd Zullinger]

Thomas Cameron wrote:
> I know that there are a ton of duplicate files between the i386 and
> the  x86_64 directories.  Is there an easy way to do something like
> hard link  duplicate files or maybe a better way to rsync so that
> duplicate files  are considered?

There is a -H option for rsync, to make it preserve hardlinks.  This
and other useful tips for mirroring are at:

http://fedoraproject.org/wiki/Infrastructure/Mirroring

Running hardlink on your mirror can also help (and is also mentioned
at the URL above).

[thomas cameron]

Todd Zullinger wrote:
> Thomas Cameron wrote:
> > I know that there are a ton of duplicate files between the i386 and
> > the  x86_64 directories.  Is there an easy way to do something like
> > hard link  duplicate files or maybe a better way to rsync so that
> > duplicate files  are considered?
>
> There is a -H option for rsync, to make it preserve hardlinks.  This
> and other useful tips for mirroring are at:
> http://fedoraproject.org/wiki/Infrastructure/Mirroring
>
> Running hardlink on your mirror can also help (and is also mentioned
> at the URL above).

Hrm - I added that and reran the sync but since each rsync job is 
separate, it isn't hardlinking between the directories.  I think I need 
to mess around with the rsycn command so that it encompasses both the 
i386 and the x86_64 directories in on sync.  Off to play with the 
syntax, I guess.

[Bruno Wolff III]

On Sat, Nov 29, 2008 at 13:38:15 -0600,
  Thomas Cameron  wrote:
> Hrm - I added that and reran the sync but since each rsync job is  
> separate, it isn't hardlinking between the directories.  I think I need  
> to mess around with the rsycn command so that it encompasses both the  
> i386 and the x86_64 directories in on sync.  Off to play with the  
> syntax, I guess.

This isn't exactly what you want, but I use it to grab rawhide's i386 and
x86_64 repositories. $MIRROR and $BASEDIR are the rysnc url for the
mirror and the local directory to put stuff in, respectively.
rsync -rltHvz --delete --include=/i386/ --include=/x86_64/ --exclude='/*' --inc
lude=/i386/os/ --include=/x86_64/os/ --exclude='/i386/*' --exclude='/x86_64/*' $
MIRROR/development/ $BASEDIR/official

[Bruno Wolff III]

On Sat, Nov 29, 2008 at 13:38:15 -0600,
  Thomas Cameron  wrote:
> Hrm - I added that and reran the sync but since each rsync job is  
> separate, it isn't hardlinking between the directories.  I think I need  
> to mess around with the rsycn command so that it encompasses both the  
> i386 and the x86_64 directories in on sync.  Off to play with the  
> syntax, I guess.

I needed to play around with this and get it set up to use Everything,
Updates and Updates-Testing instead of using Rawhide + Koji. As a bonus
I also have it pulling in RPMFusion in one rsync for both x86_64 and i386
in case you want that.

#!/bin/sh

RELEASE=10
BASEDIR=/home/f${RELEASE}
FEDORA=rsync://mirror.hiwaay.net/fedora-linux/
FUSION=rsync://fr2.rpmfind.net/linux/rpmfusion/

echo -e "\nSyncing local Fedora ${RELEASE} mirrors ...\n"
rsync -rltHvz --delete-after --include=/releases --include=/updates --exclude=/'*' --include=/releases/${RELEASE} --exclude=/releases/'*' --include=/releases/${RELEASE}/Everything --include=/releases/${RELEASE}/Fedora --exclude=/releases/${RELEASE}/'*' --include=/releases/${RELEASE}/Everything/i386/ --include=/releases/${RELEASE}/Everything/x86_64/ --exclude=/releases/${RELEASE}/Everything/'*' --include=/releases/${RELEASE}/Everything/i386/os/ --include=/releases/${RELEASE}/Everything/x86_64/os/ --exclude=/releases/${RELEASE}/Everything/i386/'*' --exclude=/releases/${RELEASE}/Everything/x86_64/'*' --include=/releases/${RELEASE}/Fedora/i386 --exclude=/releases/${RELEASE}/Fedora/'*' --include=/releases/${RELEASE}/Fedora/i386/iso --exclude=/releases/${RELEASE}/Fedora/i386/'*' --include=/releases/${RELEASE}/Fedora/i386/iso/'*'-netinst.iso --exclude=/releases/${RELEASE}/Fedora/i386/iso/'*' --include=/updates/${RELEASE} --include=/updates/testing --exclude=/updates/'*' --include=/updates/${RELEASE}/i386 --include=/updates/${RELEASE}/x86_64 --exclude=/updates/${RELEASE}/'*' --exclude=/updates/${RELEASE}/'*'/debug --include=/updates/testing/${RELEASE} --exclude=/updates/testing/'*' --include=/updates/testing/${RELEASE}/i386 --include=/updates/testing/${RELEASE}/x86_64 --exclude=/updates/testing/${RELEASE}/'*' --exclude=/updates/testing/${RELEASE}/'*'/debug ${FEDORA} ${BASEDIR}

echo -e "\nSyncing local RPM Fusion Fedora ${RELEASE} mirrors ...\n"
rsync -rltHvz --delete-after --include=/'*'free --exclude=/'*' --include=/'*'free/fedora --exclude=/'*'free/'*' --include=/'*'free/fedora/releases --include=/'*'free/fedora/updates --exclude=/'*'free/fedora/'*' --include=/'*'free/fedora/releases/${RELEASE} --exclude=/'*'free/fedora/releases/'*' --include=/'*'free/fedora/releases/${RELEASE}/Everything --include=/'*'free/fedora/releases/${RELEASE}/Fedora --exclude=/'*'free/fedora/releases/${RELEASE}/'*' --include=/'*'free/fedora/releases/${RELEASE}/Everything/i386/ --include=/'*'free/fedora/releases/${RELEASE}/Everything/x86_64/ --exclude=/'*'free/fedora/releases/${RELEASE}/Everything/'*' --include=/'*'free/fedora/releases/${RELEASE}/Everything/i386/os/ --include=/'*'free/fedora/releases/${RELEASE}/Everything/x86_64/os/ --exclude=/'*'free/fedora/releases/${RELEASE}/Everything/i386/'*' --exclude=/'*'free/fedora/releases/${RELEASE}/Everything/x86_64/'*' --include=/'*'free/fedora/releases/${RELEASE}/Fedora/i386 --exclude=/'*'free/fedora/releases/${RELEASE}/Fedora/'*' --exclude=/'*'free/fedora/releases/${RELEASE}/Fedora/i386/'*' --include=/'*'free/fedora/updates/${RELEASE} --include=/'*'free/fedora/updates/testing --exclude=/'*'free/fedora/updates/'*' --include=/'*'free/fedora/updates/${RELEASE}/i386 --include=/'*'free/fedora/updates/${RELEASE}/x86_64 --exclude=/'*'free/fedora/updates/${RELEASE}/'*' --exclude=/'*'free/fedora/updates/${RELEASE}/'*'/debug --include=/'*'free/fedora/updates/testing/${RELEASE} --exclude=/'*'free/fedora/updates/testing/'*' --include=/'*'free/fedora/updates/testing/${RELEASE}/i386 --include=/'*'free/fedora/updates/testing/${RELEASE}/x86_64 --exclude=/'*'free/fedora/updates/testing/${RELEASE}/'*' --exclude=/'*'free/fedora/updates/testing/${RELEASE}/'*'/debug ${FUSION} ${BASEDIR}