rsyncing 79gb of data to 250gb drive
Michael Mansour
micoots at yahoo.com
Sat Jun 5 08:18:12 UTC 2004
Hi,
> > At 15:20 5/28/2004, Michael Mansour wrote:
> > >Not only do I wonder if tar will respond the same
> way
> > >as rsync without the -S option, but I'm also
> wondering
> > >if backup software which use rsync to perform
> backups
> > >would also take this into account...
> >
> > I also wonder if there is any downside to using -S
> or whether it might
> > produce an undesirable result in some cases. If
> not, perhaps one should
> > always use it just in case?
>
> Sparse files are a boon or a bust.
>
> If you make a sparse file
>
> open(); write(byte); lseek(way-out-there);
> write(byte); close()
>
> you have not reserved or allocated the disk blocks
> for all the
> possible stuff in the middle. If you expect those
> blocks to be real
> and there is not enough disk space in the future
> things will bust.
>
> In some cases if you compress the file then expand
> it you can run
> out of disk space on the uncompress pass (compress,
> gzip, bzip2, zip).
> Watch out for uncompress bombs.
>
> Some backup and restore tools will run you out of
> tape or out of disk
> or both with sparse files. Gnu tar has a --sparse
> flag. You might not
> be able to restore the contents of a file system if
> you are not
> cautious/ aware.
>
> Some data base files might be sparse but you could
> run out of space
> a year later when you do not expect and need the
> blocks to be real.
>
> mkswap (to files) might generate files that look
> like they should be
> sparse to backup and restore tools but should not
> be. i.e. You should
> not use swapon on a file with holes.
>
> Holes on a file system with one size of allocation
> block may not be
> holes on another file system with a smaller
> allocation block size.
> Since allocation block size is heuristically
> determined one could
> be confused when moving files from one FS to another
> of a different
> size or type. mv does not have a --sparse flag that
> I can see.
>
> The big issue is backup and restore processes
> (including cp, mv).
>
> I guess a utility (holeify and holecount) might be
> written.
>
>
> --
> T o m M i t c h e l l
> /dev/null the ultimate in secure storage.
Thanks Tom, excellent explanation.
In regards to where these sparse files came from, I
set about to determine this and found out.
As mentioned in earlier posts, I use that data volume
under Samba, which is shared out to apps/clients to
store their files.
Azureus is a Java BitTorrent client, and when starting
torrent downloads on that data drive, the entire size
of the torrent is reserved on the data drive (so a
600Mb ISO shows up on the filesystem even though it is
empty). A "du -ks" shows the actual size (a few bytes
at the most), although an "ls -l" shows the 600Mb file
which the filesystem recognises as space taken.
So when I rsynced that data volume to another drive
(without the -S option), the rsync naturally copied
exactly what it saw, spacewise, but when using -S it
recognised the sparse files and therefore didn't take
up the space for them.
We learn something everyday :) ... I'm wondering now
if those sparse files that got "un-sparsed" on the
rsync will still be valid if, for example, Azureus was
to continue downloading to them.
Michael.
Find local movie times and trailers on Yahoo! Movies.
http://au.movies.yahoo.com
More information about the users
mailing list