RE: rsyncing 79gb of data to 250gb drive

[David Maynard]

I would suspect that some of the source files are "sparse."  Ie. there are
"holes" in the middle of them where space hasn't been allocated in the
source directory.  Some applications that use random file access will create
sparse files.  A simple copy operation that doesn't look for sparse files
will fill in the holes, causing the copy to take up more space than the
original.

I haven't tried it (at least not recently), but rsync appears to have a
"--sparse" (-S) option.  Try using that in addition to -av and see if that
changes the results.  GNU tar has a similar option.

-dpm

-----Original Message-----
From: Michael Mansour [mailto:[e-mail]]
Sent: Tuesday, May 25, 2004 11:45 PM
To: fedora-list(a)redhat.com
Subject: rsyncing 79gb of data to 250gb drive


Hi,

I have an issue which I can't seem to figure out.

I bought a 250Gb Maxtor Maxline Plus ATA133 drive,
plugged it into a server, formatted it using:

# e2fsck -m 0 -j -b 1024 /dev/hdc1

I made the partition a Linux partition id 82.

After format, 241Gb free space is recognised.

I have 80Gb of data on another drive in the system
that I wish to backup onto this drive. So after
rsyncing the drives contents from the 79Gb (which is
on a stripe) to the maxtor 250Gb drive, the following
is the result:

/dev/md6              85419328  79573952   1506236 
99% /data01
/dev/hdc1            241201359 123508888 117692471 
52% /data02

123Gb used??? this is what is confusing me, why?

I first formatted this drive using 4096 block size and
got the result above, then I thought I'll reformat it
and change it to 1024 byte block size to try and get
back some space, the same result above.

I've also made sure (with tune2fs) that the "sparse"
option is turned on for the large filesystem.

I thought maybe the rsync was following symlinks
(which I'm sure there are none), but by default the
command:

rsync -av

won't do that. I checked for pipe and socket files
using find, and also couldn't see anything suspicious.

I'm at a loss to figure this one out and thought I'd
email here for some help.

Thanks.

Michael.

Find local movie times and trailers on Yahoo! Movies.
http://au.movies.yahoo.com


-- 
fedora-list mailing list
fedora-list(a)redhat.com
To unsubscribe: http://www.redhat.com/mailman/listinfo/fedora-list

rsyncing 79gb of data to 250gb drive

[Ow Mun Heng]

On Wed, 2004-05-26 at 11:42, David Maynard wrote:
>  I would suspect that some of the source files are "sparse."
>  Ie. there are
>  "holes" in the middle of them where space hasn't been allocated in the
>  source directory.  Some applications that use random file access will create
>  sparse files.  A simple copy operation that doesn't look for sparse files
>  will fill in the holes, causing the copy to take up more space than the
>  original.
>  
>  I haven't tried it (at least not recently), but rsync appears to have a
>  "--sparse" (-S) option.  Try using that in addition to -av and see if that
>  changes the results.  GNU tar has a similar option. 
Actually, if you're trying to copy all the files from A to B, where B is
a Fresh directory, It's advisable to use GNU Tar. it'll speed up the
transfer and it'll ensure that permissions are kept as they should.

After tha Rsync them for incremental backup etc.

my 2 cents

rsyncing 79gb of data to 250gb drive

[Michael Mansour]

Hi,

 --- Ow Mun Heng <Ow.Mun.Heng(a)wdc.com&gt; wrote: > On
Wed, 2004-05-26 at 11:42, David Maynard wrote:
>  > I would suspect that some of the source files are
>  "sparse."  Ie. there are
>  > "holes" in the middle of them where space hasn't
>  been allocated in the
>  > source directory.  Some applications that use
>  random file access will create
>  > sparse files.  A simple copy operation that
>  doesn't look for sparse files
>  > will fill in the holes, causing the copy to take
>  up more space than the
>  > original.
>  > 
>  > I haven't tried it (at least not recently), but
>  rsync appears to have a
>  > "--sparse" (-S) option.  Try using that in
>  addition to -av and see if that
>  > changes the results.  GNU tar has a similar
>  option.
>  
>  Actually, if you're trying to copy all the files
>  from A to B, where B is
>  a Fresh directory, It's advisable to use GNU Tar.
>  it'll speed up the
>  transfer and it'll ensure that permissions are kept
>  as they should.
>  
>  After tha Rsync them for incremental backup etc.
>  
>  my 2 cents 
For the people following this sage here is the result
of the rsync withOUT the -S option:

/dev/md6              85419328  79678740   1401448 
99% /data01
/dev/hdf1            241263968 124262976 117000992 
52% /data02

So the problem is sparse files.

I wonder if tar takes this into account also?

Michael.


Find local movie times and trailers on Yahoo! Movies.
http://au.movies.yahoo.com

rsyncing 79gb of data to 250gb drive

[Michael Mansour]

Hi,

Just to keep things updated here:

 --- Michael Mansour <micoots(a)yahoo.com&gt; wrote: > Hi,
>  
>   --- Ow Mun Heng <Ow.Mun.Heng(a)wdc.com&gt; wrote: > On
>  Wed, 2004-05-26 at 11:42, David Maynard wrote:
>  > > I would suspect that some of the source files
>  are
>  > "sparse."  Ie. there are
>  > > "holes" in the middle of them where space hasn't
>  > been allocated in the
>  > > source directory.  Some applications that use
>  > random file access will create
>  > > sparse files.  A simple copy operation that
>  > doesn't look for sparse files
>  > > will fill in the holes, causing the copy to take
>  > up more space than the
>  > > original.
>  > > 
>  > > I haven't tried it (at least not recently), but
>  > rsync appears to have a
>  > > "--sparse" (-S) option.  Try using that in
>  > addition to -av and see if that
>  > > changes the results.  GNU tar has a similar
>  > option.
>  > 
>  > Actually, if you're trying to copy all the files
>  > from A to B, where B is
>  > a Fresh directory, It's advisable to use GNU Tar.
>  > it'll speed up the
>  > transfer and it'll ensure that permissions are
>  kept
>  > as they should.
>  > 
>  > After tha Rsync them for incremental backup etc.
>  > 
>  > my 2 cents
>  
>  For the people following this sage here is the
>  result
>  of the rsync withOUT the -S option:
>  
>  /dev/md6              85419328  79678740   1401448 
>  99% /data01
>  /dev/hdf1            241263968 124262976 117000992 
>  52% /data02
>  
>  So the problem is sparse files.
>  
>  I wonder if tar takes this into account also?
>  
>  Michael. 
I did another:

rsync -av -S /data01/ /data02

with the following result:

/dev/md6              85419328  79678740   1401448 
99% /data01
/dev/hdf1            241263968  79532584 161731384 
33% /data02

Strange how I got such files to begin with, because on
the original partition it's got 79Gb, I'm assuming
with sparse files, and only when I perform the rsync
(without the -S option) does it grow the sizes on the
rsync.

Not only do I wonder if tar will respond the same way
as rsync without the -S option, but I'm also wondering
if backup software which use rsync to perform backups
would also take this into account...

Michael.


Find local movie times and trailers on Yahoo! Movies.
http://au.movies.yahoo.com

rsyncing 79gb of data to 250gb drive

[Rodolfo J. Paiz]

At 15:20 5/28/2004, Michael Mansour wrote:
> Not only do I wonder if tar will respond the same way
> as rsync without the -S option, but I'm also wondering
> if backup software which use rsync to perform backups
> would also take this into account... 
I also wonder if there is any downside to using -S or whether it might 
produce an undesirable result in some cases. If not, perhaps one should 
always use it just in case?

Cheers,


-- 
Rodolfo J. Paiz
rpaiz(a)simpaticus.com
http://www.simpaticus.com

rsyncing 79gb of data to 250gb drive

[Michael Mansour]

Hi,

>  >Not only do I wonder if tar will respond the same
>  way
>  >as rsync without the -S option, but I'm also
>  wondering
>  >if backup software which use rsync to perform
>  backups
>  >would also take this into account...
>  
>  I also wonder if there is any downside to using -S
>  or whether it might 
>  produce an undesirable result in some cases. If not,
>  perhaps one should 
>  always use it just in case? 
Since then I've been rsyncing all my data using that
switch, and so far haven't seen any issues with it.

The man page on it says the only problem that can be
caused by it is if rsyncing TO a Solaris filesystem,
which can't handle the switch and garbles the files.

Either way, any rsync's I do from now on will contain
it.

Michael.


Find local movie times and trailers on Yahoo! Movies.
http://au.movies.yahoo.com

rsyncing 79gb of data to 250gb drive

[Nathan Ollerenshaw]

I've found the fastest way to copy from one partition to another, preserving 
symlinks, hard links, special devices, etc is to use dump/restore.

  cd <destination> ; dump -0 /dev/<source> -f - | restore -rf -

replace <source> with the device name of the source device (say, hda1) and 
replace destination with the directory you want to copy to.

Never had a problem doing it this way.

(sorry, didnt read the rest of the thread, but thought I'd toss in my 2c ;)

\n

rsyncing 79gb of data to 250gb drive

[Tom Mitchell]

On Wed, Jun 02, 2004 at 11:46:38AM -0600, Rodolfo J. Paiz wrote:
>  Date: Wed, 02 Jun 2004 11:46:38 -0600
>  To: For users of Fedora Core releases <fedora-list(a)redhat.com&gt;
>  From: "Rodolfo J. Paiz" <rpaiz(a)simpaticus.com&gt;
>  Subject: RE: rsyncing 79gb of data to 250gb drive
>  Reply-To: For users of Fedora Core releases <fedora-list(a)redhat.com&gt;
>  
>  At 15:20 5/28/2004, Michael Mansour wrote:
>  >Not only do I wonder if tar will respond the same way
>  >as rsync without the -S option, but I'm also wondering
>  >if backup software which use rsync to perform backups
>  >would also take this into account...
>  
>  I also wonder if there is any downside to using -S or whether it might 
>  produce an undesirable result in some cases. If not, perhaps one should 
>  always use it just in case? 
Sparse files are a boon or a bust.  

If you make a sparse file 

  open(); write(byte); lseek(way-out-there); write(byte); close()

you have not reserved or allocated the disk blocks for all the
possible stuff in the middle.  If you expect those blocks to be real
and there is not enough disk space in the future things will bust.

In some cases if you compress the file then expand it you can run
out of disk space on the uncompress pass (compress, gzip, bzip2, zip).
Watch out for uncompress bombs.

Some backup and restore tools will run you out of tape or out of disk
or both with sparse files. Gnu tar has a --sparse flag.  You might not
be able to restore the contents of a file system if you are not
cautious/ aware.

Some data base files might be sparse but you could run out of space
a year later when you do not expect and need the blocks to be real.

mkswap (to files) might generate files that look like they should be
sparse to backup and restore tools but should not be. i.e. You should
not use swapon on a file with holes.

Holes on a file system with one size of allocation block may not be
holes on another file system with a smaller allocation block size.
Since allocation block size is  heuristically determined one could
be confused when moving files from one FS to another of a different 
size or type.  mv does not have a --sparse flag that I can see.

The big issue is backup and restore processes (including cp, mv).

I guess a utility (holeify and holecount) might be written.


-- 
	T o m  M i t c h e l l 
	/dev/null the ultimate in secure storage.

rsyncing 79gb of data to 250gb drive

[Michael Mansour]

Hi,

>  > At 15:20 5/28/2004, Michael Mansour wrote:
>  > >Not only do I wonder if tar will respond the same
>  way
>  > >as rsync without the -S option, but I'm also
>  wondering
>  > >if backup software which use rsync to perform
>  backups
>  > >would also take this into account...
>  > 
>  > I also wonder if there is any downside to using -S
>  or whether it might 
>  > produce an undesirable result in some cases. If
>  not, perhaps one should 
>  > always use it just in case?
>  
>  Sparse files are a boon or a bust.  
>  
>  If you make a sparse file 
>  
>    open(); write(byte); lseek(way-out-there);
>  write(byte); close()
>  
>  you have not reserved or allocated the disk blocks
>  for all the
>  possible stuff in the middle.  If you expect those
>  blocks to be real
>  and there is not enough disk space in the future
>  things will bust.
>  
>  In some cases if you compress the file then expand
>  it you can run
>  out of disk space on the uncompress pass (compress,
>  gzip, bzip2, zip).
>  Watch out for uncompress bombs.
>  
>  Some backup and restore tools will run you out of
>  tape or out of disk
>  or both with sparse files. Gnu tar has a --sparse
>  flag.  You might not
>  be able to restore the contents of a file system if
>  you are not
>  cautious/ aware.
>  
>  Some data base files might be sparse but you could
>  run out of space
>  a year later when you do not expect and need the
>  blocks to be real.
>  
>  mkswap (to files) might generate files that look
>  like they should be
>  sparse to backup and restore tools but should not
>  be. i.e. You should
>  not use swapon on a file with holes.
>  
>  Holes on a file system with one size of allocation
>  block may not be
>  holes on another file system with a smaller
>  allocation block size.
>  Since allocation block size is  heuristically
>  determined one could
>  be confused when moving files from one FS to another
>  of a different 
>  size or type.  mv does not have a --sparse flag that
>  I can see.
>  
>  The big issue is backup and restore processes
>  (including cp, mv).
>  
>  I guess a utility (holeify and holecount) might be
>  written.
>  
>  
>  -- 
>  	T o m  M i t c h e l l 
>  	/dev/null the ultimate in secure storage. 
Thanks Tom, excellent explanation.

In regards to where these sparse files came from, I
set about to determine this and found out.

As mentioned in earlier posts, I use that data volume
under Samba, which is shared out to apps/clients to
store their files.

Azureus is a Java BitTorrent client, and when starting
torrent downloads on that data drive, the entire size
of the torrent is reserved on the data drive (so a
600Mb ISO shows up on the filesystem even though it is
empty). A "du -ks" shows the actual size (a few bytes
at the most), although an "ls -l" shows the 600Mb file
which the filesystem recognises as space taken.

So when I rsynced that data volume to another drive
(without the -S option), the rsync naturally copied
exactly what it saw, spacewise, but when using -S it
recognised the sparse files and therefore didn't take
up the space for them.

We learn something everyday :) ... I'm wondering now
if those sparse files that got "un-sparsed" on the
rsync will still be valid if, for example, Azureus was
to continue downloading to them.

Michael.


Find local movie times and trailers on Yahoo! Movies.
http://au.movies.yahoo.com

rsyncing 79gb of data to 250gb drive

[Tom Mitchell]

On Sat, Jun 05, 2004 at 06:18:12PM +1000, Michael Mansour wrote:
>  > > At 15:20 5/28/2004, Michael Mansour wrote:
>  > > >Not only do I wonder if tar will respond the same way
>  > > >as rsync without the -S option, but I'm also  wondering
>  > > >if backup software which use rsync to perform  backups
>  > > >would also take this into account...
>  > > 
>  > > I also wonder if there is any downside to using -S  or whether it might 
>  > > produce an undesirable result in some cases. If  not, perhaps one should 
>  > > always use it just in case?
>  > 
>  > Sparse files are a boon or a bust.  
>  > 
>  > If you make a sparse file 
>  > 
>  >   open(); write(byte); lseek(way-out-there); write(byte); close()
>  > 
>  > you have not reserved or allocated the disk blocks
>  > for all the possible stuff in the middle.  If you expect those
>  > blocks to be real and there is not enough disk space in the future
>  > things will bust.
>  >  
.....
>  
>  Thanks Tom, excellent explanation. 
You are welcome.

>  We learn something everyday :) ... I'm wondering now
>  if those sparse files that got "un-sparsed" on the
>  rsync will still be valid if, for example, Azureus was
>  to continue downloading to them. 
There should be no problem.

If the file was sparse (holes of null) blocks of null would be
delivered on demand.  In the un-sparsed situation you still get your
block of nulls, this time from physical disk.

In some cases a program that loads a large sparse data file will be
slower because it becomes disk bandwidth IO bound.  The kernel filesystem
code would see a gap in a sparse file and copy a block of nulls at
'register to memory' speed into the applications data space.  Such
applications with BIG sparse data sets are rare.

It is something to think about if you are designing a application with
big sparse data and big sparse files.


-- 
	T o m  M i t c h e l l 
	/dev/null the ultimate in secure storage.