Zbip2 Size limitatations ?

[Gene Heskett]

On Friday 30 December 2005 23:56, Franck Y wrote:
>Hi everyone,
>
>I make several backup with bzip2, it seems that he has the better
> compression.
>
>I do my backup like this for several folders.
>
>tar cvf - /home/data/ | bzip2 -9 > /backup/data.tar.bz2
>
>When i test the archive with bzip2 -t archive.tar.bz2, everything is
>correct when the archive is under 1 Go. But i get CRC error when it's
>upper ....
>
>
>Has anyone experience this ?

No, other than I've also heard that bzip2 has problems with source 
streams above 2GB, which would explain the effect your are seeing.

As an amanda user myself, it seems rather foolish to make a single 
backup file that big considering what you have lost if its 
unrecoverable, and bz2 stuff can be entirely too sensitive to the 
medium its on.  Its demonstrated that to me on many occasions.

"gzip best", OTOH, is considerably more robust in my experience, and 
faster in the compression cycle than bz2.  And, when you have a good 
backup manager, the individual files can be made fairly smallish so 
that each file in the archive remains a manageable size.  My amanda 
disklist has 44 entries scattered over 2 machines because I've 
purposely made each entry only one, under a gigabyte of raw data, 
directory tree.  Having smallish files means amanda can more easily 
manage the storage to make maximum use of the tape it is allowed to 
use each night, and I've often filled the tape to 98% capacity without 
ever hitting an EOT from the drive.  With a mixture of entries that 
compress, and those that aren't (because they already are, like rpms & 
tar.gz's etc) gzip can routinely reduce 16GB of raw data into 6GB on 
the tape, so I don't consider the better compression performance of 
bz2 to be an advantage other than in the eye of the beholder.  Storage 
today is cheap, use what works more dependably rather than that which 
may work slightly better, but only 99% of the time.  For backups, a 1% 
failure rate is unacceptable to me.

>--
>Franck

-- 
Cheers, Gene
People having trouble with vz bouncing email to me should add the word
'online' between the 'verizon', and the dot which bypasses vz's
stupid bounce rules.  I do use spamassassin too. :-)
Yahoo.com and AOL/TW attorneys please note, additions to the above
message by Gene Heskett are:
Copyright 2005 by Maurice Eugene Heskett, all rights reserved.