compression tools

[Benjamin Franz]

On Wed, 22 Jun 2005, sharif islam wrote:

> Is there any compression mechanism that's rated for tar files as big
> as, say, 100GB?

gzip and bzip2 both handle large files. What is your evaluation criteria?

Are you CPU bound (IOW how much time/power it takes to compress is your 
most important criteria and how much it compresses doesn't matter so 
much)? gzip is good for that (much faster than bzip2).

Are you space bound (IOW what is most important is how much it compresses 
- even if it takes longer/more power)? bzip2 is good for that (it often 
compresses as much as twice as well as gzip or zip). If you have multiple 
processors there is even a version that can exploit several processors to 
accelerate compression (Parallel BZIP2 
<URL:http://compression.ca/pbzip2/>).

Does the 'tar file' contain a lot of 'precompressed' data 
(zip/gz/mpg/avi/wmf/mp3/jpg/gif/png etc)? If so you may not get a lot of 
(or any) additional compression from _any_ general purpose compression 
program.

You haven't specified your problem well enough to get good recommendations.

-- 
Benjamin Franz

Simple things should be simple, complex things should be possible.
                                          - Alan Kay