compression tools
Jon Shorie
jshorie at medinaco.org
Wed Jun 22 19:58:32 UTC 2005
> On Wednesday, Jun 22nd 2005 at 19:42 +0200, quoth Gérard Milmeister:
>
> =>On Wed, 2005-06-22 at 13:25 -0400, Scot L. Harris wrote:
> =>> On Wed, 2005-06-22 at 12:54, sharif islam wrote:
> =>> > Is there any compression mechanism that's rated for tar files as big
> =>> > as, say, 100GB?
> =>>
> =>> I don't believe anyone rates compression programs based the size of the
> =>> file to be compressed.
Not actually true. Older versions of gzip (such as the one that came with
Solaris 8) could only work with tar archives (or any file) that was <2GB.
With new versions this is not the case.
> =>
> =>For files of that size, it is probably useful for compress program to be
> =>fast. Bzip2 for example would take ages to compress 100GB. Gzip with a
> =>low compress degree (< 5) would be more useful.
>
> I think you'll find that gzip and bzip2 will give very similiar results
> both in time and space by appropriately lowering the compression factor.
>
> e.g., bzip2 -1 and gzip -1 will be very comparable.
>
> The point is that the larger the chunk that any compression program has to
> operate upon the more compression you're going to end up with and the
> greater the time requirement will be for the chunk.
I have found that when compressing large files Gzip seems to be faster with
default options especially if the original is on an NFS mount.
We have a Linux backup server that backs up all of our other servers nightly.
We use gzip in a command such as the following:
tar cvf /backup/alpha/payroll/monday.tar.gz /export/alpha/payroll/* --gzip
To back up our payroll program that is running on an Alpha with Redhat 7.1.
On some of our larger archives we build the tar.gz on the server to be backed
up and then just copy over the file to the backup server.
cp /export/penuing/backup/websites.tar.gz /backup/penguin/monday.tar.gz
We found that when backing up our Solaris 8 box we had to use
tar cvf /backups/homedirectories.tar /export/home/*
compress /backups/homedirectories.tar
and then on our backup server
cp /export/sunbox/backup/homedirectories.tar.Z /backup/sunbox/monday.tar.Z
because the solaris 8 version of gzip would only work with archives < 2GB
although it is usually more efficient than compress for archives that it will
work on.
Jon Shorie
Medina County Sanitary Engineers
More information about the users
mailing list