Re: Compressing files (gz versus bz2 versus xz)

[Mike McGrath]

On Thu, 26 Aug 2010, Stephen John Smoogen wrote:

>  So I was asked to look at compression on log servers and to see if
>  changing to xz would save us some space. My test is not comprehensive
>  but showed what might happen.
>
>  Basic summary. XZ may save us up to 2% over what we are currently
>  saving but its real advantage is in speed of uncompressing files over
>  bzip2. [compression may be faster for some files also.]
>
>  File         |  Size   | Gzip  | G%   | Bunzip2 | B%   | XZ    | X%
>  messages.log |  644568 | 10992 | 98.3 |  4856   | 99.3 | 5940  | 99.1
>  mail.log     |  610816 | 65060 | 89.3 | 40836   | 93.3 | 35536 | 94.5
>  TOTAL        | 1255384 | 76052 | 93.5 | 45692   | 96.1 | 41476 | 96.5
>
>  Program      | Compression Time  | Uncompression Time
>  GZIP         | 00m43.416s        |  00m10.033s
>  BZIP         | 10m42.296s        |  01m02.525s
>  XZ           | 10m15.937s        |  00m12.565s
>
>
>  Raw data below
>
>  root@log01 smooge-b]# du -s messages.log mail.log
>  644568  messages.log
>  610816  mail.log
>  [root@log01 smooge-b]# time gzip -v -9 messages.log mail.log
>  messages.log:    98.3% -- replaced with messages.log.gz
>  mail.log:        89.3% -- replaced with mail.log.gz
>
>  real    0m43.416s
>  user    0m41.335s
>  sys     0m1.736s
>  [root@log01 smooge-b]# du -s messages.log.gz mail.log.gz
>  10992   messages.log.gz
>  65060   mail.log.gz
>  [root@log01 smooge-b]# time gunzip -v messages.log.gz mail.log.gz
>  messages.log.gz:         98.3% -- replaced with messages.log
>  mail.log.gz:     89.3% -- replaced with mail.log
>
>  real    0m10.033s
>  user    0m6.948s
>  sys     0m3.004s
>
>  [root@log01 smooge-b]# time bzip2 -v -9 messages.log mail.log
>    messages.log: 133.043:1,  0.060 bits/byte, 99.25% saved, 659381328
>  in, 4956148 out.
>    mail.log:     14.961:1,  0.535 bits/byte, 93.32% saved, 624854215
>  in, 41766136 out.
>
>  real    10m42.296s
>  user    10m36.948s
>  sys     0m1.608s
>  [root@log01 smooge-b]# du -sc messages.log.bz2 mail.log.bz2
>  4856    messages.log.bz2
>  40836   mail.log.bz2
>  45692   total
>  [root@log01 smooge-b]# time bunzip2 -v messages.log.bz2 mail.log.bz2
>    messages.log.bz2: done
>    mail.log.bz2:     done
>
>  real    1m2.525s
>  user    0m44.779s
>  sys     0m4.956s
>
>  [root@log01 smooge-b]# time xz -v -9 messages.log mail.log
>  messages.log (1/2)
>    100.0 %             5,923.6 KiB / 628.8 MiB = 0.009   3.1 MiB/s         3:21
>
>  mail.log (2/2)
>    100.0 %                34.7 MiB / 595.9 MiB = 0.058   1.4 MiB/s         6:53
>
>  real    10m15.937s
>  user    10m8.550s
>  sys     0m3.552s
>  [root@log01 smooge-b]# du -s messages.log.xz mail.log.xz
>  5940    messages.log.xz
>  35536   mail.log.xz
>  [root@log01 smooge-b]# time unxz -v messages.log.xz mail.log.xz
>  messages.log.xz (1/2)
>    100.0 %             5,923.6 KiB / 628.8 MiB = 0.009   140 MiB/s         0:04
>
>  mail.log.xz (2/2)
>    100.0 %                34.7 MiB / 595.9 MiB = 0.058    74 MiB/s         0:08
>
>  real    0m12.565s
>  user    0m8.709s
>  sys     0m3.636s
>
>  
It does take a while to grep through the bzipped logs.  if you want to
re-compress them all i say have at it.

	-Mike