F21 System Wide Change: lbzip2 as default bzip2 implementation

[Mikolaj Izdebski]

On 04/04/2014 05:26 PM, Mikolaj Izdebski wrote:
> On 04/04/2014 05:16 PM, Michal Schmidt wrote:
>> On 04/04/2014 04:15 PM, Mikolaj Izdebski wrote:
>>> Compression of payload.tar
>>> --------------------------
>>>
>>> command    |   real |   user |  sys | memory | compr. size
>>> -----------+--------+--------+------+--------+------------
>>> lbzip2     |   3.36 | 170.07 | 6.38 | 380448 | 424676188
>>> lbzip2 -u  |   6.45 | 123.14 | 3.80 | 255524 | 424518771
>>> pbzip2     |   6.78 | 288.33 | 8.90 | 491644 | 425213134
>>> bzip2      | 176.68 | 175.76 | 0.67 |   8000 | 425108407
>>>
>>>
>>> Conclusions
>>> ===========
>>> [...]
>>> "lbzip2 -u" always produced smallest files (even smaller than bzip2)
>>> while consuming the least amount of resources (CPU power and memory).
>>
>> The table above says it needs about 30 times *more* memory than bzip2.
> 
> No, it shows that it *used* that much memory.
> 
> The system had 32 GB of RAM, lbzip2 using all 56 CPUs used less than 1.2
> % of available memory.  That is *very* conservative.
> 
> Memory usage can be limited by lowering number of threads used (-n) or
> by specifying explicit memory limit (-m, undocumented for now, it will
> be fully supported in future version of lbzip2 after it gets enough
> testing).

lbzip2 can use less memory than bzip2 while still being much faster.

Example results:

	Command being timed: "bzip2 -kf linux-3.12.6.tar"
	User time (seconds): 47.70
	System time (seconds): 0.20
	Percent of CPU this job got: 95%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 0:49.92
	Maximum resident set size (kbytes): 7884

	Command being timed: "lbzip2 -kfusn1 linux-3.12.6.tar"
	User time (seconds): 31.77
	System time (seconds): 0.89
	Percent of CPU this job got: 99%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 0:32.96
	Maximum resident set size (kbytes): 7704

-- 
Mikolaj Izdebski
Software Engineer, Red Hat
IRC: mizdebsk