Adding concurrency/multithreading support to par2cmdline

[Erik van Pienbroek]

Hi everybody,

Recently I took ownership of the par2cmdline package. par2cmdline is a
program for creating and using PAR2 files to detect damage in data files
and repair them if necessary.

The current version of par2cmdline in Fedora is 0.4. This is also the
latest upstream version. There hasn't been any upstream activity ever
since the year 2004 [1].

Some time ago, par2cmdline was forked by somebody [2] and
concurrency/multithreading support was added to it using Intel's
Threading Building Blocks.

As this new feature enhances the performance of creating PAR2 files and
performing PAR2 repairs on modern hardware (with multiple CPU cores) I
think a lot of people would appreciate it if this would land in Fedora.

According to repoquery the par2cmdline program is currently being used
by the following packages:

$ repoquery --whatrequires par2cmdline | sort
dar-0:2.3.8-3.fc11.i586
hellanzb-0:0.13-8.fc11.noarch
nntpgrab-core-0:0.5.1-1.fc12.i686
pypar2-0:1.4-4.fc11.noarch

The package maintainers of these packages have been BCC'ed.

A scratch build containing the TBB version of par2cmdline has been done
at http://koji.fedoraproject.org/koji/taskinfo?taskID=1570576
Binary RPM's can also be fetched from there.

Package maintainers: could you please test whether the TBB version of
par2cmdline works fine with your packages?

Applications which are dependent on the order in which par2cmdline spits
out messages (like download programs which want to trace the PAR2
verification/repair process) are recommended to add the argument '-t0'
to the par2 command. This disables concurrent verify, but still enables
concurrent repair and generation which should be enough to prevent such
applications from getting confused.

The behaviour of the TBB version is the same as the original
par2cmdline. The TBB version can also be used on hardware which doesn't
have multiple CPU cores like Pentium 4's. On PPC environments Intel
Threading Building Blocks isn't available so for those environments the
old behaviour also is used.

I plan to commit my changes to rawhide in a few days. If people need
assistance in getting their applications operational with the TBB
version of par2cmdline feel free to email me or ping me on Freenode IRC
(my nickname is epienbro).

Regards,

Erik van Pienbroek


[1]: http://sourceforge.net/projects/parchive/files/par2cmdline/
[2]: http://www.chuchusoft.com/par2_tbb/

[Conrad Meyer]

On Friday 31 July 2009 13:03:06 Erik van Pienbroek wrote:
>  [snip]
>
>  The current version of par2cmdline in Fedora is 0.4. This is also the
>  latest upstream version. There hasn't been any upstream activity ever
>  since the year 2004 [1].
>
>  Some time ago, par2cmdline was forked by somebody [2] and
>  concurrency/multithreading support was added to it using Intel's
>  Threading Building Blocks. 
Also worth noting: AIUI the -tbb fork of par2cmdline fixes bugs that were 
present in the original version we currently have in Fedora, as well as taking 
advantage of multiple cores. There is a compile-time option to disable the
TBB portions (but leave the fixes for the bugs) that we plan on using on 
arches other than x86 and x86_64.

Regards,
-- 
Conrad Meyer <cemeyer(a)u.washington.edu&gt;