[OS-BUILD PATCHv5] redhat/kernel.spec.template: Parallelize compression

[Prarit Bhargava (via Email Bridge)]

From: Prarit Bhargava <prarit(a)redhat.com&gt;

redhat/kernel.spec.template: Parallelize compression

This line in the kernel.spec file:

find $RPM_BUILD_ROOT/lib/modules/ -type f -name '*.ko' | xargs
-P${RPM_BUILD_NCPUS} -r xz;

implies that ${RPM_BUILD_NCPUS} xz instances are run.  This is not true,
and this line actually applies ${RPM_BUILD_NCPUS} to a single instance of
xz.  This means that the compression has been done one module at a time
(ie, in serial) rather than in parallel as is implied by the code.

Use xarg's -n option to assign one cpu per process and parallelize the
compression.

Suggested-by: "Herton R. Krzesinski" <herton(a)redhat.com&gt;
Signed-off-by: Prarit Bhargava <prarit(a)redhat.com&gt;

diff --git a/redhat/kernel.spec.template b/redhat/kernel.spec.template
index blahblah..blahblah 100755
--- a/redhat/kernel.spec.template
+++ b/redhat/kernel.spec.template
@@ -2472,7 +2472,8 @@ find Documentation -type d | xargs chmod u+w
     fi \
   fi \
   if [ "%{zipmodules}" -eq "1" ]; then \
-    find $RPM_BUILD_ROOT/lib/modules/ -type f -name '*.ko' | xargs
-P${RPM_BUILD_NCPUS} -r xz; \
+    echo "Compressing kernel modules ..." \
+    find $RPM_BUILD_ROOT/lib/modules/ -type f -name '*.ko' | xargs -n 1
-P${RPM_BUILD_NCPUS} -r xz; \
   fi \
 %{nil}
 

--
https://gitlab.com/cki-project/kernel-ark/-/merge_requests/2254

[Denys Vlasenko (via Email Bridge)]

From: Denys Vlasenko on gitlab.com
https://gitlab.com/cki-project/kernel-ark/-/merge_requests/2254#note_1249...

"-n 1" will spawn one "xz" per every module. This incurs some startup
costs
(fork+exec+library loading).
How about "-n 16", so that each xz process compresses 16 modules, decreasing
startup costs by more than 90%? rhel7 and rhel8 already use this approach.

[Prarit Bhargava (via Email Bridge)]

From: Prarit Bhargava on gitlab.com
https://gitlab.com/cki-project/kernel-ark/-/merge_requests/2254#note_1249...

I'd argue then the original version (with a separate parallel script) would be
preferable than a magic number.  @hertonrk-rh, your thoughts?

[Herton R. Krzesinski (via Email Bridge)]

From: Herton R. Krzesinski on gitlab.com
https://gitlab.com/cki-project/kernel-ark/-/merge_requests/2254#note_1249...

I don't think we need to overengineer this. I agree fork costs can be
something to avoid, but we don't need a separate script. If there is a
heuristic for it we should calculate it and and give to ```-n``` option. But
I'm ok the way it is right now. If you want to determine an heuristic and
measure the difference feel free to.

[Denys Vlasenko (via Email Bridge)]

From: Denys Vlasenko on gitlab.com
https://gitlab.com/cki-project/kernel-ark/-/merge_requests/2254#note_1249...

We usually have 2000..4000 modules.
With e.g. -n 1000, you obviously risk having fewer "xz" processes than CPUs.
Even with -n 100, conceivably on a big machine, you can underutilize CPUs.
OTOH, -n 1, you can spend more time starting a new xz process than compressing
a module, if it's a small module.
So yes, -n16 is an arbitrary choice, but there is logic why it should be
somewhere in [10,100] range.