[releng] Issue #8581: installations from Live images use excessive client CPU on /dev/loop1

[Chris Murphy]

chrismurphy reported a new issue against the project: `releng` that you are following:
``
* Describe the issue

Based on the investigation on this bug, the squashfs with xz decompression causes a
significant CPU hit and is expected.
https://bugzilla.redhat.com/show_bug.cgi?id=1717728

Use this releng issue to track changes and testing for a switch to something else,
possibly zstd, which is known to have far less resource requirements for decompression,
and see if this can be improved on the client side without unacceptable negatives on the
compose side.

* When do you need this? (YYYY/MM/DD)

To be determined. Depends on how testing goes.

* When is this no longer needed or useful? (YYYY/MM/DD)

Unforeseeable future.

* If we cannot complete your request, what is the impact?

Status quo. Clients will continue to experience high CPU load during installations.
It's been this way for a while so it's not the end of the world, but thousands of
Fedora users, and openQA VMs, get hit with this every time they do an installation, every
day, weeks on end.
``

To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/8581

[Chris Murphy]

chrismurphy added a new comment to an issue you are following:
``
Looks like the minimum squashfs-tools needed to support this feature is 4.3-21
* Mon Jun 24 2019 Bruno Wolff III <bruno(a)wolff.to&gt; - 4.3-21*
*- Add zstd compression support (Sean Purcell via github.com/plougher/squashfs-tools)*

It's possible the nested ext4 image in the squashfs image is contributing to this
problem. Opened #8646 to separately track the possibility of using plain squashfs images.
``

To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/8581

[Chris Murphy]

chrismurphy added a new comment to an issue you are following:
``
Update on the testing here, summary is that it works and /dev/loop1 burn drops 10-30% CPU
instead of 65-85% CPU.
https://bugzilla.redhat.com/show_bug.cgi?id=1717728#c24

Update to that update. Size change. Still using
Fedora-Workstation-Live-x86_64-Rawhide-20190816.n.0.iso as the basis. 

6981419008 rootfs.img, is the ext4 file system image in the squashfs.img, that's
what's being compressed by mksquashfs and compared. All percentages are relative to
the original ISO's squashfs.img and represents change in final ISO size.

1927286784 +0%, squashfs.img on ISO
2138419200 +11% squashfs.img using level 15 (mksquashfs default)
2430496768 +26% squashfs.img using level 1, ~5x faster than level 15
2372837376 +23% squashfs.img using level 1, plain squashfs (omits embedded ext4 image but
contains its payload)
2326519808 +21% squashfs.img using level 1, plain squashfs, 256K block size
2020712448 +5%  squashfs.img using level 22, plain squashfs, 256K block size, ~9x slower
than level 1

Some testing to see if block size 512K helps knock down that 5% more without the consumer
side memory+cpu taking an excessive hit. And I'll ask squashfs folks if it's
possible to unlock the higher compression levels above 22. The mksquashfs with zstd does
compress in parallel, by default using the number of CPUs, I expect on the build side to
get the same or better compression compared to xz, it will take more RAM and CPU. But on
the consumer side, higher compression levels have minimal impact on resource consumption.
It's a balancing act.
``

To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/8581