It was a long time since the last message in this change proposal.
Recently I was working to reduce the impact of the increased compression
ratio on the installation image size for Fedora. I have achieved
outstanding results -- working proof of concept. With the following
, not only the
higher compression does not impact the installation time. In certain
cases, the installation time is even reduced. This is because of the
fact the filesystem internal structure aware process is used to install
the system from the SquashFS. The new process also allows for taking
advantage of the multi-core architecture of the system during
installation -- does the decompression on multiple processors in parallel.
The combination of https://github.com/rhinstaller/anaconda/pull/2292
reduce _both_ the image size and the installation time. The installation
time will be reduced in case the system is installed from the SquashFS.
This is the case in Fedora Workstation.
For optimization of the SquashFS, I will work on requesting the support
of the required functionality in the Pungi compose build software.
On 25/01/2020 16:36, Bohdan Khomutskyi wrote:
I opened a request to anaconda team with a draft patch to use
unsquashfs instead of rsync:
That should lower the installation time from Live media.
Adjustments should be made to make this patch work, I was not able to
install the image using this patch. Anaconda installer crashes.
> Could this be read amplification?
> That's probably mitigated with unsquashfs.
That could be read amplification, it will be mitigated with unsquashfs.
The memory usage should not be a problem: xz (1) manual page, states
that 2 MiB of memory required to decompress an archive with 1MiB block
size. My previous observations found that the squashfs is currently
decompressed in single thread. I welcome additional testing, but I
think the memory limit will not be a problem.
> If image size is a significant consideration, then evaluation of erofs
> seems indicated. It promises both significant compression and CPU
> performance. The intended use case is for Android device read-only
> partitions with both limited storage and CPU/power capacity.
I briefly reviewed the document and found that LZ4 is the only
supported algorithm in EROFS, even if EROFS has perfect layout of the
filesystem, it's to good to be true it can outperform the XZ in
compression ratio tests. The difference for SquashFS LZ4hc 1M vs XZ 1M
On Fri, Jan 24, 2020 at 2:17 AM Chris Murphy <lists(a)colorremedies.com
On Mon, Jan 20, 2020 at 1:38 AM Bohdan Khomutskyi
<bkhomuts(a)redhat.com <mailto:firstname.lastname@example.org>> wrote:
> In my previous message, I mentioned that CPU is underutilized
during installation. I haven't investigated further why, but I
suspect it's due to the inefficiency caused by the usage of the
loop device and/or inefficiency in the rsync itself.
Could this be read amplification?
This paper on erofs suggests read amplification can be a significant
side effect with squashfs. It could be exacerbated with random reads,
and I expect it gets worse with larger block size. That's probably
mitigated with unsquashfs.
Specifically page 4, 2nd paragraph.
This also makes me wonder about the memory consumption effect of a 1M
block size, especially for Fedora ARM where it looks like Raspberry Pi
Most of the ARM images are raw.xz but some are bootable ISOs, dvd and
netinstall. And those contain a squashfs sysroot. Even if there's no
out of memory problem, it could result in paging. All ISOs setup
swap-on-ZRAM these days, lives, DVD, and netinstall. I think the ARM
case needs testing before committing to 1M block size across all ISOs,
or implementing changes in Fedora release engineering.
Bohdan Khomutskyi, RHCE
Release configuration management engineer, PnT DevOps
Red Hat Czech s.r.o
T: +420532270289 IRC: bkhomuts