On Mon, Mar 25, 2024 at 04:50:28PM -0400, Neal Gompa wrote:
On Mon, Mar 25, 2024 at 4:40 PM Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl wrote:
On Mon, Mar 25, 2024 at 07:29:09PM +0100, Kevin Kofler via devel wrote:
Daniel Alley wrote:
One more point: createrepo_c uses zstd compression level 10, but the range goes all the way up to level 22. I would oppose making the default much computationally heavier than it is currently, but if spending 20x longer to compress the repo 10% more is desirable to the fedora project, then createrepo_c could perhaps add a the ability to select a compression level.
zstd at high compression levels is very nearly as good at compressing as xz and sometimes better, while remaining much faster to decompress. --
Considering that compression happens once on the server and downloading and decompression happens many times on many computers, I think we should use the highest possible compression level.
+1
Keep in mind we also want to make the compose process faster too, I don't know if it's worth it to spend 20x more time compressing repodata when we keep trying to get back hours and minutes in the compose time.
I wanted to write that the compression times are small enough for this not not matter, but indeed, at the very highest levels, they do become noticable.
$ mv 8e09489af54bbd4ab85470d449f0b0afa4a26fc3eb97c1665c741427bbc8f060-filelists.xml filelists.xml $ time zstd -k -9 filelists.xml filelists.xml : 5.38% ( 863 MiB => 46.4 MiB, filelists.xml.zst) zstd -k -9 4.96s user 0.18s system 103% cpu 4.971 total
$ time zstd -k -21 filelists.xml Warning : compression level higher than max, reduced to 19 filelists.xml : 4.74% ( 863 MiB => 40.9 MiB, filelists.xml.zst) zstd -k -21 321.49s user 0.31s system 99% cpu 5:22.20 total
$ time zstd -k -21 -T8 filelists.xml Warning : compression level higher than max, reduced to 19 filelists.xml : 4.74% ( 863 MiB => 40.9 MiB, filelists.xml.zst) zstd -k -21 -T8 874.57s user 0.70s system 732% cpu 1:59.51 total
$ time xz -k -v 8e09489af54bbd4ab85470d449f0b0afa4a26fc3eb97c1665c741427bbc8f060-filelists.xml 8e09489af54bbd4ab85470d449f0b0afa4a26fc3eb97c1665c741427bbc8f060-filelists.xml (1/1) 100 % 44.3 MiB / 862.9 MiB = 0.051 33 MiB/s 0:26 xz -k -v 196.88s user 0.63s system 749% cpu 26.337 total (This is multithreaded, and gives a compression ratio of 5.14%.)
Dunno, I think anything below a minute should be OK…
Zbyszek