On Wed, Jun 5, 2019 at 2:11 AM Panu Matilainen <pmatilai(a)redhat.com> wrote:
On 6/5/19 12:53 AM, Chris Murphy wrote:
> At least for small files, and there are many in any distribution,
> using a dictionary very well could improve compression/decompression
> time, compression ratio, more than threads. Adding dictionary support
> would help all the single thread hardware, and even the builders when
> zstd -T0 option dictates there's only 1 or 2 threads available. On the
> generic sample set, it's functionally like getting 4 threads on speed,
> and even compression ratio goes up by ~3x. But I have no idea how that
> sample set compares to Fedora's files.
>
Yes, but as I mentioned in another email, rpm doesn't compress the files
individually, it compresses them as one big continuous archive. The
dictionary is unlikely to help that (in my quick test yesterday it
actually made it worse)
Sorry about that I missed it. The --long/windowLog option sounds interesting.
I found this on HN today. While xz is not expressly being used within
Fedora/Red Hat packaging in an archive context, it does seem to have
quite a lot of other potential problems. But I have no idea what
lurking liabilities zstd will have.
http://lzip.nongnu.org/xz_inadequate.html
Tangentially, I think there is room for improvement with LiveOS
delivery, which right now is doing something pathological I haven't
been able to figure out compared to other distro LiveOS's: 100% CPU
usage reported by one of the /dev/loopN processes during startup and
installation. And as it's a single thread, it's a bottleneck.
Everytime I do an installation on any computer, fans go to the max. I
don't know that this is xz related, but it might be perturbing things
because decompression is so processor intensive. I'll start a separate
thread.
--
Chris Murphy