zlib compress bit-identical output on different archs - goal or non-goal?

[Georg Sauthoff]

Hello,

during packaging of a new python-img2pdf version I noticed that 2
unit-test cases fail on aarch64.

This is caused by zlib.compress() yielding different output on aarch64
and x86-64. Which triggers the failure because the unit-test case just
compares the result after compression (it's a small PDF that contains a
compressed image - and that PDF is compared against a reference PDF).

I checked the aarch64 compress output and it's valid zlib data, i.e. I can
decompress it on both aarch64 and x86-64 and get the same decompression
result (i.e. the input image data).

As far as I understand RFC 1951, it allows for different implementations
to yield differing compress output bit-streams - as long as other
requirements are met and uncompress(compress(x)) == x (in any
combination of implementations).

Funnily, the img2pdf author uses Debian, and Debian aarch64
zlib.compress() *does* yield bit-identical output (to x86-64). Perhaps
the deviating Fedora aarch64 behaviour is due to different compile
options/patches (e.g. some Neon optimization or something like that).

The img2pdf author isn't really convinced that zlib may produce
different output on different architectures:

https://gitlab.mister-muffin.de/josch/img2pdf/issues/51

Any thoughts?

Best regards
Georg
-- 
'I have long really held the opinion that the amount of noise
which any one can bear undisturbed stands in inverse proportion
to his mental capacity, and therefore may be regarded as a pretty
fair measure of it.' (Arthur Schopenhauer, The World As Will And
Idea. Vol. 2, p.199, 1883)

[Zbigniew Jędrzejewski-Szmek]

On Sun, Dec 09, 2018 at 10:38:10AM +0100, Georg Sauthoff wrote:
>  Hello,
>  
>  during packaging of a new python-img2pdf version I noticed that 2
>  unit-test cases fail on aarch64.
>  
>  This is caused by zlib.compress() yielding different output on aarch64
>  and x86-64. Which triggers the failure because the unit-test case just
>  compares the result after compression (it's a small PDF that contains a
>  compressed image - and that PDF is compared against a reference PDF).
>  
>  I checked the aarch64 compress output and it's valid zlib data, i.e. I can
>  decompress it on both aarch64 and x86-64 and get the same decompression
>  result (i.e. the input image data).
>  
>  As far as I understand RFC 1951, it allows for different implementations
>  to yield differing compress output bit-streams - as long as other
>  requirements are met and uncompress(compress(x)) == x (in any
>  combination of implementations).
>  
>  Funnily, the img2pdf author uses Debian, and Debian aarch64
>  zlib.compress() *does* yield bit-identical output (to x86-64). Perhaps
>  the deviating Fedora aarch64 behaviour is due to different compile
>  options/patches (e.g. some Neon optimization or something like that). 
IMO there's two bugs that should be independently fixed:
- img2pdf should not assume bit-indentical output since zlib 
  guarantee that. It *would* be convenient to assume that, but
  it clearly doesn't work.

- More of an RFE, but I'd really try to make zlib output identical on
  all architectures, to make things more predictable and
  repeatable. From the bug report, the compression is more efficient
  by one byte sometimes, so that sounds like an optimization
  opportunity.

Zbyszek

> The img2pdf author isn't really convinced that zlib may produce
> different output on different architectures:
> 
> https://gitlab.mister-muffin.de/josch/img2pdf/issues/51
> 
> Any thoughts?
> 
> Best regards
> Georg
> -- 
> 'I have long really held the opinion that the amount of noise
> which any one can bear undisturbed stands in inverse proportion
> to his mental capacity, and therefore may be regarded as a pretty
> fair measure of it.' (Arthur Schopenhauer, The World As Will And
> Idea. Vol. 2, p.199, 1883)
> _______________________________________________
> devel mailing list -- devel(a)lists.fedoraproject.org
> To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
https://lists.fedoraproject.org/archives/list/[e-mail]

[Pavel Raiskup]

On Sunday, December 9, 2018 10:38:10 AM CET Georg Sauthoff wrote:
>  Funnily, the img2pdf author uses Debian, and Debian aarch64
>  zlib.compress() *does* yield bit-identical output (to x86-64). Perhaps
>  the deviating Fedora aarch64 behaviour is due to different compile
>  options/patches (e.g. some Neon optimization or something like that). 
This is because we have downstream-only patches related to that
architecture, applied/maintained by Peter Robinson and Jeremy Linton
(both are in CC).  It caused problems at least in qpdf, git and R
components so far.

I'm not in favor maintaining the downstream patches (at least as long as
upstream considers this not important enough to even react).  So for now,
please consult with Peter/Jeremy.

Pavel

[Kevin Fenzi]

On 12/9/18 1:38 AM, Georg Sauthoff wrote:
>  Hello,
>  
>  during packaging of a new python-img2pdf version I noticed that 2
>  unit-test cases fail on aarch64. 
...snip...

>  Any thoughts? 
It would be nice to have bit--identical content
on different arches so
that noarch pacakges at least could be hardlinked and take up less
mirror space.

kevin