6:21 p.m.
When building a normal arch package with noarch subpackages, would it be
acceptable in Fedora if only *one* of the arches actually took
responsibility to build those noarch subpackages?
Context: the "rust" package normally builds arch-specific packages, like
"rust" for rustc, rustdoc, and their libraries, and "rust-std-static"
for the standard library static archives (*.rlib). Since Rust has no
stable ABI, these are closely tied together, so they always need to be
built from the same srpm.
I've had requests to include cross-target std builds for wasm[1] and
mingw[2] too. These would most naturally be noarch subpackages,
something like rust-std-static-$TRIPLE.noarch.rpm, and for the same ABI
reasons it should be built as part of the same rust srpm.
[1] https://src.fedoraproject.org/rpms/rust/pull-request/6
[2] https://bugzilla.redhat.com/show_bug.cgi?id=1762856
The problem is that those cross-target libraries built by two different
host arches will have different metadata hashes in the filenames,
because the hash includes the full "rustc -Vv" version output, including
the host triple. Koji will reject this when the overlapping noarch
packages are compared, since the file list doesn't match. The libraries
are still perfectly usable by rustc from a different host arch though.
So my question/proposal is, what if I just build those cross-target
noarch subpackages from a single host (probably x86_64)? My reading of
the koji's "check_noarch_rpms" seems like it would be OK with this, as
there just wouldn't be any duplicates to rpmdiff. I worry that it would
be against policy somehow though...
Another alternative is to try to remove the host information from the
metadata hash, which I've already started upstream[3], but I'm not sure
alleviate their concerns about caching and such.
[3] https://github.com/rust-lang/cargo/pull/7873
Worst case, I could just build them as arch'ed subpackages, specific to
the host which compiled them.
Thoughts?
8:17 p.m.
On Tue, Feb 11, 2020 at 7:22 PM Josh Stone <jistone(a)redhat.com> wrote:
When building a normal arch package with noarch subpackages, would it be
acceptable in Fedora if only *one* of the arches actually took
responsibility to build those noarch subpackages?
Context: the "rust" package normally builds arch-specific packages, like
"rust" for rustc, rustdoc, and their libraries, and
"rust-std-static"
for the standard library static archives (*.rlib). Since Rust has no
stable ABI, these are closely tied together, so they always need to be
built from the same srpm.
I've had requests to include cross-target std builds for wasm[1] and
mingw[2] too. These would most naturally be noarch subpackages,
something like rust-std-static-$TRIPLE.noarch.rpm, and for the same ABI
reasons it should be built as part of the same rust srpm.
[1] https://src.fedoraproject.org/rpms/rust/pull-request/6
[2] https://bugzilla.redhat.com/show_bug.cgi?id=1762856
The problem is that those cross-target libraries built by two different
host arches will have different metadata hashes in the filenames,
because the hash includes the full "rustc -Vv" version output, including
the host triple. Koji will reject this when the overlapping noarch
packages are compared, since the file list doesn't match. The libraries
are still perfectly usable by rustc from a different host arch though.
So my question/proposal is, what if I just build those cross-target
noarch subpackages from a single host (probably x86_64)? My reading of
the koji's "check_noarch_rpms" seems like it would be OK with this, as
there just wouldn't be any duplicates to rpmdiff. I worry that it would
be against policy somehow though...
Another alternative is to try to remove the host information from the
metadata hash, which I've already started upstream[3], but I'm not sure
alleviate their concerns about caching and such.
[3] https://github.com/rust-lang/cargo/pull/7873
Worst case, I could just build them as arch'ed subpackages, specific to
the host which compiled them.
Thoughts?
My instinct is that this wouldn't work, but I'm not certain. Have you
tried this change with a scratch build? Scratch builds run the same
checks that normal builds do, and would be a good way to verify if
your theory is true.
That said, this is really weird and I'd really be much more
comfortable if this was fixed properly upstream first before you do
this, as you've already started working on.
--
真実はいつも一つ！/ Always, there's only one truth!
9:26 a.m.
Neal Gompa wrote:
My instinct is that this wouldn't work, but I'm not certain.
Have you
tried this change with a scratch build? Scratch builds run the same
checks that normal builds do, and would be a good way to verify if
your theory is true.
%ifarch-ing noarch subpackages (note: noarch SUBpackages of arch-dependent
packages) actually works and does the right thing in Koji. (Koji will still
copy them to all the architectures, even if they were built only on one of
them.) As far as I know, this was implemented that way to make QEMU
firmwares work (which are built on and for a specific architecture, and then
shipped as noarch packages for all of them so that the architecture can be
emulated).
Back when we had secondary Koji instances, the secondary architecture people
used to complain about that practice because those noarch subpackages would
then be missing on their Koji instances. But now that we build alternative
architectures on the primary Koji, I do not see a good reason to not %ifarch
the noarch subpackages, at least in the cases where it works around a known
bogus comparison failure. (In the other cases, you may still want Koji to
actually do that comparison as a form of automated QA, even if it is
technically a waste of resources.)
Building, e.g., noarch documentation subpackages only on a fast architecture
such as x86_64 also helps speeding up builds on slower architectures such as
armv7hl, without actually affecting their users (as per the first
paragraph).
Kevin Kofler
11:49 a.m.
On 2/13/20 7:26 AM, Kevin Kofler wrote:
Neal Gompa wrote:
> My instinct is that this wouldn't work, but I'm not certain. Have you
> tried this change with a scratch build? Scratch builds run the same
> checks that normal builds do, and would be a good way to verify if
> your theory is true.
OK, I ran a scratch build where only x86_64 built the noarch wasm
target. I can't tell if koji actually ran the checks though.
https://koji.fedoraproject.org/koji/taskinfo?taskID=41472922
%ifarch-ing noarch subpackages (note: noarch SUBpackages of
arch-dependent
packages) actually works and does the right thing in Koji. (Koji will still
copy them to all the architectures, even if they were built only on one of
them.) As far as I know, this was implemented that way to make QEMU
firmwares work (which are built on and for a specific architecture, and then
shipped as noarch packages for all of them so that the architecture can be
emulated).
Ah, I suppose you are referring to ipxe? It looks like that package has
always been built this way, as was gpxe before it. Even further back was
etherboot which did this since etherboot-5.4.4-13.fc11, and its
changelog for release 10 said, "Koji now supports noarch subpackages."
So it seems this has worked nearly as long as we've had noarch at all.
Thanks, it's useful to have history/precedent for this kind of design.
Back when we had secondary Koji instances, the secondary architecture
people
used to complain about that practice because those noarch subpackages would
then be missing on their Koji instances. But now that we build alternative
architectures on the primary Koji, I do not see a good reason to not %ifarch
the noarch subpackages, at least in the cases where it works around a known
bogus comparison failure. (In the other cases, you may still want Koji to
actually do that comparison as a form of automated QA, even if it is
technically a waste of resources.)
Building, e.g., noarch documentation subpackages only on a fast architecture
such as x86_64 also helps speeding up builds on slower architectures such as
armv7hl, without actually affecting their users (as per the first
paragraph).
Kevin Kofler
_______________________________________________
devel mailing list -- devel(a)lists.fedoraproject.org
To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
6:44 p.m.
Josh Stone wrote:
On 2/13/20 7:26 AM, Kevin Kofler wrote:
> built only on one of them.) As far as I know, this was implemented that
> way to make QEMU firmwares work (which are built on and for a specific
> architecture, and then shipped as noarch packages for all of them so that
> the architecture can be emulated).
Ah, I suppose you are referring to ipxe?
I was mainly referring to all those BIOS equivalents that are used for
qemu-system-* full system emulation.
Kevin Kofler
2:45 a.m.
Le 2020-02-12 01:21, Josh Stone a écrit :
The problem is that those cross-target libraries built by two
different
host arches will have different metadata hashes in the filenames,
because the hash includes the full "rustc -Vv" version output,
including
the host triple.
koji is doing the right thing. For audits and QA checks a noarch package
must be produced exactly the same ways regardless of the arch of the
builder.
Either recording the triplet is meaningless, and you should inhibit
it(the same way reproducible builds inhibit date recording), or it
serves a purpose, and your builds are archfull by construction.
Regards,
--
Nicolas Mailhot
11:59 a.m.
On 2/12/20 12:45 AM, Nicolas Mailhot wrote:
Le 2020-02-12 01:21, Josh Stone a écrit :
> The problem is that those cross-target libraries built by two different
> host arches will have different metadata hashes in the filenames,
> because the hash includes the full "rustc -Vv" version output,
> including
> the host triple.
koji is doing the right thing. For audits and QA checks a noarch package
must be produced exactly the same ways regardless of the arch of the
builder.
Either recording the triplet is meaningless, and you should inhibit
it(the same way reproducible builds inhibit date recording), or it
serves a purpose, and your builds are archfull by construction.
It's not so black and white. In theory, the only thing that should
matter is the target triple, but the metadata also hashes the metadata
of all its build dependencies. That in turn may include procedural
macros (essentially compiler plugins) which are host binaries. The end
result for the target should be the same, but it's a real concern for
upstream that ignoring the host would break caching that relies on
filename differences.
1:26 p.m.
Le jeudi 13 février 2020 à 09:59 -0800, Josh Stone a écrit :
It's not so black and white. In theory, the only thing that should
matter is the target triple, but the metadata also hashes the
metadata
of all its build dependencies. That in turn may include procedural
macros (essentially compiler plugins) which are host binaries. The
end
result for the target should be the same, but it's a real concern for
upstream that ignoring the host would break caching that relies on
filename differences.
It is so black and white. If you can not produce bit-perfect identical
builds, don’t try to make the result noarch.
Neither koji, nor auditors, nor other third parties, can be expected to
check binary differences for you, and identify what differences matter
and what differences do not matter.
Neither koji, nor auditors, nor other third parties, can be expected to
trust that you did this checking for every single build that differs.
The *only* sane check strategy is a black and white equal/inequal test.
And you can not handwave those concerns aways because you feel your
builds should be noarch (feeling is not implementable in software).
When you write
it's a real concern for upstream that ignoring the host would
break
caching
you are writing that upstream thinks the result is archful (because if
it actually were noarch, it would be perfectly fine to substitute one
build output for another, and there would be no caching concerns).
The proof is in the pudding. noarch → binary differences do not matter,
they can be removed during the build process, and koji will be happy.
archful → removing differences will break something.
And, removing differences does not necessarily mean removing things.
The reproducible build project didn’t remove timestamps. It just made
sure the same timestamp would be used for all builds.
Regards,
--
Nicolas Mailhot
2 p.m.
On 2/13/20 11:26 AM, Nicolas Mailhot wrote:
Le jeudi 13 février 2020 à 09:59 -0800, Josh Stone a écrit :
>
> It's not so black and white. In theory, the only thing that should
> matter is the target triple, but the metadata also hashes the
> metadata
> of all its build dependencies. That in turn may include procedural
> macros (essentially compiler plugins) which are host binaries. The
> end
> result for the target should be the same, but it's a real concern for
> upstream that ignoring the host would break caching that relies on
> filename differences.
It is so black and white. If you can not produce bit-perfect identical
builds, don’t try to make the result noarch.
AIUI, there's no requirement that noarch has to be perfectly
reproducible -- Koji explicitly ignores file size and checksum.
Neither koji, nor auditors, nor other third parties, can be expected
to
check binary differences for you, and identify what differences matter
and what differences do not matter.
Neither koji, nor auditors, nor other third parties, can be expected to
trust that you did this checking for every single build that differs.
The *only* sane check strategy is a black and white equal/inequal test.
And you can not handwave those concerns aways because you feel your
builds should be noarch (feeling is not implementable in software).
I'm not waving it away. Believe it or not, I actually do understand the
impact here.
When you write
> it's a real concern for upstream that ignoring the host would break
> caching
you are writing that upstream thinks the result is archful (because if
it actually were noarch, it would be perfectly fine to substitute one
build output for another, and there would be no caching concerns).
If you read that upstream PR, you might understand my comment about
caching better, and see that I'm also trying to find a better solution
for those concerns.
The proof is in the pudding. noarch → binary differences do not
matter,
they can be removed during the build process, and koji will be happy.
archful → removing differences will break something.
And, removing differences does not necessarily mean removing things.
The reproducible build project didn’t remove timestamps. It just made
sure the same timestamp would be used for all builds.
Regards,
3:55 p.m.
Le jeudi 13 février 2020 à 12:00 -0800, Josh Stone a écrit :
On 2/13/20 11:26 AM, Nicolas Mailhot wrote:
> Le jeudi 13 février 2020 à 09:59 -0800, Josh Stone a écrit :
> >
> It is so black and white. If you can not produce bit-perfect
> identical
> builds, don’t try to make the result noarch.
AIUI, there's no requirement that noarch has to be perfectly
reproducible -- Koji explicitly ignores file size and checksum.
Right, out infra has some loopholes right now, they allow the kind of
breakage that the reproducible build people try to erradicate
Anyway, sorry about the tone, I should not answer while battling
upstream breakage.
--
Nicolas Mailhot
10:29 a.m.
On Tue, 2020-02-11 at 16:21 -0800, Josh Stone wrote:
Another alternative is to try to remove the host information from
the
metadata hash, which I've already started upstream[3], but I'm not sure
alleviate their concerns about caching and such.
[3] https://github.com/rust-lang/cargo/pull/7873
Worst case, I could just build them as arch'ed subpackages, specific to
the host which compiled them.
I'd lean in favor of removing at least the architecture part of the
host triple from the hash. koji is still going to build those noarch
packages on every architecture and compare them, so if you do remove
the arch from the hash and they all compare equal, it can't have
mattered which arch you built it from.
- ajax
12:02 p.m.
On 2/12/20 8:29 AM, Adam Jackson wrote:
On Tue, 2020-02-11 at 16:21 -0800, Josh Stone wrote:
> Another alternative is to try to remove the host information from the
> metadata hash, which I've already started upstream[3], but I'm not sure
> alleviate their concerns about caching and such.
>
> [3] https://github.com/rust-lang/cargo/pull/7873
>
> Worst case, I could just build them as arch'ed subpackages, specific to
> the host which compiled them.
I'd lean in favor of removing at least the architecture part of the
host triple from the hash. koji is still going to build those noarch
packages on every architecture and compare them, so if you do remove
the arch from the hash and they all compare equal, it can't have
mattered which arch you built it from.
FWIW, I don't think Koji actually cares if the files are *identical*, as
long as the file list is the same. The rpmdiff call says:
# ignore differences in file size, md5sum, and mtime
# (files may have been generated at build time and contain
# embedded dates or other insignificant differences)
d = koji.rpmdiff.Rpmdiff(joinpath(basepath, first_rpm),
joinpath(basepath, other_rpm), ignore='S5TN')
1532
days inactive
1535
days old
11 comments
5 participants
participants (5)
-
Adam Jackson -
Josh Stone -
Kevin Kofler -
Neal Gompa -
Nicolas Mailhot