Wiki - https://fedoraproject.org/wiki/Changes/Package_builds_are_expected_to_be_rep... Discussion thread - https://discussion.fedoraproject.org/t/f43-change-proposal-package-builds-ar...
This is a proposed Change for Fedora Linux. This document represents a proposed Change. As part of the Changes process, proposals are publicly announced in order to receive community feedback. This proposal will only be implemented if approved by the Fedora Engineering Steering Committee.
== Summary == Over the last few releases, we changed our build infrastructure to make package builds reproducible. This is enough to reach 90%. The remaining issues need to be fixed in individual packages. After this Change, '''package builds are expected to be reproducible'''. Bugs will be filed against packages when an irreproducibility is detected. The goal is to have no fewer than 99% of package builds reproducible.
A public service with package rebuild statistics and reports for individual packages is made available. (An instance of [https://github.com/kpcyrd/rebuilderd/ rebuilderd].) A script to make local rebuilds of historic koji builds is made available ([https://github.com/keszybz/fedora-repro-build fedora-repro-build]).
== Owner == * ** Name: [[User:Zbyszek| Zbigniew Jędrzejewski-Szmek]] ** Email: zbyszek@in.waw.pl * ** Name: Davide Cavalca ** Email: dcavalca@fedoraproject.org * ** Name: [[User:Jelly| Jelle van der Waa]] ** Email: jvanderw@redhat.com
== Detailed Description ==
We define "reproducible" as when a completely independent rebuild of a koji package produces rpms that are identical except for the build timestamps, signatures, and some associated metadata. The ''payload'' (i.e. packaged files) and important metadata are bit-for-bit identical.
In 2023, [[Changes/ReproducibleBuildsClampMtimes]] added clamping of mtimes to `$SOURCE_DATE_EPOCH` and changed the process to produce `.pyc` files to take `$SOURCE_DATE_EPOCH` into account. In 2024, [[Changes/ReproduciblePackageBuilds]] introduced `add-determinism` into package builds. Along the way, various details in rpm itself and other tools were adjusted to increase reproducibility. With those changes, about 90% of package builds are reproducible. The initial "big" issues that affect all packages are now solved, and what remains are problems that require changes in individual packages.
The next step is to '''ask maintainers to resolve reproducibility issues in their packages'''. The goal is to have 99% of packages reproducible in Fedora. To achieve this, bugzillas will be opened against packages when a rebuild reports differences.
We are aware of some issues that cannot be fixed easily: * Haskell packages are not reproducible when compiled with more than one thread. Upstream is [https://gitlab.haskell.org/ghc/ghc/-/issues/12935 working on the issue], the next release of ghc may resolve the issue completely. * mingw packages have irreproducible debug data. * golang packages have irreproducible debug data ([https://pagure.io/fedora-reproducible-builds/project/issue/15 irreproducibility#15]). * the kernel uses an ephemeral key for module signatures. See [https://lore.kernel.org/lkml/20241225-module-hashes-v1-0-d710ce7a3fd1@weisss...] for a possible solution. * packages that are signed for SecureBoot use a private key (shim, grub2). * Some BuildRequires on srpms are architecture-dependent. This is mostly an artifact of how we prepare the environment for builds. It does not directly affect binary rpms.
We will create tracker bugs for the issues that affect multiple packages (haskell, mingw, golang). We hope that those issues will be resolved either upstream or downstream.
== Feedback ==
== Benefit to Fedora ==
(''The first three paragraphs are copied unchanged from [[Changes/ReproduciblePackageBuilds]].)
Adding determinism (i.e., removing non-determinsim) enables the Fedora community to have confidence that, if given the same source code, build environment, build instructions, and metadata from the build artifacts, any party can recreate copies of the artifacts that are identical except for the signatures and some parts of metadata.
Reproducibility of builds leads to packages of higher quality. It turns out that quite often those irreproducible bits are caused by an error or sloppiness in the code. In particular, any dependence on architecture in noarch packages is almost always unwanted and/or a bug. Test builds that check reproducibility will expose such instances.
Reproducibility of builds makes it easier to develop packages: when a small change is made and a package is rebuilt (in the same environment), then with a reproducible package, the only difference is directly caused by the change. If the package is different every time it is rebuilt, making a comparison is much harder.
Build reproducibility is a topic that is gaining in popularity. Major distributions like Debian, Arch, OpenSUSE, and NixOS are trying to achieve full reproducibility. By making b-r an expectation in Fedora, we avoid driving away people who consider b-r a requirement. We may even attract additional contributors who are interested in this topic, if we achieve better results than other distros. With 90% reproducibility we're on par, with 99% we can be the leader 😀.
== Scope == * Proposal owners: ** Package [https://github.com/keszybz/fedora-repro-build fedora-repro-build] to allow local rebuilds of historical koji builds ** Make [https://github.com/kpcyrd/rebuilderd/ rebuilderd] work with Fedora packages and repos ** Stand up a public rebuilderd instance for Fedora rawhide ** Adjust `add-determinism` to handle new cases, if widespread issues are found and it's possible to handle them in the cleanup phase ** Open bugs against packages when irreproducibilities are detected, initially as a manual process ** Open a pull request for Packaging Guidelines * Other developers: ** Fix irreproducibility issues in individual packages ** Fix tooling issues that affect multiple packages, if possible ** Review the pull request for Packaging Guidelines ** Adjust various packager workflow descriptions in the wiki * Release engineering: [https://pagure.io/releng/issues #Releng issue number]
* Policies and guidelines: ** Packaging Guidelines will be changed to say that packages **should** build reproducibly, and link to our docs ([https://docs.fedoraproject.org/en-US/reproducible-builds/ reproducible builds]) and upstream docs at [https://reproducible-builds.org/documentation reproducible-builds.org].
* Trademark approval: N/A (not needed for this Change)
* Alignment with the Fedora Strategy:
== Upgrade/compatibility impact == N/A
== Early Testing (Optional) ==
We have the tooling in place to do a local rebuild of a koji build and compare the results. Packagers can do such rebuilds and verify the result or fix the issues if any are found.
== How To Test ==
When the public rebuilderd instance is up, maintainers will be able to see reports and diffoscope output for their packages. They can then use this to learn about any issues in their packages.
== User Experience == No user-visible changes. Users ''may'' do local rebuilds and expect that the results are reproducible. == Dependencies ==
== Contingency Plan ==
* Contingency mechanism: All the items in the Scope are additions to the current tooling and infrastructure, so there is no need for a contingency plan for them. If we back out of the whole idea, rather than just postponing it for example, Packaging Guidelines changes would probably need to be reverted and any bugzillas closed. * Contingency deadline: any time * Blocks release? no
== Documentation == * [https://reproducible-builds.org/docs reproducible-builds.org/docs] * https://docs.fedoraproject.org/en-US/reproducible-builds/
== Release Notes == Reproducibility of package builds has improved. TBD% of package builds are now reproducible.
Dne 19. 03. 25 v 7:48 odp. Aoife Moloney via devel-announce napsal(a):
== Scope ==
- Proposal owners:
** Package [https://github.com/keszybz/fedora-repro-build fedora-repro-build] to allow local rebuilds of historical koji builds
This
https://rpm-software-management.github.io/mock/feature-hermetic-builds
may help you.
** Make [https://github.com/kpcyrd/rebuilderd/ rebuilderd] work with Fedora packages and repos Stand up a public rebuilderd instance for Fedora rawhide
Yet another build system in fedora-infra? Can the rebuilderd be adjusted that it submits builds to Copr?
On Thu, Mar 20, 2025 at 08:04:57AM +0100, Miroslav Suchý wrote:
Dne 19. 03. 25 v 7:48 odp. Aoife Moloney via devel-announce napsal(a):
== Scope ==
- Proposal owners:
** Package [https://github.com/keszybz/fedora-repro-build fedora-repro-build] to allow local rebuilds of historical koji builds
This
https://rpm-software-management.github.io/mock/feature-hermetic-builds
Yes, thanks for the link. I think this didn't exist when I started working on my script to do rebuilds, so I just gather the rpms reported by koji to have been used for the original build and call 'createrepo_c' on the directory and point mock to that. This works fine… But having support for using a lockfile natively in mock is nice.
Though, the process described in that link seems incomplete.
# we want to build this package srpm=your-package.src.rpm
Where does the $srpm come from? The process of creating the srpm from dist-git involves running the spec, i.e. already calling "untrusted" code. How is that part handled?
** Make [https://github.com/kpcyrd/rebuilderd/ rebuilderd] work with Fedora packages and repos Stand up a public rebuilderd instance for Fedora rawhide
Yet another build system in fedora-infra? Can the rebuilderd be adjusted that it submits builds to Copr?
That is a good question. Does copr support running a build with a specified "$lockfile", i.e. with a fixed set of nvra's? It would also have to download those rpms from koji, since they are not available on mirrors.
rebuilderrd is an complete project that does things in its own way. It turned out to be not-very-complicated to teach it to use a build script that does the build locally. But talking to an external service, possibly asynchronously, might be more complicated. Maybe Jelle has more thoughts on this.
Zbyszek
Dne 20. 03. 25 v 9:39 dop. Zbigniew Jędrzejewski-Szmek napsal(a):
https://rpm-software-management.github.io/mock/feature-hermetic-builds
Yes, thanks for the link. I think this didn't exist when I started working on my script to do rebuilds, so I just gather the rpms reported by koji to have been used for the original build and call 'createrepo_c' on the directory and point mock to that. This works fine… But having support for using a lockfile natively in mock is nice.
Yes. This is several months old.
Pavel added it during his work on Konflux. And this feature will be used in Konflux.
Though, the process described in that link seems incomplete.
Can you elaborate? Or you can reach Pavel off-list to discuss the usage.
# we want to build this package srpm=your-package.src.rpm
Where does the $srpm come from? The process of creating the srpm from dist-git involves running the spec, i.e. already calling "untrusted" code. How is that part handled?
"mock --buildsrpm" or "fedpkg srpm". I am not sure. In Konflux creating SRPM is separate step. So SRPM is granted from previous step.
Pavel may provide more details later when he recovers from illness.
Yet another build system in fedora-infra? Can the rebuilderd be adjusted that it submits builds to Copr?
That is a good question. Does copr support running a build with a specified "$lockfile", i.e. with a fixed set of nvra's? It would also have to download those rpms from koji, since they are not available on mirrors.
No. Copr does not support it, because no one asked for that and there was no use-case. When you open RFE we can discuss what it would requires and when we can put it in our plan.
Or you can use Konflux that will use this natively. But Konflux is very immature now.
rebuilderrd is an complete project that does things in its own way. It turned out to be not-very-complicated to teach it to use a build script that does the build locally.
Building 30k packages locally? With frequent updates for lots of them. Does it scale?
On Thu, Mar 20, 2025 at 10:10:23AM +0100, Miroslav Suchý wrote:
Dne 20. 03. 25 v 9:39 dop. Zbigniew Jędrzejewski-Szmek napsal(a):
https://rpm-software-management.github.io/mock/feature-hermetic-builds
Yes, thanks for the link. I think this didn't exist when I started working on my script to do rebuilds, so I just gather the rpms reported by koji to have been used for the original build and call 'createrepo_c' on the directory and point mock to that. This works fine… But having support for using a lockfile natively in mock is nice.
Yes. This is several months old.
Pavel added it during his work on Konflux. And this feature will be used in Konflux.
Though, the process described in that link seems incomplete.
Can you elaborate? Or you can reach Pavel off-list to discuss the usage.
# we want to build this package srpm=your-package.src.rpm
Where does the $srpm come from? The process of creating the srpm from dist-git involves running the spec, i.e. already calling "untrusted" code. How is that part handled?
"mock --buildsrpm" or "fedpkg srpm". I am not sure. In Konflux creating SRPM is separate step. So SRPM is granted from previous step.
This "separate step" is what I was talking about. That step already requires executing some untrusted code, so it should be sandboxed.
Pavel may provide more details later when he recovers from illness.
Yet another build system in fedora-infra? Can the rebuilderd be adjusted that it submits builds to Copr?
That is a good question. Does copr support running a build with a specified "$lockfile", i.e. with a fixed set of nvra's? It would also have to download those rpms from koji, since they are not available on mirrors.
No. Copr does not support it, because no one asked for that and there was no use-case. When you open RFE we can discuss what it would requires and when we can put it in our plan.
Can we discuss it here, at least in the rough outline? As I wrote elsewhere in the thread, we currently ask koji for getBuildrootListing() and use the resulting list of rpm nvra's to populate a repo for mock. Can copr do this for us?
Building 30k packages locally? With frequent updates for lots of them. Does it scale?
Yeah, it scales surprisingly well. We skip %check, and for now only build on two architectures, so that saves a lot of time. One big issue was https://github.com/rpm-software-management/mock/issues/1394. This slows downs parallel builds considerably. I never found the time to figure this one out.
Zbyszek
Dne 20. 03. 25 v 10:26 dop. Zbigniew Jędrzejewski-Szmek napsal(a):
Can we discuss it here, at least in the rough outline?
Sure. I think Pavel will anwer it better. He should be back on Monday.
Building 30k packages locally? With frequent updates for lots of them. Does it scale?
Yeah, it scales surprisingly well. We skip %check, and for now only build on two architectures, so that saves a lot of time.
Interresting. While this is likely technically incorrect because %check may modify the buildroot
https://github.com/rpm-software-management/mock/issues/1352
it may actually reveal such cases where %check do something with the buildroot.
On Thu, Mar 20, 2025 at 11:16:55AM +0100, Miroslav Suchý wrote:
Dne 20. 03. 25 v 10:26 dop. Zbigniew Jędrzejewski-Szmek napsal(a):
Can we discuss it here, at least in the rough outline?
Sure. I think Pavel will anwer it better. He should be back on Monday.
Building 30k packages locally? With frequent updates for lots of them. Does it scale?
Yeah, it scales surprisingly well. We skip %check, and for now only build on two architectures, so that saves a lot of time.
Interresting. While this is likely technically incorrect because %check may modify the buildroot
https://github.com/rpm-software-management/mock/issues/1352
it may actually reveal such cases where %check do something with the buildroot.
Yes, it does indeed happen, but I think we should treat that as a bug. (I recall vaguely one or two cases where I found that skipping %check made a difference, some timestamps were modified or something like that. But those cases are generally easy to fix.)
Zbyszek
On Thu, Mar 20, 2025 at 4:40 AM Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl wrote:
On Thu, Mar 20, 2025 at 08:04:57AM +0100, Miroslav Suchý wrote:
Dne 19. 03. 25 v 7:48 odp. Aoife Moloney via devel-announce napsal(a):
== Scope ==
- Proposal owners:
** Package [https://github.com/keszybz/fedora-repro-build fedora-repro-build] to allow local rebuilds of historical koji builds
This
https://rpm-software-management.github.io/mock/feature-hermetic-builds
Yes, thanks for the link. I think this didn't exist when I started working on my script to do rebuilds, so I just gather the rpms reported by koji to have been used for the original build and call 'createrepo_c' on the directory and point mock to that. This works fine… But having support for using a lockfile natively in mock is nice.
Though, the process described in that link seems incomplete.
# we want to build this package srpm=your-package.src.rpm
Where does the $srpm come from? The process of creating the srpm from dist-git involves running the spec, i.e. already calling "untrusted" code. How is that part handled?
It's also not *really* hermetic either. Hermetic builds require true isolation and there is no Mock backend that provides that right now. All it does is let you pre-download the build environment and replay it multiple times. Koji can do that too, and yet nobody calls it hermetic either because chroots/containers aren't good enough for that.
-- 真実はいつも一つ！/ Always, there's only one truth!
On Thu, Mar 20, 2025 at 06:23:53AM -0400, Neal Gompa wrote:
On Thu, Mar 20, 2025 at 4:40 AM Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl wrote:
On Thu, Mar 20, 2025 at 08:04:57AM +0100, Miroslav Suchý wrote:
Dne 19. 03. 25 v 7:48 odp. Aoife Moloney via devel-announce napsal(a):
== Scope ==
- Proposal owners:
** Package [https://github.com/keszybz/fedora-repro-build fedora-repro-build] to allow local rebuilds of historical koji builds
This
https://rpm-software-management.github.io/mock/feature-hermetic-builds
Yes, thanks for the link. I think this didn't exist when I started working on my script to do rebuilds, so I just gather the rpms reported by koji to have been used for the original build and call 'createrepo_c' on the directory and point mock to that. This works fine… But having support for using a lockfile natively in mock is nice.
Though, the process described in that link seems incomplete.
# we want to build this package srpm=your-package.src.rpm
Where does the $srpm come from? The process of creating the srpm from dist-git involves running the spec, i.e. already calling "untrusted" code. How is that part handled?
It's also not *really* hermetic either. Hermetic builds require true isolation and there is no Mock backend that provides that right now. All it does is let you pre-download the build environment and replay it multiple times. Koji can do that too, and yet nobody calls it hermetic either because chroots/containers aren't good enough for that.
OK, so what do you mean by "true isolation" then? I'd say that once the build in a container with no network access, that is good enough for me :)
Zbyszek
On Thu, Mar 20, 2025 at 6:33 AM Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl wrote:
On Thu, Mar 20, 2025 at 06:23:53AM -0400, Neal Gompa wrote:
On Thu, Mar 20, 2025 at 4:40 AM Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl wrote:
On Thu, Mar 20, 2025 at 08:04:57AM +0100, Miroslav Suchý wrote:
Dne 19. 03. 25 v 7:48 odp. Aoife Moloney via devel-announce napsal(a):
== Scope ==
- Proposal owners:
** Package [https://github.com/keszybz/fedora-repro-build fedora-repro-build] to allow local rebuilds of historical koji builds
This
https://rpm-software-management.github.io/mock/feature-hermetic-builds
Yes, thanks for the link. I think this didn't exist when I started working on my script to do rebuilds, so I just gather the rpms reported by koji to have been used for the original build and call 'createrepo_c' on the directory and point mock to that. This works fine… But having support for using a lockfile natively in mock is nice.
Though, the process described in that link seems incomplete.
# we want to build this package srpm=your-package.src.rpm
Where does the $srpm come from? The process of creating the srpm from dist-git involves running the spec, i.e. already calling "untrusted" code. How is that part handled?
It's also not *really* hermetic either. Hermetic builds require true isolation and there is no Mock backend that provides that right now. All it does is let you pre-download the build environment and replay it multiple times. Koji can do that too, and yet nobody calls it hermetic either because chroots/containers aren't good enough for that.
OK, so what do you mean by "true isolation" then? I'd say that once the build in a container with no network access, that is good enough for me :)
A microvm, basically. Running as a VM prevents host characteristics from leaking into the build environment and messing things up. It also makes it so some test suites for packages will work correctly without breaking the builder.
On Thu, Mar 20, 2025 at 06:36:09AM -0400, Neal Gompa wrote:
On Thu, Mar 20, 2025 at 6:33 AM Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl wrote:
On Thu, Mar 20, 2025 at 06:23:53AM -0400, Neal Gompa wrote:
On Thu, Mar 20, 2025 at 4:40 AM Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl wrote:
On Thu, Mar 20, 2025 at 08:04:57AM +0100, Miroslav Suchý wrote:
Dne 19. 03. 25 v 7:48 odp. Aoife Moloney via devel-announce napsal(a):
== Scope ==
- Proposal owners:
** Package [https://github.com/keszybz/fedora-repro-build fedora-repro-build] to allow local rebuilds of historical koji builds
This
https://rpm-software-management.github.io/mock/feature-hermetic-builds
Yes, thanks for the link. I think this didn't exist when I started working on my script to do rebuilds, so I just gather the rpms reported by koji to have been used for the original build and call 'createrepo_c' on the directory and point mock to that. This works fine… But having support for using a lockfile natively in mock is nice.
Though, the process described in that link seems incomplete.
# we want to build this package srpm=your-package.src.rpm
Where does the $srpm come from? The process of creating the srpm from dist-git involves running the spec, i.e. already calling "untrusted" code. How is that part handled?
It's also not *really* hermetic either. Hermetic builds require true isolation and there is no Mock backend that provides that right now. All it does is let you pre-download the build environment and replay it multiple times. Koji can do that too, and yet nobody calls it hermetic either because chroots/containers aren't good enough for that.
OK, so what do you mean by "true isolation" then? I'd say that once the build in a container with no network access, that is good enough for me :)
A microvm, basically. Running as a VM prevents host characteristics from leaking into the build environment and messing things up. It also makes it so some test suites for packages will work correctly without breaking the builder.
Meh. It's _a_ solution, but I don't think it's a great solution, because I think the problem cannot be really solved in that way.
If the package is taking into account stray characteristics of the host, then this is a problem in the package build, something to be fixed there. You can try to make the build environment more uniform, but that only goes so far. The package can always start depending on some little detail (number of CPUs, file system, µarch, amount of memory, the list of devices, etc, etc, the possibilities are endless), and then this pseudorandom characteristic would become locked and the microvm cannot be changed. This is a race to the bottom that we cannot win.
Also, if "tests suite for packages […] break the builder" then this is a problem to fix in the container environment.
Zbyszek
On Thu, Mar 20, 2025 at 7:14 AM Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl wrote:
On Thu, Mar 20, 2025 at 06:36:09AM -0400, Neal Gompa wrote:
On Thu, Mar 20, 2025 at 6:33 AM Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl wrote:
On Thu, Mar 20, 2025 at 06:23:53AM -0400, Neal Gompa wrote:
On Thu, Mar 20, 2025 at 4:40 AM Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl wrote:
On Thu, Mar 20, 2025 at 08:04:57AM +0100, Miroslav Suchý wrote:
Dne 19. 03. 25 v 7:48 odp. Aoife Moloney via devel-announce napsal(a): > == Scope == > * Proposal owners: > ** Package [https://github.com/keszybz/fedora-repro-build > fedora-repro-build] to allow local rebuilds of historical koji builds
This
https://rpm-software-management.github.io/mock/feature-hermetic-builds
Yes, thanks for the link. I think this didn't exist when I started working on my script to do rebuilds, so I just gather the rpms reported by koji to have been used for the original build and call 'createrepo_c' on the directory and point mock to that. This works fine… But having support for using a lockfile natively in mock is nice.
Though, the process described in that link seems incomplete.
# we want to build this package srpm=your-package.src.rpm
Where does the $srpm come from? The process of creating the srpm from dist-git involves running the spec, i.e. already calling "untrusted" code. How is that part handled?
It's also not *really* hermetic either. Hermetic builds require true isolation and there is no Mock backend that provides that right now. All it does is let you pre-download the build environment and replay it multiple times. Koji can do that too, and yet nobody calls it hermetic either because chroots/containers aren't good enough for that.
OK, so what do you mean by "true isolation" then? I'd say that once the build in a container with no network access, that is good enough for me :)
A microvm, basically. Running as a VM prevents host characteristics from leaking into the build environment and messing things up. It also makes it so some test suites for packages will work correctly without breaking the builder.
Meh. It's _a_ solution, but I don't think it's a great solution, because I think the problem cannot be really solved in that way.
If the package is taking into account stray characteristics of the host, then this is a problem in the package build, something to be fixed there. You can try to make the build environment more uniform, but that only goes so far. The package can always start depending on some little detail (number of CPUs, file system, µarch, amount of memory, the list of devices, etc, etc, the possibilities are endless), and then this pseudorandom characteristic would become locked and the microvm cannot be changed. This is a race to the bottom that we cannot win.
Also, if "tests suite for packages […] break the builder" then this is a problem to fix in the container environment.
Well, the shared kernel puts a lot of limits on things. You can't fix anything as long as you can't make a whole new devfs for each build environment.
And it screws up both package builds and image builds in weird ways. For example, two builds of fish cannot run at the same time in Koji because the test suite will lock up and fail due to shared devfs devices from the host into the container environment. And neither kiwi nor image-builder are able to produce disk images because nspawn doesn't let us do the right thing for loop devices. And the list goes on and on.
On Thu, Mar 20, 2025 at 07:33:22AM -0400, Neal Gompa wrote:
And it screws up both package builds and image builds in weird ways. For example, two builds of fish cannot run at the same time in Koji because the test suite will lock up and fail due to shared devfs devices from the host into the container environment. And neither kiwi nor image-builder are able to produce disk images because nspawn doesn't let us do the right thing for loop devices. And the list goes on and on.
I think it'd be nice to be able to use a VM as a workaround for fragile or broken software… But we shouldn't need to make this the default option.
Zbyszek
Dne 20. 03. 25 v 11:23 dop. Neal Gompa napsal(a):
It's also notreally hermetic either. Hermetic builds require true isolation and there is no Mock backend that provides that right now. All it does is let you pre-download the build environment and replay it multiple times.
OK. We can argue if systemd-nspawn containers with disabled network is good enough isolation or not. :)
What this feature does is that it wrote down all packages used during build (including dynamic buildrequires). And can replay it without the need to download anything from net. So even bootstrap_chroot can be isolate from network. This is why we call it "hermetic".
For the reproducible build POV is important that this feature allows you to re-run build with older package that was used during a build. Despite the newever version of the package being available.
Koji can do that too, and yet nobody calls it hermetic either because chroots/containers aren't good enough for that.
I believe Koji can't do such level of isolation. DNF in bootstrap chroot has to have access to internet in Koji.
On Thu, Mar 20, 2025 at 6:42 AM Miroslav Suchý msuchy@redhat.com wrote:
Dne 20. 03. 25 v 11:23 dop. Neal Gompa napsal(a):
It's also notreally hermetic either. Hermetic builds require true isolation and there is no Mock backend that provides that right now. All it does is let you pre-download the build environment and replay it multiple times.
OK. We can argue if systemd-nspawn containers with disabled network is good enough isolation or not. :)
What this feature does is that it wrote down all packages used during build (including dynamic buildrequires). And can replay it without the need to download anything from net. So even bootstrap_chroot can be isolate from network. This is why we call it "hermetic".
For the reproducible build POV is important that this feature allows you to re-run build with older package that was used during a build. Despite the newever version of the package being available.
Koji can do that too, and yet nobody calls it hermetic either because chroots/containers aren't good enough for that.
I believe Koji can't do such level of isolation. DNF in bootstrap chroot has to have access to internet in Koji.
Only if you use external repos. If you don't, then it can be locked down like the rest of the tasks.
-- 真実はいつも一つ！/ Always, there's only one truth!
On 20/03/2025 09:39, Zbigniew Jędrzejewski-Szmek wrote:
On Thu, Mar 20, 2025 at 08:04:57AM +0100, Miroslav Suchý wrote:
Dne 19. 03. 25 v 7:48 odp. Aoife Moloney via devel-announce napsal(a):
== Scope ==
- Proposal owners:
** Package [https://github.com/keszybz/fedora-repro-build fedora-repro-build] to allow local rebuilds of historical koji builds
<snip>>>> ** Make [https://github.com/kpcyrd/rebuilderd/ rebuilderd] work with
Fedora packages and repos Stand up a public rebuilderd instance for Fedora rawhide
Yet another build system in fedora-infra? Can the rebuilderd be adjusted that it submits builds to Copr?
That is a good question. Does copr support running a build with a specified "$lockfile", i.e. with a fixed set of nvra's? It would also have to download those rpms from koji, since they are not available on mirrors.
rebuilderrd is an complete project that does things in its own way. It turned out to be not-very-complicated to teach it to use a build script that does the build locally.
Rebuilderd isn't much of a build system but more an orchestrator, it scans a Fedora repo's metadata for new packages and queue's them up for building.
Rebuilderd-worker picks up build tassk from the rebuilderd daemon, executes an arbitrary rebuild script with the to be rebuild rpm file and reports back a status to rebuilderd.
Additionally rebuilderd provides an API which includes the reproducibility status of all packages and an optionally diffoscope output of a non-reproducible package. This could be integrated in koschei?
But talking to an external service, possibly asynchronously, might be more complicated. Maybe Jelle has more thoughts on this.
The worker executes any arbitrary script so definitively this can be an API call to for example koji. This would complicate things a bit when the network is unstable.
The main benefit of having rebuilderd being operated separate is that multiple parties could rebuild Fedora and their results can be compared.
Arch Linux for example has multiple independent rebuilders and a simple tool arch-repro-status to inspect the reproducibility of your system installed packages by querying different rebuilders. [1] [2]
[1] https://wiki.archlinux.org/title/Rebuilderd#Package_rebuilders [2] https://gitlab.archlinux.org/archlinux/arch-repro-status
On Thu, Mar 20, 2025, at 3:27 PM, Jelle van der Waa wrote:
Rebuilderd isn't much of a build system but more an orchestrator, it scans a Fedora repo's metadata for new packages and queue's them up for building.
You could also listen to the appropriate channel(s) in fedmsg, that way you wouldn't need to scan repositories :)
The worker executes any arbitrary script so definitively this can be an API call to for example koji. This would complicate things a bit when the network is unstable.
You're probably already pulling stuff (RPMs, dist-git inputs, etc) from that same network.
For what it's worth I'm not super set on having builds be performed by koji. I think reproducibility *across* build systems is a nice thing.
What I would really like is for this to integrate well with Fedora's pre- existing tools and things so it has the highest chance of people actually using it and it doing what people expect :)
Hello,
We are aware of some issues that cannot be fixed easily:
- Haskell packages are not reproducible when compiled with more than
one thread. Upstream is [https://gitlab.haskell.org/ghc/ghc/-/issues/12935 working on the issue], the next release of ghc may resolve the issue completely.
Does this issue affect only Haskell packages? E.g. openssl is also built
with -j8 and switching back to single thread will significantly increase compilation time
On Thu, Mar 20, 2025 at 09:00:09AM +0100, Dmitry Belyavskiy wrote:
Hello,
We are aware of some issues that cannot be fixed easily:
- Haskell packages are not reproducible when compiled with more than
one thread. Upstream is [https://gitlab.haskell.org/ghc/ghc/-/issues/12935 working on the issue], the next release of ghc may resolve the issue completely.
Does this issue affect only Haskell packages? E.g. openssl is also built with -j8 and switching back to single thread will significantly increase compilation time
Yes.
Zbyszek
On Wed, Mar 19, 2025 at 06:48:21PM +0000, Aoife Moloney via devel-announce wrote:
== Scope ==
- Proposal owners:
** Package [https://github.com/keszybz/fedora-repro-build fedora-repro-build] to allow local rebuilds of historical koji builds ** Make [https://github.com/kpcyrd/rebuilderd/ rebuilderd] work with Fedora packages and repos
What is rebuilderd using to actually perform the builds ? Can it just be using the regular koji, so that maintainers don't have to learn yet another build tool.
** Stand up a public rebuilderd instance for Fedora rawhide ** Adjust `add-determinism` to handle new cases, if widespread issues are found and it's possible to handle them in the cleanup phase ** Open bugs against packages when irreproducibilities are detected, initially as a manual process
Can we have this run as part of the post-build gating tests, so that we have a single place to look for test results, along with the option for maintainers to waive reproducibility issues ?
I'm pretty unethusiastic about dealing with yet another standalone web service providing post-build testing.
With regards, Daniel
On Thu, Mar 20, 2025 at 08:18:15AM +0000, Daniel P. Berrangé wrote:
On Wed, Mar 19, 2025 at 06:48:21PM +0000, Aoife Moloney via devel-announce wrote:
== Scope ==
- Proposal owners:
** Package [https://github.com/keszybz/fedora-repro-build fedora-repro-build] to allow local rebuilds of historical koji builds ** Make [https://github.com/kpcyrd/rebuilderd/ rebuilderd] work with Fedora packages and repos
What is rebuilderd using to actually perform the builds ? Can it just be using the regular koji, so that maintainers don't have to learn yet another build tool.
Currently it's using my script to do rebuilds (https://github.com/keszybz/fedora-repro-build/blob/main/koji_rebuild.py). It queries koji for the buildroot contents of the (historic) build, downloads all listed rpms into a temporary directory, calls createrepo_c on this directory, and calls mock with that directory as the only repo.
Using koji to do the build is an interesting idea. koji has createBuildTarget(name, build_tag, dest_tag) and build(src, target), where the build_tag contents specify the rpms that are visible. It also get getBuildrootListing(id), which returns a list of nvra's. What we'd actually want is to connect the two, i.e. do a build using the specified buildroot id. Maybe koji could be taught to do that for us.
That said, as I wrote in a reply about copr, we also want our rebuilds to be as independent as possible. So using koji to do rebuilds would be nice for test packager rebuilds, we'd still want to have rebuilderd doing the rebuilds _not_ through koji.
** Stand up a public rebuilderd instance for Fedora rawhide ** Adjust `add-determinism` to handle new cases, if widespread issues are found and it's possible to handle them in the cleanup phase ** Open bugs against packages when irreproducibilities are detected, initially as a manual process
Can we have this run as part of the post-build gating tests, so that we have a single place to look for test results, along with the option for maintainers to waive reproducibility issues ?
I'm pretty unethusiastic about dealing with yet another standalone web service providing post-build testing.
Yes, I understand this. The same question was asked previously in the other thread and I wrote [1] the following:
Theoretically — yes. In practice — no such plans at this point. Right now, we expect some percentage of the rebuilds to fail, so it’d be too early to gate on this. After some time, once those bugs have been fixed, theoretically we try could try. But this always means a second build, which can be quite slow. (That said, the rebuilds can be done without tests, which are often the slowest part.)
The problem is that if we just rebuild the package a second time in the same build environment, we are not testing much. We really want to build it on a different variant of the same architecture, with a different file system, on a different date, etc. But I don’t think we want to set up two variants of our build environment internally.
Dunno, this could be something to consider in the future, depending on how this initial stage goes.
[1] https://discussion.fedoraproject.org/t/f43-change-proposal-package-builds-ar...
Zbyszek
On Thu, Mar 20, 2025, at 9:18 AM, Daniel P. Berrangé wrote:
Can we have this run as part of the post-build gating tests, so that we have a single place to look for test results, along with the option for maintainers to waive reproducibility issues ?
I'm pretty unethusiastic about dealing with yet another standalone web service providing post-build testing.
Same. It would however be quite nice if it reports in `bodhi`, much like the installability tests though I'm unsure if we can show 'advisory' CI there :)
On Thu, Mar 20, 2025 at 10:13:25AM +0100, Simon de Vlieger wrote:
On Thu, Mar 20, 2025, at 9:18 AM, Daniel P. Berrangé wrote:
Can we have this run as part of the post-build gating tests, so that we have a single place to look for test results, along with the option for maintainers to waive reproducibility issues ?
I'm pretty unethusiastic about dealing with yet another standalone web service providing post-build testing.
Same. It would however be quite nice if it reports in `bodhi`, much like the installability tests though I'm unsure if we can show 'advisory' CI there :)
OK, it sounds like we need to figure out if this is possible. AFAIK, it should be possible to add a non-gating test.
Zbyszek
On Wed, Mar 19, 2025 at 06:48:21PM +0000, Aoife Moloney via devel-announce wrot> * mingw packages have irreproducible debug data.
Anything more information about this? What is it about the "debug data" which is a problem, and can anything be done about it?
I assume this refers to files like /usr/lib/debug/usr/x86_64-w64-mingw32/sys-root/mingw/bin/zlib1.dll.debug which are actually PE files.
Rich.
On Thu, Mar 20, 2025 at 09:57:23AM +0000, Richard W.M. Jones wrote:
- mingw packages have irreproducible debug data.
Anything more information about this? What is it about the "debug data" which is a problem, and can anything be done about it?
I assume this refers to files like /usr/lib/debug/usr/x86_64-w64-mingw32/sys-root/mingw/bin/zlib1.dll.debug which are actually PE files.
Basically, most mingw binaries show irreproducibilities. Random example: $ diffoscope --text-color=always orig/liblerc-4.0.0-8.fc42/mingw64-liblerc-4.0.0-8.fc42.noarch.rpm rebuild/liblerc-4.0.0-8.fc42/rebuild/mingw64-liblerc-4.0.0-8.fc42.noarch.rpm
├── content │ ├── ./usr/x86_64-w64-mingw32/sys-root/mingw/bin/libLerc-4.dll │ │ ├── objdump │ │ │ @@ -26,15 +26,15 @@ │ │ │ MajorImageVersion 0 │ │ │ MinorImageVersion 0 │ │ │ MajorSubsystemVersion 5 │ │ │ MinorSubsystemVersion 2 │ │ │ Win32Version 00000000 │ │ │ SizeOfImage 00594000 │ │ │ SizeOfHeaders 00000600 │ │ │ -CheckSum 005b5e09 │ │ │ +CheckSum 005bac94 │ │ │ Subsystem 00000003 (Windows CUI) │ │ │ DllCharacteristics 00000160 │ │ │ HIGH_ENTROPY_VA │ │ │ DYNAMIC_BASE │ │ │ NX_COMPAT │ │ │ SizeOfStackReserve 0000000000200000 │ │ │ SizeOfStackCommit 0000000000001000 │ │ │ @@ -5759,15 +5759,15 @@ │ │ │ CONTENTS, ALLOC, LOAD, READONLY, DATA │ │ │ 11 .debug_aranges 000014b0 00000002dbe54000 00000002dbe54000 0008be00 2**0 │ │ │ CONTENTS, READONLY, DEBUGGING │ │ │ 12 .debug_info 002eaad4 00000002dbe56000 00000002dbe56000 0008d400 2**0 │ │ │ CONTENTS, READONLY, DEBUGGING │ │ │ 13 .debug_abbrev 00009733 00000002dc141000 00000002dc141000 00378000 2**0 │ │ │ CONTENTS, READONLY, DEBUGGING │ │ │ - 14 .debug_line 000914e2 00000002dc14b000 00000002dc14b000 00381800 2**0 │ │ │ + 14 .debug_line 000914d7 00000002dc14b000 00000002dc14b000 00381800 2**0 │ │ │ CONTENTS, READONLY, DEBUGGING │ │ │ 15 .debug_frame 0000b678 00000002dc1dd000 00000002dc1dd000 00412e00 2**0 │ │ │ CONTENTS, READONLY, DEBUGGING │ │ │ 16 .debug_str 00005ed6 00000002dc1e9000 00000002dc1e9000 0041e600 2**0 │ │ │ CONTENTS, READONLY, DEBUGGING │ │ │ 17 .debug_line_str 00003b6b 00000002dc1ef000 00000002dc1ef000 00424600 2**0 │ │ │ CONTENTS, READONLY, DEBUGGING │ │ │ @@ -7098,15 +7098,15 @@ │ │ │ [1319](sec 19)(fl 0x00)(ty 0)(scl 3) (nx 1) 0x0000000000061463 .debug_loclists │ │ │ AUX scnlen 0x3a1b nreloc 623 nlnno 0 │ │ │ [1321](sec 12)(fl 0x00)(ty 0)(scl 3) (nx 1) 0x0000000000000810 .debug_aranges │ │ │ AUX scnlen 0xc0 nreloc 11 nlnno 0 │ │ │ [1323](sec 20)(fl 0x00)(ty 0)(scl 3) (nx 1) 0x000000000000f656 .debug_rnglists │ │ │ AUX scnlen 0xb7e nreloc 155 nlnno 0 │ │ │ [1325](sec 15)(fl 0x00)(ty 0)(scl 3) (nx 1) 0x000000000002e8ac .debug_line │ │ │ -AUX scnlen 0x23fe nreloc 73 nlnno 0 │ │ │ +AUX scnlen 0x23f3 nreloc 72 nlnno 0 ...
I haven't really looked into the details. https://in.waw.pl/~zbyszek/fedora/builds-f42-after-mass-rebuild.amd64.txt and https://in.waw.pl/~zbyszek/fedora/builds-f42-after-mass-rebuild.arm64.txt have bunch of examples, search for "mingw". The two rpms are at https://in.waw.pl/~zbyszek/fedora/mingw64-liblerc-4.0.0-8.fc42.noarch.orig.r... https://in.waw.pl/~zbyszek/fedora/mingw64-liblerc-4.0.0-8.fc42.noarch.rebuil... in case you want to take a look.
Zbyszek
-
Aoife Moloney via devel-announce -
Daniel P. Berrangé -
Dmitry Belyavskiy -
Jelle van der Waa -
Miroslav Suchý -
Neal Gompa -
Richard W.M. Jones -
Simon de Vlieger -
Zbigniew Jędrzejewski-Szmek