Hi,
The effort to make package builds in Fedora reproducible has picked up steam again. We now have a new website: https://docs.fedoraproject.org/en-US/reproducible-builds and an issue tracker: https://pagure.io/fedora-reproducible-builds/project and a matrix room: https://matrix.to/#/#reproducible-builds:fedora.im
We've done a mini rebuild using [1] for the package list [2] and results are at [3]. (The result is a json dump of rpmdiff output by package. Generally, "" means the rebuild was identical except for variable metadata, and non-empty output else means that the rebuild was different.)
[1] https://github.com/keszybz/fedora-repro-build [2] https://fedorapeople.org/~zbyszek/builds-2024-02-fc41-filtered.txt [3] https://fedorapeople.org/~zbyszek/builds-2024-02-fc41-filtered.results.txt
I'm writing this mail for two purposes. First, as a heads-up: various patches and RFEs have been filed to fix issues as they are detected. Second, for the usual enticement: join and be merry.
The plan for the immediate future is to fix various issues, both those that affect a single package and also the ones that affect a swath of packages. Some of the remaining second type:
https://pagure.io/fedora-reproducible-builds/project/issue/7 — static archives do not respect $SOURCE_DATE_EPOCH, embed UID and GID https://pagure.io/fedora-reproducible-builds/project/issue/10 — Java jar files embed build timestamps https://pagure.io/fedora-reproducible-builds/project/issue/12 — Python pyc file serialization is architecture-specific https://pagure.io/fedora-reproducible-builds/project/issue/14 — noarch packages installing into %{_libdir} https://pagure.io/fedora-reproducible-builds/project/issue/15 — golang debuginfo pakages have files with .gdb_index section of varying size
If you have ideas how to tackle some of those issues, help would be very welcome. Please use the matrix room for coordination.
Currently the percentage of reproducibility is not very high. Once we fix the issues that affect swaths of packages and we're down to issues that only affect one or a very small number of packages, I hope we can make reproducibility an official effort in Fedora. But that's still some way ahead.
Zbyszek
On Thu, Mar 07, 2024 at 12:39:37PM +0000, Zbigniew Jędrzejewski-Szmek wrote:
Hi,
The effort to make package builds in Fedora reproducible has picked up steam again. We now have a new website: https://docs.fedoraproject.org/en-US/reproducible-builds
I read this page but it doesn't cover an important point (for me). What's the actual threat model you have in mind?
Rich.
On Thu, Mar 07, 2024 at 10:01:08PM +0000, Richard W.M. Jones wrote:
On Thu, Mar 07, 2024 at 12:39:37PM +0000, Zbigniew Jędrzejewski-Szmek wrote:
Hi,
The effort to make package builds in Fedora reproducible has picked up steam again. We now have a new website: https://docs.fedoraproject.org/en-US/reproducible-builds
I read this page but it doesn't cover an important point (for me). What's the actual threat model you have in mind?
The threat model is the build machine being "compromised" and modifying the build artifacts somehow. In case of a software attack, this could mean for example a rogue version of the compiler injecting additional code into the binaries. In general, any kind of "supply chain attack". But a non-malicious scenario is also possible, with the hardware being flaky or overheated or having firmware problems that result in some modification to the output binaries.
I imagine that large organizations may invest in setting up a "shadow rebuild" instance that has the full pipeline from dist-git to binary rpms and does fully independent builds of packages. Reproducibility allows independent verification that the dist-git sources actually correspond to the binaries that are delivered. With such checks, any kind of supply chain attack would be very hard to do undetected. The build infrastracture in Fedora is obviously well protected, so such an attack would be very hard to pull off, but it also is exteremely attractive because of how effective and stealthy it would be. So by making builds reproducible and allowing 3rd parties to do rebuilds, we allow more trust to be established for our packages.
Nevertheless, for me, reproducibility is interesting because it aids debugability, and "threats" are not an immediate concern. Essentially, when the builds are stable, any unexpected change in the build outputs is much easier to diagnose. We have already found and submitted a bunch of obvious fixes that would not have been found otherwise. Also, when builds are stable, when working on the tools, it is easy to do a rebuild with the patched tools and observe the diff. If the build is "unstable", i.e. there are various other unrelated changes, interesting differences often drown in noise.
E.g. today I ended up creating a pull request for intltool package [1], backporting a patch to fix a race in cache creation which was corrupting translations in files. The patch is from 2015, but seemingly nobody noticed the issue in Fedora so far.
More examples are [2,3], one of the many examples of noarch packages using arch-dependent macros, e.g. %_libdir, leading to packages that are "noarch", but actually depend on the build architecture, and misbehave when installed on a system with a different architecture than the build machine.
[1] https://src.fedoraproject.org/rpms/intltool/pull-request/2 [2] https://src.fedoraproject.org/rpms/xbyak/pull-request/5 [3] https://src.fedoraproject.org/rpms/python-virt-firmware/pull-request/3
Zbyszek
On Thu, 7 Mar 2024 at 18:38, Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl wrote:
On Thu, Mar 07, 2024 at 10:01:08PM +0000, Richard W.M. Jones wrote:
On Thu, Mar 07, 2024 at 12:39:37PM +0000, Zbigniew Jędrzejewski-Szmek
wrote:
Nevertheless, for me, reproducibility is interesting because it aids debugability, and "threats" are not an immediate concern. Essentially, when the builds are stable, any unexpected change in the build outputs is much easier to diagnose. We have already found and submitted a bunch of obvious fixes that would not have been found otherwise. Also, when builds are stable, when working on the tools, it is easy to do a rebuild with the patched tools and observe the diff. If the build is "unstable", i.e. there are various other unrelated changes, interesting differences often drown in noise.
Now this is the part which is the real important part which gets overlooked a lot. As much as some people somehow make it out that reproducibility will make things secure.. it will only do so against a threat which isn't as existent as people injecting bad source code. [Supply chain attacks are already just including bad source code which will build reproducible but still be bad. It is much cheaper to make mostly useful but compromised code into pypi, cpan, some cargo place etc and getting other people to need to include it in a distro or just straight to a developer than it is to break into Fedora/Debian/etc and compromise a gcc ]
The real fix is in Quality and catching things which silently make builds bad. A CPU problem, a memory problem, a builder kernel issue, etc are bigger problems and more likely to cause hard to fix bugs because they aren't bugs in the code. Of course this means that builds need to be built twice inside a build system to make sure that both builds match.. otherwise you can end up where someone rebuilding starts reporting problems with our build system but its because they used overclocked ram or cpu :)
E.g. today I ended up creating a pull request for intltool package [1], backporting a patch to fix a race in cache creation which was corrupting translations in files. The patch is from 2015, but seemingly nobody noticed the issue in Fedora so far.
More examples are [2,3], one of the many examples of noarch packages using arch-dependent macros, e.g. %_libdir, leading to packages that are "noarch", but actually depend on the build architecture, and misbehave when installed on a system with a different architecture than the build machine.
[1] https://src.fedoraproject.org/rpms/intltool/pull-request/2 [2] https://src.fedoraproject.org/rpms/xbyak/pull-request/5 [3] https://src.fedoraproject.org/rpms/python-virt-firmware/pull-request/3
Zbyszek
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
V Thu, Mar 07, 2024 at 12:39:37PM +0000, Zbigniew Jędrzejewski-Szmek napsal(a):
The effort to make package builds in Fedora reproducible has picked up steam again. We now have a new website: https://docs.fedoraproject.org/en-US/reproducible-builds and an issue tracker: https://pagure.io/fedora-reproducible-builds/project and a matrix room: https://matrix.to/#/#reproducible-builds:fedora.im
We've done a mini rebuild using [1] for the package list [2] and results are at [3]. (The result is a json dump of rpmdiff output by package. Generally, "" means the rebuild was identical except for variable metadata, and non-empty output else means that the rebuild was different.)
[1] https://github.com/keszybz/fedora-repro-build [2] https://fedorapeople.org/~zbyszek/builds-2024-02-fc41-filtered.txt [3] https://fedorapeople.org/~zbyszek/builds-2024-02-fc41-filtered.results.txt
Is this mini rebuild a one-shot thing, or are are you going to rebuild the packages repeatedly or use the results for something significant? I ask because I spotted some discrepancies in those text files:
(1) Some packages are listed twice, with different NEVRAs. E.g. perl-Alien-pkgconf or perl-RDF-RDFa-Generator.
(2) Both perl-Alien-pkgconf NEVRAs reports a differing /usr/lib64/perl5/vendor_perl/auto/share/dist/Alien-pkgconf/status.json content. That content looks likes this:
{"libs":"-lpkgconf","version":"2.1.0","install_type":"system","cflags":"-I/usr/include/pkgconf","dll":"/lib64/libpkgconf.so.4.0.0"}
That means you had to perform rebuilds of the same NEVRA with different libpkgconf-devel packages in the build roots. That looks like a bug in your mini rebuild scheduler.
(3) Some packages listed in builds-2024-02-fc41-filtered.txt are missing from builds-2024-02-fc41-filtered.results.txt. E.g. perl-CPAN-Plugin-Sysdeps-0.73-1.fc41 is listed as COMPLETE, yet results are missing.
(4) dnf5-5.1.13-1.fc41.src reports changes in Requires (e.g. "removed REQUIRES createrepo_c"). That again looks like you built the same NEVRA in different build roots (for some reason "%bcond_without tests" flipped).
All that means you might hunting ghosts instead of real bugs.
-- Petr
On Fri, Mar 08, 2024 at 09:07:30AM +0100, Petr Pisar wrote:
V Thu, Mar 07, 2024 at 12:39:37PM +0000, Zbigniew Jędrzejewski-Szmek napsal(a):
The effort to make package builds in Fedora reproducible has picked up steam again. We now have a new website: https://docs.fedoraproject.org/en-US/reproducible-builds and an issue tracker: https://pagure.io/fedora-reproducible-builds/project and a matrix room: https://matrix.to/#/#reproducible-builds:fedora.im
We've done a mini rebuild using [1] for the package list [2] and results are at [3]. (The result is a json dump of rpmdiff output by package. Generally, "" means the rebuild was identical except for variable metadata, and non-empty output else means that the rebuild was different.)
[1] https://github.com/keszybz/fedora-repro-build [2] https://fedorapeople.org/~zbyszek/builds-2024-02-fc41-filtered.txt [3] https://fedorapeople.org/~zbyszek/builds-2024-02-fc41-filtered.results.txt
Is this mini rebuild a one-shot thing, or are are you going to rebuild the packages repeatedly or use the results for something significant? I ask because I spotted some discrepancies in those text files:
Both ;) I'm working on the tooling to do the rebuilds, and the code is still a bit rough, but also I'm still figuring out the design. Once that done, we hope to hook this up to rebuilderd to get a continous rebuild with a nice dashboard (https://wiki.archlinux.org/title/Rebuilderd). For now, the results are being used to figure out a list of things to fix.
(1) Some packages are listed twice, with different NEVRAs. E.g. perl-Alien-pkgconf or perl-RDF-RDFa-Generator.
This is because they were built multiple times in koji.
(2) Both perl-Alien-pkgconf NEVRAs reports a differing /usr/lib64/perl5/vendor_perl/auto/share/dist/Alien-pkgconf/status.json content. That content looks likes this:
{"libs":"-lpkgconf","version":"2.1.0","install_type":"system","cflags":"-I/usr/include/pkgconf","dll":"/lib64/libpkgconf.so.4.0.0"}
That means you had to perform rebuilds of the same NEVRA with different libpkgconf-devel packages in the build roots. That looks like a bug in your mini rebuild scheduler.
Diffoscope says:
├── ./usr/lib64/perl5/vendor_perl/auto/share/dist/Alien-pkgconf/status.json │ ├── Pretty-printed │ │┄ Ordering differences only │ │ @@ -1,7 +1,7 @@ │ │ { │ │ "libs": "-lpkgconf", │ │ + "dll": "/lib64/libpkgconf.so.4.0.0", │ │ "version": "2.1.0", │ │ - "install_type": "system", │ │ "cflags": "-I/usr/include/pkgconf", │ │ - "dll": "/lib64/libpkgconf.so.4.0.0" │ │ + "install_type": "system" │ │ }
It would be great to sort the dictionary to avoid this randomness.
(3) Some packages listed in builds-2024-02-fc41-filtered.txt are missing from builds-2024-02-fc41-filtered.results.txt. E.g. perl-CPAN-Plugin-Sysdeps-0.73-1.fc41 is listed as COMPLETE, yet results are missing.
Yes. Some packages failed to build, and then I finished the build early because there were already enough interesting results.
(The few failures I looked at were caused by differences in BR between architectures. This is currently a corner case that I'm not sure how to deal with. Most of the time, using a srpm from a different architecture works fine, but in some cases the set of installed packages would differ, and then I can't figure out which version of the rpm for the local architecture would have been used and buildroot creation fails. I would be happy to describe the problem in more detail. It's also possible that other packages FTBFS, I didn't look into this and I didn't save the logs.)
(4) dnf5-5.1.13-1.fc41.src reports changes in Requires (e.g. "removed REQUIRES createrepo_c"). That again looks like you built the same NEVRA in different build roots (for some reason "%bcond_without tests" flipped).
Yes, the build root is different. I install the package set that was used for the main package build and just call mock with that and it does both srpm and the binary rpms there. But koji does the srpm build in a separate buildroot that is smaller.
It would be fairly easy to get the buildroot listing for the srpm and build the srpm separately. This wouldn't actually doesn't solve the problem fully, because koji will use a random build architecture, so we may potentially hit the problem that was described above for noarch builds.
For now, I'm taking the pragmatic approach of ignoring changes in PROVIDES/REQUIRES for the srpm. We know that those are not important. OTOH, for example changes in the source files are unexpected. I filed https://src.fedoraproject.org/rpms/stratisd/pull-request/6 for stratisd to stop rewriting the source tarballs (it was done irreproducibly, but I think it's better to not do this at all).
All that means you might hunting ghosts instead of real bugs.
The rebuild process is surprisingly reliable. Per the discussion above, we have some mix of real issues and (unexpected) randomness. Of course such randomness usually is not a bug, but I hope we can agree to get rid of any such cases to make the builds reproducible.
Zbyszek
On Fri, Mar 08, 2024 at 04:54:04PM +0000, Zbigniew Jędrzejewski-Szmek wrote:
On Fri, Mar 08, 2024 at 09:07:30AM +0100, Petr Pisar wrote:
(2) Both perl-Alien-pkgconf NEVRAs reports a differing /usr/lib64/perl5/vendor_perl/auto/share/dist/Alien-pkgconf/status.json content. That content looks likes this:
{"libs":"-lpkgconf","version":"2.1.0","install_type":"system","cflags":"-I/usr/include/pkgconf","dll":"/lib64/libpkgconf.so.4.0.0"}
That means you had to perform rebuilds of the same NEVRA with different libpkgconf-devel packages in the build roots. That looks like a bug in your mini rebuild scheduler.
Diffoscope says:
├── ./usr/lib64/perl5/vendor_perl/auto/share/dist/Alien-pkgconf/status.json │ ├── Pretty-printed │ │┄ Ordering differences only │ │ @@ -1,7 +1,7 @@ │ │ { │ │ "libs": "-lpkgconf", │ │ + "dll": "/lib64/libpkgconf.so.4.0.0", │ │ "version": "2.1.0", │ │ - "install_type": "system", │ │ "cflags": "-I/usr/include/pkgconf", │ │ - "dll": "/lib64/libpkgconf.so.4.0.0" │ │ + "install_type": "system" │ │ }
It would be great to sort the dictionary to avoid this randomness.
And perl-Module-Build has: ├── ./usr/libexec/perl-Module-Build/_build/magicnum │ @@ -1 +1 @@ │ -1016 │ +697476
It's generated via $self->_write_data('magicnum', $self->magic_number(int rand 1_000_000)); It seems strange to fix the seed for tests like this…
Zbyszek
V Fri, Mar 08, 2024 at 05:04:01PM +0000, Zbigniew Jędrzejewski-Szmek napsal(a):
And perl-Module-Build has: ├── ./usr/libexec/perl-Module-Build/_build/magicnum │ @@ -1 +1 @@ │ -1016 │ +697476
It's generated via $self->_write_data('magicnum', $self->magic_number(int rand 1_000_000)); It seems strange to fix the seed for tests like this…
That file was a byproduct of perl-Module-Build's build script. I removed it the from a binary package and perl-Module-Build-0.42.34-6.fc41 should be reproducible now.
-- Petr
On Mon, Mar 11, 2024 at 04:43:37PM +0100, Petr Pisar wrote:
V Fri, Mar 08, 2024 at 05:04:01PM +0000, Zbigniew Jędrzejewski-Szmek napsal(a):
And perl-Module-Build has: ├── ./usr/libexec/perl-Module-Build/_build/magicnum │ @@ -1 +1 @@ │ -1016 │ +697476
It's generated via $self->_write_data('magicnum', $self->magic_number(int rand 1_000_000)); It seems strange to fix the seed for tests like this…
That file was a byproduct of perl-Module-Build's build script. I removed it the from a binary package and perl-Module-Build-0.42.34-6.fc41 should be reproducible now.
It is indeed. Thanks!
Zbyszek
V Fri, Mar 08, 2024 at 04:54:04PM +0000, Zbigniew Jędrzejewski-Szmek napsal(a):
On Fri, Mar 08, 2024 at 09:07:30AM +0100, Petr Pisar wrote:
(2) Both perl-Alien-pkgconf NEVRAs reports a differing /usr/lib64/perl5/vendor_perl/auto/share/dist/Alien-pkgconf/status.json content. That content looks likes this:
{"libs":"-lpkgconf","version":"2.1.0","install_type":"system","cflags":"-I/usr/include/pkgconf","dll":"/lib64/libpkgconf.so.4.0.0"}
That means you had to perform rebuilds of the same NEVRA with different libpkgconf-devel packages in the build roots. That looks like a bug in your mini rebuild scheduler.
Diffoscope says:
├── ./usr/lib64/perl5/vendor_perl/auto/share/dist/Alien-pkgconf/status.json │ ├── Pretty-printed │ │┄ Ordering differences only │ │ @@ -1,7 +1,7 @@ │ │ { │ │ "libs": "-lpkgconf", │ │ + "dll": "/lib64/libpkgconf.so.4.0.0", │ │ "version": "2.1.0", │ │ - "install_type": "system", │ │ "cflags": "-I/usr/include/pkgconf", │ │ - "dll": "/lib64/libpkgconf.so.4.0.0" │ │ + "install_type": "system" │ │ }
It would be great to sort the dictionary to avoid this randomness.
You are right. I completely forgot the ordering. I fixed it in perl-Alien-pkgconf-0.19-10.fc41.
(3) Some packages listed in builds-2024-02-fc41-filtered.txt are missing from builds-2024-02-fc41-filtered.results.txt. E.g. perl-CPAN-Plugin-Sysdeps-0.73-1.fc41 is listed as COMPLETE, yet results are missing.
Yes. Some packages failed to build, and then I finished the build early because there were already enough interesting results.
(The few failures I looked at were caused by differences in BR between architectures. This is currently a corner case that I'm not sure how to deal with. Most of the time, using a srpm from a different architecture works fine, but in some cases the set of installed packages would differ, and then I can't figure out which version of the rpm for the local architecture would have been used and buildroot creation fails. I would be happy to describe the problem in more detail. It's also possible that other packages FTBFS, I didn't look into this and I didn't save the logs.)
(4) dnf5-5.1.13-1.fc41.src reports changes in Requires (e.g. "removed REQUIRES createrepo_c"). That again looks like you built the same NEVRA in different build roots (for some reason "%bcond_without tests" flipped).
Yes, the build root is different. I install the package set that was used for the main package build and just call mock with that and it does both srpm and the binary rpms there. But koji does the srpm build in a separate buildroot that is smaller.
Using SRPMs from a different archicture won't work as you find out. You need to unpack the SRPM and do "dnf builddep THE_UNPACKED_SPEC_FILE". But then you will have a different build root content comparing to the Koji build.
So I guess an architecture of the builder needs to be handled as a piece of the reproducibility environment (i.e. reproducing a noarch package built on s390x on s390x, not on x86_64), or you can assume that noarch builds should not differ among builder archictures and then ignore the build root content and only focus on the resulting binary package.
-- Petr
On Mon, Mar 11, 2024 at 12:29:49PM +0100, Petr Pisar wrote:
V Fri, Mar 08, 2024 at 04:54:04PM +0000, Zbigniew Jędrzejewski-Szmek napsal(a):
On Fri, Mar 08, 2024 at 09:07:30AM +0100, Petr Pisar wrote:
(2) Both perl-Alien-pkgconf NEVRAs reports a differing /usr/lib64/perl5/vendor_perl/auto/share/dist/Alien-pkgconf/status.json content. That content looks likes this:
{"libs":"-lpkgconf","version":"2.1.0","install_type":"system","cflags":"-I/usr/include/pkgconf","dll":"/lib64/libpkgconf.so.4.0.0"}
That means you had to perform rebuilds of the same NEVRA with different libpkgconf-devel packages in the build roots. That looks like a bug in your mini rebuild scheduler.
Diffoscope says:
├── ./usr/lib64/perl5/vendor_perl/auto/share/dist/Alien-pkgconf/status.json │ ├── Pretty-printed │ │┄ Ordering differences only │ │ @@ -1,7 +1,7 @@ │ │ { │ │ "libs": "-lpkgconf", │ │ + "dll": "/lib64/libpkgconf.so.4.0.0", │ │ "version": "2.1.0", │ │ - "install_type": "system", │ │ "cflags": "-I/usr/include/pkgconf", │ │ - "dll": "/lib64/libpkgconf.so.4.0.0" │ │ + "install_type": "system" │ │ }
It would be great to sort the dictionary to avoid this randomness.
You are right. I completely forgot the ordering. I fixed it in perl-Alien-pkgconf-0.19-10.fc41.
Ah, cool, thanks! I can confirm that with perl-Alien-pkgconf-0.19-10.fc41, rpmdiff reports no differences.
(4) dnf5-5.1.13-1.fc41.src reports changes in Requires (e.g. "removed REQUIRES createrepo_c"). That again looks like you built the same NEVRA in different build roots (for some reason "%bcond_without tests" flipped).
Yes, the build root is different. I install the package set that was used for the main package build and just call mock with that and it does both srpm and the binary rpms there. But koji does the srpm build in a separate buildroot that is smaller.
Using SRPMs from a different archicture won't work as you find out. You need to unpack the SRPM and do "dnf builddep THE_UNPACKED_SPEC_FILE". But then you will have a different build root content comparing to the Koji build.
So I guess an architecture of the builder needs to be handled as a piece of the reproducibility environment (i.e. reproducing a noarch package built on s390x on s390x, not on x86_64), or you can assume that noarch builds should not differ among builder archictures and then ignore the build root content and only focus on the resulting binary package.
Yes, but that'd be a bummer. If we were strict about this, the number of packages that can be rebuilt would go down by a large percentage. So I think we need to take some hybrid approach, where we try to redo the srpm build on a different architecture.
Zbyszek
P.S. I pasted this in the Matrix channel already, but here is a summary of the rebuild:
#################### SUMMARY #################### total builds: 1810 reproducible: 989 (55%) only src metadata: 76 (4%) irreproducible: 745 (41%)
by rpm: total rpms: 12259 src, non-src: 1810 (15%), 10449 (85%) reproducible: 8879 (72%) only src metadata: 279 (2%) irreproducible: 3101 (25%)
rpms with irreproducibility: src_metadata: 288 static_library: 529 jar_library: 718 mingw_binary: 92 debuginfo_metadata: 205 debuginfo_hash: 775 javadoc_html: 338 doc_pdf: 7 rpm_metadata: 24 payload_paths: 32 payload_mods: 1286 unknown: 13
If we ignore the bogus metadata on srpms, we're 59% reproducible by package, and 75% reproducible counting individual rpms. The next order of business is to handle those static libraries and jars, which should make a sizeable dent in the remaining list. I started working on some tooling for this on the weekend.
Zbyszek
On 2024-03-07 04:39, Zbigniew Jędrzejewski-Szmek wrote:
Hi,
The effort to make package builds in Fedora reproducible has picked up steam again.
I gave a talk at SCALE 21x last week covering this work, the current state and what's coming down the pipe. You can find the recording at https://www.youtube.com/watch?v=5c4gfXVPAbU if you're interested, and a copy of the slides at https://www.socallinuxexpo.org/scale/21x/presentations/making-fedora-linux-m...
Cheers Davide