Fedora 34 Change: fbrnch (Self-Contained Change proposal)
by Ben Cotton
https://fedoraproject.org/wiki/Changes/fbrnch
== Summary ==
Add the [https://github.com/juhp/fbrnch fbrnch] packager tool to Fedora.
== Owner ==
* Name: [[User:Petersen|Jens Petersen]]
* Email: <petersen(a)redhat.com>
== Detailed Description ==
fbrnch (F(ed) Branch) is a tool designed to help simplify or
semi-automate workflow processes for Fedora Packagers.
It was first presented at "Nest with Fedora" last year, and has
received some favorable feedback from users.
It is currently only available in a
[https://copr.fedorainfracloud.org/coprs/petersen/fbrnch/ Copr repo].
This Change is simply to bring the package into Fedora itself and
broaden awareness about the tool.
See the Benefit section below for more details about the functionality
that fbrnch provides.
== Benefit to Fedora ==
fbrnch is a CLI tool for Fedora Packagers, which supports some common
tasks like:
* merging and building packages across release branches
* automatic parallel builds of sets of packages in dependency order in
Koji sidetags
* srpm, scratch, local install, and mock builds from arbitrary package branches
* cloning and listing Fedora packages from Pagure
* progressive Copr builds across chroots
* creating, updating and listing one's package reviews
* requesting new repos and branches, and importing new packages
directly from package reviews
and more. See https://github.com/juhp/fbrnch#readme for example usage.
== Scope ==
* Proposal owners:
** Release fbrnch with new dependencies on the Haskell Hackage
repository and add to Stackage. (Latest current version tagged in
github and built in copr is 0.6.8.)
** Submit and complete package review(s).
** Build packages.
** Test fbrnch package working correctly.
* Other developers: N/A (not a System Wide Change)
* Release engineering: N/A (check by Release Engineering not needed:
this just adds a few new package(s))
* Policies and guidelines: N/A (not a System Wide Change)
* Trademark approval: N/A (not needed for this Change)
* Alignment with Objectives: N/A
== Upgrade/compatibility impact ==
N/A (not a System Wide Change)
== How To Test ==
fbrnch has already been tested quite bit over the last 6 months.
* dnf install fbrnch
The [https://github.com/juhp/fbrnch#readme Readme file] contains
examples of usage, and <code>fbrnch --help</code> provides help
output.
== User Experience ==
Fedora Packagers who wish to use fbrnch will be able to install it
without having to add another Copr repo,
and benefit easily from the advanced packaging workflow features
provided by fbnrch.
The package will get updated through Bodhi and with more testing, will
gain broader confidence from Packagers.
== Dependencies ==
A few package reviews will need to be completed in time.
== Contingency Plan ==
* Contingency mechanism: (What to do? Who will do it?) N/A (not a
System Wide Change)
* Contingency deadline: N/A (not a System Wide Change)
* Blocks release? N/A (not a System Wide Change)
In the case the Change is not completed, the remainder can safely be
deferred until Fedora 35.
== Documentation ==
* https://github.com/juhp/fbrnch#readme
* https://github.com/juhp/presentations/blob/master/fedora-nest-2020-fbrnch...
--
Ben Cotton
He / Him / His
Senior Program Manager, Fedora & CentOS Stream
Red Hat
TZ=America/Indiana/Indianapolis
3 years, 2 months
Fedora 34 Change: DNF/RPM Copy on Write enablement for all variants
(System-Wide Change)
by Ben Cotton
https://fedoraproject.org/wiki/Changes/RPMCoW
== Summary ==
RPM Copy on Write provides a better experience for Fedora Users as it
reduces the amount of I/O and offsets CPU cost of package
decompression. RPM Copy on Write uses reflinking capabilities in
btrfs, which is the default filesystem in Fedora 33.
== Owners ==
* Name: [[User:malmond|Matthew Almond]], [[User:dcavalca|Davide Cavalca]]
* Email: malmond(a)fb.com, dcavalca(a)fb.com
== Detailed description ==
Installing and upgrading software packages is a standard part of
managing the lifecycle of any operating system. For the entire
lifecycle of Fedora, all software is packaged and distributed using
the RPM file fomat. This proposal changes how software is downloaded
and installed, leaving the distribution process unmodified.
=== Current process ===
# Resolve packaging request into a list of packages and operations
# Download and verify new packages
# Install and/or upgrade packages sequentially using RPM files,
decompressing, and writing a copy of the new files to storage.
=== New process ===
# Resolve packaging request into a list of packages and operations
# Download and '''decompress''' packages into a '''locally optimized''' rpm file
# Install and/or upgrade packages sequentially using RPM files, using
'''reference linking''' (reflinking) to reuse data already on disk.
The outcome is intended to be the same, but the order of operations is
different.
# Decompression happens inline with download. This has a positive
effect on resource usage: downloads are typically limited by
bandwidth. Decompression and writing the full data into a single file
per rpm is essentially free. Additionally: if there is more than one
download at a time, a multi-CPU system can be better utilized. All
compression types supported in RPM work because this uses the rpm I/O
functions.
# RPMs are cached on local storage between downloading and
installation time as normal. This allows DNF to defer actual RPM
installation to when all the RPM are available. This is unchanged.
# The file format for RPMs is different with Copy on Write. The
headers are identical, but the payload is different. There is also a
footer.
## Files are converted (“transcoded”) locally during download using
<code>/usr/bin/rpm2extents</code> (part of rpm codebase). The format
is not intended to be “portable” - i.e. copying the files from the
cache is not supported.
## Regular RPMs use a compressed .cpio based payload. In contrast,
extent based RPMs contain uncompressed data aligned to the fundamental
page size of the architecture, e.g. 4KiB on x86_64. This alignment is
required for <code>FICLONERANGE</code> to work. Only files are
represented in the payload, other directory entries like symlinks,
device nodes etc are constructed entirely from rpm header information.
Files are referenced by their digest, so identical files are
de-duplicated.
## The footer currently has three sections
### Table of original (rpm) file digests, used to validate the
integrity of the download in dnf.
### Table of digest → offset used when actually installing files.
### Signature 8 bytes at the end of the file, used to differentiate
between traditional RPMs and extent based.
=== Notes ===
# The headers are preserved bit for bit during transcoding. This
preserves signatures. The signatures cover the main header blob, and
the main header blob ensures the integrity of data in two ways:
## Each file with content has a digest. Originally this was md5, but
today it’s usually sha256. In normal RPM this is only used to verify
the integrity of files, e.g. <code>rpm -V</code>. With CoW we use this
as a content key.
## There is/are one or two digests (<code>PAYLOADDIGEST</code> and
<code>PAYLOADDIGESTALT</code>) covering the payload archive
(compressed cpio). The header value is preserved, but transcoded RPMs
do not preserve the original structure so RPM’s pre-installation
verification (controlled by <code>%_pkgverify_level</code> will fail.
<code>dnf-plugin-cow</code> disables this check in dnf because it
verifies the whole file digest which is captured during
download/transcoding. The second one is likely used for delta rpm.
# This is untested, and possibly incompatible with delta RPM (drpm).
The process for reconstructing an rpm to install from a delta is
expensive from both a CPU and I/O perspective, while only providing
marginal benefits on download size. It is expected that having delta
rpm enabled (which is the default) will be handled gracefully.
# Disk space requirements are expected to be marginally higher than
before: all new packages or updates will consume their installed size
before installation instead of about half their size (regular rpms
with payloads still cost space).
# <code>rpm-plugin-reflink</code> will fall back to simple file
copying when the destination path is not on the same
filesystem/subvolume. A common example is <code>/boot</code> and/or
<code>/boot/efi</code>.
# The system will still work on other filesystem types, but will
''always'' fall back to simple copying. This is expected to be
slightly slower than not enabling CoW because the source for copying
will be the decompressed data.
# For systems that enable transparent filesystem compression: every
file will continue to be decompressed from the original rpm, and then
transparently re-compressed by the filesystem. There is no effective
change here. There is a future project to investigate alternate
distribution mechanics to provide parallel versions of file content
pre-compressed in a filesystem specific format, reducing both CPU
costs and I/O. It is expected that this will result in slightly higher
network utilization because filesystem compression is purposely
restricted to allow random I/O.
# Current implementation of <code>dnf-plugin-cow</code> is in Python,
but it looks possible to implement this in <code>libdnf</code> instead
which would make it work in <code>packagekit</code>.
=== Performance Metrics ===
Ballpark performance difference is about half the duration for file
download+install time. A lot of rpms are very small, so it’s difficult
to see/measure. Larger RPMs give much clearer signal.
(Actual numbers/charts will be supplied in Jan 2021)
=== Terminology ===
* '''Copy on Write (CoW)''' is a broad description of any technology
that reduces or eliminates data duplication by sharing the data behind
the scenes until one of the references makes changes. This has been a
cornerstone technology in memory management in Unix systems. Here we
are using it to specifically reference Copy on Write as supported in
modern filesystems, e.g. btrfs, xfs and potentially others.
* '''Reflink''' is the verb for duplicating stored data on a
filesystem. See
[https://man7.org/linux/man-pages/man2/ioctl_ficlonerange.2.html
ioctl_ficlonerange(2)] for the specific call we use on Linux
* '''Extent''' (based RPMs) refers to how payload file data is stored
in within an RPM. Normal RPMs simply contain a compressed CPIO
archive. Extent based RPMs contain the raw data uncompressed, which
can be referenced with reflink.
== Benefit to Fedora ==
Faster package installs and upgrades
== Scope ==
* Proposal owners:
** Merge changes to rpm, librepo to enable capabilities
** Add dnf-plugin-cow to available packages
** Test days
** Aid with documentation
* Other developers:
** rpm, librepo: review PRs as needed
* Release engineering: https://pagure.io/releng/issue/9914
* Policies and guidelines: N/A
* Trademark approval: N/A
== Upgrade/compatibility impact ==
None, RPM with CoW is not enabled by default.
Upgrades with <code>keepcache</code> in dnf.conf will be able to use
existing packages, but it will not convert them. This only happens at
download time.
If a system is configured to keep packages in the cache
(<code>keepcache</code> in <code>dnf.conf</code>) and
<code>dnf-plugin-cow</code> is removed then the packages will be
unusable. Recommend <code>dnf clean packages</code> to resolve this.
== How to test ==
Enable RPM with CoW with
<pre>$ sudo dnf install dnf-plugin-cow
...
$ sudo dnf install hello
...
$ hello
Hello, world!</pre>
There should be no end user visible changes, except timing.
== User experience ==
No anticipated user visible changes in this change proposal. This
makes the feature available, but does not enable it by default.
== Dependencies ==
# A copy-on-write filesystem; this Change is primarily targeting
btrfs, but RPM with CoW should work with XFS as well (untested)
# Most package install paths and the dnf package cache on the same
filesystem / subvolume.
# <code>rpm</code> with Copy on Write patch set:
https://github.com/malmond77/rpm/tree/cow
# <code>librepo</code> with transcoding support:
https://github.com/malmond77/librepo/tree/transcode_cow
# dnf-plugin-reflink (a new package):
https://github.com/facebookincubator/dnf-plugin-cow/
== Contingency plan ==
* Contingency mechanism: will not include PR patches if not merged
upstream, skip <code>dnf-plugin-cow</code>
* Contingency deadline: Final freeze
* Blocks release? No
* Blocks product? No
== Documentation ==
Documentation will be available at
https://github.com/facebookincubator/dnf-plugin-cow in the coming
weeks
== Release Notes ==
RPM with CoW is not enabled by default. To enable it:
<pre>$ sudo dnf install dnf-plugin-cow</pre>
--
Ben Cotton
He / Him / His
Senior Program Manager, Fedora & CentOS Stream
Red Hat
TZ=America/Indiana/Indianapolis
3 years, 3 months
ELN build order
by Vít Ondruch
Can the ELN be fixed to stop doing rebuild like this? Of course the
issue is not in rubygem-thin but the issue is that once the Ruby side
tag was merged, ELN did not bothered with build order and now there is
rubygem-eventmachine which was built against Ruby 2.7 where is should
have been build against Ruby 3.0. I can't rebuild it, because that would
require release bump in Fedora (I don't want to do that, because Fedora
maintainers should not be bothered with ELN, right?).
I think that ELN should try to follow the Rawhide build order. If it
can't do that, then it should wait for manual intervention.
I think that ELN should have done gating or scratch build and check if
the dependencies are the same as in Rawhide or something.
I think that ELN should check the build result for dependency issue and
stop blindly submitting builds.
Vít
-------- Přeposlaná zpráva --------
Předmět: bpeck/jenkins-continuous-infra.apps.ci.centos.org's
rubygem-thin-1.7.2-16.eln108 failed to build
Datum: Mon, 11 Jan 2021 08:49:37 +0000 (GMT)
Od: notifications(a)fedoraproject.org
Komu: vondruch(a)redhat.com
Notification time stamped 2021-01-11 08:49:35 UTC
bpeck/jenkins-continuous-infra.apps.ci.centos.org's
rubygem-thin-1.7.2-16.eln108 failed to build
http://koji.fedoraproject.org/koji/buildinfo?buildID=1666628
--
You received this message due to your preference settings at
https://apps.fedoraproject.org/notifications/vondruch.id.fedoraproject.or...
3 years, 3 months
Cfitsio soname bump
by Sergio Pascual
Hello, I'm going to build new cfitsio 3.490 in Rawhide in a week. This
involves a soname bump.
The affected packages are the following:
CCfits
LabPlot
astrometry
bes
cpl
elements-alexandria
gdal
healpix
indi-gphoto
kst
kstars
libindi
luminance-hdr
nightviewperl-Astro-FITS-CFITSIO
phd2
python-astropy
python-fitsio
python-healpy
root
siril
skyviewer
sourcextractor++
ufraw
vips
wcslib
Best regards, Sergio
3 years, 3 months
cxxtools-2.2.1 fails to compile on rawhide with gcc11 with
/usr/include/c++/11/string_view:98:21: error: static assertion failed
by Martin Gansser
Hi,
i am trying to compile cxxtools 2.2.1 [1] on Fedora 34 with gcc11 but this fails with following error messages [2]
on Fedora build server.
make[2]: Entering directory '/builddir/build/BUILD/cxxtools-2.2.1/src'
/bin/sh ../libtool --tag=CXX --mode=compile g++ -DHAVE_CONFIG_H -I. -I../src -I../include -I../include -Wno-long-long -Wall -pedantic -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -c -o settingswriter.lo settingswriter.cpp
libtool: compile: g++ -DHAVE_CONFIG_H -I. -I../src -I../include -I../include -Wno-long-long -Wall -pedantic -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -c settingswriter.cpp -fPIC -DPIC -o .libs/settingswriter.o
In file included from settingswriter.h:31,
from settingswriter.cpp:28:
../include/cxxtools/char.h: In static member function 'static std::char_traits<cxxtools::Char>::char_type* std::char_traits<cxxtools::Char>::move(std::char_traits<cxxtools::Char>::char_type*, const char_type*, std::char_traits<cxxtools::Char>::int_type)':
../include/cxxtools/char.h:337:80: warning: 'void* memmove(void*, const void*, size_t)' writing to an object of type 'std::char_traits<cxxtools::Char>::char_type' {aka 'class cxxtools::Char'} with no trivial copy-assignment; use copy-assignment or copy-initialization instead [-Wclass-memaccess]
337 | return (cxxtools::Char*)std::memmove(s1, s2, n * sizeof(cxxtools::Char));
| ^
In file included from settingswriter.h:31,
from settingswriter.cpp:28:
../include/cxxtools/char.h:65:11: note: 'std::char_traits<cxxtools::Char>::char_type' {aka 'class cxxtools::Char'} declared here
65 | class Char
| ^~~~
In file included from settingswriter.h:31,
from settingswriter.cpp:28:
../include/cxxtools/char.h: In static member function 'static std::char_traits<cxxtools::Char>::char_type* std::char_traits<cxxtools::Char>::copy(std::char_traits<cxxtools::Char>::char_type*, const char_type*, std::size_t)':
../include/cxxtools/char.h:344:79: warning: 'void* memcpy(void*, const void*, size_t)' writing to an object of type 'std::char_traits<cxxtools::Char>::char_type' {aka 'class cxxtools::Char'} with no trivial copy-assignment; use copy-assignment or copy-initialization instead [-Wclass-memaccess]
344 | return (cxxtools::Char*)std::memcpy(s1, s2, n * sizeof(cxxtools::Char));
| ^
In file included from settingswriter.h:31,
from settingswriter.cpp:28:
../include/cxxtools/char.h:65:11: note: 'std::char_traits<cxxtools::Char>::char_type' {aka 'class cxxtools::Char'} declared here
65 | class Char
| ^~~~
In file included from /usr/include/c++/11/bits/basic_string.h:48,
from /usr/include/c++/11/string:55,
from ../include/cxxtools/char.h:32,
from settingswriter.h:31,
from settingswriter.cpp:28:
/usr/include/c++/11/string_view: In instantiation of 'class std::basic_string_view<cxxtools::Char, std::char_traits<cxxtools::Char> >':
settingswriter.cpp:42:26: required from here
/usr/include/c++/11/string_view:98:21: error: static assertion failed
98 | static_assert(is_trivial_v<_CharT> && is_standard_layout_v<_CharT>);
| ^~~~~~~~~~~~~~~~~~~~
/usr/include/c++/11/string_view:98:21: note: 'std::is_trivial_v<cxxtools::Char>' evaluates to false
make[2]: *** [Makefile:740: settingswriter.lo] Error 1
[1] https://martinkg.fedorapeople.org/ErrorReports/cxxtools-2.2.1-25.fc33.src...
[2] https://koji.fedoraproject.org/koji/taskinfo?taskID=60804463
I use the following patch for gcc11:
--- include/cxxtools/char.h.orig 2021-01-28 10:15:36.017763265 +0100
+++ include/cxxtools/char.h 2021-01-28 10:16:14.833762026 +0100
@@ -68,9 +68,7 @@
typedef int32_t value_type;
//! Constructs a character with a value of 0.
- Char()
- : _value(0)
- {}
+ Char() = default;
//! Constructs a character using the given char as base for the character value.
Char(char ch)
--- src/char.cpp.orig 2021-01-29 09:50:47.876669852 +0100
+++ src/char.cpp 2021-01-29 09:51:37.747675779 +0100
@@ -140,7 +140,7 @@
cxxtools::Char ctype<cxxtools::Char>::do_widen(char ch) const
{
- return cxxtools::Char(ch);
+ return cxxtools::Char(static_cast<unsigned char>(ch));
}
but this patch didn't solve the issue.
Any ideas ?
Regards
Martin
3 years, 3 months
Fedora 34 Change: Ignore Anaconda kernel boot parameters without
'inst.' prefix (System-Wide change)
by Ben Cotton
https://fedoraproject.org/wiki/Changes/Ignore_Anaconda_kernel_boot_parame...
== Summary ==
Right now Anaconda allows usage of boot options both with the 'inst' prefix
(inst.stage2=) and without (stage2=). We would like to ignore the use of
Anaconda kernel boot parameters which do not contain the 'inst.' prefix.
== Owner ==
* Name: [[User:jkonecny| Jiří Konečný]]
* Email: <jkonecny(a)redhat.com>
== Detailed Description ==
Anaconda allows you to use kernel boot parameters with and without the
'inst.' prefix right now (e.g. inst.repo= / repo=). However, specifying
boot parameters without the 'inst' prefix has not been recommended for
years and has been deprecated since Fedora 33. We (Anaconda team) would
like to make it an official requirement now that users must specify the
'inst.' prefix all the time.
The reason for this is frequent parameter conflicts with other projects. We
already had a few issues with conflicts in the past, such as if the user
ran an installation with `debug`. In that case the installation boot would
enable debug mode for the kernel and also for Anaconda which is probably
not what the user intended.
Another reason is to make it readable at first glance what belongs to
Dracut (rd.), kernel (no prefix) or Anaconda (inst.).
== Feedback ==
We have sent mail about doing the deprecation on Fedora 33. The only issue
there was why the prefix is 'inst.' and not 'anaconda.'.
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.o...
== Benefit to Fedora ==
This change should make crystal clear what kernel boot parameters are
processed by the Anaconda installer.
It will also avoid conflicts with other kernel boot parameters.
== Scope ==
* Proposal owners: Remove support to process arguments without the 'inst.'
prefix and require the 'inst.' prefix for kernel boot parameters consumed
by Anaconda.
* Other developers:
All configurations using the not-recommended solution without prefix will
have to change the invocation of the kernel boot options consumed by
Anaconda. These users are already warned since Fedora 33 after boot.
This should not be a problem for ISOs we ship because they already use the
recommended 'inst.' prefix everywhere. However, it will probably touch some
custom PXE configurations and other custom ISO configurations which are
prone to this, because users often want to save typing and not explicitly
write the 'inst.' prefix. Fortunately, most of these configuration changes
shouldn't be that hard to change.
* Release engineering: [https://pagure.io/releng/issue/9889 #9889]
* Policies and guidelines: This should not be required. All the
documentation should use the recommended 'inst.' prefix already.
* Trademark approval: No (not needed for this Change)
* Alignment with Objectives: No
== Upgrade/compatibility impact ==
If your custom infra configuration is not updated then new Fedora
installations could not install correctly. Most probably the ISO will not
boot (use of 'stage2=') or your repositories won't be used (use of 'repo=').
== How To Test ==
# You need a Virtual Machine and ISO for testing.
# Boot the installation using the ISO and try Anaconda specific kernel boot
parameters. See here to find out the list
https://anaconda-installer.readthedocs.io/en/latest/boot-options.html.
# Parameters without the prefix should be ignored and with the prefix
should be used.
== User Experience ==
Users of Fedora official ISOs should not be impacted because all the Fedora
official ISOs should already use the recommended prefix.
== Dependencies ==
Don't know about any packages impacted by this change.
== Contingency Plan ==
* Contingency mechanism: The Anaconda team will revert code changes and get
back support for boot parameters without the 'inst.' prefix.
* Contingency deadline: Final Freeze
* Blocks release? No
* Blocks product? This won't require changes for any specific product.
== Documentation ==
I don't think we need to document this change other than in Release Notes.
As mentioned before, the solution without 'inst.' prefix has not been
recommended for years and it should not be used anywhere in the official
documentation.
--
Ben Cotton
He / Him / His
Senior Program Manager, Fedora & CentOS Stream
Red Hat
TZ=America/Indiana/Indianapolis
3 years, 3 months
btrfs hash algorithm (should xxhash be the default?)
by Matthew Miller
On my Intel i7 laptop, xxhash is a small but clear performance win over
crc32c:
$ ./hash-speedtest 10000000
Block size: 4096
Iterations: 10000000
Implementation: builtin
NULL-NOP: cycles: 1372543560, c/i 137
NULL-MEMCPY: cycles: 2844174884, c/i 284
CRC32C: cycles: 9673117404, c/i 967
XXHASH: cycles: 7129819594, c/i 712
SHA256: cycles: 649914613520, c/i 64991
BLAKE2b: cycles: 153513008046, c/i 15351
And I'm given to understand that this is even more the case on newer CPUs.
Plus, it's 64 bit instead of 32 bit. The 256-bit algorithms are obviously
much, much slower and probably not right for a default, but should we
consider making xxhash the default for Fedora Linux systems with btrfs?
--
Matthew Miller
<mattdm(a)fedoraproject.org>
Fedora Project Leader
3 years, 3 months
