Thoughts about packaging a standalone python-PyQt5-sip?
by Michel Alexandre Salim
Hi all,
Neal and I are looking at getting ButterManager packaged, and it
depends on sip and PyQt5-sip:
https://github.com/egara/buttermanager/blob/master/requirements.txt
Now, this is where things get a bit odd:
- the current sip (4.19.24) does not have autogenerated Python
provides, but sip5 does:
$ sudo dnf repoquery --provides sip
Last metadata expiration check: 1:48:00 ago on Wed 30 Dec 2020 02:50:53
PM PST.
sip = 4.19.24-1.fc33
sip(x86-64) = 4.19.24-1.fc33
sip-macros = 4.19.24-1.fc33
$ sudo dnf repoquery --provides sip5
Last metadata expiration check: 1:48:05 ago on Wed 30 Dec 2020 02:50:53
PM PST.
python-sip5 = 5.5.0-1.fc33
python3-sip5 = 5.5.0-1.fc33
python3.9-sip5 = 5.5.0-1.fc33
python3.9dist(sip) = 5.5
python3dist(sip) = 5.5
sip5 = 5.5.0-1.fc33
sip5(x86-64) = 5.5.0-1.fc33
- sip ships PyQt5 bindings with matching version, but sip 5 seems to no
longer do so
$ sudo dnf info python3-pyqt5-sip
Last metadata expiration check: 1:51:03 ago on Wed 30 Dec 2020 02:50:53
PM PST.
Installed Packages
Name : python3-pyqt5-sip
Version : 4.19.24
Release : 1.fc33
Architecture : x86_64
Size : 244 k
Source : sip-4.19.24-1.fc33.src.rpm
Repository : @System
From repo : fedora
Summary : SIP - Python 3/C++ Bindings Generator for pyqt5
URL : https://riverbankcomputing.com/software/sip/intro
License : GPLv2 or GPLv3 and (GPLv3+ with exceptions)
Description : This is the Python 3 build of pyqt5-SIP.
- https://pypi.org/project/PyQt5-sip/ has PyQt5-sip 12.8.1 but
https://pypi.org/project/sip/ has sip 5.5.0 which matches the sip5 in
Fedora (available since last month)
- there's a lot of packages that depend on python3-pyqt5-sip (though
none use the canonical python3dist(pyqt5-sip) unfortunately)
$ sudo dnf repoquery --repo rawhide,rawhide-source --whatrequires
'python3dist(pyqt5-sip)'
Last metadata expiration check: 0:15:20 ago on Wed 30 Dec 2020
04:28:29 PM PST.
$ sudo dnf repoquery --repo rawhide,rawhide-source --whatrequires
python3-pyqt5-sip
Last metadata expiration check: 0:15:30 ago on Wed 30 Dec 2020
04:28:29 PM PST.
calibre-0:4.23.0-3.fc34.x86_64
krita-0:4.4.1-1.fc34.i686
krita-0:4.4.1-1.fc34.x86_64
libarcus-0:4.8.0-1.fc34.src
mingw-python-qt5-0:5.15.0-4.fc34.src
pyqtwebengine-0:5.15.0-2.fc33.src
python-pyface-0:7.1.0-1.fc34.src
python-pynest2d-0:4.8.0-1.fc34.src
python-qt5-0:5.15.0-5.fc34.src
python3-arcus-0:4.8.0-1.fc34.x86_64
python3-arcus-lulzbot-0:3.6.21-8.fc34.x86_64
python3-poppler-qt5-0:0.75.0-6.fc33.x86_64
python3-pyqtchart-0:5.15.2-1.fc34.i686
python3-pyqtchart-0:5.15.2-1.fc34.x86_64
python3-qgis-0:3.16.1-2.fc34.i686
python3-qgis-0:3.16.1-2.fc34.x86_64
python3-qscintilla-qt5-0:2.11.5-1.fc34.x86_64
python3-qt5-base-0:5.15.0-5.fc34.i686
python3-qt5-base-0:5.15.0-5.fc34.x86_64
qhexedit2-0:0.8.9-2.fc33.src
scidavis-0:2.3.0-2.fc33.src
veusz-0:3.3.1-1.fc34.src
veusz-0:3.3.1-1.fc34.x86_64
Any idea what's the best way to handle this? and/or why PyQt5-sip's
versioning get so far ahead of sip?
Thanks,
--
Michel Alexandre Salim
profile: https://keyoxide.org/michel@michel-slm.name
chat via email: https://delta.chat/
GPG key: 5DCE 2E7E 9C3B 1CFF D335 C1D7 8B22 9D2F 7CCC 04F2
2 years, 11 months
Upstream tip wanted: CI service for Big Endian acrhes
by Miro Hrončok
Recently I've reported some Big Endian related test failures to an
upstream project [0].
I was asked by an upstream project maintainer, whether I know some free
Continuous Integration services where they can easily run their
testsuite on Big Endian.
Any tips?
* Upstream uses Travis CI to test on x86_64 Linux (Ubuntu)
* Upstream uses AppVeyor to test on Microsoft Windows
* It's a pure Python project, noarch, but some changes need to be done
when loading/saving binary data (LE) with NumPy on BE system.
What I've considered:
* COPR (but there is no big endian arch)
* (Ab)using Koji (I guess that would be considered a bad practice?)
* using QUEMU on Travis CI [1]
Any better tips? Thanks
[0] https://github.com/mikedh/trimesh/issues/249
[1]
https://developer.ibm.com/linuxonpower/2017/07/28/travis-multi-architectu...
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
3 years
Fedora 34 Change: Route all Audio to PipeWire (System-Wide Change)
by Ben Cotton
https://fedoraproject.org/wiki/Changes/DefaultPipeWire
== Summary ==
This change proposal is to route all audio from PulseAudio and JACK to
the PipeWire Audio
daemon by default.
== Owner ==
* Name: [[User:Wtaymans| Wim Taymans]]
* Email: wim.taymans(a)gmail.com
== Detailed Description ==
Currently, all desktop audio is handled by the PulseAudio daemon.
Applications make use of the
PulseAudio client library to communicate with the PulseAudio daemon
that mixes and manages the audio streams from the clients.
The desktop shell (gnome-shell) and the control panel
(gnome-control-panel) both use the
Pulseaudio client libraries to manage the volume and configuration of
the PulseAudio daemon.
This proposal is to replace the PulseAudio daemon with a functionally
compatible implementation
based on PipeWire. This means that all existing clients using the
PulseAudio client library
will continue to work as before, as well as applications shipped as Flatpak.
All PRO audio is handled with the JACK client library, which talks to
the JACK server. This
proposal will install a JACK client library replacement that talks
directly to PipeWire. All
existing PRO audio jack applications will then work on top of PipeWire.
For legacy ALSA clients, we will install an ALSA plugin that routes
the audio directly to
PipeWire.
With these 3 changes, all audio will be routed to PipeWire. There will
then be no more need to
install the PulseAudio and JACK daemons.
== Feedback ==
The owner of this proposal has been in context with both the
PulseAudio and JACK maintainers and community.
PipeWire is considered to be the successor of both projects.
== Benefit to Fedora ==
The end goal is to end up with one audio infrastructure for both
Desktop and Pro audio use cases.
This will end the fragmentation of the audio landscape.
Some of the benefits that PipeWire will bring:
* PRO Audio features
PipeWire can support both Desktop and PRO Audio use cases. PRO
Audio application tend to use
the JACK API and JACK daemon, which is hard to setup and integrates
poorly with the rest of
the system (and PulseAudio in particular).
With a replacement libjack library, PRO Audio application can run
directly on PipeWire and
integrate seamlessly with other ALSA and PulseAudio applications.
This would bring Fedora
closer to the experience of other operating systems.
* Flexibility/Integration
PipeWire is designed to be multiprocess. It separates the
processing of the multimedia graph
and the management into separate processes. This makes it possible
to better integrate with
the other system components or swap out the default policy for a
highly customized one (such as
for automotive or embedded). This is in contrast to PulseAudio,
which has all logic embedded
into the daemon with limited configuration options.
In the next phase we expect to greatly expand the user experience
and configuration of the
audio infrastructure with better integration throughout the system.
* Performance
PipeWire was designed for high performance and low-latency, using
much of the same design as
JACK. JACK application should run with comparable performance even
in low-latency situations.
* Security
PipeWire enforces per client security. Object visibility and the
actions on them can be
configured independently per client. This makes it possible to
enforce a security policy for
sandboxed applications (Flatpak) such as denying access to certain
audio capture devices or
block them from interfering with other applications.
* Maintainability
Both PulseAudio and JACK have very slow development cycles with few
new features. The more
flexible and distributed nature of the design of PipeWire should
encourage more new features
and use-cases.
== Scope ==
* Proposal owners:
We would make a pipewire-pulse package that provides the same features
as the pulseaudio (daemon) package.
We would only provide a drop-in replacement daemon, the pulseaudio
client libraries will remain unchanged.
The idea is that when installing pipewire-pulse, only the pulseaudio
package is removed and replaced by the
pipewire-pulse implementation. In the same way, installing the
pulseaudio package would remove the pipewire-pulse
package, making it possible to switch between implementations. This
will also allow for an easy rollback.
We also need to check and correct the dependencies of other packages.
As of this writing, some packages do
not state their dependencies correctly and get removed when pulseaudio
is removed. We also need to check the
JACK to make sure they still install with the replacement JACK client library.
The JACK client libraries will be installed in the same way, removing
the old JACK client libraries.
* Other developers:
The distribution needs to default to the pipewire-pulse package
instead of pulseaudio.
JACK applications need to install the pipewire-libjack client library.
* Release engineering: [https://pagure.io/releng/issues #Releng issue
number] (a check of an impact with Release Engineering is needed)
* Policies and guidelines:
* Trademark approval: N/A (not needed for this Change)
== Upgrade/compatibility impact ==
The pulseaudio package will be uninstalled and pipewire-pulse will be installed.
pipewire-pulse does not yet implement all the features of pulseaudio
but it is expected that
comparable functionality will be implemented later. Most notable
features that are likely
not going to be available for fedora 34
* avahi network discovery and audio routing. This is not enabled by
default but can be activated
with paprefs. this includes TCP and RTP transport of audio.
* make devices available as UPNP media servers. Not enabled by
default, paprefs can be used.
* easy configuration of combining all sinks. Questionable feature but
possible via paprefs.
User scripts will still work but custom configurations of the
pulseaudio daemon will not be used
anymore.
Most of the JACK workflow of managing the JACK daemon is not going to
be needed anymore as things
will work out-of-the-box. As of this writing, these things are missing
from the JACK implementation,
we hope to implement them before fedora 34:
* latency reporting: useful to align streams
* freewheeling: used when rendering a project
* jackdbus: used by some tools to manage the graph
== How To Test ==
This change needs to be tested on as many different audio cards as
possible. The same test plan applies here as with PulseAudio.
To test, one needs to install the pipewire-pulse library (which
removes the pulseaudio package).
To test the JACK support, one needs to install pipewire-libjack, which
removes the original
JACK client and server.
After these changes, a restart is needed to make sure the new
pipewire-pulse daemon is running.
Audio functionality should be like it was before with the Pulseaudio
daemon. Some things to verify:
- patcl info should now list: Server Name: PulseAudio (on PipeWire 0.3.x)
- gnome-control-center: check the audio tab, check the volume sliders
and do the audio channel test. Change the card profile, plug/unplug
headphones and observe correct
switch.
- pavucontrol: check volumes in the input devices tabs and check the
microphone volumes
- firefox: check out a website with audio/video such as youtube and
verify that audio works as
usual. Check out a website with a video chat test page
(bluejeans.com/111).
- rhythmbox: check if playback works as expected
- bluetooth devices, connect as usual and verify working behaviour
with PipeWire. Check volume changes etc.
- Regular system usage and performance should not change.
- JACK tools such as catia, carla should run and can be used to
inspect the graph.
== User Experience ==
In general, users should not be able to see any change when using
PulseAudio applications.
The big change is when using JACK application:
- They will start without having to configure and start the daemon.
this includes
the period size and sample rates.
- All devices will be visible in the graph with meaningful names. Devices will
be automatically slaved when needed without needing any configuration.
- bluetooth devices will be usable as well.
== Dependencies ==
Other packages might need to have their requirements fixed to work
with the replacement packages
but this is under our control.
== Contingency Plan ==
* Contingency mechanism: Keep existing pulseaudio and JACK client
libraries as defaults
* Contingency deadline: beta freeze
* Blocks release? No
* Blocks product? No
== Documentation ==
[https://pipewire.org](PipeWire website)
[https://www.youtube.com/watch?v=8LZt4loZu64&t=14s](Video with Current status)
[https://gitlab.freedesktop.org/pipewire/pipewire/-/blob/master/INSTALL.md...
guide)
[https://gitlab.freedesktop.org/pipewire/pipewire/-/blob/master/README.md]...
to use/test)
--
Ben Cotton
He / Him / His
Senior Program Manager, Fedora & CentOS Stream
Red Hat
TZ=America/Indiana/Indianapolis
3 years, 1 month
Fedora 34 Change: GitRepos-master-to-main (Self-Contained Change)
by Ben Cotton
https://fedoraproject.org/wiki/Changes/GitRepos-master-to-main
== Summary ==
This Change will move Fedora git repositories to use "main" as the
default git branch instead of "master". Specific repositories will be
manually moved and default git branch for new projects will be set to
use "main".
The Fedora Community strives to be open and welcoming. Some language
around our git repositories is dated and could be more inclusive. Many
git repositories currently use "master" as the default branch. This
Change will move many repositories (see below) to use a "main" branch
as default. This small bit of naming adjustment is in-line with
Fedora's vision for free and open source software built by inclusive,
welcoming, and open-minded communities.
== Owner ==
* Name: [[User:Kevin| Kevin Fenzi]], [[User:bcotton| Ben Cotton]],
[[User:pingou| pingou]], [[User:mohanboddu| Mohan Boddu]],
[[User:till| Till Maas]]
* Email: kevin(a)scrye.com, bcotton(a)redhat.com, pingou(a)pingoured.fr,
mboddu(a)bhujji.com, opensource(a)till.name
== Detailed Description ==
The Fedora Project controls a number of git repositories. This change
will move the default branch (that is, the git branch used when
nothing is specified) from 'master' to 'main'. Existing git clones
will need to pull to get the changed default branch. Existing Pull
Requests against the 'master' branch will need to be rebased against
the 'main' branch. Documentation will be updated.
== Benefit to Fedora ==
The Fedora Project will be a more welcoming place for new contributors.
== Scope ==
This change will take place in a number of phases:
=== Phase0 - 2020-12-14 ===
A guide will be published to explain how maintainers/project
managers can change the default
branch on pagure.io, which they can then do based in their projects desires.
=== Phase1 - 2020-12-15 ===
The following repos will be switched:
src.fedoraproject.org/flatpacks/*
pagure.io:
releng
releng/*
fedora-comps
fedora-kickstarts
fedora-infrastructure
fedora-lorax-templates
fedora-mediawikitheme
fedora-packager
fedora-infra/*
infra-docs
koji-fedmsg-plugin
workstation-ostree-config
pungi-fedora
github.com
fedora-infra/*
=== Phase2 - 2021-01-13 ===
src.fedoraproject.org/*
pagure.io default for new projects will be changed to 'main'
* Proposal owners:
Switching all above listed projects git repos to use 'main'
Deleting the 'master' branch
Announcing when changes have been made on devel-announce / other lists.
* Other developers:
Other developers are encouraged to change their upstream projects
on pagure.io, github or gitlab.
* Release engineering:
Releng will adjust any scripts that assume 'master' branch to use
'main' instead. The list includes and maybe few more
[https://pagure.io/releng/blob/master/f/scripts/update-critpath.py
update-critpath.py]
[https://pagure.io/releng/blob/master/f/scripts/block_retired.py
block_retired.py]
[https://pagure.io/releng/blob/master/f/scripts/update-docs.sh
update-docs.sh]
[https://pagure.io/releng/blob/master/f/scripts/find_unblocked_orphans.py
find_unblocked_orphans.py]
[https://pagure.io/releng/blob/master/f/scripts/mass-rebuild-second-run.py
mass-rebuild-second-run.py]
[https://pagure.io/releng/blob/master/f/scripts/adjust-eol-modules.py
adjust-eol-modules.py]
[https://pagure.io/releng/blob/master/f/scripts/pdc/adjust-eol-modules.py
adjust-eol-modules.py]
[https://pagure.io/releng/blob/master/f/scripts/pdc/adjust-eol-modules.py
adjust-eol-modules.py]
[https://pagure.io/releng/blob/master/f/scripts/pdc/adjust-eol.py
adjust-eol.py]
[https://pagure.io/releng/blob/master/f/scripts/pdc/create-new-release-bra...
create-new-release-branches.py]
[https://pagure.io/releng/blob/master/f/scripts/pdc/create-branch.py
create-branch.py]
[https://pagure.io/releng/blob/master/f/scripts/pdc/adjust-eol-all.py
adjust-eol-all.py]
[https://pagure.io/releng/blob/master/f/scripts/check-unretirement.py
check-unretirement.py]
[https://pagure.io/releng/blob/master/f/scripts/mass-rebuild-special.py
mass-rebuild-special.py]
[https://pagure.io/releng/blob/master/f/scripts/need-rebuild.py
need-rebuild.py]
[https://pagure.io/releng/blob/master/f/scripts/distgit-branch-unused.py
distgit-branch-unused.py]
[https://pagure.io/releng/blob/master/f/scripts/create-new-release-branche...
create-new-release-branches.py]
* Policies and guidelines: N/A
* Trademark approval: N/A
* Alignment with Objectives:
== Upgrade/compatibility impact ==
Users with old checkouts will need to update their git configuration
or re-clone repositories that have changed before they can see any new
changes.
Repos used to build content for docs.fedoraproject.org can change the
default git branch, but the Antora module version (in `antora.yml`)
should remain `master` pending support for alternate "unversioned"
versions upstream.
== How To Test ==
# git clone <one of the listed repositories>
# cd <repositoryname>
# git branch
should return: `* main`
== User Experience ==
Users and developers will see more welcoming language and that the
fedora project expended effort to be more welcoming to them.
== Dependencies ==
none
== Contingency Plan ==
* Contingency mechanism: Revert repositories, scripts, and docs back
to the unwelcoming 'master'
* Contingency date: 2020-01-13
== Documentation ==
A short guide on how to change this for pagure.io will be produced.
== Release Notes ==
Many git repositories used to create Fedora releases were moved to use
a 'main' branch instead of a 'master' branch. The Fedora Project
envisions a world where everyone benefits from free and open source
software built by inclusive, welcoming, and open-minded communities.
--
Ben Cotton
He / Him / His
Senior Program Manager, Fedora & CentOS Stream
Red Hat
TZ=America/Indiana/Indianapolis
3 years, 1 month
Fedora 34 Change: Enable btrfs transparent zstd compression by
default (System-Wide Change proposal)
by Ben Cotton
https://fedoraproject.org/wiki/Changes/BtrfsTransparentCompression
== Summary ==
On variants using btrfs as the default filesystem, enable transparent
compression using zstd. Compression saves space and can significantly
increase the lifespan of flash-based media by reducing write
amplification. It can also increase read and write performance.
== Owners ==
* Name: [[User:salimma|Michel Salim]], [[User:dcavalca|Davide
Cavalca]], [[User:josef|Josef Bacik]]
* Email: michel(a)michel-slm.name, dcavalca(a)fb.com, josef(a)toxicpanda.com
== Detailed description ==
Transparent compression is a btrfs feature that allows a btrfs
filesystem to apply compression on a per-file basis. Of the three
supported algorithms, zstd is the one with the best compression speed
and ratio. Enabling compression saves space, but it also reduces write
amplification, which is important for SSDs. Depending on the workload
and the hardware, compression can also result in an increase in read
and write performance.
See https://pagure.io/fedora-btrfs/project/issue/5 for details. This
was originally scoped as an optimization for
https://fedoraproject.org/wiki/Changes/BtrfsByDefault during Fedora
33.
== Benefit to Fedora ==
Better disk space usage, reduction of write amplification, which in
turn helps increase lifespan and performance on SSDs and other
flash-based media. It can also increase read and write performance.
== Scope ==
* Proposal owners:
** Update anaconda to perform the installation using <code>mount -o
compress=zstd:1</code>
** Set the proper option in fstab (alternatively: set the XATTR)
** Update disk image build tools to enable compression:
*** lorax
*** appliance-tools
*** osbuild
*** imagefactory
** [optional] Add support for
[https://github.com/kdave/btrfs-progs/issues/328 setting compression
level when defragmenting]
** [optional] Add support for
[https://github.com/kdave/btrfs-progs/issues/329 setting compression
level using `btrfs property`]
* Other developers:
** anaconda: review PRs as needed
* Release engineering: https://pagure.io/releng/issue/9920
* Policies and guidelines: N/A
* Trademark approval: N/A
== Upgrade/compatibility impact ==
This Change only applies to newly installed systems. Existing systems
on upgrade will be unaffected, but can be converted manually with
<code>btrfs filesystem defrag -czstd -r</code>, updating `/etc/fstab`
and remounting.
== How to test ==
Existing systems can be converted to use compression manually with
<code>btrfs filesystem defrag -czstd -r</code>, updating `/etc/fstab`
and remounting.
== User experience ==
Compression will result in file sizes (e.g. as reported by du) not
matching the actual space occupied on disk. The
[https://src.fedoraproject.org/rpms/compsize compsize] utility can be
used to examine the compression type, effective compression ration and
actual size.
== Dependencies ==
Anaconda will need to be updated to perform the installation using
<code>mount -o compress=zstd:1</code>
== Contingency plan ==
* Contingency mechanism: will not include PR patches if not merged
upstream and will not enable
* Contingency deadline: Final freeze
* Blocks release? No
* Blocks product? No
== Documentation ==
https://btrfs.wiki.kernel.org/index.php/Compression
== Release Notes ==
Transparent compression of the filesystem using zstd is now enabled by
default. Use the compsize utility to find out the actual size on disk
of a given file.
--
Ben Cotton
He / Him / His
Senior Program Manager, Fedora & CentOS Stream
Red Hat
TZ=America/Indiana/Indianapolis
3 years, 2 months
Fedora 34 Change proposal: Remove make from BuildRoot (System-Wide Change)
by Ben Cotton
https://fedoraproject.org/wiki/Changes/Remove_make_from_BuildRoot
== Summary ==
This change will remove make from the default buildroot in Koji and mock.
== Owner ==
* Name: [[User:tstellar| Tom Stellard]]
* Email: <tstellar(a)redhat.com>
== Detailed Description ==
===== Phase 1: Analysis =====
For this change, we will start by creating a list of all packages that
have a build-time dependency on make. This will be done by analyzing
spec files and also by rebuilding all packages in Fedora with make
removed from the buildroot to see which packages fail to build. Once
we have this list, we will remove packages that already have
BuildRequires:make in their spec file, and this final list[1] will be
all the packages that need to be updated in Phase 2.
[1] https://github.com/tstellar/fedora-change-make-buildroot/blob/main/needs_...
===== Phase 2: Package Updates =====
Once we have the list of packages that need to be updated, we will
proceed with adding BuildRequires: make to all spec files that require
it. This new BuildRequires will be added to the line after the last
BuildRequires in the spec file. The release number for packages will
'''*not*''' be incremented for this change.
The spec file updates will be automated and changes will be pushed
directly to dist-git once they are ready.
===== Phase 3: Update Buildroot =====
Once package spec files have been updated, then we can proceed with
removing make from the BuildRoot. This will be done by removing make
from the build group in Koji and by removing make from the
buildsys-build group in comps (fedora-comps). In order to avoid
issues with Koschei, this change will need to be made as close as
possible to the start of the mass rebuild. If possible, we will try
to first remove make from the mass rebuild side-tag and then remove it
from rawhide after the rebuild completes.
===== Phase 4: Monitor Failures =====
Once all the changes are in place, we will continue to monitor koji
builds to see if there are any failures related to this change. We
expect that the analysis done in Phase 1 would give us the complete
list of packages that need to be updated, but it is always possible
that something will be missed.
== Feedback ==
* Removing make from the Buildroot without rebuilding the packages
first has the potential to cause mass failures in Koschei. This is
because Koschei builds from the latest SRPM and not from dist-git. We
can minimize this problem by removing make from the buildroot as close
as possible to the mass rebuild. The proposal has been updated now to
account for this issue.
== Benefit to Fedora ==
Based on my research[1], Fedora Rawhide has 22,309 packages, and there
are approximately 10,378 packages that either use make explicitly or
fail to build when make is removed form the buildroot. This means
that there are around 11,931 packages that don't need make. Removing
make from the BuildRoot will reduce the network traffic, download
times, and disk usage for these builds in Koji and also for users
doing builds with mock.
Removing make (and its dependencies) will:
* Reduce the BuildRoot download size by: 7.3 MB [2]:
** make: 539k
** gc: 104k
** guile22: 6.6M
** libtool-ltdl: 36k
* Reduce the BuildRoot install size by; 46 MB [2]:
** make: 1.6M
** gc: 229k
** guile22: 44M
** libtool-ltdl: 71k
[1] https://github.com/tstellar/fedora-change-make-buildroot
[2] Package sizes are from the x86_64 packages.
== Scope ==
* '''Proposal owners:''' Tom Stellard
* '''Package Maintainers:''' Fedora package maintainers should report
bugs if they think there is a problem caused by this change, but
otherwise there will be no action required by them.
* '''Proven Packager:''' The proposal owner will need to either apply
for provenpackager status or get the help of someone with
provenpackager status in order to make the spec file changes that are
required.
* '''Release engineering:''' [https://pagure.io/releng/issue/9829
#9829] (a check of an impact with Release Engineering is needed)
* '''Policies and guidelines:''' The packager guidelines will need to
be updated to mention that BuildRequires: make is now required for all
packages that need make.
* '''Trademark approval:''' N/A (not needed for this Change)
* '''Alignment with Objectives:''' This aligns with the Fedora
Minimization [https://pagure.io/minimization/issue/12 objective].
== Upgrade/compatibility impact ==
This should not impact upgrades.
== How To Test ==
This change can be tested by rebuilding affected packages. The goal
is to complete this before the mass rebuild to ensure maximum testing
coverage.
== User Experience ==
Fedora users will not notice any difference with this change. This
will only impact Fedora package maintainers.
== Dependencies ==
gcc < 11 will fail if the -flto=auto option is used when make is not
installed. Phase 3 cannot be completed until gcc 11 lands in rawhide,
but Phase 1 and Phase 2 are not blocked by this.
== Contingency Plan ==
* '''Contingency mechanism:''' If we discover critical issues during
phase 4, then Fedora Release Engineering will need to re-add make into
the buildroot.
* '''Contingency deadline:''' 2021-02-23 (F34 Beta Freeze)
* '''Blocks release?''' No
* '''Blocks product?''' No
== Documentation ==
The packager guidelines will need to be updated to mention that
BuildRequires: make is now required for all packages that need make.
--
Ben Cotton
He / Him / His
Senior Program Manager, Fedora & CentOS Stream
Red Hat
TZ=America/Indiana/Indianapolis
3 years, 2 months
our containers with alias vim=vi
by clime
Hello,
could Fedora and CentOS containers for docker and podman come with
`alias vim=vi` in ~/.bashrc?
I would very much welcome it as I am used to type vim everywhere but
if vi starts instead I am happy too. I know that the solution is to
create a customized container but often I want to try something on
vanilla containers from the whole range.
Didn't want to write about this first but maybe there are more people
with the same problem.
clime
3 years, 2 months
Fedora 34 Change: DNF/RPM Copy on Write enablement for all variants
(System-Wide Change)
by Ben Cotton
https://fedoraproject.org/wiki/Changes/RPMCoW
== Summary ==
RPM Copy on Write provides a better experience for Fedora Users as it
reduces the amount of I/O and offsets CPU cost of package
decompression. RPM Copy on Write uses reflinking capabilities in
btrfs, which is the default filesystem in Fedora 33.
== Owners ==
* Name: [[User:malmond|Matthew Almond]], [[User:dcavalca|Davide Cavalca]]
* Email: malmond(a)fb.com, dcavalca(a)fb.com
== Detailed description ==
Installing and upgrading software packages is a standard part of
managing the lifecycle of any operating system. For the entire
lifecycle of Fedora, all software is packaged and distributed using
the RPM file fomat. This proposal changes how software is downloaded
and installed, leaving the distribution process unmodified.
=== Current process ===
# Resolve packaging request into a list of packages and operations
# Download and verify new packages
# Install and/or upgrade packages sequentially using RPM files,
decompressing, and writing a copy of the new files to storage.
=== New process ===
# Resolve packaging request into a list of packages and operations
# Download and '''decompress''' packages into a '''locally optimized''' rpm file
# Install and/or upgrade packages sequentially using RPM files, using
'''reference linking''' (reflinking) to reuse data already on disk.
The outcome is intended to be the same, but the order of operations is
different.
# Decompression happens inline with download. This has a positive
effect on resource usage: downloads are typically limited by
bandwidth. Decompression and writing the full data into a single file
per rpm is essentially free. Additionally: if there is more than one
download at a time, a multi-CPU system can be better utilized. All
compression types supported in RPM work because this uses the rpm I/O
functions.
# RPMs are cached on local storage between downloading and
installation time as normal. This allows DNF to defer actual RPM
installation to when all the RPM are available. This is unchanged.
# The file format for RPMs is different with Copy on Write. The
headers are identical, but the payload is different. There is also a
footer.
## Files are converted (“transcoded”) locally during download using
<code>/usr/bin/rpm2extents</code> (part of rpm codebase). The format
is not intended to be “portable” - i.e. copying the files from the
cache is not supported.
## Regular RPMs use a compressed .cpio based payload. In contrast,
extent based RPMs contain uncompressed data aligned to the fundamental
page size of the architecture, e.g. 4KiB on x86_64. This alignment is
required for <code>FICLONERANGE</code> to work. Only files are
represented in the payload, other directory entries like symlinks,
device nodes etc are constructed entirely from rpm header information.
Files are referenced by their digest, so identical files are
de-duplicated.
## The footer currently has three sections
### Table of original (rpm) file digests, used to validate the
integrity of the download in dnf.
### Table of digest → offset used when actually installing files.
### Signature 8 bytes at the end of the file, used to differentiate
between traditional RPMs and extent based.
=== Notes ===
# The headers are preserved bit for bit during transcoding. This
preserves signatures. The signatures cover the main header blob, and
the main header blob ensures the integrity of data in two ways:
## Each file with content has a digest. Originally this was md5, but
today it’s usually sha256. In normal RPM this is only used to verify
the integrity of files, e.g. <code>rpm -V</code>. With CoW we use this
as a content key.
## There is/are one or two digests (<code>PAYLOADDIGEST</code> and
<code>PAYLOADDIGESTALT</code>) covering the payload archive
(compressed cpio). The header value is preserved, but transcoded RPMs
do not preserve the original structure so RPM’s pre-installation
verification (controlled by <code>%_pkgverify_level</code> will fail.
<code>dnf-plugin-cow</code> disables this check in dnf because it
verifies the whole file digest which is captured during
download/transcoding. The second one is likely used for delta rpm.
# This is untested, and possibly incompatible with delta RPM (drpm).
The process for reconstructing an rpm to install from a delta is
expensive from both a CPU and I/O perspective, while only providing
marginal benefits on download size. It is expected that having delta
rpm enabled (which is the default) will be handled gracefully.
# Disk space requirements are expected to be marginally higher than
before: all new packages or updates will consume their installed size
before installation instead of about half their size (regular rpms
with payloads still cost space).
# <code>rpm-plugin-reflink</code> will fall back to simple file
copying when the destination path is not on the same
filesystem/subvolume. A common example is <code>/boot</code> and/or
<code>/boot/efi</code>.
# The system will still work on other filesystem types, but will
''always'' fall back to simple copying. This is expected to be
slightly slower than not enabling CoW because the source for copying
will be the decompressed data.
# For systems that enable transparent filesystem compression: every
file will continue to be decompressed from the original rpm, and then
transparently re-compressed by the filesystem. There is no effective
change here. There is a future project to investigate alternate
distribution mechanics to provide parallel versions of file content
pre-compressed in a filesystem specific format, reducing both CPU
costs and I/O. It is expected that this will result in slightly higher
network utilization because filesystem compression is purposely
restricted to allow random I/O.
# Current implementation of <code>dnf-plugin-cow</code> is in Python,
but it looks possible to implement this in <code>libdnf</code> instead
which would make it work in <code>packagekit</code>.
=== Performance Metrics ===
Ballpark performance difference is about half the duration for file
download+install time. A lot of rpms are very small, so it’s difficult
to see/measure. Larger RPMs give much clearer signal.
(Actual numbers/charts will be supplied in Jan 2021)
=== Terminology ===
* '''Copy on Write (CoW)''' is a broad description of any technology
that reduces or eliminates data duplication by sharing the data behind
the scenes until one of the references makes changes. This has been a
cornerstone technology in memory management in Unix systems. Here we
are using it to specifically reference Copy on Write as supported in
modern filesystems, e.g. btrfs, xfs and potentially others.
* '''Reflink''' is the verb for duplicating stored data on a
filesystem. See
[https://man7.org/linux/man-pages/man2/ioctl_ficlonerange.2.html
ioctl_ficlonerange(2)] for the specific call we use on Linux
* '''Extent''' (based RPMs) refers to how payload file data is stored
in within an RPM. Normal RPMs simply contain a compressed CPIO
archive. Extent based RPMs contain the raw data uncompressed, which
can be referenced with reflink.
== Benefit to Fedora ==
Faster package installs and upgrades
== Scope ==
* Proposal owners:
** Merge changes to rpm, librepo to enable capabilities
** Add dnf-plugin-cow to available packages
** Test days
** Aid with documentation
* Other developers:
** rpm, librepo: review PRs as needed
* Release engineering: https://pagure.io/releng/issue/9914
* Policies and guidelines: N/A
* Trademark approval: N/A
== Upgrade/compatibility impact ==
None, RPM with CoW is not enabled by default.
Upgrades with <code>keepcache</code> in dnf.conf will be able to use
existing packages, but it will not convert them. This only happens at
download time.
If a system is configured to keep packages in the cache
(<code>keepcache</code> in <code>dnf.conf</code>) and
<code>dnf-plugin-cow</code> is removed then the packages will be
unusable. Recommend <code>dnf clean packages</code> to resolve this.
== How to test ==
Enable RPM with CoW with
<pre>$ sudo dnf install dnf-plugin-cow
...
$ sudo dnf install hello
...
$ hello
Hello, world!</pre>
There should be no end user visible changes, except timing.
== User experience ==
No anticipated user visible changes in this change proposal. This
makes the feature available, but does not enable it by default.
== Dependencies ==
# A copy-on-write filesystem; this Change is primarily targeting
btrfs, but RPM with CoW should work with XFS as well (untested)
# Most package install paths and the dnf package cache on the same
filesystem / subvolume.
# <code>rpm</code> with Copy on Write patch set:
https://github.com/malmond77/rpm/tree/cow
# <code>librepo</code> with transcoding support:
https://github.com/malmond77/librepo/tree/transcode_cow
# dnf-plugin-reflink (a new package):
https://github.com/facebookincubator/dnf-plugin-cow/
== Contingency plan ==
* Contingency mechanism: will not include PR patches if not merged
upstream, skip <code>dnf-plugin-cow</code>
* Contingency deadline: Final freeze
* Blocks release? No
* Blocks product? No
== Documentation ==
Documentation will be available at
https://github.com/facebookincubator/dnf-plugin-cow in the coming
weeks
== Release Notes ==
RPM with CoW is not enabled by default. To enable it:
<pre>$ sudo dnf install dnf-plugin-cow</pre>
--
Ben Cotton
He / Him / His
Senior Program Manager, Fedora & CentOS Stream
Red Hat
TZ=America/Indiana/Indianapolis
3 years, 2 months
