F37 proposal: Add -fno-omit-frame-pointer to default compilation
flags (System-Wide Change proposal)
by Ben Cotton
https://fedoraproject.org/wiki/Changes/fno-omit-frame-pointer
This document represents a proposed Change. As part of the Changes
process, proposals are publicly announced in order to receive
community feedback. This proposal will only be implemented if approved
by the Fedora Engineering Steering Committee.
== Summary ==
Fedora will add -fno-omit-frame-pointer to the default C/C++
compilation flags, which will improve the effectiveness of profiling
and debugging tools.
== Owner ==
* Name: [[User:daandemeyer| Daan De Meyer]], [[User:Dcavalca| Davide
Cavalca]], [[ Andrii Nakryiko]]
* Email: daandemeyer(a)fb.com, dcavalca(a)fb.com, andriin(a)fb.com
== Detailed Description ==
Credits to Mirek Klimos, whose internal note on stacktrace unwinding
formed the basis for this change proposal (myreggg(a)gmail.com).
Any performance or efficiency work relies on accurate profiling data.
Sampling profilers probe the target program's call stack at regular
intervals and store the stack traces. If we collect enough of them, we
can closely approximate the real cost of a library or function with
minimal runtime overhead.
Stack trace capture what’s running on a thread. It should start with
clone - if the thread was created via clone syscall - or with _start -
if it’s the main thread of the process. The last function in the stack
trace is code that CPU is currently executing. If a stack starts with
[unknown] or any other symbol, it means it's not complete.
=== Unwinding ===
How does the profiler get the list of function names? There are two parts of it:
# Unwinding the stack - getting a list of virtual addresses pointing
to the executable code
# Symbolization - translating virtual addresses into human-readable
information, like function name, inlined functions at the address, or
file name and line number.
Unwinding is what we're interested in for the purpose of this
proposal. The important things are:
* Data on stack is split into frames, each frame belonging to one function.
* Right before each function call, the return address is put on the
stack. This is the instruction address in the caller to which we will
eventually return — and that's what we care about.
* One register, called the "frame pointer" or "base pointer" register
(RBP), is traditionally used to point to the beginning of the current
frame. Every function should back up RBP onto the stack and set it
properly at the very beginning.
The “frame pointer” part is achieved by adding push %rbp, mov
%rsp,%rbp to the beginning of every function and by adding pop %rbp
before returning. Using this knowledge, stack unwinding boils down to
traversing a linked list:
https://i.imgur.com/P6pFdPD.png
=== Where’s the catch? ===
The frame pointer register is not necessary to run a compiled binary.
It makes it easy to unwind the stack, and some debugging tools rely on
frame pointers, but the compiler knows how much data it put on the
stack, so it can generate code that doesn't need the RBP. Not using
the frame pointer register can make a program more efficient:
* We don’t need to back up the value of the register onto the stack,
which saves 3 instructions per function.
* We can treat the RBP as a general-purpose register and use it for
something else.
Whether the compiler sets frame pointer or not is controlled by the
-fomit-frame-pointer flag and the default is "omit", meaning we can’t
use this method of stack unwinding by default.
To make it possible to rely on the frame pointer being available,
we'll add -fno-omit-frame-pointer to the default C/C++ compilation
flags. This will instruct the compiler to make sure the frame pointer
is always available. This will in turn allow profiling tools to
provide accurate performance data which can drive performance
improvements in core libraries and executables.
== Feedback ==
=== Potential performance impact ===
* Meta builds all its libraries and executables with
-fno-omit-frame-pointer by default. Internal benchmarks did not show
significant impact on performance when omitting the frame pointer for
two of our most performance intensive applications.
* Firefox recently landed a change to preserve the frame pointer in
all jitted code
(https://bugzilla.mozilla.org/show_bug.cgi?id=1426134). No significant
decrease in performance was observed.
* Kernel 4.8 frame pointer benchmarks by Suse showed 5%-10%
regressions in some benchmarks
(https://lore.kernel.org/all/20170602104048.jkkzssljsompjdwy@suse.de/T/#u)
Should individual libraries or executables notice a significant
performance degradation caused by including the frame pointer
everywhere, these packages can opt-out on an individual basis as
described in https://docs.fedoraproject.org/en-US/packaging-guidelines/#_compiler_flags.
=== Alternatives to frame pointers ===
There are a few alternative ways to unwind stacks instead of using the
frame pointer:
* [https://dwarfstd.org DWARF] data - The compiler can emit extra
information that allows us to find the beginning of the frame without
the frame pointer, which means we can walk the stack exactly as
before. The problem is that we need to unwind the stack in kernel
space which isn't implemented in the kernel. Given that the kernel
implemented it's own format (ORC) instead of using DWARF, it's
unlikely that we'll see a DWARF unwinder in the kernel any time soon.
The perf tool allows you to use the DWARF data with
--call-graph=dwarf, but this means that it copies the full stack on
every event and unwinds in user space. This has very high overhead.
* [https://www.kernel.org/doc/html/v5.3/x86/orc-unwinder.html ORC]
(undwarf) - problems with unwinding in kernel led to creation of
another format with the same purpose as DWARF, just much simpler. This
can only be used to unwind kernel stack traces; it doesn't help us
with userspace stacks. More information on ORC can be found
[https://lwn.net/Articles/728339 here].
* [https://lwn.net/Articles/680985 LBR] - New Intel CPUs have a
feature that gives you source and target addresses for the last 16 (or
32, in newer CPUs) branches with no overhead. It can be configured to
record only function calls and to be used as a stack, which means it
can be used to get the stack trace. Sadly, you only get the last X
calls, and not the full stack trace, so the data can be very
incomplete. On top of that, many Fedora users might still be using
CPUs without LBR support which means we wouldn't be able to assume
working profilers on a Fedora system by default.
To summarize, if we want complete stacks with reasonably low overhead
(which we do, there's no other way to get accurate profiling data from
running services), frame pointers are currently the best option.
== Benefit to Fedora ==
Implementing this change will provide profiling tools with easy access
to stacktraces of installed libraries and executables which will lead
to more accurate profiling data in general. This in turn can be used
to implement optimizations to core libraries and executables which
will improve the overall performance of Fedora itself and the wider
Linux ecosystem.
Various debugging tools can also make use of the frame pointer to
access the current stacktrace, although tools like gdb can already do
this to some degree via embedded dwarf debugging info.
== Scope ==
* Proposal owners: Put up a PR to change the rpm macros to build
packages by default with -fno-omit-frame-pointer by default.
* Other developers: Review and merge the PR implementing the Change.
* Release engineering: [https://pagure.io/releng/issues #Releng issue
number]. A mass rebuild is required.
* Policies and guidelines: N/A (not needed for this Change)
* Trademark approval: N/A (not needed for this Change)
* Alignment with Objectives: N/A
== Upgrade/compatibility impact ==
This should not impact upgrades in any way.
== How To Test ==
# Build the package with the updated rpm macros
# Profile the binary with `perf record -g <binary>`
# Inspect the perf data with `perf report -g 'graph,0.5,caller'`
# When expanding hot functions in the perf report, perf should show
the full call graph of the hot function (at least for all functions
that are part of the binary compiled with -fno-omit-frame-pointer)
== User Experience ==
Fedora users will be more likely to have a streamlined experience when
trying to debug/profile system executables/libraries. Tools such as
perf will work out of the box instead of requiring to users to provide
extra options (e.g. --call-graph=dwarf/LBR) or requiring users to
recompile all relevant packages with -fno-omit-frame-pointer.
== Dependencies ==
The rpm macros for Fedora need to be adjusted to include
-fno-omit-frame-pointer in the default C/C++ compilation flags.
== Contingency Plan ==
* Contingency mechanism: The new version can be released without every
package being rebuilt with fno-omit-frame-pointer. Profiling will only
work perfectly once all packages have been rebuilt but there will be
no regression in behavior if not all packages have been rebuilt by the
time of the release. If the Change is found to introduce unacceptable
regressions, the PR implementing it can be reverted and affected
packages can be rebuilt.
* Contingency deadline: Final freeze
* Blocks release? No
== Documentation ==
* Original proposal for in-kernel DWARF unwinder (rejected):
https://lkml.org/lkml/2017/5/5/571
== Release Notes ==
Packages are now compiled with frame pointers included by default.
This will enable a variety of profiling and debugging tools to show
more information out of the box.
--
Ben Cotton
He / Him / His
Fedora Program Manager
Red Hat
TZ=America/Indiana/Indianapolis
1 year, 4 months
fedpkg clone fails with Permission denied (publickey).
by Richard Shaw
Long story short I lost my home directory where I do all of my packager
activities (separate from my main user) so I'm setting things up from
scratch.
I created new ssh keys and uploaded the public key to
admin.fedoraproject.org and pasted into pagure.io. It's been over an hour
and I'm still getting:
$ fedpkg clone hamlib
Cloning into 'hamlib'...
hobbes1069(a)pkgs.fedoraproject.org: Permission denied (publickey).
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
Could not execute clone: Failed to execute command.
I've also updated my API tokens, which is STILL not well documented. I
pasted them in the appropriate spot in "/etc/rpkg/fedpkg.conf" which isn't
real intuitive.
Thanks,
Richard
1 year, 5 months
F38 Proposal: SPDX License Phase 1 (Self-Contained Change proposal)
by Ben Cotton
https://fedoraproject.org/wiki/Changes/SPDX_Licenses_Phase_1
This document represents a proposed Change. As part of the Changes
process, proposals are publicly announced in order to receive
community feedback. This proposal will only be implemented if approved
by the Fedora Engineering Steering Committee.
== Summary ==
Transition from Fedora's short name of licenses to standardized
[https://spdx.org/licenses/ SPDX license]
[https://spdx.dev/specifications/ formula].
== Owner ==
* Name: [[User:msuchy| Miroslav Suchý]]
* Name: [[User:jlovejoy| Jilayne Lovejoy]]
* Name: [[User:ngompa| Neal Gompa]]
* Name: [[User:dcantrell| David Cantrell]]
* Name: [[User:rfontanaref| Richard Fontana]]
* Name: [[User:mattdm| Matthew Miller]]
<!-- Include you email address that you can be reached should people
want to contact you about helping with your change, status is
requested, or technical issues need to be resolved. If the change
proposal is owned by a SIG, please also add a primary contact person.
-->
* Email: msuchy(a)redhat.com, dcantrell(a)redhat.com, jlovejoy(a)redhat.com,
ngompa13(a)gmail.com, rfontana(a)redhat.com
== Detailed Description ==
In the past, Fedora decided to use short names for licenses. Although
we documented the short names very well. The identifiers were never
standard. In the meantime, SPDX identifiers become standard, and
[https://wiki.spdx.org/view/Business_Team/Adoption other SW vendors
start using it].
In this phase, we want to provide documentation and tooling to allow
maintainers to begin using SPDX license ids instead of the old Fedora
short names. This move is opt-in. There will be
[[Changes/SPDX_Licenses_Phase_2|Phase 2]], where we identify the
remaining packages and help them to migrate to the SPDX formula.
== Feedback ==
Ancient [https://lists.fedoraproject.org/archives/list/legal@lists.fedoraproject.o...
feedback from SPDX organization].
Summary from [https://lists.fedoraproject.org/archives/search?q=spdx&page=1&mlist=legal...
fedora-legal mailing list]: we want this to happen, but this is big
scope and likely will happen over more than one release.
Summary from packaging-committee:
* [https://pagure.io/packaging-committee/pull-request/971#]: older PR
to change packaging guidelines
* [https://pagure.io/packaging-committee/pull-request/1142]: present
PR that needs more updating
Summary from devel-list: TBD
== Benefit to Fedora ==
The use of a standardized identifier for license will align Fedora
with other distributions. And allows efficient and reliable
identification of licenses.
== Scope ==
* Proposal owners (things sorted by done/todo and by priorities):
** Miroslav Suchý: license-fedora2spdx - done
** Jilayne Lovejoy: map rest of Fedora licenses to SPDX ids - done
** David Cantrell: create machine-readable format and new repo - done
** David Cantrell: merge mapping of Fedora licenses to SPDX ids to new
data format/repo - done
** Richard Fontana & Jilayne Lovejoy: review update all licensing info
and legal pages in wiki - in process
** Jilayne Lovejoy & Richard Fontana: create and populate new Docs
pages for legal and licensing info - in process
** Miroslav Suchy - create
[https://gitlab.com/fedora/legal/fedora-license-data
fedora-license-data package] (with data from rpminspect-data-fedora) -
TODO
** David Cantrell: separate licenses from rpminspect-data-fedora
[https://bugzilla.redhat.com/show_bug.cgi?id=2077914 BZ 2077914] -
TODO
** Miroslav Suchý: allow `license-validate` to use spdx - TODO
** David Cantrell: generate from license data to new Docs page similar
to [https://fedoraproject.org/wiki/Licensing:Main#Software_License_List
Licensing:Main]
** SOMEBODY: create a webhook that updates Docs page after the merge
to fedora-license-data - TODO
** Jilayne Lovejoy: prepare PR for updates to packaging guidelines -
in the process [https://pagure.io/packaging-committee/pull-request/1142]
** SOMEBODY: help maintainers who want to change license string to
SPDX identifiers proactively.
* Out of Scope: In this phase, we do not target to move **all**
packages to SPDX identifiers. That will be done in
[[Changes/SPDX_Licenses_Phase_2|Phase 2]]. In
[[Changes/SPDX_Licenses_Phase_2|Phase 2]] we will identify the
remaining packages and open BZ or PR.
* Other developers:
Early adopters can migrate their License tag to the SPDX identifiers.
Proposal owners will gather feedback and will work on potential
problems.
We want to have all bits ready so that maintainers can start changing
the spec files just after Fedora 37 branching (summer 2022).
* Release engineering:
* Policies and guidelines: Licensing page, packaging guidelines has to
be altered.
* Trademark approval: N/A (not needed for this Change)
* Alignment with Objectives:
== Upgrade/compatibility impact ==
License strings are not used anything in run time. This change will
not affect the upgrade or runtime of Fedora.
During the transition period, developer tools like rpminspect,
licensecheck, etc. may produce false negatives. And we have to define
a date where we flip these tools from old Fedora's short names to the
SPDX formula.
== How To Test ==
Users should not need any testing. These steps are for package maintainers:
* Fetch your license string from `License` tag in SPEC file.
* Test that your current Fedora's short name is correct. E.g.
$ license-validate -v 'MIT or GPLv1'
Approved license
* Convert license string to SPDX formula:
$ license-fedora2spdx 'MIT or GPLv1'
Warning: more options how to interpret MIT. Possible options:
['Adobe-Glyph', 'MIT-CMU', 'MIT-CMU', 'HPND', 'HPND', 'no-spdx-yet
(MIT license (also X11))', 'SGI-B-2.0', 'SGI-B-2.0', 'SMLNJ',
'MIT-enna', 'MIT-feh', 'mpich2']
mpich2 or GPL-1.0-only
In this example, the short name `GPLv1` can be converted straight to
`GPL-1.0-only`. But short name `MIT` stands for several licenses with
different [https://spdx.org/licenses/ SPDX identifiers]. You have to
examine what license is package actually using. `license-fedora2spdx`
will try to convert the formula and use one of the options but without
any heuristics. You need to manually review the license.
You can check if SPDX formula is correct using:
$ license-validate -v --file FIXME "MIT-CMU or GPL-1.0-only"
== User Experience ==
Users should be able to use standard software tools that audit
licenses. E.g. for Software Bills of Materials.
== Dependencies ==
No other dependencies.
== Contingency Plan ==
* Contingency mechanism: In this first phase, if something goes wrong,
we can 'git revert' each change in dist-git. It is expected that in
the first phase, there will be only a few packages altered. It may be
a few hundred, but it is still doable to revert.
* Contingency deadline: Beta freeze. But it is expected that not all
packages will be converted by that time and the change will continue
in the next release.
* Blocks release? No. This change has no impact on runtime of any package.
== Documentation ==
N/A (not a System Wide Change)
== Release Notes ==
--
Ben Cotton
He / Him / His
Fedora Program Manager
Red Hat
TZ=America/Indiana/Indianapolis
1 year, 6 months
tinyxml2 soname bump
by Rich Mattes
Hi,
I'm planning on upgrading tinyxml2 to version 9 in rawhide in the coming
week, which will include a soname bump. Affected packages are:
Macaulay2-0:1.19.1-2.fc37.src
bullet-0:3.08-3.fc36.src
cppcheck-0:2.7.4-1.fc37.src
dvblinkremote-0:0.2.0-0.25.beta.fc36.src
gazebo-0:10.1.0-27.fc36.src
linbox-0:1.6.3-11.fc36.src
vdr-epg2vdr-0:1.2.7-1.fc37.src
vdr-osd2web-0:0.2.54-12.fc37.src
I'll take care of the rebuilds in a side tag.
Rich
1 year, 7 months
F37 proposal: Supplement of Server distributables by a KVM VM disk
image (Self-Contained Change proposal)
by Ben Cotton
https://fedoraproject.org/wiki/Changes/Supplement-server-by-kvm-vm-image
This document represents a proposed Change. As part of the Changes
process, proposals are publicly announced in order to receive
community feedback. This proposal will only be implemented if approved
by the Fedora Engineering Steering Committee.
== Summary ==
Virtualization has long been a steadily growing use case of Fedora
Server Edition, but it is still time consuming and tedious for the
system administrator to create a Fedora Server VM. Supplementing the
downloads by a KVM VM image remedies the deficiency.
== Owner ==
* Name: [[User:pboy| Peter Boy on behalf of Server WG]]
* Email: pboy(a)uni-bremen.de
== Detailed Description ==
The creation of the VM disk image, uses the same kickstart files and
installation groups as the standard full installation, except of
course for the hardware-specific items. The image is optimized for
KVM.
That way, the VM resembles a default server installation as closely as possible.
All administrative tools are available reliably from the beginning,
all administrative routines and helps (scripts) can be used in the
same way. All application services work in the identical way.
A default VM installation takes 1-2 minutes instead of about 30 mins.
== Feedback ==
The change proposal is based on an extensive discussion of the server
WG and has become part of its work program. For example, the server
documentation on virtualization has already been significantly
expanded in parallel and will continue to be supplemented and updated
on an ongoing basis. This is a response to the importance of the
topic.
== Benefit to Fedora ==
Significantly improves usability for Fedora Server Edition
administrators when deploying a Fedora Server Edtion VM. It thus makes
Fedora Server Edition more attractive, especially for new users
without extensive experience with Fedora. It thus helps to expand our
user base.
Fedora finally provides an installation path for a Fedora Server VM
that is built entirely on Fedora Resources, subject to Fedora quality
control, and compliant with Fedora principles. Until now, if a system
administrator has to install a VM preferable without the hassles of a
full default installation, we could only recommend third party binary
blob (e.g. virt-builder), which is a violation of our own core
principles. In addition, the third party products do not provide a
'Fedora Server Edition', but their own different concept and vision of
a server, under the name Fedora Server.
== Scope ==
* Proposal owners:
A kickstart file for ImageFactory is completed and locally tested.
* Other developers:
n/a
* Release engineering: [https://pagure.io/releng/issues #10768]
* Policies and guidelines: N/A (not needed for this Change)
* Trademark approval: N/A (not needed for this Change)
* Alignment with Objectives:
n/a
== Upgrade/compatibility impact ==
none
== How To Test ==
Basically, the same test procedures as for the full install apply.
1. Install a Fedora Server Edition including virtualization, follow
the Server documentation
2. Import the Fedora Server disk image following the Server
documentation (staging), either using Cockpit or cli virt-install
3. Use the VM with your intendend server applications.
== User Experience ==
Users (system administrators) will have a VM install method available,
which saves them a lot of time and efforts.
== Dependencies ==
none
== Contingency Plan ==
* Contingency mechanism: (What to do? Who will do it?) N/A (not a
System Wide Change)
* Contingency deadline: N/A (not a System Wide Change)
* Blocks release? N/A (not a System Wide Change), No
== Documentation ==
N/A (not a System Wide Change)
Fedora Server documentation is available.
== Release Notes ==
TBD
--
Ben Cotton
He / Him / His
Fedora Program Manager
Red Hat
TZ=America/Indiana/Indianapolis
1 year, 8 months
Suggestion: Use a unified kernel image by default in the future.
by Sharpened Blade
Use unified kernel images by default for new releases. This can allow for the local installation to sign the kernel and the initrd, so the boot chain can be verified until after the uefi. Currently, the initrd can be modified by attackers, so even if the / partition is encrypted, the systems data can be read on the next boot. If the kernel image, which includes the command line, and the initrd, is signed then it is harder to comprimise the system. The system can still be comprimised if the uefi is modified.
If this was used, then the kernel could no longer be signed in the package by the fedora infrastructure. To still support secure boot, the kernel image would have to be signed be key stored on disk on every update. If the disk is encrypted, the private key can still be protected from attackers. On installation, or update for existing installs, a public/private keypair would be generated, and trusted by the shim.
This has a few problems, if the root user is hacked, then the kernel can be tampered with. But this is not a very big problem because if the root user is hacked, then for example systemd can be changed, secure boot will not protect you. It will also mean that if the user want to modify the kernel command line or initrd, they have to regenerate the entire kernel image. This can also break some users install, if they use a non-default boot process, or have a buggy uefi implementation. For non-uefi architectures, this change could be ignored.
1 year, 9 months
F37 proposal: Python: Add -P to default shebangs (System-Wide Change proposal)
by Ben Cotton
https://fedoraproject.org/wiki/Changes/PythonSafePath
== Summary ==
The [https://docs.python.org/3.11/using/cmdline.html#cmdoption-P `-P`
flag] will be added to the Python shebang macros
(`%{py3_shbang_opts}`, `%{py3_shebang_flags}`, ...). Packages that
adhere to those macros will change their Python shebangs from `#!
/usr/bin/python3 -s` to `#! /usr/bin/python3 -sP` and as a result,
will no longer have the directory of the script (such as `/usr/bin`)
in `sys.path`. An opt-out mechanism exists.
== Owner ==
* Name: [[User:Churchyard|Miro Hrončok]], [[User:Vstinner|Victor Stinner]]
* Email: python-maint(a)redhat.com
== Detailed Description ==
All Python 3 shebang RPM macros will be changed to contain one more
flag: `-P`. Previously, they contained `-s`, now they will contain
`-sP`.
From the [https://docs.python.org/3.11/using/cmdline.html#cmdoption-P
documentation for the `-P` option]:
:Don’t prepend a potentially unsafe path to `sys.path`:
:
:* `python -m module` command line: Don’t prepend the current working directory.
:* `python script.py` command line: Don’t prepend the script’s
directory. If it’s a symbolic link, resolve symbolic links.
:* `python -c code` and `python` (REPL) command lines: Don’t prepend
an empty string, which means the current working directory.
In shebangs, only the middle option (''don’t prepend the script’s
directory'') is relevant.
Consider the following executbale script installed as
`/usr/bin/let-there-be-fun`:
#! /usr/bin/python3 -s
import abc
...
When the script is directly executed (e.g. by running
`let-there-be-fun` from the console), the script's directory
(`/usr/bin`) is prepended to `sys.path`. Python tries to locate an
importable `abc` module in `/usr/bin` first. This can cause real
issues: [https://bugzilla.redhat.com/2057340 python3-notebook:
ImportError: bad magic number in six] and
[https://github.com/benjaminp/six/issues/359 bad magic number in six].
When the shebang includes `-P`:
#! /usr/bin/python3 -sP
import abc
...
...the script's directory (`/usr/bin`) is '''not''' prepended to
`sys.path`. The change owners consider this approach safer for the
majority of Fedora's RPM packages.
By default, '''all standardly RPM-packaged Python packages with
scripts in `/usr/bin` will gain the `-P` flag in their shebang''',
assuming the software is packaged in a way that respects the Python
shebang RPM macros (see below for opt-out and explicit opt-in
mechanisms). Due to the variety of ways such scripts can be
created/packaged, there will likely be packages that will not be
affected by the change automatically. (In other words, the change is
applied on RPM macro level, no added mechanics to force the flag, such
as BRP scripts, are planned as part of this change.)
=== List of RPM macros that will gain `-P` ===
* `%{py3_shbang_opts}`
* `%{py3_shbang_opts_nodash}`
* `%{py3_shebang_flags}`
* `%{py_shbang_opts}`
* `%{py_shbang_opts_nodash}`
* `%{py_shebang_flags}`
=== Opting out ===
If the new behavior is not desirable to your package amend the macros
(e.g. with `sed`) to remove the `P` flag.
If you use the [https://docs.fedoraproject.org/en-US/packaging-guidelines/Python/
current Python packaging guidelines], e.g. `%pyproject_wheel` and
`%pyproject_install`, use:
# Don't add -P to Python shebang
# This package only works when /usr/bin is in sys.path (use your own
rationale here)
%global py3_shebang_flags %(echo %py3_shebang_flags | sed s/P//)
If you use the [https://docs.fedoraproject.org/en-US/packaging-guidelines/Python_201x/
201x-era Python packaging guidelines], e.g. `%py3_build` and
`%py3_install`, use:
# Don't add -P to Python shebang
# This package only works when /usr/bin is in sys.path (use your own
rationale here)
%global py3_shbang_opts %(echo %py3_shbang_opts | sed s/P//)
(The only difference is the name of the macro.)
=== Opting in ===
If you use the [https://docs.fedoraproject.org/en-US/packaging-guidelines/Python/
current Python packaging guidelines], e.g. `%pyproject_wheel` and
`%pyproject_install`, the standard set of Python shebang flags is
applied to all files with Python shebangs installed in `/usr/bin/`.
If you use the [https://docs.fedoraproject.org/en-US/packaging-guidelines/Python_201x/
201x-era Python packaging guidelines], e.g. `%py3_build` and
`%py3_install`, the standard set of Python shebang flags might be
applied to some files and not applied to others depending on the exact
structure of the packaged software.
If you wish to explicitly apply the standard set of Python shebang
flags on a certain file that is not handled automatically, use
[https://docs.fedoraproject.org/en-US/packaging-guidelines/Python/#py3_she...
the `%py3_shebang_fix` macro].
=== What if the packager changes `%__python3` to an older version of Python ===
The `-P` flag was introduced in Python 3.11. When `%__python3` is
redefined to an older Python version, e.g. `/usr/bin/python3.10`,
including the `-P` flag in shebangs would break the scripts. Hence,
the flag will be included conditionally, presumably somehow like this:
<nowiki>%py3_shbang_opts -s%(%{__python3} -Ic "import sys; print('P'
if hasattr(sys.flags, 'safe_path') else '')")</nowiki>
=== What if the admin/user changes `/usr/bin/python3` to an older
version of Python ===
The `-P` flag was introduced in Python 3.11. When an admin/user
changes `/usr/bin/python3` to point to an older version of Python,
e.g. `/usr/bin/python3.10`, including the `-P` flag in shebangs would
break the scripts.
However, changing `/usr/bin/python3` to a different Python would
''brick'' a Fedora system even now. So we don't consider that an
issue. See an example that changes `/usr/bin/python3` to Python 3.9
(don't try this at home):
[root@086a2804411a /]# head -n1 /usr/bin/dnf
#!/usr/bin/python3
[root@086a2804411a /]# dnf --version
4.12.0
...
[root@086a2804411a /]# sudo ln -sf /usr/bin/python3.9 /usr/bin/python3
[root@086a2804411a /]# dnf --version
Traceback (most recent call last):
File "/usr/bin/dnf", line 61, in <module>
from dnf.cli import main
ModuleNotFoundError: No module named 'dnf'
=== Risks ===
As with any other change, there is a risk that this will break things.
The change owners plan to test the change extensively via Copr before
they deploy the change in Rawhide. If things go badly, they are
prepared to delay or cancel the change.
The [https://docs.fedoraproject.org/en-US/packaging-guidelines/Python_201x/
201x-era Python RPM macros] set the shebang flags by a weird hack. As
a result, it is not well-defined what scripts will be affected by this
change. The change owners are aware that:
# not all Python scripts in `/usr/bin` will have the `-P` flag automatically
# some scripts '''not''' in `/usr/bin` might gain the `-P` flag as well
The first point is an acceptable gradual deployment of the default
flag. The second point is not very dangerous because we don't except
users to directly execute Python scripts via shebangs when such
scripts are not in `$PATH`. If a problematic package is found, it can
opt-out easily. If this causes too much friction, we will only change
the flag used in the `%pyproject_install` macro, leaving packages with
the "legacy" macros intact.
== Feedback ==
* Relevant upstream discussion: [https://discuss.python.org/t/13896
Should console_scripts entry points exclude the scripts directory from
sys.path?]
* Adding `-P` to Python:
[https://github.com/python/cpython/issues/57684 GitHub issue],
[https://mail.python.org/archives/list/python-dev@python.org/thread/IU5Q2A...
email thread]
* [https://github.com/rpm-software-management/dnf/pull/1815#pullrequestrevie...
dnf maintainers response for doing this explicitly in dnf first]
Generally, people seem to think that this is a good thing. They are
afraid to change the default behavior of Python but welcome using this
flag for system-installed scripts.
== Benefit to Fedora ==
Python programs in `/usr/bin` will be less fragile to other random
files being present in `/usr/bin`.
Real hard-to-debug issues like [https://bugzilla.redhat.com/2057340
python3-notebook: ImportError: bad magic number in six] and
[https://github.com/benjaminp/six/issues/359 bad magic number in six]
will not happen.
== Scope ==
* Proposal owners:
** Test everything in Copr
** If everything works, change the flags either before the Python 3.11
rebuild or before the Fedora 37 Mass Rebuild
** Provide guidance to packagers, fix bugs if needed
* Other developers:
** Observe their packages, find and report bugs, opt-out if needed
** Volunteerily opt-in by calling the `%py3_shebang_fix` macro and/or
by converting their packages to `%pyproject_install`
* Release engineering: [https://pagure.io/releng/issue/10784 #10784]
* Policies and guidelines: The new flag needs to be documented in the
Python packaging guidelines (old and new)
* Trademark approval: not needed for this Change
* Alignment with Objectives: no
== Upgrade/compatibility impact ==
No impact is anticipated.
== How To Test ==
* Low level: Examine the value of the changed RPM macros, it should
contain the `-P` flag
* Middle level: Examine the shebang lines of RPM-installed Python
scripts in `/usr/bin`, it should contain the `-P` flag
* High level: Tests that RPM-installed Python scripts still behave as
expected but don't try to import stuff from `/usr/bin`
== User Experience ==
Users of RPM-installed scripts should get a safer experience by default.
Users of Python should not observe a difference, the behavior is not a
new default: the `-P` flag needs to be explicitly used.
== Dependencies ==
We need [[Changes/Python3.11]] first.
== Contingency Plan ==
* Contingency mechanism: defer to F38; only add `-P` to shebangs in
`%pyproject_install` to keep backward compatibility of the old macros;
revert and rebuild
* Contingency deadline: 1 week before the beta freeze
* Blocks release? No
== Documentation ==
* This page
* TBD Updated Python guidelines
* https://docs.python.org/3.11/using/cmdline.html#cmdoption-P
* https://docs.python.org/3.11/whatsnew/3.11.html#summary-release-highlights
(Security improvements)
== Release Notes ==
TBD
--
Ben Cotton
He / Him / His
Fedora Program Manager
Red Hat
TZ=America/Indiana/Indianapolis
1 year, 9 months
Spring Cleaning go-sig/June 2022
by Fabio Alessandro Locati
Hello Golang packagers,
As it happened last month, we are trying to clean the situation with golang packages.
In the last month we have seen big improvements (265 ->131 packages in the list), and I really hope next month we will be able to claim such a great result again!
At the moment I've gathered all packages that depend on golang and the privilege that @go-sig has on them.
I would ask of all of you to make sure all your packages have been added to the @go-sig group on src.fedoraproject.org (at least with "commit" access), unless there is a very good reason not to do so (and if that is the case for a particular package on this list, I'd be interested in knowing the reason).
Without that, it makes it very hard for us to keep the Go stack up-to-date and in working order, because the "go-sig" list / bugzilla account does not get CC'd on new bugs that way, and your bugs do not show up in our BugZilla queries.
If you want a scripted way of adding "@go-sig" group to many packages, you can generate an API token on src.fedoraproject.org (with "Modify an existing project") access level, and use the simple Python script from this link:
https://fale.fedorapeople.org/add-go-sig.py
Below is the list of "incompletely set-up" packages, in alphabetic order, and at the bottom, is a list per package maintainer.
Thanks,
Fale
================================================================================
Maintainers per package:
- android-tools: van
- apptainer: dwd
- asciigraph: mhayden
- buildah: lsm5
- butane: bgilbert
- ceph: kkeithle
- cheat: tkorbar
- clipman: wef
- containernetworking-plugins: mheon
- cri-o: haircommander
- deepin-api: mosquito
- deepin-daemon: mosquito
- deepin-desktop-schemas: mosquito
- deepin-gir-generator: mosquito
- deepin-pw-check: cheeselee
- docker-distribution: cverna
- git-octopus: baitaand
- go-bindata: lsm5
- godep: jchaloup
- golang: jcajka
- golang-deepin-go-lib: mosquito
- golang-entgo-ent: salimma
- golang-github-acobaugh-osrelease: harrymichal
- golang-github-anaskhan96-soup: atim
- golang-github-apparentlymart-cidr: pwouters
- golang-github-apparentlymart-textseg-13: limb
- golang-github-bits-and-blooms-bitset: dcavalca
- golang-github-client9-gospell: mayorga
- golang-github-cli-gh: mikelo2
- golang-github-cli-shurcool-graphql: jdoss
- golang-github-coreos-semver: limb
- golang-github-elves-elvish: appadeia
- golang-github-enescakir-emoji: jdoss
- golang-github-facebook-time: abulimov
- golang-github-gizak-termui: atim
- golang-github-gocomply-scap: isimluk
- golang-github-googlecloudplatform-guest-logging: ericedens
- golang-github-google-dap: alexsaezm
- golang-github-google-gousb: jjelen
- golang-github-gosexy-gettext: zyga
- golang-github-hajimehoshi-mp3: atim
- golang-github-hajimehoshi-oto: atim
- golang-github-harrymichal-version: harrymichal
- golang-github-heistp-irtt: tohojo
- golang-github-hinshun-vt10x: jdoss
- golang-github-hub: jaymzh
- golang-github-hugelgupf-socketpair: dcavalca
- golang-github-ianbruene-difflib: dfateyev
- golang-github-jroimartin-gocui: atim
- golang-github-kalafut-imohash: dcavalca
- golang-github-kelvins-sunrisesunset: cheeselee
- golang-github-letsencrypt-challtestsrv: pwouters
- golang-github-letsencrypt-pebble: pwouters
- golang-github-linuxdeepin-dbus-factory: mosquito
- golang-github-linuxdeepin-go-x11-client: mosquito
- golang-github-lithammer-fuzzysearch: atim
- golang-github-lofanmi-pinyin: cheeselee
- golang-github-lpabon-godbc: lpabon
- golang-github-lunixbochs-vtclean: atim
- golang-github-mbland-hmacauth: pwouters
- golang-github-mbndr-figlet4go: atim
- golang-github-mingrammer-commonregex: athoscr
- golang-github-mitchellh-ps: szeth
- golang-github-mozillazg-pinyin: cheeselee
- golang-github-msprev-fzf-bibtex: dmoerner
- golang-github-muesli-reflow: jdoss
- golang-github-muesli-termenv: jdoss
- golang-github-mvo5-goconfigparser: zyga
- golang-github-netflix-expect: jdoss
- golang-github-openprinting-goipp: zdohnal
- golang-github-openprinting-ipp-usb: zdohnal
- golang-github-pkg-browser: thomasfedb
- golang-github-projectdiscovery-cdncheck: fab
- golang-github-projectdiscovery-sliceutil: mikelo2
- golang-github-remeh-sizedwaitgroup: mayorga
- golang-github-rickb777-date: cheeselee
- golang-github-rickb777-plural: cheeselee
- golang-github-rivo-uniseg: jdoss
- golang-github-rubyist-tracerx: clime
- golang-github-schollz-cli-2: dcavalca
- golang-github-schollz-croc: dcavalca
- golang-github-schollz-logger: dcavalca
- golang-github-schollz-mnemonicode: dcavalca
- golang-github-schollz-pake-3: dcavalca
- golang-github-schollz-peerdiscovery: dcavalca
- golang-github-segmentio-ksuid: gundersanne
- golang-github-sqshq-sampler: atim
- golang-github-teambition-rrule: cheeselee
- golang-github-termie-shutil: dfateyev
- golang-github-tomnomnom-xtermcolor: atim
- golang-github-tscholl2-siec: dcavalca
- golang-github-xiaq-persistent: appadeia
- golang-github-xrash-smetrics: mayorga
- golang-github-zyedidia-highlight: atim
- golang-gitlab-esr-fqme: dfateyev
- golang-gitlab-ianbruene-kommandant: dfateyev
- golang-nanomsg-mangos-3: fab
- golang-rsc-pdf: deparker
- golang-starlark: alexsaezm
- golie: isimluk
- gomtree: vbatts
- google-guest-agent: ericedens
- gotun: kushal
- grafana: agerstmayr
- grafana-pcp: agerstmayr
- graphviz: jskarvad
- gron: lkiesow
- ignition: dustymabe
- kata-containers: etrunko
- kompose: dustymabe
- manifest-tool: jwboyer
- oci-seccomp-bpf-hook: jnovy
- origin: jcajka
- osbuild-composer: obudai
- pack: lsm5
- podman: mheon
- podman-tui: navidys
- popub: zsun
- reg: mattia
- reposurgeon: dfateyev
- restic: copart
- runc: kir
- singularity: dwd
- skopeo: lsm5
- stargz-snapshotter: ktock
- startdde: mosquito
- swig: jplesnik
- tmux-top: ttomecek
- toolbox: rishi
- vultr-cli: mhayden
- xe-guest-utilities-latest: cheeselee
Packages per maintainer:
abulimov (1): golang-github-facebook-time
agerstmayr (2): grafana, grafana-pcp
alexsaezm (2): golang-github-google-dap, golang-starlark
appadeia (2): golang-github-elves-elvish, golang-github-xiaq-persistent
athoscr (1): golang-github-mingrammer-commonregex
atim (11): golang-github-anaskhan96-soup, golang-github-gizak-termui, golang-github-hajimehoshi-mp3, golang-github-hajimehoshi-oto, golang-github-jroimartin-gocui, golang-github-lithammer-fuzzysearch, golang-github-lunixbochs-vtclean, golang-github-mbndr-figlet4go, golang-github-sqshq-sampler, golang-github-tomnomnom-xtermcolor, golang-github-zyedidia-highlight
baitaand (1): git-octopus
bgilbert (1): butane
cheeselee (8): deepin-pw-check, golang-github-kelvins-sunrisesunset, golang-github-lofanmi-pinyin, golang-github-mozillazg-pinyin, golang-github-rickb777-date, golang-github-rickb777-plural, golang-github-teambition-rrule, xe-guest-utilities-latest
clime (1): golang-github-rubyist-tracerx
copart (1): restic
cverna (1): docker-distribution
dcavalca (10): golang-github-bits-and-blooms-bitset, golang-github-hugelgupf-socketpair, golang-github-kalafut-imohash, golang-github-schollz-cli-2, golang-github-schollz-croc, golang-github-schollz-logger, golang-github-schollz-mnemonicode, golang-github-schollz-pake-3, golang-github-schollz-peerdiscovery, golang-github-tscholl2-siec
deparker (1): golang-rsc-pdf
dfateyev (5): golang-github-ianbruene-difflib, golang-github-termie-shutil, golang-gitlab-esr-fqme, golang-gitlab-ianbruene-kommandant, reposurgeon
dmoerner (1): golang-github-msprev-fzf-bibtex
dustymabe (2): ignition, kompose
dwd (2): apptainer, singularity
ericedens (2): golang-github-googlecloudplatform-guest-logging, google-guest-agent
etrunko (1): kata-containers
fab (2): golang-github-projectdiscovery-cdncheck, golang-nanomsg-mangos-3
gundersanne (1): golang-github-segmentio-ksuid
haircommander (1): cri-o
harrymichal (2): golang-github-acobaugh-osrelease, golang-github-harrymichal-version
isimluk (2): golang-github-gocomply-scap, golie
jaymzh (1): golang-github-hub
jcajka (2): golang, origin
jchaloup (1): godep
jdoss (7): golang-github-cli-shurcool-graphql, golang-github-enescakir-emoji, golang-github-hinshun-vt10x, golang-github-muesli-reflow, golang-github-muesli-termenv, golang-github-netflix-expect, golang-github-rivo-uniseg
jjelen (1): golang-github-google-gousb
jnovy (1): oci-seccomp-bpf-hook
jplesnik (1): swig
jskarvad (1): graphviz
jwboyer (1): manifest-tool
kir (1): runc
kkeithle (1): ceph
ktock (1): stargz-snapshotter
kushal (1): gotun
limb (2): golang-github-apparentlymart-textseg-13, golang-github-coreos-semver
lkiesow (1): gron
lpabon (1): golang-github-lpabon-godbc
lsm5 (4): buildah, go-bindata, pack, skopeo
mattia (1): reg
mayorga (3): golang-github-client9-gospell, golang-github-remeh-sizedwaitgroup, golang-github-xrash-smetrics
mhayden (2): asciigraph, vultr-cli
mheon (2): containernetworking-plugins, podman
mikelo2 (2): golang-github-cli-gh, golang-github-projectdiscovery-sliceutil
mosquito (8): deepin-api, deepin-daemon, deepin-desktop-schemas, deepin-gir-generator, golang-deepin-go-lib, golang-github-linuxdeepin-dbus-factory, golang-github-linuxdeepin-go-x11-client, startdde
navidys (1): podman-tui
obudai (1): osbuild-composer
pwouters (4): golang-github-apparentlymart-cidr, golang-github-letsencrypt-challtestsrv, golang-github-letsencrypt-pebble, golang-github-mbland-hmacauth
rishi (1): toolbox
salimma (1): golang-entgo-ent
szeth (1): golang-github-mitchellh-ps
thomasfedb (1): golang-github-pkg-browser
tkorbar (1): cheat
tohojo (1): golang-github-heistp-irtt
ttomecek (1): tmux-top
van (1): android-tools
vbatts (1): gomtree
wef (1): clipman
zdohnal (2): golang-github-openprinting-goipp, golang-github-openprinting-ipp-usb
zsun (1): popub
zyga (2): golang-github-gosexy-gettext, golang-github-mvo5-goconfigparser
--
Fabio Alessandro Locati
fale.io
1 year, 9 months
Updated criteria proposal: networking requirements
by Adam Williamson
Hi folks!
Some time ago I proposed some specific networking release criteria. I
revived the thread back in February, and meeting discussion suggested
we should be more intentional and specific about wifi requirements. So,
looking at it again, I suggest adding an additional footnote:
Footnote titled "Wireless networks": Common wireless network
configurations using supported hardware as defined above are covered by
this criterion. This includes access to home and enterprise wireless
networks using 802.11 series connection protocols and WPA2 and WPA3
personal and enterprise security protocols. Bugs that are specific to
particular hardware or configurations will be assessed according to
[[Blocker_Bug_FAQ|hardware-dependent-issues|the normal considerations
for such issues]].
Here is the full proposal again, with the new footnote included:
#####
=== Network requirements ===
Each of these requirements apply to both installer and installed system
environments. For any given installer environment, the 'default network
configuration tools' are considered to be those the installer documents
as supported ways to configure networking (e.g. for anaconda-based
environments, configuration via kernel command line options, a
kickstart, or interactively in anaconda itself are included).
==== Basic networking ====
It must be possible to establish both IPv4 and IPv6 network connections
using both typical router-provided addressing systems (e.g. DHCP on
IPv4 or SLAAC or IPv6) and static addressing. The default network
configuration tools for the console, for release-blocking desktops and
for installer environments must work well enough to allow typical
network connection configuration operations without major workarounds.
Standard network functions such as address resolution and connections
with common protocols such as ping, HTTP and ssh must work as expected.
Footnote titled "Supported hardware": Supported network hardware is
hardware for which the Fedora kernel includes drivers and, where
necessary, for which a firmware package is available. If support for a
commonly-used piece or type of network hardware that would usually be
present is omitted, that may constitute a violation of this criterion,
after consideration of the [[Blocker_Bug_FAQ|hardware-dependent-
issues|normal factors for hardware-dependent issues]]. Similarly,
violations of this criteria that are hardware or configuration
dependent are, as usual, subject to consideration of those factors when
determining whether they are release-blocking.
Footnote titled "Wireless networks": Common wireless network
configurations using supported hardware as defined above are covered by
this criterion. This includes access to home and enterprise wireless
networks using 802.11 series connection protocols and WPA2 and WPA3
personal and enterprise security protocols. Bugs that are specific to
particular hardware or configurations will be assessed according to
[[Blocker_Bug_FAQ|hardware-dependent-issues|the normal considerations
for such issues]].
==== VPN connections ====
Using the default network configuration tools for the console and for
release-blocking desktops, it must be possible to establish a working
connection to common OpenVPN, openconnect-supported and vpnc-supported
VPN servers with typical configurations.
Footnote titled "Supported servers and configurations": As there are
many different VPN server applications and configurations, blocker
reviewers must use their best judgment in determining whether
violations of this criterion are likely to be encountered commonly
enough to block a release, and if so, at which milestone. As a general
principle, the more people are likely to use affected servers and the
less complicated the configuration required to hit the bug, the more
likely it is to be a blocker.
#####
Any more thoughts, comments, adjustments etc? Thanks!
--
Adam Williamson
Fedora QA
IRC: adamw | Twitter: adamw_ha
https://www.happyassassin.net
1 year, 9 months