F39 Change Proposal: Further reduce Fedora-specific build flags in non-RPM Python extensions (Self Contained)

[Aoife Moloney]

https://fedoraproject.org/wiki/Changes/Python_Extension_Flags_Reduction

This document represents a proposed Change. As part of the Changes
process, proposals are publicly announced in order to receive
community feedback. This proposal will only be implemented if approved
by the Fedora Engineering Steering Committee.

== Summary ==
<!-- A sentence or two summarizing what this change is and what it
will do. This information is used for the overall changeset summary
page for each release. Note that motivation for the change should be
in the Benefit to Fedora section below, and this part should answer
the question "What?" rather than "Why?". -->
Continuing the work started with
https://fedoraproject.org/wiki/Changes/Python_Extension_Flags, this
change is about further reducing the build and linker flags (CFLAGS
and LDFLAGS) saved internally in the Python interpreter for use by
distutils and other build systems. Compiling non-RPM Python extension
modules will carry only the compiler flags required for binary
compatibility with the interpreter they were built against and not
Fedora specific ones.

Practically that means the only Fedora derived flag will be
<code>-fexceptions</code> and Python will apply its own upstream
hardcoded ones, making the final flag set for a non-RPM compiled
Python extension as follows:

* <code>-Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG
-fexceptions</code>

Python C extensions built as rpm's will '''not''' be
affected.

The current main Python interpreter on Fedora 39 will be modified
(Python 3.12) and Python 3.6-3.11 will follow.

This change will affect every package that provides support for
extension builders via utilizing the <code>%{extension...flags}</code>
macros which at the time being is only Python.

== Owner ==

* Name: [[User:cstratak| Charalampos Stratakis]]

* Email: python-maint AT redhat.com


== Detailed Description ==
After implementing
https://fedoraproject.org/wiki/Changes/Python_Extension_Flags we
uncoupled some distro specific compilation and linker flags propagated
to C extensions.

However with an ever increasing set of compiler flags being added and
applied distro-wide, as compilers and security standards evolve (e.g.
-D_FORTIFY_SOURCE=3) it becomes an increasingly complex job to vet
each flag that might leak into user-built Python C extensions through
the Python interpreter. Instead of removing only some flags and
letting the rest follow through, we will be taking a more proactive
approach by removing all the compiler and linker flags, except the
ones that are required to maintain the binary compatibility with the
Python interpreter the extensions were built against which is
<code>-fexceptions</code>. We will also preserve the ones that Python
hardcodes itself through the Makefile.

Similarly, when a user builds their own C programs, no compiler flags
are applied by default and the user is free to making their own
decision. Bringing the compilation of Python C extensions closer to
that experience is the next logical step.

Currently a user-built Python C extension will be built with:

CFLAGS:
<code>
-Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG  -O2
-fexceptions -g -grecord-gcc-switches -pipe -Wall
-Werror=format-security -Wp,-U_FORTIFY_SOURCE,-D_FORTIFY_SOURCE=3
-Wp,-D_GLIBCXX_ASSERTIONS  -fstack-protector-strong   -m64
-mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection
-fcf-protection    -D_GNU_SOURCE -fPIC -fwrapv -O2  -fexceptions -g
-grecord-gcc-switches -pipe -Wall -Werror=format-security
-Wp,-U_FORTIFY_SOURCE,-D_FORTIFY_SOURCE=3 -Wp,-D_GLIBCXX_ASSERTIONS
-fstack-protector-strong   -m64  -mtune=generic
-fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection
  -D_GNU_SOURCE -fPIC -fwrapv  -O2  -fexceptions -g
-grecord-gcc-switches -pipe -Wall -Werror=format-security
-Wp,-U_FORTIFY_SOURCE,-D_FORTIFY_SOURCE=3 -Wp,-D_GLIBCXX_ASSERTIONS
-fstack-protector-strong   -m64  -mtune=generic
-fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection
  -D_GNU_SOURCE -fPIC -fwrapv
</code>

LDFLAGS:
<code>
'-Wl,-z,relro -Wl,--as-needed  -Wl,-z,now    -Wl,--build-id=sha1   -g
-Wl,-z,relro -Wl,--as-needed  -Wl,-z,now    -Wl,--build-id=sha1   -g'
</code>

After this change:

CFLAGS:
<code>
-Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -fexceptions
-fexceptions -fexceptions
</code>

LDFLAGS:
None

== Feedback ==

The initial thread that inspired this change was
https://lists.fedoraproject.org/archives/list/[e-mail].o...

== Benefit to Fedora ==

Python developers will get more upstream-like experience when building
Python extension modules and also closer to building vanilla C
programs. Also new decisions made about the distro-wide compiler flags
won't necessarily affect Python developers building their extension
modules.

In addition any Python developer using Fedora will have the capability
to build the extension on Fedora, test it and later ship it and build
it on a CI or other systems that are not based on Fedora.

== Scope ==
* Proposal owners: Review, merge and build the
[https://src.fedoraproject.org/rpms/redhat-rpm-config/pull-request/252
redhat-rpm-config PR] and the apply the relevant changes in the Python
interpreters ([https://src.fedoraproject.org/rpms/python3.11/pull-request/111
example from python3.11])


* Other developers: No requirements apart from welcoming testing their
C extensions

* Release engineering: No mass rebuild required and no releng impact
anticipated.[https://pagure.io/releng/issues #Releng issue number]

* Policies and guidelines: N/A (not needed for this Change)

* Trademark approval: N/A (not needed for this Change)


== Upgrade/compatibility impact ==

Not anticipated. Extension modules (built for the same Python version)
are compatible with the interpreter with or without the removed flags
back and forth.


== How To Test ==


=== For users (Python developers) ===

# build your favorite Python extension module in venv or outside venv
with your favorite build system
# observe the used flags and check that the full set of flags are
'''are not there''' as mentioned in the detailed description,
report
bugs for {{package|python3.12}} otherwise (and block our tracking bug)
# check if the extension works as expected

=== For packagers (Fedora contributors)  ===

# build your favorite RPM package with Python extension module
# observe the used flags and check that the full set of flags '''are
there''' and not the reduced one, report bugs for that package
otherwise (and block our tracking bug)
# check if the package works as expected

== User Experience ==

See '''Benefit to Fedora''' above.

== Dependencies ==

Changes are required in {{package|redhat-rpm-config}} along with the
Python interpreters.

== Contingency Plan ==

* Contingency mechanism: Change owners can revert the change at any point.
* Contingency deadline: final freeze (not a System Wide Change)
* Blocks release? No

== Documentation ==

N/A (not a System Wide Change)

== Release Notes ==





-- 
Aoife Moloney

Product Owner

Community Platform Engineering Team

Red Hat EMEA

Communications House

Cork Road

Waterford