12:39 p.m.
Hello folks,
this recent Fedora change:
https://fedoraproject.org/wiki/Changes/Add_FORTIFY_SOURCE%3D3_to_distribu...
Made me think:
Which compiler flags we need to store in Python and which can we omit?
In order to make Python extension modules binary compatible with Python, Python
saves the compiler flags at compile-time and reuses them when building
extension modules.
Historically, we had troubles with this approach because some of the flags are
unusable without redhat-rpm-config, annobin etc.
As a result, there are now 2 compiler flags macros available for RPM:
%{build_cflags} and %{extension_cflags} (same for ldflags etc.).
While Python itself is built with %{build_cflags}, it saves %{extension_cflags}
in sysconfig.
The flags differ like this:
$ diff -u <(rpm --eval '%build_cflags' | tr ' ' '\n') <(rpm
--eval
'%extension_cflags' | tr ' ' '\n') | grep ^-
--flto=auto
--ffat-lto-objects
--specs=/usr/lib/rpm/redhat/redhat-hardened-cc1
--specs=/usr/lib/rpm/redhat/redhat-annobin-cc1
$ diff -u <(rpm --eval '%build_ldflags' | tr ' ' '\n') <(rpm
--eval
'%extension_ldflags' | tr ' ' '\n') | grep ^-
--specs=/usr/lib/rpm/redhat/redhat-hardened-ld
--specs=/usr/lib/rpm/redhat/redhat-annobin-cc1
(There are also some other differences wrt
https://fedoraproject.org/wiki/Changes/Package_information_on_ELF_objects but
those are apparently harder to get from outside of a real build.)
The current set of flags from Python can be obtained by:
>> sysconfig.get_config_var('CFLAGS')
'-Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -O2 -fexceptions -g
-grecord-gcc-switches -pipe -Wall -Werror=format-security -U_FORTIFY_SOURCE
-Wp,-U_FORTIFY_SOURCE -Wp,-D_FORTIFY_SOURCE=3 -Wp,-D_GLIBCXX_ASSERTIONS
-fstack-protector-strong -m64 -mtune=generic -fasynchronous-unwind-tables
-fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -O2
-fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security
-U_FORTIFY_SOURCE -Wp,-U_FORTIFY_SOURCE -Wp,-D_FORTIFY_SOURCE=3
-Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -m64 -mtune=generic
-fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection
-D_GNU_SOURCE -fPIC -fwrapv -O2 -fexceptions -g -grecord-gcc-switches -pipe
-Wall -Werror=format-security -U_FORTIFY_SOURCE -Wp,-U_FORTIFY_SOURCE
-Wp,-D_FORTIFY_SOURCE=3 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong
-m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection
-fcf-protection -D_GNU_SOURCE -fPIC -fwrapv'
>> sysconfig.get_config_var('LDFLAGS')
'-Wl,-z,relro -Wl,--as-needed -Wl,-z,now -Wl,--build-id=sha1 -g
-Wl,-z,relro -Wl,--as-needed -Wl,-z,now -Wl,--build-id=sha1 -g'
I wonder if other flags should be removed as well.
Isn't Python built e.g. with -Werror=format-security or -Wsign-compare binary
compatible with extension modules built without it?
What about FORTIFY_SOURCE and others?
Is there a compiler flags expert here who could help me determine what flags
could (or even should) be removed from %{extension_*flags}?
Thanks,
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
12:42 p.m.
On 1/16/23 13:39, Miro Hrončok wrote:
Isn't Python built e.g. with -Werror=format-security or
-Wsign-compare
binary compatible with extension modules built without it?
That's a good question, it's almost as if "extension_flags" should be
named "abi_compatibility_flags" or something similar.
--
Kevin P. Fleming
He/Him/His
Principal Program Manager, RHEL
Red Hat US/Eastern Time Zone
1:30 p.m.
On Mon, Jan 16, 2023 at 1:40 PM Miro Hrončok <mhroncok(a)redhat.com> wrote:
Hello folks,
this recent Fedora change:
https://fedoraproject.org/wiki/Changes/Add_FORTIFY_SOURCE%3D3_to_distribu...
Made me think:
Which compiler flags we need to store in Python and which can we omit?
In order to make Python extension modules binary compatible with Python, Python
saves the compiler flags at compile-time and reuses them when building
extension modules.
Historically, we had troubles with this approach because some of the flags are
unusable without redhat-rpm-config, annobin etc.
As a result, there are now 2 compiler flags macros available for RPM:
%{build_cflags} and %{extension_cflags} (same for ldflags etc.).
While Python itself is built with %{build_cflags}, it saves %{extension_cflags}
in sysconfig.
The flags differ like this:
$ diff -u <(rpm --eval '%build_cflags' | tr ' ' '\n') <(rpm
--eval
'%extension_cflags' | tr ' ' '\n') | grep ^-
--flto=auto
--ffat-lto-objects
--specs=/usr/lib/rpm/redhat/redhat-hardened-cc1
--specs=/usr/lib/rpm/redhat/redhat-annobin-cc1
$ diff -u <(rpm --eval '%build_ldflags' | tr ' ' '\n')
<(rpm --eval
'%extension_ldflags' | tr ' ' '\n') | grep ^-
--specs=/usr/lib/rpm/redhat/redhat-hardened-ld
--specs=/usr/lib/rpm/redhat/redhat-annobin-cc1
(There are also some other differences wrt
https://fedoraproject.org/wiki/Changes/Package_information_on_ELF_objects but
those are apparently harder to get from outside of a real build.)
The current set of flags from Python can be obtained by:
>>> sysconfig.get_config_var('CFLAGS')
'-Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -O2 -fexceptions -g
-grecord-gcc-switches -pipe -Wall -Werror=format-security -U_FORTIFY_SOURCE
-Wp,-U_FORTIFY_SOURCE -Wp,-D_FORTIFY_SOURCE=3 -Wp,-D_GLIBCXX_ASSERTIONS
-fstack-protector-strong -m64 -mtune=generic -fasynchronous-unwind-tables
-fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -O2
-fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security
-U_FORTIFY_SOURCE -Wp,-U_FORTIFY_SOURCE -Wp,-D_FORTIFY_SOURCE=3
-Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -m64 -mtune=generic
-fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection
-D_GNU_SOURCE -fPIC -fwrapv -O2 -fexceptions -g -grecord-gcc-switches -pipe
-Wall -Werror=format-security -U_FORTIFY_SOURCE -Wp,-U_FORTIFY_SOURCE
-Wp,-D_FORTIFY_SOURCE=3 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong
-m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection
-fcf-protection -D_GNU_SOURCE -fPIC -fwrapv'
>>> sysconfig.get_config_var('LDFLAGS')
'-Wl,-z,relro -Wl,--as-needed -Wl,-z,now -Wl,--build-id=sha1 -g
-Wl,-z,relro -Wl,--as-needed -Wl,-z,now -Wl,--build-id=sha1 -g'
I wonder if other flags should be removed as well.
Isn't Python built e.g. with -Werror=format-security or -Wsign-compare binary
compatible with extension modules built without it?
None of the warning flags should impact ABI or for that matter, even codegen.
What about FORTIFY_SOURCE and others?
It won't impact ABI, i.e. python could be built with _FORTIFY_SOURCE
and modules without, or vice versa.
Is there a compiler flags expert here who could help me determine
what flags
could (or even should) be removed from %{extension_*flags}?
If it is for distribution packages then I reckon the flags should be
as close as possible for the mere reason of consistency within the
distribution. If they're used for custom built modules (e.g. through
a tool that python auto-generates to build modules), then a very small
subset of the above flags would be necessary to retain binary
compatibility; you should be able to remove most of it.
Sid
1:42 p.m.
On 16. 01. 23 20:30, Siddhesh Poyarekar wrote:
If it is for distribution packages then I reckon the flags should be
as close as possible for the mere reason of consistency within the
distribution.
Nope, the individual packages with extension modules all¹ use the
%{build_cflags} flags by default.
¹: Technically they are required to and it is really hard not to do that noways.
If they're used for custom built modules (e.g. through
a tool that python auto-generates to build modules), then a very small
subset of the above flags would be necessary to retain binary
compatibility; you should be able to remove most of it.
That is what I thought. Thank you.
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
2:01 p.m.
On Mon, Jan 16, 2023 at 08:42:32PM +0100, Miro Hrončok wrote:
On 16. 01. 23 20:30, Siddhesh Poyarekar wrote:
> If it is for distribution packages then I reckon the flags should be
> as close as possible for the mere reason of consistency within the
> distribution.
Nope, the individual packages with extension modules all¹ use the
%{build_cflags} flags by default.
¹: Technically they are required to and it is really hard not to do that noways.
> If they're used for custom built modules (e.g. through
> a tool that python auto-generates to build modules), then a very small
> subset of the above flags would be necessary to retain binary
> compatibility; you should be able to remove most of it.
That is what I thought. Thank you.
Options like -m64/-m32 are ABI changing (but in %{optflags} these days
mostly uselessly because we don't usually cross-compile and gcc defaults to
those), -fsigned-char/-funsigned-char (I see it in redhat-rpm-config but
for armv3l/armv4b/armv4l which we don't build - in the past it was used on
more platforms) would be ABI changing, again on arm the various
-march=, -mfpu=, -mfloat-abi=, -mabi= options (but note that e.g. on most
other arches the -march= options aren't ABI changing, one just needs hw with
all the ISAs required by the built code), -fexceptions could be considered
partly ABI changing (at least say pthread_cancel through some objects
built with -fexceptions and others without that or with -fno-exceptions
will not destruct everything that should be after reaching frames without
it or similarly throwing C++ exceptions won't work well), so I'd suggest
to keep that one in. -mlong-double-{64,128} would be ABI changing, but
we just use compiler's default. -mabi={ibm,ieee}longdouble are ABI changing
on ppc64le too, but again we use the compiler defaults.
If you look at gcc documentation, many options are marked as ABI changing,
but I don't see them used in our %{optflags}...
So, when talking about options actually in use currently in f36/f37/f38 on
Fedora supported arches, I think -fexceptions is the only one I'd list.
Jakub
2:37 p.m.
On Mon, Jan 16, 2023 at 9:01 PM Jakub Jelinek <jakub(a)redhat.com> wrote:
On Mon, Jan 16, 2023 at 08:42:32PM +0100, Miro Hrončok wrote:
> On 16. 01. 23 20:30, Siddhesh Poyarekar wrote:
> > If it is for distribution packages then I reckon the flags should be
> > as close as possible for the mere reason of consistency within the
> > distribution.
>
> Nope, the individual packages with extension modules all¹ use the
> %{build_cflags} flags by default.
>
> ¹: Technically they are required to and it is really hard not to do that noways.
>
> > If they're used for custom built modules (e.g. through
> > a tool that python auto-generates to build modules), then a very small
> > subset of the above flags would be necessary to retain binary
> > compatibility; you should be able to remove most of it.
>
> That is what I thought. Thank you.
Options like -m64/-m32 are ABI changing (but in %{optflags} these days
mostly uselessly because we don't usually cross-compile and gcc defaults to
those), -fsigned-char/-funsigned-char (I see it in redhat-rpm-config but
for armv3l/armv4b/armv4l which we don't build - in the past it was used on
more platforms) would be ABI changing, again on arm the various
-march=, -mfpu=, -mfloat-abi=, -mabi= options (but note that e.g. on most
other arches the -march= options aren't ABI changing, one just needs hw with
all the ISAs required by the built code), -fexceptions could be considered
partly ABI changing (at least say pthread_cancel through some objects
built with -fexceptions and others without that or with -fno-exceptions
will not destruct everything that should be after reaching frames without
it or similarly throwing C++ exceptions won't work well), so I'd suggest
to keep that one in. -mlong-double-{64,128} would be ABI changing, but
we just use compiler's default. -mabi={ibm,ieee}longdouble are ABI changing
on ppc64le too, but again we use the compiler defaults.
If you look at gcc documentation, many options are marked as ABI changing,
but I don't see them used in our %{optflags}...
So, when talking about options actually in use currently in f36/f37/f38 on
Fedora supported arches, I think -fexceptions is the only one I'd list.
Isn't the problem here that building Python extensions needs to work
correctly in two - possibly conflicting - scenarios:
- in RPM packages, where using system compiler flags is a MUST
- when user installs Python packages manually, for example with pip in
a venv, where using system compiler flags isn't necessary (or
necessarily a good idea)
Just dropping everything but `-fexceptions` would certainly violate
Packaging Guidelines (i.e. packages not getting built with default
compiler and hardening flags) in the first scenario, wouldn't it?
Fabio
3:02 p.m.
On 16. 01. 23 21:37, Fabio Valentini wrote:
Isn't the problem here that building Python extensions needs to
work
correctly in two - possibly conflicting - scenarios:
- in RPM packages, where using system compiler flags is a MUST
Yes and packages get *all* the Fedora RPM flags that way via:
- https://fedoraproject.org/wiki/Changes/SetBuildFlagsBuildCheck
- %py3_build, %pyproject_wheel, etc.
- when user installs Python packages manually, for example with pip
in
a venv, where using system compiler flags isn't necessary (or
necessarily a good idea)
This is the case this thread is about.
Just dropping everything but `-fexceptions` would certainly violate
Packaging Guidelines (i.e. packages not getting built with default
compiler and hardening flags) in the first scenario, wouldn't it?
Dropping it from %{extension_*flags} would not change any Fedora package except
such packages that %undefine _auto_set_build_flags and don't use macros to
build stuff but invoke the build manually somehow -- such packages should
already be recognizable by not having annobin and package ELF notes.
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
9:39 a.m.
On Mon, Jan 16, 2023 at 10:02 PM Miro Hrončok <mhroncok(a)redhat.com> wrote:
On 16. 01. 23 21:37, Fabio Valentini wrote:
> Isn't the problem here that building Python extensions needs to work
> correctly in two - possibly conflicting - scenarios:
>
> - in RPM packages, where using system compiler flags is a MUST
Yes and packages get *all* the Fedora RPM flags that way via:
- https://fedoraproject.org/wiki/Changes/SetBuildFlagsBuildCheck
- %py3_build, %pyproject_wheel, etc.
> - when user installs Python packages manually, for example with pip in
> a venv, where using system compiler flags isn't necessary (or
> necessarily a good idea)
This is the case this thread is about.
Ok, thanks for the explanation!
In that case, I think it would make sense to have C / C++ compiler
flags for user-installed Python packages to fall back to GCC defaults
(same as for user-compiled C / C++ code), and only add flags that are
necessary for compatibility with system Python (looks like this is
only "-fexceptions"?).
Fabio
9:38 a.m.
Fabio Valentini wrote:
In that case, I think it would make sense to have C / C++ compiler
flags for user-installed Python packages to fall back to GCC defaults
(same as for user-compiled C / C++ code)
The GCC defaults are completely useless. -O0, almost no warnings, etc.
Kevin Kofler
476
days inactive
479
days old
8 comments
6 participants
participants (6)
-
Fabio Valentini -
Jakub Jelinek -
Kevin Kofler -
Kevin P. Fleming -
Miro Hrončok -
Siddhesh Poyarekar