11:07 a.m.
On Fri, 2020-06-05 at 15:56 +0000, devel-request(a)lists.fedoraproject.org wrote:
Send devel mailing list submissions to
devel(a)lists.fedoraproject.org
To subscribe or unsubscribe via email, send a message with subject or
body 'help' to
devel-request(a)lists.fedoraproject.org
You can reach the person managing the list at
devel-owner(a)lists.fedoraproject.org
When replying, please edit your Subject line so it is more specific
than "Re: Contents of devel digest..."
Today's Topics:
1. Re: Fedora 33 System-Wide Change proposal: CompilerPolicy Change
(Mark Wielaard)
2. Re: Fedora 33 System-Wide Change proposal: CompilerPolicy Change
(Josh Boyer)
3. Re: Fedora 33 System-Wide Change proposal: swap on zram
(Igor Raits)
4. Re: Fedora 33 System-Wide Change proposal: CompilerPolicy Change
(Jeff Law)
----------------------------------------------------------------------
Date: Fri, 05 Jun 2020 17:47:51 +0200
From: Mark Wielaard <mjw(a)fedoraproject.org>
Subject: Re: Fedora 33 System-Wide Change proposal: CompilerPolicy
Change
To: law(a)redhat.com, Development discussions related to Fedora
<devel(a)lists.fedoraproject.org>
Message-ID:
<3c8151c120d43cfa33d1dae11ba1c2b4f6e16be9.camel(a)fedoraproject.org>
Content-Type: text/plain; charset="UTF-8"
Hi,
On Fri, 2020-06-05 at 09:25 -0600, Jeff Law wrote:
> The LTO bytecode streams do not survive past any given package build. ie,
> they
> are used within the build, then discarded. They're not supposed to show up
> in
> any installed libraries.
> Thus the fact that the two compilers use totally
different LTO
> representations
> (and always will) is a non-issue here.
One issue I am concerned about here is debuginfo quality. GCC produced
pretty bad debuginfo with LTO in older versions. The EarlyDebug work
did make it usable. And we needed some work on the DARF consumers to
correctly process GCC 10 LTO produced DWARF (I actually have two small
patches for elfutils pending so it correctly parses the cross CU-
references GCC 10 LTO now produces).
I don't know if anybody did any analysis on llvm LTO produced
debuginfo. In the past it was observed that llvm doesn't do VTA (Var
Tracking Assignments) that is default in GCC. And VTA is really
important for debugging and performance/tracing tools like systemtap.
So even without LTO I am concerned about the observability of llvm
generated binaries.
And these are reasonable concerns (even more so in an LTO world
and I expect I'll
be spending most of my summer poking at debuginfo stuff for GCC LTO :-).
IMHO this is one of the issues an upstream project needs to evaluate for their
toolchain choice -- no different than performance, diagnostics, sanitizer
support, plug-in support, etc etc. The level of observability needed by any
particular upstream project can vary. ie, the kernel may well have different
needs than the Ruby interpreter.
I have not done any evaluation of LLVM's debug info. I would not be surprised if
GCC is generally better, particularly with Alex's work over the last couple
years. But again, I think the upstream project is in the best position to
determine how to balance this against the other factors in toolchain selection.
Jeff
4:04 p.m.
New subject: Fedora 33 System-Wide Change proposal: CompilerPolicy Change
Hi Jeff,
On Fri, 2020-06-05 at 10:07 -0600, Jeff Law wrote:
On Fri, 2020-06-05 at 15:56 +0000,
devel-request(a)lists.fedoraproject.org wrote:
> One issue I am concerned about here is debuginfo quality. GCC produced
> pretty bad debuginfo with LTO in older versions. The EarlyDebug work
> did make it usable. And we needed some work on the DARF consumers to
> correctly process GCC 10 LTO produced DWARF (I actually have two small
> patches for elfutils pending so it correctly parses the cross CU-
> references GCC 10 LTO now produces).
>
> I don't know if anybody did any analysis on llvm LTO produced
> debuginfo. In the past it was observed that llvm doesn't do VTA (Var
> Tracking Assignments) that is default in GCC. And VTA is really
> important for debugging and performance/tracing tools like systemtap.
> So even without LTO I am concerned about the observability of llvm
> generated binaries.
And these are reasonable concerns (even more so in an LTO world and I expect I'll
be spending most of my summer poking at debuginfo stuff for GCC LTO :-).
IMHO this is one of the issues an upstream project needs to evaluate for their
toolchain choice -- no different than performance, diagnostics, sanitizer
support, plug-in support, etc etc. The level of observability needed by any
particular upstream project can vary. ie, the kernel may well have different
needs than the Ruby interpreter.
I think you place too much faith in upstream and give Fedora and Fedora
packagers not enough credit for creating a well integrated
distribution. For Fedora it is important that what we ship is
observable by our users. We, together with our users, sometimes need to
debug, trace or do performance analysis on the binaries as we ship
them. Some upstream projects don't care about that at all. They will
happily tell their users to just rebuild with some other
tools/optimizations/etc. Or just rebuild their whole world with some
sanitizers enabled. So then an upstream developer might prefer a
different toolchain, simply because they personally would rebuild the
whole world with it. In such cases Fedora should not simply switch to a
non-default system compiler.
I have not done any evaluation of LLVM's debug info. I would not
be surprised if
GCC is generally better, particularly with Alex's work over the last couple
years. But again, I think the upstream project is in the best position to
determine how to balance this against the other factors in toolchain selection.
GCC's is certainly better, it has much better coverage. That doesn't
mean that LLVM's debuginfo is invalid, but it is less useful. At some
point the kernel accidentially disabled VTA and it made the kernel a
lot less observable by systemtap for example.
And even if it is correct it might still not be fully supported by all
our tools. For example clang used to default to not generate
.debug_aranges, which is valid, but means a DWARF consumer needs to
construct its own aranges table by reading the all the CUs. elfutils
doesn't support that, and so some address/line lookups simply fail with
clang generated binaries. I could cerainly add support for missing
.debug_aranges, but there is so much more to do (luckily I could
convince them to switch the default to generate .debug_aranges, at
least for rust code).
In general when someone reports an issue with an elfutils or valgrind
based tool and it is against code not compiled with the default gcc
toolchain I simply have to tell people to please use the system
compiler. There are just not enough time to also try to resolve any
debuginfo issues because people use an alternative toolchain.
So I like this proposal to be a bit more strict in when it is OK to not
use the system toolchain. Just deferring to upstream preference seems
too weak. If upstream supports building with gcc, but some upstream
developers have a preference for using clang, I would prefer to make it
clear that the Fedora package should keep using gcc. We simply don't
have the resources for all other tools to have to deal with another
toolchain.
Thanks,
Mark
10:48 a.m.
New subject: Fedora 33 System-Wide Change proposal: CompilerPolicy Change
On Fri, 2020-06-05 at 23:04 +0200, Mark Wielaard wrote:
Hi Jeff,
On Fri, 2020-06-05 at 10:07 -0600, Jeff Law wrote:
> On Fri, 2020-06-05 at 15:56 +0000, devel-request(a)lists.fedoraproject.org wrote:
> > One issue I am concerned about here is debuginfo quality. GCC produced
> > pretty bad debuginfo with LTO in older versions. The EarlyDebug work
> > did make it usable. And we needed some work on the DARF consumers to
> > correctly process GCC 10 LTO produced DWARF (I actually have two small
> > patches for elfutils pending so it correctly parses the cross CU-
> > references GCC 10 LTO now produces).
> >
> > I don't know if anybody did any analysis on llvm LTO produced
> > debuginfo. In the past it was observed that llvm doesn't do VTA (Var
> > Tracking Assignments) that is default in GCC. And VTA is really
> > important for debugging and performance/tracing tools like systemtap.
> > So even without LTO I am concerned about the observability of llvm
> > generated binaries.
>
> And these are reasonable concerns (even more so in an LTO world and I expect
I'll
> be spending most of my summer poking at debuginfo stuff for GCC LTO :-).
>
> IMHO this is one of the issues an upstream project needs to evaluate for their
> toolchain choice -- no different than performance, diagnostics, sanitizer
> support, plug-in support, etc etc. The level of observability needed by any
> particular upstream project can vary. ie, the kernel may well have different
> needs than the Ruby interpreter.
I think you place too much faith in upstream and give Fedora and Fedora
packagers not enough credit for creating a well integrated
distribution. For Fedora it is important that what we ship is
observable by our users. We, together with our users, sometimes need to
debug, trace or do performance analysis on the binaries as we ship
them. Some upstream projects don't care about that at all. They will
happily tell their users to just rebuild with some other
tools/optimizations/etc. Or just rebuild their whole world with some
sanitizers enabled. So then an upstream developer might prefer a
different toolchain, simply because they personally would rebuild the
whole world with it. In such cases Fedora should not simply switch to a
non-default system compiler.
I put faith in both the upstreams and packagers to do
the right thing for their
project. Right now Fedora policy does exactly the opposite by forcing everyone
to use GCC rather than making an informed, per project, decision.
If an upstream has decided that debug-ability isn't a priority, then, the
packager for Fedora has to decide if the project is even really suitable for
Fedora and take on a degree of responsibility for the package.
> I have not done any evaluation of LLVM's debug info. I would not be surprised
if
> GCC is generally better, particularly with Alex's work over the last couple
> years. But again, I think the upstream project is in the best position to
> determine how to balance this against the other factors in toolchain selection.
GCC's is certainly better, it has much better coverage. That doesn't
mean that LLVM's debuginfo is invalid, but it is less useful. At some
point the kernel accidentially disabled VTA and it made the kernel a
lot less observable by systemtap for example.
I'm well aware of the
kernels' disabling VTA, it was done in response to a
codegen bug that was triggered by the VTA changes. Sadly the kernel's response
was a bit drastic. But that's water under the bridge.
And even if it is correct it might still not be fully supported by all
our tools. For example clang used to default to not generate
.debug_aranges, which is valid, but means a DWARF consumer needs to
construct its own aranges table by reading the all the CUs. elfutils
doesn't support that, and so some address/line lookups simply fail with
clang generated binaries. I could cerainly add support for missing
.debug_aranges, but there is so much more to do (luckily I could
convince them to switch the default to generate .debug_aranges, at
least for rust code).
In general when someone reports an issue with an elfutils or valgrind
based tool and it is against code not compiled with the default gcc
toolchain I simply have to tell people to please use the system
compiler. There are just not enough time to also try to resolve any
debuginfo issues because people use an alternative toolchain.
All valid concerns.
THe way to respond when something like this comes up is
something like:
1. Analyze the issue and report back what was found.
2. Create bugs/rfes to track additional work that may need to be performed
3. Give them possible workarounds by using different tools.
That should in turn flow to the uptream package and becomes part of the feedback
loop they use to determine what toolchain they should be using. Its no different
than a bug or missing feature -- the upstream project gets to decide if the issue
is significant enough to warrant changing recommendations or waiting for the
tools they're using to catch up.
Simply punting is no longer a real option IMHO.
So I like this proposal to be a bit more strict in when it is OK to not
use the system toolchain. Just deferring to upstream preference seems
too weak. If upstream supports building with gcc, but some upstream
developers have a preference for using clang, I would prefer to make it
clear that the Fedora package should keep using gcc. We simply don't
have the resources for all other tools to have to deal with another
toolchain.
It's probably a given that for a sufficiently large upstream that
there will be
some in that upstream that want to use GCC and others that will want to use
Clang/LLVM. The fact that individual upstream developers want to use compiler X
and others want to use compiler Y isn't the deciding factor -- what is the
upstream's project's preference/stance.
Jeff
1:57 a.m.
New subject: Fedora 33 System-Wide Change proposal: CompilerPolicy Change
Le lundi 08 juin 2020 à 09:48 -0600, Jeff Law a écrit :
I put faith in both the upstreams and packagers to do the right thing
for their project. Right now Fedora policy does exactly the opposite
by forcing everyone to use GCC rather than making an informed, per
project, decision.
Look, 9∕10th of the times the "informed" upstream decision is just one
poor dev who slapped some build automation using the first compiler he
though of in quick and dirty mode so he could get back to coding as
fast as possible (sometimes, for another system than Linux, that was
grandfathered for Linux builds later, because who cares about those
things in real life).
Sifting through build systems, possible compilers and compiler options
takes an enormous amount of care and knowledge which is lacking in the
average upstream project. Only giants like Chrome or Mozilla can afford
to do this level of analysis. I’m not even putting LibreOffice in this
class – they switched compiler for a subset of their codebase, probably
just to workaround a perf regression quickly in the hope that someone
would analyse the regression causes later. Not even sure they bothered
to check if the regression happened with all possible build flags
combinations, one report of "it’s faster using compiler XXX" (one
version of one compiler on one system with one set of build flags), is
usually sufficient for this kind of switch.
Arguably, in Fedora the average packager can not afford it either – he
relies 100% on the default compiler flags defined by the gcc team. And
the gcc team can afford to look at those closely only because of the
mass effect of a whole distribution codebase that is fully built using
the same compiler and flags everywhere.
“Upstream knows better” – for upstream’s own code sure, but for
everything else (legalities, the tird party code they depend on, build
systems and automation) certainly not.
Regards,
--
Nicolas Mailhot
10:17 a.m.
New subject: Fedora 33 System-Wide Change proposal: CompilerPolicy Change
On Tue, 2020-06-09 at 08:57 +0200, Nicolas Mailhot wrote:
Le lundi 08 juin 2020 à 09:48 -0600, Jeff Law a écrit :
> I put faith in both the upstreams and packagers to do the right thing
> for their project. Right now Fedora policy does exactly the opposite
> by forcing everyone to use GCC rather than making an informed, per
> project, decision.
Look, 9∕10th of the times the "informed" upstream decision is just one
poor dev who slapped some build automation using the first compiler he
though of in quick and dirty mode so he could get back to coding as
fast as possible (sometimes, for another system than Linux, that was
grandfathered for Linux builds later, because who cares about those
things in real life).
That certainly happens. I get to see both the good and the
bad in terms of code
quality, build systems, etc for Fedora.
I won't call out any packages by name, but some of the stuff I've seen is of such
poor quality that the author should be embarrassed. Thankfully these packages
are likely only being used by one person and when it crashes because they've
written high school quality code the impact is minimal.
In other cases the upstream code is exceedingly good and thorough in terms of
trying to do the right thing. So much in fact that it obscures the intent of the
code so much that GCC can't actually analyze it fully and GCC spits out a false
positive diagnostic.
I've seen code put in to shut up things like Coverity that is horribly wrong and
introduces new buffer overflows and undefined behavior.
But I'm also experienced enough to know there's a lot of selection bias that goes
on here. I see the old crappy code because it breaks and breaks often as we
update from one version of gcc to the next. I see the well written, security
sensitive code as well because gcc tries hard to issue diagnostics for
potentially problematic code -- and it's precisely those codes that end up being
complex as the developer tries to do the right thing and avoid undefined behavior
in the first place.
Sifting through build systems, possible compilers and compiler options
takes an enormous amount of care and knowledge which is lacking in the
average upstream project. Only giants like Chrome or Mozilla can afford
to do this level of analysis. I’m not even putting LibreOffice in this
class – they switched compiler for a subset of their codebase, probably
just to workaround a perf regression quickly in the hope that someone
would analyse the regression causes later. Not even sure they bothered
to check if the regression happened with all possible build flags
combinations, one report of "it’s faster using compiler XXX" (one
version of one compiler on one system with one set of build flags), is
usually sufficient for this kind of switch.
No doubt. It does take work, often
significant amounts of work. I get to see it
all because of the work I do to try and make sure GCC and Fedora work well
together results in me looking at hundreds or thousands of build issues every
cycle. I have to figure out the build system, intent of the sources or
testcases, then debug what's going on in the package and what should be going on
in whatever part is failing, then make a decision about whether or not the issue
is a package problem or a compiler problem, then fix the problem or give someone
else enough state to do so. Rinse and repeat.
Arguably, in Fedora the average packager can not afford it either – he
relies 100% on the default compiler flags defined by the gcc team. And
the gcc team can afford to look at those closely only because of the
mass effect of a whole distribution codebase that is fully built using
the same compiler and flags everywhere.
And that's not changing. The GCC team
still selects default flags, we still
analyze build issues to ensure things go as smoothly as possible, we still look
for ways to make things work faster, compiler to smaller code, etc. We also work
closely with our LLVM team to make sure we're doing the right thing for the
Clang/LLVM toolchain as well.
“Upstream knows better” – for upstream’s own code sure, but for
everything else (legalities, the tird party code they depend on, build
systems and automation) certainly not.
I'd disagree, particularly around build
systems and the compiler components
within build systems. THere's always exceptions, but by in large I believe
upstreams are in the best position to select the compiler toolchains.
Jeff
10:41 a.m.
New subject: Fedora 33 System-Wide Change proposal: CompilerPolicy Change
On Tue, Jun 09, 2020 at 09:17:13AM -0600, Jeff Law wrote:
I'd disagree, particularly around build systems and the compiler
components
within build systems. THere's always exceptions, but by in large I believe
upstreams are in the best position to select the compiler toolchains.
Fedora routinely trips over all manner of breakages by its aggressive
toolchain upgrades and enabling various security/hardening features that
upstreams (almost) always ignore.
Fedora's packagers carry the burden of fixing and maintaining patches
for countless packages that run into issues related to those toolchain
upgrades.
So why is this "upstream knows best" argument not valid with respect to
Fedora forcing mass upgrading to newer versions of GCC (or LLVM, or
whatever) and hardening features that expose misbehaviors?
- Solomon
--
Solomon Peachy pizza at shaftnet dot org (email&xmpp)
@pizza:shaftnet dot org (matrix)
High Springs, FL speachy (freenode)
1416
days inactive
1420
days old
5 comments
4 participants
participants (4)
-
Jeff Law -
Mark Wielaard -
Nicolas Mailhot -
Solomon Peachy