3:39 a.m.
Following is the list of topics that will be discussed in the
FESCo meeting Tuesday at 17:00UTC in #fedora-meeting on
irc.libera.chat.
To convert UTC to your local time, take a look at
http://fedoraproject.org/wiki/UTCHowto
or run:
date -d '2023-01-03 17:00 UTC'
Links to all issues to be discussed can be found at:
https://pagure.io/fesco/report/meeting_agenda
= Discussed and Voted in the Ticket =
Title of issue
https://pagure.io/fesco/issue/###
DECISION (+X, Y, -Z)
Change: Strong crypto settings: phase 3, forewarning 2/2
https://pagure.io/fesco/issue/2865
REJECTED (+1, 4, 0)
Non-responsive maintainer: miminar / Michal Minar
https://pagure.io/fesco/issue/2891
APPROVED (+4,0,-0)
Change: Perl: Replace versioned MODULE_COMPAT_ requires by RPM dependency generator
https://pagure.io/fesco/issue/2898
APPROVED (+2, 0, -0)
Nonresponsive maintainer: Steven Roberts strobert
https://pagure.io/fesco/issue/2902
APPROVED (+2, 0, -0)
Change: Golang 1.20
https://pagure.io/fesco/issue/2913
APPROVED (+6, 0, -0)
Change: Fedora Sway Spin
https://pagure.io/fesco/issue/2914
APPROVED (+6, 0, -0)
Change: libpinyin 2.8
https://pagure.io/fesco/issue/2915
APPROVED (+6, 0, -0)
Change: GNU Make version 4.4
https://pagure.io/fesco/issue/2916
APPROVED (+5, 0, -0)
Change: Add _FORTIFY_SOURCE=3 to distribution build flags
https://pagure.io/fesco/issue/2917
APPROVED: (+5, 0, 0)
Change: Boost 1.81 upgrade
https://pagure.io/fesco/issue/2918
APPROVED (+7, 0, -0)
Change: Restore stricter SSH hostkeys permissions
https://pagure.io/fesco/issue/2919
APPROVED (+4, 0, -0)
Change: ImageMagick 7
https://pagure.io/fesco/issue/2920
APPROVED (+6, 0, -0)
Change: Fedora Budgie Spin
https://pagure.io/fesco/issue/2921
APPROVED (+5, 0, -0)
Change: Use mdadm for BIOS RAID Support in Anaconda
https://pagure.io/fesco/issue/2922
= New business =
#2923 Re-vote for Change proposal: Add -fno-omit-frame-pointer to default
compilation flags
https://pagure.io/fesco/issue/2923
= Open Floor =
For more complete details, please visit each individual
issue. The report of the agenda items can be found at
https://pagure.io/fesco/report/meeting_agenda
If you would like to add something to this agenda, you can
reply to this e-mail, file a new issue at
https://pagure.io/fesco, e-mail me directly, or bring it
up at the end of the meeting, during the open floor topic. Note
that added topics may be deferred until the following meeting.
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
3:44 a.m.
On 03. 01. 23 10:39, Miro Hrončok wrote:
Title of issue
https://pagure.io/fesco/issue/###
DECISION (+X, Y, -Z)
There was no actual issue supposed to be listed here instead of the
placeholder, it was merely redundant.
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
3:47 a.m.
On 03. 01. 23 10:39, Miro Hrončok wrote:
Change: Use mdadm for BIOS RAID Support in Anaconda
https://pagure.io/fesco/issue/2922
A decision has not yet been made officially for this one, I copy pasted it to
the email and forgot to remove it before I sent it. Sorry about that.
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
3:48 a.m.
On Tue, Jan 03, 2023 at 10:39:13AM +0100, Miro Hrončok wrote:
= Discussed and Voted in the Ticket =
…
Change: Use mdadm for BIOS RAID Support in Anaconda
https://pagure.io/fesco/issue/2922
Something went wrong here. The tally is +4,0,0 on this one.
The ticket wasn't closed.
Zbyszek
11:42 a.m.
===================================
#fedora-meeting: FESCO (2023-01-03)
===================================
Meeting started by mhroncok at 17:01:53 UTC. The full logs are available
at
https://meetbot.fedoraproject.org/fedora-meeting/2023-01-03/fesco.2023-01...
.
Meeting summary
---------------
* init process (mhroncok, 17:02:08)
* #2923 Re-vote for Change proposal: Add -fno-omit-frame-pointer to
default (mhroncok, 17:06:26)
* LINK:
https://src.fedoraproject.org/rpms/redhat-rpm-config/pull-request/230
is the implementation (davide, 17:13:47)
* AGREED: APPROVED (+6,1,-1) This Change is implemented for Fedora
Linux 38 and we evaluate whether to retain it by Fedora Linux 40.
This Change must be implemented in a manner which packages are able
to trivially opt-out of retaining frame pointers during compilation
so that packages that take larger performance hits can easily
revert. (mhroncok, 17:20:38)
* Change owners please coordinate the change with the Python
Maintainers before changing the defaults (mhroncok, 17:21:20)
* Next week's chair (mhroncok, 17:21:46)
* ACTION: Conan Kudo will probably chair next meeting (mhroncok,
17:22:17)
* Open Floor (mhroncok, 17:23:10)
* #2907 Exception for spliting OpenJDK build and integration (mhroncok,
17:23:47)
* AGREED: No exception is needed (+5,1,-0) (mhroncok, 17:31:20)
* Open Floor (mhroncok, 17:31:52)
Meeting ended at 17:42:06 UTC.
Action Items
------------
* Conan Kudo will probably chair next meeting
Action Items, by person
-----------------------
* **UNASSIGNED**
* Conan Kudo will probably chair next meeting
People Present (lines said)
---------------------------
* mhroncok (66)
* zbyszek (27)
* Eighth_Doctor (25)
* zodbot (20)
* hergertme (13)
* mhayden (9)
* davide (6)
* dcantrell (4)
* music[m] (2)
* salimma (1)
* MichaelCatanzaro (1)
* nirik (0)
* decathorpe (0)
* sgallagh (0)
* music (0)
* Conan_Kudo (0)
* Pharaoh_Atem (0)
* Son_Goku (0)
* King_InuYasha (0)
* Sir_Gallantmon (0)
Generated by `MeetBot`_ 0.4
.. _`MeetBot`: https://fedoraproject.org/wiki/Zodbot#Meeting_Functions
11:43 a.m.
New subject: Summary/Minutes from today's FESCo Meeting (2023-01-03)
(once again with a proper Subject line)
===================================
#fedora-meeting: FESCO (2023-01-03)
===================================
Meeting started by mhroncok at 17:01:53 UTC. The full logs are available
at
https://meetbot.fedoraproject.org/fedora-meeting/2023-01-03/fesco.2023-01...
.
Meeting summary
---------------
* init process (mhroncok, 17:02:08)
* #2923 Re-vote for Change proposal: Add -fno-omit-frame-pointer to
default (mhroncok, 17:06:26)
* LINK:
https://src.fedoraproject.org/rpms/redhat-rpm-config/pull-request/230
is the implementation (davide, 17:13:47)
* AGREED: APPROVED (+6,1,-1) This Change is implemented for Fedora
Linux 38 and we evaluate whether to retain it by Fedora Linux 40.
This Change must be implemented in a manner which packages are able
to trivially opt-out of retaining frame pointers during compilation
so that packages that take larger performance hits can easily
revert. (mhroncok, 17:20:38)
* Change owners please coordinate the change with the Python
Maintainers before changing the defaults (mhroncok, 17:21:20)
* Next week's chair (mhroncok, 17:21:46)
* ACTION: Conan Kudo will probably chair next meeting (mhroncok,
17:22:17)
* Open Floor (mhroncok, 17:23:10)
* #2907 Exception for spliting OpenJDK build and integration (mhroncok,
17:23:47)
* AGREED: No exception is needed (+5,1,-0) (mhroncok, 17:31:20)
* Open Floor (mhroncok, 17:31:52)
Meeting ended at 17:42:06 UTC.
Action Items
------------
* Conan Kudo will probably chair next meeting
Action Items, by person
-----------------------
* **UNASSIGNED**
* Conan Kudo will probably chair next meeting
People Present (lines said)
---------------------------
* mhroncok (66)
* zbyszek (27)
* Eighth_Doctor (25)
* zodbot (20)
* hergertme (13)
* mhayden (9)
* davide (6)
* dcantrell (4)
* music[m] (2)
* salimma (1)
* MichaelCatanzaro (1)
* nirik (0)
* decathorpe (0)
* sgallagh (0)
* music (0)
* Conan_Kudo (0)
* Pharaoh_Atem (0)
* Son_Goku (0)
* King_InuYasha (0)
* Sir_Gallantmon (0)
Generated by `MeetBot`_ 0.4
.. _`MeetBot`: https://fedoraproject.org/wiki/Zodbot#Meeting_Functions
7:30 a.m.
On 03/01/2023 18:42, Miro Hrončok wrote:
* AGREED: APPROVED (+6,1,-1) This Change is implemented for
Fedora
Linux 38 and we evaluate whether to retain it by Fedora Linux 40.
This Change must be implemented in a manner which packages are able
to trivially opt-out of retaining frame pointers during compilation
so that packages that take larger performance hits can easily
revert. (mhroncok, 17:20:38)
Already rejected proposal was submitted because big corporations weren't
happy with the results. This is a VERY BAD precedent for Fedora.
--
Sincerely,
Vitaly Zaitsev (vitaly(a)easycoding.org)
7:44 a.m.
On Wed, Jan 4, 2023 at 8:30 AM Vitaly Zaitsev via devel
<devel(a)lists.fedoraproject.org> wrote:
On 03/01/2023 18:42, Miro Hrončok wrote:
> * AGREED: APPROVED (+6,1,-1) This Change is implemented for Fedora
> Linux 38 and we evaluate whether to retain it by Fedora Linux 40.
> This Change must be implemented in a manner which packages are able
> to trivially opt-out of retaining frame pointers during compilation
> so that packages that take larger performance hits can easily
> revert. (mhroncok, 17:20:38)
Already rejected proposal was submitted because big corporations weren't
happy with the results. This is a VERY BAD precedent for Fedora.
Actually, the Change owners were prepared to give up. I was the one
that pushed for it to be reconsidered because of how much benefit it
gives to desktop Linux developers. The Change owners were prepared to
do the work and assist anywhere to restore/improve performance. The
GNOME and KDE developers both have tooling that benefits from this
Change, and I wanted profiling and introspection capabilities for the
desktop (just like other OSes have). When the Change proposal came in
for FORTIFY_SOURCE=3 (which introduces similar pressure), the
resulting discussion prompted the re-vote.
Most of the users of this feature will be community volunteer
developers that need every advantage they can get.
--
真実はいつも一つ！/ Always, there's only one truth!
7:49 a.m.
On 04/01/2023 14:44, Neal Gompa wrote:
Actually, the Change owners were prepared to give up. I was the one
that pushed for it to be reconsidered because of how much benefit it
gives to desktop Linux developers.
The original proposal received a lot of negative feedback. Only a few
big corporations will get benefit and end users will get a 3-10%
performance penalty which is absolutely unacceptable.
--
Sincerely,
Vitaly Zaitsev (vitaly(a)easycoding.org)
7:56 a.m.
On Wed, Jan 4, 2023 at 8:50 AM Vitaly Zaitsev via devel
<devel(a)lists.fedoraproject.org> wrote:
On 04/01/2023 14:44, Neal Gompa wrote:
> Actually, the Change owners were prepared to give up. I was the one
> that pushed for it to be reconsidered because of how much benefit it
> gives to desktop Linux developers.
The original proposal received a lot of negative feedback. Only a few
big corporations will get benefit and end users will get a 3-10%
performance penalty which is absolutely unacceptable.
I'm not going to argue with you on this again, but suffice it to say
multiple GNOME developers have come out and said they need this
feature to do performance work.
The item where we take a significant performance hit is Python, and
that's going to opt-out:
https://src.fedoraproject.org/rpms/python3.11/pull-request/95
For what it's worth, frame pointers are required on other operating
systems for precisely this reason. Linux is the odd duck out and it
causes us many problems for real-time profiling, performance analysis,
and other development tasks.
If you want more performance gains, you need to be able to observe
real workloads in real environments and see where the bottlenecks are.
I'm confident that over the course of the next year, we'll recover
performance that was lost by FORTIFY_SOURCE=3 and frame pointers as
technology improves and we get used to having these by default.
--
真実はいつも一つ！/ Always, there's only one truth!
8:25 a.m.
On Wed, Jan 4 2023 at 02:49:30 PM +0100, Vitaly Zaitsev via devel
<devel(a)lists.fedoraproject.org> wrote:
. Only a few
big corporations will get benefit
Please, stop saying this. It's just not true. All Fedora users will
benefit from the performance improvements that we're able to make using
sysprof. Right now desktop developers do almost no profiling because
it's just too difficult. I don't think I've ever successfully profiled
any application *ever*. When I see bug reports that require profiling
to solve, I think "too bad" and move on to easier bugs. Having
profiling tools that actually work will absolutely help improve
performance for our users.
10:29 a.m.
On Tue, 3 Jan 2023 at 09:39, Miro Hrončok <mhroncok(a)redhat.com> wrote:
= New business =
#2923 Re-vote for Change proposal: Add -fno-omit-frame-pointer to default
compilation flags
https://pagure.io/fesco/issue/2923
Given the controversial nature of this one, why was it re-litigated at
short notice when a large number of the stakeholders were still on
vacation?
11:28 a.m.
* Miro Hrončok:
* #2923 Re-vote for Change proposal: Add -fno-omit-frame-pointer to
default (mhroncok, 17:06:26)
* LINK:
https://src.fedoraproject.org/rpms/redhat-rpm-config/pull-request/230
is the implementation (davide, 17:13:47)
* AGREED: APPROVED (+6,1,-1) This Change is implemented for Fedora
Linux 38 and we evaluate whether to retain it by Fedora Linux 40.
This Change must be implemented in a manner which packages are able
to trivially opt-out of retaining frame pointers during compilation
so that packages that take larger performance hits can easily
revert. (mhroncok, 17:20:38)
* Change owners please coordinate the change with the Python
Maintainers before changing the defaults (mhroncok, 17:21:20)
The change as voted simply does not work at a technical level because
-mno-omit-leaf-frame-pointer is an architecture-specific GCC option that
is not available on all Fedora architectures. I don't think
-fno-omit-frame-pointer is well-exercised on s390x, so I wouldn't want
to use it there, either.
Is it safe to assume this change (as voted) is actually intended for
x86_64 only, in both directions? Specifically, that opting out will
*not* disable frame pointers on aarch64 and ppc64le (where it would
result in an ABI break)?
We should not ask package maintainers to debug i686 build failures
related to inline assembly and register shortage. (On i686, it's
impossible to make certain system calls while preserving a frame
pointer.) As far as I understand it, building i686 with frame pointers
isn't in scope for the change, either, because it does not further its
goals.
Regarding leaf functions, on typical workloads, a significant fraction
of the profiling samples will be in glibc string functions, which do not
have frame pointers. So any profiler has to cope with leaf functions
with frame pointers anyway. As a result, do we really need the
-mno-omit-leaf-frame-pointer option?
On aarch64, we should not use -mno-omit-leaf-frame-pointer because there
is no skipped frame problem with backtraces: the link register (x30) can
be read directly even in leaf functions. The ABI only mandates frame
pointers for non-leaf functions. On ppc64le, GCC does not even support
-mno-omit-leaf-frame-pointer.
This is why I think the change is implicitly just for x86_64.
Does anybody know how this change invalidates Fedora binaries as a
reference for DWARF performance work? Do the generated .eh_frame data
change materially once frame pointers are in use?
Thanks,
Florian
11:52 a.m.
On 1/4/23 10:05, Vitaly Zaitsev via devel wrote:
On 04/01/2023 15:25, Michael Catanzaro wrote:
> All Fedora users will benefit from the performance improvements that
> we're able to make using sysprof.
Maybe. Or maybe not. And the performance penalty is here and now.
The optimizations enabled by profiling can be much larger than 3-10%.
To be clear, I would prefer a means of profiling that does not cause a
performance penalty for everyone else, but that will take much longer
to create.
--
Sincerely,
Demi Marie Obenour (she/her/hers)
12:10 p.m.
On 2023-01-04 09:28, Florian Weimer wrote:
The change as voted simply does not work at a technical level
because
-mno-omit-leaf-frame-pointer is an architecture-specific GCC option
that
is not available on all Fedora architectures. I don't think
-fno-omit-frame-pointer is well-exercised on s390x, so I wouldn't want
to use it there, either.
The latest implementation update for this in
https://src.fedoraproject.org/rpms/redhat-rpm-config/pull-request/231
uses -mbackchain on s390x and drops -mno-omit-leaf-frame-pointer for
ppc64le.
Is it safe to assume this change (as voted) is actually intended for
x86_64 only, in both directions?
The Change owners are primarily interested in x86_64 and aarch64, but we
did not intend this to be x86_64 specific.
Specifically, that opting out will
*not* disable frame pointers on aarch64 and ppc64le (where it would
result in an ABI break)?
This is correct, and I've updated the documentation to clarify. Opting
out will revert to the compiler defaults, which may or may not include
frame pointers depending on the architecture.
Cheers
Davide
1:26 p.m.
The change as voted simply does not work at a technical level
because
-mno-omit-leaf-frame-pointer is an architecture-specific GCC option that
is not available on all Fedora architectures. I don't think
-fno-omit-frame-pointer is well-exercised on s390x, so I wouldn't want
to use it there, either.
Is it safe to assume this change (as voted) is actually intended for
x86_64 only, in both directions? Specifically, that opting out will
*not* disable frame pointers on aarch64 and ppc64le (where it would
result in an ABI break)?
I'd say we should just keep the spirit of the proposal in mind: to enable frame
pointers on all platforms where it's possible. Change owners are certainly not experts
on intricacies of each possible architecture, so it would definitely help to get help and
feedback from people like you on what specifically needs to be enabled to make this work
across all arches.
Regarding leaf functions, on typical workloads, a significant
fraction
of the profiling samples will be in glibc string functions, which do not
have frame pointers. So any profiler has to cope with leaf functions
with frame pointers anyway. As a result, do we really need the
-mno-omit-leaf-frame-pointer option?
I think we should be careful about defining "typical workloads" and not assume
that it's mostly inside glibc. But even if it is, just because glibcs leaf functions
don't have frame pointers doesn't seem to imply that leaf pointers should be
generally left out. As I said above, the spirit of the proposal is to make stack traces as
accurate as possible. That includes leaf functions, right?
"any profiler has to cope with leaf functions" is true, but what is
"cope"? Some more advanced solutions combine LBR stacks with frame pointers and
are able to recover full stacks. But the space of stack trace uses (at least with BPF) is
vast. There will be BPF-based tools of various degree of sophistication, and I'm 100%
sure all of them won't go to the trouble of using LBR to recover the stack. Certainly
this won't happen with simple ad-hoc bpftrace scripts.
So that's just to say that I think we should try hard to get leaf frame pointers as
well so that stack traces are better in general. For glibc cases, we'll just get stack
traces without last level, unless we do something more complicated with LBR.
WDYT?
On aarch64, we should not use -mno-omit-leaf-frame-pointer because there
is no skipped frame problem with backtraces: the link register (x30) can
be read directly even in leaf functions. The ABI only mandates frame
pointers for non-leaf functions. On ppc64le, GCC does not even support
-mno-omit-leaf-frame-pointer.
This is why I think the change is implicitly just for x86_64.
Definitely not intentionally, might be just a bias of what we had most hands-on experience
with.
2:42 p.m.
Michael Catanzaro <mcatanzaro(a)redhat.com> writes:
[...]
Please, stop saying this. It's just not true. All Fedora users will
benefit from the performance improvements that we're able to make
using sysprof. [...]
Then perhaps the Change could have had a dead-man-switch built in:
unless performance improvements due to this profiling change do not in
fact appear by F39 or F40, the Change should be automatically reverted.
- FChE
3:12 p.m.
On Wed, Jan 4 2023 at 03:42:43 PM -0500, Frank Ch. Eigler
<fche(a)redhat.com> wrote:
Then perhaps the Change could have had a dead-man-switch built in:
unless performance improvements due to this profiling change do not in
fact appear by F39 or F40, the Change should be automatically
reverted.
Question is how to measure that? Is it sufficient to fix one or two
performance bugs to call this change a success, or do we need more?
Specifically how many? You can expect specific application-level
performance improvements probably. Benchmarks probably won't improve.
Michael
10:10 p.m.
Michael Catanzaro <mcatanzaro(a)redhat.com> writes:
> Then perhaps the Change could have had a dead-man-switch built
in:
> unless performance improvements due to this profiling change do not in
> fact appear by F39 or F40, the Change should be automatically
> reverted.
Question is how to measure that? Is it sufficient to fix one or two
performance bugs to call this change a success, or do we need more?
Specifically how many? [...]
If I understood it correctly, the claim was that enabling this
distro-user-penalizing option would make it back in terms of performance
optimizations. That it was an investement that would have a positive
return.
Let's be firm in testing this empirically rather than aspirationally.
- FChE
9:15 a.m.
On Wed, Jan 4 2023 at 11:10:54 PM -0500, Frank Ch. Eigler
<fche(a)redhat.com> wrote:
If I understood it correctly, the claim was that enabling this
distro-user-penalizing option would make it back in terms of
performance
optimizations.
Of course, but the benefit is to fix performance bugs in applications
or maybe the desktop itself. It's not going to result in general
improvements that benefit benchmarks. Benchmarks will surely get worse,
not better.
That it was an investement that would have a positive
return.
Let's be firm in testing this empirically rather than aspirationally.
I really don't know how. Suggestions welcome.
Michael
9:48 a.m.
On Wed, Jan 04, 2023 at 02:30:07PM +0100, Vitaly Zaitsev via devel wrote:
On 03/01/2023 18:42, Miro Hrončok wrote:
> * AGREED: APPROVED (+6,1,-1) This Change is implemented for Fedora
> Linux 38 and we evaluate whether to retain it by Fedora Linux 40.
> This Change must be implemented in a manner which packages are able
> to trivially opt-out of retaining frame pointers during compilation
> so that packages that take larger performance hits can easily
> revert. (mhroncok, 17:20:38)
Already rejected proposal was submitted because big corporations
weren't happy with the results. This is a VERY BAD precedent for
Fedora.
Not sure who the big bad corps are, but adding frame pointers is a
very good idea if you've ever tried using perf.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-top is 'top' for virtual machines. Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://people.redhat.com/~rjones/virt-top
10:08 a.m.
Of course, but the benefit is to fix performance bugs in
applications
or maybe the desktop itself. [...]
> Let's be firm in testing this empirically rather than
aspirationally.
I really don't know how. Suggestions welcome.
I'd put the onus on the proponents of the Change, who made predictions
like "I'm confident that over the course of the next year, we'll recover
performance that was lost by FORTIFY_SOURCE=3 and frame pointers" and
"The optimizations enabled by profiling can be much larger than 3-10%.",
and that's just in the last few days.
There needs to be substance behind such predictions if they are going
to be used as justification for slowing things down in the mean time.
- FChE
10:32 a.m.
On 04. 01. 23 17:29, Jonathan Wakely wrote:
On Tue, 3 Jan 2023 at 09:39, Miro Hrončok <mhroncok(a)redhat.com>
wrote:
>
> = New business =
>
> #2923 Re-vote for Change proposal: Add -fno-omit-frame-pointer to default
> compilation flags
> https://pagure.io/fesco/issue/2923
Given the controversial nature of this one, why was it re-litigated at
short notice when a large number of the stakeholders were still on
vacation?
Presumably because it now had majority support in FESCo and waiting extra week
would collie with the mass rebuild.
See https://pagure.io/fesco/issue/2923#comment-833432 and the meting logs
https://meetbot.fedoraproject.org/fedora-meeting/2023-01-03/fesco.2023-01...
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
11:46 a.m.
On 1/5/23 11:08, Frank Ch. Eigler wrote:
> Of course, but the benefit is to fix performance bugs in applications
> or maybe the desktop itself. [...]
>> Let's be firm in testing this empirically rather than aspirationally.
> I really don't know how. Suggestions welcome.
I'd put the onus on the proponents of the Change, who made predictions
like "I'm confident that over the course of the next year, we'll recover
performance that was lost by FORTIFY_SOURCE=3 and frame pointers" and
"The optimizations enabled by profiling can be much larger than 3-10%."
As the one who made this statement: Profiling can result in very large
gains. I cannot predict what the actual gains will be.
There needs to be substance behind such predictions if they are
going
to be used as justification for slowing things down in the mean time.
I agree.
--
Sincerely,
Demi Marie Obenour (she/her/hers)
4:24 a.m.
On Thu, 5 Jan 2023 at 17:47, Demi Marie Obenour <demiobenour(a)gmail.com> wrote:
On 1/5/23 11:08, Frank Ch. Eigler wrote:
>
>> Of course, but the benefit is to fix performance bugs in applications
>> or maybe the desktop itself. [...]
>
>>> Let's be firm in testing this empirically rather than aspirationally.
>> I really don't know how. Suggestions welcome.
>
> I'd put the onus on the proponents of the Change, who made predictions
> like "I'm confident that over the course of the next year, we'll
recover
> performance that was lost by FORTIFY_SOURCE=3 and frame pointers" and
> "The optimizations enabled by profiling can be much larger than 3-10%."
As the one who made this statement: Profiling can result in very large
gains. I cannot predict what the actual gains will be.
But they should be measurable, right? If profiling can't actually
measure performance and track improvements in performance, the change
isn't useful. So it should be possible to show the benefits over the
next release or two.
Aside: could the change proposal please be updated to show *how* to
opt out, not just state it can be done trivially?
I shouldn't have to find
https://src.fedoraproject.org/rpms/redhat-rpm-config/pull-request/231#req...
to know whether the right opt-out is to %undefine the new macro, or to
define it to 0, or something else.
> There needs to be substance behind such predictions if they are going
> to be used as justification for slowing things down in the mean time.
I agree.
--
Sincerely,
Demi Marie Obenour (she/her/hers)
_______________________________________________
devel mailing list -- devel(a)lists.fedoraproject.org
To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
5:50 a.m.
On Fri, Jan 06, 2023 at 10:24:39AM +0000, Jonathan Wakely wrote:
But they should be measurable, right? If profiling can't
actually
measure performance and track improvements in performance, the change
isn't useful. So it should be possible to show the benefits over the
next release or two.
The problem is you're confusing general gains and gains in
specific scenarios.
Perf + flamegraphs are such a useful tool that we managed to double
performance (ie. ~ 100% gain) in one particular network server case
that we investigated a few years ago. This was by spotting that the
kernel was writing to an MSR (hardware register) which was really
slow, and as it wasn't necessary we just got rid of it.
For that one use case - an incredible performance gain! Does this
mean everyone sees their machines double in speed? Of course not.
The thing is that perf + flamegraphs makes your whole system much more
visible and so it's much easier to find these kind of gains in
specific scenarios. Frame pointers need to be enabled across the
whole system for this to work properly[1].
Will we be able to say that "Fedora got N% faster" in two years?
Not at all - it depends entirely what you use Fedora for.
The overhead is also a real thing. There's a few percent overhead
everywhere for enabling frame pointers because every stack frame entry
and exit involves a couple of extra instructions.
Anyway I'd really urge you to play with these tools before judging
this proposal: https://www.brendangregg.com/flamegraphs.html
Aside: could the change proposal please be updated to show *how* to
opt out, not just state it can be done trivially?
While this should be documented, please don't do it just because you
don't like it. It makes profiling worse for everyone.
Rich.
[1] Perf claims to be able to use DWARF info for stack traces, but in
our experience it does not work at all.
I shouldn't have to find
https://src.fedoraproject.org/rpms/redhat-rpm-config/pull-request/231#req...
to know whether the right opt-out is to %undefine the new macro, or to
define it to 0, or something else.
>
> > There needs to be substance behind such predictions if they are going
> > to be used as justification for slowing things down in the mean time.
>
> I agree.
> --
> Sincerely,
> Demi Marie Obenour (she/her/hers)
> _______________________________________________
> devel mailing list -- devel(a)lists.fedoraproject.org
> To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
> Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
> Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
_______________________________________________
devel mailing list -- devel(a)lists.fedoraproject.org
To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
nbdkit - Flexible, fast NBD server with plugins
https://gitlab.com/nbdkit/nbdkit
11:11 a.m.
On 2023-01-06 02:24, Jonathan Wakely wrote:
Aside: could the change proposal please be updated to show *how* to
opt out, not just state it can be done trivially?
I shouldn't have to find
https://src.fedoraproject.org/rpms/redhat-rpm-config/pull-request/231#req...
to know whether the right opt-out is to %undefine the new macro, or to
define it to 0, or something else.
Done, thanks for the suggestion.
Cheers
Davide
7:35 p.m.
Hi -
The thing is that perf + flamegraphs makes your whole system much
more
visible and so it's much easier to find these kind of gains in
specific scenarios.
(There are exist other profiling tools and techniques that do not
require frame pointer recompilation, but whatever.)
Frame pointers need to be enabled across the whole system for this
to
work properly[1]. [...]
If that were true, then the fesco-mandated per-package opt-out option
would defeat this purpose, would it not?
[...]
[1] Perf claims to be able to use DWARF info for stack traces, but in
our experience it does not work at all.
Please share more information about this.
- FChE
11:10 p.m.
Neal Gompa wrote:
On Wed, Jan 4, 2023 at 8:30 AM Vitaly Zaitsev via devel
<devel(a)lists.fedoraproject.org> wrote:
>
> On 03/01/2023 18:42, Miro Hrončok wrote:
> > * AGREED: APPROVED (+6,1,-1) This Change is implemented for Fedora
> > Linux 38 and we evaluate whether to retain it by Fedora Linux 40.
> > This Change must be implemented in a manner which packages are
> > able to trivially opt-out of retaining frame pointers during
> > compilation so that packages that take larger performance hits can
> > easily
> > revert. (mhroncok, 17:20:38)
>
> Already rejected proposal was submitted because big corporations weren't
> happy with the results. This is a VERY BAD precedent for Fedora.
>
Actually, the Change owners were prepared to give up. I was the one
that pushed for it to be reconsidered because of how much benefit it
gives to desktop Linux developers.
The way this was done is so wrong:
* There was a vote. You were not happy with the outcome.
* So you first tried to complain in the original ticket about this. It was
clear that the consensus in that ticket was to not reconsider at this time.
* So instead, you filed up a *new* ticket, with *no* public discussion
(e.g., on this mailing list), and also with *no* link in the original
ticket, thereby excluding participants of the existing discussion that you
lost (who did not get any notification that the decision was being
reconsidered before it was too late and hence no chance to chime in).
* And then you expedited the issue to the next meeting, only 4 days after it
was opened, again precluding any kind of discussion or feedback.
* I also do not see anything that has changed since this was last discussed.
* And as in the last discussion, I still believe that 2 releases, i.e., a
whole year, is way too long an evaluation period. If anything, this needs to
be evaluated in Rawhide only with the option to revert it with a mass
rebuild before feature freeze. It does not make sense to ship pessimized
code to end users for a whole year.
When the Change proposal came in for FORTIFY_SOURCE=3 (which
introduces
similar pressure), the resulting discussion prompted the re-vote.
I do not see how this is a fair comparison because FORTIFY_SOURCE is a
security feature whereas this one is not. Of course, stricter FORTIFY_SOURCE
comes at a performance penalty, but it improves security for end users.
Frame pointers, on the other hand, bring no value whatsoever to end users,
only to developers.
In addition, if you believe the penalty is too high, then the outcome should
have been to reconsider the FORTIFY_SOURCE=3 change instead.
Kevin Kofler
11:32 p.m.
Andrii Nakryiko wrote:
> This is why I think the change is implicitly just for x86_64.
Definitely not intentionally, might be just a bias of what we had most
hands-on experience with.
Well, that is why it is so bad that you forced through this change behind
the toolchain team's back. They are the experts.
Kevin Kofler
11:36 p.m.
PS:
Kevin Kofler via devel wrote:
The way this was done is so wrong:
* There was a vote. You were not happy with the outcome.
* So you first tried to complain in the original ticket about this. It was
clear that the consensus in that ticket was to not reconsider at this
time.
* So instead, you filed up a *new* ticket, with *no* public discussion
(e.g., on this mailing list), and also with *no* link in the original
ticket, thereby excluding participants of the existing discussion that you
lost (who did not get any notification that the decision was being
reconsidered before it was too late and hence no chance to chime in).
* And then you expedited the issue to the next meeting, only 4 days after
it was opened, again precluding any kind of discussion or feedback.
* You apparently also did not even invite the toolchain team, the experts on
this issue, since judging from Florian Weimer's reply here and from Siddhesh
Poyarekar's reply in the ticket, they were caught by surprise by this sudden
complete reversal just as completely as I was.
* I also do not see anything that has changed since this was last
discussed.
* And as in the last discussion, I still believe that 2 releases, i.e., a
whole year, is way too long an evaluation period. If anything, this needs
to be evaluated in Rawhide only with the option to revert it with a mass
rebuild before feature freeze. It does not make sense to ship pessimized
Kevin Kofler
11:56 p.m.
Miro Hrončok wrote:
On 04. 01. 23 17:29, Jonathan Wakely wrote:
> On Tue, 3 Jan 2023 at 09:39, Miro Hrončok <mhroncok(a)redhat.com> wrote:
>>
>> = New business =
>>
>> #2923 Re-vote for Change proposal: Add -fno-omit-frame-pointer to
>> #default
>> compilation flags
>> https://pagure.io/fesco/issue/2923
>
> Given the controversial nature of this one, why was it re-litigated at
> short notice when a large number of the stakeholders were still on
> vacation?
Presumably because it now had majority support in FESCo and waiting extra
week would collie with the mass rebuild.
Still, it gives an extremely bad impression of rushing this through without
giving anybody the chance to object.
I also do not see why this needed to be approved for F38 on such a short
notice and could not wait for F39.
Kevin Kofler
12:01 a.m.
On Sat, Jan 7, 2023 at 12:24 AM Kevin Kofler via devel
<devel(a)lists.fedoraproject.org> wrote:
Neal Gompa wrote:
> For what it's worth, frame pointers are required on other operating
> systems for precisely this reason
What operating systems REQUIRE frame pointers? GCC supports
-fomit-frame-pointer basically everywhere.
GCC is not the official compiler on Windows or macOS. Both platforms
require frame pointers on all supported architectures with their
official compilers (MSVC for Windows, Clang for macOS).
--
真実はいつも一つ！/ Always, there's only one truth!
12:24 a.m.
Richard W.M. Jones wrote:
The problem is you're confusing general gains and gains in
specific scenarios.
But the thing is that a gain in some specific scenario is a lot less useful
than a general gain. And the latter is usually not had through profiling,
but through improvements in toolchain optimizations. -fomit-frame-pointer
was one such improvement that you have now successfully destroyed for all
Fedora users.
Perf + flamegraphs are such a useful tool that we managed to double
performance (ie. ~ 100% gain) in one particular network server case
that we investigated a few years ago. This was by spotting that the
kernel was writing to an MSR (hardware register) which was really
slow, and as it wasn't necessary we just got rid of it.
For that one use case - an incredible performance gain! Does this
mean everyone sees their machines double in speed? Of course not.
And that is why that improvement is much less impressive than it sounds at
first. Chances are it helps only a handful users, in a handful situations,
and even for those users, the overall improvement is not going to be 100%
because they will also be using other software than the one you profiled and
optimized.
Will we be able to say that "Fedora got N% faster" in two
years?
Not at all - it depends entirely what you use Fedora for.
Hence this makes the claims made by the change proponents entirely
unrealistic and impossible to ever verify. We are hitting the end users with
an overall performance penalty in exchange of potential performance
improvements that are impossible not only to predict, but even to quantify
after the fact, i.e., the claim that the latter will more than compensate
for the former is completely unsubstantiated.
The overhead is also a real thing. There's a few percent
overhead
everywhere for enabling frame pointers because every stack frame entry
and exit involves a couple of extra instructions.
Exactly.
Anyway I'd really urge you to play with these tools before
judging
this proposal: https://www.brendangregg.com/flamegraphs.html
KCachegrind, using Valgrind with the Callgrind or Cachegrind tool, gives me
more information than that even without frame pointers, and it is actually
reliable because it dynamically instruments the code and traces every single
instruction instead of just taking random samples and hoping it did not miss
anything important. It is also much more reproducible because it uses a
mathematical model for the CPU cycles instead of a wallclock time sample
that depends not only on your particular CPU, but also on things such as
background tasks, thermal throttling, etc. Yes, it is slower (up to a factor
~50), but only for the developer doing the profiling, and as explained
above, the reported cycle counts do not depend on the wallclock time anyway.
Kevin Kofler
12:31 a.m.
Neal Gompa wrote:
GCC is not the official compiler on Windows or macOS. Both platforms
require frame pointers on all supported architectures with their
official compilers (MSVC for Windows, Clang for macOS).
Frame pointers are not required by the operating system if you can compile
working programs without them.
Also, for MSVC, /Oy- is documented to be supported on everything except
"x64" (which, as I understand it, means x86_64):
https://learn.microsoft.com/en-us/cpp/build/reference/oy-frame-pointer-om...
so it requires frame pointers on x86_64 for some reason (SEH support?), but
apparently not on other architectures, or the documentation is wrong.
Kevin Kofler
12:34 a.m.
On Sat, Jan 7, 2023 at 1:31 AM Kevin Kofler via devel
<devel(a)lists.fedoraproject.org> wrote:
Neal Gompa wrote:
> GCC is not the official compiler on Windows or macOS. Both platforms
> require frame pointers on all supported architectures with their
> official compilers (MSVC for Windows, Clang for macOS).
Frame pointers are not required by the operating system if you can compile
working programs without them.
Also, for MSVC, /Oy- is documented to be supported on everything except
"x64" (which, as I understand it, means x86_64):
https://learn.microsoft.com/en-us/cpp/build/reference/oy-frame-pointer-om...
so it requires frame pointers on x86_64 for some reason (SEH support?), but
apparently not on other architectures, or the documentation is wrong.
It's on for AArch64 too (it's also always been on for AArch64 for
Linux too). But yes, it is required on x86_64 for SEH.
--
真実はいつも一つ！/ Always, there's only one truth!
12:34 a.m.
Kevin Kofler via devel wrote:
Also, for MSVC, /Oy- is documented to be supported on everything
except
/Oy actually. /Oy- is the default (= -fno-omit-frame-pointer), /Oy is the
equivalent of -fomit-frame-pointer. But both are documented as unsupported
only for "x64 compilers".
Kevin Kofler
11:06 a.m.
Kevin Kofler via devel wrote:
Still, it gives an extremely bad impression of rushing this through
without giving anybody the chance to object.
I also do not see why this needed to be approved for F38 on such a short
notice and could not wait for F39.
PS: The impression I get is that everything was deliberately rigged so that
the vote would end up the way it did:
1. A new ticket was filed, in order to exclude the participants of the
previous discussion.
2. The people watching the old ticket were NOT notified.
3. The Tools Team was NOT notified.
4. The proponents of the Change, on the other hand, WERE notified.
So, with all of the above, the discussion participants were preselected to
only include people in favor of the change.
5. The ticket was filed in the middle of the holiday season. Many people in
Europe are on vacation until today.
6. There was NO thread about the reopening of the discussion on the mailing
list. The first message that mentioned the issue on the mailing list was
"Schedule for Tuesday's FESCo Meeting (2023-01-03)" from 2023-01-03 09:39
UTC, only 7 hours 21 minutes before the meeting. This is in contrast to the
Change policy requiring at least a week between the mailing list
announcement and opening the FESCo ticket.
7. Only 4 days had elapsed between the (unannounced) opening of the ticket
and the vote. This is clearly insufficient. The one week in the Change
policy that I cited above is designed as a minimum time for discussion.
8. The change was approved only 2 weeks before the mass rebuild, leaving
little to no time to revert it in the contigency case.
So, this ensured that whoever was deliberately NOT invited had no chance to
find out by themselves and intervene before it was too late.
This strikes me as extremely intransparent and undemocratic.
Therefore, I hereby request that the vote be annulled as having happened in
violation of the Change policy.
Kevin Kofler
11:36 a.m.
Kevin Kofler via devel wrote:
7. Only 4 days had elapsed between the (unannounced) opening of the
ticket
and the vote. This is clearly insufficient. The one week in the Change
Sorry, there were actually 6 days. Still, less than a week, and there was no
mailing list announcement. The community was under the impression that the
matter was settled when it was actually not. So I still believe that the
transparency policies were not followed.
Kevin Kofler
12:31 p.m.
Vitaly Zaitsev via devel wrote:
The original proposal received a lot of negative feedback. Only a
few
big corporations will get benefit and end users will get a 3-10%
performance penalty which is absolutely unacceptable.
Not to mention the size impact. My request in the original discussion to
quantify the size impact was completely ignored by the change proponents, so
we have no idea how badly this change will bloat Fedora. This should have
been the first question by FESCo, even before performance!
Kevin Kofler
8:08 p.m.
Also, the pull request implementing the change was merged 6 days BEFORE the
change was approved by FESCo! It is unacceptable to sneak unapproved changes
into Rawhide this way in order to "create facts".
Kevin Kofler
8:11 p.m.
On Sun, Jan 8, 2023 at 9:08 PM Kevin Kofler via devel
<devel(a)lists.fedoraproject.org> wrote:
Also, the pull request implementing the change was merged 6 days BEFORE the
change was approved by FESCo! It is unacceptable to sneak unapproved changes
into Rawhide this way in order to "create facts".
Actually, no it wasn't. Adding knobs is perfectly fine when it doesn't
have a practical impact by default.
The actual change to switch the default wasn't merged until Friday:
https://src.fedoraproject.org/rpms/redhat-rpm-config/pull-request/231
--
真実はいつも一つ！/ Always, there's only one truth!
8:43 p.m.
Kevin Kofler via devel wrote:
PS: The impression I get is that everything was deliberately rigged
so
that the vote would end up the way it did:
1. A new ticket was filed, in order to exclude the participants of the
previous discussion.
2. The people watching the old ticket were NOT notified.
3. The Tools Team was NOT notified.
4. The proponents of the Change, on the other hand, WERE notified.
So, with all of the above, the discussion participants were preselected to
only include people in favor of the change.
5. The ticket was filed in the middle of the holiday season. Many people
in Europe are on vacation until today.
6. There was NO thread about the reopening of the discussion on the
mailing list. The first message that mentioned the issue on the mailing
list was "Schedule for Tuesday's FESCo Meeting (2023-01-03)" from
2023-01-03 09:39 UTC, only 7 hours 21 minutes before the meeting. This is
in contrast to the Change policy requiring at least a week between the
mailing list announcement and opening the FESCo ticket.
7. Only 4 days had elapsed between the (unannounced) opening of the ticket
and the vote. This is clearly insufficient. The one week in the Change
policy that I cited above is designed as a minimum time for discussion.
8. The change was approved only 2 weeks before the mass rebuild, leaving
little to no time to revert it in the contigency case.
So, this ensured that whoever was deliberately NOT invited had no chance
to find out by themselves and intervene before it was too late.
This strikes me as extremely intransparent and undemocratic.
PPS: This is particularly striking when you consider that the same person
who filed the new ticket and excluded one side from the discussion entirely
was the one complaining just a month earlier about the OLD discussion:
Yes, but the other stakeholder I wanted there didn't even know it
was on
the agenda yesterday, which meant we mostly heard only one side (the
toolchain people).
See: https://pagure.io/fesco/issue/2817#comment-830204
Complaining about something and then deliberately doing the same thing the
other way round, way to go!
Kevin Kofler
8:58 p.m.
New subject: Malicious communication (was: Re: Schedule for Tuesday's FESCo
Meeting (2023-01-03))
On Sun, Jan 8, 2023 at 9:44 PM Kevin Kofler via devel
<devel(a)lists.fedoraproject.org> wrote:
Kevin Kofler via devel wrote:
> PS: The impression I get is that everything was deliberately rigged so
> that the vote would end up the way it did:
>
> 1. A new ticket was filed, in order to exclude the participants of the
> previous discussion.
> 2. The people watching the old ticket were NOT notified.
> 3. The Tools Team was NOT notified.
> 4. The proponents of the Change, on the other hand, WERE notified.
>
> So, with all of the above, the discussion participants were preselected to
> only include people in favor of the change.
>
> 5. The ticket was filed in the middle of the holiday season. Many people
> in Europe are on vacation until today.
> 6. There was NO thread about the reopening of the discussion on the
> mailing list. The first message that mentioned the issue on the mailing
> list was "Schedule for Tuesday's FESCo Meeting (2023-01-03)" from
> 2023-01-03 09:39 UTC, only 7 hours 21 minutes before the meeting. This is
> in contrast to the Change policy requiring at least a week between the
> mailing list announcement and opening the FESCo ticket.
> 7. Only 4 days had elapsed between the (unannounced) opening of the ticket
> and the vote. This is clearly insufficient. The one week in the Change
> policy that I cited above is designed as a minimum time for discussion.
> 8. The change was approved only 2 weeks before the mass rebuild, leaving
> little to no time to revert it in the contigency case.
>
> So, this ensured that whoever was deliberately NOT invited had no chance
> to find out by themselves and intervene before it was too late.
>
> This strikes me as extremely intransparent and undemocratic.
PPS: This is particularly striking when you consider that the same person
who filed the new ticket and excluded one side from the discussion entirely
was the one complaining just a month earlier about the OLD discussion:
> Yes, but the other stakeholder I wanted there didn't even know it was on
> the agenda yesterday, which meant we mostly heard only one side (the
> toolchain people).
See: https://pagure.io/fesco/issue/2817#comment-830204
Complaining about something and then deliberately doing the same thing the
other way round, way to go!
You ascribe malice where there is none. The reason I didn't link it in
the original ticket when I made that one was because I didn't think of
it. I don't have some evil plan or something like that as you
describe.
Moreover, your general attitude, tone, and verbiage is unnecessarily
negative. You're also acting like a hypocrite.
I'm extremely tired of how you communicate on this mailing list. I've
been quiet about it for years and years. But people leave Fedora
because of you. Once again, I'm having discussions with people trying
to convince them Fedora isn't a bad place because of you.
You need to sit back and reflect on yourself. Go get some help on
improving your communication or something. If Linus Torvalds can
become a better person and shake off his reputation for tearing people
down, you can change and become a more constructive communicator.
Too many people give you leeway because you're not considered a native
English speaker. You know what? I've met far too many better
communicators who have English as a second, third, or even fourth
language.
There are only three other people in Fedora that give me similar
heartburn, and you should not be in that camp.
Please, for everyone's sake, do something to improve yourself.
--
真実はいつも一つ！/ Always, there's only one truth!
2:24 a.m.
New subject: Malicious communication (was: Re: Schedule for Tuesday's FESCo
Meeting (2023-01-03))
On 09/01/2023 03:58, Neal Gompa wrote:
I'm extremely tired of how you communicate on this mailing list.
I've
been quiet about it for years and years. But people leave Fedora
because of you. Once again, I'm having discussions with people trying
to convince them Fedora isn't a bad place because of you.
Please stop personal attacks on other people. By the way, you do the
same in #fedora-kde.
--
Sincerely,
Vitaly Zaitsev (vitaly(a)easycoding.org)
3:09 a.m.
New subject: Malicious communication (was: Re: Schedule for Tuesday's FESCo
Meeting (2023-01-03))
Neal Gompa wrote on 2023/01/09 11:58:
I'm extremely tired of how you communicate on this mailing list. I've
been quiet about it for years and years. But people leave Fedora
because of you. Once again, I'm having discussions with people trying
to convince them Fedora isn't a bad place because of you.
I don't think this message is appropriate, as other people already said.
Too many people give you leeway because you're not considered a native
English speaker. You know what? I've met far too many better
communicators who have English as a second, third, or even fourth
language.
Honestly saying, I cannot understand this context, why native English
speaker or not is relevant here, especially because I am also
non-native.
I don't think Fedora committers change their attitude against
between native or non-native English speakers.
Even if someone takes uncomfortable attitude against you, you also
have to refrain from attacking the person like this way.
Mamoru
3:56 a.m.
Hi Neal,
On Wed, 2023-01-04 at 08:44 -0500, Neal Gompa wrote:
On Wed, Jan 4, 2023 at 8:30 AM Vitaly Zaitsev via devel
<devel(a)lists.fedoraproject.org> wrote:
>
> Already rejected proposal was submitted because big corporations
> weren't
> happy with the results. This is a VERY BAD precedent for Fedora.
>
Actually, the Change owners were prepared to give up. I was the one
that pushed for it to be reconsidered
I must say I find this rather odd. As you say there was an agreement on
moving forward without frame pointers. And as far as I could see there
was even an healthy discussion about alternative ways to get faster and
more accurate unwinding/backtracing between the profiling and
compiler/tools hackers.
I don't mind if you would re-try to get this change in for f39 or f40
if it turns out those discussions about alternative unwinders didn't
result in faster/better profilers.
But trying to do it while multiple stackholders were away and unaware
of this because it wasn't really announced doesn't feel good.
Cheers,
Mark
4:37 a.m.
New subject: Malicious communication (was: Re: Schedule for Tuesday's FESCo
Meeting (2023-01-03))
On 9/01/2023 10:09, Mamoru TASAKA wrote:
Honestly saying, I cannot understand this context, why native
English
speaker or not is relevant here, especially because I am also
non-native.
I don't think Fedora committers change their attitude against
between native or non-native English speakers.
Mamoru
/Disclaimer: I don't want to take part in this discussion and this is
not me taking a "side"./
I want to clarify that it's often difficult for non-native English
speakers to come across as friendly. It's quite easy to write/speak
English but sounding formal or friendly is a bit more difficult.
Therefore I understand and agree that non-native English speakers are
given a bit of leeway.
--
Arthur Bols
fas/irc: principis
10:47 a.m.
New subject: FESCo revote on "Add -fno-omit-frame-pointer" Change proposal [was
Re: Schedule for Tuesday's FESCo Meeting (2023-01-03)]
On Sun, Jan 08, 2023 at 06:06:47PM +0100, Kevin Kofler via devel wrote:
PS: The impression I get is that everything was deliberately rigged
so that
the vote would end up the way it did:
1. A new ticket was filed, in order to exclude the participants of the
previous discussion.
2. The people watching the old ticket were NOT notified.
3. The Tools Team was NOT notified.
4. The proponents of the Change, on the other hand, WERE notified.
I agree with your earlier post that this did not have enough visibility,
enough notice, or enough time. I was certainly taken by surprise, and I was
trying to keep an eye on this one in particular. (Having the discussion
under "Schedule for Tuesday's FESCo Meeting" didn't help it jump out at
me
either.)
BUT, I do not think it was done with malice, as "deliberately rigged"
implies. I don't see that at all -- I see excitement and interest in moving
forward on something that already has taken a long time, and looming
practical deadlines.
Therefore, I hereby request that the vote be annulled as having
happened
in violation of the Change policy.
So, from a purely "what are the rules?" view, the Change process says:
FESCo will vote to approve or deny a change proposal in accordance with
the FESCo ticket policy.
... and I won't quote all of that, but looking at
https://docs.fedoraproject.org/en-US/fesco/#_ticket_policy...
I don't see any violations, either in the letter or the spirit of what is
written.
And, from a practical point of view, since this passed with six +1 votes,
I'm not sure what benefit canceling and re-voting would really add.
It might be useful to improve both the documented policies. The Changes
policy has nothing about reconsidering Changes in the current cycle that I
can see. (Or, actually, for that matter, for resubmitting in future cycles
either, unless i'm missing something.) And the FESCo ticket policy could
include a) some steps for wider communication, and b) maybe something about
holidays and other times which might need special consideration.
Most crucially, let's please drop the idea that anyone is acting out of malice.
I'm
quite sure that everyone arguing passionately on both sides of this issue
has the best interest of Fedora and of Fedora Linux users in mind. Let's all
keep that framing in mind in the ongoing discussion. Thank you.
--
Matthew Miller
<mattdm(a)fedoraproject.org>
Fedora Project Leader
11:34 a.m.
New subject: FESCo revote on "Add -fno-omit-frame-pointer" Change proposal [was Re: Schedule for Tuesday's FESCo Meeting (2023-01-03)]
Matthew Miller <mattdm(a)fedoraproject.org> writes:
So, from a purely "what are the rules?" view, the Change
process says:
FESCo will vote to approve or deny a change proposal in accordance with
the FESCo ticket policy.
... and I won't quote all of that, but looking at
https://docs.fedoraproject.org/en-US/fesco/#_ticket_policy...
I don't see any violations, either in the letter or the spirit of what is
written.
OTOH:
"The motivation for the Changes process is to raise the visibility of
planned changes and make coordination and planning effort easier. It is
nearly impossible to follow all changes happening in a big project such
as Fedora. By providing a mechanism for sharing changes, it is easier
for contributors to know what is coming and to ensure that we can
address impacts of changes well before the release date. Change
proposals should be shared as early as possible, before the change is
implemented and even in the very early state of the idea, to gather
community feedback and review."
One could argue that taking a change that was widely discussed and
clearly rejected, then initiating a sudden revote without at least as
much publicity, undercuts the "easier for contributors to know what is
coming" or "gather community feedback and review" point of the whole
thing.
- FChE
11:44 a.m.
New subject: FESCo revote on "Add -fno-omit-frame-pointer" Change proposal [was Re: Schedule for Tuesday's FESCo Meeting (2023-01-03)]
Am Montag, 9. Jänner 2023 17:47:54 CET schrieb Matthew Miller:
And, from a practical point of view, since this passed with six +1
votes,
I'm not sure what benefit canceling and re-voting would really add.
Canceling the vote and requiring that the Change Policy be followed would
mean that the change needs to be announced on the mailing list now and that
a FESCo ticket may only be filed no earlier than 7 days from now, i.e.,
2023-01-16. So, it could be voted no earlier than the 2023-01-17 FESCo
meeting. That is right before the mass rebuild, hence this Change must be
pushed back to Fedora 39 at the earliest. So from a practical standpoint,
even if the outcome of the vote were not to change, it would save Fedora 38
from being pessimized by this change.
But I would hope that the discussion could actually convince those FESCo
members that have switched to voting +1 due to the (IMHO unfair) comparison
with the _FORTIFY_SOURCE=3 change to switch back.
Kevin Kofler
11:53 a.m.
New subject: FESCo revote on "Add -fno-omit-frame-pointer" Change proposal
[was Re: Schedule for Tuesday's FESCo Meeting (2023-01-03)]
On 09. 01. 23 17:47, Matthew Miller wrote:
On Sun, Jan 08, 2023 at 06:06:47PM +0100, Kevin Kofler via devel
wrote:
> PS: The impression I get is that everything was deliberately rigged so that
> the vote would end up the way it did:
>
> 1. A new ticket was filed, in order to exclude the participants of the
> previous discussion.
> 2. The people watching the old ticket were NOT notified.
> 3. The Tools Team was NOT notified.
> 4. The proponents of the Change, on the other hand, WERE notified.
I agree with your earlier post that this did not have enough visibility,
enough notice, or enough time. I was certainly taken by surprise, and I was
trying to keep an eye on this one in particular. (Having the discussion
under "Schedule for Tuesday's FESCo Meeting" didn't help it jump out at
me
either.)
BUT, I do not think it was done with malice, as "deliberately rigged"
implies. I don't see that at all -- I see excitement and interest in moving
forward on something that already has taken a long time, and looming
practical deadlines.
> Therefore, I hereby request that the vote be annulled as having happened
> in violation of the Change policy.
So, from a purely "what are the rules?" view, the Change process says:
FESCo will vote to approve or deny a change proposal in accordance with
the FESCo ticket policy.
... and I won't quote all of that, but looking at
https://docs.fedoraproject.org/en-US/fesco/#_ticket_policy...
I don't see any violations, either in the letter or the spirit of what is
written.
And, from a practical point of view, since this passed with six +1 votes,
I'm not sure what benefit canceling and re-voting would really add.
It might be useful to improve both the documented policies. The Changes
policy has nothing about reconsidering Changes in the current cycle that I
can see. (Or, actually, for that matter, for resubmitting in future cycles
either, unless i'm missing something.) And the FESCo ticket policy could
include a) some steps for wider communication, and b) maybe something about
holidays and other times which might need special consideration.
Most crucially, let's please drop the idea that anyone is acting out of malice.
I'm
quite sure that everyone arguing passionately on both sides of this issue
has the best interest of Fedora and of Fedora Linux users in mind. Let's all
keep that framing in mind in the ongoing discussion. Thank you.
I feel like I need to add some words as a chair of the meeting where this was
approved.
First, let me apologize for not suggesting that we should invite all the
stakeholders to the meeting before we voted. In retrospect, this was a bad call
on my part and I sincerely regret it.
When the ticket was opened on Decmeber 28th, when many Fedora contributors were
spending time with their families etc. instead of on Fedora, I raised my
concerns about the date, as I was afraid it might be approved by gaining +3
votes in a week without the remaining FESCo members even knowing about this.
Fortunately, another FESCo member voted -1, hence this option was no longer my
concern. That immediately pushed the ticket to the agenda of the meeting on
January 3rd.
During the meeting, I felt like the proposal is gaining a (close) majority of
the required votes and I decided to conduct the vote because I felt like all
the discussion topics wrt this were already exhausted. I am not sure if this
was a good call, but I am pretty confident that postponing the vote would have
not changed the results.
Considering the mass rebuild is happening really soon, I feel like repeating
the vote later is not helpful, but if people want to, we can surely run the
vote again. However, I am confident the result will be the same.
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
11:54 a.m.
New subject: FESCo revote on "Add -fno-omit-frame-pointer" Change proposal
[was Re: Schedule for Tuesday's FESCo Meeting (2023-01-03)]
* Matthew Miller:
BUT, I do not think it was done with malice, as "deliberately
rigged"
implies. I don't see that at all -- I see excitement and interest in moving
forward on something that already has taken a long time, and looming
practical deadlines.
No one spoke out when the tools team was called untrustworthy on the
FESCo ticket. We can try explain it as an accident that the toolchain
team was sidelined afterwards. But the overall sequence of events
certainly looks rather odd.
There are infrastructure problems as well. Notifications in the Fedora
wiki system are broken (email notifications are spotty, but not
completely defunct; that did not matter here because the new FESCo
ticket was added to the change page only after the second vote), and
missing notifications for label updates on FESCo tickets (so it's hard
to spot that an issue is scheduled for discussion). So unless you are
in the in-group or invited, it's hard to contribute.
All these issues contribute to a work environment that I find very
problematic. The individual aspects are probably not deliberate, but
there's still an impact.
And, from a practical point of view, since this passed with six +1
votes,
I'm not sure what benefit canceling and re-voting would really add.
I'm pretty sure no one considered the impact on non-x86-64
architectures, given that the change as voted and submitted as a PR
would have broken the ppc64le and s390x buildroots.
I think it would save everyone a bit of time if we restricted the change
to x86-64. We do not have much experience with the -mbackchain flag
that was added at the last minute on s390x. The change owners have
stated that they aren't interested in s390x. IBM doesn't want this.
Platform Tools does not want it. I doubt the desktop team does GNOME
performance analysis on s390x on Fedora. I'm not even sure if the tools
support backchain-based unwinding; it's not a frame pointer after all.
Maybe -mbackchain won't cause any issues in after all, but we just don't
have the time to test this before the mass rebuild.
As Jakub and I have repeatedly explained, -fno-omit-frame-pointer on
i686 is known to break certain packages (although I worked around this
in glibc last year), simply because the reduced number of registers
makes it impossible for GCC to compile certain functions with inline
assembly in them. As with s390x, the concrete impact is not known at
this point, and we are out of time for test builds.
Using -mno-omit-leaf-frame-pointer for aarch64 seems to be another
last-minute addition without any clear justification. (On AArch64, the
link register allows one to recover the address of the immediate caller
even if a leaf function does not have a frame pointer. That's not
possible on x86-64, where the caller's address must be read from the
stack, and that has to be based on the frame pointer.) Just because the
compiler option is there to enable doesn't mean it does anything useful
in this context.
Thanks,
Florian
12:41 p.m.
On 1/6/23 20:35, Frank Ch. Eigler wrote:
Hi -
> The thing is that perf + flamegraphs makes your whole system much more
> visible and so it's much easier to find these kind of gains in
> specific scenarios.
(There are exist other profiling tools and techniques that do not
require frame pointer recompilation, but whatever.)
Which ones? I would love for them to exist, and I believe there is
work being done in that direction, but I am not aware of any yet.
--
Sincerely,
Demi Marie Obenour (she/her/hers)
1:10 p.m.
New subject: FESCo revote on "Add -fno-omit-frame-pointer" Change proposal
[was Re: Schedule for Tuesday's FESCo Meeting (2023-01-03)]
On Mon, Jan 09, 2023 at 11:47:54AM -0500, Matthew Miller wrote:
On Sun, Jan 08, 2023 at 06:06:47PM +0100, Kevin Kofler via devel
wrote:
> PS: The impression I get is that everything was deliberately rigged so that
> the vote would end up the way it did:
>
> 1. A new ticket was filed, in order to exclude the participants of the
> previous discussion.
> 2. The people watching the old ticket were NOT notified.
> 3. The Tools Team was NOT notified.
> 4. The proponents of the Change, on the other hand, WERE notified.
I agree with your earlier post that this did not have enough visibility,
enough notice, or enough time. I was certainly taken by surprise, and I was
trying to keep an eye on this one in particular. (Having the discussion
under "Schedule for Tuesday's FESCo Meeting" didn't help it jump out at
me
either.)
Holding a FESCo meeting and vote on the very first working day after
the long xmas / new year holiday is not exactly good timing if you
want contributors to be broadly aware it is happening[1]. I might
humbly suggest that next year, any important meetings that would
naturally fall in the 1st week, be postponed until the 2nd week
of Jan.
With regards,
Daniel
[1] Yes, I'm aware that not every part of the world will shutdown
during the xmas/new year period, but a large enough part of
our contributor base does that its worth taking into account
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
1:17 p.m.
New subject: FESCo revote on "Add -fno-omit-frame-pointer" Change proposal [was Re: Schedule for Tuesday's FESCo Meeting (2023-01-03)]
Am Montag, 9. Jänner 2023 20:10:53 CET schrieb Daniel P. Berrangé:
Holding a FESCo meeting and vote on the very first working day after
the long xmas / new year holiday is not exactly good timing if you
want contributors to be broadly aware it is happening[1]. I might
humbly suggest that next year, any important meetings that would
naturally fall in the 1st week, be postponed until the 2nd week
of Jan.
With regards,
Daniel
[1] Yes, I'm aware that not every part of the world will shutdown
during the xmas/new year period, but a large enough part of
our contributor base does that its worth taking into account
In fact, as I already mentioned, in Austria, this meeting happened *during*
the school holidays and many companies' companywide holidays. E.g., at my
employer, we resumed work today. It is commonplace here to have holidays
until Epiphany, i.e., January 6, or even until the end of the week in which
it falls. And this year, January 6 was a Friday, so even more places were
on vacation until the end of the week.
Kevin Kofler
1:18 p.m.
New subject: FESCo revote on "Add -fno-omit-frame-pointer" Change proposal
[was Re: Schedule for Tuesday's FESCo Meeting (2023-01-03)]
On Mon, Jan 9, 2023 at 2:11 PM Daniel P. Berrangé <berrange(a)redhat.com> wrote:
On Mon, Jan 09, 2023 at 11:47:54AM -0500, Matthew Miller wrote:
> On Sun, Jan 08, 2023 at 06:06:47PM +0100, Kevin Kofler via devel wrote:
> > PS: The impression I get is that everything was deliberately rigged so that
> > the vote would end up the way it did:
> >
> > 1. A new ticket was filed, in order to exclude the participants of the
> > previous discussion.
> > 2. The people watching the old ticket were NOT notified.
> > 3. The Tools Team was NOT notified.
> > 4. The proponents of the Change, on the other hand, WERE notified.
>
> I agree with your earlier post that this did not have enough visibility,
> enough notice, or enough time. I was certainly taken by surprise, and I was
> trying to keep an eye on this one in particular. (Having the discussion
> under "Schedule for Tuesday's FESCo Meeting" didn't help it jump
out at me
> either.)
Holding a FESCo meeting and vote on the very first working day after
the long xmas / new year holiday is not exactly good timing if you
want contributors to be broadly aware it is happening[1]. I might
humbly suggest that next year, any important meetings that would
naturally fall in the 1st week, be postponed until the 2nd week
of Jan.
We should push out the entire schedule one to two weeks then. We keep
losing time in the schedule, and we shouldn't lose even more.
--
真実はいつも一つ！/ Always, there's only one truth!
2:02 p.m.
New subject: FESCo revote on "Add -fno-omit-frame-pointer" Change proposal
[was Re: Schedule for Tuesday's FESCo Meeting (2023-01-03)]
On 1/9/23 11:18, Neal Gompa wrote:
On Mon, Jan 9, 2023 at 2:11 PM Daniel P. Berrangé
<berrange(a)redhat.com> wrote:
>
> On Mon, Jan 09, 2023 at 11:47:54AM -0500, Matthew Miller wrote:
>> On Sun, Jan 08, 2023 at 06:06:47PM +0100, Kevin Kofler via devel wrote:
>>> PS: The impression I get is that everything was deliberately rigged so that
>>> the vote would end up the way it did:
>>>
>>> 1. A new ticket was filed, in order to exclude the participants of the
>>> previous discussion.
>>> 2. The people watching the old ticket were NOT notified.
>>> 3. The Tools Team was NOT notified.
>>> 4. The proponents of the Change, on the other hand, WERE notified.
>>
>> I agree with your earlier post that this did not have enough visibility,
>> enough notice, or enough time. I was certainly taken by surprise, and I was
>> trying to keep an eye on this one in particular. (Having the discussion
>> under "Schedule for Tuesday's FESCo Meeting" didn't help it
jump out at me
>> either.)
>
> Holding a FESCo meeting and vote on the very first working day after
> the long xmas / new year holiday is not exactly good timing if you
> want contributors to be broadly aware it is happening[1]. I might
> humbly suggest that next year, any important meetings that would
> naturally fall in the 1st week, be postponed until the 2nd week
> of Jan.
>
We should push out the entire schedule one to two weeks then. We keep
losing time in the schedule, and we shouldn't lose even more.
I think a good solution would be to move the proposal submission deadlines
a month earlier in the schedule. There's only 3 weeks between the
"Changes Requiring Mass Rebuild" deadline and the mass rebuild. I don't
think
3 weeks is really enough time for FESCO review/approval and also getting the
necessary patches reviewed and committed.
- Tom
--
真実はいつも一つ！/ Always, there's only one truth!
_______________________________________________
devel mailing list -- devel(a)lists.fedoraproject.org
To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
2:27 p.m.
New subject: FESCo revote on "Add -fno-omit-frame-pointer" Change proposal [was Re: Schedule for Tuesday's FESCo Meeting (2023-01-03)]
Am Montag, 9. Jänner 2023 21:02:48 CET schrieb Tom Stellard:
I think a good solution would be to move the proposal submission
deadlines
a month earlier in the schedule. There's only 3 weeks between the
"Changes Requiring Mass Rebuild" deadline and the mass rebuild.
I don't think 3 weeks is really enough time for FESCO review/approval and
also getting the necessary patches reviewed and committed.
How would this have helped in this case? The original change proposal was
actually submitted more than 6 months ago! It is just that it took 5 months
to finally get to a vote (with whose outcome one FESCo member was then
unhappy).
The resubmission, on the other hand, happened one day AFTER the submission
deadline for changes requiring a mass rebuild and hence was already late
under the current process. Pushing the submission deadline earlier would
not have changed that.
If we need a rule, then it needs to be that rejected changes cannot be
resubmitted for reconsideration after the change submission deadline.
Though FESCo could just vote to accept the late change anyway, so it would
not really help if the resubmission comes from FESCo itself and if FESCo
really wants it to happen. At most it could discourage such late
reconsiderations.
Kevin Kofler
3:02 p.m.
New subject: FESCo revote on "Add -fno-omit-frame-pointer" Change proposal
[was Re: Schedule for Tuesday's FESCo Meeting (2023-01-03)]
On 1/9/23 08:47, Matthew Miller wrote:
It might be useful to improve both the documented policies. The Changes
policy has nothing about reconsidering Changes in the current cycle that I
can see. (Or, actually, for that matter, for resubmitting in future cycles
either, unless i'm missing something.) And the FESCo ticket policy could
include a) some steps for wider communication, and b) maybe something about
holidays and other times which might need special consideration.
I would recommend that reconsidering (which I would describe as modifying and revoting)
Changes like this should not be allowed after the Proposal Submission Deadlines.
Making major decisions like this late in the process introduces too much risk (imo).
Looking at this specific change, part of the reason that it was accepted is because
it has a trivial opt-out mechanism, but the decision was made so late in the release
process that there may not be enough time for maintainers to opt-out before the
mass rebuild.
-Tom
3:07 p.m.
New subject: FESCo revote on "Add -fno-omit-frame-pointer" Change proposal
[was Re: Schedule for Tuesday's FESCo Meeting (2023-01-03)]
On 1/9/23 12:27, Kevin Kofler via devel wrote:
Am Montag, 9. Jänner 2023 21:02:48 CET schrieb Tom Stellard:
> I think a good solution would be to move the proposal submission deadlines
> a month earlier in the schedule. There's only 3 weeks between the
> "Changes Requiring Mass Rebuild" deadline and the mass rebuild. I
don't think 3 weeks is really enough time for FESCO review/approval and
> also getting the necessary patches reviewed and committed.
How would this have helped in this case? The original change proposal was actually
submitted more than 6 months ago! It is just that it took 5 months to finally get to a
vote (with whose outcome one FESCo member was then unhappy).
It wouldn't have helped in this case, but if we are discussing
changing the schedule to avoid holiday conflicts, then I think
moving the "Change Proposal Deadlines" earlier would
be better than shifting the whole schedule a few weeks.
The resubmission, on the other hand, happened one day AFTER the
submission deadline for changes requiring a mass rebuild and hence was already late under
the current process. Pushing the submission deadline earlier would not have changed that.
If we need a rule, then it needs to be that rejected changes cannot be resubmitted for
reconsideration after the change submission deadline. Though FESCo could just vote to
accept the late change anyway, so it would not really help if the resubmission comes from
FESCo itself and if FESCo really wants it to happen. At most it could discourage such late
reconsiderations.
Yes, I agree with this and I actually just proposed this up-thread.
-Tom
Kevin Kofler
_______________________________________________
devel mailing list -- devel(a)lists.fedoraproject.org
To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
3:08 p.m.
New subject: FESCo revote on "Add -fno-omit-frame-pointer" Change proposal
[was Re: Schedule for Tuesday's FESCo Meeting (2023-01-03)]
Hi Matthew,
On Mon, Jan 09, 2023 at 11:47:54AM -0500, Matthew Miller wrote:
On Sun, Jan 08, 2023 at 06:06:47PM +0100, Kevin Kofler via devel
wrote:
> Therefore, I hereby request that the vote be annulled as having happened
> in violation of the Change policy.
So, from a purely "what are the rules?" view, the Change process says:
FESCo will vote to approve or deny a change proposal in accordance with
the FESCo ticket policy.
... and I won't quote all of that, but looking at
https://docs.fedoraproject.org/en-US/fesco/#_ticket_policy...
I don't see any violations, either in the letter or the spirit of what is
written.
And, from a practical point of view, since this passed with six +1 votes,
I'm not sure what benefit canceling and re-voting would really add.
It does feel to me as being against the spirit, and only not against
the letter because it is presented as a "revote" on an existing change
proposal instead of proposing a new change proposal (which this really
is IMHO).
Practically people have started preparing for the mass rebuild at the
end of last year since that was when the change checkpoint for change
proposals requiring a mass rebuild was. And at that point the Change
Proposal was already decided to not be included. So it was never
expected that the build flags would suddenly change so drastically
(and some flags are still wrong). Cancelling and backing out these
last minute changes will cause a lot less stress.
Socially I think it will be better for all involved if the policies on
revoting and/or reintroducing a change proposal are first clarified
before allowing a revote. At the moment everybody involved seems hurt
because of the unclear policy. Not having clear rules on the needed
visibility and time needed to discuss this revote/resubmission of the
change proposal caused people to assume the worst about others. Lets
reset and take the time to heal first, so people start actually
talking about real solutions again.
Cheers,
Mark
4:06 p.m.
On Sun, Jan 08, 2023 at 06:06:47PM +0100, Kevin Kofler via devel wrote:
Kevin Kofler via devel wrote:
> Still, it gives an extremely bad impression of rushing this through
> without giving anybody the chance to object.
>
> I also do not see why this needed to be approved for F38 on such a short
> notice and could not wait for F39.
PS: The impression I get is that everything was deliberately rigged so that
the vote would end up the way it did:
You're mixing up two things: a vote being "rigged" with a result you
don't
like. Absolute majority (6 out of 9) of voting members voted in favour.
The rules in Fedora are that FESCo gets to vote on certain things. There is a
time reserved for public discussion, but at some point a vote is scheduled, and
at that point we often discuss things in a meeting and vote one way or another
without futher input from outside.
It does happen occasionally that repeat votes happen, for example a resolution
is approved, but somebody immediately raises some additional concern so it is
amended. At that point we don't seek repeat opinions from all stakeholders.
Things would take forever, and most stakeholders wouldn't change their opinion
anyway for some minor detail.
The changes to this Change proposal were not considered major changes that would
require a repeat discussion on fedora-devel, but were instead a narrowing and
clarification of the proposal, so the vote was held at the earliest convenience.
It is important that *FESCo members* know about the vote, which obviously
they did in this case because absolute majority did vote.
If you were to look at FESCo meeting logs, this happens every once in a while:
a proposal is made, it get's a few +1 votes but not enough and is effectively
rejected, so a different-but-similar proposal is made and the vote repeats.
Sometimes we go through a few of those in one meeting. In this case this was
split over two meetings, but is not substantially different. Discussion was
ongoing all that time, both publicly and privately, and there is nothing that
says that FESCo members must not change their votes based on new information.
The intent of this particular proposal is to learn and adjust based on the
feedback from the initial implementation. The specific flags on different
architectures can and should be adjusted to get the desired result.
An interesting phenomenon is that before the Change was approved, most of the
feedback on fedora-devel was about whether we need the change at all, and how
horrible the performance degradation will be, and what distribution to switch
to. After it was approved, the feedback immediately became more technical, with
suggestions about specific flags and architecture differences.
If we had approved the Change 3 months ago, we would have gotten that useful
part of the feedback much earlier. For me the lesson is that we shouldn't dawdle
on high-stakes controversial votes, but instead approve ambitious proposals
early and have more time to adjust or even revert if it turns out necessary.
Zbyszek
> 1. A new ticket was filed, in order to exclude the participants of the
> previous discussion.
> 2. The people watching the old ticket were NOT notified.
> 3. The Tools Team was NOT notified.
> 4. The proponents of the Change, on the other hand, WERE notified.
>
> So, with all of the above, the discussion participants were preselected to
> only include people in favor of the change.
>
> 5. The ticket was filed in the middle of the holiday season. Many people in
> Europe are on vacation until today.
> 6. There was NO thread about the reopening of the discussion on the mailing
> list. The first message that mentioned the issue on the mailing list was
> "Schedule for Tuesday's FESCo Meeting (2023-01-03)" from 2023-01-03
09:39
> UTC, only 7 hours 21 minutes before the meeting. This is in contrast to the
> Change policy requiring at least a week between the mailing list
> announcement and opening the FESCo ticket.
> 7. Only 4 days had elapsed between the (unannounced) opening of the ticket
> and the vote. This is clearly insufficient. The one week in the Change
> policy that I cited above is designed as a minimum time for discussion.
> 8. The change was approved only 2 weeks before the mass rebuild, leaving
> little to no time to revert it in the contigency case.
>
> So, this ensured that whoever was deliberately NOT invited had no chance to
> find out by themselves and intervene before it was too late.
>
> This strikes me as extremely intransparent and undemocratic.
>
> Therefore, I hereby request that the vote be annulled as having happened in
> violation of the Change policy.
>
> Kevin Kofler
> _______________________________________________
> devel mailing list -- devel(a)lists.fedoraproject.org
> To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
> Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
> Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
4:53 p.m.
Am Montag, 9. Jänner 2023 23:06:17 CET schrieb Zbigniew Jędrzejewski-Szmek:
On Sun, Jan 08, 2023 at 06:06:47PM +0100, Kevin Kofler via devel
wrote:
> Kevin Kofler via devel wrote:
> PS: The impression I get is that everything was deliberately
> rigged so that
> the vote would end up the way it did:
You're mixing up two things: a vote being "rigged" with a result you
don't
like.
No, the result is NOT why I got the impression that the vote was rigged.
The way that result was obtained is. In fact, I had explained exactly that
in the remainder of the message, which you omitted from your quote and
pushed to the bottom of the mail (and even that quotes only half of it) for
some reason.
Absolute majority (6 out of 9) of voting members voted in favour.
As I believe, as a consequence of the incomplete and misleading data that
was provided to them in the one-sided discussion, in particular, the flawed
comparison with the _FORTIFY_SOURCE=3 change. (It might not have mattered
to you personally, but you were the only one who had been in favor of that
change from the beginning anyway.)
The changes to this Change proposal were not considered major
changes that would require a repeat discussion on fedora-devel,
And that was a fatal misjudgement. A crucial point that swayed the vote was
a comparison with another change, the _FORTIFY_SOURCE=3 one, which had not
previously come up in the discussion. Therefore, we had no chance to debunk
the flawed comparison. And flawed it was:
* It neglected that _FORTIFY_SOURCE=3 is a security feature for end users
whereas frame pointers are a developer-only feature.
* It made wrong assumptions about the performance impact of
_FORTIFY_SOURCE=3, which, compared to the already existing (!)
_FORTIFY_SOURCE=2, appears to actually have NO performance impact at all
(!), only compared to no _FORTIFY_SOURCE at all.
* It came with an implied accusation of hypocrisy (double standards)
against the Tools Team which makes no sense when you consider the above
details, and the Tools Team was given no chance to defend themselves. That
matters particularly because that false accusation was used to justify
ignoring the Tools Team's stakeholder opinion on the frame pointer change.
If you were to look at FESCo meeting logs, this happens every once
in
a while: a proposal is made, it get's a few +1 votes but not enough
and is effectively rejected, so a different-but-similar proposal is
made and the vote repeats. Sometimes we go through a few of those in
one meeting. In this case this was split over two meetings, but is not
substantially different.
This is substantially different in that it was publicly communicated that
the change was rejected and then suddenly FESCo changed their mind out of
the blue. That is different from voting on multiple proposal variants
during the same meeting and then communicating the final outcome.
(And by the way, what was ultimately accepted was not really a *different*
proposal, but effectively the same as your proposal that had been voted
down a couple weeks earlier.)
Discussion was ongoing all that time, both publicly and privately,
This was not transparent at all. We all got the impression that the
discussion was closed and the feature conclusively rejected, and had no
idea that there was still a discussion ongoing to which we were not
invited.
and there is nothing that says that FESCo members must not change
their
votes based on new information.
But, as I explained above, said "new information" was misleading or
incorrect, and the stakeholders were not given a chance to prove that.
The intent of this particular proposal is to learn and adjust based
on the
feedback from the initial implementation. The specific flags on different
architectures can and should be adjusted to get the desired result.
That is effectively treating stable Fedora releases and their users as
guinea pigs. Especially since you want to wait 2 whole release cycles
before even considering a revert (and there is already the expectation from
the Change owners that a revert will have the burden of proof reversed in
its disfavor, which I consider unacceptable, but which was neither
confirmed nor denied by FESCo – that is not what I would consider a
provisional acceptance of the change).
I really do not see why gathering data cannot be done in a side tag and/or
as a Fedora Remix. Experiments belongs into an experimental branch, not
into a stable release.
An interesting phenomenon is that before the Change was approved,
most
of the feedback on fedora-devel was about whether we need the change
at all, and how horrible the performance degradation will be, and what
distribution to switch to. After it was approved, the feedback
immediately became more technical, with suggestions about specific
flags and architecture differences.
That is because the people have apparently resignated to accept that Fedora
has decided to become unusable and are already looking for another
distribution. I know I am.
If we had approved the Change 3 months ago, we would have gotten
that
useful part of the feedback much earlier. For me the lesson is that we
shouldn't dawdle on high-stakes controversial votes, but instead approve
ambitious proposals early and have more time to adjust or even revert if
it turns out necessary.
That is also the very wrong lesson to get out of this. FESCo needs to be
more willing to actually REJECT changes that make Fedora worse. (And with
that I mean, STICK to the rejection, of course!) I am fed up of seeing
every single such detrimental change getting approved by FESCo despite all
the warnings and negative feedback from the mailing list. Often with
disastrous consequences, e.g., just see how badly mandatory Modularity
(with the intent to move some packages to module-only) has failed, for
example. The failure modes were exactly the ones I and others had warned
about from the onset. We were ignored and the change approved anyway.
Unsurprisingly, it failed, and Modularity had to be made optional and
module-only packages banned to fix the chaos. And now, for once, FESCo
finally had the courage to reject an unfixably flawed change ("unfixably"
because the performance loss is not going to go away no matter how you
reword the change, rejecting it is the only reasonable thing you can do to
it), and what happens? The decision suddenly gets reversed in a mid-holiday
commando action. What the heck?!
Why is it so bad to just say no when you have to? Why can we not accept
that some changes are just flawed by design and just cannot be fixed? Why
is an acceptance always treated as a success and a rejection as a failure?
The default reaction to a change needs to be to reject it. What has worked
for decades should only be changed if there is a really strong reason to.
Rejecting a change is always safer than accepting it.
Kevin Kofler
5:44 p.m.
New subject: FESCo revote on "Add -fno-omit-frame-pointer" Change proposal [was Re: Schedule for Tuesday's FESCo Meeting (2023-01-03)]
Miro Hrončok wrote:
Considering the mass rebuild is happening really soon, I feel like
repeating the vote later is not helpful, but if people want to, we can
surely run the vote again. However, I am confident the result will be the
same.
With all this talk about the mass rebuild being imminent, why can the change
not be pushed back to Fedora 39, to have more time for discussion, for
evaluating the impact in (post-branching) Rawhide (with almost 6 months time
to try out things before the next mass rebuild), and to have a realistic
possibility of reverting it BEFORE it reaches end users of stable releases
(if the impact is as bad as I fear)?
Kevin Kofler
6:12 p.m.
Demi Marie Obenour wrote:
On 1/6/23 20:35, Frank Ch. Eigler wrote:
> (There are exist other profiling tools and techniques that do not
> require frame pointer recompilation, but whatever.)
Which ones? I would love for them to exist, and I believe there is
work being done in that direction, but I am not aware of any yet.
For speed:
https://valgrind.org/info/tools.html#cachegrind
or
https://valgrind.org/info/tools.html#callgrind
with (in both cases)
https://apps.kde.org/kcachegrind/
For memory (RAM) usage:
https://valgrind.org/info/tools.html#massif
with
https://apps.kde.org/massif-visualizer/
Kevin Kofler
6:51 p.m.
On Mon, Jan 9, 2023 at 5:53 PM Kevin Kofler via devel
<devel(a)lists.fedoraproject.org> wrote:
* It made wrong assumptions about the performance impact of
_FORTIFY_SOURCE=3, which, compared to the already existing (!)
_FORTIFY_SOURCE=2, appears to actually have NO performance impact at all
(!), only compared to no _FORTIFY_SOURCE at all.
Your larger point is correct (and what I've been talking about too)
but for the sake of accuracy: there is no known overhead due to
_FORTIFY_SOURCE=2 either AFAIK; there's one publicly available
analysis[1] that shows a minor speedup (!) in ffmpeg due to
_FORTIFY_SOURCE=2. The ~1.3% overhead I mentioned was for
"-fstack-protector-strong -fstack-clash-protection
-D_FORTIFY_SOURCE={2,3} -D_GLIBCBXX_ASSERTIONS" and IMO it should
mainly be attributed to -fstack-protector-strong
-fstack-clash-protection and perhaps -D_GLIBCXX_ASSERTIONS.
Thanks,
Sid
[1] https://zatoichi-engineer.github.io/2017/10/06/fortify-source.html
1:25 a.m.
On Mon, Jan 09, 2023 at 11:53:17PM +0100, Kevin Kofler via devel wrote:
Am Montag, 9. Jänner 2023 23:06:17 CET schrieb Zbigniew
Jędrzejewski-Szmek:
> On Sun, Jan 08, 2023 at 06:06:47PM +0100, Kevin Kofler via devel wrote:
> > Kevin Kofler via devel wrote:
> > PS: The impression I get is that everything was deliberately rigged
> > so that the vote would end up the way it did:
>
> You're mixing up two things: a vote being "rigged" with a result you
don't
> like.
No, the result is NOT why I got the impression that the vote was rigged. The
way that result was obtained is.
Exactly, you're just confirming what I wrote above.
A "vote being rigged" means that either the people who should be allowed to
vote
couldn't, or that people who are not allowed to vote did, or that voters were
tricked or forced to vote differently, or that votes were miscounted.
None of this happened in this case. The fact that the absolute majority of FESCo
cast a vote makes the considerations simpler, because we know that the two votes
that were not cast would not have changed the result.
In fact, I had explained exactly that in
the remainder of the message, which you omitted from your quote and pushed
to the bottom of the mail (and even that quotes only half of it) for some
reason.
You message very verbosely explains why you think the result should be different.
That is not "rigged". That is other people deciding differently than you.
Zbyszek
7:30 a.m.
On Tue, Jan 10, 2023 at 3:05 AM Zbigniew Jędrzejewski-Szmek
<zbyszek(a)in.waw.pl> wrote:
Exactly, you're just confirming what I wrote above.
A "vote being rigged" means that either the people who should be allowed to
vote
couldn't, or that people who are not allowed to vote did, or that voters were
tricked or forced to vote differently, or that votes were miscounted.
What's being alleged is that many members were tricked into changing
their vote by using the false narrative about _FORTIFY_SOURCE proposal
getting an unfair pass despite performance concerns (which *I*
hypothesized months ago and I later dispelled, in the end even quoting
benchmark results for it) and creating the impression that the
toolchain team is being duplicitous about the performance question.
Further trickery involved rushing the vote, claiming that it had to be
done soon to meet the mass rebuild deadline; too bad if those who had
strong objections earlier weren't around to put their comments on
record.
In that context the vote could in fact be considered rigged. If the
voting members could come out and clarify exactly why they changed
their vote, it would make things a lot clearer.
Sid
7:51 a.m.
Am Dienstag, 10. Jänner 2023 08:25:27 CET schrieb Zbigniew
Jędrzejewski-Szmek:
On Mon, Jan 09, 2023 at 11:53:17PM +0100, Kevin Kofler via devel
wrote:
> No, the result is NOT why I got the impression that the vote
> was rigged. The way that result was obtained is.
Exactly, you're just confirming what I wrote above.
Nonsense! Where am I confirming anything? (Quite the opposite, in fact!)
Why do you keep completely missing the point of what I wrote?
A "vote being rigged" means that either the people who
should
be allowed to vote couldn't, or that people who are not allowed
to vote did, or that voters were tricked or forced to vote
differently, or that votes were miscounted.
None of this happened in this case.
The point of the message you first replied to is that I believe that voters
may have been tricked into voting differently, by inviting only one side of
the discussion and by providing the voters with a misleadingly biased
presentation of the facts (up to baseless allegations of double standards
against the Tools Team, based only on a misunderstanding of performance
measurements).
The fact that the absolute majority of FESCo cast a vote makes
the considerations simpler, because we know that the two votes
that were not cast would not have changed the result.
It would if that were the issue at hand. It is not. The issue is with those
who did vote.
> In fact, I had explained exactly that in
> the remainder of the message, which you omitted from your quote and pushed
> to the bottom of the mail (and even that quotes only half of it) for some
> reason.
You message very verbosely explains why you think the result
should be different.
That is not "rigged". That is other people deciding differently than you.
No. The message very verbosely explains errors in the process, not in the
outcome. Inviting only one side of the discussion is inherently biased. Not
allowing sufficient time for discussion is also a way to silence dissenting
opinions. No matter whether it has been done deliberately or accidentally.
The ultimate outcome is that the vote has happened under unfair
circumstances.
Sure, I also happen to think that the result should be different. For
explanations of that, see one of my mails with clear technical objections
to this change. But that was NOT the matter of the mail to which you are
replying here. That mail was about HOW that result was obtained, pointing
out clear procedural bias that you refuse to acknowledge.
Kevin Kofler
9:11 a.m.
New subject: FESCo revote on "Add -fno-omit-frame-pointer" Change proposal
[was Re: Schedule for Tuesday's FESCo Meeting (2023-01-03)]
On Mon, Jan 09, 2023 at 10:08:11PM +0100, Mark Wielaard wrote:
> ... and I won't quote all of that, but looking at
> https://docs.fedoraproject.org/en-US/fesco/#_ticket_policy...
> I don't see any violations, either in the letter or the spirit of what is
> written.
[...]
It does feel to me as being against the spirit, and only not against
the letter because it is presented as a "revote" on an existing change
proposal instead of proposing a new change proposal (which this really
is IMHO).
To be clear -- I don't think what happened is in conflict with the _FESCo_
policy about tickets (see link above). FESCo does not, as far as I can see,
have any specific policies about how votes related to a Change should be
conducted. And I don't think there is a general "FESCo can never reconsider
decisions!" rule.
But like I said, I _don't_ think this revote was as visible or transparent
as it should have been, and I agree that that doesn't really fit with the
intention of the Changes policy.
Socially I think it will be better for all involved if the policies
on
revoting and/or reintroducing a change proposal are first clarified
before allowing a revote. At the moment everybody involved seems hurt
because of the unclear policy. Not having clear rules on the needed
visibility and time needed to discuss this revote/resubmission of the
change proposal caused people to assume the worst about others. Lets
reset and take the time to heal first, so people start actually
talking about real solutions again.
Yep, I definitely agree with this.
--
Matthew Miller
<mattdm(a)fedoraproject.org>
Fedora Project Leader
10:06 a.m.
New subject: FESCo revote on "Add -fno-omit-frame-pointer" Change proposal
[was Re: Schedule for Tuesday's FESCo Meeting (2023-01-03)]
On 10. 01. 23 0:44, Kevin Kofler via devel wrote:
Miro Hrončok wrote:
> Considering the mass rebuild is happening really soon, I feel like
> repeating the vote later is not helpful, but if people want to, we can
> surely run the vote again. However, I am confident the result will be the
> same.
With all this talk about the mass rebuild being imminent, why can the change
not be pushed back to Fedora 39, to have more time for discussion, for
evaluating the impact in (post-branching) Rawhide (with almost 6 months time
to try out things before the next mass rebuild), and to have a realistic
possibility of reverting it BEFORE it reaches end users of stable releases
(if the impact is as bad as I fear)?
That is certainly a possibility, but it's not helpful to tell that to me
personally. I have not voted for this change, I have abstained, because I was
not sure the promises made by the approved statement are likely to be held.
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
1:53 p.m.
New subject: FESCo revote on "Add -fno-omit-frame-pointer" Change proposal
[was Re: Schedule for Tuesday's FESCo Meeting (2023-01-03)]
On Mon, Jan 9 2023 at 06:54:09 PM +0100, Florian Weimer
<fweimer(a)redhat.com> wrote:
No one spoke out when the tools team was called untrustworthy on the
FESCo ticket.
That was one of my comments. [1] I should have worded that more
carefully, but didn't because I was very frustrated. I apologize. What
I meant here is that I strongly disagree with how the members of the
toolchain team were evaluating the trade-offs in this issue. Of course
I don't believe that you (or any other Fedora contributors) are
generally untrustworthy.
[1] https://pagure.io/fesco/issue/2817#comment-830245
We can try explain it as an accident that the toolchain
team was sidelined afterwards. But the overall sequence of events
certainly looks rather odd.
Sounds like everyone agrees the process here did not work very well.
The FESCo meeting was held on January 3, which is first day back from
holidays for most folks in the US, and probably other countries too. An
hour or two before the revote, I was quite worried that I wouldn't be
able to contact Christian in time to attend the meeting, since it was 9
AM on first day back in his timezone. Contacting opponents of the
change proposal was just not on my radar at all.
Anyway, the good news is there is still two more weeks until the mass
rebuild begins, which is plenty of time for continued discussion and
for FESCo to cancel the change if desired. So I don't think the outcome
here was horribly unfair. I believe FESCo had an appropriate
understanding of the considerations involved at the time of the vote:
pay a performance price, gain ability to debug performance problems.
(In particular, I doubt the _FORTIFY_SOURCE=3 change was really a major
consideration here.)
Michael
1:56 p.m.
On Tue, Jan 10 2023 at 01:12:35 AM +0100, Kevin Kofler via devel
<devel(a)lists.fedoraproject.org> wrote:
For speed:
https://valgrind.org/info/tools.html#cachegrind
or
https://valgrind.org/info/tools.html#callgrind
with (in both cases)
https://apps.kde.org/kcachegrind/
For memory (RAM) usage:
https://valgrind.org/info/tools.html#massif
with
https://apps.kde.org/massif-visualizer/
None of these are acceptable alternatives to sysprof. We need to be
able to profile the entire desktop all at once; that point has been
repeated many times and heavily emphasized throughout this discussion.
So valgrind tools might be great at what they do, but they are not
adequate replacements for sysprof. We also need to be able to profile
applications that are already running, since sometimes we don't know
how an application gets into a bad state that triggers a performance
problem: if you can't initiate profiling once you've noticed the
application running slow, then fixing the problem in impractical.
You've previously indicated that developers should just 'dnf
distro-sync' to an alternative Fedora that has frame pointers, as if
building two alternate versions of Fedora, one for developers and one
for users, is a reasonable thing to do. The cost of this is just too
high. We'd need double build infrastructure.
A 2.5% runtime slowdown won't make Fedora "unusable" like you claim. It
will make us look mildly bad on benchmarks. The cost is clearly well
worth the benefit in my opinion, but it's OK to disagree on this.
Michael
3:30 p.m.
On Tue, Jan 10, 2023 at 08:30:25AM -0500, Siddhesh Poyarekar wrote:
On Tue, Jan 10, 2023 at 3:05 AM Zbigniew Jędrzejewski-Szmek
<zbyszek(a)in.waw.pl> wrote:
> Exactly, you're just confirming what I wrote above.
>
> A "vote being rigged" means that either the people who should be allowed
to vote
> couldn't, or that people who are not allowed to vote did, or that voters were
> tricked or forced to vote differently, or that votes were miscounted.
What's being alleged is that many members were tricked into changing
their vote by using the false narrative about _FORTIFY_SOURCE proposal
getting an unfair pass despite performance concerns (which *I*
hypothesized months ago and I later dispelled, in the end even quoting
benchmark results for it) and creating the impression that the
toolchain team is being duplicitous about the performance question.
Further trickery involved rushing the vote, claiming that it had to be
done soon to meet the mass rebuild deadline; too bad if those who had
strong objections earlier weren't around to put their comments on
record.
That description assumes that FESCo members are preschoolers who are
easy to trick and also need to be reminded who said what every day.
That's certainly not the case. The objections against the proposal
were made very clearly and they certainly weren't forgotten over a few
days. Those objections also didn't *change* over those few days, so
repeating them wouldn't actually change anything.
Speaking for myself, I heard the objections from various sides, and I
think I understand them. In particular I think that the objections from
the compiler team are based on correct evaluation of the effect of the
change. But that evaluation is hyperfocused on benchmark performance and
doesn't look at the needs of the whole ecosystem. I think that the
advantages of the proposal and the gains I hope will be realized outweigh
the drawbacks.
I can't speak for other people, but I assume that they made a similar
evaluation. FWIW, I voted the same both times, but I didn't make up my
mind to vote +1 until relatively late before the vote after I had time
to read up on the topic and go through various options that we have.
Zbyszek
5:13 p.m.
Michael Catanzaro wrote:
You've previously indicated that developers should just 'dnf
distro-sync' to an alternative Fedora that has frame pointers, as if
building two alternate versions of Fedora, one for developers and one
for users, is a reasonable thing to do. The cost of this is just too
high. We'd need double build infrastructure.
+100% CPU power for the Koji build farm is still a lot less overall cost
than +2.5% CPU power for *all* Fedora users in the entire world.
Kevin Kofler
5:14 p.m.
New subject: FESCo revote on "Add -fno-omit-frame-pointer" Change proposal [was Re: Schedule for Tuesday's FESCo Meeting (2023-01-03)]
Michael Catanzaro wrote:
(In particular, I doubt the _FORTIFY_SOURCE=3 change was really a
major
consideration here.)
What was, then? That was literally the only thing that has changed between
the two diametrically different votes.
Kevin Kofler
5:22 p.m.
Zbigniew Jędrzejewski-Szmek wrote:
That description assumes that FESCo members are preschoolers who are
easy to trick and also need to be reminded who said what every day.
That's certainly not the case. The objections against the proposal
were made very clearly and they certainly weren't forgotten over a few
days. Those objections also didn't *change* over those few days, so
repeating them wouldn't actually change anything.
The objections did change because the arguments brought up by the other side
did. I already pointed that out once. I do not want to sound like a broken
record, so instead of a copy&paste, here is the link:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.o...
Kevin Kofler
6:16 p.m.
New subject: FESCo revote on "Add -fno-omit-frame-pointer" Change proposal
[was Re: Schedule for Tuesday's FESCo Meeting (2023-01-03)]
On Wed, Jan 11, 2023 at 12:15 AM Kevin Kofler via devel
<devel(a)lists.fedoraproject.org> wrote:
Michael Catanzaro wrote:
> (In particular, I doubt the _FORTIFY_SOURCE=3 change was really a major
> consideration here.)
What was, then? That was literally the only thing that has changed between
the two diametrically different votes.
Speaking for myself, the only way in which the _FORTIFY_SOURCE change
impacted my opinion on -fno-omit-frame-pointers is that it made me
think about it again, and that the level of scrutiny myself - and
other members of FESCo - had given that change, had been
disproportionate. And you might like it or not, I had just changed my
mind on the topic since we had the first vote.
At that point, the only thing that could have convinced me to *not* to
vote for enabling frame pointers would have been substantial arguments
*against* the change from the toolchain team (and by substantial I
mean something more than "the team is against it", "we don't like
it",
or "it's going to have a small performance impact"). Given that there
had been *months* for people to speak up, the likelihood of any such
arguments materializing at the last minute is pretty small (and even
then, the change could have been reverted between approval and mass
rebuild, if necessary).
Fabio
8:15 p.m.
On Tue, Jan 10, 2023 at 4:31 PM Zbigniew Jędrzejewski-Szmek
<zbyszek(a)in.waw.pl> wrote:
That description assumes that FESCo members are preschoolers who are
easy to trick and also need to be reminded who said what every day.
That's certainly not the case. The objections against the proposal
were made very clearly and they certainly weren't forgotten over a few
days. Those objections also didn't *change* over those few days, so
repeating them wouldn't actually change anything.
They don't need to be preschoolers; it's not that hard to manufacture
an opinion among well informed adults, even unintentionally by just
having a lot of conviction about it.
The seeds for the revote were placed in the _FORTIFY_SOURCE=3 change
discussion and throughout the discussion, repeated explanations of why
the proposals are not comparable were ignored, instead of which the
focus seemed to be on driving consensus towards getting a revote on
the frame pointer proposal and trying to paint the tools team's
position as being duplicitous.
In the _FORTIFY_SOURCE=3 ticket one of the FESCo voters (who also
drove the revote FWIW) took a hard negative stand only because they
perceived a double standard on performance, which had, by then,
already been debunked a couple of times in the devel thread. While he
did change his vote to +1 later, he appeared to do so only after other
members voiced their support. If that's not influencing narrative
then I don't know what is.
Multiple other FESCo voters, when voting for the _FORTIFY_SOURCE
proposal, talked about the frame pointer proposal, again clearly
indicating that there is a cross-influence.
Finally, another voting member commented, this time on the re-vote
ticket[1], again indicating that the reason for the revote is the
misdirection in the _FORTIFY_SOURCE proposal discussion.
Christian did make an impassioned plea on the re-vote ticket for the
case of frame pointers and it's perfectly understandable if that was a
turning point for those who changed their vote (and please say it if
that was what it was; I'd disagree but that's a different matter then)
but the thing is, that plea needs counterarguments and further
discussion and there was no opportunity for that to happen. Even
then, the only reason why the revote happened at all was because of
the persistent misdirection in the _FORTIFY_SOURCE proposal.
Speaking for myself, I heard the objections from various sides, and
I
think I understand them. In particular I think that the objections from
the compiler team are based on correct evaluation of the effect of the
change. But that evaluation is hyperfocused on benchmark performance and
doesn't look at the needs of the whole ecosystem. I think that the
advantages of the proposal and the gains I hope will be realized outweigh
the drawbacks.
Ack, I respect that even if I don't agree with the conclusion.
Thanks,
Sid
[1] https://pagure.io/fesco/issue/2923#comment-833708
8:26 p.m.
On Tue, Jan 10, 2023 at 09:15:41PM -0500, Siddhesh Poyarekar wrote:
...snip...
Finally, another voting member commented, this time on the re-vote
ticket[1], again indicating that the reason for the revote is the
misdirection in the _FORTIFY_SOURCE proposal discussion.
Just to set the record straight here: I made that comment, and I was
wondering/suggesting (or as I note in the comment "guessing") as to why
the revote was taking place. I was not involved in the revote and
indeed I was trying to say "if this is because of the _FORTIFY_SOURCE proposal,
there's a lot of difference between them and I can't see why it would change
things."
Indeed it didn't change anything for me, as I kept my -1 vote.
kevin
8:43 p.m.
New subject: FESCo revote on "Add -fno-omit-frame-pointer" Change proposal
[was Re: Schedule for Tuesday's FESCo Meeting (2023-01-03)]
On Tue, Jan 10, 2023 at 7:17 PM Fabio Valentini <decathorpe(a)gmail.com> wrote:
Speaking for myself, the only way in which the _FORTIFY_SOURCE
change
impacted my opinion on -fno-omit-frame-pointers is that it made me
think about it again, and that the level of scrutiny myself - and
other members of FESCo - had given that change, had been
disproportionate.
The _FORTIFY_SOURCE change really shouldn't even have prompted a
discussion on the frame pointers change, let alone a reflection
because there is absolutely nothing in common there. Especially in
the context of scrutiny level; one is a security change that has no
broad slowdown and another is a developer experience change that is
shown to have a broad slowdown (characterized by minor or major
depending on which side of the aisle you stand on).
And you might like it or not, I had just changed my
mind on the topic since we had the first vote.
I wish you had spoken up sooner then, using the _FORTIFY_SOURCE change
as the opportunity to speak up not only confused the whole thing, it
also makes it harder to convince that the change of mind came
independently.
Sid
5:16 a.m.
So shouldn't there be some policy for revoting? E.g.:
~~~
If revote is initiated (by somebody?), the revote is going to be
announced on devel(-announce) and can happen as soon as in one week.
~~~
And maybe it should not happen in the ticket, but on the meeting?
Vít
Dne 11. 01. 23 v 3:15 Siddhesh Poyarekar napsal(a):
On Tue, Jan 10, 2023 at 4:31 PM Zbigniew Jędrzejewski-Szmek
<zbyszek(a)in.waw.pl> wrote:
> That description assumes that FESCo members are preschoolers who are
> easy to trick and also need to be reminded who said what every day.
> That's certainly not the case. The objections against the proposal
> were made very clearly and they certainly weren't forgotten over a few
> days. Those objections also didn't *change* over those few days, so
> repeating them wouldn't actually change anything.
They don't need to be preschoolers; it's not that hard to manufacture
an opinion among well informed adults, even unintentionally by just
having a lot of conviction about it.
The seeds for the revote were placed in the _FORTIFY_SOURCE=3 change
discussion and throughout the discussion, repeated explanations of why
the proposals are not comparable were ignored, instead of which the
focus seemed to be on driving consensus towards getting a revote on
the frame pointer proposal and trying to paint the tools team's
position as being duplicitous.
In the _FORTIFY_SOURCE=3 ticket one of the FESCo voters (who also
drove the revote FWIW) took a hard negative stand only because they
perceived a double standard on performance, which had, by then,
already been debunked a couple of times in the devel thread. While he
did change his vote to +1 later, he appeared to do so only after other
members voiced their support. If that's not influencing narrative
then I don't know what is.
Multiple other FESCo voters, when voting for the _FORTIFY_SOURCE
proposal, talked about the frame pointer proposal, again clearly
indicating that there is a cross-influence.
Finally, another voting member commented, this time on the re-vote
ticket[1], again indicating that the reason for the revote is the
misdirection in the _FORTIFY_SOURCE proposal discussion.
Christian did make an impassioned plea on the re-vote ticket for the
case of frame pointers and it's perfectly understandable if that was a
turning point for those who changed their vote (and please say it if
that was what it was; I'd disagree but that's a different matter then)
but the thing is, that plea needs counterarguments and further
discussion and there was no opportunity for that to happen. Even
then, the only reason why the revote happened at all was because of
the persistent misdirection in the _FORTIFY_SOURCE proposal.
> Speaking for myself, I heard the objections from various sides, and I
> think I understand them. In particular I think that the objections from
> the compiler team are based on correct evaluation of the effect of the
> change. But that evaluation is hyperfocused on benchmark performance and
> doesn't look at the needs of the whole ecosystem. I think that the
> advantages of the proposal and the gains I hope will be realized outweigh
> the drawbacks.
Ack, I respect that even if I don't agree with the conclusion.
Thanks,
Sid
[1] https://pagure.io/fesco/issue/2923#comment-833708
_______________________________________________
devel mailing list -- devel(a)lists.fedoraproject.org
To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
5:24 a.m.
Dne 11. 01. 23 v 12:16 Vít Ondruch napsal(a):
So shouldn't there be some policy for revoting? E.g.:
~~~
If revote is initiated (by somebody?), the revote is going to be
announced on devel(-announce) and can happen as soon as in one week.
~~~
Also, I am not quite sure who and how initiated the revote. Maybe the
policy should be about initiating the revote instead. E.g. if I disagree
with some FESCo decision and I think it should be revoted, I would
announce my intention to initiate the revote on devel(-announce).
Vít
And maybe it should not happen in the ticket, but on the meeting?
Vít
Dne 11. 01. 23 v 3:15 Siddhesh Poyarekar napsal(a):
> On Tue, Jan 10, 2023 at 4:31 PM Zbigniew Jędrzejewski-Szmek
> <zbyszek(a)in.waw.pl> wrote:
>> That description assumes that FESCo members are preschoolers who are
>> easy to trick and also need to be reminded who said what every day.
>> That's certainly not the case. The objections against the proposal
>> were made very clearly and they certainly weren't forgotten over a few
>> days. Those objections also didn't *change* over those few days, so
>> repeating them wouldn't actually change anything.
> They don't need to be preschoolers; it's not that hard to manufacture
> an opinion among well informed adults, even unintentionally by just
> having a lot of conviction about it.
>
> The seeds for the revote were placed in the _FORTIFY_SOURCE=3 change
> discussion and throughout the discussion, repeated explanations of why
> the proposals are not comparable were ignored, instead of which the
> focus seemed to be on driving consensus towards getting a revote on
> the frame pointer proposal and trying to paint the tools team's
> position as being duplicitous.
>
> In the _FORTIFY_SOURCE=3 ticket one of the FESCo voters (who also
> drove the revote FWIW) took a hard negative stand only because they
> perceived a double standard on performance, which had, by then,
> already been debunked a couple of times in the devel thread. While he
> did change his vote to +1 later, he appeared to do so only after other
> members voiced their support. If that's not influencing narrative
> then I don't know what is.
>
> Multiple other FESCo voters, when voting for the _FORTIFY_SOURCE
> proposal, talked about the frame pointer proposal, again clearly
> indicating that there is a cross-influence.
>
> Finally, another voting member commented, this time on the re-vote
> ticket[1], again indicating that the reason for the revote is the
> misdirection in the _FORTIFY_SOURCE proposal discussion.
>
> Christian did make an impassioned plea on the re-vote ticket for the
> case of frame pointers and it's perfectly understandable if that was a
> turning point for those who changed their vote (and please say it if
> that was what it was; I'd disagree but that's a different matter then)
> but the thing is, that plea needs counterarguments and further
> discussion and there was no opportunity for that to happen. Even
> then, the only reason why the revote happened at all was because of
> the persistent misdirection in the _FORTIFY_SOURCE proposal.
>
>> Speaking for myself, I heard the objections from various sides, and I
>> think I understand them. In particular I think that the objections from
>> the compiler team are based on correct evaluation of the effect of the
>> change. But that evaluation is hyperfocused on benchmark performance
>> and
>> doesn't look at the needs of the whole ecosystem. I think that the
>> advantages of the proposal and the gains I hope will be realized
>> outweigh
>> the drawbacks.
> Ack, I respect that even if I don't agree with the conclusion.
>
> Thanks,
> Sid
>
> [1] https://pagure.io/fesco/issue/2923#comment-833708
> _______________________________________________
> devel mailing list -- devel(a)lists.fedoraproject.org
> To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
> Do not reply to spam, report it:
> https://pagure.io/fedora-infrastructure/new_issue
6:28 a.m.
On Wed, Jan 11, 2023 at 6:25 AM Vít Ondruch <vondruch(a)redhat.com> wrote:
Dne 11. 01. 23 v 12:16 Vít Ondruch napsal(a):
> So shouldn't there be some policy for revoting? E.g.:
>
>
> ~~~
>
> If revote is initiated (by somebody?), the revote is going to be
> announced on devel(-announce) and can happen as soon as in one week.
>
> ~~~
>
>
Also, I am not quite sure who and how initiated the revote. Maybe the
policy should be about initiating the revote instead. E.g. if I disagree
with some FESCo decision and I think it should be revoted, I would
announce my intention to initiate the revote on devel(-announce).
That came up in yesterday's FESCO meeting, so hopefully at least that
workflow hole will be plugged:
https://meetbot.fedoraproject.org/fedora-meeting/2023-01-10/fesco.2023-01...
Sid
7:48 p.m.
On 1/10/23 18:13, Kevin Kofler via devel wrote:
Michael Catanzaro wrote:
> You've previously indicated that developers should just 'dnf
> distro-sync' to an alternative Fedora that has frame pointers, as if
> building two alternate versions of Fedora, one for developers and one
> for users, is a reasonable thing to do. The cost of this is just too
> high. We'd need double build infrastructure.
+100% CPU power for the Koji build farm is still a lot less overall cost
than +2.5% CPU power for *all* Fedora users in the entire world.
Kevin Kofler
Absolutely correct.
--
Sincerely,
Demi Marie Obenour (she/her/hers)
461
days inactive
472
days old
89 comments
24 participants
participants (24)
-
Andrii Nakryiko -
Arthur Bols -
Daniel P. Berrangé -
Davide Cavalca -
Demi Marie Obenour -
Fabio Valentini -
Florian Weimer -
Frank Ch. Eigler -
Jonathan Wakely -
Kevin Fenzi -
Kevin Kofler -
Mamoru TASAKA -
Mark Wielaard -
Mark Wielaard -
Matthew Miller -
Michael Catanzaro -
Miro Hrončok -
Neal Gompa -
Richard W.M. Jones -
Siddhesh Poyarekar -
Tom Stellard -
Vitaly Zaitsev -
Vít Ondruch -
Zbigniew Jędrzejewski-Szmek