Following is the list of topics that will be discussed in the FESCo meeting Tuesday at 17:00UTC in #fedora-meeting on irc.libera.chat.
To convert UTC to your local time, take a look at http://fedoraproject.org/wiki/UTCHowto
or run: date -d '2023-01-03 17:00 UTC'
Links to all issues to be discussed can be found at: https://pagure.io/fesco/report/meeting_agenda
= Discussed and Voted in the Ticket =
Title of issue https://pagure.io/fesco/issue/### DECISION (+X, Y, -Z)
Change: Strong crypto settings: phase 3, forewarning 2/2 https://pagure.io/fesco/issue/2865 REJECTED (+1, 4, 0)
Non-responsive maintainer: miminar / Michal Minar https://pagure.io/fesco/issue/2891 APPROVED (+4,0,-0)
Change: Perl: Replace versioned MODULE_COMPAT_ requires by RPM dependency generator https://pagure.io/fesco/issue/2898 APPROVED (+2, 0, -0)
Nonresponsive maintainer: Steven Roberts strobert https://pagure.io/fesco/issue/2902 APPROVED (+2, 0, -0)
Change: Golang 1.20 https://pagure.io/fesco/issue/2913 APPROVED (+6, 0, -0)
Change: Fedora Sway Spin https://pagure.io/fesco/issue/2914 APPROVED (+6, 0, -0)
Change: libpinyin 2.8 https://pagure.io/fesco/issue/2915 APPROVED (+6, 0, -0)
Change: GNU Make version 4.4 https://pagure.io/fesco/issue/2916 APPROVED (+5, 0, -0)
Change: Add _FORTIFY_SOURCE=3 to distribution build flags https://pagure.io/fesco/issue/2917 APPROVED: (+5, 0, 0)
Change: Boost 1.81 upgrade https://pagure.io/fesco/issue/2918 APPROVED (+7, 0, -0)
Change: Restore stricter SSH hostkeys permissions https://pagure.io/fesco/issue/2919 APPROVED (+4, 0, -0)
Change: ImageMagick 7 https://pagure.io/fesco/issue/2920 APPROVED (+6, 0, -0)
Change: Fedora Budgie Spin https://pagure.io/fesco/issue/2921 APPROVED (+5, 0, -0)
Change: Use mdadm for BIOS RAID Support in Anaconda https://pagure.io/fesco/issue/2922
= New business =
#2923 Re-vote for Change proposal: Add -fno-omit-frame-pointer to default compilation flags https://pagure.io/fesco/issue/2923
= Open Floor =
For more complete details, please visit each individual issue. The report of the agenda items can be found at https://pagure.io/fesco/report/meeting_agenda
If you would like to add something to this agenda, you can reply to this e-mail, file a new issue at https://pagure.io/fesco, e-mail me directly, or bring it up at the end of the meeting, during the open floor topic. Note that added topics may be deferred until the following meeting.
On 03. 01. 23 10:39, Miro Hrončok wrote:
Title of issue https://pagure.io/fesco/issue/### DECISION (+X, Y, -Z)
There was no actual issue supposed to be listed here instead of the placeholder, it was merely redundant.
On 03. 01. 23 10:39, Miro Hrončok wrote:
Change: Use mdadm for BIOS RAID Support in Anaconda https://pagure.io/fesco/issue/2922
A decision has not yet been made officially for this one, I copy pasted it to the email and forgot to remove it before I sent it. Sorry about that.
On Tue, Jan 03, 2023 at 10:39:13AM +0100, Miro Hrončok wrote:
= Discussed and Voted in the Ticket =
…
Change: Use mdadm for BIOS RAID Support in Anaconda https://pagure.io/fesco/issue/2922
Something went wrong here. The tally is +4,0,0 on this one. The ticket wasn't closed.
Zbyszek
=================================== #fedora-meeting: FESCO (2023-01-03) ===================================
Meeting started by mhroncok at 17:01:53 UTC. The full logs are available at https://meetbot.fedoraproject.org/fedora-meeting/2023-01-03/fesco.2023-01-03... .
Meeting summary --------------- * init process (mhroncok, 17:02:08)
* #2923 Re-vote for Change proposal: Add -fno-omit-frame-pointer to default (mhroncok, 17:06:26) * LINK: https://src.fedoraproject.org/rpms/redhat-rpm-config/pull-request/230 is the implementation (davide, 17:13:47) * AGREED: APPROVED (+6,1,-1) This Change is implemented for Fedora Linux 38 and we evaluate whether to retain it by Fedora Linux 40. This Change must be implemented in a manner which packages are able to trivially opt-out of retaining frame pointers during compilation so that packages that take larger performance hits can easily revert. (mhroncok, 17:20:38) * Change owners please coordinate the change with the Python Maintainers before changing the defaults (mhroncok, 17:21:20)
* Next week's chair (mhroncok, 17:21:46) * ACTION: Conan Kudo will probably chair next meeting (mhroncok, 17:22:17)
* Open Floor (mhroncok, 17:23:10)
* #2907 Exception for spliting OpenJDK build and integration (mhroncok, 17:23:47) * AGREED: No exception is needed (+5,1,-0) (mhroncok, 17:31:20)
* Open Floor (mhroncok, 17:31:52)
Meeting ended at 17:42:06 UTC.
Action Items ------------ * Conan Kudo will probably chair next meeting
Action Items, by person ----------------------- * **UNASSIGNED** * Conan Kudo will probably chair next meeting
People Present (lines said) --------------------------- * mhroncok (66) * zbyszek (27) * Eighth_Doctor (25) * zodbot (20) * hergertme (13) * mhayden (9) * davide (6) * dcantrell (4) * music[m] (2) * salimma (1) * MichaelCatanzaro (1) * nirik (0) * decathorpe (0) * sgallagh (0) * music (0) * Conan_Kudo (0) * Pharaoh_Atem (0) * Son_Goku (0) * King_InuYasha (0) * Sir_Gallantmon (0)
Generated by `MeetBot`_ 0.4
.. _`MeetBot`: https://fedoraproject.org/wiki/Zodbot#Meeting_Functions
On 03/01/2023 18:42, Miro Hrončok wrote:
* AGREED: APPROVED (+6,1,-1) This Change is implemented for Fedora Linux 38 and we evaluate whether to retain it by Fedora Linux 40. This Change must be implemented in a manner which packages are able to trivially opt-out of retaining frame pointers during compilation so that packages that take larger performance hits can easily revert. (mhroncok, 17:20:38)
Already rejected proposal was submitted because big corporations weren't happy with the results. This is a VERY BAD precedent for Fedora.
On Wed, Jan 4, 2023 at 8:30 AM Vitaly Zaitsev via devel devel@lists.fedoraproject.org wrote:
On 03/01/2023 18:42, Miro Hrončok wrote:
- AGREED: APPROVED (+6,1,-1) This Change is implemented for Fedora Linux 38 and we evaluate whether to retain it by Fedora Linux 40. This Change must be implemented in a manner which packages are able to trivially opt-out of retaining frame pointers during compilation so that packages that take larger performance hits can easily revert. (mhroncok, 17:20:38)
Already rejected proposal was submitted because big corporations weren't happy with the results. This is a VERY BAD precedent for Fedora.
Actually, the Change owners were prepared to give up. I was the one that pushed for it to be reconsidered because of how much benefit it gives to desktop Linux developers. The Change owners were prepared to do the work and assist anywhere to restore/improve performance. The GNOME and KDE developers both have tooling that benefits from this Change, and I wanted profiling and introspection capabilities for the desktop (just like other OSes have). When the Change proposal came in for FORTIFY_SOURCE=3 (which introduces similar pressure), the resulting discussion prompted the re-vote.
Most of the users of this feature will be community volunteer developers that need every advantage they can get.
On 04/01/2023 14:44, Neal Gompa wrote:
Actually, the Change owners were prepared to give up. I was the one that pushed for it to be reconsidered because of how much benefit it gives to desktop Linux developers.
The original proposal received a lot of negative feedback. Only a few big corporations will get benefit and end users will get a 3-10% performance penalty which is absolutely unacceptable.
On Wed, Jan 4, 2023 at 8:50 AM Vitaly Zaitsev via devel devel@lists.fedoraproject.org wrote:
On 04/01/2023 14:44, Neal Gompa wrote:
Actually, the Change owners were prepared to give up. I was the one that pushed for it to be reconsidered because of how much benefit it gives to desktop Linux developers.
The original proposal received a lot of negative feedback. Only a few big corporations will get benefit and end users will get a 3-10% performance penalty which is absolutely unacceptable.
I'm not going to argue with you on this again, but suffice it to say multiple GNOME developers have come out and said they need this feature to do performance work.
The item where we take a significant performance hit is Python, and that's going to opt-out: https://src.fedoraproject.org/rpms/python3.11/pull-request/95
For what it's worth, frame pointers are required on other operating systems for precisely this reason. Linux is the odd duck out and it causes us many problems for real-time profiling, performance analysis, and other development tasks.
If you want more performance gains, you need to be able to observe real workloads in real environments and see where the bottlenecks are.
I'm confident that over the course of the next year, we'll recover performance that was lost by FORTIFY_SOURCE=3 and frame pointers as technology improves and we get used to having these by default.
-- 真実はいつも一つ!/ Always, there's only one truth!
On Sat, Jan 7, 2023 at 12:24 AM Kevin Kofler via devel devel@lists.fedoraproject.org wrote:
Neal Gompa wrote:
For what it's worth, frame pointers are required on other operating systems for precisely this reason
What operating systems REQUIRE frame pointers? GCC supports -fomit-frame-pointer basically everywhere.
GCC is not the official compiler on Windows or macOS. Both platforms require frame pointers on all supported architectures with their official compilers (MSVC for Windows, Clang for macOS).
-- 真実はいつも一つ!/ Always, there's only one truth!
Neal Gompa wrote:
GCC is not the official compiler on Windows or macOS. Both platforms require frame pointers on all supported architectures with their official compilers (MSVC for Windows, Clang for macOS).
Frame pointers are not required by the operating system if you can compile working programs without them.
Also, for MSVC, /Oy- is documented to be supported on everything except "x64" (which, as I understand it, means x86_64): https://learn.microsoft.com/en-us/cpp/build/reference/oy-frame-pointer-omiss... so it requires frame pointers on x86_64 for some reason (SEH support?), but apparently not on other architectures, or the documentation is wrong.
Kevin Kofler
On Sat, Jan 7, 2023 at 1:31 AM Kevin Kofler via devel devel@lists.fedoraproject.org wrote:
Neal Gompa wrote:
GCC is not the official compiler on Windows or macOS. Both platforms require frame pointers on all supported architectures with their official compilers (MSVC for Windows, Clang for macOS).
Frame pointers are not required by the operating system if you can compile working programs without them.
Also, for MSVC, /Oy- is documented to be supported on everything except "x64" (which, as I understand it, means x86_64): https://learn.microsoft.com/en-us/cpp/build/reference/oy-frame-pointer-omiss... so it requires frame pointers on x86_64 for some reason (SEH support?), but apparently not on other architectures, or the documentation is wrong.
It's on for AArch64 too (it's also always been on for AArch64 for Linux too). But yes, it is required on x86_64 for SEH.
-- 真実はいつも一つ!/ Always, there's only one truth!
Kevin Kofler via devel wrote:
Also, for MSVC, /Oy- is documented to be supported on everything except
/Oy actually. /Oy- is the default (= -fno-omit-frame-pointer), /Oy is the equivalent of -fomit-frame-pointer. But both are documented as unsupported only for "x64 compilers".
Kevin Kofler
On Wed, Jan 4 2023 at 02:49:30 PM +0100, Vitaly Zaitsev via devel devel@lists.fedoraproject.org wrote:
. Only a few big corporations will get benefit
Please, stop saying this. It's just not true. All Fedora users will benefit from the performance improvements that we're able to make using sysprof. Right now desktop developers do almost no profiling because it's just too difficult. I don't think I've ever successfully profiled any application *ever*. When I see bug reports that require profiling to solve, I think "too bad" and move on to easier bugs. Having profiling tools that actually work will absolutely help improve performance for our users.
On 1/4/23 10:05, Vitaly Zaitsev via devel wrote:
On 04/01/2023 15:25, Michael Catanzaro wrote:
All Fedora users will benefit from the performance improvements that we're able to make using sysprof.
Maybe. Or maybe not. And the performance penalty is here and now.
The optimizations enabled by profiling can be much larger than 3-10%. To be clear, I would prefer a means of profiling that does not cause a performance penalty for everyone else, but that will take much longer to create.
Michael Catanzaro mcatanzaro@redhat.com writes:
[...] Please, stop saying this. It's just not true. All Fedora users will benefit from the performance improvements that we're able to make using sysprof. [...]
Then perhaps the Change could have had a dead-man-switch built in: unless performance improvements due to this profiling change do not in fact appear by F39 or F40, the Change should be automatically reverted.
- FChE
On Wed, Jan 4 2023 at 03:42:43 PM -0500, Frank Ch. Eigler fche@redhat.com wrote:
Then perhaps the Change could have had a dead-man-switch built in: unless performance improvements due to this profiling change do not in fact appear by F39 or F40, the Change should be automatically reverted.
Question is how to measure that? Is it sufficient to fix one or two performance bugs to call this change a success, or do we need more? Specifically how many? You can expect specific application-level performance improvements probably. Benchmarks probably won't improve.
Michael
Michael Catanzaro mcatanzaro@redhat.com writes:
Then perhaps the Change could have had a dead-man-switch built in: unless performance improvements due to this profiling change do not in fact appear by F39 or F40, the Change should be automatically reverted.
Question is how to measure that? Is it sufficient to fix one or two performance bugs to call this change a success, or do we need more? Specifically how many? [...]
If I understood it correctly, the claim was that enabling this distro-user-penalizing option would make it back in terms of performance optimizations. That it was an investement that would have a positive return.
Let's be firm in testing this empirically rather than aspirationally.
- FChE
On Wed, Jan 4 2023 at 11:10:54 PM -0500, Frank Ch. Eigler fche@redhat.com wrote:
If I understood it correctly, the claim was that enabling this distro-user-penalizing option would make it back in terms of performance optimizations.
Of course, but the benefit is to fix performance bugs in applications or maybe the desktop itself. It's not going to result in general improvements that benefit benchmarks. Benchmarks will surely get worse, not better.
That it was an investement that would have a positive return.
Let's be firm in testing this empirically rather than aspirationally.
I really don't know how. Suggestions welcome.
Michael
Of course, but the benefit is to fix performance bugs in applications or maybe the desktop itself. [...]
Let's be firm in testing this empirically rather than aspirationally.
I really don't know how. Suggestions welcome.
I'd put the onus on the proponents of the Change, who made predictions like "I'm confident that over the course of the next year, we'll recover performance that was lost by FORTIFY_SOURCE=3 and frame pointers" and "The optimizations enabled by profiling can be much larger than 3-10%.", and that's just in the last few days.
There needs to be substance behind such predictions if they are going to be used as justification for slowing things down in the mean time.
- FChE
On 1/5/23 11:08, Frank Ch. Eigler wrote:
Of course, but the benefit is to fix performance bugs in applications or maybe the desktop itself. [...]
Let's be firm in testing this empirically rather than aspirationally.
I really don't know how. Suggestions welcome.
I'd put the onus on the proponents of the Change, who made predictions like "I'm confident that over the course of the next year, we'll recover performance that was lost by FORTIFY_SOURCE=3 and frame pointers" and "The optimizations enabled by profiling can be much larger than 3-10%."
As the one who made this statement: Profiling can result in very large gains. I cannot predict what the actual gains will be.
There needs to be substance behind such predictions if they are going to be used as justification for slowing things down in the mean time.
I agree.
On Thu, 5 Jan 2023 at 17:47, Demi Marie Obenour demiobenour@gmail.com wrote:
On 1/5/23 11:08, Frank Ch. Eigler wrote:
Of course, but the benefit is to fix performance bugs in applications or maybe the desktop itself. [...]
Let's be firm in testing this empirically rather than aspirationally.
I really don't know how. Suggestions welcome.
I'd put the onus on the proponents of the Change, who made predictions like "I'm confident that over the course of the next year, we'll recover performance that was lost by FORTIFY_SOURCE=3 and frame pointers" and "The optimizations enabled by profiling can be much larger than 3-10%."
As the one who made this statement: Profiling can result in very large gains. I cannot predict what the actual gains will be.
But they should be measurable, right? If profiling can't actually measure performance and track improvements in performance, the change isn't useful. So it should be possible to show the benefits over the next release or two.
Aside: could the change proposal please be updated to show *how* to opt out, not just state it can be done trivially?
I shouldn't have to find https://src.fedoraproject.org/rpms/redhat-rpm-config/pull-request/231#reques... to know whether the right opt-out is to %undefine the new macro, or to define it to 0, or something else.
There needs to be substance behind such predictions if they are going to be used as justification for slowing things down in the mean time.
I agree.
Sincerely, Demi Marie Obenour (she/her/hers) _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
On Fri, Jan 06, 2023 at 10:24:39AM +0000, Jonathan Wakely wrote:
But they should be measurable, right? If profiling can't actually measure performance and track improvements in performance, the change isn't useful. So it should be possible to show the benefits over the next release or two.
The problem is you're confusing general gains and gains in specific scenarios.
Perf + flamegraphs are such a useful tool that we managed to double performance (ie. ~ 100% gain) in one particular network server case that we investigated a few years ago. This was by spotting that the kernel was writing to an MSR (hardware register) which was really slow, and as it wasn't necessary we just got rid of it.
For that one use case - an incredible performance gain! Does this mean everyone sees their machines double in speed? Of course not.
The thing is that perf + flamegraphs makes your whole system much more visible and so it's much easier to find these kind of gains in specific scenarios. Frame pointers need to be enabled across the whole system for this to work properly[1].
Will we be able to say that "Fedora got N% faster" in two years? Not at all - it depends entirely what you use Fedora for.
The overhead is also a real thing. There's a few percent overhead everywhere for enabling frame pointers because every stack frame entry and exit involves a couple of extra instructions.
Anyway I'd really urge you to play with these tools before judging this proposal: https://www.brendangregg.com/flamegraphs.html
Aside: could the change proposal please be updated to show *how* to opt out, not just state it can be done trivially?
While this should be documented, please don't do it just because you don't like it. It makes profiling worse for everyone.
Rich.
[1] Perf claims to be able to use DWARF info for stack traces, but in our experience it does not work at all.
I shouldn't have to find https://src.fedoraproject.org/rpms/redhat-rpm-config/pull-request/231#reques... to know whether the right opt-out is to %undefine the new macro, or to define it to 0, or something else.
There needs to be substance behind such predictions if they are going to be used as justification for slowing things down in the mean time.
I agree.
Sincerely, Demi Marie Obenour (she/her/hers) _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Hi -
The thing is that perf + flamegraphs makes your whole system much more visible and so it's much easier to find these kind of gains in specific scenarios.
(There are exist other profiling tools and techniques that do not require frame pointer recompilation, but whatever.)
Frame pointers need to be enabled across the whole system for this to work properly[1]. [...]
If that were true, then the fesco-mandated per-package opt-out option would defeat this purpose, would it not?
[...] [1] Perf claims to be able to use DWARF info for stack traces, but in our experience it does not work at all.
Please share more information about this.
- FChE
On 1/6/23 20:35, Frank Ch. Eigler wrote:
Hi -
The thing is that perf + flamegraphs makes your whole system much more visible and so it's much easier to find these kind of gains in specific scenarios.
(There are exist other profiling tools and techniques that do not require frame pointer recompilation, but whatever.)
Which ones? I would love for them to exist, and I believe there is work being done in that direction, but I am not aware of any yet.
Demi Marie Obenour wrote:
On 1/6/23 20:35, Frank Ch. Eigler wrote:
(There are exist other profiling tools and techniques that do not require frame pointer recompilation, but whatever.)
Which ones? I would love for them to exist, and I believe there is work being done in that direction, but I am not aware of any yet.
For speed: https://valgrind.org/info/tools.html#cachegrind or https://valgrind.org/info/tools.html#callgrind with (in both cases) https://apps.kde.org/kcachegrind/
For memory (RAM) usage: https://valgrind.org/info/tools.html#massif with https://apps.kde.org/massif-visualizer/
Kevin Kofler
On Tue, Jan 10 2023 at 01:12:35 AM +0100, Kevin Kofler via devel devel@lists.fedoraproject.org wrote:
For speed: https://valgrind.org/info/tools.html#cachegrind or https://valgrind.org/info/tools.html#callgrind with (in both cases) https://apps.kde.org/kcachegrind/
For memory (RAM) usage: https://valgrind.org/info/tools.html#massif with https://apps.kde.org/massif-visualizer/
None of these are acceptable alternatives to sysprof. We need to be able to profile the entire desktop all at once; that point has been repeated many times and heavily emphasized throughout this discussion. So valgrind tools might be great at what they do, but they are not adequate replacements for sysprof. We also need to be able to profile applications that are already running, since sometimes we don't know how an application gets into a bad state that triggers a performance problem: if you can't initiate profiling once you've noticed the application running slow, then fixing the problem in impractical.
You've previously indicated that developers should just 'dnf distro-sync' to an alternative Fedora that has frame pointers, as if building two alternate versions of Fedora, one for developers and one for users, is a reasonable thing to do. The cost of this is just too high. We'd need double build infrastructure.
A 2.5% runtime slowdown won't make Fedora "unusable" like you claim. It will make us look mildly bad on benchmarks. The cost is clearly well worth the benefit in my opinion, but it's OK to disagree on this.
Michael
Michael Catanzaro wrote:
You've previously indicated that developers should just 'dnf distro-sync' to an alternative Fedora that has frame pointers, as if building two alternate versions of Fedora, one for developers and one for users, is a reasonable thing to do. The cost of this is just too high. We'd need double build infrastructure.
+100% CPU power for the Koji build farm is still a lot less overall cost than +2.5% CPU power for *all* Fedora users in the entire world.
Kevin Kofler
On 1/10/23 18:13, Kevin Kofler via devel wrote:
Michael Catanzaro wrote:
You've previously indicated that developers should just 'dnf distro-sync' to an alternative Fedora that has frame pointers, as if building two alternate versions of Fedora, one for developers and one for users, is a reasonable thing to do. The cost of this is just too high. We'd need double build infrastructure.
+100% CPU power for the Koji build farm is still a lot less overall cost than +2.5% CPU power for *all* Fedora users in the entire world.
Kevin Kofler
Absolutely correct.
Richard W.M. Jones wrote:
The problem is you're confusing general gains and gains in specific scenarios.
But the thing is that a gain in some specific scenario is a lot less useful than a general gain. And the latter is usually not had through profiling, but through improvements in toolchain optimizations. -fomit-frame-pointer was one such improvement that you have now successfully destroyed for all Fedora users.
Perf + flamegraphs are such a useful tool that we managed to double performance (ie. ~ 100% gain) in one particular network server case that we investigated a few years ago. This was by spotting that the kernel was writing to an MSR (hardware register) which was really slow, and as it wasn't necessary we just got rid of it.
For that one use case - an incredible performance gain! Does this mean everyone sees their machines double in speed? Of course not.
And that is why that improvement is much less impressive than it sounds at first. Chances are it helps only a handful users, in a handful situations, and even for those users, the overall improvement is not going to be 100% because they will also be using other software than the one you profiled and optimized.
Will we be able to say that "Fedora got N% faster" in two years? Not at all - it depends entirely what you use Fedora for.
Hence this makes the claims made by the change proponents entirely unrealistic and impossible to ever verify. We are hitting the end users with an overall performance penalty in exchange of potential performance improvements that are impossible not only to predict, but even to quantify after the fact, i.e., the claim that the latter will more than compensate for the former is completely unsubstantiated.
The overhead is also a real thing. There's a few percent overhead everywhere for enabling frame pointers because every stack frame entry and exit involves a couple of extra instructions.
Exactly.
Anyway I'd really urge you to play with these tools before judging this proposal: https://www.brendangregg.com/flamegraphs.html
KCachegrind, using Valgrind with the Callgrind or Cachegrind tool, gives me more information than that even without frame pointers, and it is actually reliable because it dynamically instruments the code and traces every single instruction instead of just taking random samples and hoping it did not miss anything important. It is also much more reproducible because it uses a mathematical model for the CPU cycles instead of a wallclock time sample that depends not only on your particular CPU, but also on things such as background tasks, thermal throttling, etc. Yes, it is slower (up to a factor ~50), but only for the developer doing the profiling, and as explained above, the reported cycle counts do not depend on the wallclock time anyway.
Kevin Kofler
On 2023-01-06 02:24, Jonathan Wakely wrote:
Aside: could the change proposal please be updated to show *how* to opt out, not just state it can be done trivially?
I shouldn't have to find https://src.fedoraproject.org/rpms/redhat-rpm-config/pull-request/231#reques... to know whether the right opt-out is to %undefine the new macro, or to define it to 0, or something else.
Done, thanks for the suggestion.
Cheers Davide
Vitaly Zaitsev via devel wrote:
The original proposal received a lot of negative feedback. Only a few big corporations will get benefit and end users will get a 3-10% performance penalty which is absolutely unacceptable.
Not to mention the size impact. My request in the original discussion to quantify the size impact was completely ignored by the change proponents, so we have no idea how badly this change will bloat Fedora. This should have been the first question by FESCo, even before performance!
Kevin Kofler
Neal Gompa wrote:
On Wed, Jan 4, 2023 at 8:30 AM Vitaly Zaitsev via devel devel@lists.fedoraproject.org wrote:
On 03/01/2023 18:42, Miro Hrončok wrote:
- AGREED: APPROVED (+6,1,-1) This Change is implemented for Fedora Linux 38 and we evaluate whether to retain it by Fedora Linux 40. This Change must be implemented in a manner which packages are able to trivially opt-out of retaining frame pointers during compilation so that packages that take larger performance hits can easily revert. (mhroncok, 17:20:38)
Already rejected proposal was submitted because big corporations weren't happy with the results. This is a VERY BAD precedent for Fedora.
Actually, the Change owners were prepared to give up. I was the one that pushed for it to be reconsidered because of how much benefit it gives to desktop Linux developers.
The way this was done is so wrong: * There was a vote. You were not happy with the outcome. * So you first tried to complain in the original ticket about this. It was clear that the consensus in that ticket was to not reconsider at this time. * So instead, you filed up a *new* ticket, with *no* public discussion (e.g., on this mailing list), and also with *no* link in the original ticket, thereby excluding participants of the existing discussion that you lost (who did not get any notification that the decision was being reconsidered before it was too late and hence no chance to chime in). * And then you expedited the issue to the next meeting, only 4 days after it was opened, again precluding any kind of discussion or feedback. * I also do not see anything that has changed since this was last discussed. * And as in the last discussion, I still believe that 2 releases, i.e., a whole year, is way too long an evaluation period. If anything, this needs to be evaluated in Rawhide only with the option to revert it with a mass rebuild before feature freeze. It does not make sense to ship pessimized code to end users for a whole year.
When the Change proposal came in for FORTIFY_SOURCE=3 (which introduces similar pressure), the resulting discussion prompted the re-vote.
I do not see how this is a fair comparison because FORTIFY_SOURCE is a security feature whereas this one is not. Of course, stricter FORTIFY_SOURCE comes at a performance penalty, but it improves security for end users. Frame pointers, on the other hand, bring no value whatsoever to end users, only to developers.
In addition, if you believe the penalty is too high, then the outcome should have been to reconsider the FORTIFY_SOURCE=3 change instead.
Kevin Kofler
PS:
Kevin Kofler via devel wrote:
The way this was done is so wrong:
- There was a vote. You were not happy with the outcome.
- So you first tried to complain in the original ticket about this. It was
clear that the consensus in that ticket was to not reconsider at this time.
- So instead, you filed up a *new* ticket, with *no* public discussion
(e.g., on this mailing list), and also with *no* link in the original ticket, thereby excluding participants of the existing discussion that you lost (who did not get any notification that the decision was being reconsidered before it was too late and hence no chance to chime in).
- And then you expedited the issue to the next meeting, only 4 days after
it was opened, again precluding any kind of discussion or feedback.
* You apparently also did not even invite the toolchain team, the experts on this issue, since judging from Florian Weimer's reply here and from Siddhesh Poyarekar's reply in the ticket, they were caught by surprise by this sudden complete reversal just as completely as I was.
- I also do not see anything that has changed since this was last
discussed.
- And as in the last discussion, I still believe that 2 releases, i.e., a
whole year, is way too long an evaluation period. If anything, this needs to be evaluated in Rawhide only with the option to revert it with a mass rebuild before feature freeze. It does not make sense to ship pessimized
Kevin Kofler
Hi Neal,
On Wed, 2023-01-04 at 08:44 -0500, Neal Gompa wrote:
On Wed, Jan 4, 2023 at 8:30 AM Vitaly Zaitsev via devel devel@lists.fedoraproject.org wrote:
Already rejected proposal was submitted because big corporations weren't happy with the results. This is a VERY BAD precedent for Fedora.
Actually, the Change owners were prepared to give up. I was the one that pushed for it to be reconsidered
I must say I find this rather odd. As you say there was an agreement on moving forward without frame pointers. And as far as I could see there was even an healthy discussion about alternative ways to get faster and more accurate unwinding/backtracing between the profiling and compiler/tools hackers.
I don't mind if you would re-try to get this change in for f39 or f40 if it turns out those discussions about alternative unwinders didn't result in faster/better profilers.
But trying to do it while multiple stackholders were away and unaware of this because it wasn't really announced doesn't feel good.
Cheers,
Mark
On Wed, Jan 04, 2023 at 02:30:07PM +0100, Vitaly Zaitsev via devel wrote:
On 03/01/2023 18:42, Miro Hrončok wrote:
* AGREED: APPROVED (+6,1,-1) This Change is implemented for Fedora Linux 38 and we evaluate whether to retain it by Fedora Linux 40. This Change must be implemented in a manner which packages are able to trivially opt-out of retaining frame pointers during compilation so that packages that take larger performance hits can easily revert. (mhroncok, 17:20:38)
Already rejected proposal was submitted because big corporations weren't happy with the results. This is a VERY BAD precedent for Fedora.
Not sure who the big bad corps are, but adding frame pointers is a very good idea if you've ever tried using perf.
Rich.
* Miro Hrončok:
- #2923 Re-vote for Change proposal: Add -fno-omit-frame-pointer to default (mhroncok, 17:06:26)
- LINK: https://src.fedoraproject.org/rpms/redhat-rpm-config/pull-request/230 is the implementation (davide, 17:13:47)
- AGREED: APPROVED (+6,1,-1) This Change is implemented for Fedora Linux 38 and we evaluate whether to retain it by Fedora Linux 40. This Change must be implemented in a manner which packages are able to trivially opt-out of retaining frame pointers during compilation so that packages that take larger performance hits can easily revert. (mhroncok, 17:20:38)
- Change owners please coordinate the change with the Python Maintainers before changing the defaults (mhroncok, 17:21:20)
The change as voted simply does not work at a technical level because -mno-omit-leaf-frame-pointer is an architecture-specific GCC option that is not available on all Fedora architectures. I don't think -fno-omit-frame-pointer is well-exercised on s390x, so I wouldn't want to use it there, either.
Is it safe to assume this change (as voted) is actually intended for x86_64 only, in both directions? Specifically, that opting out will *not* disable frame pointers on aarch64 and ppc64le (where it would result in an ABI break)?
We should not ask package maintainers to debug i686 build failures related to inline assembly and register shortage. (On i686, it's impossible to make certain system calls while preserving a frame pointer.) As far as I understand it, building i686 with frame pointers isn't in scope for the change, either, because it does not further its goals.
Regarding leaf functions, on typical workloads, a significant fraction of the profiling samples will be in glibc string functions, which do not have frame pointers. So any profiler has to cope with leaf functions with frame pointers anyway. As a result, do we really need the -mno-omit-leaf-frame-pointer option?
On aarch64, we should not use -mno-omit-leaf-frame-pointer because there is no skipped frame problem with backtraces: the link register (x30) can be read directly even in leaf functions. The ABI only mandates frame pointers for non-leaf functions. On ppc64le, GCC does not even support -mno-omit-leaf-frame-pointer.
This is why I think the change is implicitly just for x86_64.
Does anybody know how this change invalidates Fedora binaries as a reference for DWARF performance work? Do the generated .eh_frame data change materially once frame pointers are in use?
Thanks, Florian
On 2023-01-04 09:28, Florian Weimer wrote:
The change as voted simply does not work at a technical level because -mno-omit-leaf-frame-pointer is an architecture-specific GCC option that is not available on all Fedora architectures. I don't think -fno-omit-frame-pointer is well-exercised on s390x, so I wouldn't want to use it there, either.
The latest implementation update for this in https://src.fedoraproject.org/rpms/redhat-rpm-config/pull-request/231 uses -mbackchain on s390x and drops -mno-omit-leaf-frame-pointer for ppc64le.
Is it safe to assume this change (as voted) is actually intended for x86_64 only, in both directions?
The Change owners are primarily interested in x86_64 and aarch64, but we did not intend this to be x86_64 specific.
Specifically, that opting out will *not* disable frame pointers on aarch64 and ppc64le (where it would result in an ABI break)?
This is correct, and I've updated the documentation to clarify. Opting out will revert to the compiler defaults, which may or may not include frame pointers depending on the architecture.
Cheers Davide
The change as voted simply does not work at a technical level because -mno-omit-leaf-frame-pointer is an architecture-specific GCC option that is not available on all Fedora architectures. I don't think -fno-omit-frame-pointer is well-exercised on s390x, so I wouldn't want to use it there, either.
Is it safe to assume this change (as voted) is actually intended for x86_64 only, in both directions? Specifically, that opting out will *not* disable frame pointers on aarch64 and ppc64le (where it would result in an ABI break)?
I'd say we should just keep the spirit of the proposal in mind: to enable frame pointers on all platforms where it's possible. Change owners are certainly not experts on intricacies of each possible architecture, so it would definitely help to get help and feedback from people like you on what specifically needs to be enabled to make this work across all arches.
Regarding leaf functions, on typical workloads, a significant fraction of the profiling samples will be in glibc string functions, which do not have frame pointers. So any profiler has to cope with leaf functions with frame pointers anyway. As a result, do we really need the -mno-omit-leaf-frame-pointer option?
I think we should be careful about defining "typical workloads" and not assume that it's mostly inside glibc. But even if it is, just because glibcs leaf functions don't have frame pointers doesn't seem to imply that leaf pointers should be generally left out. As I said above, the spirit of the proposal is to make stack traces as accurate as possible. That includes leaf functions, right?
"any profiler has to cope with leaf functions" is true, but what is "cope"? Some more advanced solutions combine LBR stacks with frame pointers and are able to recover full stacks. But the space of stack trace uses (at least with BPF) is vast. There will be BPF-based tools of various degree of sophistication, and I'm 100% sure all of them won't go to the trouble of using LBR to recover the stack. Certainly this won't happen with simple ad-hoc bpftrace scripts.
So that's just to say that I think we should try hard to get leaf frame pointers as well so that stack traces are better in general. For glibc cases, we'll just get stack traces without last level, unless we do something more complicated with LBR.
WDYT?
On aarch64, we should not use -mno-omit-leaf-frame-pointer because there is no skipped frame problem with backtraces: the link register (x30) can be read directly even in leaf functions. The ABI only mandates frame pointers for non-leaf functions. On ppc64le, GCC does not even support -mno-omit-leaf-frame-pointer.
This is why I think the change is implicitly just for x86_64.
Definitely not intentionally, might be just a bias of what we had most hands-on experience with.
Andrii Nakryiko wrote:
This is why I think the change is implicitly just for x86_64.
Definitely not intentionally, might be just a bias of what we had most hands-on experience with.
Well, that is why it is so bad that you forced through this change behind the toolchain team's back. They are the experts.
Kevin Kofler
(once again with a proper Subject line)
=================================== #fedora-meeting: FESCO (2023-01-03) ===================================
Meeting started by mhroncok at 17:01:53 UTC. The full logs are available at https://meetbot.fedoraproject.org/fedora-meeting/2023-01-03/fesco.2023-01-03... .
Meeting summary --------------- * init process (mhroncok, 17:02:08)
* #2923 Re-vote for Change proposal: Add -fno-omit-frame-pointer to default (mhroncok, 17:06:26) * LINK: https://src.fedoraproject.org/rpms/redhat-rpm-config/pull-request/230 is the implementation (davide, 17:13:47) * AGREED: APPROVED (+6,1,-1) This Change is implemented for Fedora Linux 38 and we evaluate whether to retain it by Fedora Linux 40. This Change must be implemented in a manner which packages are able to trivially opt-out of retaining frame pointers during compilation so that packages that take larger performance hits can easily revert. (mhroncok, 17:20:38) * Change owners please coordinate the change with the Python Maintainers before changing the defaults (mhroncok, 17:21:20)
* Next week's chair (mhroncok, 17:21:46) * ACTION: Conan Kudo will probably chair next meeting (mhroncok, 17:22:17)
* Open Floor (mhroncok, 17:23:10)
* #2907 Exception for spliting OpenJDK build and integration (mhroncok, 17:23:47) * AGREED: No exception is needed (+5,1,-0) (mhroncok, 17:31:20)
* Open Floor (mhroncok, 17:31:52)
Meeting ended at 17:42:06 UTC.
Action Items ------------ * Conan Kudo will probably chair next meeting
Action Items, by person ----------------------- * **UNASSIGNED** * Conan Kudo will probably chair next meeting
People Present (lines said) --------------------------- * mhroncok (66) * zbyszek (27) * Eighth_Doctor (25) * zodbot (20) * hergertme (13) * mhayden (9) * davide (6) * dcantrell (4) * music[m] (2) * salimma (1) * MichaelCatanzaro (1) * nirik (0) * decathorpe (0) * sgallagh (0) * music (0) * Conan_Kudo (0) * Pharaoh_Atem (0) * Son_Goku (0) * King_InuYasha (0) * Sir_Gallantmon (0)
Generated by `MeetBot`_ 0.4
.. _`MeetBot`: https://fedoraproject.org/wiki/Zodbot#Meeting_Functions
On Tue, 3 Jan 2023 at 09:39, Miro Hrončok mhroncok@redhat.com wrote:
= New business =
#2923 Re-vote for Change proposal: Add -fno-omit-frame-pointer to default compilation flags https://pagure.io/fesco/issue/2923
Given the controversial nature of this one, why was it re-litigated at short notice when a large number of the stakeholders were still on vacation?
On 04. 01. 23 17:29, Jonathan Wakely wrote:
On Tue, 3 Jan 2023 at 09:39, Miro Hrončok mhroncok@redhat.com wrote:
= New business =
#2923 Re-vote for Change proposal: Add -fno-omit-frame-pointer to default compilation flags https://pagure.io/fesco/issue/2923
Given the controversial nature of this one, why was it re-litigated at short notice when a large number of the stakeholders were still on vacation?
Presumably because it now had majority support in FESCo and waiting extra week would collie with the mass rebuild.
See https://pagure.io/fesco/issue/2923#comment-833432 and the meting logs https://meetbot.fedoraproject.org/fedora-meeting/2023-01-03/fesco.2023-01-03...
Miro Hrončok wrote:
On 04. 01. 23 17:29, Jonathan Wakely wrote:
On Tue, 3 Jan 2023 at 09:39, Miro Hrončok mhroncok@redhat.com wrote:
= New business =
#2923 Re-vote for Change proposal: Add -fno-omit-frame-pointer to #default compilation flags https://pagure.io/fesco/issue/2923
Given the controversial nature of this one, why was it re-litigated at short notice when a large number of the stakeholders were still on vacation?
Presumably because it now had majority support in FESCo and waiting extra week would collie with the mass rebuild.
Still, it gives an extremely bad impression of rushing this through without giving anybody the chance to object.
I also do not see why this needed to be approved for F38 on such a short notice and could not wait for F39.
Kevin Kofler
Kevin Kofler via devel wrote:
Still, it gives an extremely bad impression of rushing this through without giving anybody the chance to object.
I also do not see why this needed to be approved for F38 on such a short notice and could not wait for F39.
PS: The impression I get is that everything was deliberately rigged so that the vote would end up the way it did:
1. A new ticket was filed, in order to exclude the participants of the previous discussion. 2. The people watching the old ticket were NOT notified. 3. The Tools Team was NOT notified. 4. The proponents of the Change, on the other hand, WERE notified.
So, with all of the above, the discussion participants were preselected to only include people in favor of the change.
5. The ticket was filed in the middle of the holiday season. Many people in Europe are on vacation until today. 6. There was NO thread about the reopening of the discussion on the mailing list. The first message that mentioned the issue on the mailing list was "Schedule for Tuesday's FESCo Meeting (2023-01-03)" from 2023-01-03 09:39 UTC, only 7 hours 21 minutes before the meeting. This is in contrast to the Change policy requiring at least a week between the mailing list announcement and opening the FESCo ticket. 7. Only 4 days had elapsed between the (unannounced) opening of the ticket and the vote. This is clearly insufficient. The one week in the Change policy that I cited above is designed as a minimum time for discussion. 8. The change was approved only 2 weeks before the mass rebuild, leaving little to no time to revert it in the contigency case.
So, this ensured that whoever was deliberately NOT invited had no chance to find out by themselves and intervene before it was too late.
This strikes me as extremely intransparent and undemocratic.
Therefore, I hereby request that the vote be annulled as having happened in violation of the Change policy.
Kevin Kofler
Kevin Kofler via devel wrote:
- Only 4 days had elapsed between the (unannounced) opening of the ticket
and the vote. This is clearly insufficient. The one week in the Change
Sorry, there were actually 6 days. Still, less than a week, and there was no mailing list announcement. The community was under the impression that the matter was settled when it was actually not. So I still believe that the transparency policies were not followed.
Kevin Kofler
Also, the pull request implementing the change was merged 6 days BEFORE the change was approved by FESCo! It is unacceptable to sneak unapproved changes into Rawhide this way in order to "create facts".
Kevin Kofler
On Sun, Jan 8, 2023 at 9:08 PM Kevin Kofler via devel devel@lists.fedoraproject.org wrote:
Also, the pull request implementing the change was merged 6 days BEFORE the change was approved by FESCo! It is unacceptable to sneak unapproved changes into Rawhide this way in order to "create facts".
Actually, no it wasn't. Adding knobs is perfectly fine when it doesn't have a practical impact by default.
The actual change to switch the default wasn't merged until Friday: https://src.fedoraproject.org/rpms/redhat-rpm-config/pull-request/231
Kevin Kofler via devel wrote:
PS: The impression I get is that everything was deliberately rigged so that the vote would end up the way it did:
- A new ticket was filed, in order to exclude the participants of the
previous discussion. 2. The people watching the old ticket were NOT notified. 3. The Tools Team was NOT notified. 4. The proponents of the Change, on the other hand, WERE notified.
So, with all of the above, the discussion participants were preselected to only include people in favor of the change.
- The ticket was filed in the middle of the holiday season. Many people
in Europe are on vacation until today. 6. There was NO thread about the reopening of the discussion on the mailing list. The first message that mentioned the issue on the mailing list was "Schedule for Tuesday's FESCo Meeting (2023-01-03)" from 2023-01-03 09:39 UTC, only 7 hours 21 minutes before the meeting. This is in contrast to the Change policy requiring at least a week between the mailing list announcement and opening the FESCo ticket. 7. Only 4 days had elapsed between the (unannounced) opening of the ticket and the vote. This is clearly insufficient. The one week in the Change policy that I cited above is designed as a minimum time for discussion. 8. The change was approved only 2 weeks before the mass rebuild, leaving little to no time to revert it in the contigency case.
So, this ensured that whoever was deliberately NOT invited had no chance to find out by themselves and intervene before it was too late.
This strikes me as extremely intransparent and undemocratic.
PPS: This is particularly striking when you consider that the same person who filed the new ticket and excluded one side from the discussion entirely was the one complaining just a month earlier about the OLD discussion:
Yes, but the other stakeholder I wanted there didn't even know it was on the agenda yesterday, which meant we mostly heard only one side (the toolchain people).
See: https://pagure.io/fesco/issue/2817#comment-830204
Complaining about something and then deliberately doing the same thing the other way round, way to go!
Kevin Kofler
On Sun, Jan 8, 2023 at 9:44 PM Kevin Kofler via devel devel@lists.fedoraproject.org wrote:
Kevin Kofler via devel wrote:
PS: The impression I get is that everything was deliberately rigged so that the vote would end up the way it did:
- A new ticket was filed, in order to exclude the participants of the
previous discussion. 2. The people watching the old ticket were NOT notified. 3. The Tools Team was NOT notified. 4. The proponents of the Change, on the other hand, WERE notified.
So, with all of the above, the discussion participants were preselected to only include people in favor of the change.
- The ticket was filed in the middle of the holiday season. Many people
in Europe are on vacation until today. 6. There was NO thread about the reopening of the discussion on the mailing list. The first message that mentioned the issue on the mailing list was "Schedule for Tuesday's FESCo Meeting (2023-01-03)" from 2023-01-03 09:39 UTC, only 7 hours 21 minutes before the meeting. This is in contrast to the Change policy requiring at least a week between the mailing list announcement and opening the FESCo ticket. 7. Only 4 days had elapsed between the (unannounced) opening of the ticket and the vote. This is clearly insufficient. The one week in the Change policy that I cited above is designed as a minimum time for discussion. 8. The change was approved only 2 weeks before the mass rebuild, leaving little to no time to revert it in the contigency case.
So, this ensured that whoever was deliberately NOT invited had no chance to find out by themselves and intervene before it was too late.
This strikes me as extremely intransparent and undemocratic.
PPS: This is particularly striking when you consider that the same person who filed the new ticket and excluded one side from the discussion entirely was the one complaining just a month earlier about the OLD discussion:
Yes, but the other stakeholder I wanted there didn't even know it was on the agenda yesterday, which meant we mostly heard only one side (the toolchain people).
See: https://pagure.io/fesco/issue/2817#comment-830204
Complaining about something and then deliberately doing the same thing the other way round, way to go!
You ascribe malice where there is none. The reason I didn't link it in the original ticket when I made that one was because I didn't think of it. I don't have some evil plan or something like that as you describe.
Moreover, your general attitude, tone, and verbiage is unnecessarily negative. You're also acting like a hypocrite.
I'm extremely tired of how you communicate on this mailing list. I've been quiet about it for years and years. But people leave Fedora because of you. Once again, I'm having discussions with people trying to convince them Fedora isn't a bad place because of you.
You need to sit back and reflect on yourself. Go get some help on improving your communication or something. If Linus Torvalds can become a better person and shake off his reputation for tearing people down, you can change and become a more constructive communicator.
Too many people give you leeway because you're not considered a native English speaker. You know what? I've met far too many better communicators who have English as a second, third, or even fourth language.
There are only three other people in Fedora that give me similar heartburn, and you should not be in that camp.
Please, for everyone's sake, do something to improve yourself.
On 09/01/2023 03:58, Neal Gompa wrote:
I'm extremely tired of how you communicate on this mailing list. I've been quiet about it for years and years. But people leave Fedora because of you. Once again, I'm having discussions with people trying to convince them Fedora isn't a bad place because of you.
Please stop personal attacks on other people. By the way, you do the same in #fedora-kde.
Neal Gompa wrote on 2023/01/09 11:58:
I'm extremely tired of how you communicate on this mailing list. I've been quiet about it for years and years. But people leave Fedora because of you. Once again, I'm having discussions with people trying to convince them Fedora isn't a bad place because of you.
I don't think this message is appropriate, as other people already said.
Too many people give you leeway because you're not considered a native English speaker. You know what? I've met far too many better communicators who have English as a second, third, or even fourth language.
Honestly saying, I cannot understand this context, why native English speaker or not is relevant here, especially because I am also non-native.
I don't think Fedora committers change their attitude against between native or non-native English speakers.
Even if someone takes uncomfortable attitude against you, you also have to refrain from attacking the person like this way.
Mamoru
On 9/01/2023 10:09, Mamoru TASAKA wrote:
Honestly saying, I cannot understand this context, why native English speaker or not is relevant here, especially because I am also non-native.
I don't think Fedora committers change their attitude against between native or non-native English speakers.
Mamoru
/Disclaimer: I don't want to take part in this discussion and this is not me taking a "side"./
I want to clarify that it's often difficult for non-native English speakers to come across as friendly. It's quite easy to write/speak English but sounding formal or friendly is a bit more difficult. Therefore I understand and agree that non-native English speakers are given a bit of leeway.
On Sun, Jan 08, 2023 at 06:06:47PM +0100, Kevin Kofler via devel wrote:
PS: The impression I get is that everything was deliberately rigged so that the vote would end up the way it did:
- A new ticket was filed, in order to exclude the participants of the
previous discussion. 2. The people watching the old ticket were NOT notified. 3. The Tools Team was NOT notified. 4. The proponents of the Change, on the other hand, WERE notified.
I agree with your earlier post that this did not have enough visibility, enough notice, or enough time. I was certainly taken by surprise, and I was trying to keep an eye on this one in particular. (Having the discussion under "Schedule for Tuesday's FESCo Meeting" didn't help it jump out at me either.)
BUT, I do not think it was done with malice, as "deliberately rigged" implies. I don't see that at all -- I see excitement and interest in moving forward on something that already has taken a long time, and looming practical deadlines.
Therefore, I hereby request that the vote be annulled as having happened in violation of the Change policy.
So, from a purely "what are the rules?" view, the Change process says:
FESCo will vote to approve or deny a change proposal in accordance with the FESCo ticket policy.
... and I won't quote all of that, but looking at https://docs.fedoraproject.org/en-US/fesco/#_ticket_policy... I don't see any violations, either in the letter or the spirit of what is written.
And, from a practical point of view, since this passed with six +1 votes, I'm not sure what benefit canceling and re-voting would really add.
It might be useful to improve both the documented policies. The Changes policy has nothing about reconsidering Changes in the current cycle that I can see. (Or, actually, for that matter, for resubmitting in future cycles either, unless i'm missing something.) And the FESCo ticket policy could include a) some steps for wider communication, and b) maybe something about holidays and other times which might need special consideration.
Most crucially, let's please drop the idea that anyone is acting out of malice. I'm quite sure that everyone arguing passionately on both sides of this issue has the best interest of Fedora and of Fedora Linux users in mind. Let's all keep that framing in mind in the ongoing discussion. Thank you.
Matthew Miller mattdm@fedoraproject.org writes:
So, from a purely "what are the rules?" view, the Change process says: FESCo will vote to approve or deny a change proposal in accordance with the FESCo ticket policy.
... and I won't quote all of that, but looking at https://docs.fedoraproject.org/en-US/fesco/#_ticket_policy... I don't see any violations, either in the letter or the spirit of what is written.
OTOH:
"The motivation for the Changes process is to raise the visibility of planned changes and make coordination and planning effort easier. It is nearly impossible to follow all changes happening in a big project such as Fedora. By providing a mechanism for sharing changes, it is easier for contributors to know what is coming and to ensure that we can address impacts of changes well before the release date. Change proposals should be shared as early as possible, before the change is implemented and even in the very early state of the idea, to gather community feedback and review."
One could argue that taking a change that was widely discussed and clearly rejected, then initiating a sudden revote without at least as much publicity, undercuts the "easier for contributors to know what is coming" or "gather community feedback and review" point of the whole thing.
- FChE
Am Montag, 9. Jänner 2023 17:47:54 CET schrieb Matthew Miller:
And, from a practical point of view, since this passed with six +1 votes, I'm not sure what benefit canceling and re-voting would really add.
Canceling the vote and requiring that the Change Policy be followed would mean that the change needs to be announced on the mailing list now and that a FESCo ticket may only be filed no earlier than 7 days from now, i.e., 2023-01-16. So, it could be voted no earlier than the 2023-01-17 FESCo meeting. That is right before the mass rebuild, hence this Change must be pushed back to Fedora 39 at the earliest. So from a practical standpoint, even if the outcome of the vote were not to change, it would save Fedora 38 from being pessimized by this change.
But I would hope that the discussion could actually convince those FESCo members that have switched to voting +1 due to the (IMHO unfair) comparison with the _FORTIFY_SOURCE=3 change to switch back.
Kevin Kofler
On 09. 01. 23 17:47, Matthew Miller wrote:
On Sun, Jan 08, 2023 at 06:06:47PM +0100, Kevin Kofler via devel wrote:
PS: The impression I get is that everything was deliberately rigged so that the vote would end up the way it did:
- A new ticket was filed, in order to exclude the participants of the
previous discussion. 2. The people watching the old ticket were NOT notified. 3. The Tools Team was NOT notified. 4. The proponents of the Change, on the other hand, WERE notified.
I agree with your earlier post that this did not have enough visibility, enough notice, or enough time. I was certainly taken by surprise, and I was trying to keep an eye on this one in particular. (Having the discussion under "Schedule for Tuesday's FESCo Meeting" didn't help it jump out at me either.)
BUT, I do not think it was done with malice, as "deliberately rigged" implies. I don't see that at all -- I see excitement and interest in moving forward on something that already has taken a long time, and looming practical deadlines.
Therefore, I hereby request that the vote be annulled as having happened in violation of the Change policy.
So, from a purely "what are the rules?" view, the Change process says:
FESCo will vote to approve or deny a change proposal in accordance with the FESCo ticket policy.
... and I won't quote all of that, but looking at https://docs.fedoraproject.org/en-US/fesco/#_ticket_policy... I don't see any violations, either in the letter or the spirit of what is written.
And, from a practical point of view, since this passed with six +1 votes, I'm not sure what benefit canceling and re-voting would really add.
It might be useful to improve both the documented policies. The Changes policy has nothing about reconsidering Changes in the current cycle that I can see. (Or, actually, for that matter, for resubmitting in future cycles either, unless i'm missing something.) And the FESCo ticket policy could include a) some steps for wider communication, and b) maybe something about holidays and other times which might need special consideration.
Most crucially, let's please drop the idea that anyone is acting out of malice. I'm quite sure that everyone arguing passionately on both sides of this issue has the best interest of Fedora and of Fedora Linux users in mind. Let's all keep that framing in mind in the ongoing discussion. Thank you.
I feel like I need to add some words as a chair of the meeting where this was approved.
First, let me apologize for not suggesting that we should invite all the stakeholders to the meeting before we voted. In retrospect, this was a bad call on my part and I sincerely regret it.
When the ticket was opened on Decmeber 28th, when many Fedora contributors were spending time with their families etc. instead of on Fedora, I raised my concerns about the date, as I was afraid it might be approved by gaining +3 votes in a week without the remaining FESCo members even knowing about this.
Fortunately, another FESCo member voted -1, hence this option was no longer my concern. That immediately pushed the ticket to the agenda of the meeting on January 3rd.
During the meeting, I felt like the proposal is gaining a (close) majority of the required votes and I decided to conduct the vote because I felt like all the discussion topics wrt this were already exhausted. I am not sure if this was a good call, but I am pretty confident that postponing the vote would have not changed the results.
Considering the mass rebuild is happening really soon, I feel like repeating the vote later is not helpful, but if people want to, we can surely run the vote again. However, I am confident the result will be the same.
Miro Hrončok wrote:
Considering the mass rebuild is happening really soon, I feel like repeating the vote later is not helpful, but if people want to, we can surely run the vote again. However, I am confident the result will be the same.
With all this talk about the mass rebuild being imminent, why can the change not be pushed back to Fedora 39, to have more time for discussion, for evaluating the impact in (post-branching) Rawhide (with almost 6 months time to try out things before the next mass rebuild), and to have a realistic possibility of reverting it BEFORE it reaches end users of stable releases (if the impact is as bad as I fear)?
Kevin Kofler
On 10. 01. 23 0:44, Kevin Kofler via devel wrote:
Miro Hrončok wrote:
Considering the mass rebuild is happening really soon, I feel like repeating the vote later is not helpful, but if people want to, we can surely run the vote again. However, I am confident the result will be the same.
With all this talk about the mass rebuild being imminent, why can the change not be pushed back to Fedora 39, to have more time for discussion, for evaluating the impact in (post-branching) Rawhide (with almost 6 months time to try out things before the next mass rebuild), and to have a realistic possibility of reverting it BEFORE it reaches end users of stable releases (if the impact is as bad as I fear)?
That is certainly a possibility, but it's not helpful to tell that to me personally. I have not voted for this change, I have abstained, because I was not sure the promises made by the approved statement are likely to be held.
* Matthew Miller:
BUT, I do not think it was done with malice, as "deliberately rigged" implies. I don't see that at all -- I see excitement and interest in moving forward on something that already has taken a long time, and looming practical deadlines.
No one spoke out when the tools team was called untrustworthy on the FESCo ticket. We can try explain it as an accident that the toolchain team was sidelined afterwards. But the overall sequence of events certainly looks rather odd.
There are infrastructure problems as well. Notifications in the Fedora wiki system are broken (email notifications are spotty, but not completely defunct; that did not matter here because the new FESCo ticket was added to the change page only after the second vote), and missing notifications for label updates on FESCo tickets (so it's hard to spot that an issue is scheduled for discussion). So unless you are in the in-group or invited, it's hard to contribute.
All these issues contribute to a work environment that I find very problematic. The individual aspects are probably not deliberate, but there's still an impact.
And, from a practical point of view, since this passed with six +1 votes, I'm not sure what benefit canceling and re-voting would really add.
I'm pretty sure no one considered the impact on non-x86-64 architectures, given that the change as voted and submitted as a PR would have broken the ppc64le and s390x buildroots.
I think it would save everyone a bit of time if we restricted the change to x86-64. We do not have much experience with the -mbackchain flag that was added at the last minute on s390x. The change owners have stated that they aren't interested in s390x. IBM doesn't want this. Platform Tools does not want it. I doubt the desktop team does GNOME performance analysis on s390x on Fedora. I'm not even sure if the tools support backchain-based unwinding; it's not a frame pointer after all. Maybe -mbackchain won't cause any issues in after all, but we just don't have the time to test this before the mass rebuild.
As Jakub and I have repeatedly explained, -fno-omit-frame-pointer on i686 is known to break certain packages (although I worked around this in glibc last year), simply because the reduced number of registers makes it impossible for GCC to compile certain functions with inline assembly in them. As with s390x, the concrete impact is not known at this point, and we are out of time for test builds.
Using -mno-omit-leaf-frame-pointer for aarch64 seems to be another last-minute addition without any clear justification. (On AArch64, the link register allows one to recover the address of the immediate caller even if a leaf function does not have a frame pointer. That's not possible on x86-64, where the caller's address must be read from the stack, and that has to be based on the frame pointer.) Just because the compiler option is there to enable doesn't mean it does anything useful in this context.
Thanks, Florian
On Mon, Jan 9 2023 at 06:54:09 PM +0100, Florian Weimer fweimer@redhat.com wrote:
No one spoke out when the tools team was called untrustworthy on the FESCo ticket.
That was one of my comments. [1] I should have worded that more carefully, but didn't because I was very frustrated. I apologize. What I meant here is that I strongly disagree with how the members of the toolchain team were evaluating the trade-offs in this issue. Of course I don't believe that you (or any other Fedora contributors) are generally untrustworthy.
[1] https://pagure.io/fesco/issue/2817#comment-830245
We can try explain it as an accident that the toolchain team was sidelined afterwards. But the overall sequence of events certainly looks rather odd.
Sounds like everyone agrees the process here did not work very well. The FESCo meeting was held on January 3, which is first day back from holidays for most folks in the US, and probably other countries too. An hour or two before the revote, I was quite worried that I wouldn't be able to contact Christian in time to attend the meeting, since it was 9 AM on first day back in his timezone. Contacting opponents of the change proposal was just not on my radar at all.
Anyway, the good news is there is still two more weeks until the mass rebuild begins, which is plenty of time for continued discussion and for FESCo to cancel the change if desired. So I don't think the outcome here was horribly unfair. I believe FESCo had an appropriate understanding of the considerations involved at the time of the vote: pay a performance price, gain ability to debug performance problems. (In particular, I doubt the _FORTIFY_SOURCE=3 change was really a major consideration here.)
Michael
Michael Catanzaro wrote:
(In particular, I doubt the _FORTIFY_SOURCE=3 change was really a major consideration here.)
What was, then? That was literally the only thing that has changed between the two diametrically different votes.
Kevin Kofler
On Wed, Jan 11, 2023 at 12:15 AM Kevin Kofler via devel devel@lists.fedoraproject.org wrote:
Michael Catanzaro wrote:
(In particular, I doubt the _FORTIFY_SOURCE=3 change was really a major consideration here.)
What was, then? That was literally the only thing that has changed between the two diametrically different votes.
Speaking for myself, the only way in which the _FORTIFY_SOURCE change impacted my opinion on -fno-omit-frame-pointers is that it made me think about it again, and that the level of scrutiny myself - and other members of FESCo - had given that change, had been disproportionate. And you might like it or not, I had just changed my mind on the topic since we had the first vote.
At that point, the only thing that could have convinced me to *not* to vote for enabling frame pointers would have been substantial arguments *against* the change from the toolchain team (and by substantial I mean something more than "the team is against it", "we don't like it", or "it's going to have a small performance impact"). Given that there had been *months* for people to speak up, the likelihood of any such arguments materializing at the last minute is pretty small (and even then, the change could have been reverted between approval and mass rebuild, if necessary).
Fabio
On Tue, Jan 10, 2023 at 7:17 PM Fabio Valentini decathorpe@gmail.com wrote:
Speaking for myself, the only way in which the _FORTIFY_SOURCE change impacted my opinion on -fno-omit-frame-pointers is that it made me think about it again, and that the level of scrutiny myself - and other members of FESCo - had given that change, had been disproportionate.
The _FORTIFY_SOURCE change really shouldn't even have prompted a discussion on the frame pointers change, let alone a reflection because there is absolutely nothing in common there. Especially in the context of scrutiny level; one is a security change that has no broad slowdown and another is a developer experience change that is shown to have a broad slowdown (characterized by minor or major depending on which side of the aisle you stand on).
And you might like it or not, I had just changed my mind on the topic since we had the first vote.
I wish you had spoken up sooner then, using the _FORTIFY_SOURCE change as the opportunity to speak up not only confused the whole thing, it also makes it harder to convince that the change of mind came independently.
Sid
On Mon, Jan 09, 2023 at 11:47:54AM -0500, Matthew Miller wrote:
On Sun, Jan 08, 2023 at 06:06:47PM +0100, Kevin Kofler via devel wrote:
PS: The impression I get is that everything was deliberately rigged so that the vote would end up the way it did:
- A new ticket was filed, in order to exclude the participants of the
previous discussion. 2. The people watching the old ticket were NOT notified. 3. The Tools Team was NOT notified. 4. The proponents of the Change, on the other hand, WERE notified.
I agree with your earlier post that this did not have enough visibility, enough notice, or enough time. I was certainly taken by surprise, and I was trying to keep an eye on this one in particular. (Having the discussion under "Schedule for Tuesday's FESCo Meeting" didn't help it jump out at me either.)
Holding a FESCo meeting and vote on the very first working day after the long xmas / new year holiday is not exactly good timing if you want contributors to be broadly aware it is happening[1]. I might humbly suggest that next year, any important meetings that would naturally fall in the 1st week, be postponed until the 2nd week of Jan.
With regards, Daniel
[1] Yes, I'm aware that not every part of the world will shutdown during the xmas/new year period, but a large enough part of our contributor base does that its worth taking into account
Am Montag, 9. Jänner 2023 20:10:53 CET schrieb Daniel P. Berrangé:
Holding a FESCo meeting and vote on the very first working day after the long xmas / new year holiday is not exactly good timing if you want contributors to be broadly aware it is happening[1]. I might humbly suggest that next year, any important meetings that would naturally fall in the 1st week, be postponed until the 2nd week of Jan.
With regards, Daniel
[1] Yes, I'm aware that not every part of the world will shutdown during the xmas/new year period, but a large enough part of our contributor base does that its worth taking into account
In fact, as I already mentioned, in Austria, this meeting happened *during* the school holidays and many companies' companywide holidays. E.g., at my employer, we resumed work today. It is commonplace here to have holidays until Epiphany, i.e., January 6, or even until the end of the week in which it falls. And this year, January 6 was a Friday, so even more places were on vacation until the end of the week.
Kevin Kofler
On Mon, Jan 9, 2023 at 2:11 PM Daniel P. Berrangé berrange@redhat.com wrote:
On Mon, Jan 09, 2023 at 11:47:54AM -0500, Matthew Miller wrote:
On Sun, Jan 08, 2023 at 06:06:47PM +0100, Kevin Kofler via devel wrote:
PS: The impression I get is that everything was deliberately rigged so that the vote would end up the way it did:
- A new ticket was filed, in order to exclude the participants of the
previous discussion. 2. The people watching the old ticket were NOT notified. 3. The Tools Team was NOT notified. 4. The proponents of the Change, on the other hand, WERE notified.
I agree with your earlier post that this did not have enough visibility, enough notice, or enough time. I was certainly taken by surprise, and I was trying to keep an eye on this one in particular. (Having the discussion under "Schedule for Tuesday's FESCo Meeting" didn't help it jump out at me either.)
Holding a FESCo meeting and vote on the very first working day after the long xmas / new year holiday is not exactly good timing if you want contributors to be broadly aware it is happening[1]. I might humbly suggest that next year, any important meetings that would naturally fall in the 1st week, be postponed until the 2nd week of Jan.
We should push out the entire schedule one to two weeks then. We keep losing time in the schedule, and we shouldn't lose even more.
-- 真実はいつも一つ!/ Always, there's only one truth!
On 1/9/23 11:18, Neal Gompa wrote:
On Mon, Jan 9, 2023 at 2:11 PM Daniel P. Berrangé berrange@redhat.com wrote:
On Mon, Jan 09, 2023 at 11:47:54AM -0500, Matthew Miller wrote:
On Sun, Jan 08, 2023 at 06:06:47PM +0100, Kevin Kofler via devel wrote:
PS: The impression I get is that everything was deliberately rigged so that the vote would end up the way it did:
- A new ticket was filed, in order to exclude the participants of the
previous discussion. 2. The people watching the old ticket were NOT notified. 3. The Tools Team was NOT notified. 4. The proponents of the Change, on the other hand, WERE notified.
I agree with your earlier post that this did not have enough visibility, enough notice, or enough time. I was certainly taken by surprise, and I was trying to keep an eye on this one in particular. (Having the discussion under "Schedule for Tuesday's FESCo Meeting" didn't help it jump out at me either.)
Holding a FESCo meeting and vote on the very first working day after the long xmas / new year holiday is not exactly good timing if you want contributors to be broadly aware it is happening[1]. I might humbly suggest that next year, any important meetings that would naturally fall in the 1st week, be postponed until the 2nd week of Jan.
We should push out the entire schedule one to two weeks then. We keep losing time in the schedule, and we shouldn't lose even more.
I think a good solution would be to move the proposal submission deadlines a month earlier in the schedule. There's only 3 weeks between the "Changes Requiring Mass Rebuild" deadline and the mass rebuild. I don't think 3 weeks is really enough time for FESCO review/approval and also getting the necessary patches reviewed and committed.
- Tom
-- 真実はいつも一つ!/ Always, there's only one truth! _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Am Montag, 9. Jänner 2023 21:02:48 CET schrieb Tom Stellard:
I think a good solution would be to move the proposal submission deadlines a month earlier in the schedule. There's only 3 weeks between the "Changes Requiring Mass Rebuild" deadline and the mass rebuild. I don't think 3 weeks is really enough time for FESCO review/approval and also getting the necessary patches reviewed and committed.
How would this have helped in this case? The original change proposal was actually submitted more than 6 months ago! It is just that it took 5 months to finally get to a vote (with whose outcome one FESCo member was then unhappy).
The resubmission, on the other hand, happened one day AFTER the submission deadline for changes requiring a mass rebuild and hence was already late under the current process. Pushing the submission deadline earlier would not have changed that.
If we need a rule, then it needs to be that rejected changes cannot be resubmitted for reconsideration after the change submission deadline. Though FESCo could just vote to accept the late change anyway, so it would not really help if the resubmission comes from FESCo itself and if FESCo really wants it to happen. At most it could discourage such late reconsiderations.
Kevin Kofler
On 1/9/23 12:27, Kevin Kofler via devel wrote:
Am Montag, 9. Jänner 2023 21:02:48 CET schrieb Tom Stellard:
I think a good solution would be to move the proposal submission deadlines a month earlier in the schedule. There's only 3 weeks between the "Changes Requiring Mass Rebuild" deadline and the mass rebuild. I don't think 3 weeks is really enough time for FESCO review/approval and also getting the necessary patches reviewed and committed.
How would this have helped in this case? The original change proposal was actually submitted more than 6 months ago! It is just that it took 5 months to finally get to a vote (with whose outcome one FESCo member was then unhappy).
It wouldn't have helped in this case, but if we are discussing changing the schedule to avoid holiday conflicts, then I think moving the "Change Proposal Deadlines" earlier would be better than shifting the whole schedule a few weeks.
The resubmission, on the other hand, happened one day AFTER the submission deadline for changes requiring a mass rebuild and hence was already late under the current process. Pushing the submission deadline earlier would not have changed that.
If we need a rule, then it needs to be that rejected changes cannot be resubmitted for reconsideration after the change submission deadline. Though FESCo could just vote to accept the late change anyway, so it would not really help if the resubmission comes from FESCo itself and if FESCo really wants it to happen. At most it could discourage such late reconsiderations.
Yes, I agree with this and I actually just proposed this up-thread.
-Tom
Kevin Kofler _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
On 1/9/23 08:47, Matthew Miller wrote:
It might be useful to improve both the documented policies. The Changes policy has nothing about reconsidering Changes in the current cycle that I can see. (Or, actually, for that matter, for resubmitting in future cycles either, unless i'm missing something.) And the FESCo ticket policy could include a) some steps for wider communication, and b) maybe something about holidays and other times which might need special consideration.
I would recommend that reconsidering (which I would describe as modifying and revoting) Changes like this should not be allowed after the Proposal Submission Deadlines. Making major decisions like this late in the process introduces too much risk (imo).
Looking at this specific change, part of the reason that it was accepted is because it has a trivial opt-out mechanism, but the decision was made so late in the release process that there may not be enough time for maintainers to opt-out before the mass rebuild.
-Tom
Hi Matthew,
On Mon, Jan 09, 2023 at 11:47:54AM -0500, Matthew Miller wrote:
On Sun, Jan 08, 2023 at 06:06:47PM +0100, Kevin Kofler via devel wrote:
Therefore, I hereby request that the vote be annulled as having happened in violation of the Change policy.
So, from a purely "what are the rules?" view, the Change process says:
FESCo will vote to approve or deny a change proposal in accordance with the FESCo ticket policy.
... and I won't quote all of that, but looking at https://docs.fedoraproject.org/en-US/fesco/#_ticket_policy... I don't see any violations, either in the letter or the spirit of what is written.
And, from a practical point of view, since this passed with six +1 votes, I'm not sure what benefit canceling and re-voting would really add.
It does feel to me as being against the spirit, and only not against the letter because it is presented as a "revote" on an existing change proposal instead of proposing a new change proposal (which this really is IMHO).
Practically people have started preparing for the mass rebuild at the end of last year since that was when the change checkpoint for change proposals requiring a mass rebuild was. And at that point the Change Proposal was already decided to not be included. So it was never expected that the build flags would suddenly change so drastically (and some flags are still wrong). Cancelling and backing out these last minute changes will cause a lot less stress.
Socially I think it will be better for all involved if the policies on revoting and/or reintroducing a change proposal are first clarified before allowing a revote. At the moment everybody involved seems hurt because of the unclear policy. Not having clear rules on the needed visibility and time needed to discuss this revote/resubmission of the change proposal caused people to assume the worst about others. Lets reset and take the time to heal first, so people start actually talking about real solutions again.
Cheers,
Mark
On Mon, Jan 09, 2023 at 10:08:11PM +0100, Mark Wielaard wrote:
... and I won't quote all of that, but looking at https://docs.fedoraproject.org/en-US/fesco/#_ticket_policy... I don't see any violations, either in the letter or the spirit of what is written.
[...]
It does feel to me as being against the spirit, and only not against the letter because it is presented as a "revote" on an existing change proposal instead of proposing a new change proposal (which this really is IMHO).
To be clear -- I don't think what happened is in conflict with the _FESCo_ policy about tickets (see link above). FESCo does not, as far as I can see, have any specific policies about how votes related to a Change should be conducted. And I don't think there is a general "FESCo can never reconsider decisions!" rule.
But like I said, I _don't_ think this revote was as visible or transparent as it should have been, and I agree that that doesn't really fit with the intention of the Changes policy.
Socially I think it will be better for all involved if the policies on revoting and/or reintroducing a change proposal are first clarified before allowing a revote. At the moment everybody involved seems hurt because of the unclear policy. Not having clear rules on the needed visibility and time needed to discuss this revote/resubmission of the change proposal caused people to assume the worst about others. Lets reset and take the time to heal first, so people start actually talking about real solutions again.
Yep, I definitely agree with this.
On Sun, Jan 08, 2023 at 06:06:47PM +0100, Kevin Kofler via devel wrote:
Kevin Kofler via devel wrote:
Still, it gives an extremely bad impression of rushing this through without giving anybody the chance to object.
I also do not see why this needed to be approved for F38 on such a short notice and could not wait for F39.
PS: The impression I get is that everything was deliberately rigged so that the vote would end up the way it did:
You're mixing up two things: a vote being "rigged" with a result you don't like. Absolute majority (6 out of 9) of voting members voted in favour.
The rules in Fedora are that FESCo gets to vote on certain things. There is a time reserved for public discussion, but at some point a vote is scheduled, and at that point we often discuss things in a meeting and vote one way or another without futher input from outside.
It does happen occasionally that repeat votes happen, for example a resolution is approved, but somebody immediately raises some additional concern so it is amended. At that point we don't seek repeat opinions from all stakeholders. Things would take forever, and most stakeholders wouldn't change their opinion anyway for some minor detail.
The changes to this Change proposal were not considered major changes that would require a repeat discussion on fedora-devel, but were instead a narrowing and clarification of the proposal, so the vote was held at the earliest convenience. It is important that *FESCo members* know about the vote, which obviously they did in this case because absolute majority did vote.
If you were to look at FESCo meeting logs, this happens every once in a while: a proposal is made, it get's a few +1 votes but not enough and is effectively rejected, so a different-but-similar proposal is made and the vote repeats. Sometimes we go through a few of those in one meeting. In this case this was split over two meetings, but is not substantially different. Discussion was ongoing all that time, both publicly and privately, and there is nothing that says that FESCo members must not change their votes based on new information.
The intent of this particular proposal is to learn and adjust based on the feedback from the initial implementation. The specific flags on different architectures can and should be adjusted to get the desired result.
An interesting phenomenon is that before the Change was approved, most of the feedback on fedora-devel was about whether we need the change at all, and how horrible the performance degradation will be, and what distribution to switch to. After it was approved, the feedback immediately became more technical, with suggestions about specific flags and architecture differences.
If we had approved the Change 3 months ago, we would have gotten that useful part of the feedback much earlier. For me the lesson is that we shouldn't dawdle on high-stakes controversial votes, but instead approve ambitious proposals early and have more time to adjust or even revert if it turns out necessary.
Zbyszek
- A new ticket was filed, in order to exclude the participants of the
previous discussion. 2. The people watching the old ticket were NOT notified. 3. The Tools Team was NOT notified. 4. The proponents of the Change, on the other hand, WERE notified.
So, with all of the above, the discussion participants were preselected to only include people in favor of the change.
- The ticket was filed in the middle of the holiday season. Many people in
Europe are on vacation until today. 6. There was NO thread about the reopening of the discussion on the mailing list. The first message that mentioned the issue on the mailing list was "Schedule for Tuesday's FESCo Meeting (2023-01-03)" from 2023-01-03 09:39 UTC, only 7 hours 21 minutes before the meeting. This is in contrast to the Change policy requiring at least a week between the mailing list announcement and opening the FESCo ticket. 7. Only 4 days had elapsed between the (unannounced) opening of the ticket and the vote. This is clearly insufficient. The one week in the Change policy that I cited above is designed as a minimum time for discussion. 8. The change was approved only 2 weeks before the mass rebuild, leaving little to no time to revert it in the contigency case.
So, this ensured that whoever was deliberately NOT invited had no chance to find out by themselves and intervene before it was too late.
This strikes me as extremely intransparent and undemocratic.
Therefore, I hereby request that the vote be annulled as having happened in violation of the Change policy.
Kevin Kofler
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Am Montag, 9. Jänner 2023 23:06:17 CET schrieb Zbigniew Jędrzejewski-Szmek:
On Sun, Jan 08, 2023 at 06:06:47PM +0100, Kevin Kofler via devel wrote:
Kevin Kofler via devel wrote: PS: The impression I get is that everything was deliberately rigged so that the vote would end up the way it did:
You're mixing up two things: a vote being "rigged" with a result you don't like.
No, the result is NOT why I got the impression that the vote was rigged. The way that result was obtained is. In fact, I had explained exactly that in the remainder of the message, which you omitted from your quote and pushed to the bottom of the mail (and even that quotes only half of it) for some reason.
Absolute majority (6 out of 9) of voting members voted in favour.
As I believe, as a consequence of the incomplete and misleading data that was provided to them in the one-sided discussion, in particular, the flawed comparison with the _FORTIFY_SOURCE=3 change. (It might not have mattered to you personally, but you were the only one who had been in favor of that change from the beginning anyway.)
The changes to this Change proposal were not considered major changes that would require a repeat discussion on fedora-devel,
And that was a fatal misjudgement. A crucial point that swayed the vote was a comparison with another change, the _FORTIFY_SOURCE=3 one, which had not previously come up in the discussion. Therefore, we had no chance to debunk the flawed comparison. And flawed it was: * It neglected that _FORTIFY_SOURCE=3 is a security feature for end users whereas frame pointers are a developer-only feature. * It made wrong assumptions about the performance impact of _FORTIFY_SOURCE=3, which, compared to the already existing (!) _FORTIFY_SOURCE=2, appears to actually have NO performance impact at all (!), only compared to no _FORTIFY_SOURCE at all. * It came with an implied accusation of hypocrisy (double standards) against the Tools Team which makes no sense when you consider the above details, and the Tools Team was given no chance to defend themselves. That matters particularly because that false accusation was used to justify ignoring the Tools Team's stakeholder opinion on the frame pointer change.
If you were to look at FESCo meeting logs, this happens every once in a while: a proposal is made, it get's a few +1 votes but not enough and is effectively rejected, so a different-but-similar proposal is made and the vote repeats. Sometimes we go through a few of those in one meeting. In this case this was split over two meetings, but is not substantially different.
This is substantially different in that it was publicly communicated that the change was rejected and then suddenly FESCo changed their mind out of the blue. That is different from voting on multiple proposal variants during the same meeting and then communicating the final outcome.
(And by the way, what was ultimately accepted was not really a *different* proposal, but effectively the same as your proposal that had been voted down a couple weeks earlier.)
Discussion was ongoing all that time, both publicly and privately,
This was not transparent at all. We all got the impression that the discussion was closed and the feature conclusively rejected, and had no idea that there was still a discussion ongoing to which we were not invited.
and there is nothing that says that FESCo members must not change their votes based on new information.
But, as I explained above, said "new information" was misleading or incorrect, and the stakeholders were not given a chance to prove that.
The intent of this particular proposal is to learn and adjust based on the feedback from the initial implementation. The specific flags on different architectures can and should be adjusted to get the desired result.
That is effectively treating stable Fedora releases and their users as guinea pigs. Especially since you want to wait 2 whole release cycles before even considering a revert (and there is already the expectation from the Change owners that a revert will have the burden of proof reversed in its disfavor, which I consider unacceptable, but which was neither confirmed nor denied by FESCo – that is not what I would consider a provisional acceptance of the change).
I really do not see why gathering data cannot be done in a side tag and/or as a Fedora Remix. Experiments belongs into an experimental branch, not into a stable release.
An interesting phenomenon is that before the Change was approved, most of the feedback on fedora-devel was about whether we need the change at all, and how horrible the performance degradation will be, and what distribution to switch to. After it was approved, the feedback immediately became more technical, with suggestions about specific flags and architecture differences.
That is because the people have apparently resignated to accept that Fedora has decided to become unusable and are already looking for another distribution. I know I am.
If we had approved the Change 3 months ago, we would have gotten that useful part of the feedback much earlier. For me the lesson is that we shouldn't dawdle on high-stakes controversial votes, but instead approve ambitious proposals early and have more time to adjust or even revert if it turns out necessary.
That is also the very wrong lesson to get out of this. FESCo needs to be more willing to actually REJECT changes that make Fedora worse. (And with that I mean, STICK to the rejection, of course!) I am fed up of seeing every single such detrimental change getting approved by FESCo despite all the warnings and negative feedback from the mailing list. Often with disastrous consequences, e.g., just see how badly mandatory Modularity (with the intent to move some packages to module-only) has failed, for example. The failure modes were exactly the ones I and others had warned about from the onset. We were ignored and the change approved anyway. Unsurprisingly, it failed, and Modularity had to be made optional and module-only packages banned to fix the chaos. And now, for once, FESCo finally had the courage to reject an unfixably flawed change ("unfixably" because the performance loss is not going to go away no matter how you reword the change, rejecting it is the only reasonable thing you can do to it), and what happens? The decision suddenly gets reversed in a mid-holiday commando action. What the heck?!
Why is it so bad to just say no when you have to? Why can we not accept that some changes are just flawed by design and just cannot be fixed? Why is an acceptance always treated as a success and a rejection as a failure? The default reaction to a change needs to be to reject it. What has worked for decades should only be changed if there is a really strong reason to. Rejecting a change is always safer than accepting it.
Kevin Kofler
On Mon, Jan 9, 2023 at 5:53 PM Kevin Kofler via devel devel@lists.fedoraproject.org wrote:
- It made wrong assumptions about the performance impact of
_FORTIFY_SOURCE=3, which, compared to the already existing (!) _FORTIFY_SOURCE=2, appears to actually have NO performance impact at all (!), only compared to no _FORTIFY_SOURCE at all.
Your larger point is correct (and what I've been talking about too) but for the sake of accuracy: there is no known overhead due to _FORTIFY_SOURCE=2 either AFAIK; there's one publicly available analysis[1] that shows a minor speedup (!) in ffmpeg due to _FORTIFY_SOURCE=2. The ~1.3% overhead I mentioned was for "-fstack-protector-strong -fstack-clash-protection -D_FORTIFY_SOURCE={2,3} -D_GLIBCBXX_ASSERTIONS" and IMO it should mainly be attributed to -fstack-protector-strong -fstack-clash-protection and perhaps -D_GLIBCXX_ASSERTIONS.
Thanks, Sid
[1] https://zatoichi-engineer.github.io/2017/10/06/fortify-source.html
On Mon, Jan 09, 2023 at 11:53:17PM +0100, Kevin Kofler via devel wrote:
Am Montag, 9. Jänner 2023 23:06:17 CET schrieb Zbigniew Jędrzejewski-Szmek:
On Sun, Jan 08, 2023 at 06:06:47PM +0100, Kevin Kofler via devel wrote:
Kevin Kofler via devel wrote: PS: The impression I get is that everything was deliberately rigged so that the vote would end up the way it did:
You're mixing up two things: a vote being "rigged" with a result you don't like.
No, the result is NOT why I got the impression that the vote was rigged. The way that result was obtained is.
Exactly, you're just confirming what I wrote above.
A "vote being rigged" means that either the people who should be allowed to vote couldn't, or that people who are not allowed to vote did, or that voters were tricked or forced to vote differently, or that votes were miscounted.
None of this happened in this case. The fact that the absolute majority of FESCo cast a vote makes the considerations simpler, because we know that the two votes that were not cast would not have changed the result.
In fact, I had explained exactly that in the remainder of the message, which you omitted from your quote and pushed to the bottom of the mail (and even that quotes only half of it) for some reason.
You message very verbosely explains why you think the result should be different. That is not "rigged". That is other people deciding differently than you.
Zbyszek
On Tue, Jan 10, 2023 at 3:05 AM Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl wrote:
Exactly, you're just confirming what I wrote above.
A "vote being rigged" means that either the people who should be allowed to vote couldn't, or that people who are not allowed to vote did, or that voters were tricked or forced to vote differently, or that votes were miscounted.
What's being alleged is that many members were tricked into changing their vote by using the false narrative about _FORTIFY_SOURCE proposal getting an unfair pass despite performance concerns (which *I* hypothesized months ago and I later dispelled, in the end even quoting benchmark results for it) and creating the impression that the toolchain team is being duplicitous about the performance question. Further trickery involved rushing the vote, claiming that it had to be done soon to meet the mass rebuild deadline; too bad if those who had strong objections earlier weren't around to put their comments on record.
In that context the vote could in fact be considered rigged. If the voting members could come out and clarify exactly why they changed their vote, it would make things a lot clearer.
Sid
On Tue, Jan 10, 2023 at 08:30:25AM -0500, Siddhesh Poyarekar wrote:
On Tue, Jan 10, 2023 at 3:05 AM Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl wrote:
Exactly, you're just confirming what I wrote above.
A "vote being rigged" means that either the people who should be allowed to vote couldn't, or that people who are not allowed to vote did, or that voters were tricked or forced to vote differently, or that votes were miscounted.
What's being alleged is that many members were tricked into changing their vote by using the false narrative about _FORTIFY_SOURCE proposal getting an unfair pass despite performance concerns (which *I* hypothesized months ago and I later dispelled, in the end even quoting benchmark results for it) and creating the impression that the toolchain team is being duplicitous about the performance question. Further trickery involved rushing the vote, claiming that it had to be done soon to meet the mass rebuild deadline; too bad if those who had strong objections earlier weren't around to put their comments on record.
That description assumes that FESCo members are preschoolers who are easy to trick and also need to be reminded who said what every day. That's certainly not the case. The objections against the proposal were made very clearly and they certainly weren't forgotten over a few days. Those objections also didn't *change* over those few days, so repeating them wouldn't actually change anything.
Speaking for myself, I heard the objections from various sides, and I think I understand them. In particular I think that the objections from the compiler team are based on correct evaluation of the effect of the change. But that evaluation is hyperfocused on benchmark performance and doesn't look at the needs of the whole ecosystem. I think that the advantages of the proposal and the gains I hope will be realized outweigh the drawbacks.
I can't speak for other people, but I assume that they made a similar evaluation. FWIW, I voted the same both times, but I didn't make up my mind to vote +1 until relatively late before the vote after I had time to read up on the topic and go through various options that we have.
Zbyszek
Zbigniew Jędrzejewski-Szmek wrote:
That description assumes that FESCo members are preschoolers who are easy to trick and also need to be reminded who said what every day. That's certainly not the case. The objections against the proposal were made very clearly and they certainly weren't forgotten over a few days. Those objections also didn't *change* over those few days, so repeating them wouldn't actually change anything.
The objections did change because the arguments brought up by the other side did. I already pointed that out once. I do not want to sound like a broken record, so instead of a copy&paste, here is the link: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/...
Kevin Kofler
On Tue, Jan 10, 2023 at 4:31 PM Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl wrote:
That description assumes that FESCo members are preschoolers who are easy to trick and also need to be reminded who said what every day. That's certainly not the case. The objections against the proposal were made very clearly and they certainly weren't forgotten over a few days. Those objections also didn't *change* over those few days, so repeating them wouldn't actually change anything.
They don't need to be preschoolers; it's not that hard to manufacture an opinion among well informed adults, even unintentionally by just having a lot of conviction about it.
The seeds for the revote were placed in the _FORTIFY_SOURCE=3 change discussion and throughout the discussion, repeated explanations of why the proposals are not comparable were ignored, instead of which the focus seemed to be on driving consensus towards getting a revote on the frame pointer proposal and trying to paint the tools team's position as being duplicitous.
In the _FORTIFY_SOURCE=3 ticket one of the FESCo voters (who also drove the revote FWIW) took a hard negative stand only because they perceived a double standard on performance, which had, by then, already been debunked a couple of times in the devel thread. While he did change his vote to +1 later, he appeared to do so only after other members voiced their support. If that's not influencing narrative then I don't know what is.
Multiple other FESCo voters, when voting for the _FORTIFY_SOURCE proposal, talked about the frame pointer proposal, again clearly indicating that there is a cross-influence.
Finally, another voting member commented, this time on the re-vote ticket[1], again indicating that the reason for the revote is the misdirection in the _FORTIFY_SOURCE proposal discussion.
Christian did make an impassioned plea on the re-vote ticket for the case of frame pointers and it's perfectly understandable if that was a turning point for those who changed their vote (and please say it if that was what it was; I'd disagree but that's a different matter then) but the thing is, that plea needs counterarguments and further discussion and there was no opportunity for that to happen. Even then, the only reason why the revote happened at all was because of the persistent misdirection in the _FORTIFY_SOURCE proposal.
Speaking for myself, I heard the objections from various sides, and I think I understand them. In particular I think that the objections from the compiler team are based on correct evaluation of the effect of the change. But that evaluation is hyperfocused on benchmark performance and doesn't look at the needs of the whole ecosystem. I think that the advantages of the proposal and the gains I hope will be realized outweigh the drawbacks.
Ack, I respect that even if I don't agree with the conclusion.
Thanks, Sid
On Tue, Jan 10, 2023 at 09:15:41PM -0500, Siddhesh Poyarekar wrote: ...snip...
Finally, another voting member commented, this time on the re-vote ticket[1], again indicating that the reason for the revote is the misdirection in the _FORTIFY_SOURCE proposal discussion.
Just to set the record straight here: I made that comment, and I was wondering/suggesting (or as I note in the comment "guessing") as to why the revote was taking place. I was not involved in the revote and indeed I was trying to say "if this is because of the _FORTIFY_SOURCE proposal, there's a lot of difference between them and I can't see why it would change things."
Indeed it didn't change anything for me, as I kept my -1 vote.
kevin
So shouldn't there be some policy for revoting? E.g.:
~~~
If revote is initiated (by somebody?), the revote is going to be announced on devel(-announce) and can happen as soon as in one week.
~~~
And maybe it should not happen in the ticket, but on the meeting?
Vít
Dne 11. 01. 23 v 3:15 Siddhesh Poyarekar napsal(a):
On Tue, Jan 10, 2023 at 4:31 PM Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl wrote:
That description assumes that FESCo members are preschoolers who are easy to trick and also need to be reminded who said what every day. That's certainly not the case. The objections against the proposal were made very clearly and they certainly weren't forgotten over a few days. Those objections also didn't *change* over those few days, so repeating them wouldn't actually change anything.
They don't need to be preschoolers; it's not that hard to manufacture an opinion among well informed adults, even unintentionally by just having a lot of conviction about it.
The seeds for the revote were placed in the _FORTIFY_SOURCE=3 change discussion and throughout the discussion, repeated explanations of why the proposals are not comparable were ignored, instead of which the focus seemed to be on driving consensus towards getting a revote on the frame pointer proposal and trying to paint the tools team's position as being duplicitous.
In the _FORTIFY_SOURCE=3 ticket one of the FESCo voters (who also drove the revote FWIW) took a hard negative stand only because they perceived a double standard on performance, which had, by then, already been debunked a couple of times in the devel thread. While he did change his vote to +1 later, he appeared to do so only after other members voiced their support. If that's not influencing narrative then I don't know what is.
Multiple other FESCo voters, when voting for the _FORTIFY_SOURCE proposal, talked about the frame pointer proposal, again clearly indicating that there is a cross-influence.
Finally, another voting member commented, this time on the re-vote ticket[1], again indicating that the reason for the revote is the misdirection in the _FORTIFY_SOURCE proposal discussion.
Christian did make an impassioned plea on the re-vote ticket for the case of frame pointers and it's perfectly understandable if that was a turning point for those who changed their vote (and please say it if that was what it was; I'd disagree but that's a different matter then) but the thing is, that plea needs counterarguments and further discussion and there was no opportunity for that to happen. Even then, the only reason why the revote happened at all was because of the persistent misdirection in the _FORTIFY_SOURCE proposal.
Speaking for myself, I heard the objections from various sides, and I think I understand them. In particular I think that the objections from the compiler team are based on correct evaluation of the effect of the change. But that evaluation is hyperfocused on benchmark performance and doesn't look at the needs of the whole ecosystem. I think that the advantages of the proposal and the gains I hope will be realized outweigh the drawbacks.
Ack, I respect that even if I don't agree with the conclusion.
Thanks, Sid
[1] https://pagure.io/fesco/issue/2923#comment-833708 _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Dne 11. 01. 23 v 12:16 Vít Ondruch napsal(a):
So shouldn't there be some policy for revoting? E.g.:
If revote is initiated (by somebody?), the revote is going to be announced on devel(-announce) and can happen as soon as in one week.
Also, I am not quite sure who and how initiated the revote. Maybe the policy should be about initiating the revote instead. E.g. if I disagree with some FESCo decision and I think it should be revoted, I would announce my intention to initiate the revote on devel(-announce).
Vít
And maybe it should not happen in the ticket, but on the meeting?
Vít
Dne 11. 01. 23 v 3:15 Siddhesh Poyarekar napsal(a):
On Tue, Jan 10, 2023 at 4:31 PM Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl wrote:
That description assumes that FESCo members are preschoolers who are easy to trick and also need to be reminded who said what every day. That's certainly not the case. The objections against the proposal were made very clearly and they certainly weren't forgotten over a few days. Those objections also didn't *change* over those few days, so repeating them wouldn't actually change anything.
They don't need to be preschoolers; it's not that hard to manufacture an opinion among well informed adults, even unintentionally by just having a lot of conviction about it.
The seeds for the revote were placed in the _FORTIFY_SOURCE=3 change discussion and throughout the discussion, repeated explanations of why the proposals are not comparable were ignored, instead of which the focus seemed to be on driving consensus towards getting a revote on the frame pointer proposal and trying to paint the tools team's position as being duplicitous.
In the _FORTIFY_SOURCE=3 ticket one of the FESCo voters (who also drove the revote FWIW) took a hard negative stand only because they perceived a double standard on performance, which had, by then, already been debunked a couple of times in the devel thread. While he did change his vote to +1 later, he appeared to do so only after other members voiced their support. If that's not influencing narrative then I don't know what is.
Multiple other FESCo voters, when voting for the _FORTIFY_SOURCE proposal, talked about the frame pointer proposal, again clearly indicating that there is a cross-influence.
Finally, another voting member commented, this time on the re-vote ticket[1], again indicating that the reason for the revote is the misdirection in the _FORTIFY_SOURCE proposal discussion.
Christian did make an impassioned plea on the re-vote ticket for the case of frame pointers and it's perfectly understandable if that was a turning point for those who changed their vote (and please say it if that was what it was; I'd disagree but that's a different matter then) but the thing is, that plea needs counterarguments and further discussion and there was no opportunity for that to happen. Even then, the only reason why the revote happened at all was because of the persistent misdirection in the _FORTIFY_SOURCE proposal.
Speaking for myself, I heard the objections from various sides, and I think I understand them. In particular I think that the objections from the compiler team are based on correct evaluation of the effect of the change. But that evaluation is hyperfocused on benchmark performance and doesn't look at the needs of the whole ecosystem. I think that the advantages of the proposal and the gains I hope will be realized outweigh the drawbacks.
Ack, I respect that even if I don't agree with the conclusion.
Thanks, Sid
[1] https://pagure.io/fesco/issue/2923#comment-833708 _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
On Wed, Jan 11, 2023 at 6:25 AM Vít Ondruch vondruch@redhat.com wrote:
Dne 11. 01. 23 v 12:16 Vít Ondruch napsal(a):
So shouldn't there be some policy for revoting? E.g.:
If revote is initiated (by somebody?), the revote is going to be announced on devel(-announce) and can happen as soon as in one week.
Also, I am not quite sure who and how initiated the revote. Maybe the policy should be about initiating the revote instead. E.g. if I disagree with some FESCo decision and I think it should be revoted, I would announce my intention to initiate the revote on devel(-announce).
That came up in yesterday's FESCO meeting, so hopefully at least that workflow hole will be plugged:
https://meetbot.fedoraproject.org/fedora-meeting/2023-01-10/fesco.2023-01-10...
Sid
Am Dienstag, 10. Jänner 2023 08:25:27 CET schrieb Zbigniew Jędrzejewski-Szmek:
On Mon, Jan 09, 2023 at 11:53:17PM +0100, Kevin Kofler via devel wrote:
No, the result is NOT why I got the impression that the vote was rigged. The way that result was obtained is.
Exactly, you're just confirming what I wrote above.
Nonsense! Where am I confirming anything? (Quite the opposite, in fact!) Why do you keep completely missing the point of what I wrote?
A "vote being rigged" means that either the people who should be allowed to vote couldn't, or that people who are not allowed to vote did, or that voters were tricked or forced to vote differently, or that votes were miscounted.
None of this happened in this case.
The point of the message you first replied to is that I believe that voters may have been tricked into voting differently, by inviting only one side of the discussion and by providing the voters with a misleadingly biased presentation of the facts (up to baseless allegations of double standards against the Tools Team, based only on a misunderstanding of performance measurements).
The fact that the absolute majority of FESCo cast a vote makes the considerations simpler, because we know that the two votes that were not cast would not have changed the result.
It would if that were the issue at hand. It is not. The issue is with those who did vote.
In fact, I had explained exactly that in the remainder of the message, which you omitted from your quote and pushed to the bottom of the mail (and even that quotes only half of it) for some reason.
You message very verbosely explains why you think the result should be different. That is not "rigged". That is other people deciding differently than you.
No. The message very verbosely explains errors in the process, not in the outcome. Inviting only one side of the discussion is inherently biased. Not allowing sufficient time for discussion is also a way to silence dissenting opinions. No matter whether it has been done deliberately or accidentally. The ultimate outcome is that the vote has happened under unfair circumstances.
Sure, I also happen to think that the result should be different. For explanations of that, see one of my mails with clear technical objections to this change. But that was NOT the matter of the mail to which you are replying here. That mail was about HOW that result was obtained, pointing out clear procedural bias that you refuse to acknowledge.
Kevin Kofler