On Tue, Nov 29, 2022 at 09:31:01PM +0100, Kevin Kofler via devel wrote:
Hi,
Stephen Gallagher wrote:
> Following is the list of topics that will be discussed in the
> FESCo meeting Tuesday at 17:00UTC in #fedora-meeting on
> irc.libera.chat.
does anyone have a complete log? It does not show up on
meetbot.fedoraproject.org, and the raw log at
https://meetbot-raw.fedoraproject.org/fedora-meeting/2022-11-29/fesco.202...
is truncated.
--- Log opened Tue Nov 29 17:07:07 2022
17:07 -!- zbyszek [~zbyszek@fedora/zbyszek] has joined #fedora-meeting
17:07 -!- Irssi: #fedora-meeting: Total of 183 nicks [1 ops, 0 halfops, 62 voices, 120
normal]
17:07 -!- Irssi: Join to #fedora-meeting was synced in 1 secs
17:07 < zbyszek> .hello2
17:07 < zodbot> zbyszek: zbyszek 'Zbigniew Jędrzejewski-Szmek'
<zbyszek(a)in.waw.pl>
17:07 < zbyszek> Sorry, another meeting refusing ETERM signalling.
17:08 < nirik> ok, back sorry.
17:10 <+sgallagh> #topic #2817 Change proposal: Add -fno-omit-frame-pointer to
default compilation flags
17:10 <+sgallagh> .fesco 2817
17:10 -!- zodbot changed the topic of #fedora-meeting to: #2817 Change proposal: Add
-fno-omit-frame-pointer to default compilation flags (Meeting topic: FESCO (2022-11-29))
17:10 < zodbot> sgallagh: Issue #2817: Change proposal: Add -fno-omit-frame-pointer
to default compilation flags - fesco - Pagure.io -
https://pagure.io/fesco/issue/2817
17:10 < mhroncok> I'm back, thank you
17:10 -!- jontyms2707749 [~jontyms(a)76-242-160-235.lightspeed.dybhfl.sbcglobal.net] has
quit [Read error: Connection reset by peer]
17:11 < decathorpe> hey o/ sorry for being late
17:11 -!- jontyms2707749 [~jontyms(a)76-242-160-235.lightspeed.dybhfl.sbcglobal.net] has
joined #fedora-meeting
17:11 -!- zodbot [~supybot@fedora/bot/zodbot] has quit [Read error: Connection reset by
peer]
17:11 -!- jmracek57 [~jmracek(a)185.16.81.139] has quit [Quit: Client closed]
17:11 < decathorpe> or, reading backlog, not late after all :)
17:12 < mhroncok> check out the latest comment
17:12 < nirik> so... lots of discussion on this one.
17:12 < nirik> right, I was going to ask if the tools folks had really weighed in.
17:12 <+DaanDeMeyer[m]> Hi everyone, I'm joining as well for the frame pointer
proposal
17:12 < dcantrell> I was already leaning towards voting no on this one, but if the
tools team advises against it as well, that's a solid no from me
17:13 < codonell> :-)
17:13 < dcantrell> I saw codonell join as well
17:13 < nirik> I was trying to think of a middle ground type thing, but it would
need a second mass rebuild, which I am not sure would fly.
17:13 < decathorpe> yeah :( if there's objections that strong against it from
the people who maintain the affected tools ...
17:13 <+sgallagh> Let's give daandemeyer[m] a minute to state their case, but I
think we should make a final decision today in this meeting
17:14 < zbyszek> So, what is the basis for opposition from tools folks?
17:14 < Eighth_Doctor> on the other hand, the tools team might not be solely in the
right here, given that performance teams are saying they need this
17:14 < Eighth_Doctor> so there's clearly a big disconnect going on
17:14 <+music[m]> fweimer: Thanks for the update from the platform tools team
(
https://pagure.io/fesco/issue/2817#comment-829918). That was one thing on which I had
meant to request an update, but I hadn't had a chance.
17:15 < Eighth_Doctor> my beef with this isn't with actually doing the change,
it's that I didn't know whether people would use the new data to actually do perf
stuff
17:15 < Eighth_Doctor> at least from the desktop developers I've talked to, they
consider this change highly valuable
17:15 < fweimer> zbyszek: It's substantial slowdown across the board. And
there's no interest in frame pointers downstream.
17:15 < Eighth_Doctor> both on GNOME and KDE Plasma
17:15 < fweimer> Not from partners nor customers.
17:15 < fweimer> Or our internal performance team.
17:15 < zbyszek> Eighth_Doctor: I talked to some people privately who use perf for
various things, and they were rather enthusiastic about this.
17:16 < Eighth_Doctor> on the other hand, I also wonder why nobody has tried to make
-O3 better either
17:16 < zbyszek> fweimer: So those are very weak arguments, sorry.
17:16 < nirik> did I read somewhere that ppcl4le/aarch64 do have frame-pointers by
default?
17:16 <+DaanDeMeyer[m]> I can only say that we run upstream glibc compiled with
frame pointers internally and haven't run into any noticeable issues, we still get
tremendous value out of building with frame pointers
17:16 < fweimer> nirik: Correct.
17:17 < Eighth_Doctor> the other think I've heard from some folks is that the
reason we have this problem is similar to why we had this when spectre/meltdown
mitigations were first implemented
17:17 < codonell> DaanDeMeyer[m], Do you have patches you can contribute upstream to
fix all the missing frame pointer in the assembly code? :-)
17:17 < nirik> would perf work done on them translate to x86_64?
17:17 < Eighth_Doctor> nobody bothered to do performance with frame pointers for so
long that the only real way to make it back is to actually start doing it again
17:17 < fweimer> zbyszek: Why is this a weak argument? I think we're fairly
well-connected.
17:17 < Eighth_Doctor> s/think/thing/
17:18 < zbyszek> fweimer: re "slowdown": the slowdowns are not that big
and the expectation is that most of them will be work-around as this is implemented.
17:18 < Eighth_Doctor> fweimer: because partners and customers don't know what
any of it means
17:18 < zbyszek> re "no interest": there is clearly interest from people
who care and use this. That people who don't use it and don't care don't care,
doesn't mean much.
17:18 -!- anitazha [~anitazha@2001:470:69fc:105::fd32] has joined #fedora-meeting
17:18 -!- mode/#fedora-meeting [+v anitazha] by ChanServ
17:19 < fweimer> Eighth_Doctor: That's extremely condescending. I assume they
have learnt how to work with the tools we provide.
17:19 < Eighth_Doctor> and we're going to take the hit for Python anyway in 3.12
most likely
17:19 < Eighth_Doctor> which is actually where the biggest pain is
17:19 <+davide> .hello dcavalca
17:19 < zbyszek> Eighth_Doctor: … python upstream will work on making the pain go
away.
17:19 < codonell> zbyszek, Eighth_Doctor, The slowdowns are real. There is real
register pressure on x86_64, that is different from other architectures that have
different ISA designs.
17:19 < Eighth_Doctor> fweimer: it's borne from experience... this stuff is
weirdly deep and arcane
17:20 < michel-slm> .hi
17:20 < Eighth_Doctor> and a lot of it is cargo culting
17:20 -!- anitazha [~anitazha@2001:470:69fc:105::fd32] has left #fedora-meeting []
17:20 -!- anitazha [~anitazha@2001:470:69fc:105::fd32] has joined #fedora-meeting
17:20 -!- mode/#fedora-meeting [+v anitazha] by ChanServ
17:20 <+DaanDeMeyer[m]> codonell: We don't have any glibc patches for fixing the
assembly frame pointers issues, we run upstream glibc as far as I'm aware
17:20 < codonell> Eighth_Doctor, What is cargo culting? That we should not enable
frame pointers?
17:20 < decathorpe> Would it be possible to do something similar to what eln does
with rawhide? I.e. rebuild (a relevant subset of) rawhide in rawhide-fp? That would let
people opt-in to these packages if they need to. And when / if the worst problems have
been mitigated, it could be reconsidered for a default change?
17:20 < Eighth_Doctor> Fabio Valentini: it has to be the whole distro
17:20 < Eighth_Doctor> otherwise it's not useful for developers
17:20 <+davide> yeah, a subset isn't actually useful
17:20 < Eighth_Doctor> especially desktop devs
17:21 < michel-slm> if the register pressure is heaviest on x86_64, there's also
the possibility of enabling it for other arches first, right
17:21 < Eighth_Doctor> codonell: -O2 vs -O3, no frame pointers, other compilation
flags we have
17:21 <+sgallagh> Fabio Valentini: FWIW, if we wanted to add a new target to Koji
for this, the tools I wrote for ELN would work just fine for those builds.
17:21 < Eighth_Doctor> most people have no clue what that even means and lots of
options are used or unused based on lore
17:21 < decathorpe> Davide Cavalca: I think you missed the "relevant"
specifier for "subset" ...
17:21 < fweimer> michel-slm: It's already enabled for aarch64 & ppc64le.
17:22 < Eighth_Doctor> michel-slm: it's already on for aarch64 and power
17:22 <+sgallagh> michel-slm: From what was said above, the other arches already
have frame pointers
17:22 < Eighth_Doctor> the missing ones are x86 and s390x
17:22 < decathorpe> Stephen Gallagher: right, that's why I thought that this
might be a way to salvage all the work that went into this proposal
17:22 < michel-slm> I wonder what the consideration is for s390x
17:22 < nirik> right, thus my question... could we just ask folks to use those
arches for perf work? or does the kind of things they are doing not really translate?
17:22 <+davide> Fabio Valentini: that's my point, "relevant" in this
context is the whole distro, due to the dependency web for any non-trivial application
17:22 <+davide> this is especially true for continuous profiling applications
17:23 < decathorpe> I meant: excluding stuff like noarch packages and ocaml which
doesn't support frame pointers ...
17:23 < zbyszek> nirik: you can't really do profiling of desktop stuff and such…
Access to ppc64el and arm64 machines is hard for most poeple.
17:23 <+davide> ah got it, thanks for clarifying
17:23 < nirik> rpi4 seems like it would be pretty doable, but sure...
17:23 < codonell> Eighth_Doctor, They aren't "missing", the x86 and
s390x choices are specific.
17:24 < Eighth_Doctor> rpi4 is useless for profiling for other reasons
17:24 <+music[m]> Yeah, if I'm J. Random Developer, I'm definitely not going
to go buy an exotic workstation for profiling. And rpi4 is going to have different
bottlenecks from a desktop-class system.
17:24 < Eighth_Doctor> it's a horrible arm64 platform
17:24 <+music[m]> Much less a server-class system.
17:24 < nirik> music[m]: would you install a fedora remix?
17:24 <+davide> profiling work is pretty system specific, I don't think it's
realistic to expect folks to keep a rpi on their desk for this, and the results
wouldn't be comparable anyway
17:24 < Eighth_Doctor> codonell: yeah, if that's true, they're not
documented
17:25 < Eighth_Doctor> I literally only was able to learn about why we have the omit
frame pointer thing from Brendan Gregg
17:25 < Eighth_Doctor> nobody has docs anywhere about it
17:25 <+DaanDeMeyer[m]> Even a remix wouldn't work, because it's a different
system, you can't even ask a bug reporter to do a profile for you and provide the
results because they'd have to switch to the remix
17:25 < codonell> Eighth_Doctor, Would you like me to document that somewhere?
17:26 < Eighth_Doctor> yes, please
17:26 < Eighth_Doctor> document all of our compilation options and their rationales
17:26 <+music[m]> nirik: if a remix existed, i think it would be of some use, but
very few people would download and install in a VM to do profiling in practice
17:26 < Eighth_Doctor> I only know the cargo-cult-y reason why we've never done
-O3, not the real reason why we never tried, for example
17:27 <+music[m]> i think that it's likely a remix it would not be maintained
long-term in practice
17:27 < nirik> yep, its a lot of work
17:27 -!- jontyms2707749 [~jontyms(a)76-242-160-235.lightspeed.dybhfl.sbcglobal.net] has
quit [Read error: Connection reset by peer]
17:27 < dcantrell> looking at other distributions and platforms, profiling builds of
everything have usually been offered alongside the "regular" builds. the
general understanding being they are slower but they are also available for those
interested
17:27 < Eighth_Doctor> also there's legal problems with remixes
17:27 < Eighth_Doctor> in that the whole trademark policy is a mess and nobody knows
how to deal with it right now
17:27 < fweimer> codonell: A while back, I documented all the Fedora-specific
choices (buildflags.md). But this one is not Fedora-specific, it comes straight from
upstream.
17:27 -!- jontyms2707749 [~jontyms(a)76-242-160-235.lightspeed.dybhfl.sbcglobal.net] has
joined #fedora-meeting
17:27 <+davide> a remix also doesn't help you if you're currently running
Fedora, have an issue in production and want to use profiling to troubleshoot
17:27 <+music[m]> and as daandemeyer pointed out, you're either installing the
remix on different hardware or on a vm; in either case it isn't the original system of
interest
17:28 < mhroncok> brainstorming: would it make sense to enable this in rawhide only
and evaluate the results in time for a quick revert and rebuild before F38 beta?
17:28 <+davide> it's not realistic to switch a system on the fly to the remix,
and a separate system with the remix wouldn't necessarily help in troubleshooting (or
provide the same results)
17:28 < dcantrell> yeah, not really in favor of that
17:28 < fweimer> Eighth_Doctor: -O3 is not generally faster than -O2. GCC 12 did
enable auto-vectorization at -O2 though, with a cost model that is beneficial.
17:28 < mhroncok> (rather than having it as an experiment for 1 year which indeed
seems rather long)
17:28 < nirik> mhroncok: would need a mass rebuild...
17:28 < fweimer> So there is movement.
17:28 < Eighth_Doctor> we get observability principle problems when you do that
17:28 < michel-slm> mhroncok: you'll need an extra mass rebuild, yeah
17:28 < codonell> Eighth_Doctor, The knowledge is largely maintained by downstream
and IHV toolchain teams, and the CPU design teams that support them. We are here to
support today's decision with that knowledge and experience.
17:29 < gotmax> Well, it would be possible to distro-sync a regular rawhide system
to the theoretical rawhide-fp repos w/o a reinstall.
17:29 < zbyszek> gotmax: you'd still likely get different versions of packages.
17:30 < Eighth_Doctor> one outcome I'd prefer to see is figuring out how to make
-fno-omit-frame-pointers more performant if that's the core issue
17:30 <+music[m]> agree with fweimer re: -03 and Davide Cavalca re: remixes
17:30 < zbyszek> Also… not everybody runs rawhide everywhere, so we're back to
the square that we want to do this on production systems and not a fresh system.
17:30 <+sgallagh> Note: we are at 20 minutes on this topic.
17:30 < zbyszek> So… considering that we need a mass rebuild to really see the
effects, and we are worried that some specific applications might be negatively impacted,
17:30 -!- jontyms2707749 [~jontyms(a)76-242-160-235.lightspeed.dybhfl.sbcglobal.net] has
quit [Read error: Connection reset by peer]
17:30 < Eighth_Doctor> like register pressure is a real thing, I get it, but at the
same time, having Linux be worse than Windows and macOS is bad too
17:30 <+sgallagh> Does anyone want to make a proposal more detailed than
"accept or reject the Change"?
17:30 < gotmax> zbyszek: Fair point about not running rawhide, but automatic
rebuilds should keep versions in sync if they're done properly.
17:31 < mhroncok> can we do some preliminary vote to see if the discussion even
makes sense?
17:31 < fweimer> Eighth_Doctor: There's also the option to find a way to create
useful flamegraphs *without* frame pointers.
17:31 < Eighth_Doctor> gotmax: with what tooling?
17:31 < fweimer> I don't understand why that's not really on the table.
17:31 < gotmax> There's the ELN tooling that sgallagh brought up
17:31 <+sgallagh> Conan Kudo: presumably they'd use DistroBuildSync like ELN
does
17:32 < Eighth_Doctor> fweimer: does anyone care in the toolchain team to enable
that? my understanding from gnome developers is that they've been brushed off on this
for over a decade
17:32 < nirik> fweimer: is anyone actually working on that?
17:32 < Eighth_Doctor> so unless there's a change in attitude there, then 🤷♂️
17:32 < Eighth_Doctor> and there's also problems at the kernel level, since no
dwarf unwinding is ever going to happen
17:33 -!- jontyms2707749 [~jontyms(a)76-242-160-235.lightspeed.dybhfl.sbcglobal.net] has
joined #fedora-meeting
17:33 < Eighth_Doctor> Stephen Gallagher: it doesn't do build chains properly,
though, I see that fail a lot
17:33 <+sgallagh> That's a tangent and can be discussed IF we end up taking that
approach
17:33 < codonell> Eighth_Doctor, The kernel has specific tradeoffs that make it
choose Orc or CTF. Those formats are growing in size every year, approaching Dwarf in
complexity as they attempt feature parity :-)
17:34 <+sgallagh> Last change to make a more detailed proposal, else I'm holding
the Accept or Reject vote in 60s
17:34 < nirik> FWIW, i'm -1 to the change as is, open to revisiting if python is
no longer so affected/benchmarks show its not as bad.
17:34 < zbyszek> proposal: enable this for F38 and watch out for performance
regressions. If there are big performance regressions that can't be resolved, we can
quickly rebuild specific packages with -fomit-frame-pointer, and if the overall outlook is
bad, revert it altogether.
17:35 < Eighth_Doctor> codonell: yeah, but until they do dwarf, we're stuck
17:35 < nirik> zbyszek: by 'enable' you just mean for things that rebuild?
17:35 -!- besser82[m] [besser82@fedora/besser82] has quit [Quit: Freedom, Friends,
Features, First --->
https://getfedora.org]
17:35 < zbyszek> By "enable" I mean make the change.
17:35 < codonell> Eighth_Doctor, I have no objection to setting up a collaboration
with GNOME if they need support form toolchain people to implement what they need.
17:35 < gotmax> zbyszek: Here we come back to the people don't run rawhide
problem. Issues will likely come up later in the release cycle when it's kind of late
to revert things.
17:35 < zbyszek> Flip the switch.
17:35 < mhroncok> zbyszek: does your proposal consider that Python 3.11 would
opt-out?
17:35 <+sgallagh> zbyszek: Or do you mean mass rebuild
17:35 < fweimer> Eighth_Doctor: We could work with GNOME. We've put in a lot of
effort to get fast backtraces going over the years.
17:35 < zbyszek> OK, let me rephrase.
17:35 < mhroncok> and, who "watches out for performance regressions"?
17:36 < fweimer> It's just that it's not about frame pointers, so it gets
dismissed instanatly by many.
17:36 < Eighth_Doctor> fweimer: and KDE? because they have similar issues
17:36 < codonell> Eighth_Doctor, "Brushed off" is certainly not how we
want to engage with users of our tooling.
17:36 <+DaanDeMeyer[m]> zbyszek: As mentioned in the fesco ticket, we're in
favor of this approach
17:36 <+davide> the Change owners would be happy to help investigate performance
regressions that arise from this
17:37 <+music[m]> as mhroncok said, the hard part is, who is going to do the work to
find the regressions before release
17:37 < zbyszek> proposal: Flip the switch as proposed for F38 and watch out for
performance regressions. If there are big performance regressions reported that can't
be resolved, managers can opt-out for specific packages and rebuild with
-fomit-frame-pointer, and if the overall outlook is bad, we can consider reverting the
change altogether.
17:37 <+davide> and we had already started doing so for things like Python that had
already been surfaced
17:37 < fweimer> Eighth_Doctor: Well, the plan is to enable this in the perf APIs,
so it shouldn't matter much which frontend you use.
17:37 < decathorpe> problem is that most things will only apply the change after
mass rebuild, and at that point we'd need a second mass rebuild to revert it.
17:37 * nirik agrees with decathorpe
17:38 < zbyszek> decathorpe: yeah, but we don't need to rebuild everything.
It's most likely to be some small subset of packages that have problems. Maybe python,
maybe glibc.
17:38 <+davide> when is the mass rebuild planned for?
17:38 < dcantrell> I feel we do way too many mass rebuilds as it is
17:38 < Eighth_Doctor> fweimer: in practice, that's not fully true
17:38 <+sgallagh> zbyszek: "Flip the switch" isn't sufficiently clear.
Do you mean "change the macro and let things rebuild when they happen'" or
do you mean "run a mass-rebuild to ensure everything is built with this flag"
17:38 < zbyszek> The former.
17:38 < Eighth_Doctor> dcantrell: we should be able to do so many that they're a
nonevent
17:38 < Eighth_Doctor> the reason they're hard for us is that our tooling makes
it hard
17:38 < dcantrell> Eighth_Doctor: I disagree
17:38 <+sgallagh> zbyszek: -1
17:38 < fweimer> sgallagh: Flip the switch is clear, but it doesn't mean that
you get frame pointers.
17:38 < nirik> Wed 2023-01-18 is mass rebuild for f38
17:39 < fweimer> sgallagh: So it's not clear to me whether the proposal is about
getting frame pointers, or just some textual change to redhat-rpm-config.
17:39 * nirik also notes few people will be looking at this over the holidays either
17:39 < dcantrell> nirik: a very valid point
17:39 < gotmax> I don't think changes that we plan to revert if things go wrong
are helpful. If there's not confidence in them, they probably shouldn't be
approved.
17:39 < dcantrell> gotmax: I agree
17:40 <+sgallagh> We're now at 30 minutes. Let's at least see where votes
fall on a strict "yes or no" vote on the Change.
17:40 <+sgallagh> I'm -1 at this time
17:40 < dcantrell> -1
17:40 < Eighth_Doctor> +1
17:40 < zbyszek> +1
17:40 < gotmax> People were unhappy about this part of the SHA-1 change. I don't
see why this should be treated differently.
17:40 < decathorpe> 0
17:41 < nirik> -1
17:41 < Eighth_Doctor> I don't find the toolchain team's arguments
compelling without a credible alternative from them
17:41 < mhroncok> -1
17:41 < Eighth_Doctor> and so far, we have nothing, and a lot of suffering
developers who can't do perf work
17:42 <+sgallagh> I count (+2, 1, -4) so far.
17:42 <+sgallagh> mhayden ?
17:42 < Eighth_Doctor> in many respects, this is more useful than annobin, package
notes, and all the other weird changes we've made to the compiler stack
17:42 < Eighth_Doctor> and both of those examples have regularly broken the
toolchain
17:43 <+sgallagh> (Note that a 0 vote is functionally a -1 here, since it
doesn't result in a change)
17:43 < zbyszek> Eighth_Doctor: yeah, this has the potential to be quite useful, and
we won't actually see this potential until it's been available for a while and
people learn to use it more widely.
17:43 -!- daandemeyer [~daandemey@2a02:a03f:864b:8201:e534:34f4:1c34:8de7] has joined
#fedora-meeting
17:43 <+sgallagh> It's not possible for this to pass, given the votes it has
received.
17:44 < fweimer> We'll raise it with our IHVs nevertheless.
17:44 < Eighth_Doctor> laying the cards on the table: I don't even like this
change because I don't like taking a perf hit
17:44 < Eighth_Doctor> but what I like even less is how badly desktop developers get
screwed for development tooling
17:45 < Eighth_Doctor> I know Meta wants this for perfing their workloads, but I
want it to make desktop developers' lives better
17:45 < mhroncok> fweimer: What does IHVs mean?
17:45 < michel-slm> indep hardware vendor?
17:45 < Eighth_Doctor> mhroncok: Independent Hardware Vendors
17:45 < nirik> Hopefully we get some better options for them...
17:45 < fweimer> mhroncok: Independent Hardware Vendors (silicon in this case)
17:45 <+sgallagh> mhayden, music: Last chance to vote
17:46 < mhroncok> cool, I was confused because I found the Institute of Human
Virology
17:46 < Eighth_Doctor> nirik: I'm honestly not optimistic
17:46 < mhroncok> is it important to get this now?
17:46 < Eighth_Doctor> desktop Linux investment is lower than it has ever been
overall
17:46 < mhroncok> why don't we wait for the Python 3.12 situation to unfold
17:46 <+sgallagh> #agreed REJECTED - Add -fno-omit-frame-pointer to default
compilation flags (+2, 1, -4)
17:46 < mhroncok> and reconsider this change then?
17:46 < Eighth_Doctor> I just don't see anyone caring enough to add new tools to
make their lives better
17:47 <+sgallagh> A new proposal can be submitted at a later date if new
information comes to light
17:47 < mhayden> sorry y'all, got a new puppy and had a bathroom emergency 🤦♂️
17:47 < Eighth_Doctor> if I were skilled in such a way, I'd do something about
it :(
17:47 < michel-slm> mhayden: aww, congrats
17:47 < Eighth_Doctor> but I'm not
17:47 <+sgallagh> #2870 Change proposal: Replace DNF with DNF5
17:47 <+sgallagh> .fesco 2870
17:48 * decathorpe waits for music to finish typing
17:48 <+music[m]> I didn't get my vote in quickly enough, but I'm going to
go on record as weakly in favor (despite compelling arguments from both positions), for
similar reasons to Conan Kudo .
17:48 <+sgallagh> #info music votes +1, belatedly
17:48 <+sgallagh> I'll include that in the tally on the ticket
17:48 < Eighth_Doctor> it'd be almost a tie if mhayden voted in favor :P
17:48 <+sgallagh> Let's move on, as we're already approaching the top of the
hour
17:49 < mhroncok> dnf -- I more or less agree with what Stephen Gallagher commented
recently in the ticket
17:49 < gotmax> It seems zodbot didn't respond to the .fesco command...
17:49 <+sgallagh> .fesco 2870
17:49 < nirik> on this one, theres some dispute about the replacement?
17:50 * dcantrell is catching up on the ticket
17:50 < Eighth_Doctor> meh, sure I'll go with what Stephen Gallagher said
17:50 < Eighth_Doctor> I don't know if I want machine-readable output from dnf
though
17:50 < Eighth_Doctor> that would encourage bad things from people
17:50 < Eighth_Doctor> I'd rather they use the API and build things around it
17:50 < decathorpe> problem: the API is horrible, CLI is not as bad
17:50 -!- jontyms2707749 [~jontyms(a)76-242-160-235.lightspeed.dybhfl.sbcglobal.net] has
quit [Read error: Connection reset by peer]
17:50 < Eighth_Doctor> the only reason zypper has machine readable output is because
zypper's API is absolute crap
17:51 < mhroncok> OTOH I am afraid that the change owners are not committed to
"to at least enumerate the major pieces that will need to be modified (infra and
releng in particular) as part of the Change page"
17:51 <+sgallagh> Let's not go into the weeds on that request.
17:51 < Eighth_Doctor> Stephen Gallagher: you put it in there :P
17:51 <+sgallagh> jmracek: Are you around?
17:51 < decathorpe> mhroncok: I agree, detemining change impact is not job for the
community
17:51 < gotmax> They have updated the scope section a bit
17:51 <+sgallagh> I did, but not as something conditional for approval
17:51 < jmracek> Yes
17:52 <+sgallagh> That may have been unclear
17:53 -!- jontyms2707749 [~jontyms(a)76-242-160-235.lightspeed.dybhfl.sbcglobal.net] has
joined #fedora-meeting
17:53 -!- daandemeyer [~daandemey@2a02:a03f:864b:8201:e534:34f4:1c34:8de7] has quit [Quit:
Client closed]
17:54 <+sgallagh> jmracek: If you and your team could do the legwork to at least
identify the major infra and releng pieces that might be broken and need to be kept in
mind, I think that would be sufficient for me to vote +1 here.
17:54 * nirik is in favor of the idea of this change, but not sure about some of the
details/etc
17:54 <+sgallagh> Could you commit to that?
17:54 < jmracek> I tried my best to answer all the questions in all chanels and I
improved proposal according to suggestions. Hope I succeed
17:54 < decathorpe> "Repoquery command with unknown scope" lists that
repoquery options that are essential for Fedora development are still missing, things like
"--whatrequires" and "--requires" ... that doesn't really make me
confident
17:54 < Eighth_Doctor> I would not vote until we have that list enumerated to
evaluate
17:54 < Eighth_Doctor> cart before the horse after all
17:55 <+music[m]> decathorpe: Where are you reading this from?
17:55 < gotmax>
https://fedoraproject.org/wiki/Changes/ReplaceDnfWithDnf5#Scope
17:55 < decathorpe> music:
https://github.com/rpm-software-management/dnf5/issues/122
17:56 < jmracek> python3-dnf will be arrond, dnf-3 binary, therefore minimal list is
quite short
17:56 < decathorpe> the list of things that are still missing compared to dnf4 looks
rather bleak right now
17:56 < jmracek> The list of all dependencies is in proposal
17:56 -!- anakryiko [~anakryiko(a)163.114.132.128] has quit [Quit: Client closed]
17:56 < dcantrell> the proposal presents a lot of the work items and lacks a lot of
"how this will look to the end user" details. that's what I would like to
see. emphasis on minimizing impact and not breaking things, obviously
17:57 < gotmax> jmracek: That's true, but if dnf5 is the default, "dnf-3
still exists" is not a solution to "there's a lot of missing
functionality"
17:57 < decathorpe> yeah,dependencies are listed, but not dependents
17:57 < jmracek> We got a minimal responce from comunity what they need
17:58 <+sgallagh> Yes they are
17:58 < jmracek> We have some time to delivery missing functionality
17:59 <+sgallagh> My biggest concerns are the pungi, lorax and mock use-cases.
17:59 < jmracek> Without the approval, no one will care about DNF5
17:59 < gotmax> Perhaps there should be another release in between "dnf5 is
available in the repos" and "dnf5 is the default"?
17:59 <+sgallagh> User-interaction is honestly low on my list; users are more
adaptable
17:59 <+music[m]> Maybe I misunderstand that statement, but I'm not sure that
keeping existing features should be considered "opt-in" based on feedback from
the relatively small group of people that are paying attention to dnf5 at this point.
17:59 < decathorpe> The way I see it, voting on making dnf5 take over from dnf
before it can make essential functionality work is way premature
18:00 <+sgallagh> gotmax: Reading between the lines, I assume that DNF5 is wanted
for RHEL 10, which is expected to branch from Fedora 40, IIRC
18:00 < Eighth_Doctor> I'm also not super pleased about them ripping out the
abbreviations from dnf5
18:00 < gotmax> Eighth_Doctor: You mean the zypper style ones?
18:00 < Eighth_Doctor> yes
18:00 <+sgallagh> And I for one would not want RHEL 10 to ship without at least one
full Fedora release to bake it first
18:01 < jmracek> I don't say that the change will be not painfull, but this is
not the first time. Right now only tools that use commandline will be directly affected,
but DNF5 will be compatible in many aspects
18:01 < gotmax> sgallagh: Sure, but Fedora makes decisions based on its interests
first and foremost.
18:01 < nirik> this is for f39 right?
18:01 < Eighth_Doctor> Stephen Gallagher: yes... that's been a point in the last
couple dnf community meetings
18:01 <+music[m]> I feel like I don't have a good idea of what kind of
functionality dnf5 will and won't have by the time it would take over as default.
18:02 < nirik> music[m]: yeah.
18:02 < Eighth_Doctor> nirik: yes
18:02 <+sgallagh> gotmax: I try not to make decisions in a vacuum. (Particularly
since RHEL is my $DAYJOB).
18:02 < gotmax> <jmracek> Without the approval, no one will care about DNF5 |
I don't think that's necessarily true
18:02 < jmracek> mock can already test and we are in contact
18:03 < jmracek> If I am not mistaken pungi, lorax uses python3-dnf (dnf api)
therefore not effected
18:03 < nirik> in releng land: releng scripts call dnf, pungi needs to work, koji of
course needs to work (meaning mock, etc too), the weird image and container stuff we have
18:04 < Eighth_Doctor> kiwi needs to work for fedora asahi and hyperscale stuff
18:04 < Eighth_Doctor> but that I expect will be fine
18:04 < Eighth_Doctor> since kiwi and dnf have been working together for a while
18:04 <+music[m]> can we clarify @nirik's question about F39 vs F38? there was
discussion in the issue about doing this for F39, but the Change page is for F38
18:04 < decathorpe> service which provides broken dependencies report for the
packager dashboard uses dnf repoclosure, which seems to be gone entirely
18:04 < jmracek> You can agree and we can set that those tools must work at certain
time point. Is it fair for you?
18:04 <+sgallagh> I'm also somewhat concerned that FESCo members are falling
prey to the "nirvana fallacy"
18:04 <+sgallagh> music: The Change page says F39
18:05 < Eighth_Doctor> well, I have to go now... dental appointment awaits
18:05 < Eighth_Doctor> bye y'all
18:05 < dcantrell> take care
18:05 < nirik> jmracek: it would definitely be good to try and get things all
working before f38 cycle... if possible.
18:05 < nirik> f39 I mean
18:05 < nirik> Eighth_Doctor: ouch. Good luck.
18:05 <+music[m]> Ah, I was in the wrong place. Got my tabs straightened out.
Thanks.
18:05 < dcantrell> the previously mentioned suggestion of landing dnf5 parallel for
one release and then have a separate change to switch over to it as the default would make
me feel better
18:06 < decathorpe> but aren't we back to "we shouldn't enable it if if
we're not confident that it can work", aren't we?
18:06 < dcantrell> switch over to it in the following release, that is
18:06 < gotmax> dcantrell: Isn't that what we already have?
18:06 <+sgallagh> There's no need to have a Change for F38 to land it
18:06 < zbyszek> dcantrell: that's already happening.
18:06 < mhroncok> I am sorry but in my worldview if somebody proposes to change
things, they evaluate the impact and propose a solution to the negative impact.
18:06 < zbyszek> dcantrell: it's already in rawhide, i.e. it'll be in F38 if
nothing else changes.
18:07 < dcantrell> alright, so this change proposal isn't even that clear to me
then
18:07 < decathorpe> yeah, I don't think "let's approve this and wait
for people to come screaming because everything is on fire" is a good approach
18:07 < jmracek> In Fedora 38, DNF5 replaces microdnf
18:07 <+sgallagh> dcantrell: What are you missing?
18:07 < dcantrell> decathorpe: agreed
18:07 <+sgallagh> Fabio Valentini: That's different from other approvals...
how?
18:07 <+sgallagh> (Sorry)
18:08 < decathorpe> for other Changes, impact and scope are defined much clearer
18:08 < decathorpe> here we're talking about replacing the package manager,
which affects everything and everybody
18:08 < gotmax> dcantrell: The change is to obsolete the current dnf and replace the
dnf binary with a symlink to dnf5
18:08 <+sgallagh> Well, we're talking about the basic package manager. They
could just write "everything" and it wouldn't really be a lie
18:09 < dcantrell> ok, so doing that without existing functionality implemented
seems like a pretty easy decision to me. don't make that change until everything we
need works
18:09 < decathorpe> true. so there should also be a higher burden to make impact and
scope clearer ...
18:10 < nirik> isn't 'everything we need working' what the proposal
says? or in not enough detail? I am not sure how easy it is to define 'everything we
need working'
18:10 <+sgallagh> dcantrell: And we're back to the Nirvana Fallacy
18:10 < jmracek> I tried to get the list of required functionality from comunity an
all requirements were mentioned in proposal.
18:10 < zbyszek> Another thing which is not clear to me: if tools that depend on
python3-dnf continue to use that, but other tools use dnf5, how likely are we to get
unexpected discrepancies in behaviour?
18:10 -!- jontyms2707749 [~jontyms(a)76-242-160-235.lightspeed.dybhfl.sbcglobal.net] has
quit [Read error: Connection reset by peer]
18:10 < decathorpe> Stephen Gallagher: isn't approaching Nirvana a desirable
thing? I don't know that idiom.
18:10 < gotmax> Yeah, currently the contingency plan section is blank. At the very
least, the proposal should say that the symlink will only be changed over *after* the
listed features are implemented.
18:11 < nirik> jmracek: I am sure we will hit things we didn't know
about/notice/realize. ;)
18:11 <+sgallagh> Fabio Valentini:
https://en.wikipedia.org/wiki/Nirvana_fallacy
18:11 < dcantrell> I think it can be further grouped by must haves and nice to
haves.
18:11 < jmracek> We are developing RFE according to priority and community
requirements.
18:11 < decathorpe> jmracek: "I tried to get the list of required functionality
from comunity" how did you ask for community feedback? Community Blog post? Fedora
Magazine Article?
18:11 < jmracek> nirik: sure and we are ready to face that chalenge
18:12 < gotmax> I started that mailing list thread about dnf5 blockers, because I
felt the community's feedback wasn't being collected
18:12 < dcantrell> nirik: I would say that the required functionality to make a
release should be a must have, as an example
18:12 < jmracek> We already have 2 fedora-devel topics
18:12 < zbyszek> jmracek: I see "The scope of the features for Fedora 39".
This is pretty clear. But there's also long list in "Dependencies". What is
the plan for those?
18:13 < jmracek> Upstream have open discussions and issues.
18:13 -!- jontyms2707749 [~jontyms(a)76-242-160-235.lightspeed.dybhfl.sbcglobal.net] has
joined #fedora-meeting
18:13 < jmracek> I had a presentation about DNF5
18:13 < bcotton> The F39 System-Wide Change proposal deadline is 2023-06-27. Could
we defer this decision until, say, the F38 Beta release and see where dnf5 stands at that
point?
18:13 < jmracek> I think it was Fedora Hatch
18:14 < gotmax> <jmracek> Upstream have open discussions and issues. |
That's true, but I don't think that counts as *collecting*
18:14 < decathorpe> jmracek: these are all things with pretty limited / targeted
audience ... and probably won't reach the community.
18:14 < jmracek> I am open for any ideas
18:14 < nirik> bcotton: or we could approve it now, but add a 'check in' for
then to see how it's going?
18:15 < dcantrell> I am losing my ability to think. It's past lunch time here
and I had breakfast at 6am
18:16 < gotmax> jmracek: decathorpe already listed some
18:16 < decathorpe> that would mean we need to know what amount of
"brokenness" would be enough to result in a revert to dnf 4, right?
18:16 < jmracek> If we will be unable to deliver what we promise the proposal can be
always postponed
18:16 <+sgallagh> Let me rephrase the question: is anyone opposed to including DNF5
as the default package manager in F39, assuming it meets certain TBD requirements?
18:16 < gotmax> Starting a forum thread might also be a good idea
18:17 < decathorpe> Stephen Gallagher: depends on the TBD requirements. that
question doesn't make sense to me
18:17 < mhroncok> Stephen Gallagher: such question is potentially useless.
18:17 < zbyszek> sgallagh: would "TBD" be resolved before we make the
vote?
18:17 < gotmax> sgallagh: In addition to having the requirements properly
enumerated, I'm worried that there won't be enough testing by that point.
18:17 <+sgallagh> The intent of the question is to determine if we're quibbling
over the details or the plan as a whole
18:18 < decathorpe> if "TBD requirements == it's perfect", then
I'm +1. otherwise, I'm leaning towards the negative
18:18 < decathorpe> :D
18:18 <+sgallagh> If we have general agreement that we want DNF5 to happen and just
disagree on specifics of how, that'
18:18 < mhroncok> exactly
18:18 < zbyszek> sgallagh: I think we're quibbling over the *lack* of details.
18:18 <+sgallagh> s very different from "we are opposed to DNF 5"
18:18 -!- fweimer [~fweimer(a)albireo.enyo.de] has quit [Quit: .]
18:19 < zbyszek> I don't want to agree to a switch when we can't answer
basic questions about what tools we use to build and manage the distro will work.
18:19 < decathorpe> Stephen Gallagher: I don't think that makes much sense.
people working on packaging tools clearly want to ship DNF5. if we say "we don't
want that" we're going to be stuck with an unmaintained dnf4, aren't we?
18:19 < mhroncok> zbyszek: +1
18:19 < dcantrell> zbyszek: +1
18:19 < gotmax> sgallagh: I don't think the problem is opposition to dnf5,
it's compatibility and missing functionality
18:20 < zbyszek> decathorpe: Yeah, I think the switch is inevitable in the long run.
But it may be better for Fedora to *delay* the switch if some important things are
missing.
18:20 <+sgallagh> You're all demanding extra information, but is there literally
any information that would change your ultimate decision about allowing DNF5?
18:20 < jmracek> I guess that this is a requirement - tools we use to build and
manage the distro will work
18:20 < dcantrell> sgallagh: a switch to zip files
18:20 <+sgallagh> OK, then the question is not "Should we switch?" but
"When should we switch?"
18:20 < mhroncok> I don't think there is anybody on FESCo who would argue that
we should avoid eventually switching to dnf5
18:21 < jmracek> DNF team will not release DNF5 by the way that we will break
Fedora
18:21 < gotmax> How do you even define breaking Fedora?
18:22 < mhroncok> does "tools we use to build and manage the distro"
include tools our packagers use?
18:22 <+sgallagh> mhroncok: So what we're really determining is "what
requirements should we set to switch in F39 vs. deferring to F40?"
18:22 < jmracek> Without setting firm plan we can repeate the issue with yum,
because it remained unsupported in distro for too long
18:22 < dcantrell> (brb, have to get sandwich)
18:22 < decathorpe> A list of features that are present in dnf but removed from /
not implemented in / not planned for DNF5 would be great. I can't find that right now.
That would likely be more actionable than "we want to know what you need", i.e.
people can just check that list if the things they use are missing from DNF5
18:22 < gotmax> "How do you even define breaking Fedora?" | I feel people
have asked this type of question in several different ways, and it hasn't been
satisfactorily answered.
18:23 < jmracek> I am sorry, I have to disconnect
18:23 < mhroncok> Stephen Gallagher: I think that is more or less what I've been
saying since day 1
18:23 <+sgallagh> So let's provide some minimum requirements "Such as: ODCS
must be able to complete a full compose of all blocking media"
18:23 <+sgallagh> mhroncok: Maybe, but I'm trying to make sure we're all
speaking the same language.
18:23 < zbyszek> decathorpe: +
18:23 < zbyszek> decathorpe: +1
18:23 < gotmax> decathorpe: Strongly agree
18:23 < mhroncok> sure, np
18:24 < jmracek> The list is already in preparation
18:24 <+sgallagh> Another requirement: "The above compose must be comprised of
packages built from Koji using DNF5 for mock"
18:25 <+sgallagh> I don't think we want to set requirements on individual
features. I think we need use-cases that they can support.
18:25 < gotmax> Well specific features are needed to support those use cases
18:26 < jmracek> Builddep is already implemented therefore I dont see a problem with
mock. But during testing there could appear some issues, but this is normal life
18:26 -!- jmracek [~jmracek(a)185.16.81.139] has quit [Quit: Konversation terminated!]
18:26 < mhroncok> "The list is already in preparation" -- let's wait
for it? I don't think this discussion is very productive at this point
18:26 < dcantrell> (back)
18:26 < gotmax> Yup
18:26 * nirik nods
18:26 < decathorpe> +1
18:27 < gotmax> jmracek has left. Not sure if that shows up on Matrix
18:27 < decathorpe> And collecting requirements is probably also going to be more
productive once users know what they're about to lose ;)
18:28 <+sgallagh> jmracek: I'm not asking you to defend the current state. Just
trying to give you something to actually work with
18:29 * decathorpe 's brian starts to feel like mush
18:29 < dcantrell> also your brain?
18:29 < mhroncok> Fabio Valentini: unfortunately, mush is not supported in dnf5 :D
18:30 < dcantrell> :)
18:30 < decathorpe> apparently. is there something else we need to discuss today?
I'd say postpone dnf5 stuff until at least next week ...
18:30 < zbyszek> decathorpe: +1
18:30 <+sgallagh> Let's also take the FESCo interview questions to next week,
since we're well over time
18:31 -!- jontyms2707749 [~jontyms(a)76-242-160-235.lightspeed.dybhfl.sbcglobal.net] has
quit [Read error: Connection reset by peer]
18:31 < mhroncok> #info a list of things missing from dnf5 is already in
preparation, let's postpone the dnf5 discussion until we see it
18:31 <+sgallagh> +1
18:31 < mhroncok> Stephen Gallagher: we cannot do that
18:31 < mhroncok> I am afraid
18:31 * mhroncok checks
18:31 <+sgallagh> Sorry, I meant "tot he ticket"
18:31 < mhroncok> "In case the selection is not complete by 30 November, I will
use the same set of questions as the last election cycle"
18:32 < decathorpe> tag it with "OMG!Urgent!!111"
18:32 < zbyszek> We have another 4.5 h.
18:33 -!- jontyms2707749 [~jontyms(a)76-242-160-235.lightspeed.dybhfl.sbcglobal.net] has
joined #fedora-meeting
18:33 * sgallagh sighs
18:33 <+sgallagh> #2895 Election interview questions: FESCo (F37)
18:33 <+sgallagh> .fesco 2895
18:34 < mhroncok> judging by the list of candidates, we can keep the questions as
they were, copy our interviews from last year and get ourselves reelected
18:34 < mhroncok> :(
18:34 < decathorpe> mhroncok: ᕕ( ᐛ )ᕗ
18:34 <+sgallagh> Proposal: Replace the CentOS Stream question with "How do you
handle disagreements when working as part of a team?"
18:34 < zbyszek> sgallagh: +1
18:34 <+sgallagh> (as suggested by mhayden)
18:34 <+sgallagh> +1
18:35 -!- Guest34324 [~Guest3432@2001:a61:2483:c401:ceb8:afad:3f04:b9e8] has joined
#fedora-meeting
18:35 < mhroncok> I don't love "when working as part of a team" in
this context but I won't block this. +1
18:35 < nirik> same
18:35 <+sgallagh> mhroncok: Are you suggesting that FESCo doesn't work as a
team?!
18:35 <+music[m]> +1, that question may or may not be enlightening in practice but
it at least makes some kind of sense
18:36 < mhroncok> Stephen Gallagher: I am suggesting that we often need to handle
disagreements from outside FESCo
18:36 < mhroncok> so in that con text, the team is Fedora
18:36 < mhroncok> but it just reads wrong for me, no big deal
18:36 < gotmax> "disagreements within the community"?
18:37 < decathorpe> I trust that bcotton can come up with a sufficiently good
alternative formulation ;)
18:38 < mhroncok> sorry for speaking up, let's approve what we have before we
bike-shed for another hour
18:38 < zbyszek> Yes, please.
18:38 < decathorpe> it's fine with me. +1
18:38 < nirik> sure, +1
18:38 < dcantrell> +1
18:39 < mhroncok> it's not like people will actually read our answers anyway :D
18:39 < zbyszek> mhroncok: your optimism is shocking
18:39 < mhroncok> it's not me, it's the mush
18:40 <+sgallagh> #agreed Replace the CentOS Stream question with "How do you
handle disagreements when working as part of a team?" (+6, 0, -0)
18:40 <+sgallagh> #topic Next week's chair
18:40 < mhroncok> I'll do it
18:40 <+sgallagh> Thanks
18:40 <+sgallagh> #action mhroncok to chair next meeting
18:40 <+sgallagh> #topic Open Floor
18:40 <+sgallagh> Anything for Open Floor?
18:40 <+sgallagh> (Please say no)
18:41 < decathorpe> (no)
18:41 < mhroncok> no
18:41 < zbyszek> If we have a minute
18:41 < zbyszek> OK, just joking.
18:41 < zbyszek> Badly.
18:42 <+sgallagh> #endmeeting
18:42 <+sgallagh> Thanks folks
18:42 -!- pablomh [~pablomh(a)55.red-83-43-84.dynamicip.rima-tde.net] has quit [Quit:
Leaving]
18:42 <+sgallagh> Of course, I think Zodbot died somewhere in there
18:42 < mhroncok> Stephen Gallagher++
18:42 < mhroncok> I am not afraid
18:42 < decathorpe> sgallagh++
18:42 < mhroncok> *surprised
18:42 < decathorpe> zodbot--
18:42 < mhroncok> if I were zodbot, I'd die as well
18:43 < nirik> hum.
18:43 < decathorpe> see you next week o/
18:44 -!- besser82[m] [besser82@fedora/besser82] has joined #fedora-meeting
18:46 < nirik> zodbot should be back in a min... but it will not remember the
meeting.
18:46 < gotmax> I pasted my IRC logs from the FESCo meeting if someone wants:
https://paste.sr.ht/~gotmax23/3576b21a357f5da14cc5871925d1f45999197b62
I just saw gotmax's comment after pasting this in ;)
Zbyszek