So it looks like what you folks are doing is actually very similar to
what
Facebook is doing. That is interesting, and explains why some GNOME
developers are jumping on the bandwagon of this Change proposal.
To be fair, we've been complaining about it internally in GNOME for probably around a
decade. So it's not a new thing from our standpoint. What is new is that it appears
others came to the similar-in-spirit solutions.
There are certainly places where it falls down still. Things like libffi, libc,
encryption, hand-rolled assembler, etc as you mention. But since we still capture those
stack traces as counting against the proper pid, it's often enough to allow you to
dive deeper or see collateral damage. You might need to sort your callgraph a bit
differently, but it's certainly possible in sysprof given the flexibility of the
callgraph display.
Does profiling individual applications file under "profiling by
intuition"
for you? Because that is what I would expect developers to go back to if
systemwide profiling stops being viable.
I would say yes because you have to be intuitive about which application(s) or libraries
matter.
When sysprof is working correctly, you can click record and have a decent understanding of
where things are going wrong. That's a tough thing to replicate in a handful of
terminals simultaneously displaying information which likely exacerbate desktop workloads
on their own.
And is it such a problem to require the handful developers who need
to do
systemwide profiling to do that, instead of slowing down the production
installation for all users?
I think our goal should be to make it so easy that it's not just a handful of people
doing system-wide profiling like it is today.
Using sysprof (or similar tool) as the first step in triage makes a lot of sense to me
because it gives upstream a way to capture correlating information and visualize it in a
useful manner. Despite being the author of the modern incarnation of Sysprof, I'm not
against using pretty much anything else that works.
But here we are at an existential choice of what Fedora is. Are we for developers creating
the platform(s)? Are we an optimized end-user distribution? If so where do the developers
go that need to build these systems? Because it's clear to me that the status quo is
often getting in the way for some of us.