F27 System Wide Change: Parallel Installable Debuginfo

[Jan Kurik]

= System Wide Change: Parallel Installable Debuginfo =
https://fedoraproject.org/wiki/Changes/ParallelInstallableDebuginfo

Change owner(s):
* Mark Wielaard <mjw AT fedoraproject DOT org>

debuginfo packages can be installed in parallel to make it easier to
trace, profile and observe what programs are doing or to debug when
they have crashed. That way debugging, tracing or profiling programs
can be done independent of whether they are 32bit, 64bit, a slightly
newer or older version than currently installed or even from a
different architecture.


== Detailed Description ==
Currently only one version of a debuginfo package can be installed for
a given package. Even on a multi-lib system you cannot install the
64-bit and 32-bit versions of a debuginfo package in parallel
(technically you sometimes can, because of RPM file coloring, the
64bit version of the .debug files win over the 32bit version - causing
lots of confusion). But there are various situation where having
multiple versions of the debuginfo package installed help with
tracing, profiling, debugging and/or crash analysis (see the Benefit
to Fedora section below). There are various things provided by a
debuginfo file that might conflict preventing parallel installation of
different versions:

* build-id file /usr/lib/debug/.build-id/xx/yyyy...yyy which is a
symlink to the main ELF file.
* build-id.debug file /usr/lib/debug/.build-id/xx/yyyy...yyy.debug
which is a symlink to the .debug ELF file.
* The .debug files under /usr/lib/debug/ with file path names
mirroring the main ELF file paths under / with .debug added.
* The source files under /usr/src/debug/<name>-<version>/

They can be made non-conflicting in the following ways:

* The main build-id file should not be in the debuginfo file, but in
the main package (this was always a problem since the package and
debuginfo package installed might not match). If we want to make
usr/lib/debug/ a network resource then we will need to move the
symlink to another location (maybe /usr/lib/.build-id). Unfortunately
this means a change will be necessary for debuginfo consumers to that
depend on the old location. We could keep the old symlink and point it
to the new location to work around it. But I will audit the consumers
to see which depend on it and discuss if we can have a new standard
location.
* build-ids are globally unique identifiers. They will be different
across arches. But might match between minor releases if the exact
same ELF image is produced. The linker will get an option to hash in
the full nvr to make sure all build-ids are always fully unique.
* The .debug file names will be changed to main ELF file
name-vr.debug. This name will also be set in the .gnu_debuglink
section of the main file by changing the options given to eu-strip in
the rpm find-debuginfo.sh script.
* The source files will be moved under
/usr/src/debug/<name>-<version>-<release>.<arch>/. This needs
changes
to the rpm debugedit program which rewrites the DWARF source file
information.

These changes will make all files in any debuginfo file unique so they
don't conflict when installed in parallel. There should be no changes
necessary to programs (gdb, perf, valgrind, systemtap,
systemd-coredump, eu-stack, abrt-hook-ccpp, etc.) that use build-ids
or .gnu_debuglink to lookup DWARF debug information and source
references for tracing, profiling and debugging.

It would be good to tweak dnf debuginfo-install to know about parallel
installable debuginfo packages and maybe have an easy option to
install the debuginfo for a core file or for the packages running in a
container.

Alternative solutions currently rejected:

* Move main ELF image build-id file under
/usr/lib/.build-id/xx/yyyy...yyy when moving into main pages. Because
existing programs probably depend on the link being under
/usr/lib/debug/.
* Since when the build-id is identifical also the ELF file is
identical we could mark all build-id.debug files as replacable in the
rpm. It isn't clear that works for symlinks though (but we could
reverse the symlink direction from debug file to build-id file). And
currently you can identify the exact package nvr installed given just
one build-id. That would be impossible if multiple packages could
contain the same build-id/ELF image file.
* Do away with the old .gnu_debuglink way of accessing files under
/usr/lib/debug and just not install .debug files and only support
build-id based debug lookups. Because it isn't clear build-ids are
100% available and all programs work with build-id lookups instead
through .gnu_debuglink names.
* Move the .debug files under a subdir like the sources.
/usr/lib/debug/<name>-<version>-<release>.<arch>/. This cannot
easily
be expressed in .gnu_debuglink, which officially only allows a
basename.


== Scope ==
* Proposal owners: Patches have been developed against rpm debugedit
to accept a hash value to seed the build-id calculation, rewrite
source paths (currently source paths can only be smaller, this change
might create larger paths) and the rpm find-debuginfo.sh script to
change the paths, symlinks and .gnu_debuglink names as outlined in the
Detailed Description. And the dnf debuginfo-install plugin might be
patches to provide subcommands for pulling in debuginfo packages found
by build-id in core files and/or programs running in containers.

* Other developers: Upstream rpm and dnf maintainers have to review
the proposed patches. If accepted the package maintainers will have to
decide whether those patches can be backported for the next fedora
release. Once all changes are in a package debuginfo needs to be
regenerated before it becomes parallel installable. (Note most of this
has been done already during the F25 cycle. Only one outstanding patch
is not yet ready.)

* Release engineering: Needs to be discussed. In theory no changes
apart from those listed above are needed. But if we want to support
installing cross-architectures (not just multi-lib arch) debuginfo
then some way needs to be found to get those in the right repodata.

* List of deliverables: N/A (Still Unknown)

* Policies and guidelines: No changes, the debuginfo related rpm
macros won't change. They will just start producing parallel
installable debuginfo packages once all changes are in place.

* Trademark approval: N/A (not needed for this Change)

-- 
Jan Kuřík
Platform & Fedora Program Manager
Red Hat Czech s.r.o., Purkynova 99/71, 612 45 Brno, Czech Republic

[Jason L Tibbitts III]

I know the boat has sailed on this and no amount of complaining will
help at this point, but I want to make sure this gets on the record
wherever this feature is proposed.

> >>>> "JK" == Jan Kurik <jkurik(a)redhat.com&gt;
> writes: 
JK> * build-id file /usr/lib/debug/.build-id/xx/yyyy...yyy which is a
JK> symlink to the main ELF file.
JK> * build-id.debug file /usr/lib/debug/.build-id/xx/yyyy...yyy.debug
JK> which is a symlink to the .debug ELF file.
JK> * The .debug files under /usr/lib/debug/ with file path names
JK> mirroring the main ELF file paths under / with .debug added.

Please do not use a hidden directory for this.  There is no reason for
it, and it only adds confusion and rpmlint complaints that packagers
can't control at all.

 - J<

[Matthew Miller]

On Tue, Jun 20, 2017 at 01:08:17PM -0500, Jason L Tibbitts III wrote:
>  JK> * build-id file /usr/lib/debug/.build-id/xx/yyyy...yyy which
> is a
>  JK> symlink to the main ELF file.
>  JK> * build-id.debug file /usr/lib/debug/.build-id/xx/yyyy...yyy.debug
>  JK> which is a symlink to the .debug ELF file.
>  JK> * The .debug files under /usr/lib/debug/ with file path names
>  JK> mirroring the main ELF file paths under / with .debug added.
>  Please do not use a hidden directory for this.  There is no reason for
>  it, and it only adds confusion and rpmlint complaints that packagers
>  can't control at all. 
I don't know if there are any advantages to a hidden directory here,
but... at the very least we should fix rpmlint to not give errors, as
part of this change.

-- 
Matthew Miller
<mattdm(a)fedoraproject.org&gt;
Fedora Project Leader

[Mark Wielaard]

On Tue, 2017-06-20 at 14:12 -0400, Matthew Miller wrote:
>  On Tue, Jun 20, 2017 at 01:08:17PM -0500, Jason L Tibbitts III
> wrote:
>  > JK> * build-id file /usr/lib/debug/.build-id/xx/yyyy...yyy which is a
>  > JK> symlink to the main ELF file.
>  > JK> * build-id.debug file /usr/lib/debug/.build-id/xx/yyyy...yyy.debug
>  > JK> which is a symlink to the .debug ELF file.
>  > JK> * The .debug files under /usr/lib/debug/ with file path names
>  > JK> mirroring the main ELF file paths under / with .debug added.
>  
>  > Please do not use a hidden directory for this.  There is no reason for
>  > it, and it only adds confusion and rpmlint complaints that packagers
>  > can't control at all.
>  
>  I don't know if there are any advantages to a hidden directory here,
>  but... at the very least we should fix rpmlint to not give errors, as
>  part of this change. 
I believe this has always been a hidden directory name that is expected
to be called that way by various tools. The origin of the name probably
hidden somewhere in history. BTW there is a bug and a patch for rpmlint
to not complain about it already:
https://bugzilla.redhat.com/show_bug.cgi?id=1431408

Cheers,

Mark

[Jason L Tibbitts III]

> >>>> "MW" == Mark Wielaard
> <mjw(a)fedoraproject.org&gt; writes: 
MW> I believe this has always been a hidden directory name that is
MW> expected to be called that way by various tools.

I don't recall having seen it before, and I've been doing packaging work
for a very long time.  The confusion about rpmlint complaints started
when this change hit rawhide.

If it really is historical, we're through such a sea change with the way
debuginfo is being done now that we should fix things to not require a
hidden directory.  It's simply a bad idea.

"Fixing" rpmlint to complain about such things isn't really the right
answer, since the complaints are valid.

 _ J<

[Mark Wielaard]

On Tue, 2017-06-20 at 15:34 -0500, Jason L Tibbitts III wrote:
>  If it really is historical, we're through such a sea change with
> the way
>  debuginfo is being done now that we should fix things to not require a
>  hidden directory.  It's simply a bad idea.
>  
>  "Fixing" rpmlint to complain about such things isn't really the right
>  answer, since the complaints are valid. 
The rpmlint fix really is just a two line patch adding the dir name to
the expected file list.

It does seem a bit late to change the name now. And it would require
coordination with the various tools that depend on and search for
".build-id" directories now (rpm, binutils, elfutils, valgrind,
systemtap, gdb, etc.). But if there is a concrete proposal we could try
to push it upstream. Maybe do that as a separate change request though
when all upstreams have been adjusted?

Cheers,

Mark