rpm debuginfo improvements for rawhide/f27

[Mark Wielaard]

Hi packagers,

rawhide rpmbuild contains various debuginfo improvements that hopefully
will make various hacks in spec files redundant.

If you have your own way of handling debuginfo packages, calling
find-debuginfo.sh directly, need hacks for working around debugedit
limitations or split your debuginfo package by hand then please try out
rpmbuild in rawhide and read below for some macros you can set to tweak
debuginfo package generation.

If you still need hacks in your spec file because setting macros isn't
enough to get the debuginfo packages you want then please let us know.
Also please let us know about packages that need to set debuginfo rpm
macros to non-default values because they would crash and burn with the
default settings (best to file a bug against rpmbuild).

The improvements have been mainly driven by the following two change
proposals for f27 (some inspired by what other distros do):

https://fedoraproject.org/wiki/Changes/ParallelInstallableDebuginfo
https://fedoraproject.org/wiki/Changes/SubpackageAndSourceDebuginfo

The first is completely done and has been enabled by default for some
months now in rawhide. The second introduces two new macros to enable
separate debugsource and sub-debuginfo packages, but has not been
enabled by default yet. If people like the change and no bugs are found
(and fesco and releng agree) we can enable them for the f27 mass
rebuild.

If your package already splits debuginfo packages in a (common) source
package and/or sub-debuginfo packages, please try out the new macros
introduced by the second change. You can enable the standard splitting
by adding the following to your spec file:

%global _debugsource_packages 1
%global _debuginfo_subpackages 1

Besides the above two changes debuginfo packages can now (and are by
default in rawhide) build by running debug extraction in parallel. This
should speed up building with lots of binaries/libraries. If you do
invoke find-debuginfo.sh by hand you most likely will want to add
%{?_smp_mflags} as argument to get the parallel processing speedup.

If your package is invoking find-debuginfo.sh by hand also please take a
look at all the new options that have been added. Also note that almost
all options can be changed by setting (or undefining) rpm macros now.
Using the rpm macros is preferred over invoking find-debuginfo.sh
directly since it means you get any defaults and improvements that might
need new find-debuginfo.sh arguments automatically. 

Here is an overview of various debuginfo rpm macros that you can define
undefine in your spec file with the latest rpmbuild:

#
# Should an ELF file processed by find-debuginfo.sh having no build ID
# terminate a build?  This is left undefined to disable it and defined to
# enable.
#
%_missing_build_ids_terminate_build    1

#
# Include minimal debug information in build binaries.
# Requires _enable_debug_packages.
#
%_include_minidebuginfo        1

#
# Include a .gdb_index section in the .debug files.
# Requires _enable_debug_packages and gdb-add-index installed.
#
%_include_gdb_index    1

#
# Defines how and if build_id links are generated for ELF files.
# The following settings are supported:
#
# - none
#   No build_id links are generated.
#
# - alldebug
#   build_id links are generated only when the __debug_package global is
#   defined. This will generate build_id links in the -debuginfo package
#   for both the main file as /usr/lib/debug/.build-id/xx/yyy and for
#   the .debug file as /usr/lib/debug/.build-id/xx/yyy.debug.
#   This is the old style build_id links as generated by the original
#   find-debuginfo.sh script.
#
# - separate
#   build_id links are generate for all binary packages. If this is a
#   main package (the __debug_package global isn't set) then the
#   build_id link is generated as /usr/lib/.build-id/xx/yyy. If this is
#   a -debuginfo package (the __debug_package global is set) then the
#   build_id link is generated as /usr/lib/debug/.build-id/xx/yyy.
#
# - compat
#   Same as for "separate" but if the __debug_package global is set then
#   the -debuginfo package will have a compatibility link for the main
#   ELF /usr/lib/debug/.build-id/xx/yyy -> /usr/lib/.build-id/xx/yyy
%_build_id_links compat

# Whether build-ids should be made unique between package version/releases
# when generating debuginfo packages. If set to 1 this will pass
# --build-id-seed "%{VERSION}-%{RELEASE}" to find-debuginfo.sh which will
# pass it onto debugedit --build-id-seed to be used to prime the build-id
# note hash.
%_unique_build_ids      1

# Do not recompute build-ids but keep whatever is in the ELF file already.
# Cannot be used together with _unique_build_ids (which forces recomputation).
# Defaults to undefined (unset).
#%_no_recompute_build_ids 1

# Whether .debug files should be made unique between package version,
# release and architecture. If set to 1 this will pass
# --unique-debug-suffix "-%{VERSION}-%{RELEASE}.%{_arch} find-debuginfo.sh
# to create debuginfo files which end in -<ver>-<rel>.<arch>.debug
# Requires _unique_build_ids.
%_unique_debug_names    1

# Whether the /usr/debug/src/<package> directories should be unique between
# package version, release and architecture. If set to 1 this will pass
# --unique-debug-src-base "%{name}-%{VERSION}-%{RELEASE}.%{_arch}" to
# find-debuginfo.sh to name the directory under /usr/debug/src as
# <name>-<ver>-<rel>.<arch>.
%_unique_debug_srcs     1

# Whether rpm should put debug source files into its own subpackage
#%_debugsource_packages 1

# Whether rpm should create extra debuginfo packages for each subpackage
#%_debuginfo_subpackages 1

# Number of debugging information entries (DIEs) above which
# dwz will stop considering file for multifile optimizations
# and enter a low memory mode, in which it will optimize
# in about half the memory needed otherwise.
%_dwz_low_mem_die_limit          10000000
# Number of DIEs above which dwz will stop processing
# a file altogether.
%_dwz_max_die_limit              50000000

%_find_debuginfo_dwz_opts --run-dwz\\\
   --dwz-low-mem-die-limit %{_dwz_low_mem_die_limit}\\\
   --dwz-max-die-limit %{_dwz_max_die_limit}

If there are settings missing that would be useful, bugs with the
default settings or defaults that should be changed please do file a bug
report.

Thanks,

Mark

[Mark Wielaard]

Hi packagers,

On Mon, Jun 26, 2017 at 04:13:02PM +0200, Mark Wielaard wrote:
>  The second introduces two new macros to enable
>  separate debugsource and sub-debuginfo packages, but has not been
>  enabled by default yet. If people like the change and no bugs are found
>  (and fesco and releng agree) we can enable them for the f27 mass
>  rebuild.
>  
>  If your package already splits debuginfo packages in a (common) source
>  package and/or sub-debuginfo packages, please try out the new macros
>  introduced by the second change. You can enable the standard splitting
>  by adding the following to your spec file:
>  
>  %global _debugsource_packages 1
>  %global _debuginfo_subpackages 1 
rel-eng is still figuring out whether any updates are necessary to
pick up both the -debuginfo and -debugsource pacakges:
https://pagure.io/releng/issue/6863

But if you tried out the above I would really like to hear your
experiences.

>  If there are settings missing that would be useful, bugs with the
>  default settings or defaults that should be changed please do file a bug
>  report. 
There was one request for keeping a special section for rust packages.
"Need a way to exclude sections from eu-strip"
https://bugzilla.redhat.com/show_bug.cgi?id=1465997
That was implemented and should be generally useful.
It is now possible to add --keep-section=NAME or --remove-section=NAME
as _find_debuginfo_opts. Which will make sure a section is explicitly
kept in the main binary or explicitly (re)moved to the .debug file.

For completeness here are the different argument that a package can
add when using %global _find_debuginfo_opts to tweak how rpm
find-debuginfo.sh works which don't have a separate rpm macro:

# The -g flag says to use strip -g instead of full strip on DSOs or EXEs.
#
# The -r flag says to use eu-strip --reloc-debug-sections.
# (this is only really useful for kernel modules)
#
# Use --keep-section SECTION or --remove-section SECTION to explicitly
# keep a (non-allocated) section in the main executable or explicitly
# remove it into the .debug file. SECTION is an extended wildcard pattern.
# Both options can be given more than once.

The following options should normally not be given directly, but will
be used when the corresponding rpm macros are defined:

# %_smp_mflags
# The -j N option will spawn N processes to do the debuginfo extraction
# in parallel.
#
# %_missing_build_ids_terminate_build
# The --strict-build-id flag says to exit with failure status if
# any ELF binary processed fails to contain a build-id note.
#
# %_no_recompute_build_ids
# The -n flag says to not recompute the build-id.
#
# %_include_minidebuginfo
# The -m flag says to include a .gnu_debugdata section in the main binary.
#
# %_include_gdb_index
# The -i flag says to include a .gdb_index section in the .debug file.
#
# %_unique_build_ids
# If --build-id-seed SEED is given then debugedit is called to
# update the build-ids it finds adding the SEED as seed to recalculate
# the build-id hash.  This makes sure the build-ids in the ELF files
# are unique between versions and releases of the same package.
# (Use --build-id-seed "%{VERSION}-%{RELEASE}".)
#
# %_unique_debug_names
# If --unique-debug-suffix SUFFIX is given then the debug files created
# for <FILE> will be named <FILE>-<SUFFIX>.debug.  This makes sure
.debug
# are unique between package version, release and architecture.
# (Use --unique-debug-suffix "-%{VERSION}-%{RELEASE}.%{_arch}".)
#
# %_unique_debug_srcs
# If --unique-debug-src-base BASE is given then the source directory
# will be called /usr/debug/src/<BASE>.  This makes sure the debug source
# directories are unique between package version, release and architecture.
# (Use --unique-debug-src-base "%{name}-%{VERSION}-%{RELEASE}.%{_arch}".)
#
# %_find_debuginfo_dwz_opts
# %_dwz_low_mem_die_limit
# %_dwz_max_die_limit
# The --run-dwz flag instructs find-debuginfo.sh to run the dwz utility
# if available, and --dwz-low-mem-die-limit and --dwz-max-die-limit
# provide detailed limits.  See dwz(1) -l and -L option for details.
#
# %_debugsource_packages
# The -S debugsourcefiles.list option will create a debugsourcefiles.list
# file containing all source files found in the .debug packages.
# This will also exclude all source files from the debugfiles.list.

But please don't invoke find-debuginfo.sh by hand, but use the provided
rpm macros. Finally when you are invoking find-debuginfo.sh by hand then
the following can be also be used:

# A single -o switch before any -l or -p switches simply renames
# the primary output file from debugfiles.list to something else.
# A -o switch that follows a -p switch or some -l switches produces
# an additional output file with the debuginfo for the files in
# the -l filelist file, or whose names match the -p pattern.
# The -p argument is an grep -E -style regexp matching the a file name,
# and must not use anchors (^ or $).

I would be interested in feedback from packagers that do need to invoke
find-debuginfo.sh by hand. And what they believe would be necessary to
create their packages without needing to invokle find-debuginfo.sh by hand.

Thanks,

Mark

[Richard W.M. Jones]

Could this be causing:

  error: Empty %files file /builddir/build/BUILD/cduce-0.6.0/debugsourcefiles.list

in https://koji.fedoraproject.org/koji/taskinfo?taskID=20742651 ?

OCaml generates DWARF information which is picked up for debuginfo
files (and is useful), but our debuginfo tools have never been able to
locate the source files correctly -- perhaps they need to be told that
*.ml files are source files?

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine.  Supports Linux and Windows.
http://people.redhat.com/~rjones/virt-df/

[Mark Wielaard]

On Wed, 2017-07-26 at 11:37 +0100, Richard W.M. Jones wrote:
>  Could this be causing:
>  
>    error: Empty %files file /builddir/build/BUILD/cduce-0.6.0/debugsourcefiles.list
>  
>  in https://koji.fedoraproject.org/koji/taskinfo?taskID=20742651 ? 
Yes, if %_debugsource_packages is defined and the rpm debugedit cannot
find any debug sources then that error would be created. That is
obviously a bug. Someone else also just reported it. But I haven't had
time to look into it yet.

Note that that the fedora rawhide rpm package now has enabled
%_debugsource_packages by default. So you will have to %undefine
it in your spec file if you don't want a -debugsource package.

>  OCaml generates DWARF information which is picked up for debuginfo
>  files (and is useful), but our debuginfo tools have never been able to
>  locate the source files correctly -- perhaps they need to be told that
>  *.ml files are source files? 
The problem seems to be that the generated DWARF doesn't use
a .debug_line directory table but only plain file names. Those file
names are all relative to the debuginfo compile unit comp_dir attribute.
But the comp_dir is always the full absolute path:

$ eu-readelf
--debug-dump=info ./usr/lib/debug/usr/bin/cduce-0.6.0-23.fc27.x86_64.debug 2>&1 |
grep comp_dir
           comp_dir             (strp)
"/builddir/build/BUILD/ocamlnet-4.1.2/src/netsys"

This confuses rpm debugedit which assumes anything with an absolute path
is outside the build dir and so a system file belonging to another
package (e.g. /usr/include/...) and so gets excluded.

That should be fixable in rpm debugedit. I'll try to come up with a
patch.

Thanks for the report,

Mark

[Richard W.M. Jones]

On Wed, Jul 26, 2017 at 01:21:29PM +0200, Mark Wielaard wrote:
>  > OCaml generates DWARF information which is picked up for
> debuginfo
>  > files (and is useful), but our debuginfo tools have never been able to
>  > locate the source files correctly -- perhaps they need to be told that
>  > *.ml files are source files?
>  
>  The problem seems to be that the generated DWARF doesn't use
>  a .debug_line directory table but only plain file names. Those file
>  names are all relative to the debuginfo compile unit comp_dir attribute.
>  But the comp_dir is always the full absolute path:
>  
>  $ eu-readelf
>  --debug-dump=info ./usr/lib/debug/usr/bin/cduce-0.6.0-23.fc27.x86_64.debug 2>&1 |
> grep comp_dir
>             comp_dir             (strp)
>  "/builddir/build/BUILD/ocamlnet-4.1.2/src/netsys"
>  
>  This confuses rpm debugedit which assumes anything with an absolute path
>  is outside the build dir and so a system file belonging to another
>  package (e.g. /usr/include/...) and so gets excluded.
>  
>  That should be fixable in rpm debugedit. I'll try to come up with a
>  patch.
>  
>  Thanks for the report, 
Upstream OCaml doesn't generate that directive either, but I might
have a go at adding it ...

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW

[Richard W.M. Jones]

On Wed, Jul 26, 2017 at 01:21:29PM +0200, Mark Wielaard wrote:
>  On Wed, 2017-07-26 at 11:37 +0100, Richard W.M. Jones wrote:
>  > Could this be causing:
>  > 
>  >   error: Empty %files file /builddir/build/BUILD/cduce-0.6.0/debugsourcefiles.list
>  > 
>  > in https://koji.fedoraproject.org/koji/taskinfo?taskID=20742651 ?
>  
>  Yes, if %_debugsource_packages is defined and the rpm debugedit cannot
>  find any debug sources then that error would be created. That is
>  obviously a bug. Someone else also just reported it. But I haven't had
>  time to look into it yet.
>  
>  Note that that the fedora rawhide rpm package now has enabled
>  %_debugsource_packages by default. So you will have to %undefine
>  it in your spec file if you don't want a -debugsource package.
>  
>  > OCaml generates DWARF information which is picked up for debuginfo
>  > files (and is useful), but our debuginfo tools have never been able to
>  > locate the source files correctly -- perhaps they need to be told that
>  > *.ml files are source files?
>  
>  The problem seems to be that the generated DWARF doesn't use
>  a .debug_line directory table but only plain file names. Those file
>  names are all relative to the debuginfo compile unit comp_dir attribute.
>  But the comp_dir is always the full absolute path:
>  
>  $ eu-readelf
>  --debug-dump=info ./usr/lib/debug/usr/bin/cduce-0.6.0-23.fc27.x86_64.debug 2>&1 |
> grep comp_dir
>             comp_dir             (strp)
>  "/builddir/build/BUILD/ocamlnet-4.1.2/src/netsys"
>  
>  This confuses rpm debugedit which assumes anything with an absolute path
>  is outside the build dir and so a system file belonging to another
>  package (e.g. /usr/include/...) and so gets excluded.
>  
>  That should be fixable in rpm debugedit. I'll try to come up with a
>  patch. 
I think actually the compiler is correct, but there's another thing
going on.

Firstly, here's an example of the compiler being (ISTM) correct:

  $ pwd
  /var/tmp
  $ cat comp/test.ml 
  print_endline "hello world"
  $ ocamlopt.opt -g ./comp/test.ml -o ./comp/test-ocaml
  $ eu-readelf --debug-dump=info ./comp/test-ocaml |& less
   Compilation unit at offset 0:
  ...
             name                 (strp) "pervasives.ml"
             comp_dir             (strp)
"/builddir/build/BUILD/ocaml-4.04.2/stdlib"
   Compilation unit at offset 46:
  ...
             name                 (strp) "./comp/test.ml"
             comp_dir             (strp) "/var/tmp"

What's happened is that some functions from the pervasives.ml stdlib
have been statically linked into the final binary.

The same happens in the cduce binary, with a bunch of stuff being
statically linked.

However the problem is that cduce itself was compiled without the ‘-g’
flag, so none of the debuginfo refers to the main binary, only to
statically linked libraries.  That's why the paths appear to only
refer to random external directories (the ones which the libraries
were compiled under) and nothing under the cduce build directory.

That's a bug in cduce.spec which I'll fix shortly.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-builder quickly builds VMs from scratch
http://libguestfs.org/virt-builder.1.html

[Richard W.M. Jones]

On Wed, Jul 26, 2017 at 01:27:59PM +0100, Richard W.M. Jones wrote:
>  The same happens in the cduce binary, with a bunch of stuff being
>  statically linked.
>  
>  However the problem is that cduce itself was compiled without the ‘-g’
>  flag, so none of the debuginfo refers to the main binary, only to
>  statically linked libraries.  That's why the paths appear to only
>  refer to random external directories (the ones which the libraries
>  were compiled under) and nothing under the cduce build directory.
>  
>  That's a bug in cduce.spec which I'll fix shortly. 
So I fixed that:

 
http://pkgs.fedoraproject.org/cgit/rpms/cduce.git/commit/?id=7be2ca682411...

and it fixes the debuginfo problem, yay!

And we even get a debugsource package which contains sources, even better!

This isn't a problem in either ocaml or the debuginfo scripts after all.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW

[Richard W.M. Jones]

I should say there is still a kind of problem, although I don't think
it's a real bug.

Because of the statically linked sources, during debuginfo generation
it prints a bunch of warnings:

cpio: big_int.ml: Cannot stat: No such file or directory
cpio: buffer.ml: Cannot stat: No such file or directory
cpio: bytes.ml: Cannot stat: No such file or directory
cpio: curl.ml: Cannot stat: No such file or directory
[etc]

At least I now understand where those come from and why they can be
ignored.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine.  Supports Linux and Windows.
http://people.redhat.com/~rjones/virt-df/

[Kamil Dudka]

On Monday, June 26, 2017 16:13:02 Mark Wielaard wrote:
>  Hi packagers,
>  
>  rawhide rpmbuild contains various debuginfo improvements that hopefully
>  will make various hacks in spec files redundant. 
These improvements break build of packages that use the RemovePathPostfixes 
feature of RPM, such as coreutils or curl:

https://koji.fedoraproject.org/koji/getfile?taskID=20745024&volume=DE...
https://koji.fedoraproject.org/koji/getfile?taskID=20745984&volume=DE...

Is this expected?

Kamil
 
> If you have your own way of handling debuginfo packages, calling
> find-debuginfo.sh directly, need hacks for working around debugedit
> limitations or split your debuginfo package by hand then please try out
> rpmbuild in rawhide and read below for some macros you can set to tweak
> debuginfo package generation.
> 
> If you still need hacks in your spec file because setting macros isn't
> enough to get the debuginfo packages you want then please let us know.
> Also please let us know about packages that need to set debuginfo rpm
> macros to non-default values because they would crash and burn with the
> default settings (best to file a bug against rpmbuild).
> 
> The improvements have been mainly driven by the following two change
> proposals for f27 (some inspired by what other distros do):
> 
> https://fedoraproject.org/wiki/Changes/ParallelInstallableDebuginfo
> https://fedoraproject.org/wiki/Changes/SubpackageAndSourceDebuginfo
> 
> The first is completely done and has been enabled by default for some
> months now in rawhide. The second introduces two new macros to enable
> separate debugsource and sub-debuginfo packages, but has not been
> enabled by default yet. If people like the change and no bugs are found
> (and fesco and releng agree) we can enable them for the f27 mass
> rebuild.
> 
> If your package already splits debuginfo packages in a (common) source
> package and/or sub-debuginfo packages, please try out the new macros
> introduced by the second change. You can enable the standard splitting
> by adding the following to your spec file:
> 
> %global _debugsource_packages 1
> %global _debuginfo_subpackages 1
> 
> Besides the above two changes debuginfo packages can now (and are by
> default in rawhide) build by running debug extraction in parallel. This
> should speed up building with lots of binaries/libraries. If you do
> invoke find-debuginfo.sh by hand you most likely will want to add
> %{?_smp_mflags} as argument to get the parallel processing speedup.
> 
> If your package is invoking find-debuginfo.sh by hand also please take a
> look at all the new options that have been added. Also note that almost
> all options can be changed by setting (or undefining) rpm macros now.
> Using the rpm macros is preferred over invoking find-debuginfo.sh
> directly since it means you get any defaults and improvements that might
> need new find-debuginfo.sh arguments automatically. 
> 
> Here is an overview of various debuginfo rpm macros that you can define
> undefine in your spec file with the latest rpmbuild:
> 
> #
> # Should an ELF file processed by find-debuginfo.sh having no build ID
> # terminate a build?  This is left undefined to disable it and defined to
> # enable.
> #
> %_missing_build_ids_terminate_build    1
> 
> #
> # Include minimal debug information in build binaries.
> # Requires _enable_debug_packages.
> #
> %_include_minidebuginfo        1
> 
> #
> # Include a .gdb_index section in the .debug files.
> # Requires _enable_debug_packages and gdb-add-index installed.
> #
> %_include_gdb_index    1
> 
> #
> # Defines how and if build_id links are generated for ELF files.
> # The following settings are supported:
> #
> # - none
> #   No build_id links are generated.
> #
> # - alldebug
> #   build_id links are generated only when the __debug_package global is
> #   defined. This will generate build_id links in the -debuginfo package
> #   for both the main file as /usr/lib/debug/.build-id/xx/yyy and for
> #   the .debug file as /usr/lib/debug/.build-id/xx/yyy.debug.
> #   This is the old style build_id links as generated by the original
> #   find-debuginfo.sh script.
> #
> # - separate
> #   build_id links are generate for all binary packages. If this is a
> #   main package (the __debug_package global isn't set) then the
> #   build_id link is generated as /usr/lib/.build-id/xx/yyy. If this is
> #   a -debuginfo package (the __debug_package global is set) then the
> #   build_id link is generated as /usr/lib/debug/.build-id/xx/yyy.
> #
> # - compat
> #   Same as for "separate" but if the __debug_package global is set then
> #   the -debuginfo package will have a compatibility link for the main
> #   ELF /usr/lib/debug/.build-id/xx/yyy -> /usr/lib/.build-id/xx/yyy
> %_build_id_links compat
> 
> # Whether build-ids should be made unique between package version/releases
> # when generating debuginfo packages. If set to 1 this will pass
> # --build-id-seed "%{VERSION}-%{RELEASE}" to find-debuginfo.sh which will
> # pass it onto debugedit --build-id-seed to be used to prime the build-id
> # note hash.
> %_unique_build_ids      1
> 
> # Do not recompute build-ids but keep whatever is in the ELF file already.
> # Cannot be used together with _unique_build_ids (which forces
> recomputation).
> # Defaults to undefined (unset).
> #%_no_recompute_build_ids 1
> 
> # Whether .debug files should be made unique between package version,
> # release and architecture. If set to 1 this will pass
> # --unique-debug-suffix "-%{VERSION}-%{RELEASE}.%{_arch} find-debuginfo.sh
> # to create debuginfo files which end in -<ver>-<rel>.<arch>.debug
> # Requires _unique_build_ids.
> %_unique_debug_names    1
> 
> # Whether the /usr/debug/src/<package> directories should be unique between
> # package version, release and architecture. If set to 1 this will pass #
> --unique-debug-src-base "%{name}-%{VERSION}-%{RELEASE}.%{_arch}" to #
> find-debuginfo.sh to name the directory under /usr/debug/src as #
> <name>-<ver>-<rel>.<arch>.
> %_unique_debug_srcs     1
> 
> # Whether rpm should put debug source files into its own subpackage
> #%_debugsource_packages 1
> 
> # Whether rpm should create extra debuginfo packages for each subpackage
> #%_debuginfo_subpackages 1
> 
> # Number of debugging information entries (DIEs) above which
> # dwz will stop considering file for multifile optimizations
> # and enter a low memory mode, in which it will optimize
> # in about half the memory needed otherwise.
> %_dwz_low_mem_die_limit          10000000
> # Number of DIEs above which dwz will stop processing
> # a file altogether.
> %_dwz_max_die_limit              50000000
> 
> %_find_debuginfo_dwz_opts --run-dwz\\\
>    --dwz-low-mem-die-limit %{_dwz_low_mem_die_limit}\\\
>    --dwz-max-die-limit %{_dwz_max_die_limit}
> 
> If there are settings missing that would be useful, bugs with the
> default settings or defaults that should be changed please do file a bug
> report.
> 
> Thanks,
> 
> Mark

[Mark Wielaard]

On Wed, 2017-07-26 at 14:38 +0200, Kamil Dudka wrote:
>  On Monday, June 26, 2017 16:13:02 Mark Wielaard wrote:
>  > rawhide rpmbuild contains various debuginfo improvements that hopefully
>  > will make various hacks in spec files redundant.
>  
>  These improvements break build of packages that use the RemovePathPostfixes 
>  feature of RPM, such as coreutils or curl:
>  
>
> https://koji.fedoraproject.org/koji/getfile?taskID=20745024&volume=DE...
>
> https://koji.fedoraproject.org/koji/getfile?taskID=20745984&volume=DE...
>  
>  Is this expected? 
No, please file a bug and don't enable (or disable) any features that
cause trouble for your package. Note that since rpm-4.13.0.1-37 both
%_debugsource_packages and %_debuginfo_subpackages have been enabled
by default in rawhide.

Thanks,

Mark

[Igor Gnatenko]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On Wed, 2017-07-26 at 14:38 +0200, Kamil Dudka wrote:
>  On Monday, June 26, 2017 16:13:02 Mark Wielaard wrote:
>  > Hi packagers,
>  > 
>  > rawhide rpmbuild contains various debuginfo improvements that
>  > hopefully
>  > will make various hacks in spec files redundant.
>  
>  These improvements break build of packages that use the
>  RemovePathPostfixes 
>  feature of RPM, such as coreutils or curl:
>  
>  https://koji.fedoraproject.org/koji/getfile?taskID=20745024&volume=DE
>  FAULT&name=build.log&offset=-4000
>  https://koji.fedoraproject.org/koji/getfile?taskID=20745984&volume=DE
>  FAULT&name=build.log&offset=-4000
>  
>  Is this expected? 
Definitely not, but thanks a lot for reporting!

I've created issue for rpm[0] to track this problem. For now, feel free
to add %undefine _debuginfo_subpackages on top of your spec files.

Sorry for breakage.

[0]https://github.com/rpm-software-management/rpm/issues/280
>  
>  Kamil
>   
>  > If you have your own way of handling debuginfo packages, calling
>  > find-debuginfo.sh directly, need hacks for working around debugedit
>  > limitations or split your debuginfo package by hand then please try
>  > out
>  > rpmbuild in rawhide and read below for some macros you can set to
>  > tweak
>  > debuginfo package generation.
>  > 
>  > If you still need hacks in your spec file because setting macros
>  > isn't
>  > enough to get the debuginfo packages you want then please let us
>  > know.
>  > Also please let us know about packages that need to set debuginfo
>  > rpm
>  > macros to non-default values because they would crash and burn with
>  > the
>  > default settings (best to file a bug against rpmbuild).
>  > 
>  > The improvements have been mainly driven by the following two
>  > change
>  > proposals for f27 (some inspired by what other distros do):
>  > 
>  > https://fedoraproject.org/wiki/Changes/ParallelInstallableDebuginfo
>  > https://fedoraproject.org/wiki/Changes/SubpackageAndSourceDebuginfo
>  > 
>  > The first is completely done and has been enabled by default for
>  > some
>  > months now in rawhide. The second introduces two new macros to
>  > enable
>  > separate debugsource and sub-debuginfo packages, but has not been
>  > enabled by default yet. If people like the change and no bugs are
>  > found
>  > (and fesco and releng agree) we can enable them for the f27 mass
>  > rebuild.
>  > 
>  > If your package already splits debuginfo packages in a (common)
>  > source
>  > package and/or sub-debuginfo packages, please try out the new
>  > macros
>  > introduced by the second change. You can enable the standard
>  > splitting
>  > by adding the following to your spec file:
>  > 
>  > %global _debugsource_packages 1
>  > %global _debuginfo_subpackages 1
>  > 
>  > Besides the above two changes debuginfo packages can now (and are
>  > by
>  > default in rawhide) build by running debug extraction in parallel.
>  > This
>  > should speed up building with lots of binaries/libraries. If you do
>  > invoke find-debuginfo.sh by hand you most likely will want to add
>  > %{?_smp_mflags} as argument to get the parallel processing speedup.
>  > 
>  > If your package is invoking find-debuginfo.sh by hand also please
>  > take a
>  > look at all the new options that have been added. Also note that
>  > almost
>  > all options can be changed by setting (or undefining) rpm macros
>  > now.
>  > Using the rpm macros is preferred over invoking find-debuginfo.sh
>  > directly since it means you get any defaults and improvements that
>  > might
>  > need new find-debuginfo.sh arguments automatically. 
>  > 
>  > Here is an overview of various debuginfo rpm macros that you can
>  > define
>  > undefine in your spec file with the latest rpmbuild:
>  > 
>  > #
>  > # Should an ELF file processed by find-debuginfo.sh having no build
>  > ID
>  > # terminate a build?  This is left undefined to disable it and
>  > defined to
>  > # enable.
>  > #
>  > %_missing_build_ids_terminate_build    1
>  > 
>  > #
>  > # Include minimal debug information in build binaries.
>  > # Requires _enable_debug_packages.
>  > #
>  > %_include_minidebuginfo        1
>  > 
>  > #
>  > # Include a .gdb_index section in the .debug files.
>  > # Requires _enable_debug_packages and gdb-add-index installed.
>  > #
>  > %_include_gdb_index    1
>  > 
>  > #
>  > # Defines how and if build_id links are generated for ELF files.
>  > # The following settings are supported:
>  > #
>  > # - none
>  > #   No build_id links are generated.
>  > #
>  > # - alldebug
>  > #   build_id links are generated only when the __debug_package
>  > global is
>  > #   defined. This will generate build_id links in the -debuginfo
>  > package
>  > #   for both the main file as /usr/lib/debug/.build-id/xx/yyy and
>  > for
>  > #   the .debug file as /usr/lib/debug/.build-id/xx/yyy.debug.
>  > #   This is the old style build_id links as generated by the
>  > original
>  > #   find-debuginfo.sh script.
>  > #
>  > # - separate
>  > #   build_id links are generate for all binary packages. If this is
>  > a
>  > #   main package (the __debug_package global isn't set) then the
>  > #   build_id link is generated as /usr/lib/.build-id/xx/yyy. If
>  > this is
>  > #   a -debuginfo package (the __debug_package global is set) then
>  > the
>  > #   build_id link is generated as /usr/lib/debug/.build-id/xx/yyy.
>  > #
>  > # - compat
>  > #   Same as for "separate" but if the __debug_package global is set
>  > then
>  > #   the -debuginfo package will have a compatibility link for the
>  > main
>  > #   ELF /usr/lib/debug/.build-id/xx/yyy -> /usr/lib/.build-
>  > id/xx/yyy
>  > %_build_id_links compat
>  > 
>  > # Whether build-ids should be made unique between package
>  > version/releases
>  > # when generating debuginfo packages. If set to 1 this will pass
>  > # --build-id-seed "%{VERSION}-%{RELEASE}" to find-debuginfo.sh
>  > which will
>  > # pass it onto debugedit --build-id-seed to be used to prime the
>  > build-id
>  > # note hash.
>  > %_unique_build_ids      1
>  > 
>  > # Do not recompute build-ids but keep whatever is in the ELF file
>  > already.
>  > # Cannot be used together with _unique_build_ids (which forces
>  > recomputation).
>  > # Defaults to undefined (unset).
>  > #%_no_recompute_build_ids 1
>  > 
>  > # Whether .debug files should be made unique between package
>  > version,
>  > # release and architecture. If set to 1 this will pass
>  > # --unique-debug-suffix "-%{VERSION}-%{RELEASE}.%{_arch} find-
>  > debuginfo.sh
>  > # to create debuginfo files which end in
> -<ver>-<rel>.<arch>.debug
>  > # Requires _unique_build_ids.
>  > %_unique_debug_names    1
>  > 
>  > # Whether the /usr/debug/src/<package> directories should be unique
>  > between
>  > # package version, release and architecture. If set to 1 this will
>  > pass #
>  > --unique-debug-src-base "%{name}-%{VERSION}-%{RELEASE}.%{_arch}" to
>  > #
>  > find-debuginfo.sh to name the directory under /usr/debug/src as #
>  > <name>-<ver>-<rel>.<arch>.
>  > %_unique_debug_srcs     1
>  > 
>  > # Whether rpm should put debug source files into its own subpackage
>  > #%_debugsource_packages 1
>  > 
>  > # Whether rpm should create extra debuginfo packages for each
>  > subpackage
>  > #%_debuginfo_subpackages 1
>  > 
>  > # Number of debugging information entries (DIEs) above which
>  > # dwz will stop considering file for multifile optimizations
>  > # and enter a low memory mode, in which it will optimize
>  > # in about half the memory needed otherwise.
>  > %_dwz_low_mem_die_limit          10000000
>  > # Number of DIEs above which dwz will stop processing
>  > # a file altogether.
>  > %_dwz_max_die_limit              50000000
>  > 
>  > %_find_debuginfo_dwz_opts --run-dwz\\\
>  >    --dwz-low-mem-die-limit %{_dwz_low_mem_die_limit}\\\
>  >    --dwz-max-die-limit %{_dwz_max_die_limit}
>  > 
>  > If there are settings missing that would be useful, bugs with the
>  > default settings or defaults that should be changed please do file
>  > a bug
>  > report.
>  > 
>  > Thanks,
>  > 
>  > Mark
>  
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org 
- -- 
- -Igor Gnatenko
> -----BEGIN PGP SIGNATURE-----
>
> iQIzBAEBCAAdFiEEhLFO09aHZVqO+CM6aVcUvRu8X0wFAll4kUYACgkQaVcUvRu8
> X0xOWhAAu5GGfH9ZsXzeZx/lARpVWsWzou+SiQ34ECzIu6wrWJXlc/l8xEXShSFI
> Ye5lNtxOlsdACvI+IIsRplLG7NHPLoyX6593yYNS5TquUU6dSVIw2PH8wAy3uk00
> mvO65HPlYyMsA9WK/MvcXVhdeCfPAJT9q0VxIrjrwpwa1nJbj4+gi8JEMqQSptPF
> mgFEUa/HuE4rBb2kSQQDEfxOlcI7P39/FgmZf6eYqv7JwVWJ8S2vMgNnKU7gZvgP
> oPSP7D52MHjZRJWUky3GtpyicNZafikpnt8iCJc1wItBeMl5uRm2zElqxPsEP7Sp
> 6pr7PKgDp1/B5XELyOfhsvEWCw4rVHzoWiuhyLqRcrGrNXLQbXCas9vKag1GLWDX
> n+WutrsyxvMaRf9pGf4q+8RXEdpMlaMKgImigmK1FFCVcELhVkVF5jAs6VDr78Yp
> rm82JQQ6OeFi55lj0Hl4aHT0cjYq34birfaJTreJ/VXGDbGb8IfEEvNqDetuX1gM
> vl5VWfLZgCdgtX8JYrjxx7pSTZsZurw0UD+0sddvSYQlYAYF4lnRVPqL3wmhv5QK
> P7U9xxdfKCM42s7qq20eexbgVPp0PB5/tvJrjL4AGviRQATrXY4NXKN0j7TKy04Q
> oXHBmbBG4Co3gDEQt+OUcOIOMTMalkC+IuZpbwjM7hRww0j1qEo=
> =UcUg
> -----END PGP SIGNATURE-----

[Kamil Dudka]

On Wednesday, July 26, 2017 14:55:34 Igor Gnatenko wrote:
>  On Wed, 2017-07-26 at 14:38 +0200, Kamil Dudka wrote:
>  
>  > On Monday, June 26, 2017 16:13:02 Mark Wielaard wrote:
>  > 
>  > > Hi packagers,
>  > >
>  > >
>  > >
>  > > rawhide rpmbuild contains various debuginfo improvements that
>  > > hopefully
>  > > will make various hacks in spec files redundant.
>  >
>  >
>  >
>  > These improvements break build of packages that use the
>  > RemovePathPostfixes
>  > feature of RPM, such as coreutils or curl:
>  >
>  >
>  >
>  > https://koji.fedoraproject.org/koji/getfile?taskID=20745024&volume=DE
>  > FAULT&name=build.log&offset=-4000
>  > https://koji.fedoraproject.org/koji/getfile?taskID=20745984&volume=DE
>  > FAULT&name=build.log&offset=-4000
>  >
>  >
>  >
>  > Is this expected?
>  
>  Definitely not, but thanks a lot for reporting!
>  
>  I've created issue for rpm[0] to track this problem. For now, feel free
>  to add %undefine _debuginfo_subpackages on top of your spec files. 
Thanks for the quick reply on this!  Using the above trick helped me to
make both coreutils and curl build successfully again.

Kamil

> Sorry for breakage.
> 
> [0]https://github.com/rpm-software-management/rpm/issues/280
> 
> >
> >
> > Kamil
> >
> >
> >
> > > If you have your own way of handling debuginfo packages, calling
> > > find-debuginfo.sh directly, need hacks for working around debugedit
> > > limitations or split your debuginfo package by hand then please try
> > > out
> > > rpmbuild in rawhide and read below for some macros you can set to
> > > tweak
> > > debuginfo package generation.
> > >
> > >
> > >
> > > If you still need hacks in your spec file because setting macros
> > > isn't
> > > enough to get the debuginfo packages you want then please let us
> > > know.
> > > Also please let us know about packages that need to set debuginfo
> > > rpm
> > > macros to non-default values because they would crash and burn with
> > > the
> > > default settings (best to file a bug against rpmbuild).
> > >
> > >
> > >
> > > The improvements have been mainly driven by the following two
> > > change
> > > proposals for f27 (some inspired by what other distros do):
> > >
> > >
> > >
> > > https://fedoraproject.org/wiki/Changes/ParallelInstallableDebuginfo
> > > https://fedoraproject.org/wiki/Changes/SubpackageAndSourceDebuginfo
> > >
> > >
> > >
> > > The first is completely done and has been enabled by default for
> > > some
> > > months now in rawhide. The second introduces two new macros to
> > > enable
> > > separate debugsource and sub-debuginfo packages, but has not been
> > > enabled by default yet. If people like the change and no bugs are
> > > found
> > > (and fesco and releng agree) we can enable them for the f27 mass
> > > rebuild.
> > >
> > >
> > >
> > > If your package already splits debuginfo packages in a (common)
> > > source
> > > package and/or sub-debuginfo packages, please try out the new
> > > macros
> > > introduced by the second change. You can enable the standard
> > > splitting
> > > by adding the following to your spec file:
> > >
> > >
> > >
> > > %global _debugsource_packages 1
> > > %global _debuginfo_subpackages 1
> > >
> > >
> > >
> > > Besides the above two changes debuginfo packages can now (and are
> > > by
> > > default in rawhide) build by running debug extraction in parallel.
> > > This
> > > should speed up building with lots of binaries/libraries. If you do
> > > invoke find-debuginfo.sh by hand you most likely will want to add
> > > %{?_smp_mflags} as argument to get the parallel processing speedup.
> > >
> > >
> > >
> > > If your package is invoking find-debuginfo.sh by hand also please
> > > take a
> > > look at all the new options that have been added. Also note that
> > > almost
> > > all options can be changed by setting (or undefining) rpm macros
> > > now.
> > > Using the rpm macros is preferred over invoking find-debuginfo.sh
> > > directly since it means you get any defaults and improvements that
> > > might
> > > need new find-debuginfo.sh arguments automatically.
> > >
> > >
> > >
> > > Here is an overview of various debuginfo rpm macros that you can
> > > define
> > > undefine in your spec file with the latest rpmbuild:
> > >
> > >
> > >
> > > #
> > > # Should an ELF file processed by find-debuginfo.sh having no build
> > > ID
> > > # terminate a build?  This is left undefined to disable it and
> > > defined to
> > > # enable.
> > > #
> > > %_missing_build_ids_terminate_build    1
> > >
> > >
> > >
> > > #
> > > # Include minimal debug information in build binaries.
> > > # Requires _enable_debug_packages.
> > > #
> > > %_include_minidebuginfo        1
> > >
> > >
> > >
> > > #
> > > # Include a .gdb_index section in the .debug files.
> > > # Requires _enable_debug_packages and gdb-add-index installed.
> > > #
> > > %_include_gdb_index    1
> > >
> > >
> > >
> > > #
> > > # Defines how and if build_id links are generated for ELF files.
> > > # The following settings are supported:
> > > #
> > > # - none
> > > #   No build_id links are generated.
> > > #
> > > # - alldebug
> > > #   build_id links are generated only when the __debug_package
> > > global is
> > > #   defined. This will generate build_id links in the -debuginfo
> > > package
> > > #   for both the main file as /usr/lib/debug/.build-id/xx/yyy and
> > > for
> > > #   the .debug file as /usr/lib/debug/.build-id/xx/yyy.debug.
> > > #   This is the old style build_id links as generated by the
> > > original
> > > #   find-debuginfo.sh script.
> > > #
> > > # - separate
> > > #   build_id links are generate for all binary packages. If this is
> > > a
> > > #   main package (the __debug_package global isn't set) then the
> > > #   build_id link is generated as /usr/lib/.build-id/xx/yyy. If
> > > this is
> > > #   a -debuginfo package (the __debug_package global is set) then
> > > the
> > > #   build_id link is generated as /usr/lib/debug/.build-id/xx/yyy.
> > > #
> > > # - compat
> > > #   Same as for "separate" but if the __debug_package global is
set
> > > then
> > > #   the -debuginfo package will have a compatibility link for the
> > > main
> > > #   ELF /usr/lib/debug/.build-id/xx/yyy -> /usr/lib/.build-
> > > id/xx/yyy
> > > %_build_id_links compat
> > >
> > >
> > >
> > > # Whether build-ids should be made unique between package
> > > version/releases
> > > # when generating debuginfo packages. If set to 1 this will pass
> > > # --build-id-seed "%{VERSION}-%{RELEASE}" to find-debuginfo.sh
> > > which will
> > > # pass it onto debugedit --build-id-seed to be used to prime the
> > > build-id
> > > # note hash.
> > > %_unique_build_ids      1
> > >
> > >
> > >
> > > # Do not recompute build-ids but keep whatever is in the ELF file
> > > already.
> > > # Cannot be used together with _unique_build_ids (which forces
> > > recomputation).
> > > # Defaults to undefined (unset).
> > > #%_no_recompute_build_ids 1
> > >
> > >
> > >
> > > # Whether .debug files should be made unique between package
> > > version,
> > > # release and architecture. If set to 1 this will pass
> > > # --unique-debug-suffix "-%{VERSION}-%{RELEASE}.%{_arch} find-
> > > debuginfo.sh
> > > # to create debuginfo files which end in
-<ver>-<rel>.<arch>.debug
> > > # Requires _unique_build_ids.
> > > %_unique_debug_names    1
> > >
> > >
> > >
> > > # Whether the /usr/debug/src/<package> directories should be unique
> > > between
> > > # package version, release and architecture. If set to 1 this will
> > > pass #
> > > --unique-debug-src-base "%{name}-%{VERSION}-%{RELEASE}.%{_arch}"
to
> > > #
> > > find-debuginfo.sh to name the directory under /usr/debug/src as #
> > > <name>-<ver>-<rel>.<arch>.
> > > %_unique_debug_srcs     1
> > >
> > >
> > >
> > > # Whether rpm should put debug source files into its own subpackage
> > > #%_debugsource_packages 1
> > >
> > >
> > >
> > > # Whether rpm should create extra debuginfo packages for each
> > > subpackage
> > > #%_debuginfo_subpackages 1
> > >
> > >
> > >
> > > # Number of debugging information entries (DIEs) above which
> > > # dwz will stop considering file for multifile optimizations
> > > # and enter a low memory mode, in which it will optimize
> > > # in about half the memory needed otherwise.
> > > %_dwz_low_mem_die_limit          10000000
> > > # Number of DIEs above which dwz will stop processing
> > > # a file altogether.
> > > %_dwz_max_die_limit              50000000
> > >
> > >
> > >
> > > %_find_debuginfo_dwz_opts --run-dwz\\\
> > > 
> > >    --dwz-low-mem-die-limit %{_dwz_low_mem_die_limit}\\\
> > >    --dwz-max-die-limit %{_dwz_max_die_limit}
> > >
> > >
> > >
> > > If there are settings missing that would be useful, bugs with the
> > > default settings or defaults that should be changed please do file
> > > a bug
> > > report.
> > >
> > >
> > >
> > > Thanks,
> > >
> > >
> > >
> > > Mark
> >
> >
> >
> > _______________________________________________
> > devel mailing list -- devel(a)lists.fedoraproject.org
> > To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
> 
> 
> --
> -Igor Gnatenko