F36 Change: Remove .la files from buildroot (Self-Contained Change proposal)

[Ben Cotton]

https://fedoraproject.org/wiki/Changes/RemoveLaFiles

== Summary ==
Autools/libtool-based projects frequently install files ending in
`.la` in their `make install`. These files are usually unwanted. Many
projects therefore end up with a variation of `find $RPM_BUILD_ROOT
-name "*.la" -delete` in their `%install section`.

`*.la` files are "libtool archives" and provide additional metadata
for library files and come from a time before the introduction of the
ELF format. Today, they are only consumed by libtool itself. Refer to
https://autotools.io/libtool/lafiles.html.

This changes proposes to instead use the `%__brp_remove_la_files`
macro in `redhat-rpm-config`'s `%__os_install_post` to remove the
`*.la` files automatically. This has been added to RPM 4.17.



== Owner ==
* Name: [[User:FASAcountName| Timm Bäder]]
* Email: tbaeder(a)redhat.com


== Detailed Description ==
(not provided)

== Benefit to Fedora ==
This change removes a widely used line of shell script from many spec
files. The advantage is cleaner and easier to maintain spec files as
well as more sensible defaults for rpm package builds.

While looking at what packages will be affected by this change, I
already found several packages that ship and install `.la` files by
accident. These packages will be fixed to not ship them.

== Scope ==
* Proposal owners:
** Update packaging guidelines to mention the automatic removal of
`*.la` files and mention the mechanism for opting out of this
behavior.
* Other developers:
** For the packages already removing their `*.la` files manually,
there should be no change. For packages that want to install such
packages, the package maintainers need to opt out of the automatic
removal.

* Release engineering: [https://pagure.io/releng/issue/10353 #10353]
* Policies and guidelines: N/A (not needed for this Change)
* Alignment with Objectives:

== Upgrade/compatibility impact ==
The following packages ship `.la` files currently (queried via `$
repoquery --repo=rawhide -f '*.la' --source | pkgname | sort | uniq`):

<!--
* apr
* apr-util
* aqbanking
* arts
* avr-gcc
* binutils
* calf
* cross-gcc
* djview4
* filezilla
* flatpak
* gambas3
* gforth
* gnome-do
* gnome-subtitles
* google-authenticator
* GraphicsMagick
* gretl
* gstreamer1-doc
* gutenprint
* gwenhywfar
* chafa
* ImageMagick
* jpilot
* kdebase3
* kdegames3
* kdelibs3
* kdepim3
* kdewebdev
* kdissert
* kguitar
* koffice-kivio
* libmodsecurity
* libsecp256k1
* liferea
* mcabber
* mingw-sane-backends
* mingw-speexdsp
* mousepad
* neon
* octave
* opencryptoki
* OpenIPMI
* openldap
* owfs
* pinball
* pragha
* qt5-qtbase
* qt5-qtfeedback
* qt5-qtremoteobjects
* qt5-qttools
* subversion
* taxipilot
* unicornscan
* util-linux
* xfce4-calculator-plugin
* xfce4-timer-plugin

-->

`*.la` files explicitly listed in `%files` (this includes packages
that modify the `*.la` file(s) but don't list them in `%files`
explicitly):
* arts
* gambas3
* chafa
* ImageMagick
* kdebase3
* kdegames3
* kdelibs3
* kdewebdev
* kdissert
* libmodsecurity
* libsecp256k1
* mingw-sane-backends
* mingw-speexdsp
* neon
* openldap
* pinball
* qt5-qtbase
* qt5-qtfeedback
* qt5-qtremoteobjects
* qt5-qttools
* subversion
* unicornscan
* xfce4-calculator-plugin



No mention of `*.la` files in the spec file:
* gcc-avr
* calf
* cross-gcc
* djview4
* filezilla
* gforth
* gnome-do
* gnome-subtitles
* google-authenticator
* GraphicsMagick - But probably needs them just like ImageMagick
* gstreamer1-doc
* jpilot
* kguitar
* liferea
* mcabber
* mousepad
* octave
* opencryptoki
* pragha
* taxipilot
* xfce4-timer-plugin



Tries to delete them but fails:
* aqbanking
* binutils (fixed)
* flatpak
* gretl
* gutenprint
* gwenhywfar
* kdepim3
* koffice-kivio
* OpenIPMI
* owfs
* util-linux



== How To Test ==

Whether a RPM package ships `.la` files can be checked either via the
`repoquery` command from above or via querying a local `.rpm` file
directly: `rpm -q --files ./my-package.rpm | grep '\.la$'`. The latter
command can be used for local testing.

If a package currently ships any `*.la` files but only does so
accidentally, nothing needs to be done and the package will
automatically stop shipping those files after it is rebuilt with this
change in effect.

If a package whishes to keep shipping `*.la` files, the package
maintainer can opt out of the automatic removal by setting
`%__brp_remove_la_files` to `%nil`:  `%global __brp_remove_la_files
%nil`

If the package currently removes all `*.la` files manually via some
form of `find $RPM_BUILD_ROOT -name "*.la" -delete` or similar, it is
recommended (but not required) to remove that line.



== User Experience ==

Users should not notice any change.

== Dependencies ==
There are no dependencies. Only `redhat-rpm-config` needs to adapt to
the change.

== Contingency Plan ==

* Contingency mechanism: The change can simply be reverted in the
`redhat-rpm-config` package. Packages that have already removed the
manual deletion of `*.la` files need to revert this change too.
Packages that have opted out of the automatic `*.la` file removal
don't need to do anything.

* Contingency deadline: beta freeze
* Blocks release? Yes


== Documentation ==
Pull request implementing `%__brp_remove_la_files` in the upstream rpm
repository: https://github.com/rpm-software-management/rpm/pull/1674


== Release Notes ==

The [https://rpm.org/wiki/Releases/4.17.0 RPM 4.17 release notes]
simply state "Add policy for removing .la files from buildroot by
default".


-- 
Ben Cotton
He / Him / His
Fedora Program Manager
Red Hat
TZ=America/Indiana/Indianapolis

[Richard W.M. Jones]

On Mon, Nov 01, 2021 at 09:37:40AM -0400, Ben Cotton wrote:
>  https://fedoraproject.org/wiki/Changes/RemoveLaFiles
>  
>  == Summary ==
>  Autools/libtool-based projects frequently install files ending in
>  `.la` in their `make install`. These files are usually unwanted. Many
>  projects therefore end up with a variation of `find $RPM_BUILD_ROOT
>  -name "*.la" -delete` in their `%install section`.
>  
>  `*.la` files are "libtool archives" and provide additional metadata
>  for library files and come from a time before the introduction of the
>  ELF format. Today, they are only consumed by libtool itself. Refer to
>  https://autotools.io/libtool/lafiles.html.
>  
>  This changes proposes to instead use the `%__brp_remove_la_files`
>  macro in `redhat-rpm-config`'s `%__os_install_post` to remove the
>  `*.la` files automatically. This has been added to RPM 4.17. 
Yes!

There are quite a few of these "junk files" left around by build
systems that many packages have to remove:

 - .packlist (Perl)

 - .dune-keep (OCaml/dune)

 - *.cmti (OCaml)

 - *.bs (Perl)

 - *~ (gcc build removes these)

Should we have a more systematic way to remove them?

Rich.

>  
>  
>  == Owner ==
>  * Name: [[User:FASAcountName| Timm Bäder]]
>  * Email: tbaeder(a)redhat.com
>  
>  
>  == Detailed Description ==
>  (not provided)
>  
>  == Benefit to Fedora ==
>  This change removes a widely used line of shell script from many spec
>  files. The advantage is cleaner and easier to maintain spec files as
>  well as more sensible defaults for rpm package builds.
>  
>  While looking at what packages will be affected by this change, I
>  already found several packages that ship and install `.la` files by
>  accident. These packages will be fixed to not ship them.
>  
>  == Scope ==
>  * Proposal owners:
>  ** Update packaging guidelines to mention the automatic removal of
>  `*.la` files and mention the mechanism for opting out of this
>  behavior.
>  * Other developers:
>  ** For the packages already removing their `*.la` files manually,
>  there should be no change. For packages that want to install such
>  packages, the package maintainers need to opt out of the automatic
>  removal.
>  
>  * Release engineering: [https://pagure.io/releng/issue/10353 #10353]
>  * Policies and guidelines: N/A (not needed for this Change)
>  * Alignment with Objectives:
>  
>  == Upgrade/compatibility impact ==
>  The following packages ship `.la` files currently (queried via `$
>  repoquery --repo=rawhide -f '*.la' --source | pkgname | sort | uniq`):
>  
>  <!--
>  * apr
>  * apr-util
>  * aqbanking
>  * arts
>  * avr-gcc
>  * binutils
>  * calf
>  * cross-gcc
>  * djview4
>  * filezilla
>  * flatpak
>  * gambas3
>  * gforth
>  * gnome-do
>  * gnome-subtitles
>  * google-authenticator
>  * GraphicsMagick
>  * gretl
>  * gstreamer1-doc
>  * gutenprint
>  * gwenhywfar
>  * chafa
>  * ImageMagick
>  * jpilot
>  * kdebase3
>  * kdegames3
>  * kdelibs3
>  * kdepim3
>  * kdewebdev
>  * kdissert
>  * kguitar
>  * koffice-kivio
>  * libmodsecurity
>  * libsecp256k1
>  * liferea
>  * mcabber
>  * mingw-sane-backends
>  * mingw-speexdsp
>  * mousepad
>  * neon
>  * octave
>  * opencryptoki
>  * OpenIPMI
>  * openldap
>  * owfs
>  * pinball
>  * pragha
>  * qt5-qtbase
>  * qt5-qtfeedback
>  * qt5-qtremoteobjects
>  * qt5-qttools
>  * subversion
>  * taxipilot
>  * unicornscan
>  * util-linux
>  * xfce4-calculator-plugin
>  * xfce4-timer-plugin
>  
>  -->
>  
>  `*.la` files explicitly listed in `%files` (this includes packages
>  that modify the `*.la` file(s) but don't list them in `%files`
>  explicitly):
>  * arts
>  * gambas3
>  * chafa
>  * ImageMagick
>  * kdebase3
>  * kdegames3
>  * kdelibs3
>  * kdewebdev
>  * kdissert
>  * libmodsecurity
>  * libsecp256k1
>  * mingw-sane-backends
>  * mingw-speexdsp
>  * neon
>  * openldap
>  * pinball
>  * qt5-qtbase
>  * qt5-qtfeedback
>  * qt5-qtremoteobjects
>  * qt5-qttools
>  * subversion
>  * unicornscan
>  * xfce4-calculator-plugin
>  
>  
>  
>  No mention of `*.la` files in the spec file:
>  * gcc-avr
>  * calf
>  * cross-gcc
>  * djview4
>  * filezilla
>  * gforth
>  * gnome-do
>  * gnome-subtitles
>  * google-authenticator
>  * GraphicsMagick - But probably needs them just like ImageMagick
>  * gstreamer1-doc
>  * jpilot
>  * kguitar
>  * liferea
>  * mcabber
>  * mousepad
>  * octave
>  * opencryptoki
>  * pragha
>  * taxipilot
>  * xfce4-timer-plugin
>  
>  
>  
>  Tries to delete them but fails:
>  * aqbanking
>  * binutils (fixed)
>  * flatpak
>  * gretl
>  * gutenprint
>  * gwenhywfar
>  * kdepim3
>  * koffice-kivio
>  * OpenIPMI
>  * owfs
>  * util-linux
>  
>  
>  
>  == How To Test ==
>  
>  Whether a RPM package ships `.la` files can be checked either via the
>  `repoquery` command from above or via querying a local `.rpm` file
>  directly: `rpm -q --files ./my-package.rpm | grep '\.la$'`. The latter
>  command can be used for local testing.
>  
>  If a package currently ships any `*.la` files but only does so
>  accidentally, nothing needs to be done and the package will
>  automatically stop shipping those files after it is rebuilt with this
>  change in effect.
>  
>  If a package whishes to keep shipping `*.la` files, the package
>  maintainer can opt out of the automatic removal by setting
>  `%__brp_remove_la_files` to `%nil`:  `%global __brp_remove_la_files
>  %nil`
>  
>  If the package currently removes all `*.la` files manually via some
>  form of `find $RPM_BUILD_ROOT -name "*.la" -delete` or similar, it is
>  recommended (but not required) to remove that line.
>  
>  
>  
>  == User Experience ==
>  
>  Users should not notice any change.
>  
>  == Dependencies ==
>  There are no dependencies. Only `redhat-rpm-config` needs to adapt to
>  the change.
>  
>  == Contingency Plan ==
>  
>  * Contingency mechanism: The change can simply be reverted in the
>  `redhat-rpm-config` package. Packages that have already removed the
>  manual deletion of `*.la` files need to revert this change too.
>  Packages that have opted out of the automatic `*.la` file removal
>  don't need to do anything.
>  
>  * Contingency deadline: beta freeze
>  * Blocks release? Yes
>  
>  
>  == Documentation ==
>  Pull request implementing `%__brp_remove_la_files` in the upstream rpm
>  repository: https://github.com/rpm-software-management/rpm/pull/1674
>  
>  
>  == Release Notes ==
>  
>  The [https://rpm.org/wiki/Releases/4.17.0 RPM 4.17 release notes]
>  simply state "Add policy for removing .la files from buildroot by
>  default".
>  
>  
>  -- 
>  Ben Cotton
>  He / Him / His
>  Fedora Program Manager
>  Red Hat
>  TZ=America/Indiana/Indianapolis
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]
>  Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-p2v converts physical machines to virtual machines.  Boot with a
live CD or over the network (PXE) and turn machines into KVM guests.
http://libguestfs.org/virt-v2v

[Petr Pisar]

V Mon, Nov 01, 2021 at 02:50:27PM +0000, Richard W.M. Jones napsal(a):
>  There are quite a few of these "junk files" left around by
> build systems
>  that many packages have to remove:
>  
>   - .packlist (Perl)
>   
These are actually useful for a few Perl tools which need to enumarate
installed CPAN distributions. Usually various packers and bundlers. But the
files are incompatible with RPM because their content changes with each newly
installed Perl module, both with DNF and CPAN. Terefore we delete them and
sacrifies fearures of the packer tools. Theoretically we could manage the
.packlist file with an RPM file trigger but I worry it would be too slow. For
the full disclosure, those files are a local database of installed Perl
modules and a mapping to a (CPAN) distribution. Nevertheless their creation
can be supressed with proper build options.

>   - *.bs (Perl)
>   
When the files are empty (which has been true on Linux for ages), they can be
removed. Ideally we could change a Perl build machinery not to produce them
at all. But I hazily remember that it's not easy because they are used for
side effects, e.g. as Makefile targets. The files are used for setting dynamic
linker before dlopening Perl binary modules. (An anolog to to the very libtool
archives.)

>  Should we have a more systematic way to remove them?
>   
We could. But I'm not sure we should clutter rpmbuild with language specific
hacks.

-- Petr

[Björn Persson]

>  Pull request implementing `%__brp_remove_la_files` in the upstream
> rpm
>  repository: https://github.com/rpm-software-management/rpm/pull/1674 
This looks like it risks deleting more files than intended. If some
package uses country codes or domain names in filenames, then this
change could silently delete files specific to Laos.

Language codes occur as filename suffixes. Apache can use them to serve
web pages in the client's preferred language. This change risks deleting
files written in latin.

If there is a more reliable way to recognize a Libtool archive by
inspecting the file's content, then I think the script should do that
to verify that files with a ".la" suffix really are Libtool archives
before deleting them.

Björn Persson

[Zbigniew Jędrzejewski-Szmek]

On Mon, Nov 01, 2021 at 09:37:40AM -0400, Ben Cotton wrote:
>  == Summary ==
>  Autools/libtool-based projects frequently install files ending in
>  `.la` in their `make install`. These files are usually unwanted. Many
>  projects therefore end up with a variation of `find $RPM_BUILD_ROOT
>  -name "*.la" -delete` in their `%install section`.
>  
>  `*.la` files are "libtool archives" and provide additional metadata
>  for library files and come from a time before the introduction of the
>  ELF format. Today, they are only consumed by libtool itself. Refer to
>  https://autotools.io/libtool/lafiles.html. 
I read the mythbuster page, and I still don't understand if removing
the file has any effect or not. Will there be any difference in builds
(for package builds and end-user builds)?

>  == Scope ==
>  * Proposal owners:
>  ** Update packaging guidelines to mention the automatic removal of
>  `*.la` files and mention the mechanism for opting out of this
>  behavior.
>  * Other developers:
>  ** For the packages already removing their `*.la` files manually,
>  there should be no change. For packages that want to install such
>  packages, the package maintainers need to opt out of the automatic
>  removal. 
If this change is implemented, manual removal in packages becomes unnecessary.
Will you do a 'mass change' sweep to drop those removals?

>  If a package whishes to keep shipping `*.la` files, the package
>  maintainer can opt out of the automatic removal by setting
>  `%__brp_remove_la_files` to `%nil`:  `%global __brp_remove_la_files
>  %nil` 
Why would anyone want to do that? (I'm not talking about the case
mentioned elsewhere in the thread were a non-libtool file is removed
by a mistake, but the actual case where one would want to keep
distributing a libtool file.)

Zbyszek

[Vitaly Zaitsev]

On 01/11/2021 14:37, Ben Cotton wrote:
>  Autools/libtool-based projects frequently install files ending in
>  `.la` in their `make install`. These files are usually unwanted. Many
>  projects therefore end up with a variation of `find $RPM_BUILD_ROOT
>  -name "*.la" -delete` in their `%install section`. 
+1 for this change. SPECs can be simplified.

-- 
Sincerely,
   Vitaly Zaitsev (vitaly(a)easycoding.org)

[Kevin Kofler]

Zbigniew Jędrzejewski-Szmek wrote:
>  Why would anyone want to do that? (I'm not talking about the
> case
>  mentioned elsewhere in the thread were a non-libtool file is removed
>  by a mistake, but the actual case where one would want to keep
>  distributing a libtool file.) 
The kdelibs3 plugin loader breaks down horribly if you remove the .la file 
under it. (This was fixed in kdelibs 4, but the kdelibs3 compat library 
stack still ships those .la files for that reason.)

        Kevin Kofler

[Ben Beasley]

I think Björn Persson’s concerns are reasonable. Hopefully they can be 
easily addressed, since I would be happy to see this change implemented. 
At least six packages I maintain or regularly contribute to could be 
simplified.

I spot-checked a libtool .la file to see what typical contents looked 
like. I’ve pasted the entire file for libfakekey inline at the bottom of 
this message as an example.

The following page has some additional information about the format: 
https://autotools.io/libtool/lafiles.html

I’m not aware of a formal specification for this format—and I haven’t 
tested this suggestions against a corpus of packages—but I suspect that 
“files ending in .la that contain a line matching /^dlname=/” would be a 
very successful heuristic.

While it’s not always the best tool for automation, libmagic/file(1) was 
also able to confirm the type:

$ file libfakekey.la
libfakekey.la: libtool library file, ASCII text

----- Sample libtool library file begins here -----
# libfakekey.la - a libtool library file
# Generated by libtool (GNU libtool) 2.4.6
#
# Please DO NOT delete this file!
# It is necessary for linking the library.

# The name that we can dlopen(3).
dlname='libfakekey.so.0'

# Names of this library.
library_names='libfakekey.so.0.0.1 libfakekey.so.0 libfakekey.so'

# The name of the static archive.
old_library=''

# Linker flags that cannot go in dependency_libs.
inherited_linker_flags=''

# Libraries that this one depends upon.
dependency_libs=' -lX11 -lXtst'

# Names of additional weak libraries provided by this library
weak_library_names=''

# Version information for libfakekey.
current=0
age=0
revision=1

# Is this an already installed library?
installed=no

# Should we warn about portability when linking against -modules?
shouldnotlink=no

# Files to dlopen/dlpreopen
dlopen=''
dlpreopen=''

# Directory that this library needs to be installed in:
libdir='/usr/lib64'
----- Sample libtool library file ends here -----

On 11/1/21 13:50, Björn Persson wrote:
>> Pull request implementing `%__brp_remove_la_files` in the upstream rpm
>> repository: https://github.com/rpm-software-management/rpm/pull/1674
> 
> This looks like it risks deleting more files than intended. If some
> package uses country codes or domain names in filenames, then this
> change could silently delete files specific to Laos.
> 
> Language codes occur as filename suffixes. Apache can use them to serve
> web pages in the client's preferred language. This change risks deleting
> files written in latin.
> 
> If there is a more reliable way to recognize a Libtool archive by
> inspecting the file's content, then I think the script should do that
> to verify that files with a ".la" suffix really are Libtool archives
> before deleting them.
> 
> Björn Persson
> 
> 
> _______________________________________________
> devel mailing list -- devel(a)lists.fedoraproject.org
> To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
> Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
https://lists.fedoraproject.org/archives/list/[e-mail]
> Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
>

[Florian Weimer]

* Zbigniew Jędrzejewski-Szmek:

>  I read the mythbuster page, and I still don't understand if
> removing
>  the file has any effect or not. Will there be any difference in builds
>  (for package builds and end-user builds)? 
As far as I understand it, linking with static libraries using libtool
will not handle indirect dependencies anymore.

Thanks,
Florian

[Rex Dieter]

Kevin Kofler via devel wrote:

>  Zbigniew Jędrzejewski-Szmek wrote:
> > Why would anyone want to do that? (I'm not talking about the case
> > mentioned elsewhere in the thread were a non-libtool file is removed
> > by a mistake, but the actual case where one would want to keep
> > distributing a libtool file.)
>  
>  The kdelibs3 plugin loader breaks down horribly if you remove the .la file
>  under it. (This was fixed in kdelibs 4, but the kdelibs3 compat library
>  stack still ships those .la files for that reason.) 
I'm sure there's a way to opt-out of this behavior (right?)

-- Rex

[Jason Tibbitts]

> >>>> Rex Dieter <rdieter(a)gmail.com&gt; writes: 
>  I'm sure there's a way to opt-out of this behavior (right?)
There is a standard way to opt out of any of the brp scripts:

https://docs.fedoraproject.org/en-US/packaging-guidelines/#_brp_buildroot...

 - J<

[Timm Bäder]

>  This looks like it risks deleting more files than intended. If some
>  package uses country codes or domain names in filenames, then this
>  change could silently delete files specific to Laos. 
None of the packages I inspected looked like they would do this, but I opened
https://github.com/rpm-software-management/rpm/pull/1819
upstream to try to minimize the risk.

[Timm Bäder]

>  If this change is implemented, manual removal in packages becomes
> unnecessary.
>  Will you do a 'mass change' sweep to drop those removals? 
I've already looked at all the packages I listed, so looking at them again
shouldn't be
a problem.
But that's just the list of packages that currently ship .la files and not the list of
packages
that *don't* ship them because they successfully remove them. I don't have a list
for
that and I'm not sure how I would generate one.

[Richard Shaw]

On Fri, Nov 5, 2021 at 5:47 AM Timm Bäder <tbaeder(a)redhat.com&gt; wrote:

>  > If this change is implemented, manual removal in packages becomes
>  unnecessary.
>  > Will you do a 'mass change' sweep to drop those removals?
>
>  I've already looked at all the packages I listed, so looking at them again
>  shouldn't be
>  a problem.
>  But that's just the list of packages that currently ship .la files and not
>  the list of packages
>  that *don't* ship them because they successfully remove them. I don't have
>  a list for
>  that and I'm not sure how I would generate one.
>  
You'd have to grep through all the spec files. Most of the time 'find' is
used so you want to return all lines with 'find' and then grep through
those results for 'la'.

Thanks,
Richard