Is this allowed? Files and libs duplicated in subpackages

[Michael Schwendt]

https://bugzilla.redhat.com/1003196

Based on this suspicious output

  mate-dictionary  from  mate-utils
      provides libmatedict.so.6()(64bit)
  mate-utils  from  mate-utils
      provides libmatedict.so.6()(64bit)
      required by: mate-dictionary-devel-1.6.0-7.fc20.x86_64
      required by: mate-utils-devel-1.6.0-7.fc20.x86_64

I've only verified in koji that lots of files are included in both
sub-packages. Even the descriptions overlap.

And there are even more subpackages, which only contain copies of
files included in the base mate-utils package already.
Why is that done? Why aren't RPM dependencies used to have the
base-package depend on the multiple subpackages?

So far, it has always been a packaging mistake to duplicate files (and
their Provides as a consequence) in multiple subpackages.

[Michael Schwendt]

>  https://bugzilla.redhat.com/1003196 
This package seems to have been rushed through the review process
just like a few others recently. :-/

The duplication of libs and files is even worse (but my checker only
looks at Provides so far).

If I wanted to link with libmatedict, currently I can either

  BuildRequires: mate-dictionary-devel

or

  BuildRequires: mate-utils-devel

and even the pkgconfig file is duplicated. Actually, the two
-devel package contents don't differ at all:

mate-utils-devel
  http://koji.fedoraproject.org/koji/rpminfo?rpmID=4317455
mate-dictionary-devel
  http://koji.fedoraproject.org/koji/rpminfo?rpmID=4317456

[Stephen Gallagher]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 09/01/2013 03:09 PM, Michael Schwendt wrote:
>  https://bugzilla.redhat.com/1003196
>  
>  Based on this suspicious output
>  
>  mate-dictionary  from  mate-utils provides
>  libmatedict.so.6()(64bit) mate-utils  from  mate-utils provides
>  libmatedict.so.6()(64bit) required by:
>  mate-dictionary-devel-1.6.0-7.fc20.x86_64 required by:
>  mate-utils-devel-1.6.0-7.fc20.x86_64
>  
>  I've only verified in koji that lots of files are included in both 
>  sub-packages. Even the descriptions overlap.
>  
>  And there are even more subpackages, which only contain copies of 
>  files included in the base mate-utils package already. Why is that
>  done? Why aren't RPM dependencies used to have the base-package
>  depend on the multiple subpackages?
>  
>  So far, it has always been a packaging mistake to duplicate files
>  (and their Provides as a consequence) in multiple subpackages. 

Well, there are a few places where I can see duplicating files making
sense (but certainly not to the degree demonstrated in the mate packages).

For example, in the SSSD package, we duplicate the 'sssd_pac' libexec
binary in both the 'sssd-provider-ad' and 'sssd-provider-ipa' plugin
subpackages, rather than add useless metadata for an extra common
subpackage for both to depend on. It seems wasteful to have a whole
subpackage for one 150k binary.
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.14 (GNU/Linux)
> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>
> iEYEARECAAYFAlIl2FIACgkQeiVVYja6o6OCJwCeLlChllYfkw2XotaWFUsNN8bX
> ok8An2tDqieGQcnwXP8t9beDjTMLNBdB
> =pW9X
> -----END PGP SIGNATURE-----

[Susi Lehtola]

On Tue, 03 Sep 2013 08:38:42 -0400
Stephen Gallagher <sgallagh(a)redhat.com&gt; wrote:
>  On 09/01/2013 03:09 PM, Michael Schwendt wrote:
>  > https://bugzilla.redhat.com/1003196
>  > So far, it has always been a packaging mistake to duplicate files
>  > (and their Provides as a consequence) in multiple subpackages.
>  
>  Well, there are a few places where I can see duplicating files making
>  sense (but certainly not to the degree demonstrated in the mate
>  packages).
>  
>  For example, in the SSSD package, we duplicate the 'sssd_pac' libexec
>  binary in both the 'sssd-provider-ad' and 'sssd-provider-ipa' plugin
>  subpackages, rather than add useless metadata for an extra common
>  subpackage for both to depend on. It seems wasteful to have a whole
>  subpackage for one 150k binary. 
You may need to duplicate license files, but I really can't see why you
would duplicate binaries. It just makes no sense. That's what packages
are for: eliminating redundancies and tracking dependency info.
-- 
Susi Lehtola
Fedora Project Contributor
jussilehtola(a)fedoraproject.org

[Toshio Kuratomi]

On Tue, Sep 03, 2013 at 08:38:42AM -0400, Stephen Gallagher wrote:
>  -----BEGIN PGP SIGNED MESSAGE-----
>  Hash: SHA1
>  
>  On 09/01/2013 03:09 PM, Michael Schwendt wrote:
>  > https://bugzilla.redhat.com/1003196
>  > 
>  > Based on this suspicious output
>  > 
>  > mate-dictionary  from  mate-utils provides
>  > libmatedict.so.6()(64bit) mate-utils  from  mate-utils provides
>  > libmatedict.so.6()(64bit) required by:
>  > mate-dictionary-devel-1.6.0-7.fc20.x86_64 required by:
>  > mate-utils-devel-1.6.0-7.fc20.x86_64
>  > 
>  > I've only verified in koji that lots of files are included in both 
>  > sub-packages. Even the descriptions overlap.
>  > 
>  > And there are even more subpackages, which only contain copies of 
>  > files included in the base mate-utils package already. Why is that
>  > done? Why aren't RPM dependencies used to have the base-package
>  > depend on the multiple subpackages?
>  > 
>  > So far, it has always been a packaging mistake to duplicate files
>  > (and their Provides as a consequence) in multiple subpackages.
>  
>  
>  Well, there are a few places where I can see duplicating files making
>  sense (but certainly not to the degree demonstrated in the mate packages).
>  
>  For example, in the SSSD package, we duplicate the 'sssd_pac' libexec
>  binary in both the 'sssd-provider-ad' and 'sssd-provider-ipa' plugin
>  subpackages, rather than add useless metadata for an extra common
>  subpackage for both to depend on. It seems wasteful to have a whole
>  subpackage for one 150k binary.
https://fedoraproject.org/wiki/Packaging:Guidelines?rd=Packaging/Guidelin...

So that would also be a packaging mistake.  It's been many years since this
was last touched though.  IIRC, mschwendt raised the last issue with it so
he may be best able to recall the justifications for this rule and whether
the FPC should consider relaxing it.

-Toshio

[Michael Schwendt]

On Wed, 4 Sep 2013 07:47:55 -0700, Toshio Kuratomi wrote:

> https://fedoraproject.org/wiki/Packaging:Guidelines?rd=Packaging/Guidelin...
>   
Embarrassing. :) I was so focused on rpmbuild's "file listed twice"
warning that I didn't find this section in the guidelines. And rpmbuild's
warning is only per subpackage. It doesn't warn, if a file is included in
multiple subpackages.

>  So that would also be a packaging mistake.  It's been many years
> since this
>  was last touched though.  IIRC, mschwendt raised the last issue with it so
>  he may be best able to recall the justifications for this rule and whether
>  the FPC should consider relaxing it. 
Relaxing it would be bad.

I agree with the interest in including a small file (other than the
license file or small %doc files) in multiple subpackages. That's okay.
It doesn't cause any RPM file conflicts.

However, it boils down to: The packager must know what he/she is doing,
and particularly must be aware of the consequences. For example, if the
duplicate files result in duplicate Provides (e.g. for shared library
names), this may affect dependencies in non-obvious ways (e.g. pulling
in the wrong package because it happens to include a copy of the needed
lib but missing other run-time bits).


To build multiple -devel subpackages, which contain exactly the same
set of files, only adds confusion and doesn't add any value at all.

To build multiple subpackages, and in the base package include copies of
the files from those subpackages, doesn't make any sense. And it's
a pitfall. In bugzilla I've added a comment on the scenario where
you want to rename/replace a subpackage.


Originally, avoiding duplicate files in %files sections has been made a
checklist item, because of the risks and to have packagers verify
painstakingly which file belongs into which (sub)package. Not because
duplicated files are always a grave mistake, but because avoiding
packaging mistakes (accidents) leads to cleaner packages. Examples:
Splitting off a noarch -data subpackage is of no benefit, if the base
package includes the data files, too. Easy to fix, though. 
Splitting off individual plugin subpackages is one of the pitfalls, if
the base package contains copies of all plugins (just think about how to
recover from that).

[Michael Schwendt]

On Wed, 4 Sep 2013 22:00:06 +0200, Michael Schwendt wrote:

>  Originally, avoiding duplicate files in %files sections has been made
> a
>  checklist item, ... 
... and, of course, also to not include a file more than once in a single
%files section. That's a scenario such as with this application package,
but not limited to that:

    %files
    %{_bindir}/*
    %{_datadir}/*
    %{_libdir}/*
    %doc COPYING README NEWS
    %{_datadir}/applications/*.desktop
    %dir %{_libdir}/%{name}
    %{_libdir}/%{name}/plugins/

The entry for %_datadir/* also includes the .desktop file (well, perhaps
even other stuff in %_datadir/applications but that's a different prob ;).
The entry for %_libdir/* also includes %_libdir/%name and everything
below it. This confusing %files section bears the risk that the packager
makes mistakes when splitting of the plugins into subpackages, for example.

[The case when files are duplicated in several subpackages is worse.]

[Stephen Gallagher]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 09/04/2013 10:47 AM, Toshio Kuratomi wrote:
>  On Tue, Sep 03, 2013 at 08:38:42AM -0400, Stephen Gallagher wrote:
> > -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
> > 
> > On 09/01/2013 03:09 PM, Michael Schwendt wrote:
> >> https://bugzilla.redhat.com/1003196
> >> 
> >> Based on this suspicious output
> >> 
> >> mate-dictionary  from  mate-utils provides 
> >> libmatedict.so.6()(64bit) mate-utils  from  mate-utils
> >> provides libmatedict.so.6()(64bit) required by: 
> >> mate-dictionary-devel-1.6.0-7.fc20.x86_64 required by: 
> >> mate-utils-devel-1.6.0-7.fc20.x86_64
> >> 
> >> I've only verified in koji that lots of files are included in
> >> both sub-packages. Even the descriptions overlap.
> >> 
> >> And there are even more subpackages, which only contain copies
> >> of files included in the base mate-utils package already. Why
> >> is that done? Why aren't RPM dependencies used to have the
> >> base-package depend on the multiple subpackages?
> >> 
> >> So far, it has always been a packaging mistake to duplicate
> >> files (and their Provides as a consequence) in multiple
> >> subpackages.
> > 
> > 
> > Well, there are a few places where I can see duplicating files
> > making sense (but certainly not to the degree demonstrated in the
> > mate packages).
> > 
> > For example, in the SSSD package, we duplicate the 'sssd_pac'
> > libexec binary in both the 'sssd-provider-ad' and
> > 'sssd-provider-ipa' plugin subpackages, rather than add useless
> > metadata for an extra common subpackage for both to depend on. It
> > seems wasteful to have a whole subpackage for one 150k binary.
> > 
>
> https://fedoraproject.org/wiki/Packaging:Guidelines?rd=Packaging/Guidelin...
>
>   So that would also be a packaging mistake.  It's been many years
>  since this was last touched though.  IIRC, mschwendt raised the
>  last issue with it so he may be best able to recall the
>  justifications for this rule and whether the FPC should consider
>  relaxing it.
>   

For the record, I sent a patch to the SSSD upstream today to add a new
common sub-package for just that one file. It'll be cleaned up in the
next build.

> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.14 (GNU/Linux)
> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>
> iEYEARECAAYFAlInlmQACgkQeiVVYja6o6PeBACfdhsGLWJmYiyysLY7SCzJfuzC
> Y+sAn0m/csQLcFaPTYmO5ZFMXN1cNmUk
> =bbM7
> -----END PGP SIGNATURE-----

[Till Maas]

On Wed, Sep 04, 2013 at 07:47:55AM -0700, Toshio Kuratomi wrote:

> https://fedoraproject.org/wiki/Packaging:Guidelines?rd=Packaging/Guidelin...
>  
>  So that would also be a packaging mistake.  It's been many years since this
>  was last touched though.  IIRC, mschwendt raised the last issue with it so
>  he may be best able to recall the justifications for this rule and whether
>  the FPC should consider relaxing it. 
The python guidelines contain an example where %doc is used to include
the same doc files both in the python2 and python3 package:

https://fedoraproject.org/wiki/Packaging:Python#Example_spec_file

It sounds like it should be mentioned as an exception in the duplicate
files guidelines.

Also the python guidelines still use %{__python} but say at the same
time that %{__python2} should be used instead.

Regards
Till

[Michael Schwendt]

On Sat, 7 Sep 2013 17:55:46 +0200, Till Maas wrote:

>  On Wed, Sep 04, 2013 at 07:47:55AM -0700, Toshio Kuratomi wrote:
>  
>  >
> https://fedoraproject.org/wiki/Packaging:Guidelines?rd=Packaging/Guidelin...
>  > 
>  > So that would also be a packaging mistake.  It's been many years since this
>  > was last touched though.  IIRC, mschwendt raised the last issue with it so
>  > he may be best able to recall the justifications for this rule and whether
>  > the FPC should consider relaxing it.
>  
>  The python guidelines contain an example where %doc is used to include
>  the same doc files both in the python2 and python3 package:
>  
>  https://fedoraproject.org/wiki/Packaging:Python#Example_spec_file
>  
>  It sounds like it should be mentioned as an exception in the duplicate
>  files guidelines. 
Or drop that section from the Packaging and Review Guidelines and replace
it with something else. Exceptions soften the guidelines to a degree they
become useless. It's a subsection of "File and Directory Ownership":

| In most cases, it should not be necessary for multiple packages to
| contain identical copies of the same file. However, if it is necessary,

... that's the backdoor already ... ;)

| multiple packages may contain identical copies of the same file, as long
| as the following requirements are met:
|
| * The packages sharing ownership of the identical files are built from
|   a single SRPM. 
|
| OR
|
| * The packages sharing ownership of the identical files are not in a
|   dependency chain

... sometimes we _create_ package inter-dependencies when we split off
-common packages, for example - or a meta-package that depends on
multiple packages ...

|   (e.g. if package A requires package B, they should
|   not both contain identical files, either A or B must own the common
|   files, but not both.) 

The ReviewGuidelines refer to it in a second place:

| MUST: Packages must not own files or directories already owned by other
| packages. The rule of thumb here is that the first package to be installed
| should own the files or directories that other packages may rely
| upon. This means, for example, that no package in Fedora should ever share
| ownership with any of the files or directories owned by the filesystem or
| man package. If you feel that you have a good reason to own a file or
| directory that another package owns, then please present that at package
| review time.

That links the same "File and Directory Ownership" section and covers
subpackages, too, of course.

We need to ask ourselves, do we want to create tiny subpackages for
every single file that may be shared by several subpackages? Apparently
not according to above section and the exceptions for license/%doc files.
And we already have exceptions for directory ownership, too.

A proposal only for the Review Guidelines:

  MUST: The spec %files lists must be crafted carefully to ensure that no
  files or directories get included in the wrong packages.

  MUST: Don't duplicate files in more than one subpackage, unless there is
  a specific requirement to do so. If there is, give the rationale in a
  comment in the spec file. If it's many files, splitting of a shared package
  would be preferred. 

Known cases here are: the subpackage licensing guidelines, multiple alternative
packages that may want to include the same files and/or a few key %doc files.