To Require or not to Require?

[David Howells]

Hi,

I have a package (keyutils) that produces three RPMs: keyutils (programs),
keyutils-libs and keyutils-devel.  The programs in the keyutils RPM depend on
the libraries in the keyutils-libs RPM and pick up implicit dependencies thus:

	warthog>rpm -qRp keyutils-1.5.1-1.fc14.x86_64.rpm
	...
	config(keyutils) = 1.5.1-1.fc14
	...
	libkeyutils.so.1()(64bit)  
	libkeyutils.so.1(KEYUTILS_0.3)(64bit)  
	libkeyutils.so.1(KEYUTILS_1.0)(64bit)  
	libkeyutils.so.1(KEYUTILS_1.3)(64bit)  
	libkeyutils.so.1(KEYUTILS_1.4)(64bit)  
	...

So does the keyutils rpm need an explicit Requires on the keyutils-libs rpm in
the specfile or is the implicit library dependency sufficient?

rpmlint does not complain, but, for RHEL, rpmdiff does.

David

[Paul Howarth]

On 08/11/2011 01:19 PM, David Howells wrote:
>  I have a package (keyutils) that produces three RPMs: keyutils
> (programs),
>  keyutils-libs and keyutils-devel.  The programs in the keyutils RPM depend on
>  the libraries in the keyutils-libs RPM and pick up implicit dependencies thus:
>
>  	warthog>rpm -qRp keyutils-1.5.1-1.fc14.x86_64.rpm
>  	...
>  	config(keyutils) = 1.5.1-1.fc14
>  	...
>  	libkeyutils.so.1()(64bit)
>  	libkeyutils.so.1(KEYUTILS_0.3)(64bit)
>  	libkeyutils.so.1(KEYUTILS_1.0)(64bit)
>  	libkeyutils.so.1(KEYUTILS_1.3)(64bit)
>  	libkeyutils.so.1(KEYUTILS_1.4)(64bit)
>  	...
>
>  So does the keyutils rpm need an explicit Requires on the keyutils-libs rpm in
>  the specfile or is the implicit library dependency sufficient? 
Library requirements should be implicit unless there's a good reason 
otherwise; see:
http://fedoraproject.org/wiki/Packaging:Guidelines#Explicit_Requires

>  rpmlint does not complain, but, for RHEL, rpmdiff does. 
What is the rpmdiff "complaint"?

Paul.

[David Howells]

Paul Howarth <paul(a)city-fan.org&gt; wrote:

>  > rpmlint does not complain, but, for RHEL, rpmdiff does.
>  
>  What is the rpmdiff "complaint"? 
Report from TEST_REQUIRES:
[VERIFY] [keyutils] Subpackage keyutils on i686 ppc s390 consumes libraries
libkeyutils.so.1 libkeyutils.so.1(KEYUTILS_0.3) libkeyutils.so.1(KEYUTILS_1.0)
libkeyutils.so.1(KEYUTILS_1.3) from subpackage keyutils-libs but does not have explicit
package version requirement.
Please add Requires: keyutils-libs = %{version}-%{release} to keyutils in the specfile to
avoid the need to test interoperability between the various combinations of old and new
subpackages.

I guess this is more of a RHEL policy thing.

David

[Michael Schwendt]

On Thu, 11 Aug 2011 13:29:52 +0100, PH (Paul) wrote:

>  Library requirements should be implicit unless there's a good
> reason 
>  otherwise; see:
>  http://fedoraproject.org/wiki/Packaging:Guidelines#Explicit_Requires 
Not true, or "not the full story".

Library SONAME requirement for _external_ builds ought to stay
implicit/automatic, but _libraries and subpackages_ are a different
problem space. A library update may add stuff without changing its SONAME
and while staying compatible with existing executables in external packages.
But executables shipped with the library package (in subpackages) may
strictly need the very latest library.

I've posted another reply in this thread.

[Andreas Schwab]

Michael Schwendt <mschwendt(a)gmail.com&gt; writes:

>  Library SONAME requirement for _external_ builds ought to stay
>  implicit/automatic, but _libraries and subpackages_ are a different
>  problem space. A library update may add stuff without changing its SONAME
>  and while staying compatible with existing executables in external packages.
>  But executables shipped with the library package (in subpackages) may
>  strictly need the very latest library. 
There is no difference between "external" packages and subpackages: they
both will need the new symbol some day.

Andreas.

-- 
Andreas Schwab, schwab(a)redhat.com
GPG Key fingerprint = D4E8 DBE3 3813 BB5D FA84  5EC7 45C6 250E 6F00 984E
"And now for something completely different."

[Paul Howarth]

On 08/11/2011 02:03 PM, Andreas Schwab wrote:
>  Michael Schwendt<mschwendt(a)gmail.com>  writes:
>
> > Library SONAME requirement for _external_ builds ought to stay
> > implicit/automatic, but _libraries and subpackages_ are a different
> > problem space. A library update may add stuff without changing its SONAME
> > and while staying compatible with existing executables in external packages.
> > But executables shipped with the library package (in subpackages) may
> > strictly need the very latest library.
>
>  There is no difference between "external" packages and subpackages: they
>  both will need the new symbol some day. 
Indeed, which is why we have version-specific library dependencies on 
libssh2 in curl: https://bugzilla.redhat.com/show_bug.cgi?id=642796

Paul.

[Michael Schwendt]

On Thu, 11 Aug 2011 14:10:44 +0100, PH (Paul) wrote:

>  >> Library SONAME requirement for _external_ builds ought to
> stay
>  >> implicit/automatic, but _libraries and subpackages_ are a different
>  >> problem space. A library update may add stuff without changing its SONAME
>  >> and while staying compatible with existing executables in external packages.
>  >> But executables shipped with the library package (in subpackages) may
>  >> strictly need the very latest library.
>  >
>  > There is no difference between "external" packages and subpackages: they
>  > both will need the new symbol some day.
>  
>  Indeed, which is why we have version-specific library dependencies on 
>  libssh2 in curl: https://bugzilla.redhat.com/show_bug.cgi?id=642796 
This is just
http://fedoraproject.org/wiki/Packaging:Guidelines#Explicit_Requires
where a specific library version is needed actually and differs from
what has been released before. It is neither the norm nor mandatory.

[Michael Schwendt]

On Thu, 11 Aug 2011 15:03:47 +0200, AS (Andreas) wrote:

>  > Library SONAME requirement for _external_ builds ought to stay
>  > implicit/automatic, but _libraries and subpackages_ are a different
>  > problem space. A library update may add stuff without changing its SONAME
>  > and while staying compatible with existing executables in external packages.
>  > But executables shipped with the library package (in subpackages) may
>  > strictly need the very latest library.
>  
>  There is no difference between "external" packages and subpackages: they
>  both will need the new symbol some day. 
The difference is that the subpackages may need the new symbols immediately
when installing the packages, whereas future builds of external packages
would link with the latest library version that has been released before
and is the one available in the build system's buildroot.

Mind you, no external packages can be built against the library update before
it enters the buildroot (as a stable update).

[Andreas Schwab]

Michael Schwendt <mschwendt(a)gmail.com&gt; writes:

>  The difference is that the subpackages may need the new symbols
> immediately
>  when installing the packages, whereas future builds of external packages
>  would link with the latest library version that has been released before
>  and is the one available in the build system's buildroot. 
I don't understand.  How is the contents of the buildroot relevant to
"yum install foo"?

Andreas.

-- 
Andreas Schwab, schwab(a)redhat.com
GPG Key fingerprint = D4E8 DBE3 3813 BB5D FA84  5EC7 45C6 250E 6F00 984E
"And now for something completely different."

[Michael Schwendt]

On Thu, 11 Aug 2011 15:24:30 +0200, AS (Andreas) wrote:

>  > The difference is that the subpackages may need the new symbols
> immediately
>  > when installing the packages, whereas future builds of external packages
>  > would link with the latest library version that has been released before
>  > and is the one available in the build system's buildroot.
>  
>  I don't understand.  How is the contents of the buildroot relevant to
>  "yum install foo"? 
I think you do understand, but probably you just want to reinvent the 
wheel. Let's see:

No external package can build with new features of the new "foo-libs" package
prior to making that package available in the buildroot.

If "foo" is the base package of "foo-libs", and this -libs package
has
added symbols needed by programs in package "foo", an older release
of "foo-libs" would not suffice. On the contrary, all programs
linked with an older foo library will continue to work.

For the corner-case that an update of a program in an arbitrary package
strictly needs a specific minimum version of foo-libs, the packager may
add an explicit dependency according to:
http://fedoraproject.org/wiki/Packaging:Guidelines#Explicit_Requires 

You don't want to enforce explicit dependencies on library pkg NEVR
in _all_ packages that use the library, do you?

[Andreas Schwab]

Michael Schwendt <mschwendt(a)gmail.com&gt; writes:

>  No external package can build with new features of the new
> "foo-libs" package
>  prior to making that package available in the buildroot. 
How is the contents of the buildroot relevant to "yum install bar"?

>  If "foo" is the base package of "foo-libs", and
> this -libs package has
>  added symbols needed by programs in package "foo", an older release
>  of "foo-libs" would not suffice. On the contrary, all programs
>  linked with an older foo library will continue to work. 
How is that different from the above?

Andreas.

-- 
Andreas Schwab, schwab(a)redhat.com
GPG Key fingerprint = D4E8 DBE3 3813 BB5D FA84  5EC7 45C6 250E 6F00 984E
"And now for something completely different."

[Michael Schwendt]

On Thu, 11 Aug 2011 16:13:48 +0200, AS (Andreas) wrote:

>  > No external package can build with new features of the new
> "foo-libs" package
>  > prior to making that package available in the buildroot.
>  
>  How is the contents of the buildroot relevant to "yum install bar"? 
"bar" has been built with an older libfoo. The newer libfoo isn't strictly
needed, because any available package providing the needed ABI will suffice.

"foo", on the other hand, would be first to actually use the very latest
API extensions. Not always, but often (and more often that many ordinary
app bug-fix updates that have build dependencies on much older lib versions).
The explicit dep on the lib pkg is _not_ mandatory.

>  > If "foo" is the base package of "foo-libs",
> and this -libs package has
>  > added symbols needed by programs in package "foo", an older release
>  > of "foo-libs" would not suffice. On the contrary, all programs
>  > linked with an older foo library will continue to work.
>  
>  How is that different from the above? 
Well, feel free to toggle on/off the explicit dep whenever you think
that's appropriate. Have fun with the typical breakage when subpackages
get out of sync with base packages (such as in the case of a patched
-devel package not depending on the patched library base package -- which
has resulted in a lot of head-scratching when rebuilds against a new
-devel package didn't fix an issue because the fixed library wasn't
installed yet). Only a full "yum update" before and after any "yum
install" action will avoid many issues with dependencies.

[Andreas Schwab]

Michael Schwendt <mschwendt(a)gmail.com&gt; writes:

>  On Thu, 11 Aug 2011 16:13:48 +0200, AS (Andreas) wrote:
>
> > > No external package can build with new features of the new "foo-libs"
> package
> > > prior to making that package available in the buildroot.
> > 
> > How is the contents of the buildroot relevant to "yum install bar"?
>
>  "bar" has been built with an older libfoo. 
No.

Andreas.

-- 
Andreas Schwab, schwab(a)redhat.com
GPG Key fingerprint = D4E8 DBE3 3813 BB5D FA84  5EC7 45C6 250E 6F00 984E
"And now for something completely different."

[Michael Schwendt]

On Thu, 11 Aug 2011 17:23:06 +0200, AS (Andreas) wrote:

>  > On Thu, 11 Aug 2011 16:13:48 +0200, AS (Andreas) wrote:
>  >
>  >> > No external package can build with new features of the new
> "foo-libs" package
>  >> > prior to making that package available in the buildroot.
>  >> 
>  >> How is the contents of the buildroot relevant to "yum install bar"?
>  >
>  > "bar" has been built with an older libfoo.
>  
>  No. 
http://www.youtube.com/watch?v=kQFKtI6gn9Y
or what?

Any previously released "bar" has been built with an older libfoo.
Do you agree here?

Any update (or upgrade) of "bar" can only be built against a newer
libfoo if that enters the buildroot. Do you agree here?

For any update of "bar", you need to reconsider
https://fedoraproject.org/wiki/Packaging:Guidelines#Explicit_Requires

So, where are we now?

[Andreas Schwab]

Michael Schwendt <mschwendt(a)gmail.com&gt; writes:

>  So, where are we now? 
yum install bar doesn't update foo-libs automagically.

Andreas.

-- 
Andreas Schwab, schwab(a)redhat.com
GPG Key fingerprint = D4E8 DBE3 3813 BB5D FA84  5EC7 45C6 250E 6F00 984E
"And now for something completely different."

[Michael Schwendt]

On Fri, 12 Aug 2011 09:32:26 +0200, AS (Andreas) wrote:

>  > So, where are we now?
>  
>  yum install bar doesn't update foo-libs automagically. 
Which is why you may benefit from an explicit dependency *if* you
publish such an updated "bar" that needs a specific minimum version
of foo-libs:
http://fedoraproject.org/wiki/Packaging:Guidelines#Explicit_Requires

In this thread Paul has given a real-world example:
https://bugzilla.redhat.com/show_bug.cgi?id=642796

$ rpm -qR libcurl|grep ssh
libssh2(x86-64) >= 1.2.7
libssh2.so.1()(64bit)  

Such explicit deps are neither the norm nor mandatory.
In Fedora-land, a full "yum update" (or the equivalent thereof with a
different tool) is needed frequently anyway because of the updates-flood
and because of "yum install" not considering newest packages only. It
isn't any secret that Fedora users encounter lots of dep problems when
installing without updating frequently.

Back to subpackages. I could repeat my 1st reply in this thread, but
let's see.
We try to keep builds from the same src.rpm (and not limited to that) in
sync for various reasons. Not just because a program/extension included in
a subpkg may strictly require the very latest base pkg. Or vice versa
(the guidelines even mention the pitfall of creating a circular dep):
https://fedoraproject.org/wiki/Packaging/Guidelines#RequiringBasePackage

A foo.src.rpm that builds libs *and* programs doesn't need to be split into
several packages. It could also be split into foo + foo-utils or libfoo + foo,
and the naming scheme doesn't change which is the base pkg and which is the
subpkg.
So, if the build creates multiple subpackages, you *could* treat the libs
pkg like a separate/external build. However, library subpkg builds are
special because they build from the same src.rpm, with no risk that the
library package will change its name and break the explicit subpkg dep
created within the same package set, with no versioned BuildRequires on an
external library package, with no configure script checking for a minimum
library version -> any programs built within the same src.rpm may
immediately use this very latest lib API.

So, what is safer?

* To treat a foo-libs subpackage like a base package and have foo (or e.g.
foo-utils) depend on it with full NEVR?

* Or to treat it like an ordinary/individual library package with just the
automatic SONAME dep and reconsider an explicit dep only when building
against new library updates?
http://fedoraproject.org/wiki/Packaging:Guidelines#Explicit_Requires

[Andreas Schwab]

Michael Schwendt <mschwendt(a)gmail.com&gt; writes:

>  So, what is safer? 
Neither fixes the missing symbol.

Andreas.

-- 
Andreas Schwab, schwab(a)redhat.com
GPG Key fingerprint = D4E8 DBE3 3813 BB5D FA84  5EC7 45C6 250E 6F00 984E
"And now for something completely different."

[Michael Schwendt]

On Fri, 12 Aug 2011 12:12:15 +0200, AS (Andreas) wrote:

>  > So, what is safer?
>  
>  Neither fixes the missing symbol. 
There is no missing symbol.
Be more verbose. Your very brief replies don't give enough context.

[Andreas Schwab]

Michael Schwendt <mschwendt(a)gmail.com&gt; writes:

>  There is no missing symbol. 
Of course there is.  See
<http://permalink.gmane.org/gmane.linux.redhat.fedora.devel/152427>;.

Andreas.

-- 
Andreas Schwab, schwab(a)redhat.com
GPG Key fingerprint = D4E8 DBE3 3813 BB5D FA84  5EC7 45C6 250E 6F00 984E
"And now for something completely different."

[Michael Schwendt]

On Fri, 12 Aug 2011 13:00:57 +0200, AS (Andreas) wrote:

>  > There is no missing symbol.
>  
>  Of course there is.  See
>  <http://permalink.gmane.org/gmane.linux.redhat.fedora.devel/152427>;. 
No, there isn't, because of a subpackage dependency with full NEVR, even
if that may not be needed as a default. Elsewhere in this thread, Rex Dieter
has also recommended that one with this rationale:

| Since it's a subpkg from the same srpm, I'd opt to be on the safe side,
| and add an explicit versioned Requires

So, I think I understand your mission now. You're either just practising
some troll skills or pedantry. :-/

The separate "bar" package still is entirely irrelevant up to the point
when it is updated/upgraded and actually needs new symbols found only
in a latest library update. Then we're back at
https://fedoraproject.org/wiki/Packaging:Guidelines#Explicit_Requires
and you're free to act accordingly.

If you prefer Explicit Library NEVR Requires in _all_ packages, take that
up with the packaging committee.

[Andreas Schwab]

Michael Schwendt <mschwendt(a)gmail.com&gt; writes:

>  The separate "bar" package still is entirely irrelevant
It kills your entire argument.

Andreas.

-- 
Andreas Schwab, schwab(a)redhat.com
GPG Key fingerprint = D4E8 DBE3 3813 BB5D FA84  5EC7 45C6 250E 6F00 984E
"And now for something completely different."

[Michael Schwendt]

On Fri, 12 Aug 2011 14:30:46 +0200, Andreas Schwab <schwab(a)redhat.com&gt; wrote:

>  > The separate "bar" package still is entirely
> irrelevant
>  
>  It kills your entire argument.
>  
>  Andreas. 
It still doesn't.
libfoo update would need to come first. First come, first served. And
what packaging techniques to apply in the libfoo update has been the topic
of this thread:

| I have a package (keyutils) that produces three RPMs: keyutils
| (programs), keyutils-libs and keyutils-devel.
|
| ...
|
| So does the keyutils rpm need an explicit Requires on the keyutils-libs
| rpm in the specfile or is the implicit library dependency sufficient?

Whether any other update built against libfoo would also benefit from
an explicit dep is covered by the guidelines:
https://fedoraproject.org/wiki/Packaging:Guidelines#Explicit_Requires

http://permalink.gmane.org/gmane.linux.redhat.fedora.devel/152426

[Andreas Schwab]

Michael Schwendt <mschwendt(a)gmail.com&gt; writes:

>  libfoo update would need to come first. 
How?

Andreas.

-- 
Andreas Schwab, schwab(a)redhat.com
GPG Key fingerprint = D4E8 DBE3 3813 BB5D FA84  5EC7 45C6 250E 6F00 984E
"And now for something completely different."

[Michael Schwendt]

On Fri, 12 Aug 2011 15:54:45 +0200, Andreas Schwab <schwab(a)redhat.com&gt; wrote:

>  > libfoo update would need to come first.
>  
>  How?
>  
>  Andreas.
>   
If you're serious about discussing this further, show that.
I'm not going to reply to this thread anymore before tomorrow.

With your single-word reply you [once again] demonstrate that we're
likely talking past eachother. Perhaps it's due to language barriers
or attitude. Dunno. I don't want to read between the lines to guess
whether you understand the sentence you've quoted above. I think
you've misunderstood it. You sound as if you now refer to the process
of installing package updates rather than preparing and releasing packages.

[Andreas Schwab]

Your whole argument is just void.  There is not a single difference
between "external" packages and (sub-)packages.

Andreas.

-- 
Andreas Schwab, schwab(a)redhat.com
GPG Key fingerprint = D4E8 DBE3 3813 BB5D FA84  5EC7 45C6 250E 6F00 984E
"And now for something completely different."

[Miloslav Trmač]

On Fri, Aug 12, 2011 at 4:15 PM, Andreas Schwab <schwab(a)redhat.com&gt; wrote:
>  Your whole argument is just void.  There is not a single difference
>  between "external" packages and (sub-)packages. 
There is one difference:
with subpackages it is trivial to write a
Requires: that 1) is always strict enough and 2) is always satisfied.
For external packages achieving both requires non-trivial human
effort.
   Mirek

[Andreas Schwab]

Miloslav Trmač <mitr(a)volny.cz&gt; writes:

>  There is one difference: with subpackages it is trivial to write a
>  Requires: that 1) is always strict enough and 2) is always satisfied.
>  For external packages achieving both requires non-trivial human
>  effort. 
Yet you insist that your "external" package is always working.

Andreas.

-- 
Andreas Schwab, schwab(a)redhat.com
GPG Key fingerprint = D4E8 DBE3 3813 BB5D FA84  5EC7 45C6 250E 6F00 984E
"And now for something completely different."

[Matthew Garrett]

On Fri, Aug 12, 2011 at 04:18:34PM +0200, Miloslav Trmač wrote:
>  On Fri, Aug 12, 2011 at 4:15 PM, Andreas Schwab
> <schwab(a)redhat.com&gt; wrote:
>  > Your whole argument is just void.  There is not a single difference
>  > between "external" packages and (sub-)packages.
>  There is one difference: with subpackages it is trivial to write a
>  Requires: that 1) is always strict enough and 2) is always satisfied.
>  For external packages achieving both requires non-trivial human
>  effort. 
Not at all. The simplest thing to do is to ensure that any binary 
depends on at least the version of any libraries it was linked against, 
which is the way Debian handle this in the absence of maintainer 
overrides.

-- 
Matthew Garrett | mjg59(a)srcf.ucam.org

[Toshio Kuratomi]

On Fri, Aug 12, 2011 at 04:15:49PM +0200, Andreas Schwab wrote:
>  Your whole argument is just void.  There is not a single difference
>  between "external" packages and (sub-)packages.
>   
Here's a couple differences:

(libfoo-1.0 building subpackage foo-utils) is licensed LGPLv2
(libfoo-1.0.1 building subpackage foo-utils) is licensed LGPLv3

Our guidelines let you package the license information in the base package
when the subpackage depends on it.  In this case, the subpackage won't pull
in the updated license information based on soname alone.


libfoo-1.0 has private data:

foopublic.h
struct Data {
  int public;
  void *private;
}

fooprivate.h

struct PrivateData {
  [...]
}

Rightly or wrongly, upstream libfoo-1.0 has some additional utilities that
access the PrivateData.  Because the utilities are built from the libfoo
source, they can include the fooprivate.h header file and do this.  When
libfoo goes to 1.0.1, upstream changes the definition of PrivateData and
updates the utilities to work with the new datastructure.  Since the public
ABI stayed the same, the SONAME doesn't change and external programs
compiled against libfoo-1.0 continue to work but the utilities built as
a subpackage would be broken without stricter versioning.

-Toshio

[Matthew Garrett]

On Fri, Aug 12, 2011 at 08:27:13AM -0700, Toshio Kuratomi wrote:

>  Rightly or wrongly, upstream libfoo-1.0 has some additional utilities
> that
>  access the PrivateData.  Because the utilities are built from the libfoo
>  source, they can include the fooprivate.h header file and do this.  When
>  libfoo goes to 1.0.1, upstream changes the definition of PrivateData and
>  updates the utilities to work with the new datastructure.  Since the public
>  ABI stayed the same, the SONAME doesn't change and external programs
>  compiled against libfoo-1.0 continue to work but the utilities built as
>  a subpackage would be broken without stricter versioning. 
Upstream can change the ABI as much as they want without bumping the 
SONAME providing that the old interfaces are also present. It's entirely 
possible to end up with a situation where external binaries built 
against 1.0.1 won't run on 1.0.0 - the problem isn't limited to 
subpackages.

-- 
Matthew Garrett | mjg59(a)srcf.ucam.org

[Simo Sorce]

On Fri, 2011-08-12 at 16:40 +0100, Matthew Garrett wrote:
>  On Fri, Aug 12, 2011 at 08:27:13AM -0700, Toshio Kuratomi wrote:
>  
>  > Rightly or wrongly, upstream libfoo-1.0 has some additional utilities that
>  > access the PrivateData.  Because the utilities are built from the libfoo
>  > source, they can include the fooprivate.h header file and do this.  When
>  > libfoo goes to 1.0.1, upstream changes the definition of PrivateData and
>  > updates the utilities to work with the new datastructure.  Since the public
>  > ABI stayed the same, the SONAME doesn't change and external programs
>  > compiled against libfoo-1.0 continue to work but the utilities built as
>  > a subpackage would be broken without stricter versioning.
>  
>  Upstream can change the ABI as much as they want without bumping the 
>  SONAME providing that the old interfaces are also present. It's entirely 
>  possible to end up with a situation where external binaries built 
>  against 1.0.1 won't run on 1.0.0 - the problem isn't limited to 
>  subpackages. 
If rpmbuild does not add an implicit requires with libraryX >= <version
we built against> then it is certainly broken.

ABI *additions* w/o change of the SONAME happens every day in every
library.

If you do not want to set a require on the fedora package version, then
you could check what is the highest symbol version for all symbols in
the lib and require that. Unfortunately not all libraries do symbol
versioning so that would not work in many cases.

Simo.

-- 
Simo Sorce * Red Hat, Inc * New York

[Michael Schwendt]

On Fri, 12 Aug 2011 12:08:50 -0400, SS (Simo) wrote:

>  If rpmbuild does not add an implicit requires with libraryX >=
> <version
>  we built against> then it is certainly broken. 
One could also argue that an activity like "yum install ..." ought to
search for and apply the latest available updates of needed packages. Such
behaviour has been rejected some years ago, if memory serves correctly. 

[Obviously.  Because if something provides SONAME and is installed
already, it can be assumed that it isn't necessary to look for something
newer that also provides SONAME. A matter of definition. In the buildsys
we build against latest packages, so it would be natural to let package
installers drag in updates always, too -- instead of adding extra deps
everywhere, which aren't future-proof.]

[Seth Vidal]

On Sat, 2011-08-13 at 09:19 +0200, Michael Schwendt wrote:
>  On Fri, 12 Aug 2011 12:08:50 -0400, SS (Simo) wrote:
>  
>  > If rpmbuild does not add an implicit requires with libraryX >= <version
>  > we built against> then it is certainly broken.
>  
>  One could also argue that an activity like "yum install ..." ought to
>  search for and apply the latest available updates of needed packages. Such
>  behaviour has been rejected some years ago, if memory serves correctly.  

It was rejected with good reason, I think. I do not think we should be
adding functionality into yum which:

1. violates the principle of least surprise for the admin
2. covers up for not-specific-enough requirements in the packaging.


It just feels like it would be fixing a "problem" at the wrong layer.


-sv

[Michael Schwendt]

On Mon, 15 Aug 2011 16:20:47 -0400, SV (seth) wrote:

>  On Sat, 2011-08-13 at 09:19 +0200, Michael Schwendt wrote:
>  > On Fri, 12 Aug 2011 12:08:50 -0400, SS (Simo) wrote:
>  > 
>  > > If rpmbuild does not add an implicit requires with libraryX >= <version
>  > > we built against> then it is certainly broken.
>  > 
>  > One could also argue that an activity like "yum install ..." ought to
>  > search for and apply the latest available updates of needed packages. Such
>  > behaviour has been rejected some years ago, if memory serves correctly. 
>  
>  
>  It was rejected with good reason, I think. I do not think we should be
>  adding functionality into yum which:
>  
>  1. violates the principle of least surprise for the admin
>  2. covers up for not-specific-enough requirements in the packaging. 
Well, I can understand that point of view partially. The rationale
isn't entirely convincing, though. The more explicitly versioned dependencies
we would add to packages (either manually or automatically during rpmbuild),
the more updates a "yum install" would pull in. It won't be a full update,
but could break other installed packages.

Running out-of-date installations is not only a problem when a newly
installed package works only with latest updates. Ordinary bugs in
dependencies can also be a reason why the newly installed package will
need a full "yum update" before it would work [at all or correctly].
Ever seen users with an app startup crash and "yum update APP-PKG-NAME"
not fixing it becuase "yum update LIB-PKG-NAME" was necessary?

>  It just feels like it would be fixing a "problem" at the
> wrong layer. 
Where is the right layer?
Do we need to adjust the packaging policies wrt updates?

Fedora 15 by default displays update notifications less frequently.

The package installer could at least notify the user about available
updates immediately after installing a package. Some updates could be
important, especially if they are a dependency of the newly installed
package.

Users with out-of-date installations often are harder to support not just
because of old issues they run into.

If application misbehaviour is fixed in a system library, there is a
library package update, but the application package normally isn't rebuilt
just to add to it a "specific-enough requirement" on the updated library.
This isn't limited to one programming language. Even a Python app could
crash at run-time because of a bug in a Python module with an update that
has not been installed yet.

I think we are ill-advised if we publish a steady flood of updates (even for
old dist releases), but want users to install updates less frequently.

[Bryn M. Reeves]

On 08/12/2011 04:40 PM, Matthew Garrett wrote:
>  On Fri, Aug 12, 2011 at 08:27:13AM -0700, Toshio Kuratomi wrote:
>  
> > Rightly or wrongly, upstream libfoo-1.0 has some additional utilities that
> > access the PrivateData.  Because the utilities are built from the libfoo
> > source, they can include the fooprivate.h header file and do this.  When
> > libfoo goes to 1.0.1, upstream changes the definition of PrivateData and
> > updates the utilities to work with the new datastructure.  Since the public
> > ABI stayed the same, the SONAME doesn't change and external programs
> > compiled against libfoo-1.0 continue to work but the utilities built as
> > a subpackage would be broken without stricter versioning.
>  
>  Upstream can change the ABI as much as they want without bumping the 
>  SONAME providing that the old interfaces are also present. It's entirely 
>  possible to end up with a situation where external binaries built 
>  against 1.0.1 won't run on 1.0.0 - the problem isn't limited to 
>  subpackages.
>   
I think Toshio is talking about the case where the subpackage executables are
using things that are explicitly outside of any defined ABI (since they are
compiled from the same source tree and have access to non-public headers and
types defined within them). In his example PrivateData was never part of the
interface that external code using -devel sees.

Third party code built against -devel and depending only on the SONAME is fine
in this situation as it sticks to the published ABI. In-tree code that plays
with non-ABI symbols will break and so may need a stricter dep.

It's easy to argue that this is a misuse of the interface or that the interface
is inadequate/broken but I'm sure there are packages out there with this problem.

Bryn.

[Matthew Garrett]

On Fri, Aug 12, 2011 at 05:25:17PM +0100, Bryn M. Reeves wrote:

>  Third party code built against -devel and depending only on the
> SONAME is fine
>  in this situation as it sticks to the published ABI. In-tree code that plays
>  with non-ABI symbols will break and so may need a stricter dep. 
It is in this situation, but there are other situations where depending 
on the SONAME will cause breakage. If libfoo 1.1 adds a new symbol, 
anything built against it may fail to run against libfoo 1.0. But how 
will you know that in advance if all you have in your dependencies is 
the SONAME?

-- 
Matthew Garrett | mjg59(a)srcf.ucam.org

[Adam Jackson]

On 8/12/11 12:28 PM, Matthew Garrett wrote:
>  On Fri, Aug 12, 2011 at 05:25:17PM +0100, Bryn M. Reeves wrote:
>
> > Third party code built against -devel and depending only on the SONAME is fine
> > in this situation as it sticks to the published ABI. In-tree code that plays
> > with non-ABI symbols will break and so may need a stricter dep.
>
>  It is in this situation, but there are other situations where depending
>  on the SONAME will cause breakage. If libfoo 1.1 adds a new symbol,
>  anything built against it may fail to run against libfoo 1.0. But how
>  will you know that in advance if all you have in your dependencies is
>  the SONAME? 
In fairness, this is why rpm elaborates soname dependencies to also 
include symbol versions.  It's a pity that symbol versions are so 
painful to use that really only glibc makes any effort to do it.

Hilariously gcc _does_ let you specify symbol version in a __attribute__ 
tag, but only on HP/UX on ia64.  Thanks for that.

- ajax

[Doug Ledford]

On 8/12/2011 1:28 PM, Adam Jackson wrote:
>  On 8/12/11 12:28 PM, Matthew Garrett wrote:
> > On Fri, Aug 12, 2011 at 05:25:17PM +0100, Bryn M. Reeves wrote:
> >
> >> Third party code built against -devel and depending only on the SONAME is fine
> >> in this situation as it sticks to the published ABI. In-tree code that plays
> >> with non-ABI symbols will break and so may need a stricter dep.
> >
> > It is in this situation, but there are other situations where depending
> > on the SONAME will cause breakage. If libfoo 1.1 adds a new symbol,
> > anything built against it may fail to run against libfoo 1.0. But how
> > will you know that in advance if all you have in your dependencies is
> > the SONAME?
>  
>  In fairness, this is why rpm elaborates soname dependencies to also 
>  include symbol versions.  It's a pity that symbol versions are so 
>  painful to use that really only glibc makes any effort to do it. 
libibverbs uses symbol versions quite nicely.

>  Hilariously gcc _does_ let you specify symbol version in a
> __attribute__ 
>  tag, but only on HP/UX on ia64.  Thanks for that.
>  
>  - ajax 

-- 
Doug Ledford <dledford(a)redhat.com&gt;
              GPG KeyID: CFBFF194
	      http://people.redhat.com/dledford

Infiniband specific RPMs available at
	      http://people.redhat.com/dledford/Infiniband

[Daniel P. Berrange]

On Fri, Aug 12, 2011 at 01:28:59PM -0400, Adam Jackson wrote:
>  On 8/12/11 12:28 PM, Matthew Garrett wrote:
>  > On Fri, Aug 12, 2011 at 05:25:17PM +0100, Bryn M. Reeves wrote:
>  >
>  >> Third party code built against -devel and depending only on the SONAME is fine
>  >> in this situation as it sticks to the published ABI. In-tree code that plays
>  >> with non-ABI symbols will break and so may need a stricter dep.
>  >
>  > It is in this situation, but there are other situations where depending
>  > on the SONAME will cause breakage. If libfoo 1.1 adds a new symbol,
>  > anything built against it may fail to run against libfoo 1.0. But how
>  > will you know that in advance if all you have in your dependencies is
>  > the SONAME?
>  
>  In fairness, this is why rpm elaborates soname dependencies to also 
>  include symbol versions.  It's a pity that symbol versions are so 
>  painful to use that really only glibc makes any effort to do it. 
We make use of them in libvirt, via the '-Wl,--version-script=libvirt.syms'.
While it is slightly unfortunate to have to specify versions outside the
header file definitions, it isn't a significant problem. It is immediately
obvious if you forget to add new public APIs to sym file, since you won't
be able to link/test against them. So I wouldn't really call symbol
versoning painful.

What is unfortunate, is that if your library wants to also be maintained
on non-Linux platforms, then you can't make full use of the things that
glibc's versioning lets you do. eg introducing new APIs with the same
name, but different ABI, while not breaking existing linked apps.

Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|

[Simo Sorce]

On Fri, 2011-08-12 at 13:28 -0400, Adam Jackson wrote:
>  On 8/12/11 12:28 PM, Matthew Garrett wrote:
>  > On Fri, Aug 12, 2011 at 05:25:17PM +0100, Bryn M. Reeves wrote:
>  >
>  >> Third party code built against -devel and depending only on the SONAME is fine
>  >> in this situation as it sticks to the published ABI. In-tree code that plays
>  >> with non-ABI symbols will break and so may need a stricter dep.
>  >
>  > It is in this situation, but there are other situations where depending
>  > on the SONAME will cause breakage. If libfoo 1.1 adds a new symbol,
>  > anything built against it may fail to run against libfoo 1.0. But how
>  > will you know that in advance if all you have in your dependencies is
>  > the SONAME?
>  
>  In fairness, this is why rpm elaborates soname dependencies to also 
>  include symbol versions.  It's a pity that symbol versions are so 
>  painful to use that really only glibc makes any effort to do it. 
FWIW: talloc, tdb, tevent, and ldb and all of samba4 also do symbol
versioning since recently, thanks to waf build extensions that make it
simple to do.

>  Hilariously gcc _does_ let you specify symbol version in a
> __attribute__ 
>  tag, but only on HP/UX on ia64.  Thanks for that. 
[fail]

Simo.

-- 
Simo Sorce * Red Hat, Inc * New York

[Toshio Kuratomi]

On Fri, Aug 12, 2011 at 05:28:56PM +0100, Matthew Garrett wrote:
>  On Fri, Aug 12, 2011 at 05:25:17PM +0100, Bryn M. Reeves wrote:
>  
>  > Third party code built against -devel and depending only on the SONAME is fine
>  > in this situation as it sticks to the published ABI. In-tree code that plays
>  > with non-ABI symbols will break and so may need a stricter dep.
>  
>  It is in this situation, but there are other situations where depending 
>  on the SONAME will cause breakage. If libfoo 1.1 adds a new symbol, 
>  anything built against it may fail to run against libfoo 1.0. But how 
>  will you know that in advance if all you have in your dependencies is 
>  the SONAME?
>   
Yeah, this is what mschwendt was talking about with the potential of yum
install <package> leading to brokenness unless yum update is also performed.

My reply was really to refute Andreas's assertion that a subpackage doesn't
have any further concerns than an external package wrt version dependencies
which is wrong for both code-related reasons like this and for other factors
(like the relicensing example).

This thread has shown that we should probably update the guidelines to not
appear so draconian about explicit library dependencies (and highlight the
ABI/API implications and possibly get changes to rpmbuild), though.  Working
on an FPC ticket now.

Simo or mjg, one of you want to file a bug against rpmbuild since you guys
seem to know how Debian has already implemented this?

-Toshio

[Toshio Kuratomi]

On Fri, Aug 12, 2011 at 04:40:20PM +0100, Matthew Garrett wrote:
>  On Fri, Aug 12, 2011 at 08:27:13AM -0700, Toshio Kuratomi wrote:
>  
>  > Rightly or wrongly, upstream libfoo-1.0 has some additional utilities that
>  > access the PrivateData.  Because the utilities are built from the libfoo
>  > source, they can include the fooprivate.h header file and do this.  When
>  > libfoo goes to 1.0.1, upstream changes the definition of PrivateData and
>  > updates the utilities to work with the new datastructure.  Since the public
>  > ABI stayed the same, the SONAME doesn't change and external programs
>  > compiled against libfoo-1.0 continue to work but the utilities built as
>  > a subpackage would be broken without stricter versioning.
>  
>  Upstream can change the ABI as much as they want without bumping the 
>  SONAME providing that the old interfaces are also present. It's entirely 
>  possible to end up with a situation where external binaries built 
>  against 1.0.1 won't run on 1.0.0 - the problem isn't limited to 
>  subpackages.
>   
Sure.  But in this case, upstream isn't changing the public ABI.

It's a different level of mistake that's being practiced here.

-Toshio

[Matthew Garrett]

On Fri, Aug 12, 2011 at 09:26:33AM -0700, Toshio Kuratomi wrote:
>  On Fri, Aug 12, 2011 at 04:40:20PM +0100, Matthew Garrett wrote:
>  > Upstream can change the ABI as much as they want without bumping the 
>  > SONAME providing that the old interfaces are also present. It's entirely 
>  > possible to end up with a situation where external binaries built 
>  > against 1.0.1 won't run on 1.0.0 - the problem isn't limited to 
>  > subpackages.
>  > 
>  Sure.  But in this case, upstream isn't changing the public ABI.
>  
>  It's a different level of mistake that's being practiced here. 
What difference does it make? Even if you stick to the public ABI you 
can't guarantee that a matching SONAME is sufficient. You need to depend 
on the package version you build against.

-- 
Matthew Garrett | mjg59(a)srcf.ucam.org

[Michael Schwendt]

On Fri, 12 Aug 2011 17:30:10 +0100, MG (Matthew) wrote:

>  On Fri, Aug 12, 2011 at 09:26:33AM -0700, Toshio Kuratomi wrote:
>  > On Fri, Aug 12, 2011 at 04:40:20PM +0100, Matthew Garrett wrote:
>  > > Upstream can change the ABI as much as they want without bumping the 
>  > > SONAME providing that the old interfaces are also present. It's entirely 
>  > > possible to end up with a situation where external binaries built 
>  > > against 1.0.1 won't run on 1.0.0 - the problem isn't limited to 
>  > > subpackages.
>  > > 
>  > Sure.  But in this case, upstream isn't changing the public ABI.
>  > 
>  > It's a different level of mistake that's being practiced here.
>  
>  What difference does it make? Even if you stick to the public ABI you 
>  can't guarantee that a matching SONAME is sufficient. You need to depend 
>  on the package version you build against. 
With the difference that a library subpackage dependency can be _exact_
always, being an "equal to" dependency on full NEVR and staying correct
throughout the lifetime of this package set.

On the contrary, any separate package could at most depend on a library
package with a ">=" dependency on full NEV (or NEVR), or else a minor
version upgrade of the lib would break that dependency already. The
dependency would still only be a snapshot made at some point in time,
which would not be refreshed with updates of the library package, and
which would still not protect against unexpected accidental ABI/API
changes in future library updates -- and not against Epoch bumps either
(e.g. with an Epoch bump, 1.0.0 suddenly would be larger than 1.0.1 as
required by the external program package).

An "equal to" dep on full NEVR (not even NEV might suffice) would be an
expensive work-around only, because it forces packages to be rebuilt for
every minor lib update.

[Toshio Kuratomi]

On Fri, Aug 12, 2011 at 02:30:46PM +0200, Andreas Schwab wrote:
>  Michael Schwendt <mschwendt(a)gmail.com&gt; writes:
>  
>  > The separate "bar" package still is entirely irrelevant
>  
>  It kills your entire argument.
>   
AFAICS, your insistance that it is a blocker means that you aren't seeing
something correctly.

-Toshio

[Michael Schwendt]

On Thu, 11 Aug 2011 13:19:49 +0100, DH (David) wrote:

>  
>  Hi,
>  
>  I have a package (keyutils) that produces three RPMs: keyutils (programs),
>  keyutils-libs and keyutils-devel.  The programs in the keyutils RPM depend on
>  the libraries in the keyutils-libs RPM and pick up implicit dependencies thus:
>  
>  	warthog>rpm -qRp keyutils-1.5.1-1.fc14.x86_64.rpm
>  	...
>  	config(keyutils) = 1.5.1-1.fc14
>  	...
>  	libkeyutils.so.1()(64bit)  
>  	libkeyutils.so.1(KEYUTILS_0.3)(64bit)  
>  	libkeyutils.so.1(KEYUTILS_1.0)(64bit)  
>  	libkeyutils.so.1(KEYUTILS_1.3)(64bit)  
>  	libkeyutils.so.1(KEYUTILS_1.4)(64bit)  
>  	...
>  
>  So does the keyutils rpm need an explicit Requires on the keyutils-libs rpm in
>  the specfile or is the implicit library dependency sufficient?
>  
>  rpmlint does not complain, but, for RHEL, rpmdiff does.
>  
>  David 
It's a grey area. Basically, it boils down to these two guidelines:

  https://fedoraproject.org/wiki/Packaging/Guidelines#RequiringBasePackage
    under consideration of
  https://fedoraproject.org/wiki/Packaging:Guidelines#Explicit_Requires

In your case, the -libs package is the real base package in disguise.
The executables in the "keyutils" pkg may need the latest libkeyutils.so.1
API, or else they would fail to link at run-time.

[...]

Occasionally, one can observe users running into this pitfall as it is not
just a theoretical problem. They read about an update notification for
package "foo" and try to apply that update manually with "yum update
foo".
However, due to weak dependencies (or not dependencies) on subpackages
containing important stuff, such as the latest version-release of a "foo-libs"
subpackage, the applied update is only a partial one -- and not even a
"yum update foo\*" might have helped, because some subpackage names are
within a different prefix.

[Rex Dieter]

>  I have a package (keyutils) that produces three RPMs: keyutils
> (programs),
>  keyutils-libs and keyutils-devel.  The programs in the keyutils RPM depend on
>  the libraries in the keyutils-libs RPM and pick up implicit dependencies thus:
>  
>  	warthog>rpm -qRp keyutils-1.5.1-1.fc14.x86_64.rpm
>  	...
>  	config(keyutils) = 1.5.1-1.fc14
>  	...
>  	libkeyutils.so.1()(64bit)   
>  	...
>  So does the keyutils rpm need an explicit Requires on the keyutils-libs rpm in
>  the specfile or is the implicit library dependency sufficient? 
Since it's a subpkg from the same srpm, I'd opt to be on the safe side, and add an

explicit versioned Requires

-- rex