Re: introducing curl-minimal and libcurl-minimal RPM packages

[Kamil Dudka]

On Thursday, March 17, 2016 14:44:21 Przemek Klosowski wrote:
>  On 03/17/2016 02:00 PM, Kamil Dudka wrote:
>  > On Thursday 17 March 2016 13:21:42 Przemek Klosowski wrote:
>  >> Could you summarize the argument against something like
>  >> 
>  >> libcurl_version=REGULAR;
>  >> if(!(lp=dlopen("/usr/lib64/libcurl.so",RTLD_NOW))  {
>  >> 
>  >>        lp=dlopen("/usr/lib64/libcurl-minimal.so",RTLD_NOW);
>  >>        libcurl_version=MINIMAL;
>  >> 
>  >> }
>  >> if (!lp) abort("Can't load libcurl because ",dlerror());
>  > 
>  > I was (by mistake) speaking about loading libcurl's run-time dependencies
>  > by dlopen(), which is implemented for OpenLDAP in RHEL-5.  It used to
>  > cause
>  > problems and was removed from upstream curl long time ago.
>  > 
>  > Nevertheless, most of the reasons are valid for loading libcurl itself,
>  > too:
>  > 
>  > - Your example would require libcurl-devel to be installed because it
>  > provides> 
>  >    the /usr/lib64/libcurl.so symlink.  This would be yet another
>  >    *run-time*
>  >    
>  >    dependency of the package you patched.  See the following bug for 
example:
>  >      https://bugzilla.redhat.com/215928
>  
>  wait, why does libcurl.so sit in libcurl-devel? could it be simply in
>  libcurl? 
According to Fedora Packaging Guidelines, unversioned shared library files 
should be installed by -devel packages:

https://fedoraproject.org/wiki/Packaging:Guidelines?rd=Packaging/Guidelin...

If you click on the link above your question, you will see that there was 
exactly the same problem with openldap-devel.  So it is nothing specific
to libcurl.

>  of 548 /usr/lib64/lib*.so  files on my system, 358 are in
>  lib*-devel packages and 190 are in regular lib* packages.
>  
>  Is it a broader problem with how packages keep their libs, i.e. is one
>  group or the other violating some policy? 
Different packages have different policies for maintaining ABI compatibility.

>  > - This approach is not compatible with the dependency scanner
> of
>  > rmp-build.
>  > 
>  > - You would need to patch this way every single package to be able to load
>  > 
>  >    libcurl-minimal.  It is very likely that maintainers of upstream
>  >    projects
>  >    would reject such Fedora-specific patches.
>  
>  No, this would be only required for the curl---all the other packages
>  should expect regular libcurl.so 
Then it would not solve the problem with dnf because dnf uses libcurl via the 
python-pycurl binding (not the curl executable).

>  > - Loading libcurl by dlopen() changes the order of global
> initialization
>  > 
>  >    of system libraries (such as OpenSSL, NSS, and OpenLDAP), which itself
>  >    causes bugs from time to time.
>  > 
>  > There was a similar proposal for libcurl to load TLS libraries by
>  > dlopen().
>  > 
>  > You can read the response of the upstream curl maintainer:
>  >      https://lists.mayfirst.org/pipermail/vtls/2015-February/000020.html
>  
>  Yes, the portability argument.... I see the problem here....
>  
>  
>  
>  What if you created a 'minimal curl' package that would jam
>  libcurl-minimal into curl:
>  
>      LD_PRELOAD=/usr/lib64/libcurl-minimal.so curl 
I will not support any solution that would allow to install multiple instances
of libcurl on a single system.  We (as curl maintainers) have no control about 
which packages link which libraries and it could sooner or later happen that 
both instances of libcurl are loaded in a single process through higher-level 
libraries, which is not a scenario supported by upstream.

Apart from that, LD_PRELOAD is IMO as evil as RPATH, which we are discouraged 
from using in Fedora packages.  LD_PRELOAD is pretty useful while debugging.  
However, if you use it in production, it usually causes more problems than it 
solves.

Kamil

introducing curl-minimal and libcurl-minimal RPM packages

[Petr Pisar]

On 2016-03-18, Kamil Dudka <kdudka(a)redhat.com&gt; wrote:
>  On Thursday, March 17, 2016 14:44:21 Przemek Klosowski wrote:
>
>  According to Fedora Packaging Guidelines, unversioned shared library files 
>  should be installed by -devel packages:
>
>
> https://fedoraproject.org/wiki/Packaging:Guidelines?rd=Packaging/Guidelin...
>  
No. The guidelines are badly written. "unversioned shared library files"
means "shared library unversioned file name". Spliting files between
-libs and -devel packages is not based on sonames, but on purpose of the
file.

Dlopening unversioned file name is as bad as linking against librires
without sonames. To express package dependencies on soanames, one has to
dlopen the versioned file name. Of course that means to compile in the
soname when building an application. At the end that's what linker (ld)
does. Yes, it would mean to patch each application.

>  I will not support any solution that would allow to install multiple
>  instances of libcurl on a single system.  We (as curl maintainers)
>  have no control about which packages link which libraries and it could
>  sooner or later happen that both instances of libcurl are loaded in
>  a single process through higher-level libraries, which is not
>  a scenario supported by upstream.
>  
I think the solution is have more packages delivering the same-named
shared library file with the same soname. Each of the packages
conflicting each other. Then the non-minimal package would provide RPM
symbols declaring compiled-in features like "Provides: libcurl(LDAP)"
and then each application package requiring specific feature would
explicitly run-require it ("Requied: libcurl(LDAP)"), besides
automatically genererated dependency on the soname.

The magic of prefering the minimal package over non-minimal package
would be kept on package manager. For example it could sort the
candidates on size or number of dependencies.

-- Petr

introducing curl-minimal and libcurl-minimal RPM packages

[Kamil Dudka]

On Friday 18 March 2016 08:29:27 Petr Pisar wrote:
>  On 2016-03-18, Kamil Dudka <kdudka(a)redhat.com&gt; wrote:
>  > On Thursday, March 17, 2016 14:44:21 Przemek Klosowski wrote:
>  > 
>  > According to Fedora Packaging Guidelines, unversioned shared library files
>  > should be installed by -devel packages:
>  > 
>  > https://fedoraproject.org/wiki/Packaging:Guidelines?rd=Packaging/Guideline
>  > s#Devel_Packages
>  No. The guidelines are badly written. "unversioned shared library files"
>  means "shared library unversioned file name". Spliting files between
>  -libs and -devel packages is not based on sonames, but on purpose of the
>  file.
>  
>  Dlopening unversioned file name is as bad as linking against librires
>  without sonames. To express package dependencies on soanames, one has to
>  dlopen the versioned file name. Of course that means to compile in the
>  soname when building an application. At the end that's what linker (ld)
>  does. Yes, it would mean to patch each application.
>  
>  > I will not support any solution that would allow to install multiple
>  > instances of libcurl on a single system.  We (as curl maintainers)
>  > have no control about which packages link which libraries and it could
>  > sooner or later happen that both instances of libcurl are loaded in
>  > a single process through higher-level libraries, which is not
>  > a scenario supported by upstream.
>  
>  I think the solution is have more packages delivering the same-named
>  shared library file with the same soname. Each of the packages
>  conflicting each other. 
Up to this point, you are describing the originally proposed solution.  
Testing packages are available in the following Copr repository if you
want to give it a try:

https://copr.fedorainfracloud.org/coprs/kdudka/curl-minimal/

>  Then the non-minimal package would provide RPM
>  symbols declaring compiled-in features like "Provides: libcurl(LDAP)"
>  and then each application package requiring specific feature would
>  explicitly run-require it ("Requied: libcurl(LDAP)"), besides
>  automatically genererated dependency on the soname. 
Sounds like a reasonable improvement to me.  Thanks for the suggestion!

Kamil

> The magic of prefering the minimal package over non-minimal package
> would be kept on package manager. For example it could sort the
> candidates on size or number of dependencies.
> 
> -- Petr

introducing curl-minimal and libcurl-minimal RPM packages

[Nico Kadel-Garcia]

On Fri, Mar 18, 2016 at 4:29 AM, Petr Pisar <ppisar(a)redhat.com&gt; wrote:

>  I think the solution is have more packages delivering the same-named
>  shared library file with the same soname. Each of the packages
>  conflicting each other. Then the non-minimal package would provide RPM
>  symbols declaring compiled-in features like "Provides: libcurl(LDAP)"
>  and then each application package requiring specific feature would
>  explicitly run-require it ("Requied: libcurl(LDAP)"), besides
>  automatically genererated dependency on the soname. 
I suspect the better solution is to stop burning effort rewriting the
structure of core utilities for something that does not actually
improve the utility, but only helps with edge cases, in this case in
minimalizing build environments. This is extremely difficult to test,
and as likely to cause fracturing of unexpected test environments as
the reduced versions of "parted" caused in anaconda years ago.

>  The magic of prefering the minimal package over non-minimal package
>  would be kept on package manager. For example it could sort the
>  candidates on size or number of dependencies. 
Maintaining it is likely to break the ability to test expected
features when compiling in userland, then expect them to work the same
way in mock or koji at build time.

introducing curl-minimal and libcurl-minimal RPM packages

[Kamil Dudka]

On Friday 18 March 2016 05:14:28 Nico Kadel-Garcia wrote:
>  On Fri, Mar 18, 2016 at 4:29 AM, Petr Pisar <ppisar(a)redhat.com&gt;
> wrote:
>  > I think the solution is have more packages delivering the same-named
>  > shared library file with the same soname. Each of the packages
>  > conflicting each other. Then the non-minimal package would provide RPM
>  > symbols declaring compiled-in features like "Provides: libcurl(LDAP)"
>  > and then each application package requiring specific feature would
>  > explicitly run-require it ("Requied: libcurl(LDAP)"), besides
>  > automatically genererated dependency on the soname.
>  
>  I suspect the better solution is to stop burning effort rewriting the
>  structure of core utilities for something that does not actually
>  improve the utility 
Exactly.  That is why I am not proposing a crazy plug-in system
for libcurl :-)

>  , but only helps with edge cases, in this case in
>  minimalizing build environments. 
Neither me, nor Petr were talking about minimalizing build environments.

>  This is extremely difficult to test, 
Really?  What exactly were the problems you encountered while you
were testing the curl-minimal and libcurl-minimal packages?

Kamil

> and as likely to cause fracturing of unexpected test environments as
> the reduced versions of "parted" caused in anaconda years ago.
> 
> > The magic of prefering the minimal package over non-minimal package
> > would be kept on package manager. For example it could sort the
> > candidates on size or number of dependencies.
> 
> Maintaining it is likely to break the ability to test expected
> features when compiling in userland, then expect them to work the same
> way in mock or koji at build time.