JNI shared objects in %{_prefix}/lib

[Mikolaj Izdebski]

Hello,

I want to propose updating Java Packaging Guidelines [1] to allow
installation of JNI shared objects in %{_prefix}/lib/%{name} or in
%{_libdir}/%{name}, according to packager preference.

%{_libdir} expands to either /usr/lib or /usr/lib64 (depending on
system architecture), while %{_prefix}/lib always expands to /usr/lib.
To see what %{_libdir} expands to on each platform, you can run: grep
%_libdir -r /usr/lib/rpm/platform/

Currently Java Packaging Guidelines require all packages to install
JNI shared objects in %{_libdir}/%{name}, which causes issues for
noarch packages that want to refer to such shared objects. For example
Maven needs to install a symlink to libjansi.so. Since Maven is a
noarch package, the symlink must technically point to one fixed
location, but libjansi.so ought to be installed in different locations
on different architectures. Moving libjansi.so to one location across
all architectures solves this problem. An alternative solution
involves usage of a ghost symlink manipulated by scriplets, which in
my opinion is an inferior solution.

Java has been granted multilib exemption by FESCo [2], therefore I
would like to allow installation of JNI shared objects in
%{_prefix}/lib/%{name} or %{_libdir}/%{name}.

Does anyone have any objections or feedback before I go with this
request to FPC?

[1] https://docs.fedoraproject.org/en-US/packaging-guidelines/Java/
[2] https://pagure.io/fesco/issue/961

[Severin Gehwolf]

On Thu, 2021-10-07 at 08:33 +0200, Mikolaj Izdebski wrote:
>  Hello,
>  
>  I want to propose updating Java Packaging Guidelines [1] to allow
>  installation of JNI shared objects in %{_prefix}/lib/%{name} or in
>  %{_libdir}/%{name}, according to packager preference.
>  
>  %{_libdir} expands to either /usr/lib or /usr/lib64 (depending on
>  system architecture), while %{_prefix}/lib always expands to /usr/lib.
>  To see what %{_libdir} expands to on each platform, you can run: grep
>  %_libdir -r /usr/lib/rpm/platform/
>  
>  Currently Java Packaging Guidelines require all packages to install
>  JNI shared objects in %{_libdir}/%{name}, which causes issues for
>  noarch packages that want to refer to such shared objects. For example
>  Maven needs to install a symlink to libjansi.so. Since Maven is a
>  noarch package, the symlink must technically point to one fixed
>  location, but libjansi.so ought to be installed in different locations
>  on different architectures. Moving libjansi.so to one location across
>  all architectures solves this problem. An alternative solution
>  involves usage of a ghost symlink manipulated by scriplets, which in
>  my opinion is an inferior solution.
>  
>  Java has been granted multilib exemption by FESCo [2], therefore I
>  would like to allow installation of JNI shared objects in
>  %{_prefix}/lib/%{name} or %{_libdir}/%{name}.
>  
>  Does anyone have any objections or feedback before I go with this
>  request to FPC? 
Seems reasonable to me.

Thanks,
Severin

>  [1] https://docs.fedoraproject.org/en-US/packaging-guidelines/Java/
>  [2] https://pagure.io/fesco/issue/961
>  _______________________________________________
>  java-devel mailing list -- java-devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to java-devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]...
>  Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure

[Mat Booth]

On Thu, 7 Oct 2021 at 07:34, Mikolaj Izdebski <mizdebsk(a)redhat.com&gt; wrote:
>  Hello,
>
>  I want to propose updating Java Packaging Guidelines [1] to allow
>  installation of JNI shared objects in %{_prefix}/lib/%{name} or in
>  %{_libdir}/%{name}, according to packager preference.
>
>  %{_libdir} expands to either /usr/lib or /usr/lib64 (depending on
>  system architecture), while %{_prefix}/lib always expands to /usr/lib.
>  To see what %{_libdir} expands to on each platform, you can run: grep
>  %_libdir -r /usr/lib/rpm/platform/
>
>  Currently Java Packaging Guidelines require all packages to install
>  JNI shared objects in %{_libdir}/%{name}, which causes issues for
>  noarch packages that want to refer to such shared objects. For example
>  Maven needs to install a symlink to libjansi.so. Since Maven is a
>  noarch package, the symlink must technically point to one fixed
>  location, but libjansi.so ought to be installed in different locations
>  on different architectures. Moving libjansi.so to one location across
>  all architectures solves this problem. An alternative solution
>  involves usage of a ghost symlink manipulated by scriplets, which in
>  my opinion is an inferior solution.
>
>  Java has been granted multilib exemption by FESCo [2], therefore I
>  would like to allow installation of JNI shared objects in
>  %{_prefix}/lib/%{name} or %{_libdir}/%{name}.
>
>  Does anyone have any objections or feedback before I go with this
>  request to FPC?
>
>  [1] https://docs.fedoraproject.org/en-US/packaging-guidelines/Java/
>  [2] https://pagure.io/fesco/issue/961 
Could it not be argued that JNI objects should *always* be in
%{_prefix}/lib because Java itself is not multilib-aware?.

-- 
Mat Booth
http://fedoraproject.org/get-fedora

[Mikolaj Izdebski]

On Thu, Oct 7, 2021 at 11:08 AM Mat Booth <fedora(a)matbooth.co.uk&gt; wrote:
>  On Thu, 7 Oct 2021 at 07:34, Mikolaj Izdebski <mizdebsk(a)redhat.com&gt; wrote:
>  >
>  > Hello,
>  >
>  > I want to propose updating Java Packaging Guidelines [1] to allow
>  > installation of JNI shared objects in %{_prefix}/lib/%{name} or in
>  > %{_libdir}/%{name}, according to packager preference.
>  >
>  > %{_libdir} expands to either /usr/lib or /usr/lib64 (depending on
>  > system architecture), while %{_prefix}/lib always expands to /usr/lib.
>  > To see what %{_libdir} expands to on each platform, you can run: grep
>  > %_libdir -r /usr/lib/rpm/platform/
>  >
>  > Currently Java Packaging Guidelines require all packages to install
>  > JNI shared objects in %{_libdir}/%{name}, which causes issues for
>  > noarch packages that want to refer to such shared objects. For example
>  > Maven needs to install a symlink to libjansi.so. Since Maven is a
>  > noarch package, the symlink must technically point to one fixed
>  > location, but libjansi.so ought to be installed in different locations
>  > on different architectures. Moving libjansi.so to one location across
>  > all architectures solves this problem. An alternative solution
>  > involves usage of a ghost symlink manipulated by scriplets, which in
>  > my opinion is an inferior solution.
>  >
>  > Java has been granted multilib exemption by FESCo [2], therefore I
>  > would like to allow installation of JNI shared objects in
>  > %{_prefix}/lib/%{name} or %{_libdir}/%{name}.
>  >
>  > Does anyone have any objections or feedback before I go with this
>  > request to FPC?
>  >
>  > [1] https://docs.fedoraproject.org/en-US/packaging-guidelines/Java/
>  > [2] https://pagure.io/fesco/issue/961
>
>  Could it not be argued that JNI objects should *always* be in
>  %{_prefix}/lib because Java itself is not multilib-aware?. 
Indeed, myself I would always put them in %{_prefix}/lib in most (if
not all) cases, but there may be reasons for putting them in
%{_libdir}, for example when JNI code is only part of otherwise
multilib-aware package and you want to avoid file conflicts.

--
Mikolaj

>  --
>  Mat Booth
>  http://fedoraproject.org/get-fedora
>

[Florian Weimer]

* Mikolaj Izdebski:

>  I want to propose updating Java Packaging Guidelines [1] to allow
>  installation of JNI shared objects in %{_prefix}/lib/%{name} or in
>  %{_libdir}/%{name}, according to packager preference.
>
>  %{_libdir} expands to either /usr/lib or /usr/lib64 (depending on
>  system architecture), while %{_prefix}/lib always expands to /usr/lib.
>  To see what %{_libdir} expands to on each platform, you can run: grep
>  %_libdir -r /usr/lib/rpm/platform/ 
Isn't this a consequence of using absolute paths for JNI loading?

The issue goes away if the system default or OpenJDK default paths are
used.  Neither of these paths include %{name}, of course.

>  Since Maven is a noarch package, the symlink must technically point
> to
>  one fixed location, but libjansi.so ought to be installed in different
>  locations on different architectures. 
This also suggests a different problem: Why does Maven as a noarch
package even have to know that libjansi.so exists?  A future version of
jansi could very well be noarch.  I suspect it's related to dependencies
bubbling up to consumers (which is related to the Class-Path: attribute
ban).

Thanks,
Florian

[Mikolaj Izdebski]

On Fri, Oct 8, 2021 at 1:23 PM Florian Weimer <fweimer(a)redhat.com&gt; wrote:
>  * Mikolaj Izdebski:
>
>  > I want to propose updating Java Packaging Guidelines [1] to allow
>  > installation of JNI shared objects in %{_prefix}/lib/%{name} or in
>  > %{_libdir}/%{name}, according to packager preference.
>  >
>  > %{_libdir} expands to either /usr/lib or /usr/lib64 (depending on
>  > system architecture), while %{_prefix}/lib always expands to /usr/lib.
>  > To see what %{_libdir} expands to on each platform, you can run: grep
>  > %_libdir -r /usr/lib/rpm/platform/
>
>  Isn't this a consequence of using absolute paths for JNI loading?
>
>  The issue goes away if the system default or OpenJDK default paths are
>  used.  Neither of these paths include %{name}, of course. 
JNI shared objects are internal DSOs, which are expected to be loaded
and called only by corresponding Java bytecode. As such they should be
placed outside of dynamic linker search path, in a subdirectory called
%{name}. OpenJDK does not have code to load such DSOs. The Java
counterpart of particular JNI DSO typically contains specific code to
search for the DSO and load it.

A typical Java application is installed into a single directory tree
containing everything needed to run that application - Java bytecode,
native libraries, configuration files, data etc. Java applications
packaged as Fedora RPM packages install each of these into different
system directories - bytecode in /usr/share/java, native libraries in
/usr/lib or /usr/lib64, configuration files in /etc, data in /var and
so on. The single upstream directory is then created in
/usr/share/%{name}, but instead of regular files there are symbolic
links pointing to the actual location where the files are installed.

>  > Since Maven is a noarch package, the symlink must technically
> point to
>  > one fixed location, but libjansi.so ought to be installed in different
>  > locations on different architectures.
>
>  This also suggests a different problem: Why does Maven as a noarch
>  package even have to know that libjansi.so exists?  A future version of
>  jansi could very well be noarch.  I suspect it's related to dependencies
>  bubbling up to consumers (which is related to the Class-Path: attribute
>  ban). 
This is the way Maven is distributed and installed in binary form -
with all dependencies bundled in a single directory tree. Fedora maven
package needs to recreate this directory structure, but separate
native code from Java bytecode. Symbolic links are used to achieve
this split. And indeed, the latest Maven 3.8.3 (not yet in Fedora)
does not include libjansi for Linux, only for Windows.

--
Mikolaj Izdebski

[Florian Weimer]

* Mikolaj Izdebski:

>  On Fri, Oct 8, 2021 at 1:23 PM Florian Weimer
> <fweimer(a)redhat.com&gt; wrote:
> >
> > * Mikolaj Izdebski:
> >
> > > I want to propose updating Java Packaging Guidelines [1] to allow
> > > installation of JNI shared objects in %{_prefix}/lib/%{name} or in
> > > %{_libdir}/%{name}, according to packager preference.
> > >
> > > %{_libdir} expands to either /usr/lib or /usr/lib64 (depending on
> > > system architecture), while %{_prefix}/lib always expands to /usr/lib.
> > > To see what %{_libdir} expands to on each platform, you can run: grep
> > > %_libdir -r /usr/lib/rpm/platform/
> >
> > Isn't this a consequence of using absolute paths for JNI loading?
> >
> > The issue goes away if the system default or OpenJDK default paths are
> > used.  Neither of these paths include %{name}, of course.
>
>  JNI shared objects are internal DSOs, which are expected to be loaded
>  and called only by corresponding Java bytecode. As such they should be
>  placed outside of dynamic linker search path, in a subdirectory called
>  %{name}. OpenJDK does not have code to load such DSOs. 
I believe OpenJDK as packaged for Fedora tries to load from
/usr/java/packages/lib.  Unfortunately, this path has been somewhat
unstable (1.8.0 used /usr/java/packages/lib/amd64, it seems, and I've
also seen /usr/lib/jni and other variants), but that should not matter
for current Fedora Java packaging because 11 and 17 use the same path.

Furthermore, in the current glibc implementation, private shared
objects do not really exist because the dynamic loader tries too hard
to avoid loading objects with the same soname.  So even if the JVM
tries to partition the C symbol namespace with RTLD_LOCAL on a
per-class-loader basis, I doubt that this enables conflicting JNI
libraries.

> > > Since Maven is a noarch package, the symlink must
> technically point to
> > > one fixed location, but libjansi.so ought to be installed in different
> > > locations on different architectures.
> >
> > This also suggests a different problem: Why does Maven as a noarch
> > package even have to know that libjansi.so exists?  A future version of
> > jansi could very well be noarch.  I suspect it's related to dependencies
> > bubbling up to consumers (which is related to the Class-Path: attribute
> > ban).
>
>  This is the way Maven is distributed and installed in binary form -
>  with all dependencies bundled in a single directory tree. 
That makes sense for installing things from a ZIP file, without a
package manager.

>  Fedora maven package needs to recreate this directory structure, but
>  separate native code from Java bytecode. Symbolic links are used to
>  achieve this split. And indeed, the latest Maven 3.8.3 (not yet in
>  Fedora) does not include libjansi for Linux, only for Windows. 
Why do Fedora packages have to do this?  I thought that Maven does not
come with a module system.

I think it should be possible to use consistent JAR names, do away
with the symbolic links, and rely on the Class-Path attribute for
linking.  Doesn't Debian already do this?

[Mikolaj Izdebski]

On Mon, Oct 11, 2021 at 8:50 AM Florian Weimer <fw(a)deneb.enyo.de&gt; wrote:
>  * Mikolaj Izdebski:
>
>  > On Fri, Oct 8, 2021 at 1:23 PM Florian Weimer <fweimer(a)redhat.com&gt; wrote:
>  >>
>  >> * Mikolaj Izdebski:
>  >>
>  >> > I want to propose updating Java Packaging Guidelines [1] to allow
>  >> > installation of JNI shared objects in %{_prefix}/lib/%{name} or in
>  >> > %{_libdir}/%{name}, according to packager preference.
>  >> >
>  >> > %{_libdir} expands to either /usr/lib or /usr/lib64 (depending on
>  >> > system architecture), while %{_prefix}/lib always expands to /usr/lib.
>  >> > To see what %{_libdir} expands to on each platform, you can run: grep
>  >> > %_libdir -r /usr/lib/rpm/platform/
>  >>
>  >> Isn't this a consequence of using absolute paths for JNI loading?
>  >>
>  >> The issue goes away if the system default or OpenJDK default paths are
>  >> used.  Neither of these paths include %{name}, of course.
>  >
>  > JNI shared objects are internal DSOs, which are expected to be loaded
>  > and called only by corresponding Java bytecode. As such they should be
>  > placed outside of dynamic linker search path, in a subdirectory called
>  > %{name}. OpenJDK does not have code to load such DSOs.
>
>  I believe OpenJDK as packaged for Fedora tries to load from
>  /usr/java/packages/lib.  Unfortunately, this path has been somewhat
>  unstable (1.8.0 used /usr/java/packages/lib/amd64, it seems, and I've
>  also seen /usr/lib/jni and other variants), but that should not matter
>  for current Fedora Java packaging because 11 and 17 use the same path.
>
>  Furthermore, in the current glibc implementation, private shared
>  objects do not really exist because the dynamic loader tries too hard
>  to avoid loading objects with the same soname.  So even if the JVM
>  tries to partition the C symbol namespace with RTLD_LOCAL on a
>  per-class-loader basis, I doubt that this enables conflicting JNI
>  libraries.
>
>  >> > Since Maven is a noarch package, the symlink must technically point to
>  >> > one fixed location, but libjansi.so ought to be installed in different
>  >> > locations on different architectures.
>  >>
>  >> This also suggests a different problem: Why does Maven as a noarch
>  >> package even have to know that libjansi.so exists?  A future version of
>  >> jansi could very well be noarch.  I suspect it's related to dependencies
>  >> bubbling up to consumers (which is related to the Class-Path: attribute
>  >> ban).
>  >
>  > This is the way Maven is distributed and installed in binary form -
>  > with all dependencies bundled in a single directory tree.
>
>  That makes sense for installing things from a ZIP file, without a
>  package manager.
>
>  > Fedora maven package needs to recreate this directory structure, but
>  > separate native code from Java bytecode. Symbolic links are used to
>  > achieve this split. And indeed, the latest Maven 3.8.3 (not yet in
>  > Fedora) does not include libjansi for Linux, only for Windows.
>
>  Why do Fedora packages have to do this?  I thought that Maven does not
>  come with a module system. 
Because Maven directory structure can be considered as part of its
API, which other software relies upon. For example, users can
configure Jenkins or Eclipse to use Maven by pointing at Maven home
directory (/usr/share/maven). These tools may rely on directory layout
to do various things, such as getting version of Maven by opening
certain JAR files and reading embedded resources.

>  I think it should be possible to use consistent JAR names, do away
>  with the symbolic links, and rely on the Class-Path attribute for
>  linking.  Doesn't Debian already do this? 
That's not impossible, but upstream should make this change, not
downstream Fedora maintainers.
Like Fedora, Debian maven package also has /usr/share/maven directory
with symlinks.

--
Mikolaj Izdebski