javapackages-tools auto-require: Why?

[Severin Gehwolf]

Hi,

See this bug for some background:
https://bugzilla.redhat.com/show_bug.cgi?id=1600426

Any Java package gets a "Requires: javapackages-tools" whether they
want it or not. It's part of the Java Packaging Guidelines[1]. Does
anybody remember the rationale as to why that exists? If the answer is
directory ownership for dirs like /usr/lib/jvm/ et.al. I'd consider
changing the guidelines to auto-requires javapackages-filesystem in
F29+.

Thoughts?

Thanks,
Severin

[1] https://fedoraproject.org/wiki/Packaging:Java#BuildRequires_and_Requires

[Aleksandar Kurtakov]

On Fri, Jul 13, 2018 at 12:00 PM, Severin Gehwolf <sgehwolf(a)redhat.com&gt;
wrote:

>  Hi,
>
>  See this bug for some background:
>  https://bugzilla.redhat.com/show_bug.cgi?id=1600426
>
>  Any Java package gets a "Requires: javapackages-tools" whether they
>  want it or not. It's part of the Java Packaging Guidelines[1]. Does
>  anybody remember the rationale as to why that exists? If the answer is
>  directory ownership for dirs like /usr/lib/jvm/ et.al. I'd consider
>  changing the guidelines to auto-requires javapackages-filesystem in
>  F29+.
>  
It makes perfect sense to switch it IMHO.


>  Thoughts?
>
>  Thanks,
>  Severin
>
>  [1] https://fedoraproject.org/wiki/Packaging:Java#
>  BuildRequires_and_Requires
>  _______________________________________________
>  java-devel mailing list -- java-devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to java-devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives: https://lists.fedoraproject.org/archives/list/java-devel@
>  lists.fedoraproject.org/message/A5UIG565C3LPB63ZIITNK6TBYMYT3GAZ/
>  


-- 
Alexander Kurtakov
Red Hat Eclipse Team

[Mikolaj Izdebski]

On 07/13/2018 11:00 AM, Severin Gehwolf wrote:
>  See this bug for some background:
>  https://bugzilla.redhat.com/show_bug.cgi?id=1600426
>  
>  Any Java package gets a "Requires: javapackages-tools" whether they
>  want it or not. It's part of the Java Packaging Guidelines[1]. Does
>  anybody remember the rationale as to why that exists? If the answer is
>  directory ownership for dirs like /usr/lib/jvm/ et.al. I'd consider
>  changing the guidelines to auto-requires javapackages-filesystem in
>  F29+. 
Typically packages require javapackages-tools during runtime for one or
both of two following major reasons:

1. Directory ownership. javapackages-tools requires
javapackages-filesystem, which owns many directories into which Java
packages install their files.

2. Java functions library. javapackages-tools contains library of shell
functions at /usr/share/java-utils/java-functions which is imported by
application launcher scripts written in shell. These functions may be
used to help loading system-wide or application-specific configuration,
constructing application classpath, determining proper JVM/JDK to use
with the application, and finally invoking JVM with appropriate flags,
including ABRT agent flags, and finally invoking JVM.

(There are also other minor, but valid reasons to require
javapckages-tools, such as for configuration files it contains.)

Changing auto-requires from javapackages-tools to
javapackages-filesystem and rebuilding all packages would break most of
launcher scripts. Explicit requires on javapackages-tools would have to
be added to packages that have launchers in order to fix them, reverting
some of benefits of the change.

Lets look at case of byteman, mentioned in the bug. Switching
auto-requires to javapackages-filesystem would not achieve much, as
byteman ships launcher scripts in /usr/bin/ that require
javapackages-tools to function properly, so explicit requires on
javapackages-tools would probably need to be added. Moreover, byteman
requires objectweb-asm, which also contains launcher script and
therefore would still need to require javapackages-tools for reason nr
2. above. For the auto-requries to have desired effect many packages
would need to be split into library and application parts. Like for
example, maven binary package contains only launcher script (plus
manpage and bash completion associated with the script), while maven-lib
contains library code.

>  Thoughts? 
Implementing this change would require modifying many Java packages to
add explicit requires to spec files. It would not affect users of Java
applications, which would still need to require javapackages-tools. It
would only possibly affect installations with Java libraries, but no
applications, provided that these libraries don't have dependencies on
applications.

So to me it seems like a lot of effort for not much benefit. But I'm not
opposed to the change, as long as people implementing it will take
responsibility for fixing all packages broken by it. Due to nature of
the change, affecting many packages across the distribution, I think it
should be a system-wide change approved by FESCo.

-- 
Mikolaj Izdebski
Senior Software Engineer, Red Hat
IRC: mizdebsk

[Severin Gehwolf]

On Fri, 2018-07-13 at 15:30 +0200, Mikolaj Izdebski wrote:
>  On 07/13/2018 11:00 AM, Severin Gehwolf wrote:
>  > See this bug for some background:
>  > https://bugzilla.redhat.com/show_bug.cgi?id=1600426
>  > 
>  > Any Java package gets a "Requires: javapackages-tools" whether they
>  > want it or not. It's part of the Java Packaging Guidelines[1]. Does
>  > anybody remember the rationale as to why that exists? If the answer is
>  > directory ownership for dirs like /usr/lib/jvm/ et.al. I'd consider
>  > changing the guidelines to auto-requires javapackages-filesystem in
>  > F29+.
>  
>  Typically packages require javapackages-tools during runtime for one or
>  both of two following major reasons:
>  
>  1. Directory ownership. javapackages-tools requires
>  javapackages-filesystem, which owns many directories into which Java
>  packages install their files.
>  
>  2. Java functions library. javapackages-tools contains library of shell
>  functions at /usr/share/java-utils/java-functions which is imported by
>  application launcher scripts written in shell. These functions may be
>  used to help loading system-wide or application-specific configuration,
>  constructing application classpath, determining proper JVM/JDK to use
>  with the application, and finally invoking JVM with appropriate flags,
>  including ABRT agent flags, and finally invoking JVM.
>  
>  (There are also other minor, but valid reasons to require
>  javapckages-tools, such as for configuration files it contains.)
>  
>  Changing auto-requires from javapackages-tools to
>  javapackages-filesystem and rebuilding all packages would break most of
>  launcher scripts. Explicit requires on javapackages-tools would have to
>  be added to packages that have launchers in order to fix them, reverting
>  some of benefits of the change. 
Question is how many of the Java packages use javapackages-tools for
2). Any ideas how we'd figure this out in some automated way?

>  Lets look at case of byteman, mentioned in the bug. Switching
>  auto-requires to javapackages-filesystem would not achieve much, as
>  byteman ships launcher scripts in /usr/bin/ that require
>  javapackages-tools to function properly, so explicit requires on
>  javapackages-tools would probably need to be added. Moreover, byteman
>  requires objectweb-asm, which also contains launcher script and
>  therefore would still need to require javapackages-tools for reason nr
>  2. above. 
Not really. byteman does not use what you describe as 2) from
javapackages-tools. That's just upstream functionality. Byteman also
doesn't require objectweb-asm (at runtime; its BR only), so that's not
good of an argument either. It would run fine with just java-openjdk
(JDK 10).

>  For the auto-requries to have desired effect many packages
>  would need to be split into library and application parts. Like for
>  example, maven binary package contains only launcher script (plus
>  manpage and bash completion associated with the script), while maven-lib
>  contains library code. 
Perhaps that would be a change for the better?

>  > Thoughts?
>  
>  Implementing this change would require modifying many Java packages to
>  add explicit requires to spec files. It would not affect users of Java
>  applications, which would still need to require javapackages-tools. It
>  would only possibly affect installations with Java libraries, but no
>  applications, provided that these libraries don't have dependencies on
>  applications.
>  
>  So to me it seems like a lot of effort for not much benefit. But I'm not
>  opposed to the change, as long as people implementing it will take
>  responsibility for fixing all packages broken by it. Due to nature of
>  the change, affecting many packages across the distribution, I think it
>  should be a system-wide change approved by FESCo. 
It sounds like a system-wide-change would be good to do for this one
way or another. One issue here is that javapackage-tools drags in java-
1.8.0-openjdk-headless. At some point that package will be gone in
Fedora. So looking ahead, it would be good to start thinking about this
now.

I'd be happy to help fix some of those packages. Looks like this
proposal would be F20 at the earliest.

Thanks,
Severin

[Mikolaj Izdebski]

On 07/13/2018 03:55 PM, Severin Gehwolf wrote:
>  On Fri, 2018-07-13 at 15:30 +0200, Mikolaj Izdebski wrote:
> > On 07/13/2018 11:00 AM, Severin Gehwolf wrote:
> >> See this bug for some background:
> >> https://bugzilla.redhat.com/show_bug.cgi?id=1600426
> >>
> >> Any Java package gets a "Requires: javapackages-tools" whether they
> >> want it or not. It's part of the Java Packaging Guidelines[1]. Does
> >> anybody remember the rationale as to why that exists? If the answer is
> >> directory ownership for dirs like /usr/lib/jvm/ et.al. I'd consider
> >> changing the guidelines to auto-requires javapackages-filesystem in
> >> F29+.
> >
> > Typically packages require javapackages-tools during runtime for one or
> > both of two following major reasons:
> >
> > 1. Directory ownership. javapackages-tools requires
> > javapackages-filesystem, which owns many directories into which Java
> > packages install their files.
> >
> > 2. Java functions library. javapackages-tools contains library of shell
> > functions at /usr/share/java-utils/java-functions which is imported by
> > application launcher scripts written in shell. These functions may be
> > used to help loading system-wide or application-specific configuration,
> > constructing application classpath, determining proper JVM/JDK to use
> > with the application, and finally invoking JVM with appropriate flags,
> > including ABRT agent flags, and finally invoking JVM.
> >
> > (There are also other minor, but valid reasons to require
> > javapckages-tools, such as for configuration files it contains.)
> >
> > Changing auto-requires from javapackages-tools to
> > javapackages-filesystem and rebuilding all packages would break most of
> > launcher scripts. Explicit requires on javapackages-tools would have to
> > be added to packages that have launchers in order to fix them, reverting
> > some of benefits of the change.
>  
>  Question is how many of the Java packages use javapackages-tools for
>  2). Any ideas how we'd figure this out in some automated way? 
Should be easy. Check packages that require javapackages-tools and check
if their executable files import /usr/share/java-utils/java-functions.

>  
> > Lets look at case of byteman, mentioned in the bug. Switching
> > auto-requires to javapackages-filesystem would not achieve much, as
> > byteman ships launcher scripts in /usr/bin/ that require
> > javapackages-tools to function properly, so explicit requires on
> > javapackages-tools would probably need to be added. Moreover, byteman
> > requires objectweb-asm, which also contains launcher script and
> > therefore would still need to require javapackages-tools for reason nr
> > 2. above.
>  
>  Not really. byteman does not use what you describe as 2) from
>  javapackages-tools. That's just upstream functionality. Byteman also
>  doesn't require objectweb-asm (at runtime; its BR only), so that's not
>  good of an argument either. It would run fine with just java-openjdk
>  (JDK 10). 
Right, byteman does not use java-functions library. After I checked its
file lists wrongly assumed it followed packaging guidelines for launcher
scripts, but it does not.

Also from what I can see byteman does require objectweb-asm during
runtime, but this is probably a bug as byteman bundles ASM, IIRC.

$ dnf -q --repo rawhide repoquery --requires byteman | grep asm
mvn(org.ow2.asm:asm)
mvn(org.ow2.asm:asm-analysis)
mvn(org.ow2.asm:asm-commons)
mvn(org.ow2.asm:asm-tree)

$ dnf -q --repo rawhide repoquery --whatprovides 'mvn(org.ow2.asm:asm)'
objectweb-asm-0:6.2-1.fc29.noarch

> > For the auto-requries to have desired effect many packages
> > would need to be split into library and application parts. Like for
> > example, maven binary package contains only launcher script (plus
> > manpage and bash completion associated with the script), while maven-lib
> > contains library code.
>  
>  Perhaps that would be a change for the better? 
Perhaps.

-- 
Mikolaj Izdebski
Senior Software Engineer, Red Hat
IRC: mizdebsk

[Severin Gehwolf]

On Sat, 2018-07-14 at 12:00 +0200, Mikolaj Izdebski wrote:
>  $ dnf -q --repo rawhide repoquery --requires byteman | grep asm
>  mvn(org.ow2.asm:asm)
>  mvn(org.ow2.asm:asm-analysis)
>  mvn(org.ow2.asm:asm-commons)
>  mvn(org.ow2.asm:asm-tree) 
This must have been a caching issue. The above command comes back empty
for me. That's not very surprising, since I've fixed this with byteman-
4.0.4-1[1].

Thanks,
Severin

[1] https://koji.fedoraproject.org/koji/buildinfo?buildID=1104966

[Severin Gehwolf]

Hi Mikolaj,

On Fri, 2018-07-13 at 15:30 +0200, Mikolaj Izdebski wrote:
>  On 07/13/2018 11:00 AM, Severin Gehwolf wrote:
>  > See this bug for some background:
>  > https://bugzilla.redhat.com/show_bug.cgi?id=1600426
>  > 
>  > Any Java package gets a "Requires: javapackages-tools" whether they
>  > want it or not. It's part of the Java Packaging Guidelines[1]. Does
>  > anybody remember the rationale as to why that exists? If the answer is
>  > directory ownership for dirs like /usr/lib/jvm/ et.al. I'd consider
>  > changing the guidelines to auto-requires javapackages-filesystem in
>  > F29+.
>  
>  Typically packages require javapackages-tools during runtime for one or
>  both of two following major reasons:
>  
>  1. Directory ownership. javapackages-tools requires
>  javapackages-filesystem, which owns many directories into which Java
>  packages install their files.
>  
>  2. Java functions library. javapackages-tools contains library of shell
>  functions at /usr/share/java-utils/java-functions which is imported by
>  application launcher scripts written in shell. These functions may be
>  used to help loading system-wide or application-specific configuration,
>  constructing application classpath, determining proper JVM/JDK to use
>  with the application, and finally invoking JVM with appropriate flags,
>  including ABRT agent flags, and finally invoking JVM.
>  
>  (There are also other minor, but valid reasons to require
>  javapckages-tools, such as for configuration files it contains.)
>  
>  Changing auto-requires from javapackages-tools to
>  javapackages-filesystem and rebuilding all packages would break most of
>  launcher scripts. Explicit requires on javapackages-tools would have to
>  be added to packages that have launchers in order to fix them, reverting
>  some of benefits of the change. 
[...]

Some data points from a rawhide query from today.

A) Total packages with "Requires: javapackages-tools": 4204
B) Total packages with "Requires: javapackages-tools" that are *not*
   -javadoc sub-packages: 2991
C) Total packages with "Requires: javapackages-tools", not a
   -javadoc sub-package and installing something in {,/usr}/bin: 145

>  Implementing this change would require modifying many Java packages
> to
>  add explicit requires to spec files. It would not affect users of Java
>  applications, which would still need to require javapackages-tools. It
>  would only possibly affect installations with Java libraries, but no
>  applications, provided that these libraries don't have dependencies on
>  applications. 
Given from the data above it would entail to change on the order of 145
(binary) packages[1].

I've also run specific analysis on those 145 packages in order to
figure out whether or not they source /usr/share/java-utils/java-
functions from javapackages-tools. It appears there are (roughly) 82
SRPM packages which may need changing[2]. 5 of those already use
explicit requires for javapackages-tools[3].

Note that this doesn't seem to capture packages like ant. ant itself
doesn't depend on javapackages-tools, but it depends on ant-lib which
does. Yet, the ant package installs binaries which use functions
exported by javapackages-tools, not ant-lib. There is also a case of
installed symlinks like for package eclipse-platform which symlinks to
ant. While eclipse-platform is a false positive, ant is a false
negative.

>  So to me it seems like a lot of effort for not much benefit. But
> I'm not
>  opposed to the change, as long as people implementing it will take
>  responsibility for fixing all packages broken by it. Due to nature of
>  the change, affecting many packages across the distribution, I think it
>  should be a system-wide change approved by FESCo. 
If you think a system-wide change is needed for this, then so be it.
I'll help get packages fixed and get explicit javapackages-tools
requirement added for the packages that need it.

Given that there are currently about 80 SRPM to potentially change, it
seems a reasonable effort. After all, all other packages will benefit
from the change.

Thanks,
Severin

[1] See attachment "bin_containing_packages_javapackages_tools.txt"
    for a list of specific binary RPM packages.
[2] See attachment "pkgs_needing_javapackages-tools_srpms_name.txt"
[3] console-image-viewer
    eclipse
    gradle
    liquibase
    will-crash