Hi Tom,
First, thank you for contacting with us, providing feedback and sharing
ideas. I'm glad you wrote - it's quite uncommon for Java upstreams to
see value in having their projects available in Fedora or other
GNU/Linux distributions.
On 11/11/2014 03:05 AM, Tom Dunstan wrote:
After a few emails going around a couple of weeks ago, I would like
to
re-kick-off a discussion about how to get a functional Gradle build into
Fedora. Given the recent Fedora 21 beta, it also seems like a good time to
try to work out what might be achievable for Fedora 22.
I think that packaging latest Gradle 2.x should be doable for Fedora 22.
I will prepare an official Fedora change for Gadle in Fedora 22.
We also considered packaging 1.x line (as compat package "gradle1"), but
for now my focus was mainly on 2.1. (I will uptade to 2.2 soon.)
I've had a look at both the plan (
https://fedoraproject.org/wiki/User:Mizdebsk/PackagingGradle) and Mikolaj's
work here
https://github.com/mizdebsk/gradle-packaging, which look great,
and seems like a sane path to getting Gradle into Fedora.
Feedback
---
Some feedback on the existing TODO list and work done so far:
- TODO item "report usage of non-free JSON library (org.json:json)" - we
no longer use this
The non-free JSON lib is a dependency of XStream. We have reported this
as XSTR-763 [1]. This lib seems to be used during Gradle 2.1 build -
building Gradle pulls it into cache at $HOME/.gradle. This lib is also
hosted in Gradle repo [2]. Perhaps it should be explicitly excluded as
XStream upstream claims it's a test dependency only?
- Porting to aether-ant-tasks - we would probably accept a pull
request to
upstream this, as we'll need to do it at some point anyway. There may be
some work on our end to work out / around any differences between the
implementations. Has the Fedora Java team found much in the way of
incompatibility when porting between the two?
I have the patch mostly ready. I am definitely going to open a pull
request in future (I've already signed CLA), but the patch needs some
polishing and extra testing first.
There are two incompatibilities I discovered so far:
Gradle exposes maven-model in API. maven-model 2.x and 3.x are mostly
compatible, but not entirely, so I suppose that some third-party plugins
or build scripts would be affected by this change.
Second smaller issue: some internal fields of maven-ant-tasks (like
authentication info) can be injected from DSL. I found out that one
field was renamed from "userName" to "username". I think this should
be
solvable without breaking build scripts.
- Local mode / xmvn integration - this looks a lot cleaner than the
gradle
1.0 patches! I wonder if it would be possible to add the /usr/share/java
repo without patching gradle itself - maybe using a local init script to
add the repo when doing a Fedora build. But then the repo wouldn't be
available outside the Fedora build process.
The utlimate goal is to have two installations of Gradle. First
upstream-like, for standard use by Fedora users and second, with some
necessary Fedora-specific extensions enabled. The second installation
would have local repository (xmvn) enabled by default so that we don't
have to patch build scripts of every software we want to build for
Fedora using Gradle.
- Upgrading Jetty. Mostly we're pointing people to use the
external Gretty
plugin. See below for concerns.
AFAIK Gradle is at Java 1.5, but any Jetty version above 6 requires Java
1.6 or newer. Unfortunatelly this alone makes our Jetty patch not
upstreamable :(
One concern that we have is that any user-visible changes made to
Gradle
will result in non-portable build scripts. A user using e.g. Gradle 2.2
that they installed using yum will end up using Jetty 9, which is very
different to the Jetty 6 that users of the standard Gradle distribution
will use. This may then cause issues for users using the standard
distribution to build the target project.
We are aware of that. We'll try to minimize changes we apply to Gradle
(possibly by pushing patches upstream), but in some cases this may not
be possible.
In case of Jetty 6, from technical PoV we could add it to Fedora, but it
would be against our goals (for many reasons Fedora doesn't want to ship
legacy, unmaintained software). Current solution (porting to Jetty 9) is
definitely not ideal either and I'll try to think of some better way of
solving the problem.
The same goes for any plugins
that might not be enabled (there were lots of these in the earlier
Fedora-Gradle 1.0 patches),
The goal is to enable all stardard plugins. As of now all plugins are
enabled. Only Scala Zinc compiler is disabled, but this should be
eventually fixed.
or non-standard repositories such as xmvn.
xmvn repository was added temporarly only, until this problem is solved
the right way. (As I mentioned above, custom repository will be enabled
only in local mode, used only during package build and activated by
running Gradle with a different command, like "gradle-local").
While we would obviously prefer perfect compatibility, I can
certainly
understand not wanting to keep older versions of certain packages around
forever in Fedora. It would be nice if, when using different packages such
as Jetty, a warning were emitted in the build to notify the user that they
are using a different version to the normal Gradle distribution and that
the build may be non-portable.
That's an interesting idea. We can add a warning.
Another thing to consider would be modifying the version string to
include
"-fedora" or -"rpm" or something like that so that users with issues
turning up at
forums.gradle.org can report when they're using the rpm-based
distribution. Is there a standard Fedora policy for doing such things?
This is a good idea. For most packages we don't change upstream versions
(but usually we don't patch code either, besides bugfixes). In some
cases we add "Fedora" or "Red Hat" suffixes (for examples see
"gcc
--version" or "mvn -version" in RHEL).
I already tried adding Fedora version suffix, but GradleVersion class
implements strict version checking which rejected my first attempts to
use customized version. But I'll find a way to do this properly.
Do you have any recommendation how the version should look like to
satisfy Gradle versioning rules?
Testing
---
We have a large set of tests that we run as part of our continuous
integration and release processes. These tests take some hours to complete.
It looks like at least some rpm specs in fedora have the ability to run
bundled tests. What is the policy for when these tests are run? Does the
fedora build machinery run full test suites even if they take that long? We
would really like to see the full set of tests run against the patched
Gradle to get some level of confidence that the patches applied have not
harmed build compatibility.
During RPM package build we try to run all unit and integration tests
included in project sources. All test dependencies must be packaged as
Fedora package. If any test fails, the build is aborted and RPMs are not
created until maintainer fixes or disables failing test.
For external integration tests we have Taskotron [1]. It is still in
development and not ready for general use, but once it's ready we can
think of adding Gradle IT suite there. Test dependencies don't have to
be packaged in Fedora. Tasktron tests are ran after in testing phase
(after package is built, but before shipped) or on demand. Failing tests
don't prevent package from being shipped.
So far I haven't even been running Gradle unit tests during build. One
reason for that is we don't have all test dependencies packaged and some
tests would have to be ported to newer dependency versions. Some cases I
ported code to different Maven or Jetty versions, but left tests
unchanged. This is on my TODO list, but lower priority. Another reason
is that I am experimenting a bit with different patches and I don't want
to spend too much time on porting or fixing tests if the patch is going
to be reverted or changed anyways.
In short: with time we'll try to enable as many tests as possible, but
for now we may skip tests to speed up development.
Opportunities / Ideas
---
One idea that came to mind was enhancing the application plugin to create
application distributions that have symlinks to the correct jar inside
/usr/share/java rather than copying the jar into the distribution - that
might make life easier for devs / packagers trying to package Gradle-built
java applications.
That could be done.
There is xmvn-subst utility, which can be used to replace JAR with a
symlink, assuming that the JAR is coming from Fedora and has Maven
metadata installed.
Another idea to consider is allowing a user to add the Xmvn repo to their
buildscript / build classpaths without having to patch their project build
file. This could be achieved by dropping a file into $GRADLE_HOME/init.d
which adds the Xmvn repo when a particular system property is set. Another
system property could disable all other repos - this would be
building-rpm-mode. See
http://www.gradle.org/docs/current/userguide/init_scripts.html for an
example of how to do this.
As explained above, this is the goal, but I don't know yet how it will
be implemented. Most likely a separate Gradle home (sharing all or most
of JARs by symlinking them), with some extensions added.
Maven is a good example of such separation. We ship vanilla Maven (yes,
all our patches have been upstreamed!) in /usr/share/maven. This is used
when you invoke "mvn" command. But during package build "xmvn"
command
is invoked instead. It uses Maven home at /usr/share/xmvn, which is
identical to Maven home except that it has Fedora-specific extensions
enabled.
Questions
---
Are you folks aware of particular packages that Gradle uses or other issues
that are likely to cause problems while iterating through phase 3 of the
bootstrap plan? In particular if there's anything that we can do to help
that process along, it would be good to know about it.
Phase 3 is already done (I forgot to update the Wiki page). All we need
to do before Gradle can be pushed to Fedora is polishing packages,
reviewing them and fixing a few remaining bugs.
One that leaps to
mind is "shading" dependent libraries - we do this in a few places to stop
the api available to the buildscript from being polluted.
Shading is indeed a big problem, in general. I believe that shading
sould be done only as last resort and that there are better techniques
(like class loaders) to prevent conflicts of different library versions
or leaking of implementation classes through APIs.
I have already unshaded Maven 3 JARs (this was possible after I ported
Gradle not to use maven-ant-tasks). I will look into unshading all
remaining libraries.
A few random problems I found working on packaging Gradle (there are
more, but this email is already getting very long...):
* Ivy resolvers can't be used in Gradle 2.x any longer (and are
deprecated in 1.x). There is no public API for defining custom
repository. (My temporary solution: use internal repository APIs.)
* There is no official way for providing Gradle extensions (by extension
I mean addon that modifies Gradle behavior). Long term I think it would
be great if Gradle could use some IoC thechniques to inject
implementation classes, allowing extensions to easily override some
standard Gradle components. (My temporary solution: patch Gradle code.)
* Internal Gradle artifacts (such as core-impl) are not published in
artifact repositories. This makes more difficult to write extensions
that use internal APIs. (I assume this done intentionally to discourage
from using Gradle internals. My solution: create my own repo with
missing artifacts.)
* gradle-launcher has "Class-Path" entry in manifest, which is not
allowed in Fedora. I had to patch Gradle launchers to work without this
entry, but I am affraid that there may be third-party launchers that
assume Gradle can be ran with "java -jar gradle-launcher.jar", which
won't work in Fedora. It would be nice if this could somehow be solved
with upstream cooperation.
Although I guess it's a little bit off-topic for this list, do
you know if
the Xmvn repo structure is compatible with debian's /usr/share/maven-repo?
How different is the Xmvn repo from a normal maven or ivy repo on the
filesystem?
This is pefectly fine for java-devel list.
/usr/share/maven-repo/ in Debian is a standard Maven repository and
Gradle should be able to use it out of the box (through Ivy, without any
custom code).
Fedora "repository" is not a typical artifact repository at all. It
doesn't have a strict layout (artifacts can have arbitrary file names
and be placed at arbitrary locations in the file system). XMvn knows
where to find them by reading special metadada files stored in
/usr/share/maven-metadata/. Besides that we use custom version
resolution scheme, meaning that POMs from Fedora can't be directly used
by Gradle or Maven as they are. Some artifacts don't have POMs at all -
we have to generate effective POM from metadata on demand.
Thanks,
--
Mikolaj Izdebski
Software Engineer, Red Hat
IRC: mizdebsk
[1]
http://jira.codehaus.org/browse/XSTR-763
[2]
https://repo.gradle.org/gradle/libs/org/json/json/20140107/json-20140107.pom