Wildfly installation

[Peter Boy]

I hope I’m posting to right list here.

I would like to draw your attention to the fact that the Fedora Server Working Group has started working on ‚official‘ support for the installation of Java Application Server, especially Wildfly.

Since a Wildfly RPM is outside the scope of what is feasible, Ansible artifacts are to be used.

See: 
* https://fedoraproject.org/wiki/Server/Using_Ansible  (first use case Wildfly)
* https://meetbot.fedoraproject.org/fedora-meeting/2021-05-19/fedora-server.20... (17:29 ff)
* https://lists.fedoraproject.org/archives/list/[e-mail]...

There are technical challenges, of course. But there are also organizational challenges to doing that in a Fedora Policy compliant way.

It would be helpful if some Java experts would join or assist, who are not only more familiar with Wildfly than I am, but also with Fedora specific Java stuff.

Best
Peter

[Adam Young]

Why is an RPM not feasible?

On Mon, May 31, 2021, 2:31 PM Peter Boy  wrote:
> I hope I’m posting to right list here.
>
> I would like to draw your attention to the fact that the Fedora Server
> Working Group has started working on ‚official‘ support for the
> installation of Java Application Server, especially Wildfly.
>
> Since a Wildfly RPM is outside the scope of what is feasible, Ansible
> artifacts are to be used.
>
> See:
> https://fedoraproject.org/wiki/Server/Using_Ansible  (first use case
>
> Wildfly)
>
> https://meetbot.fedoraproject.org/fedora-meeting/2021-05-19/fedora-server.20...
> (17:29 ff)
>
> https://lists.fedoraproject.org/archives/list/[e-mail]...
>
> There are technical challenges, of course. But there are also
> organizational challenges to doing that in a Fedora Policy compliant way.
>
> It would be helpful if some Java experts would join or assist, who are not
> only more familiar with Wildfly than I am, but also with Fedora specific
> Java stuff.
>
> Best
> Peter
>
> java-devel mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]...
> Do not reply to spam on the list, report it:
> https://pagure.io/fedora-infrastructure

[Peter Boy]

> Am 01.06.2021 um 02:22 schrieb Adam Young :
>
> Why is an RPM not feasible?

I don’t know all the details. Please correct me if my knowledge is incomplete or incorrect. But as to my knowledge, according to Fedora packaging policy the source code must be completely compiled on Fedora infrastructure and each library must become a separate package. For very large and complex Java programs like Wildfly, this is an enormous amount of work and also fits poorly with the Java build system. In this respect, building a rpm is theoretically possible, but practically not feasible with the available resources. There was an attempt a few years ago, but it was not completed as to my knowledge.

If there is something wrong with this argument, please correct or add to it. This would be important for the attempt of a simple Wildfly installation.

[Carlo de Wolf]

On 01-06-2021 08:38, Peter Boy wrote:
> > Am 01.06.2021 um 02:22 schrieb Adam Young :
> >
> > Why is an RPM not feasible?
>
> I don’t know all the details. Please correct me if my knowledge is incomplete or incorrect. But as to my knowledge, according to Fedora packaging policy the source code must be completely compiled on Fedora infrastructure and each library must become a separate package. For very large and complex Java programs like Wildfly, this is an enormous amount of work and also fits poorly with the Java build system. In this respect, building a rpm is theoretically possible, but practically not feasible with the available resources. There was an attempt a few years ago, but it was not completed as to my knowledge.

Yes, the packaging guidelines are pretty strict. 
https://docs.fedoraproject.org/en-US/packaging-guidelines/Java/#_pre_built_d...

It was completed up to the point of a working Web Profile with some 
extensions. See https://fedoraproject.org/wiki/Features/JBossAS7

To give an indication of the amount of dependencies that would need to 
be packaged you can look at 
https://fedoraproject.org/wiki/JBossAS7#Current_progress

Nowadays we acknowledge 277 components in EAP 7.4 (which is based on 
WildFly 23).

The most important thing to understand is Fedora uses Xmvn to build 
Maven projects, not Apache Maven. In a very small nutshell it boils down 
to GAVs in poms being ignored and only (RPM) installed GAs are used.

Carlo
> If there is something wrong with this argument, please correct or add to it. This would be important for the attempt of a simple Wildfly installation.

[Mat Booth]

On Tue, 1 Jun 2021 at 10:10, Carlo de Wolf  wrote:
> On 01-06-2021 08:38, Peter Boy wrote:
> > > Am 01.06.2021 um 02:22 schrieb Adam Young :
> > >
> > > Why is an RPM not feasible?
> >
> > I don’t know all the details. Please correct me if my knowledge is incomplete or incorrect. But as to my knowledge, according to Fedora packaging policy the source code must be completely compiled on Fedora infrastructure and each library must become a separate package. For very large and complex Java programs like Wildfly, this is an enormous amount of work and also fits poorly with the Java build system. In this respect, building a rpm is theoretically possible, but practically not feasible with the available resources. There was an attempt a few years ago, but it was not completed as to my knowledge.
>
> Yes, the packaging guidelines are pretty strict.
> https://docs.fedoraproject.org/en-US/packaging-guidelines/Java/#_pre_built_d...
>
> It was completed up to the point of a working Web Profile with some
> extensions. See https://fedoraproject.org/wiki/Features/JBossAS7
>
> To give an indication of the amount of dependencies that would need to
> be packaged you can look at
> https://fedoraproject.org/wiki/JBossAS7#Current_progress
>
> Nowadays we acknowledge 277 components in EAP 7.4 (which is based on
> WildFly 23).
>
> The most important thing to understand is Fedora uses Xmvn to build
> Maven projects, not Apache Maven. In a very small nutshell it boils down
> to GAVs in poms being ignored and only (RPM) installed GAs are used.
>
> Carlo

Note that while Wildfly *was* packaging in Fedora, it no longer is
because it amounted to more than one person's full time job to keep it
and its dependencies updated and working in Fedora. For some history,
please see Marek's email from 2015:
https://lists.fedoraproject.org/archives/list/[e-mail]...

[Peter Boy]

> Am 01.06.2021 um 11:47 schrieb Mat Booth :
>
> Note that while Wildfly *was* packaging in Fedora, it no longer is
> because it amounted to more than one person's full time job to keep it
> and its dependencies updated and working in Fedora.

Thanks, so my reconstruction is basically correct (minus my impression it never got to fly).
> For some history,
> please see Marek's email from 2015:
> https://lists.fedoraproject.org/archives/list/[e-mail]...

Max Rydahl Andersen made some interesting remarks at the end, e.g. 
"drop the need for individual synced rpm's of each jar,“

Was there an attempt to do this? And would that really reduce the load so much that an rpm would be possible?

Sorry, may be a dumb question. I’m just trying to (re)sort the options.

The copr rpm at https://copr.fedorainfracloud.org/coprs/g/wildfly/wildfly/ will certainly never make it into an official Fedora release or other distributable because of the automatic download of an external binary.

Our idea is to make the system admin download the binary and when he decides to to that, assist him ("officially") to  install it properly (and easily).

It it works, I don’t know.

Di you see alternative options?

[Carlo de Wolf]

On 01-06-2021 13:36, Peter Boy wrote:
> > Am 01.06.2021 um 11:47 schrieb Mat Booth :
> >
> > Note that while Wildfly *was* packaging in Fedora, it no longer is
> > because it amounted to more than one person's full time job to keep it
> > and its dependencies updated and working in Fedora.
>
> Thanks, so my reconstruction is basically correct (minus my impression it never got to fly).
> > For some history,
> > please see Marek's email from 2015:
> > https://lists.fedoraproject.org/archives/list/[e-mail]...
>
> Max Rydahl Andersen made some interesting remarks at the end, e.g.
> "drop the need for individual synced rpm's of each jar,“
>
> Was there an attempt to do this? And would that really reduce the load so much that an rpm would be possible?

Yes, https://github.com/srcdeps/srcdeps

In theory it would reduce the load of creating the one rpm, but it 
doesn't solve the real problem of dependency alignment.

It is also not scale-able to the point of becoming realistically 
operational. You are effectively building an extremely large codebase.
> Sorry, may be a dumb question. I’m just trying to (re)sort the options.
>
> The copr rpm at https://copr.fedorainfracloud.org/coprs/g/wildfly/wildfly/ will certainly never make it into an official Fedora release or other distributable because of the automatic download of an external binary.
>
> Our idea is to make the system admin download the binary and when he decides to to that, assist him ("officially") to  install it properly (and easily).
>
> It it works, I don’t know.

What you describe is exactly how copr works. :)
An admin enables the copr repo and installs a custom rpm. Whatever 
happens next, happens.

Carlo
> Di you see alternative options?

[Adam Young]

What is the binary that is automatically downloaded?  That seems a High 
risk issue.

On 6/1/21 7:36 AM, Peter Boy wrote:
> > Am 01.06.2021 um 11:47 schrieb Mat Booth :
> >
> > Note that while Wildfly *was* packaging in Fedora, it no longer is
> > because it amounted to more than one person's full time job to keep it
> > and its dependencies updated and working in Fedora.
>
> Thanks, so my reconstruction is basically correct (minus my impression it never got to fly).
> > For some history,
> > please see Marek's email from 2015:
> > https://lists.fedoraproject.org/archives/list/[e-mail]...
>
> Max Rydahl Andersen made some interesting remarks at the end, e.g.
> "drop the need for individual synced rpm's of each jar,“
>
> Was there an attempt to do this? And would that really reduce the load so much that an rpm would be possible?
>
> Sorry, may be a dumb question. I’m just trying to (re)sort the options.
>
> The copr rpm at https://copr.fedorainfracloud.org/coprs/g/wildfly/wildfly/ will certainly never make it into an official Fedora release or other distributable because of the automatic download of an external binary.
>
> Our idea is to make the system admin download the binary and when he decides to to that, assist him ("officially") to  install it properly (and easily).
>
> It it works, I don’t know.
>
> Di you see alternative options?
>
> java-devel mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedoraproject.org/archives/list/[e-mail]...
> Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure

[Peter Boy]

> Am 01.06.2021 um 11:09 schrieb Carlo de Wolf :
>
> On 01-06-2021 08:38, Peter Boy wrote:
> > > Am 01.06.2021 um 02:22 schrieb Adam Young :
> > >
> > > Why is an RPM not feasible?
> >
> > I don’t know all the details. Please correct me if my knowledge is incomplete or incorrect. But as to my knowledge, according to Fedora packaging policy the source code must be completely compiled on Fedora infrastructure and each library must become a separate package. For very large and complex Java programs like Wildfly, this is an enormous amount of work and also fits poorly with the Java build system. In this respect, building a rpm is theoretically possible, but practically not feasible with the available resources. There was an attempt a few years ago, but it was not completed as to my knowledge.
>
> Yes, the packaging guidelines are pretty strict. https://docs.fedoraproject.org/en-US/packaging-guidelines/Java/#_pre_built_d...

Was there ever an initiative to make this more workable? Above a certain size, the maintenance advantage of splitting becomes a de facto disadvantage due to the sheer number of modules. So there should be exceptions for specific types of software.
> ...
> Nowadays we acknowledge 277 components in EAP 7.4 (which is based on WildFly 23).

An impressive number. How does RH distribute Wildfly rsp JBoss? Is everything a container now?
> The most important thing to understand is Fedora uses Xmvn to build Maven projects, not Apache Maven. In a very small nutshell it boils down to GAVs in poms being ignored and only (RPM) installed GAs are used.

Thanks. Didn’t know that. What I have read on the quick, it sounds very interesting. The last release is from 2019. Are there still more features being developed or is the tool considered mature / no need for additional features?

Peter

[Florian Weimer]

* Carlo de Wolf:
> Yes, the packaging guidelines are pretty
> strict. https://docs.fedoraproject.org/en-US/packaging-guidelines/Java/#_pre_built_d...

I expect that the building from source requirement is something that
cannot be negotiated at away.

But is this is really the problem?  That is, that it is currently not
possible to build Wildfly and its dependencies from source?

If you have sources that are buildable, perhaps bundling is a feasible
approach.

Thanks,
Florian

[Peter Boy]

> Am 01.06.2021 um 14:31 schrieb Florian Weimer :
> Carlo de Wolf:
> > Yes, the packaging guidelines are pretty
> > strict. https://docs.fedoraproject.org/en-US/packaging-guidelines/Java/#_pre_built_d...
>
> I expect that the building from source requirement is something that
> cannot be negotiated at away.

Agreed. In principle, this makes a lot of sense.
> But is this is really the problem?  That is, that it is currently not
> possible to build Wildfly and its dependencies from source?

As I understand it the issue is the sheer number of dependencies, obviously about 250+. The reason is that a separate rpm must be generated for each lib/jar.

Perhaps this could be negotiated for certain cases, like Wildfly. If that really reduces the workload drastically, that would be progress.

Peter

[Florian Weimer]

* Peter Boy:
> As I understand it the issue is the sheer number of dependencies,
> obviously about 250+. The reason is that a separate rpm must be
> generated for each lib/jar.
>
> Perhaps this could be negotiated for certain cases, like Wildfly. If
> that really reduces the workload drastically, that would be progress.

The guidelines currently say this:

Bundling and Duplication of system libraries
  https://docs.fedoraproject.org/en-US/packaging-guidelines/#_avoid_bundling_of_fonts_in_other_packages

This obviously has been written with C programs in mind.

The Java-specific guidelines do not say anythign abound bundling as far
as I can see.

Thanks,
Florian

[Carlo de Wolf]

On 01-06-2021 14:58, Florian Weimer wrote:
> Peter Boy:
> > As I understand it the issue is the sheer number of dependencies,
> > obviously about 250+. The reason is that a separate rpm must be
> > generated for each lib/jar.
> >
> > Perhaps this could be negotiated for certain cases, like Wildfly. If
> > that really reduces the workload drastically, that would be progress.
>
> The guidelines currently say this:
>
> Bundling and Duplication of system libraries
>    https://docs.fedoraproject.org/en-US/packaging-guidelines/#_avoid_bundling_of_fonts_in_other_packages
>
> This obviously has been written with C programs in mind.
>
> The Java-specific guidelines do not say anythign abound bundling as far
> as I can see.
>
> Thanks,
> Florian

https://docs.fedoraproject.org/en-US/packaging-guidelines/Java/#_pre_built_d...
"In particular *.class and *.jar files from upstream releases MUST NOT 
be used during build of Fedora packages and they MUST NOT be included in 
binary RPM."

In effect you need to build every dependency from source.

The main problem doesn't lie with the creation of RPMs, although it is a 
choir to get through. Getting through the build step is the real issue.

Maven builds with whatever it can get its hands on from Maven Central or 
anywhere else defined in poms. Going to a build which can only use 
whatever is actually available on a given target platform makes 
interesting issues pop. The actual available dependency may not be API 
compatible with whatever is defined in the pom. (Note that building with 
a target platform in mind is a good thing.)

This may have gotten better over the years, but I wouldn't count on it.

Carlo

[Peter Boy]

> Am 01.06.2021 um 14:58 schrieb Florian Weimer :
> Peter Boy:
> > As I understand it the issue is the sheer number of dependencies,
> > obviously about 250+. The reason is that a separate rpm must be
> > generated for each lib/jar.
> >
> > Perhaps this could be negotiated for certain cases, like Wildfly. If
> > that really reduces the workload drastically, that would be progress.
>
> The guidelines currently say this:
>
> Bundling and Duplication of system libraries
>  https://docs.fedoraproject.org/en-US/packaging-guidelines/#_avoid_bundling_of_fonts_in_other_packages
>
> This obviously has been written with C programs in mind.

Indeed
> The Java-specific guidelines do not say anythign abound bundling as far
> as I can see.

https://docs.fedoraproject.org/en-US/packaging-guidelines/Java/#_pre_built_d...

therein the following passus:

== JAR file installation ==
The following applies to all JAR files except JNI-using JAR files and application-specific JAR files (i.e., JAR files that can only reasonably be used as part of an application and therefore constitute application-private data).
=== Split JAR files ===
If a project offers the choice of packaging it as a single monolithic JAR or several ones, the split packaging SHOULD be preferred.

seems to be the problem.

[Florian Weimer]

* Peter Boy:
> https://docs.fedoraproject.org/en-US/packaging-guidelines/Java/#_pre_built_d...
>
> therein the following passus:
>
> == JAR file installation ==
> The following applies to all JAR files except JNI-using JAR files and application-specific JAR files (i.e., JAR files that can only reasonably be used as part of an application and therefore constitute application-private data).
> === Split JAR files ===
> If a project offers the choice of packaging it as a single monolithic JAR or several ones, the split packaging SHOULD be preferred.
>
> seems to be the problem.

Does Wildfly offer any packaging choices at all?

And it seems that the issue blocking Wildfly is actually *building*
stuff from source, so the delivery mechanism doesn't matter, I think.

Thanks,
Florian

[Carlo de Wolf]

On 01-06-2021 15:59, Florian Weimer wrote:
> Peter Boy:
> > https://docs.fedoraproject.org/en-US/packaging-guidelines/Java/#_pre_built_d...
> >
> > therein the following passus:
> >
> > == JAR file installation ==
> > The following applies to all JAR files except JNI-using JAR files and application-specific JAR files (i.e., JAR files that can only reasonably be used as part of an application and therefore constitute application-private data).
> > === Split JAR files ===
> > If a project offers the choice of packaging it as a single monolithic JAR or several ones, the split packaging SHOULD be preferred.
> >
> > seems to be the problem.

The splitting is all about what you need at the end of the day. Take 
https://github.com/hibernate/hibernate-orm/ as an example. WildFly 
doesn't need everything so it makes sense to split Hibernate ORM into 
different RPMs (or not produce RPMs for certain jars).

WildFly can be split in similar vain, however I would choose a higher 
granularity and take as much as you can into a single RPM to start with.
> Does Wildfly offer any packaging choices at all?

Yes, you can focus on a subset like the servlet distribution. 
https://github.com/wildfly/wildfly/blob/master/servlet-feature-pack/galleon-...

Probably then going for additional individual extensions to build up to 
what amounts to the full distribution.
> And it seems that the issue blocking Wildfly is actually *building*
> stuff from source, so the delivery mechanism doesn't matter, I think.
>
> Thanks,
> Florian

I'm not sure what you mean by this. Could you elaborate?

Carlo

[Neal Gompa]

On Tue, Jun 1, 2021 at 8:59 AM Florian Weimer  wrote:
> Peter Boy:
> > As I understand it the issue is the sheer number of dependencies,
> > obviously about 250+. The reason is that a separate rpm must be
> > generated for each lib/jar.
> >
> > Perhaps this could be negotiated for certain cases, like Wildfly. If
> > that really reduces the workload drastically, that would be progress.
>
> The guidelines currently say this:
>
> Bundling and Duplication of system libraries
>   https://docs.fedoraproject.org/en-US/packaging-guidelines/#_avoid_bundling_of_fonts_in_other_packages
>
> This obviously has been written with C programs in mind.

Please stop repeating this obvious falsehood. It's not true. It
started with C/C++ sure, but it has applied reasonably well to most
language ecosystems too. Patching one thing once and rebuild (or
rebuilds each reverse dep if static linked) is better than patching
one thing >500 different ways and rebuilding each instance.

--
真実はいつも一つ！/ Always, there's only one truth!

[Peter Boy]

> Am 03.06.2021 um 15:43 schrieb Neal Gompa :
>
> On Tue, Jun 1, 2021 at 8:59 AM Florian Weimer  wrote:
> > The guidelines currently say this:
> >
> > Bundling and Duplication of system libraries
> >  https://docs.fedoraproject.org/en-US/packaging-guidelines/#_avoid_bundling_of_fonts_in_other_packages
> >
> > This obviously has been written with C programs in mind.
>
> Please stop repeating this obvious falsehood. It's not true. It
> started with C/C++ sure, but it has applied reasonably well to most
> language ecosystems too.

It is that „Reasonable" what might be worth discussing (not a C/C++ ancestry), or whether, what was reasonable at the time still is today. Perhaps some "reasonable" adjustments are worth considering.
> Patching one thing once and rebuild (or
> rebuilds each reverse dep if static linked) is better than patching
> one thing >500 different ways and rebuilding each instance.

Unfortunately, that is not the problem in the case of Wildfly building. Instead of having >500 different ways and rebuilding, we have nothing to build at all, because it is not buildable that way (with existing resources). So, letting Wildfly build everything it needs into one (or some) rpms would make nothing worse, but only better. Of course, this is a very specific individual case and not suitable for generalization.