Packaging Webapps for Fedora

[Adam Young]

I'm one of the developers working on the next version of Dogtag, the 
Public Key Infrastructure packages (pki-*)  which is, as far as I can 
tell, the only Java Web app that actually ships in Fedora.

PKI is an old project, and it has vestiges of many  aspects of Java 
development that are just done differently today.  We are looking to 
close the gap between where it is and where the Standard lies.   As 
such, we have to figure out how to attack packaging the application.

Part of that, for Fedora, is to establish the standard in the first place.

Dogtag ships with a utility called  pkicreate.  This is an application 
that creates a Tomcat instance in (by default) /var/lib/pki-*  and sets 
the ports, the SELinux policy, the config, etc.  While I think it does 
more than it should, certain aspects of what it does are essential.  For 
example,  it handles all of the SELinux configuration for the Tomcat 
instance.  Tomcat ships with SSL  disabled, and  without an AJP 
connector to apache.  Thus,  Tomcat can't serve over port 80 or 443, 
meaning that they can't be viewed by many people behind firewalls.  
pkicreate configures mod_proxy_ajp support.   It performs the SELinux 
configuration for the HTTP  ports (if they are different from 8080 and 
8443 which should be handled by system policy), as well as the AJP port.

Russ Alberry, on the Debian Java team, has propsed a deployment utility 
similar in scope to what pkicreate does, but for the general case:

http://dep.debian.net/deps/dep7/

Fedora 17 and later will have  JBoss, which has a different set of 
deployment rules.  Geronimo is already part of Fedora, although I have 
to admit ignorance as far as how it would interact with Tomcat.  Resin  
is a possiblity as well.  So we don't want to hardcode the deployment 
approach to the Tomcat  server.  But, the Tomcat install should be 
supported first.

For a directory standard, I would say that a Fedora Web app should not 
have to provide a .war file, but should be able to use the tool to 
produce one .

Many web applications allow for post install customization.  THe 
simplest case is that a Company wants to theme the application with its 
own images and CSS.  If the web application is delivered in  a .war file 
and expects to deploy this way on the Servlet engine, there is no way to 
customize it.  Thus, Russ' approach listed above seems right on.  The 
one thing I would adjust is that deployment as a war file should be an 
option, but not necessarily the prefereed option.  MOst application 
servers have to unzip war files anyway, and it seems a waste and 
troublesome to zip up the file only to unzip it.  I say war is the 
exception but not the rule.

pkicreate is a Perl application.  As such, I don't think it will gain 
wide acceptance in the Java community.  But, it can server as a 
prototype for A Java based java-deploy-webapp.

Here's a short list of functions that I think the deployment tool should 
handle beyond what is listed in Russ' propsal:

1.  Configuring Security Realms
2.  Configuring JDBC connections (It would be nice if it configured 
Postgresql and MySQL for JDBC, too....)
3.  Configuring LDAP  connections
4.  Opening additional ports and providing SELinux configuration for them
5.  Creating Symlinks for shared libraries  to /usr/share/java and 
/usr/lib[64]/java
6.  Registering generic Global scoped components that can be accessed 
via JNDI names.

It  is possible that the workflow for java-deploy-webapp could best be 
done using either Ant or Maven.  However, I would not like to require a 
.pom or build.xml file as part of the deployment:  Web apps should 
follow the standard, and requiring additional configuration should be 
supported but not the norm.  Perhaps we look for something like 
/etc/webapp/name/pom.xml  and, if it doesn't exist, generate it from a 
skeleton.  Or, we can say that if they provide the pom.xml, then we 
support deploying this way, otherwise you are on your own.

[Stanislav Ochotnicky]

Excerpts from Adam Young's message of Wed Sep 28 16:13:06 +0200 2011:
> I'm one of the developers working on the next version of Dogtag, the
> Public Key Infrastructure packages (pki-*)  which is, as far as I can
> tell, the only Java Web app that actually ships in Fedora.
>
> PKI is an old project, and it has vestiges of many  aspects of Java
> development that are just done differently today.  We are looking to
> close the gap between where it is and where the Standard lies.   As
> such, we have to figure out how to attack packaging the application.
>
> Part of that, for Fedora, is to establish the standard in the first place.

I like your approach. It's great that you are trying to get the
packaging updated instead of working around it.
> Dogtag ships with a utility called  pkicreate.  This is an application
> that creates a Tomcat instance in (by default) /var/lib/pki-*  and sets
> the ports, the SELinux policy, the config, etc.  While I think it does
> more than it should, certain aspects of what it does are essential.  For
> example,  it handles all of the SELinux configuration for the Tomcat
> instance.  Tomcat ships with SSL  disabled, and  without an AJP
> connector to apache.  Thus,  Tomcat can't serve over port 80 or 443,
> meaning that they can't be viewed by many people behind firewalls.
> pkicreate configures mod_proxy_ajp support.   It performs the SELinux
> configuration for the HTTP  ports (if they are different from 8080 and
> 8443 which should be handled by system policy), as well as the AJP port.

I once prepared a tomcat webapp[1] for review and even though it never
got in in the end because I backed off...it should still be a good
starting point. Especially tomcat6 subpackage that installs one simple
XML into Catalina directory and that's all that's needed for tomcat to
recognize new webapp.

I can't really comment on different ports and such, but if you are
right then I guess system policy should be modified either when
installing tomcat itself or globally. I really don't think a second
tomcat instance is the way to go.
> Russ Alberry, on the Debian Java team, has propsed a deployment utility
> similar in scope to what pkicreate does, but for the general case:
>
> http://dep.debian.net/deps/dep7/

You might want to have a look at one of our SIG meeting logs[2] from
December of last year. We discussed said deployment utility and java
webapps in general there (though we never finished the discussion).
> Fedora 17 and later will have  JBoss, which has a different set of
> deployment rules.  Geronimo is already part of Fedora, although I have
> to admit ignorance as far as how it would interact with Tomcat.  Resin
> is a possiblity as well.  So we don't want to hardcode the deployment
> approach to the Tomcat  server.  But, the Tomcat install should be
> supported first.

I'd be careful about saying "Fedora 17 will have JBoss". Marek
Goldmann is trying hard and he's doing a great job, but there's still
a lot of work ahead of us.
> For a directory standard, I would say that a Fedora Web app should not
> have to provide a .war file, but should be able to use the tool to
> produce one .
>
> Many web applications allow for post install customization.  THe
> simplest case is that a Company wants to theme the application with its
> own images and CSS.  If the web application is delivered in  a .war file
> and expects to deploy this way on the Servlet engine, there is no way to
> customize it.  Thus, Russ' approach listed above seems right on.  The
> one thing I would adjust is that deployment as a war file should be an
> option, but not necessarily the prefereed option.  MOst application
> servers have to unzip war files anyway, and it seems a waste and
> troublesome to zip up the file only to unzip it.  I say war is the
> exception but not the rule.
>
> pkicreate is a Perl application.  As such, I don't think it will gain
> wide acceptance in the Java community.  But, it can server as a
> prototype for A Java based java-deploy-webapp.
>
> Here's a short list of functions that I think the deployment tool should
> handle beyond what is listed in Russ' propsal:
> Configuring Security Realms
> Configuring JDBC connections (It would be nice if it configured
>
> Postgresql and MySQL for JDBC, too....)
> 3.  Configuring LDAP  connections
> 4.  Opening additional ports and providing SELinux configuration for them
> 5.  Creating Symlinks for shared libraries  to /usr/share/java and
> /usr/lib[64]/java
> 6.  Registering generic Global scoped components that can be accessed
> via JNDI names.

I believe our initial idea for deployment tool was much more
simple. Maybe we just never understood the real needs. I believe the
discussion could/should be started again. Getting in touch with Russ
would be a good idea too I guess.
> It  is possible that the workflow for java-deploy-webapp could best be
> done using either Ant or Maven.  However, I would not like to require a
> .pom or build.xml file as part of the deployment:  Web apps should
> follow the standard, and requiring additional configuration should be
> supported but not the norm.  Perhaps we look for something like
> /etc/webapp/name/pom.xml  and, if it doesn't exist, generate it from a
> skeleton.  Or, we can say that if they provide the pom.xml, then we
> support deploying this way, otherwise you are on your own.

I don't think an ant/maven approach is viable, though I might be
wrong. Deployment using pom.xml is fine for upstreams, but we'd have
to create our own anyway because we are not really deploying to
upstream servers.

[1] https://bugzilla.redhat.com/show_bug.cgi?id=693646
[2] http://meetbot.fedoraproject.org/fedora-meeting-1/2010-12-08/fedora-meeting-...

--
Stanislav Ochotnicky 
Software Engineer - Base Operating Systems Brno

PGP: 7B087241
Red Hat Inc.                               http://cz.redhat.com

[Guido Grazioli]

> 2011/9/29 Stanislav Ochotnicky:
> > Excerpts from Adam Young's message of Wed Sep 28 16:13:06 +0200 2011:
>
> I once prepared a tomcat webapp[1] for review and even though it never
> got in in the end because I backed off...it should still be a good
> starting point. Especially tomcat6 subpackage that installs one simple
> XML into Catalina directory and that's all that's needed for tomcat to
> recognize new webapp.

That's true if you want to set some non-default paths, just dropping the
war in webapps dir is enough to make tomcat be aware of that and try to
start the new context. I think (but im not sure about that) that putting
xml files in Catalina directory, in particular when not inside 'localhost',
requires a tomcat restart.
> I can't really comment on different ports and such, but if you are
> right then I guess system policy should be modified either when
> installing tomcat itself or globally. I really don't think a second
> tomcat instance is the way to go.

Agree with that, but what do you have in mind for packaging webapps,
creating a single -tomcat | -jboss | -jetty | -whatever rpm during build
for each provided/supported container built from the webapp srpm?
> > Many web applications allow for post install customization.  THe
> > simplest case is that a Company wants to theme the application with its
> > own images and CSS.  If the web application is delivered in  a .war file
> > and expects to deploy this way on the Servlet engine, there is no way to
> > customize it.  Thus, Russ' approach listed above seems right on.  The
> > one thing I would adjust is that deployment as a war file should be an
> > option, but not necessarily the prefereed option.  MOst application
> > servers have to unzip war files anyway, and it seems a waste and
> > troublesome to zip up the file only to unzip it.  I say war is the
> > exception but not the rule.

My knoledge of JBoss is still on v4.3. On that one, i'd say that war/ear IS
actually the rule and not the exception, as JBoss does extract the
archive in a working
directory without custom configuration, and for istance there are other
discrepancies (allowed by specification), as getServletContent().getRealPath()
returning the actual absolute path in tomcat, and just null in JBoss.

Relying on the fact that wars have to be exploded in the context-s directory
is also very bad for clustering in my opinion.
> > pkicreate is a Perl application.  As such, I don't think it will gain
> > wide acceptance in the Java community.  But, it can server as a
> > prototype for A Java based java-deploy-webapp.
> >
> > Here's a short list of functions that I think the deployment tool should
> > handle beyond what is listed in Russ' propsal:
> > Configuring Security Realms
> > Configuring JDBC connections (It would be nice if it configured
> >
> > Postgresql and MySQL for JDBC, too....)
> > 3.  Configuring LDAP  connections
> > 4.  Opening additional ports and providing SELinux configuration for them
> > 5.  Creating Symlinks for shared libraries  to /usr/share/java and
> > /usr/lib[64]/java
> > 6.  Registering generic Global scoped components that can be accessed
> > via JNDI names.
>
> I believe our initial idea for deployment tool was much more
> simple. Maybe we just never understood the real needs. I believe the
> discussion could/should be started again. Getting in touch with Russ
> would be a good idea too I guess.

I dont think sysadministrators will likely rely on a script or gui to
edit their configuration files. But, a deploy manager could be very useful
for the casual user (I mean, anyone like developers or final users of
applications
written as webapps).

That's my hypothetical use-case:
1- in my fedora system i installed tomcat7 and jboss 7
2- i install a mywebapp rpm using yum (just like installing phpmyadmin,
but in this case the war is just sitting somewhere, no deploy)
3a- i install a mywebapp-tomcat7 bridge rpm using yum that does most
configuration, but not everything
or
3b- use a newly written deploy manager to deploy whatever in wherever

That could rise a lot of dependant questions:
- WEB-INF/lib: the proposal of linking to system jars just wont work so often,
as many webapps are written to particular library versions (of
commons- or just log4j)
- many webapps in the past needed endorsed libs or configuration of bootstrap
or jvm keystore
- most webapps need a database and come with hibernate, so which
backing db engine
to configure to be used with hibernate?
- how to make the user aware that the user/pass used for database creation
have to be changed? and you still have to get somewhere the grant for
creating such user
- what about specific settings as -Xms or -XX:PermGen for the jvm which could
be in conflict among webapps

Imho, a fully automated deploy manager is just too complex to implement,
and nevertheless, that would probably not  let you avoid small manual
configurations.
What could be really a gift, is an automated tool which does a maximum common
set of configurations, or even better, some sort of wizard that in a few steps
could let you have running some webapp, that you just had sitting
somewhere after
installing the relevant packages.

[Adam Young]

On 09/29/2011 07:06 AM, Guido Grazioli wrote:
> > 2011/9/29 Stanislav Ochotnicky:
> > > Excerpts from Adam Young's message of Wed Sep 28 16:13:06 +0200 2011:
> >
> > I once prepared a tomcat webapp[1] for review and even though it never
> > got in in the end because I backed off...it should still be a good
> > starting point. Especially tomcat6 subpackage that installs one simple
> > XML into Catalina directory and that's all that's needed for tomcat to
> > recognize new webapp.
>
> That's true if you want to set some non-default paths, just dropping the
> war in webapps dir is enough to make tomcat be aware of that and try to
> start the new context. I think (but im not sure about that) that putting
> xml files in Catalina directory, in particular when not inside 'localhost',
> requires a tomcat restart.

Restarting Tomcat for initial deploy is probably OK.  For hot deploy 
scenarios, people  will likely want JBoss, and we shouldn't be trying to 
reimplement that here.  Lets focus on the simple cases first.
> > I can't really comment on different ports and such, but if you are
> > right then I guess system policy should be modified either when
> > installing tomcat itself or globally. I really don't think a second
> > tomcat instance is the way to go.
>
> Agree with that, but what do you have in mind for packaging webapps,
> creating a single -tomcat | -jboss | -jetty | -whatever rpm during build
> for each provided/supported container built from the webapp srpm?

No.  I don't think this should be done at RPM package time.  I see three 
supported scenarios

1.  RPM creates a war file and that is what the tool  deploys as is.
2.  RPM create an exploded war directory.  Tool  merges in  config files 
from /etc/tomcat6/<webapp> with the code from /usr/share/java/webapps  
and creates a war file and deploys it
3. Same as 2 but code is deployed in exploded form.

For 2, copies of dependancy lbraries are put into WEB-INF/lib,  for 3, 
it is done with symlinks.
> > > Many web applications allow for post install customization.  THe
> > > simplest case is that a Company wants to theme the application with its
> > > own images and CSS.  If the web application is delivered in  a .war file
> > > and expects to deploy this way on the Servlet engine, there is no way to
> > > customize it.  Thus, Russ' approach listed above seems right on.  The
> > > one thing I would adjust is that deployment as a war file should be an
> > > option, but not necessarily the prefereed option.  MOst application
> > > servers have to unzip war files anyway, and it seems a waste and
> > > troublesome to zip up the file only to unzip it.  I say war is the
> > > exception but not the rule.
>
> My knoledge of JBoss is still on v4.3. On that one, i'd say that war/ear IS
> actually the rule and not the exception, as JBoss does extract the
> archive in a working
> directory without custom configuration, and for istance there are other
> discrepancies (allowed by specification), as getServletContent().getRealPath()
> returning the actual absolute path in tomcat, and just null in JBoss.
>
> Relying on the fact that wars have to be exploded in the context-s directory
> is also very bad for clustering in my opinion.

Agreed.  We should support both exploded and packaged deployments.
> > > pkicreate is a Perl application.  As such, I don't think it will gain
> > > wide acceptance in the Java community.  But, it can server as a
> > > prototype for A Java based java-deploy-webapp.
> > >
> > > Here's a short list of functions that I think the deployment tool should
> > > handle beyond what is listed in Russ' propsal:
> > > Configuring Security Realms
> > > Configuring JDBC connections (It would be nice if it configured
> > >
> > > Postgresql and MySQL for JDBC, too....)
> > > 3.  Configuring LDAP  connections
> > > 4.  Opening additional ports and providing SELinux configuration for them
> > > 5.  Creating Symlinks for shared libraries  to /usr/share/java and
> > > /usr/lib[64]/java
> > > 6.  Registering generic Global scoped components that can be accessed
> > > via JNDI names.
> >
> > I believe our initial idea for deployment tool was much more
> > simple. Maybe we just never understood the real needs. I believe the
> > discussion could/should be started again. Getting in touch with Russ
> > would be a good idea too I guess.
>
> I dont think sysadministrators will likely rely on a script or gui to
> edit their configuration files. But, a deploy manager could be very useful
> for the casual user (I mean, anyone like developers or final users of
> applications
> written as webapps).

A good sysadmin should be using something like puppet to ensure that all 
servers are kept in sync,  and so on.  Automated deployments should 
provide a reasonable set of defaults for a web application.It may be 
that we want to "wizardize it"  for some choices, like "you have both 
postgres and mysql installed, which do you want to use"
> That's my hypothetical use-case:
> 1- in my fedora system i installed tomcat7 and jboss 7
> 2- i install a mywebapp rpm using yum (just like installing phpmyadmin,
> but in this case the war is just sitting somewhere, no deploy)
> 3a- i install a mywebapp-tomcat7 bridge rpm using yum that does most
> configuration, but not everything
> or
> 3b- use a newly written deploy manager to deploy whatever in wherever
>
> That could rise a lot of dependant questions:
> WEB-INF/lib: the proposal of linking to system jars just wont work so often,
>
> as many webapps are written to particular library versions (of
> commons- or just log4j)

In exploded view do the linking,  for a war view, do a copy.  This is 
for Fedora applications, so they should be deploying with the Fedora 
versions of standard libraries.  A web app that wants to deploy with 
F17   and uses log4j needs to work with the version deployed with F17 as 
well.  We can provide a way to override this for people that want to use 
the tool for non-standard library versions, but that is not the goal, as 
I see it.
> many webapps in the past needed endorsed libs or configuration of bootstrap
>
> or jvm keystore
> most webapps need a database and come with hibernate, so which
>
> backing db engine
> to configure to be used with hibernate?
> how to make the user aware that the user/pass used for database creation
>
> have to be changed? and you still have to get somewhere the grant for
> creating such user
> what about specific settings as -Xms or -XX:PermGen for the jvm which could
>
> be in conflict among webapps
>
> Imho, a fully automated deploy manager is just too complex to implement,
> and nevertheless, that would probably not  let you avoid small manual
> configurations.

I don't see it being fully automated.  More of a guide-you-through-it 
with-reasonable-defaults  tool.  A user should be able to run in in 
unattend mode by providing certain command line switches...or, 
conversely,  if we wnat the default to be  run it unattended, provide a 
switch --interactive and prompt them.  If run in unattended mode, and 
one of the values is invalid, the whole thing should fail as a unit.

We should also track the files we modify, provide copies of the 
originals, and clean up on uninstall of the webapp.
> What could be really a gift, is an automated tool which does a maximum common
> set of configurations, or even better, some sort of wizard that in a few steps
> could let you have running some webapp, that you just had sitting
> somewhere after
> installing the relevant packages.

Note that whey I suggested using maven, I did not mean that we should 
use the same pom as is used to build the application.  What would make 
more sense is to provide a fragment of a pom used just for deployment.  
I'd see it something like this:

in /etc/jwebapp/myproject.pom  (hey. why not!  seriously, though, 
where?  Should not be /etc/tomcat6...yes, I did this in JSON, to make it 
easier to read )

{
   {common:
      {format:war},  # as opposed to exploded
      {libs:[commons-codec,commons-logging]},
      {realm: simple},
   {tomcat6: ... }
   {jbossas5: ... }
}

and then

/usr/bin/jwebapp  myproject  --deploy  --server  tomcat6

Or something.  Probably should keep  these .poms in 
/usr/share/java/webapps,  but again allow users to customize them.

[Adam Young]

I wrote this up, in an attempt to solicit comments from a wider audience.

http://adam.younglogic.com/2011/10/java-web-applications-fedora/