New file layout for nativified rpms

[Gary Benson]

Hi all,

The way bc-compiled rpms are made is currently time-consuming and
fiddly, and all too easy to make mistakes.  I'm writing a script to
automate this, but there's a number of changes to the filesystem
layout I'd like to introduce at the same time, partly out of necessity
and partly for future-proofing.

Currently, bc-compiled rpms have solibs in /usr/lib, one per jarfile,
and class mappings in /usr/lib/gcj-4.0.0/classmap.db.d, one per binary
rpm.  For instance, ant has these two jarfiles:

  /usr/share/java/ant-1.6.2.jar
  /usr/share/java/ant-launcher-1.6.2.jar

so it has these native files:
  
  /usr/lib/gcj-4.0.0/classmap.db.d/ant-1.6.2.db
  /usr/lib/libant-1.6.2.jar.so
  /usr/lib/libant-launcher-1.6.2.jar.so

Fully-automatic compilation will require one mapping database per
jarfile, and I'd like to stick all the solibs and databases in
package-specific databases under /usr/lib/gcj, so to continue the
example ant would have:

  /usr/lib/gcj/ant/libant-1.6.2.jar.db
  /usr/lib/gcj/ant/libant-1.6.2.jar.so
  /usr/lib/gcj/ant/libant-launcher-1.6.2.jar.db
  /usr/lib/gcj/ant/libant-launcher-1.6.2.jar.so

My reason for moving from /usr/lib/gcj-4.0.0/classmap.db.d as the
mappings directory is that this name will of course change when gcj
is upgraded -- I don't wish to rebuild my ninety-plus rpms every time
gcj bumps unless I actually have to.  We can make rebuild-gcj-db pick
up files from both the old and new directories until all rpms have
been rebuilt.

My reason for moving the solibs into this directory is simply because
then nativified rpms need only one line in their %files section to
pick up the native binaries.

Can anyone see any problems with this layout?

Cheers,
Gary

[Matthias Klose]

Gary Benson writes:
>  My reason for moving from /usr/lib/gcj-4.0.0/classmap.db.d as the
>  mappings directory is that this name will of course change when gcj
>  is upgraded -- I don't wish to rebuild my ninety-plus rpms every time
>  gcj bumps unless I actually have to.  We can make rebuild-gcj-db pick
>  up files from both the old and new directories until all rpms have
>  been rebuilt. 
Are you able to use the bc-compiled files, built using gcj-4.0.x, with
gcj-4.1.x?

    Matthias

[Andrew Haley]

Matthias Klose writes:
>  Gary Benson writes:
>  > My reason for moving from /usr/lib/gcj-4.0.0/classmap.db.d as the
>  > mappings directory is that this name will of course change when gcj
>  > is upgraded -- I don't wish to rebuild my ninety-plus rpms every time
>  > gcj bumps unless I actually have to.  We can make rebuild-gcj-db pick
>  > up files from both the old and new directories until all rpms have
>  > been rebuilt.
>  
>  Are you able to use the bc-compiled files, built using gcj-4.0.x, with
>  gcj-4.1.x? 
Probably, but ATM I don't think we can guarantee for sure.

Andrew.

[Gary Benson]

Andrew Haley wrote:
>  Matthias Klose writes:
>  > Gary Benson writes:
>  > > My reason for moving from /usr/lib/gcj-4.0.0/classmap.db.d as
>  > > the mappings directory is that this name will of course change
>  > > when gcj is upgraded -- I don't wish to rebuild my ninety-plus
>  > > rpms every time gcj bumps unless I actually have to.  We can
>  > > make rebuild-gcj-db pick up files from both the old and new
>  > > directories until all rpms have been rebuilt.
>  > 
>  > Are you able to use the bc-compiled files, built using gcj-4.0.x,
>  > with gcj-4.1.x?
>  
>  Probably, but ATM I don't think we can guarantee for sure. 
I agree. But in the same way we probably can't guarantee compatibility
between gcj-4.0.0-x and gcj-4.0.0-y.

Cheers,
Gary

[Tom Tromey]

> >>>> "Gary" == Gary Benson
> <gbenson(a)redhat.com&gt; writes: 
> > Probably, but ATM I don't think we can guarantee for sure.
Gary> I agree. But in the same way we probably can't guarantee compatibility
Gary> between gcj-4.0.0-x and gcj-4.0.0-y.

We can always make mistakes... but we do try to guarantee that we
don't break compatibility in this case, even in the 4.0.x series.

So far, as far as I know, 4.1 can use any BC-compiled object generated
by 4.0.  However, we don't want to really promise that we won't break
this in the future, as we might find some reason that we can't keep
this working.

Our hope is that starting with 4.1 we can guarantee compatibility.

Tom

[Gary Benson]

Tom Tromey wrote:
>  >>>>> "Gary" == Gary Benson
> <gbenson(a)redhat.com&gt; writes:
>  Gary> I agree. But in the same way we probably can't guarantee
>  Gary> compatibility between gcj-4.0.0-x and gcj-4.0.0-y.
>  
>  We can always make mistakes... but we do try to guarantee that we
>  don't break compatibility in this case, even in the 4.0.x series. 
Ah, ok.  Is the BC-ABI immune to things like adding extra
members/methods to classes then?

>  So far, as far as I know, 4.1 can use any BC-compiled object
>  generated by 4.0.  However, we don't want to really promise that we
>  won't break this in the future, as we might find some reason that we
>  can't keep this working.
>  
>  Our hope is that starting with 4.1 we can guarantee compatibility. 
If compatibility _is_ broken, will it fail gracefully?

Cheers,
Gary

[Andrew Haley]

Gary Benson writes:
>  Tom Tromey wrote:
>  > >>>>> "Gary" == Gary Benson <gbenson(a)redhat.com&gt;
> writes:
>  > Gary> I agree. But in the same way we probably can't guarantee
>  > Gary> compatibility between gcj-4.0.0-x and gcj-4.0.0-y.
>  > 
>  > We can always make mistakes... but we do try to guarantee that we
>  > don't break compatibility in this case, even in the 4.0.x series.
>  
>  Ah, ok.  Is the BC-ABI immune to things like adding extra
>  members/methods to classes then? 
That's what it's for...

Andrew.

[Gary Benson]

Andrew Haley wrote:
>  Gary Benson writes:
>  > Tom Tromey wrote:
>  > > >>>>> "Gary" == Gary Benson <gbenson(a)redhat.com&gt;
> writes:
>  > > Gary> I agree. But in the same way we probably can't guarantee
>  > > Gary> compatibility between gcj-4.0.0-x and gcj-4.0.0-y.
>  > > 
>  > > We can always make mistakes... but we do try to guarantee that
>  > > we don't break compatibility in this case, even in the 4.0.x
>  > > series.
>  > 
>  > Ah, ok.  Is the BC-ABI immune to things like adding extra
>  > members/methods to classes then?
>  
>  That's what it's for... 
Sweet :)

[Bryce McKinlay]

Gary Benson wrote:

> >Gary> I agree. But in the same way we probably can't
> guarantee
> >Gary> compatibility between gcj-4.0.0-x and gcj-4.0.0-y.
> >
> >We can always make mistakes... but we do try to guarantee that we
> >don't break compatibility in this case, even in the 4.0.x series.
> >    
> >
>
> Ah, ok.  Is the BC-ABI immune to things like adding extra
> members/methods to classes then?
>   
>  
Yes. In general the BC-ABI conforms to chapter 13 of the JLS.

> >So far, as far as I know, 4.1 can use any BC-compiled object
> >generated by 4.0.  However, we don't want to really promise that we
> >won't break this in the future, as we might find some reason that we
> >can't keep this working.
> >
> >Our hope is that starting with 4.1 we can guarantee compatibility.
> >    
> >
>
> If compatibility _is_ broken, will it fail gracefully?
>   
>  
libgcj 4.0.0 would crash if it saw a compiled class with an incompatible 
version ID, but this is fixed in 4.0.1 and 4.1. Now, we will silently 
refuse to load the class and fall back to the bytecode interpreter when 
a native-compiled class could not be loaded for some reason.

But, I'm not sure that this is really the most "graceful" behaviour - 
the user could easily be unaware that their .so did not load and wonder 
why performance is bad. Other alternatives we could consider include:

1. Issue a warning before falling back to the bytecode interpreter
2. Throw a ClassFormatError/ClassNotFoundException/VerifyError

Note that this will most likely happen when you try to run a newwer 
compiled class against an old runtime. In general, we will try to 
preserve backwards-compatibility so that old compiled classes will 
continue to work with new runtimes - although, as Tom points out, we 
don't yet guarantee it.

Bryce

[Gary Benson]

Bryce McKinlay wrote:
>  Gary Benson wrote:
>  > If compatibility _is_ broken, will it fail gracefully?
>  
>  libgcj 4.0.0 would crash if it saw a compiled class with an
>  incompatible version ID, but this is fixed in 4.0.1 and 4.1. Now, we
>  will silently refuse to load the class and fall back to the bytecode
>  interpreter when a native-compiled class could not be loaded for
>  some reason.
>  
>  But, I'm not sure that this is really the most "graceful" behaviour
>  - the user could easily be unaware that their .so did not load and
>  wonder why performance is bad. Other alternatives we could consider
>  include:
>  
>  1. Issue a warning before falling back to the bytecode interpreter
>  2. Throw a ClassFormatError/ClassNotFoundException/VerifyError 
My vote would go for #1.

>  Note that this will most likely happen when you try to run a newwer
>  compiled class against an old runtime. In general, we will try to
>  preserve backwards-compatibility so that old compiled classes will
>  continue to work with new runtimes - although, as Tom points out, we
>  don't yet guarantee it. 
Cool.

[Andrew Haley]

Gary Benson writes:
>  Andrew Haley wrote:
>  > Matthias Klose writes:
>  > > Gary Benson writes:
>  > > > My reason for moving from /usr/lib/gcj-4.0.0/classmap.db.d as
>  > > > the mappings directory is that this name will of course change
>  > > > when gcj is upgraded -- I don't wish to rebuild my ninety-plus
>  > > > rpms every time gcj bumps unless I actually have to.  We can
>  > > > make rebuild-gcj-db pick up files from both the old and new
>  > > > directories until all rpms have been rebuilt.
>  > > 
>  > > Are you able to use the bc-compiled files, built using gcj-4.0.x,
>  > > with gcj-4.1.x?
>  > 
>  > Probably, but ATM I don't think we can guarantee for sure.
>  
>  I agree. But in the same way we probably can't guarantee compatibility
>  between gcj-4.0.0-x and gcj-4.0.0-y. 
I think that's a very different case.  We're trying very hard to do
just that.

Andrew.