gcj-dbtool --rebuild

[Thomas Fitzsimmons]

Hi,

Here's what "gcj-dbtool --rebuild" needs to do:

Merge all db files in "$prefix/lib/gcj-$gcc_version/classmap.db.d/" into
"$prefix/lib/gcj-$gcc_version/classmap.db".

Even better for FC5 would be eliminating the need for individual .db
files altogether.  We could have "gcj-dbtool --rebuild" compare the .sos
in /usr/lib with the database contents and add/remove any new/missing
entries to/from classmap.db.  Shipping a .db file per package is one
divergence from JPackage spec files that Fernando's proposal doesn't
address but that an improved gcj-dbtool would.

Tom

[Gary Benson]

Thomas Fitzsimmons wrote:
>  Here's what "gcj-dbtool --rebuild" needs to do:
>  
>  Merge all db files in "$prefix/lib/gcj-$gcc_version/classmap.db.d/" into
>  "$prefix/lib/gcj-$gcc_version/classmap.db".
>
>  Even better for FC5 would be eliminating the need for individual .db
>  files altogether. 
This would require the hashes to be stored in the .so, which I don't
think they are at present.  Is this something which is possible?

Also, the per-package databases are in $prefix/lib/gcj/$name these
days.

Cheers,
Gary

[Andrew Haley]

Gary Benson writes:
>  Thomas Fitzsimmons wrote:
>  > Here's what "gcj-dbtool --rebuild" needs to do:
>  > 
>  > Merge all db files in "$prefix/lib/gcj-$gcc_version/classmap.db.d/" into
>  > "$prefix/lib/gcj-$gcc_version/classmap.db".
>  >
>  > Even better for FC5 would be eliminating the need for individual .db
>  > files altogether.
>  
>  This would require the hashes to be stored in the .so, which I don't
>  think they are at present.  Is this something which is possible? 
Yes.  It is part of the Grand Plan.  But reading the hashes out of a
.so is going to be far slower than out of a .db.  It wouldn't be
practical if you had thousands of them.

Andrew.

[David Walluck]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Gary Benson wrote:
>  Also, the per-package databases are in $prefix/lib/gcj/$name these
>  days. 
Someone mentioned that files in /usr/lib are not necessarily writable,
so that could break both the rebuild-gcj-db and rebuild-security
providers. But these scripts are only (normally) called  by rpm when it
installs or uninstalls packages and /usr/lib must be writable at this point.

I don't know how I feel about the Makefile idea. In any case, it should
never require user intervention. What besides sendmail is using this
method anyhow? IMO, `make' should be touched by packagers only,
certainly not by end users, and not even by system admins. Their work is
supposed to be done by rpm out of the box must of the time. One of the
good points about the existing scripts is that they are (a) completely
managed by rpm (b) transparent to the user *unless they fail of course).

- --
Sincerely,

David Walluck
<david(a)zarb.org&gt;
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.2 (GNU/Linux)
>
> iD8DBQFDdQ+marJDwJ6gwowRAsc0AJ9xjjoy2jp3scuswhbFRvQKps4/KQCff3gT
> rxDvijQ5ue2V5g2VlMTRMKY=
> =w3hT
> -----END PGP SIGNATURE-----

[Andrew Haley]

David Walluck writes:
>  -----BEGIN PGP SIGNED MESSAGE-----
>  Hash: SHA1
>  
>  Gary Benson wrote:
>  > Also, the per-package databases are in $prefix/lib/gcj/$name these
>  > days.
>  
>  Someone mentioned that files in /usr/lib are not necessarily writable,
>  so that could break both the rebuild-gcj-db and rebuild-security
>  providers. But these scripts are only (normally) called  by rpm when it
>  installs or uninstalls packages and /usr/lib must be writable at this point. 
Ah, but /usr might be *shared* between several computers, and each one
needs to have its own master database.  So that master database can't
be in /usr.

>  I don't know how I feel about the Makefile idea. In any case, it
>  should never require user intervention.  What besides sendmail is
>  using this method anyhow? IMO, `make' should be touched by
>  packagers only, certainly not by end users, and not even by system
>  admins. 
They won't need to.  The question here is how we fix our existing
scripts so that they don't do unnecessary work.  If we do that, then
they can be called unconditionally.  make is the right answer here,
because it's a tool that already knows how to do the dependency
analysis that's needed.

>  Their work is supposed to be done by rpm out of the box must of the
>  time. One of the good points about the existing scripts is that
>  they are (a) completely managed by rpm (b) transparent to the user
>  *unless they fail of course). 
And no-one wants to change that.

Andrew.

[Gary Benson]

Andrew Haley wrote:
>  David Walluck writes:
>  > I don't know how I feel about the Makefile idea. In any case, it
>  > should never require user intervention.  What besides sendmail is
>  > using this method anyhow? IMO, `make' should be touched by
>  > packagers only, certainly not by end users, and not even by system
>  > admins.
>  
>  They won't need to.  The question here is how we fix our existing
>  scripts so that they don't do unnecessary work.  If we do that, then
>  they can be called unconditionally.  make is the right answer here,
>  because it's a tool that already knows how to do the dependency
>  analysis that's needed. 
All it has to do is read every .db file under /usr/lib/gcj and write
what it finds to a .db file in /var/lib/gcj.  Where's the dependency
analysis in that?

Cheers,
Gary

[Andrew Haley]

Gary Benson writes:
>  Andrew Haley wrote:
>  > David Walluck writes:
>  > > I don't know how I feel about the Makefile idea. In any case, it
>  > > should never require user intervention.  What besides sendmail is
>  > > using this method anyhow? IMO, `make' should be touched by
>  > > packagers only, certainly not by end users, and not even by system
>  > > admins.
>  > 
>  > They won't need to.  The question here is how we fix our existing
>  > scripts so that they don't do unnecessary work.  If we do that, then
>  > they can be called unconditionally.  make is the right answer here,
>  > because it's a tool that already knows how to do the dependency
>  > analysis that's needed.
>  
>  All it has to do is read every .db file under /usr/lib/gcj and write
>  what it finds to a .db file in /var/lib/gcj.  Where's the dependency
>  analysis in that? 
We can already do that.  

My suggestion was a way to only do that when we know we really have
to.  So, if you install a RPM that doesn't add or remove any .db
files, nothing gets rebuilt because make doesn't see any changed
dependencies.

This completely solves the problem of

>  That aside, having the database rebuilding as an alternative (so
>  possibly a no-op) would also cause problems. Imagine this:
>  
>    1. User has GCJ as their JVM.
>    2. a) User installs some Fedora packages.
>       b) GCJ database is rebuilt into a consistent state.
>    3. User switches to some other JVM.
>    4. a) User installs some more Fedora packages.
>       b) GCJ database is not rebuilt.
>    5. User switches to GCJ as their JVM.
>  
>  The user has ended up with a broken database.  His applications will
>  be slower, and in some (admittedly broken) cases will suddenly start
>  to fail.
>  
>  Rather than being an alternative, the rebuild command would need to
>  rebuild all databases that could be installed on the system.  This
>  could be done by for example having all JVMs drop scripts in some
>  database, and having 'aot-compile --rebuild' or whatever call them all
>  in turn. 
which was clipped from the mail.

Don't make database rebuilding an alternative.  Use make.

Andrew.

[Bryce McKinlay]

Andrew Haley wrote:

> Gary Benson writes:
>  > Thomas Fitzsimmons wrote:
>  > > Here's what "gcj-dbtool --rebuild" needs to do:
>  > > 
>  > > Merge all db files in "$prefix/lib/gcj-$gcc_version/classmap.db.d/"
> into
>  > > "$prefix/lib/gcj-$gcc_version/classmap.db".
>  > >
>  > > Even better for FC5 would be eliminating the need for individual .db
>  > > files altogether.
>  > 
>  > This would require the hashes to be stored in the .so, which I don't
>  > think they are at present.  Is this something which is possible?
>
> Yes.  It is part of the Grand Plan.  But reading the hashes out of a
> .so is going to be far slower than out of a .db.  It wouldn't be
> practical if you had thousands of them.
>   
>  
With hashes stored in the .so's, we'd still want to keep the .db as a 
sort of cache. The advantage would be that updating/rebuilding the db 
becomes much faster, as dbtool just needs to compare the hashes in each 
.so with what is in the database rather than reading all the bytecode 
and recomputing the hashes.

It would be too slow for the general case to have libgcj reading hashes 
from all the shared libraries at runtime, but it would allow things to 
work better in the case where the classes in a particular .so at runtime 
don't match the contents of the db - something that could happen during 
development or in case someone forgets to run dbtool. Currently we just 
quietly fall back to the interpreter in that case, resulting in 
(seemingly) mysterious performance problems.

Bryce

[Andrew Haley]

Bryce McKinlay writes:
>  Andrew Haley wrote:
>  
>  >Gary Benson writes:
>  > > Thomas Fitzsimmons wrote:
>  > > > Here's what "gcj-dbtool --rebuild" needs to do:
>  > > > 
>  > > > Merge all db files in
> "$prefix/lib/gcj-$gcc_version/classmap.db.d/" into
>  > > > "$prefix/lib/gcj-$gcc_version/classmap.db".
>  > > >
>  > > > Even better for FC5 would be eliminating the need for individual .db
>  > > > files altogether.
>  > > 
>  > > This would require the hashes to be stored in the .so, which I don't
>  > > think they are at present.  Is this something which is possible?
>  >
>  >Yes.  It is part of the Grand Plan.  But reading the hashes out of a
>  >.so is going to be far slower than out of a .db.  It wouldn't be
>  >practical if you had thousands of them.
>
>  With hashes stored in the .so's, we'd still want to keep the .db as a 
>  sort of cache. The advantage would be that updating/rebuilding the db 
>  becomes much faster, as dbtool just needs to compare the hashes in each 
>  .so with what is in the database rather than reading all the bytecode 
>  and recomputing the hashes. 
Well, we only compute the hashes at package build time, so this won't
affect installation time.

>  It would be too slow for the general case to have libgcj reading
>  hashes from all the shared libraries at runtime, but it would allow
>  things to work better in the case where the classes in a particular
>  .so at runtime don't match the contents of the db - something that
>  could happen during development or in case someone forgets to run
>  dbtool. Currently we just quietly fall back to the interpreter in
>  that case, resulting in (seemingly) mysterious performance
>  problems. 
Right.  This is effectively "closing the loop", and it's one reason
for putting the hashes into the DSOs.  It's just one less weird thing
that can go wrong.

Andrew.