[Fedora-directory-users] Recovering from database corruption?

[Josh Kelley]

I'm afraid that I may have messed up our FDS installation and would
greatly appreciate advice on how to fix things.

We have two Fedora Directory Servers, urim and thummim, set up to
replicate changes to each other.  Following a combination of hardware
failure and administrator error (i.e., I thought that the server was
hung and killed it, possibly while it was in the middle of recovery),
the database got corrupted on urim, and it refused to start, giving
the following errors in its log file:

[27/Mar/2007:17:10:18 -0400] - libdb: Ignoring log file:
/opt/fedora-ds/slapd-urim/db/log.0000000164: magic number 0, not 40988
[27/Mar/2007:17:10:20 -0400] - libdb: Invalid log file:
log.0000000164: Invalid argument
[27/Mar/2007:17:10:20 -0400] - libdb: PANIC: Invalid argument
[27/Mar/2007:17:10:20 -0400] - libdb: PANIC: DB_RUNRECOVERY: Fatal
error, run database recovery
[27/Mar/2007:17:10:20 -0400] - Opening database environment
(/opt/fedora-ds/slapd-urim/db) failed. err=-30978: DB_RUNRECOVERY:
Fatal error, run database recovery
[27/Mar/2007:17:10:20 -0400] - start: Failed to init database,
err=-30978 DB_RUNRECOVERY: Fatal error, run database recovery

So then I moved the invalid log file out of the way and successfully
started FDS.  Since urim was now out of date and had some database
inconsistencies, I opened the administrative console on thummim,
selected the replication agreement to urim, and told it to
(re)initialize the consumer.

Everything appears to be correct now; however, in the error logs on
urim, I got the following warning/error:
[27/Mar/2007:17:23:43 -0400] NSMMReplicationPlugin -
replica_reload_ruv: Warning: new data for replica dc=local does not
match the data in the changelog.
 Recreating the changelog file. This could affect replication with
replica's  consumers in which case the consumers should be
reinitialized.

Does this mean that I now need to reinitialize thummim as well?  Or is
this warning the result of urim's changelog forcibly being sync'ed
with thummim, and everything's okay now?

Thank you.

Josh Kelley

[Rich Megginson]

Josh Kelley wrote:
>  I'm afraid that I may have messed up our FDS installation and
> would
>  greatly appreciate advice on how to fix things.
>
>  We have two Fedora Directory Servers, urim and thummim, set up to
>  replicate changes to each other.  Following a combination of hardware
>  failure and administrator error (i.e., I thought that the server was
>  hung and killed it, possibly while it was in the middle of recovery),
>  the database got corrupted on urim, and it refused to start, giving
>  the following errors in its log file:
>
>  [27/Mar/2007:17:10:18 -0400] - libdb: Ignoring log file:
>  /opt/fedora-ds/slapd-urim/db/log.0000000164: magic number 0, not 40988
>  [27/Mar/2007:17:10:20 -0400] - libdb: Invalid log file:
>  log.0000000164: Invalid argument
>  [27/Mar/2007:17:10:20 -0400] - libdb: PANIC: Invalid argument
>  [27/Mar/2007:17:10:20 -0400] - libdb: PANIC: DB_RUNRECOVERY: Fatal
>  error, run database recovery
>  [27/Mar/2007:17:10:20 -0400] - Opening database environment
>  (/opt/fedora-ds/slapd-urim/db) failed. err=-30978: DB_RUNRECOVERY:
>  Fatal error, run database recovery
>  [27/Mar/2007:17:10:20 -0400] - start: Failed to init database,
>  err=-30978 DB_RUNRECOVERY: Fatal error, run database recovery
>
>  So then I moved the invalid log file out of the way and successfully
>  started FDS.  Since urim was now out of date and had some database
>  inconsistencies, I opened the administrative console on thummim,
>  selected the replication agreement to urim, and told it to
>  (re)initialize the consumer.
>
>  Everything appears to be correct now; however, in the error logs on
>  urim, I got the following warning/error:
>  [27/Mar/2007:17:23:43 -0400] NSMMReplicationPlugin -
>  replica_reload_ruv: Warning: new data for replica dc=local does not
>  match the data in the changelog.
>  Recreating the changelog file. This could affect replication with
>  replica's  consumers in which case the consumers should be
>  reinitialized.
>
>  Does this mean that I now need to reinitialize thummim as well?  Or is
>  this warning the result of urim's changelog forcibly being sync'ed
>  with thummim, and everything's okay now? 
The latter, I think.  I think you
should be ok now.
>  Thank you.
>
>  Josh Kelley
>
>  -- 
>  Fedora-directory-users mailing list
>  Fedora-directory-users(a)redhat.com
>  https://www.redhat.com/mailman/listinfo/fedora-directory-users