[Fedora-directory-users] db_verify

[Ville Silventoinen]

After I import about 1400 accounts to a new database (ebiRoot, People 
subtree), I get lot of errors when I run verify-db.pl (slapd has been stopped):

Verify log files in db ... Good
Verify db/ebiRoot/uid.db4 ... Good
Verify db/ebiRoot/mail.db4 ...
DB ERROR: db_verify: Page 37: out-of-order key at entry 247
DB ERROR: db_verify: Page 37: out-of-order key at entry 503
...

Same error for ancestorid.db4, objectclass.db4, parentid.db4, cn.db4, 
givenName.db4 and sn.db4.

I have run db2index and re-run verify-db.pl but I don't see any 
difference. Here is what db2index says about ebiRoot:

[29/Mar/2007:12:04:26 +0100] upgrade DB - ebiRoot: Start upgradedb.
[29/Mar/2007:12:04:26 +0100] - WARNING: Import is running with
nsslapd-db-private-import-mem on; No other process is allowed to access the database
[29/Mar/2007:12:04:26 +0100] - import ebiRoot: Index buffering enabled with bucket size
100
[29/Mar/2007:12:04:27 +0100] - import ebiRoot: Workers finished; cleaning up...
[29/Mar/2007:12:04:28 +0100] - import ebiRoot: Workers cleaned up.
[29/Mar/2007:12:04:28 +0100] - import ebiRoot: Cleaning up producer thread...
[29/Mar/2007:12:04:28 +0100] - import ebiRoot: Indexing complete.  Post-processing...
[29/Mar/2007:12:04:28 +0100] - import ebiRoot: Flushing caches...
[29/Mar/2007:12:04:28 +0100] - import ebiRoot: Closing files...
[29/Mar/2007:12:04:29 +0100] - import ebiRoot: Import complete.  Processed 
1424 entries in 3 seconds. (474.67 entries/sec)


Does that WARNING "No other process is alloed to access the database" mean 
something is wrong?

How can I locate those "out-of order keys" the db_verify lists? I tried 
with dbscan but I don't think I'm giving the right entry id:

$ ./dbscan -K 247 -f db/ebiRoot/mail.db4
Can't set cursor to returned item: DB_NOTFOUND: No matching key/data pair found

Is there a way to find out which entries are causing the problem? Can 
there be illegal characters in the entries?

If I import a considerably smaller set of entries (120), I get no errors. 
I noticed there was a similar thread here but no conclusion:

http://www.mail-archive.com/[e-mail]/msg04461.html

Sorry for so many questions, I've spent couple of days trying to 
solve the problem.

If I delete a database with the Console, it leaves behind couple of index files:

-rw-------  1 w3secure systems 16384 Mar 28 17:05 ancestorid.db4
-rw-------  1 w3secure systems    18 Mar 28 17:03 DBVERSION
-rw-------  1 w3secure systems 32768 Mar 28 17:05 id2entry.db4

These index files don't seem to shrink when new entries are imported. 
dbscan still shows the deleted entries in id2entry.

I noticed a problem when I import a small set of entries, delete the 
database, import large set of entries and if I query the entries, I get 
the entries from the first set (they don't exist in the second set). I can 
reproduce the problem. If I delete ancestorid.db4 and id2entry.db4 
manually when I delete the database, I don't have this problem. Is there a 
reason why those two files are not deleted? Or can this whole thing be 
caused by corrupted data?


Ville

[Noriko Hosoi]

Hello, Ville;

Ville Silventoinen wrote:

>  After I import about 1400 accounts to a new database (ebiRoot, People
>
>  subtree), I get lot of errors when I run verify-db.pl (slapd has been 
>  stopped):
>
>  Verify log files in db ... Good
>  Verify db/ebiRoot/uid.db4 ... Good
>  Verify db/ebiRoot/mail.db4 ...
>  DB ERROR: db_verify: Page 37: out-of-order key at entry 247
>  DB ERROR: db_verify: Page 37: out-of-order key at entry 503
>  ...
>
>  Same error for ancestorid.db4, objectclass.db4, parentid.db4, cn.db4, 
>  givenName.db4 and sn.db4. 
How about id2entry.db4?  Is it broken?  (It's a primary db file.)

>  I have run db2index and re-run verify-db.pl but I don't see any 
>  difference. Here is what db2index says about ebiRoot:
>
>  [29/Mar/2007:12:04:26 +0100] upgrade DB - ebiRoot: Start upgradedb.
>  [29/Mar/2007:12:04:26 +0100] - WARNING: Import is running with 
>  nsslapd-db-private-import-mem on; No other process is allowed to 
>  access the database
>  [29/Mar/2007:12:04:26 +0100] - import ebiRoot: Index buffering enabled 
>  with bucket size 100
>  [29/Mar/2007:12:04:27 +0100] - import ebiRoot: Workers finished; 
>  cleaning up...
>  [29/Mar/2007:12:04:28 +0100] - import ebiRoot: Workers cleaned up.
>  [29/Mar/2007:12:04:28 +0100] - import ebiRoot: Cleaning up producer 
>  thread...
>  [29/Mar/2007:12:04:28 +0100] - import ebiRoot: Indexing complete.  
>  Post-processing...
>  [29/Mar/2007:12:04:28 +0100] - import ebiRoot: Flushing caches...
>  [29/Mar/2007:12:04:28 +0100] - import ebiRoot: Closing files...
>  [29/Mar/2007:12:04:29 +0100] - import ebiRoot: Import complete.  
>  Processed 1424 entries in 3 seconds. (474.67 entries/sec)
>
>
>  Does that WARNING "No other process is alloed to access the database" 
>  mean something is wrong? 
No, that's just a warning not to access the backend ebiRoot.

>  How can I locate those "out-of order keys" the db_verify lists? I 
>  tried with dbscan but I don't think I'm giving the right entry id: 
Right.  247 is the Berkeley DB's internal id.

>  $ ./dbscan -K 247 -f db/ebiRoot/mail.db4
>  Can't set cursor to returned item: DB_NOTFOUND: No matching key/data 
>  pair found 
What happens if you just run dbscan for all the keys in mail.db4 
(without the -K option)? 
E.g., ./dbscan -n -r db/ebiRoot/mail.db4
Do you get any errors?

>  Is there a way to find out which entries are causing the problem? Can
>
>  there be illegal characters in the entries? 
Could it be possible to share your data with us?  (sample data would be 
good.)
Thanks,
--noriko

>  If I import a considerably smaller set of entries (120), I get no 
>  errors. I noticed there was a similar thread here but no conclusion:
>
>  http://www.mail-archive.com/[e-mail]/msg04461.html 
>
>
>  Sorry for so many questions, I've spent couple of days trying to solve 
>  the problem.
>
>  If I delete a database with the Console, it leaves behind couple of 
>  index files:
>
>  -rw-------  1 w3secure systems 16384 Mar 28 17:05 ancestorid.db4
>  -rw-------  1 w3secure systems    18 Mar 28 17:03 DBVERSION
>  -rw-------  1 w3secure systems 32768 Mar 28 17:05 id2entry.db4
>
>  These index files don't seem to shrink when new entries are imported. 
>  dbscan still shows the deleted entries in id2entry.
>
>  I noticed a problem when I import a small set of entries, delete the 
>  database, import large set of entries and if I query the entries, I 
>  get the entries from the first set (they don't exist in the second 
>  set). I can reproduce the problem. If I delete ancestorid.db4 and 
>  id2entry.db4 manually when I delete the database, I don't have this 
>  problem. Is there a reason why those two files are not deleted? Or can 
>  this whole thing be caused by corrupted data?
>
>
>  Ville
>
>  -- 
>  Fedora-directory-users mailing list
>  Fedora-directory-users(a)redhat.com
>  https://www.redhat.com/mailman/listinfo/fedora-directory-users

[Ville Silventoinen]

Hi Noriko, thanks for you reply.

On Thu, 29 Mar 2007, Noriko Hosoi wrote:

>  Ville Silventoinen wrote:
>
> > Same error for ancestorid.db4, objectclass.db4, parentid.db4, cn.db4, 
> > givenName.db4 and sn.db4.
>
>  How about id2entry.db4?  Is it broken?  (It's a primary db file.) 
No, id2entry.db4 is Good.


> > How can I locate those "out-of order keys" the
> db_verify lists? I tried 
> > with dbscan but I don't think I'm giving the right entry id:
>
>  Right.  247 is the Berkeley DB's internal id. 
OK, so there's no way to use it to locate which entry is causing the problem?


> > $ ./dbscan -K 247 -f db/ebiRoot/mail.db4
> > Can't set cursor to returned item: DB_NOTFOUND: No matching key/data pair 
> > found
>
>  What happens if you just run dbscan for all the keys in mail.db4 (without the 
>  -K option)? E.g., ./dbscan -n -r db/ebiRoot/mail.db4
>  Do you get any errors? 
I assume you meant "dbscan -n -r -f db/ebiRoot/mail.db4". I see lot of 
numbers, no errors as far as I can tell.


> > Is there a way to find out which entries are causing the problem?
> Can there 
> > be illegal characters in the entries?
>
>  Could it be possible to share your data with us?  (sample data would be 
>  good.) 
I asked my manager but he doesn't think it's a good idea for security 
reasons. The problem is that the data is our NIS mail.aliases and passwd, 
and we don't want to distribute them to the internet. He suggested I'll 
modify the data, so I can send a sample to you. I'll do that next week.

What would you be looking for in the data? Perhaps I could do it here?

Today I have been trying to narrow down which entries in our 
mail.aliases cause the problem. We have about 8400 aliases (includes 
majordomo and mailman aliases) and if I sort them alphabetically and 
import only entries 250-500, I get following errors from verify-db.pl:

Verify log files in db ... Good
Verify db/ebiRoot/ancestorid.db4 ...
DB ERROR: db_verify: Page 2: out-of-order key at entry 254
DB ERROR: db_verify: DB->verify: db/ebiRoot/ancestorid.db4: DB_VERIFY_BAD: 
Database verification failed
Secondary index file ancestorid.db4 in db/ebiRoot is corrupted.
Please run db2index(.pl) for reindexing.
Verify db/ebiRoot/objectclass.db4 ...
DB ERROR: db_verify: Page 2: out-of-order key at entry 255
DB ERROR: db_verify: DB->verify: db/ebiRoot/objectclass.db4: DB_VERIFY_BAD: Database
verification failed
Secondary index file objectclass.db4 in db/ebiRoot is corrupted.
Please run db2index(.pl) for reindexing.
Verify db/ebiRoot/nsuniqueid.db4 ... Good
Verify db/ebiRoot/parentid.db4 ...
DB ERROR: db_verify: Page 1: unsorted duplicate set in sorted-dup database
DB ERROR: db_verify: DB->verify: db/ebiRoot/parentid.db4: DB_VERIFY_BAD: Database
verification failed
Secondary index file parentid.db4 in db/ebiRoot is corrupted.
Please run db2index(.pl) for reindexing.
Verify db/ebiRoot/cn.db4 ...
DB ERROR: db_verify: Page 2: unsorted duplicate set in sorted-dup database
DB ERROR: db_verify: Page 6: unsorted duplicate set in sorted-dup database
DB ERROR: db_verify: Page 8: unsorted duplicate set in sorted-dup database
DB ERROR: db_verify: Page 11: unsorted duplicate set in sorted-dup database
DB ERROR: db_verify: Page 3: unsorted duplicate set in sorted-dup database
DB ERROR: db_verify: Page 12: unsorted duplicate set in sorted-dup database
DB ERROR: db_verify: Page 4: unsorted duplicate set in sorted-dup database
DB ERROR: db_verify: Page 10: unsorted duplicate set in sorted-dup database
DB ERROR: db_verify: Page 5: unsorted duplicate set in sorted-dup database
DB ERROR: db_verify: Page 7: unsorted duplicate set in sorted-dup database
DB ERROR: db_verify: DB->verify: db/ebiRoot/cn.db4: DB_VERIFY_BAD: Database
verification failed
Secondary index file cn.db4 in db/ebiRoot is corrupted.
Please run db2index(.pl) for reindexing.


So there's a new error "unsorted duplicate set in sorted-dup database". 
What does it mean?

Curious thing is that I cannot narrow down the problematic entries any 
further. I've tried importing entries 300-450 and 275-475, but there's no 
error. Also tried with two different sets from various ranges, but no 
success yet. It only seems to happen when I import at least 250 entries.

This is how a single entry looks like:

dn: cn=foobar-dev-local,ou=Aliases,dc=ebi,dc=ac,dc=uk
cn: foobar-dev-local
objectClass: top
objectClass: nisMailAlias
rfc822MailMember: "|/homes/majordom/wrapper stripmime.pl|/homes/majordom/wra
  pper resend -l foobar-dev  foobar-dev-outgoing"

I also tried skipping all entries with double quotes, but I still got 
errors. There can be several rfc822MailMember attribute values, as you 
probably know.

Thanks for your time, I really appreciate it.


Ville

[Noriko Hosoi]

Ville Silventoinen wrote:
>  Hi Noriko, thanks for you reply.
>
>  On Thu, 29 Mar 2007, Noriko Hosoi wrote:
>  [...]
> >> Is there a way to find out which entries are causing the problem? 
> >> Can there be illegal characters in the entries?
> >
> > Could it be possible to share your data with us? (sample data would 
> > be good.)
>
>  I asked my manager but he doesn't think it's a good idea for security 
>  reasons. The problem is that the data is our NIS mail.aliases and 
>  passwd, and we don't want to distribute them to the internet. He 
>  suggested I'll modify the data, so I can send a sample to you. I'll do 
>  that next week. 
That would be great. Thanks! I'm interested in what type of
characters 
your data contain. E.g., character set is UTF-8? Some of your DNs could 
contain any special characters such as '\'? etc...
>  What would you be looking for in the data? Perhaps I could do it
> here?
>
>  Today I have been trying to narrow down which entries in our 
>  mail.aliases cause the problem. We have about 8400 aliases (includes 
>  majordomo and mailman aliases) and if I sort them alphabetically and 
>  import only entries 250-500, I get following errors from verify-db.pl:
>
>  Verify log files in db ... Good
>  Verify db/ebiRoot/ancestorid.db4 ...
>  DB ERROR: db_verify: Page 2: out-of-order key at entry 254
>  DB ERROR: db_verify: DB->verify: db/ebiRoot/ancestorid.db4: 
>  DB_VERIFY_BAD: Database verification failed
>  Secondary index file ancestorid.db4 in db/ebiRoot is corrupted.
>  Please run db2index(.pl) for reindexing.
>  Verify db/ebiRoot/objectclass.db4 ...
>  DB ERROR: db_verify: Page 2: out-of-order key at entry 255
>  DB ERROR: db_verify: DB->verify: db/ebiRoot/objectclass.db4: 
>  DB_VERIFY_BAD: Database verification failed
>  Secondary index file objectclass.db4 in db/ebiRoot is corrupted.
>  Please run db2index(.pl) for reindexing.
>  Verify db/ebiRoot/nsuniqueid.db4 ... Good
>  Verify db/ebiRoot/parentid.db4 ...
>  DB ERROR: db_verify: Page 1: unsorted duplicate set in sorted-dup 
>  database
>  DB ERROR: db_verify: DB->verify: db/ebiRoot/parentid.db4: 
>  DB_VERIFY_BAD: Database verification failed
>  Secondary index file parentid.db4 in db/ebiRoot is corrupted.
>  Please run db2index(.pl) for reindexing.
>  Verify db/ebiRoot/cn.db4 ...
>  DB ERROR: db_verify: Page 2: unsorted duplicate set in sorted-dup 
>  database
>  DB ERROR: db_verify: Page 6: unsorted duplicate set in sorted-dup 
>  database
>  DB ERROR: db_verify: Page 8: unsorted duplicate set in sorted-dup 
>  database
>  DB ERROR: db_verify: Page 11: unsorted duplicate set in sorted-dup 
>  database
>  DB ERROR: db_verify: Page 3: unsorted duplicate set in sorted-dup 
>  database
>  DB ERROR: db_verify: Page 12: unsorted duplicate set in sorted-dup 
>  database
>  DB ERROR: db_verify: Page 4: unsorted duplicate set in sorted-dup 
>  database
>  DB ERROR: db_verify: Page 10: unsorted duplicate set in sorted-dup 
>  database
>  DB ERROR: db_verify: Page 5: unsorted duplicate set in sorted-dup 
>  database
>  DB ERROR: db_verify: Page 7: unsorted duplicate set in sorted-dup 
>  database
>  DB ERROR: db_verify: DB->verify: db/ebiRoot/cn.db4: DB_VERIFY_BAD: 
>  Database verification failed
>  Secondary index file cn.db4 in db/ebiRoot is corrupted.
>  Please run db2index(.pl) for reindexing.
>
>
>  So there's a new error "unsorted duplicate set in sorted-dup 
>  database". What does it mean? 
It does not say much... Our secondary indexes
are sorted-dup database. 
OIDs in the entries sharing the same key are supposed to be sorted in 
each key-value. It looks the internal sorting is broken.

I have a couple of questions...

So, in your ldif data, the mail attribute also has this type of value: 
"|/homes/majordom/wrapper stripmime.pl|/homes/majordom/wrapper resend -l 
foobar-dev foobar-dev-outgoing"? And your mail index has the default 
indexing type: presence, equality, and substring?

What type of indexing does the rfc822MailMember attribute have?

Have we already heard what platform you are running the FDS on?
Thanks,
--noriko

>  Curious thing is that I cannot narrow down the problematic entries
> any 
>  further. I've tried importing entries 300-450 and 275-475, but there's 
>  no error. Also tried with two different sets from various ranges, but 
>  no success yet. It only seems to happen when I import at least 250 
>  entries.
>
>  This is how a single entry looks like:
>
>  dn: cn=foobar-dev-local,ou=Aliases,dc=ebi,dc=ac,dc=uk
>  cn: foobar-dev-local
>  objectClass: top
>  objectClass: nisMailAlias
>  rfc822MailMember: "|/homes/majordom/wrapper 
>  stripmime.pl|/homes/majordom/wra
>  pper resend -l foobar-dev foobar-dev-outgoing"
>
>  I also tried skipping all entries with double quotes, but I still got 
>  errors. There can be several rfc822MailMember attribute values, as you 
>  probably know.
>
>  Thanks for your time, I really appreciate it.
>
>
>  Ville
>
>  -- 
>  Fedora-directory-users mailing list
>  Fedora-directory-users(a)redhat.com
>  https://www.redhat.com/mailman/listinfo/fedora-directory-users

[Ville Silventoinen]

Hi Noriko,

sorry it took so long to reply, I've been busy with other work.

On Fri, 30 Mar 2007, Noriko Hosoi wrote:

>  Ville Silventoinen wrote:
> > I asked my manager but he doesn't think it's a good idea for security 
> > reasons. The problem is that the data is our NIS mail.aliases and passwd, 
> > and we don't want to distribute them to the internet. He suggested I'll 
> > modify the data, so I can send a sample to you. I'll do that next week.
>  That would be great. Thanks! I'm interested in what type of characters your 
>  data contain. E.g., character set is UTF-8? Some of your DNs could contain 
>  any special characters such as '\'? etc... 
The character set should be plain ASCII. I created an imaginary 
mail.aliases file. You can download it from here:

http://www.ebi.ac.uk/systems-srv/mp/file-exchange/

Type in "fedorads" to the Pass Phrase input box and click Go. You should 
see three files: mail.aliases, mail.aliases.ldif and 99user.ldif.

I can reproduce my problem with the above files, for example, I've tested 
like this:

1. Delete existing ebiRoot database (you could use userRoot).
2. Delete db/ebiRoot directory.
3. Create ebiRoot database.
4. Shutdown slapd.
5. Run db2index and verify-db.pl. No errors.
6. Start slapd.
7. Import mail aliases. I've tried with the Console and my own CLI, which 
can import LDIF and add entries one-by-one. The method doesn't seem to matter.
8. Shutdown slapd.
9. Run db2index and verify-db.pl, verify gives errors:

Verify log files in db ... Good
Verify db/ebiRoot/ancestorid.db4 ...
DB ERROR: db_verify: Page 2: out-of-order key at entry 254
DB ERROR: db_verify: DB->verify: db/ebiRoot/ancestorid.db4: DB_VERIFY_BAD: Database
verification failed
Secondary index file ancestorid.db4 in db/ebiRoot is corrupted.
Please run db2index(.pl) for reindexing.
Verify db/ebiRoot/objectclass.db4 ...
DB ERROR: db_verify: Page 2: out-of-order key at entry 255
DB ERROR: db_verify: DB->verify: db/ebiRoot/objectclass.db4: DB_VERIFY_BAD: Database
verification failed
Secondary index file objectclass.db4 in db/ebiRoot is corrupted.
Please run db2index(.pl) for reindexing.
Verify db/ebiRoot/nsuniqueid.db4 ... Good
Verify db/ebiRoot/parentid.db4 ...
DB ERROR: db_verify: Page 1: unsorted duplicate set in sorted-dup database
DB ERROR: db_verify: DB->verify: db/ebiRoot/parentid.db4: DB_VERIFY_BAD: Database
verification failed
Secondary index file parentid.db4 in db/ebiRoot is corrupted.
Please run db2index(.pl) for reindexing.
Verify db/ebiRoot/cn.db4 ...
DB ERROR: db_verify: Page 10: out-of-order key at entry 249
DB ERROR: db_verify: DB->verify: db/ebiRoot/cn.db4: DB_VERIFY_BAD: Database
verification failed
Secondary index file cn.db4 in db/ebiRoot is corrupted.
Please run db2index(.pl) for reindexing.
Verify db/ebiRoot/id2entry.db4 ... Good
Verify db/ebiRoot/entrydn.db4 ... Good
Verify db/ebiRoot/rfc822mailmember.db4 ...
DB ERROR: db_verify: Page 2: unsorted duplicate set in sorted-dup database
DB ERROR: db_verify: Page 3: unsorted duplicate set in sorted-dup database
DB ERROR: db_verify: DB->verify: db/ebiRoot/rfc822mailmember.db4: DB_VERIFY_BAD:
Database verification failed
Secondary index file rfc822mailmember.db4 in db/ebiRoot is corrupted.
Please run db2index(.pl) for reindexing.

>  So, in your ldif data, the mail attribute also has this type of
> value: 
>  "|/homes/majordom/wrapper stripmime.pl|/homes/majordom/wrapper resend -l 
>  foobar-dev foobar-dev-outgoing"? 
No, the People entries have a simpler mail value, like "foo(a)ebi.ac.uk".

>  And your mail index has the default indexing type: presence,
> equality, and substring? 
Yes.

>  What type of indexing does the rfc822MailMember attribute have?
I've tried without any indexing, with presence and equality and with 
presence, equality and substring. The above errors are from verify-db.pl 
when I have presence and equality indeces. If I have presence, equality 
and substring, I get these errors for rfc822MailMember:

Verify db/ebiRoot/rfc822mailmember.db4 ...
DB ERROR: db_verify: Page 13: unsorted duplicate set in sorted-dup database
DB ERROR: db_verify: Page 6: unsorted duplicate set in sorted-dup database
DB ERROR: db_verify: Page 8: unsorted duplicate set in sorted-dup database
DB ERROR: db_verify: Page 12: unsorted duplicate set in sorted-dup database
DB ERROR: db_verify: Page 3: unsorted duplicate set in sorted-dup database
DB ERROR: db_verify: Page 7: unsorted duplicate set in sorted-dup database
DB ERROR: db_verify: Page 10: unsorted duplicate set in sorted-dup database
DB ERROR: db_verify: Page 15: unsorted duplicate set in sorted-dup database
DB ERROR: db_verify: Page 4: unsorted duplicate set in sorted-dup database
DB ERROR: db_verify: Page 14: unsorted duplicate set in sorted-dup database
DB ERROR: db_verify: Page 5: unsorted duplicate set in sorted-dup database
DB ERROR: db_verify: Page 9: unsorted duplicate set in sorted-dup database
DB ERROR: db_verify: Page 11: unsorted duplicate set in sorted-dup database
DB ERROR: db_verify: DB->verify: db/ebiRoot/rfc822mailmember.db4: DB_VERIFY_BAD:
Database verification failed
Secondary index file rfc822mailmember.db4 in db/ebiRoot is corrupted.
Please run db2index(.pl) for reindexing.

>  Have we already heard what platform you are running the FDS on?
CentOS release 4.4, Linux 2.6.9-42.ELsmp. Pentium III 2x1266MHz CPUs, 2GB 
memory, SCSI disks. I'm using FDS 1.0.4.

I'm away this week Wed-Fri, so I'll get back to you next week.

Thanks for the help!

Ville

[Noriko Hosoi]

Thank you, Ville, for the test data.  I could reproduce the db_verify 
problem.

I have good news and bad news. :)  Good news, first...  Your db is not 
corrupted.  The error report from verify-db.pl is bogus.

Bad news, next.  Please take a look at this bug.  We are going to 
provide a fixed utility some time soon.

Summary: verify-db.pl (db_verify) does not work on a little endian machine

https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=236256

Sorry about this inconvenience, and thank you for reporting the problem!
--noriko

Ville Silventoinen wrote:
>  Hi Noriko,
>
>  sorry it took so long to reply, I've been busy with other work.
>
>  On Fri, 30 Mar 2007, Noriko Hosoi wrote:
>
> > Ville Silventoinen wrote:
> >> I asked my manager but he doesn't think it's a good idea for 
> >> security reasons. The problem is that the data is our NIS 
> >> mail.aliases and passwd, and we don't want to distribute them to the 
> >> internet. He suggested I'll modify the data, so I can send a sample 
> >> to you. I'll do that next week.
> > That would be great. Thanks! I'm interested in what type of 
> > characters your data contain. E.g., character set is UTF-8? Some of 
> > your DNs could contain any special characters such as '\'? etc...
>
>  The character set should be plain ASCII. I created an imaginary 
>  mail.aliases file. You can download it from here:
>
>  http://www.ebi.ac.uk/systems-srv/mp/file-exchange/
>
>  Type in "fedorads" to the Pass Phrase input box and click Go. You 
>  should see three files: mail.aliases, mail.aliases.ldif and 99user.ldif.
>
>  I can reproduce my problem with the above files, for example, I've 
>  tested like this:
>
>  1. Delete existing ebiRoot database (you could use userRoot).
>  2. Delete db/ebiRoot directory.
>  3. Create ebiRoot database.
>  4. Shutdown slapd.
>  5. Run db2index and verify-db.pl. No errors.
>  6. Start slapd.
>  7. Import mail aliases. I've tried with the Console and my own CLI, 
>  which can import LDIF and add entries one-by-one. The method doesn't 
>  seem to matter.
>  8. Shutdown slapd.
>  9. Run db2index and verify-db.pl, verify gives errors:
>
>  Verify log files in db ... Good
>  Verify db/ebiRoot/ancestorid.db4 ...
>  DB ERROR: db_verify: Page 2: out-of-order key at entry 254
>  DB ERROR: db_verify: DB->verify: db/ebiRoot/ancestorid.db4: 
>  DB_VERIFY_BAD: Database verification failed
>  Secondary index file ancestorid.db4 in db/ebiRoot is corrupted.
>  Please run db2index(.pl) for reindexing.
>  Verify db/ebiRoot/objectclass.db4 ...
>  DB ERROR: db_verify: Page 2: out-of-order key at entry 255
>  DB ERROR: db_verify: DB->verify: db/ebiRoot/objectclass.db4: 
>  DB_VERIFY_BAD: Database verification failed
>  Secondary index file objectclass.db4 in db/ebiRoot is corrupted.
>  Please run db2index(.pl) for reindexing.
>  Verify db/ebiRoot/nsuniqueid.db4 ... Good
>  Verify db/ebiRoot/parentid.db4 ...
>  DB ERROR: db_verify: Page 1: unsorted duplicate set in sorted-dup 
>  database
>  DB ERROR: db_verify: DB->verify: db/ebiRoot/parentid.db4: 
>  DB_VERIFY_BAD: Database verification failed
>  Secondary index file parentid.db4 in db/ebiRoot is corrupted.
>  Please run db2index(.pl) for reindexing.
>  Verify db/ebiRoot/cn.db4 ...
>  DB ERROR: db_verify: Page 10: out-of-order key at entry 249
>  DB ERROR: db_verify: DB->verify: db/ebiRoot/cn.db4: DB_VERIFY_BAD: 
>  Database verification failed
>  Secondary index file cn.db4 in db/ebiRoot is corrupted.
>  Please run db2index(.pl) for reindexing.
>  Verify db/ebiRoot/id2entry.db4 ... Good
>  Verify db/ebiRoot/entrydn.db4 ... Good
>  Verify db/ebiRoot/rfc822mailmember.db4 ...
>  DB ERROR: db_verify: Page 2: unsorted duplicate set in sorted-dup 
>  database
>  DB ERROR: db_verify: Page 3: unsorted duplicate set in sorted-dup 
>  database
>  DB ERROR: db_verify: DB->verify: db/ebiRoot/rfc822mailmember.db4: 
>  DB_VERIFY_BAD: Database verification failed
>  Secondary index file rfc822mailmember.db4 in db/ebiRoot is corrupted.
>  Please run db2index(.pl) for reindexing.
>
> > So, in your ldif data, the mail attribute also has this type of 
> > value: "|/homes/majordom/wrapper stripmime.pl|/homes/majordom/wrapper 
> > resend -l foobar-dev foobar-dev-outgoing"?
>
>  No, the People entries have a simpler mail value, like "foo(a)ebi.ac.uk".
>
> > And your mail index has the default indexing type: presence, 
> > equality, and substring?
>
>  Yes.
>
> > What type of indexing does the rfc822MailMember attribute have?
>
>  I've tried without any indexing, with presence and equality and with 
>  presence, equality and substring. The above errors are from 
>  verify-db.pl when I have presence and equality indeces. If I have 
>  presence, equality and substring, I get these errors for 
>  rfc822MailMember:
>
>  Verify db/ebiRoot/rfc822mailmember.db4 ...
>  DB ERROR: db_verify: Page 13: unsorted duplicate set in sorted-dup 
>  database
>  DB ERROR: db_verify: Page 6: unsorted duplicate set in sorted-dup 
>  database
>  DB ERROR: db_verify: Page 8: unsorted duplicate set in sorted-dup 
>  database
>  DB ERROR: db_verify: Page 12: unsorted duplicate set in sorted-dup 
>  database
>  DB ERROR: db_verify: Page 3: unsorted duplicate set in sorted-dup 
>  database
>  DB ERROR: db_verify: Page 7: unsorted duplicate set in sorted-dup 
>  database
>  DB ERROR: db_verify: Page 10: unsorted duplicate set in sorted-dup 
>  database
>  DB ERROR: db_verify: Page 15: unsorted duplicate set in sorted-dup 
>  database
>  DB ERROR: db_verify: Page 4: unsorted duplicate set in sorted-dup 
>  database
>  DB ERROR: db_verify: Page 14: unsorted duplicate set in sorted-dup 
>  database
>  DB ERROR: db_verify: Page 5: unsorted duplicate set in sorted-dup 
>  database
>  DB ERROR: db_verify: Page 9: unsorted duplicate set in sorted-dup 
>  database
>  DB ERROR: db_verify: Page 11: unsorted duplicate set in sorted-dup 
>  database
>  DB ERROR: db_verify: DB->verify: db/ebiRoot/rfc822mailmember.db4: 
>  DB_VERIFY_BAD: Database verification failed
>  Secondary index file rfc822mailmember.db4 in db/ebiRoot is corrupted.
>  Please run db2index(.pl) for reindexing.
>
> > Have we already heard what platform you are running the FDS on?
>
>  CentOS release 4.4, Linux 2.6.9-42.ELsmp. Pentium III 2x1266MHz CPUs, 
>  2GB memory, SCSI disks. I'm using FDS 1.0.4.
>
>  I'm away this week Wed-Fri, so I'll get back to you next week.
>
>  Thanks for the help!
>
>  Ville
>
>  -- 
>  Fedora-directory-users mailing list
>  Fedora-directory-users(a)redhat.com
>  https://www.redhat.com/mailman/listinfo/fedora-directory-users

[Edward Capriolo]

I figured as much my FDS complains too about db verify. Even after initial
import.

On 4/12/07, Noriko Hosoi <nhosoi(a)redhat.com&gt; wrote:
>  Thank you, Ville, for the test data.  I could reproduce the db_verify
>  problem.
>
>  I have good news and bad news. :)  Good news, first...  Your db is not
>  corrupted.  The error report from verify-db.pl is bogus.
>
>  Bad news, next.  Please take a look at this bug.  We are going to
>  provide a fixed utility some time soon.
>
>  Summary: verify-db.pl (db_verify) does not work on a little endian machine
>
>  https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=236256
>
>  Sorry about this inconvenience, and thank you for reporting the problem!
>  --noriko
>
>  Ville Silventoinen wrote:
>  > Hi Noriko,
>  >
>  > sorry it took so long to reply, I've been busy with other work.
>  >
>  > On Fri, 30 Mar 2007, Noriko Hosoi wrote:
>  >
>  >> Ville Silventoinen wrote:
>  >>> I asked my manager but he doesn't think it's a good idea for
>  >>> security reasons. The problem is that the data is our NIS
>  >>> mail.aliases and passwd, and we don't want to distribute them to the
>  >>> internet. He suggested I'll modify the data, so I can send a sample
>  >>> to you. I'll do that next week.
>  >> That would be great. Thanks! I'm interested in what type of
>  >> characters your data contain. E.g., character set is UTF-8? Some of
>  >> your DNs could contain any special characters such as '\'? etc...
>  >
>  > The character set should be plain ASCII. I created an imaginary
>  > mail.aliases file. You can download it from here:
>  >
>  > http://www.ebi.ac.uk/systems-srv/mp/file-exchange/
>  >
>  > Type in "fedorads" to the Pass Phrase input box and click Go. You
>  > should see three files: mail.aliases, mail.aliases.ldif and 99user.ldif.
>  >
>  > I can reproduce my problem with the above files, for example, I've
>  > tested like this:
>  >
>  > 1. Delete existing ebiRoot database (you could use userRoot).
>  > 2. Delete db/ebiRoot directory.
>  > 3. Create ebiRoot database.
>  > 4. Shutdown slapd.
>  > 5. Run db2index and verify-db.pl. No errors.
>  > 6. Start slapd.
>  > 7. Import mail aliases. I've tried with the Console and my own CLI,
>  > which can import LDIF and add entries one-by-one. The method doesn't
>  > seem to matter.
>  > 8. Shutdown slapd.
>  > 9. Run db2index and verify-db.pl, verify gives errors:
>  >
>  > Verify log files in db ... Good
>  > Verify db/ebiRoot/ancestorid.db4 ...
>  > DB ERROR: db_verify: Page 2: out-of-order key at entry 254
>  > DB ERROR: db_verify: DB->verify: db/ebiRoot/ancestorid.db4:
>  > DB_VERIFY_BAD: Database verification failed
>  > Secondary index file ancestorid.db4 in db/ebiRoot is corrupted.
>  > Please run db2index(.pl) for reindexing.
>  > Verify db/ebiRoot/objectclass.db4 ...
>  > DB ERROR: db_verify: Page 2: out-of-order key at entry 255
>  > DB ERROR: db_verify: DB->verify: db/ebiRoot/objectclass.db4:
>  > DB_VERIFY_BAD: Database verification failed
>  > Secondary index file objectclass.db4 in db/ebiRoot is corrupted.
>  > Please run db2index(.pl) for reindexing.
>  > Verify db/ebiRoot/nsuniqueid.db4 ... Good
>  > Verify db/ebiRoot/parentid.db4 ...
>  > DB ERROR: db_verify: Page 1: unsorted duplicate set in sorted-dup
>  > database
>  > DB ERROR: db_verify: DB->verify: db/ebiRoot/parentid.db4:
>  > DB_VERIFY_BAD: Database verification failed
>  > Secondary index file parentid.db4 in db/ebiRoot is corrupted.
>  > Please run db2index(.pl) for reindexing.
>  > Verify db/ebiRoot/cn.db4 ...
>  > DB ERROR: db_verify: Page 10: out-of-order key at entry 249
>  > DB ERROR: db_verify: DB->verify: db/ebiRoot/cn.db4: DB_VERIFY_BAD:
>  > Database verification failed
>  > Secondary index file cn.db4 in db/ebiRoot is corrupted.
>  > Please run db2index(.pl) for reindexing.
>  > Verify db/ebiRoot/id2entry.db4 ... Good
>  > Verify db/ebiRoot/entrydn.db4 ... Good
>  > Verify db/ebiRoot/rfc822mailmember.db4 ...
>  > DB ERROR: db_verify: Page 2: unsorted duplicate set in sorted-dup
>  > database
>  > DB ERROR: db_verify: Page 3: unsorted duplicate set in sorted-dup
>  > database
>  > DB ERROR: db_verify: DB->verify: db/ebiRoot/rfc822mailmember.db4:
>  > DB_VERIFY_BAD: Database verification failed
>  > Secondary index file rfc822mailmember.db4 in db/ebiRoot is corrupted.
>  > Please run db2index(.pl) for reindexing.
>  >
>  >> So, in your ldif data, the mail attribute also has this type of
>  >> value: "|/homes/majordom/wrapper stripmime.pl|/homes/majordom/wrapper
>  >> resend -l foobar-dev foobar-dev-outgoing"?
>  >
>  > No, the People entries have a simpler mail value, like &quot;foo(a)ebi.ac.uk&quot;.
>  >
>  >> And your mail index has the default indexing type: presence,
>  >> equality, and substring?
>  >
>  > Yes.
>  >
>  >> What type of indexing does the rfc822MailMember attribute have?
>  >
>  > I've tried without any indexing, with presence and equality and with
>  > presence, equality and substring. The above errors are from
>  > verify-db.pl when I have presence and equality indeces. If I have
>  > presence, equality and substring, I get these errors for
>  > rfc822MailMember:
>  >
>  > Verify db/ebiRoot/rfc822mailmember.db4 ...
>  > DB ERROR: db_verify: Page 13: unsorted duplicate set in sorted-dup
>  > database
>  > DB ERROR: db_verify: Page 6: unsorted duplicate set in sorted-dup
>  > database
>  > DB ERROR: db_verify: Page 8: unsorted duplicate set in sorted-dup
>  > database
>  > DB ERROR: db_verify: Page 12: unsorted duplicate set in sorted-dup
>  > database
>  > DB ERROR: db_verify: Page 3: unsorted duplicate set in sorted-dup
>  > database
>  > DB ERROR: db_verify: Page 7: unsorted duplicate set in sorted-dup
>  > database
>  > DB ERROR: db_verify: Page 10: unsorted duplicate set in sorted-dup
>  > database
>  > DB ERROR: db_verify: Page 15: unsorted duplicate set in sorted-dup
>  > database
>  > DB ERROR: db_verify: Page 4: unsorted duplicate set in sorted-dup
>  > database
>  > DB ERROR: db_verify: Page 14: unsorted duplicate set in sorted-dup
>  > database
>  > DB ERROR: db_verify: Page 5: unsorted duplicate set in sorted-dup
>  > database
>  > DB ERROR: db_verify: Page 9: unsorted duplicate set in sorted-dup
>  > database
>  > DB ERROR: db_verify: Page 11: unsorted duplicate set in sorted-dup
>  > database
>  > DB ERROR: db_verify: DB->verify: db/ebiRoot/rfc822mailmember.db4:
>  > DB_VERIFY_BAD: Database verification failed
>  > Secondary index file rfc822mailmember.db4 in db/ebiRoot is corrupted.
>  > Please run db2index(.pl) for reindexing.
>  >
>  >> Have we already heard what platform you are running the FDS on?
>  >
>  > CentOS release 4.4, Linux 2.6.9-42.ELsmp. Pentium III 2x1266MHz CPUs,
>  > 2GB memory, SCSI disks. I'm using FDS 1.0.4.
>  >
>  > I'm away this week Wed-Fri, so I'll get back to you next week.
>  >
>  > Thanks for the help!
>  >
>  > Ville
>  >
>  > --
>  > Fedora-directory-users mailing list
>  > Fedora-directory-users(a)redhat.com
>  > https://www.redhat.com/mailman/listinfo/fedora-directory-users
>
>
>  --
>  Fedora-directory-users mailing list
>  Fedora-directory-users(a)redhat.com
>  https://www.redhat.com/mailman/listinfo/fedora-directory-users
>
>
>

[Ville Silventoinen]

Thanks Noriko, that explains a lot!

I still have that other problem to solve, related to re-creating the 
database (see the end of my first email). I will try to reproduce this 
problem and create some test data this week.

Thanks,
Ville


On Thu, 12 Apr 2007, Noriko Hosoi wrote:

>  Thank you, Ville, for the test data.  I could reproduce the db_verify
>
>  problem.
>
>  I have good news and bad news. :)  Good news, first...  Your db is not 
>  corrupted.  The error report from verify-db.pl is bogus.
>
>  Bad news, next.  Please take a look at this bug.  We are going to provide a 
>  fixed utility some time soon.
>
>  Summary: verify-db.pl (db_verify) does not work on a little endian machine
>
>  https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=236256
>
>  Sorry about this inconvenience, and thank you for reporting the problem!
>  --noriko
>
>  Ville Silventoinen wrote:
> > Hi Noriko,
> > 
> > sorry it took so long to reply, I've been busy with other work.
> > 
> > On Fri, 30 Mar 2007, Noriko Hosoi wrote:
> > 
> >> Ville Silventoinen wrote:
> >>> I asked my manager but he doesn't think it's a good idea for security
>
> >>> reasons. The problem is that the data is our NIS mail.aliases and passwd, 
> >>> and we don't want to distribute them to the internet. He suggested
> I'll 
> >>> modify the data, so I can send a sample to you. I'll do that next week.
> >> That would be great. Thanks! I'm interested in what type of characters 
> >> your data contain. E.g., character set is UTF-8? Some of your DNs could 
> >> contain any special characters such as '\'? etc...
> > 
> > The character set should be plain ASCII. I created an imaginary 
> > mail.aliases file. You can download it from here:
> > 
> > http://www.ebi.ac.uk/systems-srv/mp/file-exchange/
> > 
> > Type in "fedorads" to the Pass Phrase input box and click Go. You should 
> > see three files: mail.aliases, mail.aliases.ldif and 99user.ldif.
> > 
> > I can reproduce my problem with the above files, for example, I've tested 
> > like this:
> > 
> > 1. Delete existing ebiRoot database (you could use userRoot).
> > 2. Delete db/ebiRoot directory.
> > 3. Create ebiRoot database.
> > 4. Shutdown slapd.
> > 5. Run db2index and verify-db.pl. No errors.
> > 6. Start slapd.
> > 7. Import mail aliases. I've tried with the Console and my own CLI, which 
> > can import LDIF and add entries one-by-one. The method doesn't seem to 
> > matter.
> > 8. Shutdown slapd.
> > 9. Run db2index and verify-db.pl, verify gives errors:
> > 
> > Verify log files in db ... Good
> > Verify db/ebiRoot/ancestorid.db4 ...
> > DB ERROR: db_verify: Page 2: out-of-order key at entry 254
> > DB ERROR: db_verify: DB->verify: db/ebiRoot/ancestorid.db4: DB_VERIFY_BAD: 
> > Database verification failed
> > Secondary index file ancestorid.db4 in db/ebiRoot is corrupted.
> > Please run db2index(.pl) for reindexing.
> > Verify db/ebiRoot/objectclass.db4 ...
> > DB ERROR: db_verify: Page 2: out-of-order key at entry 255
> > DB ERROR: db_verify: DB->verify: db/ebiRoot/objectclass.db4: DB_VERIFY_BAD: 
> > Database verification failed
> > Secondary index file objectclass.db4 in db/ebiRoot is corrupted.
> > Please run db2index(.pl) for reindexing.
> > Verify db/ebiRoot/nsuniqueid.db4 ... Good
> > Verify db/ebiRoot/parentid.db4 ...
> > DB ERROR: db_verify: Page 1: unsorted duplicate set in sorted-dup database
> > DB ERROR: db_verify: DB->verify: db/ebiRoot/parentid.db4: DB_VERIFY_BAD: 
> > Database verification failed
> > Secondary index file parentid.db4 in db/ebiRoot is corrupted.
> > Please run db2index(.pl) for reindexing.
> > Verify db/ebiRoot/cn.db4 ...
> > DB ERROR: db_verify: Page 10: out-of-order key at entry 249
> > DB ERROR: db_verify: DB->verify: db/ebiRoot/cn.db4: DB_VERIFY_BAD: Database 
> > verification failed
> > Secondary index file cn.db4 in db/ebiRoot is corrupted.
> > Please run db2index(.pl) for reindexing.
> > Verify db/ebiRoot/id2entry.db4 ... Good
> > Verify db/ebiRoot/entrydn.db4 ... Good
> > Verify db/ebiRoot/rfc822mailmember.db4 ...
> > DB ERROR: db_verify: Page 2: unsorted duplicate set in sorted-dup database
> > DB ERROR: db_verify: Page 3: unsorted duplicate set in sorted-dup database
> > DB ERROR: db_verify: DB->verify: db/ebiRoot/rfc822mailmember.db4: 
> > DB_VERIFY_BAD: Database verification failed
> > Secondary index file rfc822mailmember.db4 in db/ebiRoot is corrupted.
> > Please run db2index(.pl) for reindexing.
> > 
> >> So, in your ldif data, the mail attribute also has this type of value: 
> >> "|/homes/majordom/wrapper stripmime.pl|/homes/majordom/wrapper resend -l 
> >> foobar-dev foobar-dev-outgoing"?
> > 
> > No, the People entries have a simpler mail value, like "foo(a)ebi.ac.uk".
> > 
> >> And your mail index has the default indexing type: presence, equality, and 
> >> substring?
> > 
> > Yes.
> > 
> >> What type of indexing does the rfc822MailMember attribute have?
> > 
> > I've tried without any indexing, with presence and equality and with 
> > presence, equality and substring. The above errors are from verify-db.pl 
> > when I have presence and equality indeces. If I have presence, equality and 
> > substring, I get these errors for rfc822MailMember:
> > 
> > Verify db/ebiRoot/rfc822mailmember.db4 ...
> > DB ERROR: db_verify: Page 13: unsorted duplicate set in sorted-dup database
> > DB ERROR: db_verify: Page 6: unsorted duplicate set in sorted-dup database
> > DB ERROR: db_verify: Page 8: unsorted duplicate set in sorted-dup database
> > DB ERROR: db_verify: Page 12: unsorted duplicate set in sorted-dup database
> > DB ERROR: db_verify: Page 3: unsorted duplicate set in sorted-dup database
> > DB ERROR: db_verify: Page 7: unsorted duplicate set in sorted-dup database
> > DB ERROR: db_verify: Page 10: unsorted duplicate set in sorted-dup database
> > DB ERROR: db_verify: Page 15: unsorted duplicate set in sorted-dup database
> > DB ERROR: db_verify: Page 4: unsorted duplicate set in sorted-dup database
> > DB ERROR: db_verify: Page 14: unsorted duplicate set in sorted-dup database
> > DB ERROR: db_verify: Page 5: unsorted duplicate set in sorted-dup database
> > DB ERROR: db_verify: Page 9: unsorted duplicate set in sorted-dup database
> > DB ERROR: db_verify: Page 11: unsorted duplicate set in sorted-dup database
> > DB ERROR: db_verify: DB->verify: db/ebiRoot/rfc822mailmember.db4: 
> > DB_VERIFY_BAD: Database verification failed
> > Secondary index file rfc822mailmember.db4 in db/ebiRoot is corrupted.
> > Please run db2index(.pl) for reindexing.
> > 
> >> Have we already heard what platform you are running the FDS on?
> > 
> > CentOS release 4.4, Linux 2.6.9-42.ELsmp. Pentium III 2x1266MHz CPUs, 2GB 
> > memory, SCSI disks. I'm using FDS 1.0.4.
> > 
> > I'm away this week Wed-Fri, so I'll get back to you next week.
> > 
> > Thanks for the help!
> > 
> > Ville
> > 
> > -- 
> > Fedora-directory-users mailing list
> > Fedora-directory-users(a)redhat.com
> > https://www.redhat.com/mailman/listinfo/fedora-directory-users
>
>

[Noriko Hosoi]

Ville Silventoinen wrote:

>  Thanks Noriko, that explains a lot!
>
>  I still have that other problem to solve, related to re-creating the 
>  database (see the end of my first email). I will try to reproduce this 
>  problem and create some test data this week. 
Hi Ville,

>  If I delete a database with the Console, it leaves behind couple of 
>  index files:
>
>  -rw-------  1 w3secure systems 16384 Mar 28 17:05 ancestorid.db4
>  -rw-------  1 w3secure systems    18 Mar 28 17:03 DBVERSION
>  -rw-------  1 w3secure systems 32768 Mar 28 17:05 id2entry.db4
>
>  These index files don't seem to shrink when new entries are imported. 
>  dbscan still shows the deleted entries in id2entry.
>
>  I noticed a problem when I import a small set of entries, delete the 
>  database, import large set of entries and if I query the entries, I 
>  get the entries from the first set (they don't exist in the second 
>  set). I can reproduce the problem. If I delete ancestorid.db4 and 
>  id2entry.db4 manually when I delete the database, I don't have this 
>  problem. Is there a reason why those two files are not deleted? Or can 
>  this whole thing be caused by corrupted data? 
I wonder this might be your case?

http://www.redhat.com/docs/manuals/dir-server/ag/7.1/entry_dist.html#22293
Deleting a Database
The following procedure describes deleting a directory database using 
the Directory Server Console. Deleting a database deletes the 
configuration information and entries for that database only, *not the 
physical database itself.*

http://www.redhat.com/docs/manuals/dir-server/ag/7.1/dbmanage.html#1117312
Importing a Database from the Console
When you perform an import operation from the Directory Server Console, 
an ldapmodify operation is executed to *append data*, as well as to 
modify and delete entries.

To overwrite the existing data, please take this step before importing 
the data.
http://www.redhat.com/docs/manuals/dir-server/ag/7.1/dbmanage.html#1117339
Initializing a Database from the Console

Or you could do the same thing from the command-line interface:
http://www.redhat.com/docs/manuals/dir-server/ag/7.1/dbmanage.html#1117378
Importing from the Command-Line
You can use three methods for importing data through the command-line:
. Using ldif2db - This import method overwrites the contents of your 
database and requires the server to be stopped.
. Using ldif2db.pl - This import method overwrites the contents of your 
database while the server is still running.

Hope it helps,
--noriko

>  Thanks,
>  Ville
>
>
>  On Thu, 12 Apr 2007, Noriko Hosoi wrote:
>
> > Thank you, Ville, for the test data.  I could reproduce the db_verify 
> > problem.
> >
> > I have good news and bad news. :)  Good news, first...  Your db is 
> > not corrupted.  The error report from verify-db.pl is bogus.
> >
> > Bad news, next.  Please take a look at this bug.  We are going to 
> > provide a fixed utility some time soon.
> >
> > Summary: verify-db.pl (db_verify) does not work on a little endian 
> > machine
> >
> > https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=236256
> >
> > Sorry about this inconvenience, and thank you for reporting the problem!
> > --noriko
> >
> > Ville Silventoinen wrote:
> >
> >> Hi Noriko,
> >>
> >> sorry it took so long to reply, I've been busy with other work.
> >>
> >> On Fri, 30 Mar 2007, Noriko Hosoi wrote:
> >>
> >>> Ville Silventoinen wrote:
> >>>
> >>>> I asked my manager but he doesn't think it's a good idea for 
> >>>> security reasons. The problem is that the data is our NIS 
> >>>> mail.aliases and passwd, and we don't want to distribute them to 
> >>>> the internet. He suggested I'll modify the data, so I can send a 
> >>>> sample to you. I'll do that next week.
> >>>
> >>> That would be great. Thanks! I'm interested in what type of 
> >>> characters your data contain. E.g., character set is UTF-8? Some of 
> >>> your DNs could contain any special characters such as '\'? etc...
> >>
> >>
> >> The character set should be plain ASCII. I created an imaginary 
> >> mail.aliases file. You can download it from here:
> >>
> >> http://www.ebi.ac.uk/systems-srv/mp/file-exchange/
> >>
> >> Type in "fedorads" to the Pass Phrase input box and click Go. You 
> >> should see three files: mail.aliases, mail.aliases.ldif and 
> >> 99user.ldif.
> >>
> >> I can reproduce my problem with the above files, for example, I've 
> >> tested like this:
> >>
> >> 1. Delete existing ebiRoot database (you could use userRoot).
> >> 2. Delete db/ebiRoot directory.
> >> 3. Create ebiRoot database.
> >> 4. Shutdown slapd.
> >> 5. Run db2index and verify-db.pl. No errors.
> >> 6. Start slapd.
> >> 7. Import mail aliases. I've tried with the Console and my own CLI, 
> >> which can import LDIF and add entries one-by-one. The method doesn't 
> >> seem to matter.
> >> 8. Shutdown slapd.
> >> 9. Run db2index and verify-db.pl, verify gives errors:
> >>
> >> Verify log files in db ... Good
> >> Verify db/ebiRoot/ancestorid.db4 ...
> >> DB ERROR: db_verify: Page 2: out-of-order key at entry 254
> >> DB ERROR: db_verify: DB->verify: db/ebiRoot/ancestorid.db4: 
> >> DB_VERIFY_BAD: Database verification failed
> >> Secondary index file ancestorid.db4 in db/ebiRoot is corrupted.
> >> Please run db2index(.pl) for reindexing.
> >> Verify db/ebiRoot/objectclass.db4 ...
> >> DB ERROR: db_verify: Page 2: out-of-order key at entry 255
> >> DB ERROR: db_verify: DB->verify: db/ebiRoot/objectclass.db4: 
> >> DB_VERIFY_BAD: Database verification failed
> >> Secondary index file objectclass.db4 in db/ebiRoot is corrupted.
> >> Please run db2index(.pl) for reindexing.
> >> Verify db/ebiRoot/nsuniqueid.db4 ... Good
> >> Verify db/ebiRoot/parentid.db4 ...
> >> DB ERROR: db_verify: Page 1: unsorted duplicate set in sorted-dup 
> >> database
> >> DB ERROR: db_verify: DB->verify: db/ebiRoot/parentid.db4: 
> >> DB_VERIFY_BAD: Database verification failed
> >> Secondary index file parentid.db4 in db/ebiRoot is corrupted.
> >> Please run db2index(.pl) for reindexing.
> >> Verify db/ebiRoot/cn.db4 ...
> >> DB ERROR: db_verify: Page 10: out-of-order key at entry 249
> >> DB ERROR: db_verify: DB->verify: db/ebiRoot/cn.db4: DB_VERIFY_BAD: 
> >> Database verification failed
> >> Secondary index file cn.db4 in db/ebiRoot is corrupted.
> >> Please run db2index(.pl) for reindexing.
> >> Verify db/ebiRoot/id2entry.db4 ... Good
> >> Verify db/ebiRoot/entrydn.db4 ... Good
> >> Verify db/ebiRoot/rfc822mailmember.db4 ...
> >> DB ERROR: db_verify: Page 2: unsorted duplicate set in sorted-dup 
> >> database
> >> DB ERROR: db_verify: Page 3: unsorted duplicate set in sorted-dup 
> >> database
> >> DB ERROR: db_verify: DB->verify: db/ebiRoot/rfc822mailmember.db4: 
> >> DB_VERIFY_BAD: Database verification failed
> >> Secondary index file rfc822mailmember.db4 in db/ebiRoot is corrupted.
> >> Please run db2index(.pl) for reindexing.
> >>
> >>> So, in your ldif data, the mail attribute also has this type of 
> >>> value: "|/homes/majordom/wrapper 
> >>> stripmime.pl|/homes/majordom/wrapper resend -l foobar-dev 
> >>> foobar-dev-outgoing"?
> >>
> >>
> >> No, the People entries have a simpler mail value, like
> "foo(a)ebi.ac.uk".
> >>
> >>> And your mail index has the default indexing type: presence, 
> >>> equality, and substring?
> >>
> >>
> >> Yes.
> >>
> >>> What type of indexing does the rfc822MailMember attribute have?
> >>
> >>
> >> I've tried without any indexing, with presence and equality and with 
> >> presence, equality and substring. The above errors are from 
> >> verify-db.pl when I have presence and equality indeces. If I have 
> >> presence, equality and substring, I get these errors for 
> >> rfc822MailMember:
> >>
> >> Verify db/ebiRoot/rfc822mailmember.db4 ...
> >> DB ERROR: db_verify: Page 13: unsorted duplicate set in sorted-dup 
> >> database
> >> DB ERROR: db_verify: Page 6: unsorted duplicate set in sorted-dup 
> >> database
> >> DB ERROR: db_verify: Page 8: unsorted duplicate set in sorted-dup 
> >> database
> >> DB ERROR: db_verify: Page 12: unsorted duplicate set in sorted-dup 
> >> database
> >> DB ERROR: db_verify: Page 3: unsorted duplicate set in sorted-dup 
> >> database
> >> DB ERROR: db_verify: Page 7: unsorted duplicate set in sorted-dup 
> >> database
> >> DB ERROR: db_verify: Page 10: unsorted duplicate set in sorted-dup 
> >> database
> >> DB ERROR: db_verify: Page 15: unsorted duplicate set in sorted-dup 
> >> database
> >> DB ERROR: db_verify: Page 4: unsorted duplicate set in sorted-dup 
> >> database
> >> DB ERROR: db_verify: Page 14: unsorted duplicate set in sorted-dup 
> >> database
> >> DB ERROR: db_verify: Page 5: unsorted duplicate set in sorted-dup 
> >> database
> >> DB ERROR: db_verify: Page 9: unsorted duplicate set in sorted-dup 
> >> database
> >> DB ERROR: db_verify: Page 11: unsorted duplicate set in sorted-dup 
> >> database
> >> DB ERROR: db_verify: DB->verify: db/ebiRoot/rfc822mailmember.db4: 
> >> DB_VERIFY_BAD: Database verification failed
> >> Secondary index file rfc822mailmember.db4 in db/ebiRoot is corrupted.
> >> Please run db2index(.pl) for reindexing.
> >>
> >>> Have we already heard what platform you are running the FDS on?
> >>
> >>
> >> CentOS release 4.4, Linux 2.6.9-42.ELsmp. Pentium III 2x1266MHz 
> >> CPUs, 2GB memory, SCSI disks. I'm using FDS 1.0.4.
> >>
> >> I'm away this week Wed-Fri, so I'll get back to you next week.
> >>
> >> Thanks for the help!
> >>
> >> Ville
> >>
> >> -- 
> >> Fedora-directory-users mailing list
> >> Fedora-directory-users(a)redhat.com
> >> https://www.redhat.com/mailman/listinfo/fedora-directory-users
> >
> >
> >
>
>  -- 
>  Fedora-directory-users mailing list
>  Fedora-directory-users(a)redhat.com
>  https://www.redhat.com/mailman/listinfo/fedora-directory-users