[389-users] How to tell when database backup has finished?
MATON Brett
Brett.Maton at nrb.be
Wed Apr 11 15:16:10 UTC 2012
Thanks for the suggestions.
For the record, status 32 was an error on my part, I hadn’t quoted the bind DN so ldapsearch was actually complaining about the bind DN not being found, not the backup task.
Noriko, I’ve searched “cn=backup,cn=tasks,cn=config” and can’t find nstaskstatus anywhere. I’m assuming that it also gets deleted once the task is complete...
cn=backup,cn=tasks,cn=config is the parent entry for all backup task entries. When you create an entry under this parent, it starts the backup task. It is the backup task entry that has the nsTaskStatus attribute, not the parent entry "cn=backup,cn=tasks,cn=config". When the backup task is complete, and the ttl is expired, the backup task entry is automatically removed.
Hi Rich, I appreciate your clarification. I did however mean that I was searching “cn=backup,cn=tasks,cn=config” with no additional filters which should have returned everything, nsTaskStatus included for any backup job?
Exactly what search parameters (e.g. ldapsearch command line arguments) did you use?
I might just be having a dumb moment but I couldn’t think of an efficient way to check the error log for a Backup Finished entry, anyone with suggestions?
The best way is to do the ldapsearch as described above.
I’m still having issues, $taskDN for example would be something like cn=backup_2012_4_5_13_36_23, cn=backup, cn=tasks, cn=config
COUNT=1
while [ ${COUNT} -ne 0 ]; do
#
# OOPS: -b ${taskDN}
# -b "cn=backup,cn=tasks,cn=config" "cn=<task>"
#
COUNT=$(ldapsearch -x -H ${HOST} -D "${BINDAS}" -y ${PWFILE} -b "cn=backup,cn=tasks,cn=config" "(${taskDN})" dn | grep -c '^dn')
so this would expand to something like
COUNT=$(ldapsearch -x -H ldap.example.com -D "cn=directory manager" -y pwfile.txt -b "cn=backup,cn=tasks,cn=config" "(cn=backup_2012_4_5_13_36_23, cn=backup, cn=tasks, cn=config)" dn | grep -c '^dn')
This is incorrect in the following ways:
1) the argument to -H must be a valid LDAP URL - so -H ldap://ldap.example.com $HOST is just a variable name, it’s actually the URL
2) "(cn=backup_2012_4_5_13_36_23, cn=backup, cn=tasks, cn=config)" is not a valid LDAP search filter - if you want to search for the backup entry, do one of the following
a) ldapsearch ... -s base -b "cn=backup_2012_4_5_13_36_23, cn=backup, cn=tasks, cn=config" ...
or
b) ldapsearch ... -b "cn=backup, cn=tasks, cn=config" "(cn=backup_2012_4_5_13_36_23)" ...
OK, search terms corrected.
The command expands to and returns this:
ldapsearch -x -H ldaps://<host fqdn>:636 -D "cn=Directory Manager" -y /root/.389ds-pw -s base -b "cn=backup_2012_4_11_7_28_40, cn=backup, cn=tasks, cn=config"
# extended LDIF
#
# LDAPv3
# base <cn=backup_2012_4_11_7_28_40, cn=backup, cn=tasks, cn=config> with scope baseObject
# filter: (objectclass=*)
# requesting: ALL
#
# search result
search: 2
result: 32 No such object
# numResponses: 1
I tweaked the db2bak.pl script to include ttl: 600 as I wanted some time to examine the backup task entry and search results:
...
$dn = "dn: cn=$taskname, cn=backup, cn=tasks, cn=config\n";
$misc = "changetype: add\nobjectclass: top\nobjectclass: extensibleObject\nttl: 600\n";
$cn = "cn: $taskname\n";
...
The entry gets added, but still disappears near instantaneously (db2bak.pl called standalone for output):
Back up directory: /var/lib/dirsrv/slapd-<host>/bak/<host>-2012_4_11_7_42_4
ldap_initialize( <DEFAULT> )
add objectclass:
top
extensibleObject
add ttl:
600
add cn:
backup_2012_4_11_7_42_4
add nsArchiveDir:
/var/lib/dirsrv/slapd-<host>/bak/<host>-2012_4_11_7_42_4
add nsDatabaseType:
ldbm database
adding new entry "cn=backup_2012_4_11_7_42_4, cn=backup, cn=tasks, cn=config"
modify complete
As you can see from the ls output below, tar is still being called before the task has completed:
[root@<host> backup]# ls -ltr
-rw-r--r--. 1 root root 6788 Apr 11 07:28 20120411072840-<host>.tgz
drwx------. 5 nobody nobody 4096 Apr 11 07:28 20120411072840-<host>
Thanks for your time....
ok - I'd like you to try two more things:
1) alter your search parameters - do this instead:
ldapsearch -x -H ldaps://<host fqdn>:636 -D "cn=Directory Manager" -y /root/.389ds-pw -b "cn=backup, cn=tasks, cn=config"
do you see anything?
Ok this is a bit freaky:
Executing db2pk.pl manually from the command line I get this in /var/log/dirsrv/<instance>/error:
[11/Apr/2012:15:33:11 +0200] - Beginning backup of 'ldbm database'
[11/Apr/2012:15:33:12 +0200] - Backing up file 35 (/var/lib/dirsrv/slapd-<instance>/bak/<instance>-2012_4_11_15_33_12/.repl_changelog_backup/changelogdb/6b85db82-5df911e1-ab138798-28541438_4f45faa60000004b0000.db4)
[11/Apr/2012:15:33:12 +0200] - Backing up file 36 (/var/lib/dirsrv/slapd-<instance>/bak/<instance>-2012_4_11_15_33_12/.repl_changelog_backup/changelogdb/DBVERSION)
[11/Apr/2012:15:33:12 +0200] - Backup finished.
ldapsearch -x -H ldaps://<instance>.nrb.be:636 -D "cn=Directory Manager" -y /root/.389ds-pw -b "cn=backup_2012_4_11_15_33_12,cn=backup,cn=tasks,cn=config" # extended LDIF
#
# LDAPv3
# base <cn=backup_2012_4_11_15_33_12,cn=backup,cn=tasks,cn=config> with scope subtree
# filter: (objectclass=*)
# requesting: ALL
#
# backup_2012_4_11_15_33_12, backup, tasks, config
dn: cn=backup_2012_4_11_15_33_12,cn=backup,cn=tasks,cn=config
objectClass: top
objectClass: extensibleObject
ttl: 600
cn: backup_2012_4_11_15_33_12
nsarchivedir: /var/lib/dirsrv/slapd-<instance>/bak/<instance>-2012_4_11_15_33_12
nsdatabasetype: ldbm database
nstaskcurrentitem: 0
nstasktotalitems: 1
nstasklog:: <snipped>
nstaskstatus: Backup finished.
nstaskexitcode: 0
# search result
search: 2
result: 0 Success
# numResponses: 2
# numEntries: 1
Scripted version:
/var/log/dirsrv/<instance>/error:
[11/Apr/2012:15:35:06 +0200] - Beginning backup of 'ldbm database'
[11/Apr/2012:15:35:07 +0200] - Backing up file 1 (/usr/backup/20120411153507-<instance>/userRoot/telephoneNumber.db4)
[11/Apr/2012:15:35:07 +0200] - Backing up file 2 (/usr/backup/20120411153507-<instance>/userRoot/ntUserDomainId.db4)
[11/Apr/2012:15:35:07 +0200] - Backing up file 3 (/usr/backup/20120411153507-<instance>/userRoot/numsubordinates.db4)
[11/Apr/2012:15:35:07 +0200] - Backing up file 4 (/usr/backup/20120411153507-<instance>/userRoot/objectclass.db4)
[11/Apr/2012:15:35:07 +0200] - Backing up file 5 (/usr/backup/20120411153507-<instance>/userRoot/mail.db4)
[11/Apr/2012:15:35:07 +0200] - Backing up file 6 (/usr/backup/20120411153507-<instance>/userRoot/ntUniqueId.db4)
[11/Apr/2012:15:35:07 +0200] - Backing up file 7 (/usr/backup/20120411153507-<instance>/userRoot/parentid.db4)
[11/Apr/2012:15:35:07 +0200] - Backing up file 8 (/usr/backup/20120411153507-<instance>/userRoot/aci.db4)
[11/Apr/2012:15:35:07 +0200] - Backing up file 9 (/usr/backup/20120411153507-<instance>/userRoot/cn.db4)
[11/Apr/2012:15:35:07 +0200] - Backing up file 10 (/usr/backup/20120411153507-<instance>/userRoot/seeAlso.db4)
[11/Apr/2012:15:35:07 +0200] - Backing up file 11 (/usr/backup/20120411153507-<instance>/userRoot/entryrdn.db4)
[11/Apr/2012:15:35:07 +0200] - Backing up file 12 (/usr/backup/20120411153507-<instance>/userRoot/uid.db4)
[11/Apr/2012:15:35:07 +0200] - Backing up file 13 (/usr/backup/20120411153507-<instance>/userRoot/ancestorid.db4)
[11/Apr/2012:15:35:07 +0200] - Backing up file 14 (/usr/backup/20120411153507-<instance>/userRoot/givenName.db4)
[11/Apr/2012:15:35:07 +0200] - Backing up file 15 (/usr/backup/20120411153507-<instance>/userRoot/sn.db4)
[11/Apr/2012:15:35:07 +0200] - Backing up file 16 (/usr/backup/20120411153507-<instance>/userRoot/uniquemember.db4)
[11/Apr/2012:15:35:07 +0200] - Backing up file 17 (/usr/backup/20120411153507-<instance>/userRoot/nsuniqueid.db4)
[11/Apr/2012:15:35:07 +0200] - Backing up file 18 (/usr/backup/20120411153507-<instance>/userRoot/nscpEntryDN.db4)
[11/Apr/2012:15:35:07 +0200] - Backing up file 19 (/usr/backup/20120411153507-<instance>/userRoot/id2entry.db4)
[11/Apr/2012:15:35:07 +0200] - Backing up file 20 (/usr/backup/20120411153507-<instance>/userRoot/DBVERSION)
[11/Apr/2012:15:35:07 +0200] - Backing up file 21 (/usr/backup/20120411153507-<instance>/NetscapeRoot/numsubordinates.db4)
[11/Apr/2012:15:35:07 +0200] - Backing up file 22 (/usr/backup/20120411153507-<instance>/NetscapeRoot/objectclass.db4)
[11/Apr/2012:15:35:07 +0200] - Backing up file 23 (/usr/backup/20120411153507-<instance>/NetscapeRoot/parentid.db4)
[11/Apr/2012:15:35:07 +0200] - Backing up file 24 (/usr/backup/20120411153507-<instance>/NetscapeRoot/aci.db4)
[11/Apr/2012:15:35:07 +0200] - Backing up file 25 (/usr/backup/20120411153507-<instance>/NetscapeRoot/cn.db4)
[11/Apr/2012:15:35:07 +0200] - Backing up file 26 (/usr/backup/20120411153507-<instance>/NetscapeRoot/entryrdn.db4)
[11/Apr/2012:15:35:07 +0200] - Backing up file 27 (/usr/backup/20120411153507-<instance>/NetscapeRoot/uid.db4)
[11/Apr/2012:15:35:07 +0200] - Backing up file 28 (/usr/backup/20120411153507-<instance>/NetscapeRoot/ancestorid.db4)
[11/Apr/2012:15:35:07 +0200] - Backing up file 29 (/usr/backup/20120411153507-<instance>/NetscapeRoot/givenName.db4)
[11/Apr/2012:15:35:07 +0200] - Backing up file 30 (/usr/backup/20120411153507-<instance>/NetscapeRoot/sn.db4)
[11/Apr/2012:15:35:07 +0200] - Backing up file 31 (/usr/backup/20120411153507-<instance>/NetscapeRoot/uniquemember.db4)
[11/Apr/2012:15:35:07 +0200] - Backing up file 32 (/usr/backup/20120411153507-<instance>/NetscapeRoot/nsuniqueid.db4)
[11/Apr/2012:15:35:07 +0200] - Backing up file 33 (/usr/backup/20120411153507-<instance>/NetscapeRoot/id2entry.db4)
[11/Apr/2012:15:35:07 +0200] - Backing up file 34 (/usr/backup/20120411153507-<instance>/NetscapeRoot/DBVERSION)
[11/Apr/2012:15:35:07 +0200] - Backing up file 35 (/usr/backup/20120411153507-<instance>/.repl_changelog_backup/changelogdb/6b85db82-5df911e1-ab138798-28541438_4f45faa60000004b0000.db4)
[11/Apr/2012:15:35:07 +0200] - Backing up file 36 (/usr/backup/20120411153507-<instance>/.repl_changelog_backup/changelogdb/DBVERSION)
[11/Apr/2012:15:35:07 +0200] - Backing up file 37 (/usr/backup/20120411153507-<instance>/log.0000000018)
[11/Apr/2012:15:35:07 +0200] - Backing up file 38 (/usr/backup/20120411153507-<instance>/DBVERSION)
[11/Apr/2012:15:35:07 +0200] - Backup finished.
Search result:
ldapsearch -x -H ldaps://<host fqdn>:636 -D "cn=Directory Manager" -y /root/.389ds-pw -s base -b "cn=backup_2012_4_11_15_35_7, cn=backup, cn=tasks, cn=config"
# extended LDIF
#
# LDAPv3
# base <cn=backup_2012_4_11_15_35_7, cn=backup, cn=tasks, cn=config> with scope baseObject
# filter: (objectclass=*)
# requesting: ALL
#
# search result
search: 2
result: 32 No such object
# numResponses: 1
Search without taskid:
ldapsearch -x -H ldaps://<host fqdn>:636 -D "cn=Directory Manager" -y /root/.389ds-pw -s base -b "cn=backup,cn=tasks,cn=config"
# extended LDIF
#
# LDAPv3
# base <cn=backup,cn=tasks,cn=config> with scope baseObject
# filter: (objectclass=*)
# requesting: ALL
#
# backup, tasks, config
dn: cn=backup,cn=tasks,cn=config
objectClass: top
objectClass: extensibleObject
cn: backup
# search result
search: 2
result: 0 Success
# numResponses: 2
# numEntries: 1
The only difference I can see is that the scripted version specifies a backup path with the –a option
Another test including the –a option from the command line:
[11/Apr/2012:16:10:31 +0200] - Beginning backup of 'ldbm database'
[11/Apr/2012:16:10:31 +0200] - Backing up file 35 (/tmp/test/.repl_changelog_backup/changelogdb/6b85db82-5df911e1-ab138798-28541438_4f45faa60000004b0000.db4)
[11/Apr/2012:16:10:31 +0200] - Backing up file 36 (/tmp/test/.repl_changelog_backup/changelogdb/DBVERSION)
[11/Apr/2012:16:10:31 +0200] - Backup finished.
ldapsearch does return the backup task, although the error log is only listing backup files 35 / 36 I created tgz’s of the backups and both command line / scripted version were identical in size.
Even more confused J
As am I. I'm just not sure what's going on, nor do I know why adding -a would make any difference here. If you think there is a bug here, please file a ticket at https://fedorahosted.org/389
I dig around at it some more tomorrow to see if I can nail it down any further first.
2) can you confirm using the errors log (manually, not automated with a script) that your backup task was started successfully and completes successfully? Just to be sure that it is indeed being started and finishing correctly
Error logs above.
if [ ${COUNT} -ne 0 ]; then
sleep 5s
fi
done
tar czf ${BACKUPPATH}${BKTSTAMP}-${INSTANCE}.tgz -C / ${DSBACKUPDIR#/*}
However the tar file is incomplete, because (assumption) the backup has not finished writing to disk before tar starts....
For the time being I’ve settled on two scripts, one to start backups and another to archive yesterdays (or older) backups. Not ideal but workable
Thanks again,
Brett
From: 389-users-bounces at lists.fedoraproject.org [mailto:389-users-bounces at lists.fedoraproject.org] On Behalf Of Noriko Hosoi
Sent: 05 April 2012 18:40
To: General discussion list for the 389 Directory server project.
Subject: Re: [389-users] How to tell when database backup has finished?
You could search the task.with nstaskstatus attribute.
$ ldapsearch -LLLx -h localhost -p <port> -D 'cn=directory manager' -w <pw> -b "cn=backup,cn=tasks,cn=config" "(cn=backup_*)" nstaskstatus
When the task is done, you'll see the "Backup finished." status:
dn: cn=backup_2012_4_5_9_35_35,cn=backup,cn=tasks,cn=config
nstaskstatus: Backup finished.
Another way would be checking the errors log, which logs the end of the back up.
[05/Apr/2012:09:32:09 -0700] - Backing up file 31 (/var/lib/dirsrv/slapd-totoro/bak/totoro-2012_4_5_9_32_8/DBVERSION)
[05/Apr/2012:09:32:09 -0700] - Backup finished.
Thanks,
--noriko
MATON Brett wrote:
Hi guys,
I want to tar up the backup set once a db2bak.pl backup job has finished.
however because it writes an entry to the LDAP and doesn’t wait for the backup to finish I was wondering what the best way to determine when the backup has actually finished?
I was thinking that I could
Ldapsearch at intervals for the backup cn entry, when it’s not there anymore backup has finished
Ro maybe watch the audit log for a delete action on the backup cn entry..
Any thoughts appreciated.
Cheers,
Brett
-------------------------------------------------------------------
GreeNRB
NRB considers its environmental responsibility and goes for green IT.
May we ask you to consider yours before printing this e-mail?
NRB, daring to commit
This e-mail and any attachments, which may contain information that is confidential and/or protected by intellectual property rights, are intended for the exclusive use of the above-mentioned addressee(s). Any use (including reproduction, disclosure and whole or partial distribution in any form whatsoever) of their content is prohibited without prior authorization of NRB. If you have received this message by error, please contact the sender promptly by resending this e-mail back to him (her), or by calling the above number. Thank you for subsequently deleting this e-mail and any files attached thereto.
-------------------------------------------------------------------
This e-mail and any attachments, which may contain information that is confidential and/or protected by intellectual property rights, are intended for the exclusive use of the above-mentioned addressee(s). Any use (including reproduction, disclosure and whole or partial distribution in any form whatsoever) of their content is prohibited without prior authorization of NRB. If you have received this message by error, please contact the sender promptly by resending this e-mail back to him (her), or by calling the above number. Thank you for subsequently deleting this e-mail and any files attached thereto.
-------------------------------------------------------------------
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.fedoraproject.org/pipermail/389-users/attachments/20120411/c131d7eb/attachment.html>
More information about the 389-users
mailing list