[Freeipa-users] Re: possible issue with ipa-backup on RHEL 9.3

[Alexander Bokovoy]

On Пят, 22 сне 2023, Charles Hedrick via FreeIPA-users wrote:
> A bit more info. Looking at errors, a normal backup terminates with
>
> [20/Dec/2023:23:01:32.943228301 -0500] - INFO - archive_copyfile - Copying
> /etc/dirsrv/slapd-CS-RUTGERS-EDU/pwdfile.txt to /var/lib/dirsrv/slapd-\
> CS-RUTGERS-EDU/bak/CS-RUTGERS-EDU/config_files/pwdfile.txt
> [20/Dec/2023:23:01:32.957342035 -0500] - INFO - archive_copyfile - Copying
> /etc/dirsrv/slapd-CS-RUTGERS-EDU/certmap.conf to /var/lib/dirsrv/slapd\
> -CS-RUTGERS-EDU/bak/CS-RUTGERS-EDU/config_files/certmap.conf
> [20/Dec/2023:23:01:32.969828971 -0500] - INFO - archive_copyfile - Copying
> /etc/dirsrv/slapd-CS-RUTGERS-EDU/slapd-collations.conf to /var/lib/dir\
> srv/slapd-CS-RUTGERS-EDU/bak/CS-RUTGERS-EDU/config_files/slapd-collations.conf
> [20/Dec/2023:23:01:32.983763256 -0500] - INFO - task_backup_thread - Backup finished.
> [2
>
> The backup that hung is missing the last line, "Backup finished." ldap
> stopped giving normal responses about a minute later, according to the
> access log. 
This looks like a thing internal to 389-ds. If you'd see it reproduced,
make sure to have debuginfo packages for 389-ds and freeipa installed
and then attempt to get a backtrace from 389-ds processes before you'd
kill them.


> ________________________________
> From: Charles Hedrick
> Sent: Friday, December 22, 2023 9:56 AM
> To: freeipa-users(a)lists.fedorahosted.org <freeipa-users(a)lists.fedorahosted.org&gt;
> Subject: possible issue with ipa-backup on RHEL 9.3
>
> I just upgraded one of three servers from RHEL 9.2. to 9.3. I have a clone of our three
> servers, on which all three have been upgraded to 9.3.
>
> All of the servers run a cron job
>
> /sbin/ipa-backup --online --data > /usr/local/scripts/ipa-backup.log 2>&1
>
> The LDAP server hung (needed kill -9) at about the time that job ran, on the production
> server but not the testing copy. Obviously I can't prove that the backup caused the
> hang, but it's suspicious. I've commented out the cron job, since the backup
> isn't actually all the useful. If we have to restore we'd use a snapshot of the
> VM.
>
> The backup completed successfully on the clone. On the production server it failed. Here
> is the log:
>
> Preparing backup on krb4.cs.rutgers.edu
> Local roles match globally used roles, proceeding.
> Backing up userRoot in CS-RUTGERS-EDU to LDIF
> Waiting for LDIF to finish
> Backing up CS-RUTGERS-EDU
> Waiting for BAK to finish
> cannot connect to 'ldapi://%2Frun%2Fslapd-CS-RUTGERS-EDU.socket':
> The ipa-backup command failed. See /var/log/ipabackup.log for more information
>
> I'm wondering whether there's a bug that only happens under load.
>
> We're been doing this in production for years with no trouble up to RHEL 9.2.
>  



-- 
/ Alexander Bokovoy
Sr. Principal Software Engineer
Security / Identity Management Engineering
Red Hat Limited, Finland