Re: IPA Reinstall

Hi Florence,
Thank you for taking the time to respond.

We have a number of replicas in various campus locations around our region, and our state and country is currently locked-down with COVID restrictions, and it would take days to get to the replicas even without restrictions. We have had wide-scale WAN outages over weeks for this system.

To describe our problem in a simplified manner;
We had the IPA replicas ‘Location1R1’ and ‘Location2R1’ talking happily when the WAN between them went away, while it was down we added ‘Location2R2’ to ‘Location2R1’, and manually added some user-ranges for accounts at Location2. We also added a number of ipa-clients at Location2. When the WAN came back, ‘Location1R1’ and ‘Location2R1’ stopped sharing data because the CSN updates were dropped weeks ago. We re-initialized ‘Location2R1’ from ‘Location1R1’, however that has resulted in isolation of ‘Location2R2’. Most logins at Location2 fail, depending on where the DNS round-robin points SSSD at the time of login.

We have multiple locations, each with a flavor of this issue. No replica has a complete picture of all replication agreements, host accounts and user accounts. Though we have had some success with continually re-initializing replicas.

It was my hope, once the WAN was stabilized to remove and re ipa-server-install at our first server, then remove and re-run ipa-client-install then ipa-replica-install at each secondary server. I believe this would re-constitute the system, effectively by re-installation of that specific service. And yes, this would all be done via SSH.

However, we meet the obstacles described in my initial email when performing the install a second time.

Sincerely, and with many thanks in advance,
Rob

From: Florence Renaud via FreeIPA-users 
Sent: Wednesday, 2 June 2021 2:16 AM
To: FreeIPA users list 
Cc: Mattson, Robert (FP) ; Florence Renaud 
Subject: [EXTERNAL] [Freeipa-users] Re: IPA Reinstall

Hi,

the recommended way to uninstall a replica and reinstall it is described in the doc:
1. Uninstall the replica (https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/htm...) with ipa server-del and ipa-server-install --uninstall
2. re-install the replica as if it was a new one: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/htm...

Was there any reason to backup files and restore them? The replica installation should re-create everything.
flo

On Wed, May 26, 2021 at 7:32 AM Robert.Mattson--- via FreeIPA-users <[e-mail]> wrote:
Dear Community,

I'd like to uninstall and reinstall IPA from a CentOS box because its easier than reinstalling the OS completely.
We have a number of replicas, and this host is installed using ipa-client-install and then ipa-replica-install.
To remove it, I backup some data like /var/kerberos/krb5kdc/{cacert.pem,kd*} and /etc/httpd/conf/password.conf
and then run '/usr/sbin/ipa-server-install --uninstall -U --ignore-topology-disconnect'.
I then sed '/Environment=K/d',  '/ExecStartPre/d', '/ExecStopPost/d' /etc/systemd/system/httpd.service

I recreate the host-account on another replica using ipa host-add, then ipa hostgroup-add-member.

On the now-removed host, I do some housekeeping like restoring the backed up files and then I run;
/usr/sbin/ipa-client-install  \
  --password=${otp} \
  --mkhomedir \
  --no-ntp \
  --unattended \
  --domain=realm.namehttp://realm.name \
  --realm=REALM.NAMEhttp://REALM.NAME \
  --ca-cert-file=/etc/pki/ca-trust/source/ca.crt

then

/usr/sbin/ipa-replica-install \
--dirsrv-cert-file=/etc/pki/tls/private/ipa.pkcs12 \
--http-cert-file=/etc/pki/tls/private/ipa.pkcs12 \
--dirsrv-pin=pwd \
--http-pin=pwd \
--unattended \
--no-pkinit \
--no-ntp

I seem to get the following keytab request problem followed by dirsrv failure. from ipa-replica-install (4.6.4-10.el7.centos.3.x86_64). If I upgrade to 4.6.8-5.el7.centos.4.noarch.rpm, I get the same problem.[1]
On serverb, the host which receives the binding request for the reinstall, I get permission denied the bind dn “” does not have permission in dirsrv error log…?

Does anyone have any thoughts,

Cheers and many thanks in advance,
Rob

[1]
2021-05-26T02:50:56Z DEBUG Backing up system configuration file '/etc/httpd/conf.d/ipa.conf'
2021-05-26T02:50:56Z DEBUG   -> Not backing up - '/etc/httpd/conf.d/ipa.conf' doesn't exist
2021-05-26T02:50:56Z DEBUG Backing up system configuration file '/etc/httpd/conf.d/ipa-rewrite.conf'
2021-05-26T02:50:56Z DEBUG   -> Not backing up - '/etc/httpd/conf.d/ipa-rewrite.conf' doesn't exist
2021-05-26T02:50:56Z DEBUG   duration: 0 seconds
2021-05-26T02:50:56Z DEBUG   [10/21]: setting up httpd keytab
2021-05-26T02:50:56Z DEBUG raw: service_add(u'HTTP/[e-mail]', force=True, version=u'2.230')
2021-05-26T02:50:56Z DEBUG service_add(ipapython.kerberos.Principal('HTTP/[e-mail]'), force=True, all=False, raw=False, version=u'2.230', no_members=False)
2021-05-26T02:50:56Z DEBUG flushing ldapi://%2Fvar%2Frun%2Fslapd-REALM-NAME.socket from SchemaCache
2021-05-26T02:50:56Z DEBUG retrieving schema for SchemaCache url=ldapi://%2Fvar%2Frun%2Fslapd-REALM-NAME.socket conn=<ldap.ldapobject.SimpleLDAPObject instance at 0x7f0d29f50368>
2021-05-26T02:50:57Z DEBUG raw: host_show(u'servera.system', version=u'2.230')
2021-05-26T02:50:57Z DEBUG host_show(u'servera.system', rights=False, all=False, raw=False, version=u'2.230', no_members=False)
2021-05-26T02:50:57Z DEBUG Backing up system configuration file '/var/lib/ipa/gssproxy/http.keytab'
2021-05-26T02:50:57Z DEBUG   -> Not backing up - '/var/lib/ipa/gssproxy/http.keytab' doesn't exist
2021-05-26T02:50:57Z DEBUG Starting external process
2021-05-26T02:50:57Z DEBUG args=/usr/sbin/ipa-getkeytab -k /var/lib/ipa/gssproxy/http.keytab -p HTTP/[e-mail] -H ldapi://%2Fvar%2Frun%2Fslapd-REALM-NAME.socket -Y EXTERNAL
2021-05-26T02:50:57Z DEBUG Process finished, return code=9
2021-05-26T02:50:57Z DEBUG stdout=
2021-05-26T02:50:57Z DEBUG stderr=Failed to parse result: unsupported extended operation
Retrying with pre-4.0 keytab retrieval method...
Failed to parse result: unsupported extended operation
Failed to get keytab!
Failed to get keytab

2021-05-26T02:50:57Z DEBUG Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 570, in start_creation
    run_step(full_msg, method)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 560, in run_step
    method()
  File "/usr/lib/python2.7/site-packages/ipaserver/install/httpinstance.py", line 637, in request_service_keytab
    super(HTTPInstance, self).request_service_keytab()
  File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 742, in request_service_keytab
    self.run_getkeytab(self.api.env.ldap_uri, self.keytab, self.principal)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 732, in run_getkeytab
    ipautil.run(args, nolog=nolog)
  File "/usr/lib/python2.7/site-packages/ipapython/ipautil.py", line 562, in run
    raise CalledProcessError(p.returncode, arg_string, str(output))
CalledProcessError: Command '/usr/sbin/ipa-getkeytab -k /var/lib/ipa/gssproxy/http.keytab -p HTTP/[e-mail] -H ldapi://%2Fvar%2Frun%2Fslapd-REALM-NAME.socket -Y EXTERNAL' returned non-zero exit status 9

2021-05-26T02:50:57Z DEBUG   [error] CalledProcessError: Command '/usr/sbin/ipa-getkeytab -k /var/lib/ipa/gssproxy/http.keytab -p HTTP/[e-mail] -H ldapi://%2Fvar%2Frun%2Fslapd-REALM-NAME.socket -Y EXTERNAL' returned non-zero exit status 9
2021-05-26T02:50:57Z DEBUG   File "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 178, in execute
    return_value = self.run()
  File "/usr/lib/python2.7/site-packages/ipapython/install/cli.py", line 319, in run
    return cfgr.run()
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 364, in run
    return self.execute()
    exc_handler(exc_info)
<snip />
  File "/usr/lib/python2.7/site-packages/ipapython/install/common.py", line 65, in _install
    for unused in self._installer(self.parent):
  File "/usr/lib/python2.7/site-packages/ipaserver/install/server/__init__.py", line 622, in main
    replica_install(self)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/server/replicainstall.py", line 406, in decorated
    func(installer)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/server/replicainstall.py", line 1487, in install
    fstore=fstore)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/server/replicainstall.py", line 173, in install_http
    subject_base=config.subject_base, master_fqdn=config.master_host_name)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/httpinstance.py", line 188, in create_instance
    self.start_creation()
  File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 570, in start_creation
    run_step(full_msg, method)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 560, in run_step
    method()
  File "/usr/lib/python2.7/site-packages/ipaserver/install/httpinstance.py", line 637, in request_service_keytab
    super(HTTPInstance, self).request_service_keytab()
  File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 742, in request_service_keytab
    self.run_getkeytab(self.api.env.ldap_uri, self.keytab, self.principal)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 732, in run_getkeytab
    ipautil.run(args, nolog=nolog)
  File "/usr/lib/python2.7/site-packages/ipapython/ipautil.py", line 562, in run
    raise CalledProcessError(p.returncode, arg_string, str(output))

2021-05-26T02:50:57Z DEBUG The ipa-replica-install command failed, exception: CalledProcessError: Command '/usr/sbin/ipa-getkeytab -k /var/lib/ipa/gssproxy/http.keytab -p HTTP/[e-mail] -H ldapi://%2Fvar%2Frun%2Fslapd-REALM-NAME.socket -Y EXTERNAL' returned non-zero exit status 9
2021-05-26T02:50:57Z ERROR Command '/usr/sbin/ipa-getkeytab -k /var/lib/ipa/gssproxy/http.keytab -p HTTP/[e-mail] -H ldapi://%2Fvar%2Frun%2Fslapd-REALM-NAME.socket -Y EXTERNAL' returned non-zero exit status 9
2021-05-26T02:50:57Z ERROR The ipa-replica-install command failed. See /var/log/ipareplica-install.log for more information

[2]
[26/May/2021:12:50:47.240285166 +1000] - WARN - NSMMReplicationPlugin - repl5_inc_run - agmt="cn=meToservera.system" (servera:389): The remote replica has a different database generation ID than the local database.  You may have to reinitialize the remote replica, or the local replica.
[26/May/2021:12:50:47.858057379 +1000] - INFO - NSMMReplicationPlugin - repl5_tot_run - Beginning total update of replica "agmt="cn=meToservera.system" (servera:389)".
[26/May/2021:12:50:50.679652092 +1000] - INFO - NSMMReplicationPlugin - repl5_tot_run - Finished total update of replica "agmt="cn=meToservera.system" (servera:389)". Sent 582 entries.
[26/May/2021:12:50:52.158394667 +1000] - ERR - NSMMReplicationPlugin - acquire_replica - agmt="cn=meToservera.system" (servera:389): Unable to acquire replica: permission denied. The bind dn "" does not have permission to supply replication updates to the replica. Will retry later.
[26/May/2021:12:50:55.079367688 +1000] - ERR - NSMMReplicationPlugin - acquire_replica - agmt="cn=meToservera.system" (servera:389): Unable to acquire replica: permission denied. The bind dn "" does not have permission to supply replication updates to the replica. Will retry later.
[26/May/2021:12:50:58.084381230 +1000] - ERR - NSMMReplicationPlugin - acquire_replica - agmt="cn=meToservera.system" (servera:389): Unable to acquire replica: permission denied. The bind dn "" does not have permission to supply replication updates to the replica. Will retry later.
[26/May/2021:12:51:01.092727541 +1000] - ERR - NSMMReplicationPlugin - acquire_replica - agmt="cn=meToservera.system" (servera:389): Unable to acquire replica: permission denied. The bind dn "" does not have permission to supply replication updates to the replica. Will retry later.

CONFIDENTIALITY NOTICE: This email and any attachments are for the sole use of the intended recipient and may contain material that is proprietary, confidential, privileged or otherwise legally protected or restricted under applicable government laws. Any review, disclosure, distributing or other use without expressed permission of the sender is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies without reading, printing, or saving.

_______________________________________________
FreeIPA-users mailing list -- [e-mail]
To unsubscribe send an email to [e-mail]
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedorahosted.org/archives/list/[e-mail]...
Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure