Hi.
Into my environment I have two IPA server, replicating each other. They are both 7.6 OS systems, ipa-server RPM version is 4.6.4-10.0.1.el7_6.2.x86_64.
The first server installed was srv01 (many years ago), then I installed the replica into srv02 (like a year later the 1st node). When I had a single server I did also a trust with my corporate Active Directory. VMs are running in 2 different hypervisor clusters.
Now the replication doesn't works. Into log files I have this error:
*[16/Apr/2020:12:25:36.856632697 +0200] - ERR - csngen_adjust_time - Adjustment limit exceeded; value - 23221226, limit - 86400[16/Apr/2020:12:25:36.857909222 +0200] - ERR - NSMMReplicationPlugin - repl5_inc_run - agmt="cn=meTosrv01.ipa.mydomain.com http://meTosrv01.ipa.mydomain.com" (srv01:389): Fatal error - too much time skew between replicas![16/Apr/2020:12:25:36.862233147 +0200] - ERR - NSMMReplicationPlugin - repl5_inc_run - agmt="cn=meTosrv01.ipa.mydomain.com http://meTosrv01.ipa.mydomain.com" (srv01:389): Incremental update failed and requires administrator action*
I tried to force the replica, but the limit exceeded problem doesn't allow the sync. I know that the problem is that CSN generator has become grossly skewed. Using the external script readNsState.py I found that there was as offset time for about a month, so ... I waited for a month and then the issue disappeared. But now the offset is about 9 months ... I can't wait so much time :)
*[root@srv01 scripts]# ./readNsState.py /etc/dirsrv/slapd-IPA-MYDOMAIN-COM/dse.ldifnsState is BAAAAAAAAACCN/xfAAAAAHbiBAAAAAAABCgAAAAAAAANdQAAAAAAAA==Little EndianFor replica cn=replica,cn=dc\3Dipa\2Cdc\3Dmydomain\2Cdc\3Dcom,cn=mapping tree,cn=con fmtstr=[H6x3QH6x] size=40 len of nsstate is 40 CSN generator state: Replica ID : 4 Sampled Time : 1610364802 Gen as csn : 5ffc37822996500040000 Time as str : Mon Jan 11 12:33:22 2021 Local Offset : 320118 Remote Offset : 10244 Seq. num : 29965 System time : Tue Apr 21 15:03:45 2020 Diff in sec. : -22890577 Day:sec diff : -265:5423nsState is YAAAAAAAAAADLZheAAAAAAAAAAAAAAAAXSgAAAAAAAATAAAAAAAAAA==Little EndianFor replica cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config fmtstr=[H6x3QH6x] size=40 len of nsstate is 40 CSN generator state: Replica ID : 96 Sampled Time : 1587031299 Gen as csn : 5e982d03001900960000 Time as str : Thu Apr 16 12:01:39 2020 Local Offset : 0 Remote Offset : 10333 Seq. num : 19 System time : Tue Apr 21 15:03:45 2020 Diff in sec. : 442926 Day:sec diff : 5:10926[root@srv02 scripts]# ./readNsState.py /etc/dirsrv/slapd-IPA-MYDOMAIN-COM/dse.ldifnsState is AwAAAAAAAABU7p5eAAAAAAAAAAAAAAAAsVNiAQAAAAAAAAAAAAAAAA==Little EndianFor replica cn=replica,cn=dc\3Dipa\2Cdc\3Dmydomain\2Cdc\3Dcom,cn=mapping tree,cn=con fmtstr=[H6x3QH6x] size=40 len of nsstate is 40 CSN generator state: Replica ID : 3 Sampled Time : 1587474004 Gen as csn : 5e9eee54000000030000 Time as str : Tue Apr 21 15:00:04 2020 Local Offset : 0 Remote Offset : 23221169 Seq. num : 0 System time : Tue Apr 21 15:02:38 2020 Diff in sec. : 154 Day:sec diff : 0:154nsState is YQAAAAAAAAAuLZheAAAAAEUBAAAAAAAA7SYAAAAAAAASAAAAAAAAAA==Little EndianFor replica cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config fmtstr=[H6x3QH6x] size=40 len of nsstate is 40 CSN generator state: Replica ID : 97 Sampled Time : 1587031342 Gen as csn : 5e982d2e001800970000 Time as str : Thu Apr 16 12:02:22 2020 Local Offset : 325 Remote Offset : 9965 Seq. num : 18 System time : Tue Apr 21 15:02:38 2020 Diff in sec. : 442816 Day:sec diff : 5:10816*
As you can see in the 1st node the Time as str is Jan 11 of 2021. With timedatectl command I see that both VMs use the same Time zone and the clock is correct.
I found this old article to fix my issue: *https://www.redhat.com/archives/freeipa-users/2014-February/msg00007.html https://www.redhat.com/archives/freeipa-users/2014-February/msg00007.html*
But ... I had the same issue in the past, always in the 1st server. So, in my mind I don't want to try to use that fix. I have a new hypervisor cluster, so I would prefer to reinstall the 1st server, using these steps:
1) check if all roles (also the CA) is installed in srv02 You can find here some data about the VMs:
*[root@srv01 ~]# ipa server-show srv01.ipa.mydomain.com http://srv01.ipa.mydomain.com Server name: srv01.ipa.mydomain.com http://srv01.ipa.mydomain.com Managed suffixes: domain, ca Min domain level: 0 Max domain level: 1 Enabled server roles: CA server, IPA master, DNS server, NTP server, AD trust controller[root@srv02 ~]# ipa server-show srv02.ipa.mydomain.com http://srv02.ipa.mydomain.com Server name: srv02.ipa.mydomain.com http://srv02.ipa.mydomain.com Managed suffixes: domain, ca Min domain level: 0 Max domain level: 1 Enabled server roles: CA server, IPA master, DNS server, NTP server[root@srv01 ~]# ipa config-show Maximum username length: 32 Home directory base: /home Default shell: /bin/bash Default users group: ipausers Default e-mail domain: ipa.mydomain.com http://ipa.mydomain.com Search time limit: 2 Search size limit: 100 User search fields: uid,givenname,sn,telephonenumber,ou,title Group search fields: cn,description Enable migration mode: FALSE Certificate Subject base: O=IPA.MYDOMAIN.COM http://IPA.MYDOMAIN.COM Password Expiration Notification (days): 4 Password plugin features: AllowNThash SELinux user map order: guest_u:s0$xguest_u:s0$user_u:s0$staff_u:s0-s0:c0.c1023$unconfined_u:s0-s0:c0.c1023 Default SELinux user: unconfined_u:s0-s0:c0.c1023 Default PAC types: MS-PAC, nfs:NONE IPA masters: srv01.ipa.mydomain.com http://srv01.ipa.mydomain.com, srv02.ipa.mydomain.com http://srv02.ipa.mydomain.com IPA CA servers: srv01.ipa.mydomain.com http://srv01.ipa.mydomain.com, srv02.ipa.mydomain.com http://srv02.ipa.mydomain.com IPA NTP servers: srv01.ipa.mydomain.com http://srv01.ipa.mydomain.com, srv02.ipa.mydomain.com http://srv02.ipa.mydomain.com IPA CA renewal master: srv01.ipa.mydomain.com http://srv01.ipa.mydomain.com[root@srv02 ~]# ipa config-show Maximum username length: 32 Home directory base: /home Default shell: /bin/bash Default users group: ipausers Default e-mail domain: ipa.mydomain.com http://ipa.mydomain.com Search time limit: 2 Search size limit: 100 User search fields: uid,givenname,sn,telephonenumber,ou,title Group search fields: cn,description Enable migration mode: FALSE Certificate Subject base: O=IPA.MYDOMAIN.COM http://IPA.MYDOMAIN.COM Password Expiration Notification (days): 4 Password plugin features: AllowNThash SELinux user map order: guest_u:s0$xguest_u:s0$user_u:s0$staff_u:s0-s0:c0.c1023$unconfined_u:s0-s0:c0.c1023 Default SELinux user: unconfined_u:s0-s0:c0.c1023 Default PAC types: MS-PAC, nfs:NONE IPA masters: srv01.ipa.mydomain.com http://srv01.ipa.mydomain.com, srv02.ipa.mydomain.com http://srv02.ipa.mydomain.com IPA CA servers: srv01.ipa.mydomain.com http://srv01.ipa.mydomain.com, srv02.ipa.mydomain.com http://srv02.ipa.mydomain.com IPA NTP servers: srv01.ipa.mydomain.com http://srv01.ipa.mydomain.com, srv02.ipa.mydomain.com http://srv02.ipa.mydomain.com IPA CA renewal master: srv01.ipa.mydomain.com http://srv01.ipa.mydomain.com[root@srv01 ~]# ipactl statusDirectory Service: RUNNINGkrb5kdc Service: RUNNINGkadmin Service: RUNNINGnamed Service: RUNNINGhttpd Service: RUNNINGipa-custodia Service: RUNNINGntpd Service: RUNNINGpki-tomcatd Service: STOPPEDsmb Service: RUNNINGwinbind Service: RUNNINGipa-otpd Service: RUNNINGipa-dnskeysyncd Service: RUNNINGipa: INFO: The ipactl command was successful[root@srv02 ~]# ipactl statusDirectory Service: RUNNINGkrb5kdc Service: RUNNINGkadmin Service: RUNNINGnamed Service: RUNNINGhttpd Service: RUNNINGipa-custodia Service: RUNNINGntpd Service: RUNNINGpki-tomcatd Service: STOPPEDipa-otpd Service: RUNNINGipa-dnskeysyncd Service: RUNNINGipa: INFO: The ipactl command was successful[root@srv01 ~]# certutil -L -d /etc/pki/pki-tomcat/aliasCertificate Nickname Trust Attributes SSL,S/MIME,JAR/XPIServer-Cert cert-pki-ca u,u,usubsystemCert cert-pki-ca u,u,ucaSigningCert cert-pki-ca CTu,Cu,CuocspSigningCert cert-pki-ca u,u,uauditSigningCert cert-pki-ca u,u,Pu[root@srv02 ~]# certutil -L -d /etc/pki/pki-tomcat/aliasCertificate Nickname Trust Attributes
SSL,S/MIME,JAR/XPIServer-Cert cert-pki-ca u,u,usubsystemCert cert-pki-ca u,u,ucaSigningCert cert-pki-ca CTu,u,uocspSigningCert cert-pki-ca u,u,uauditSigningCert cert-pki-ca u,u,Pu*
It seems that AD trust controller role, IPA CA renewal master, smb and windbind are only in the 1st server. And also caSigningCert cert-pki-ca entry is different (CTu,Cu,Cu vs CTu,u,u).
I can see only in the 1st server these DNS records:
*_kerberos._tcp.Default-First-Site-Name._sites.dc._msdcs SRV 0 100 88 srv01_kerberos._tcp.dc._msdcs SRV 0 100 88 srv01_kerberos._udp.Default-First-Site-Name._sites.dc._msdcs SRV 0 100 88 srv01_kerberos._udp.dc._msdcs SRV 0 100 88 srv01_ldap._tcp.Default-First-Site-Name._sites.dc._msdcs 0 100 389 srv01_ldap._tcp.dc._msdcs 0 100 389 srv01*
Srv01 is the first master, I know, but is the server VM that has clock problems, in both situations. So I want to keep srv02 and install a new one.
What do I have to do to let the 2nd VM be a single server? Could I use these URLs?
*https://www.freeipa.org/page/Backup_and_Restore#One_Server_Loss_-_First_Mast... https://www.freeipa.org/page/Backup_and_Restore#One_Server_Loss_-_First_Masterhttps://www.freeipa.org/page/V4/Server_Roles#Upgrade https://www.freeipa.org/page/V4/Server_Roles#Upgrade*
2) uninstall ipa-server from the 1st server (srv01) and then powering off it, assuming that all data into the 2nd one are ok (srv02)
3) update freeipa and all other RPM packages into the VM srv02
4) install a new fresh VM, always with 7 release, and create a new replica Could I use the same old hostname (srv01) and IP address for this new VM? Or is better to use the same IP but a new name, like srv03?
Do you think this is the right way to solve my issue? Or do you have any better idea?
Please let me know, thanks. Bye, Morgan
Hi,
CSN generator time skew is a pending issue still under investigation.
At the moment the way your csn generator is messed up looks not fatal. You can allow replication to continue with the setting of nsslapd-ignore-time-skew on all servers. (https://access.redhat.com/solutions/1162703)
If it does not allow replication to continue there is a recovery procedure but I would recommend to first try ignore-time-skew (https://access.redhat.com/solutions/3543811)
NTP tuning or specific VMs are suspected to contribute to time skew. What type of VMs are you using (local or cloud (AWS)) ?
best regards thierry
On 4/21/20 5:42 PM, Morgan Marodin via FreeIPA-users wrote:
Hi.
Into my environment I have two IPA server, replicating each other. They are both 7.6 OS systems, ipa-server RPM version is 4.6.4-10.0.1.el7_6.2.x86_64.
The first server installed was srv01 (many years ago), then I installed the replica into srv02 (like a year later the 1st node). When I had a single server I did also a trust with my corporate Active Directory. VMs are running in 2 different hypervisor clusters.
Now the replication doesn't works. Into log files I have this error: /[16/Apr/2020:12:25:36.856632697 +0200] - ERR - csngen_adjust_time - Adjustment limit exceeded; value - 23221226, limit - 86400 [16/Apr/2020:12:25:36.857909222 +0200] - ERR - NSMMReplicationPlugin - repl5_inc_run - agmt="cn=meTosrv01.ipa.mydomain.com http://meTosrv01.ipa.mydomain.com" (srv01:389): Fatal error - too much time skew between replicas! [16/Apr/2020:12:25:36.862233147 +0200] - ERR - NSMMReplicationPlugin - repl5_inc_run - agmt="cn=meTosrv01.ipa.mydomain.com http://meTosrv01.ipa.mydomain.com" (srv01:389): Incremental update failed and requires administrator action/
I tried to force the replica, but the limit exceeded problem doesn't allow the sync. I know that the problem is that CSN generator has become grossly skewed. Using the external script readNsState.py I found that there was as offset time for about a month, so ... I waited for a month and then the issue disappeared. But now the offset is about 9 months ... I can't wait so much time :)
/[root@srv01 scripts]# ./readNsState.py /etc/dirsrv/slapd-IPA-MYDOMAIN-COM/dse.ldif nsState is BAAAAAAAAACCN/xfAAAAAHbiBAAAAAAABCgAAAAAAAANdQAAAAAAAA== Little Endian For replica cn=replica,cn=dc\3Dipa\2Cdc\3Dmydomain\2Cdc\3Dcom,cn=mapping tree,cn=con fmtstr=[H6x3QH6x] size=40 len of nsstate is 40 CSN generator state: Replica ID : 4 Sampled Time : 1610364802 Gen as csn : 5ffc37822996500040000 *Time as str : Mon Jan 11 12:33:22 2021* Local Offset : 320118 Remote Offset : 10244 Seq. num : 29965 System time : Tue Apr 21 15:03:45 2020 Diff in sec. : -22890577 Day:sec diff : -265:5423
nsState is YAAAAAAAAAADLZheAAAAAAAAAAAAAAAAXSgAAAAAAAATAAAAAAAAAA== Little Endian For replica cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config fmtstr=[H6x3QH6x] size=40 len of nsstate is 40 CSN generator state: Replica ID : 96 Sampled Time : 1587031299 Gen as csn : 5e982d03001900960000 Time as str : Thu Apr 16 12:01:39 2020 Local Offset : 0 Remote Offset : 10333 Seq. num : 19 System time : Tue Apr 21 15:03:45 2020 Diff in sec. : 442926 Day:sec diff : 5:10926
[root@srv02 scripts]# ./readNsState.py /etc/dirsrv/slapd-IPA-MYDOMAIN-COM/dse.ldif nsState is AwAAAAAAAABU7p5eAAAAAAAAAAAAAAAAsVNiAQAAAAAAAAAAAAAAAA== Little Endian For replica cn=replica,cn=dc\3Dipa\2Cdc\3Dmydomain\2Cdc\3Dcom,cn=mapping tree,cn=con fmtstr=[H6x3QH6x] size=40 len of nsstate is 40 CSN generator state: Replica ID : 3 Sampled Time : 1587474004 Gen as csn : 5e9eee54000000030000 Time as str : Tue Apr 21 15:00:04 2020 Local Offset : 0 Remote Offset : 23221169 Seq. num : 0 System time : Tue Apr 21 15:02:38 2020 Diff in sec. : 154 Day:sec diff : 0:154
nsState is YQAAAAAAAAAuLZheAAAAAEUBAAAAAAAA7SYAAAAAAAASAAAAAAAAAA== Little Endian For replica cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config fmtstr=[H6x3QH6x] size=40 len of nsstate is 40 CSN generator state: Replica ID : 97 Sampled Time : 1587031342 Gen as csn : 5e982d2e001800970000 Time as str : Thu Apr 16 12:02:22 2020 Local Offset : 325 Remote Offset : 9965 Seq. num : 18 System time : Tue Apr 21 15:02:38 2020 Diff in sec. : 442816 Day:sec diff : 5:10816/
As you can see in the 1st node the Time as str is Jan 11 of 2021. With timedatectl command I see that both VMs use the same Time zone and the clock is correct.
I found this old article to fix my issue: /https://www.redhat.com/archives/freeipa-users/2014-February/msg00007.html/
But ... I had the same issue in the past, always in the 1st server. So, in my mind I don't want to try to use that fix. I have a new hypervisor cluster, so I would prefer to reinstall the 1st server, using these steps:
- check if all roles (also the CA) is installed in srv02
You can find here some data about the VMs:
/[root@srv01 ~]# ipa server-show srv01.ipa.mydomain.com http://srv01.ipa.mydomain.com Server name: srv01.ipa.mydomain.com http://srv01.ipa.mydomain.com Managed suffixes: domain, ca Min domain level: 0 Max domain level: 1 Enabled server roles: CA server, IPA master, DNS server, NTP server, *AD trust controller*
[root@srv02 ~]# ipa server-show srv02.ipa.mydomain.com http://srv02.ipa.mydomain.com Server name: srv02.ipa.mydomain.com http://srv02.ipa.mydomain.com Managed suffixes: domain, ca Min domain level: 0 Max domain level: 1 Enabled server roles: CA server, IPA master, DNS server, NTP server
[root@srv01 ~]# ipa config-show Maximum username length: 32 Home directory base: /home Default shell: /bin/bash Default users group: ipausers Default e-mail domain: ipa.mydomain.com http://ipa.mydomain.com Search time limit: 2 Search size limit: 100 User search fields: uid,givenname,sn,telephonenumber,ou,title Group search fields: cn,description Enable migration mode: FALSE Certificate Subject base: O=IPA.MYDOMAIN.COM http://IPA.MYDOMAIN.COM Password Expiration Notification (days): 4 Password plugin features: AllowNThash SELinux user map order: guest_u:s0$xguest_u:s0$user_u:s0$staff_u:s0-s0:c0.c1023$unconfined_u:s0-s0:c0.c1023 Default SELinux user: unconfined_u:s0-s0:c0.c1023 Default PAC types: MS-PAC, nfs:NONE IPA masters: srv01.ipa.mydomain.com http://srv01.ipa.mydomain.com, srv02.ipa.mydomain.com http://srv02.ipa.mydomain.com IPA CA servers: srv01.ipa.mydomain.com http://srv01.ipa.mydomain.com, srv02.ipa.mydomain.com http://srv02.ipa.mydomain.com IPA NTP servers: srv01.ipa.mydomain.com http://srv01.ipa.mydomain.com, srv02.ipa.mydomain.com http://srv02.ipa.mydomain.com IPA CA renewal master: srv01.ipa.mydomain.com http://srv01.ipa.mydomain.com
[root@srv02 ~]# ipa config-show Maximum username length: 32 Home directory base: /home Default shell: /bin/bash Default users group: ipausers Default e-mail domain: ipa.mydomain.com http://ipa.mydomain.com Search time limit: 2 Search size limit: 100 User search fields: uid,givenname,sn,telephonenumber,ou,title Group search fields: cn,description Enable migration mode: FALSE Certificate Subject base: O=IPA.MYDOMAIN.COM http://IPA.MYDOMAIN.COM Password Expiration Notification (days): 4 Password plugin features: AllowNThash SELinux user map order: guest_u:s0$xguest_u:s0$user_u:s0$staff_u:s0-s0:c0.c1023$unconfined_u:s0-s0:c0.c1023 Default SELinux user: unconfined_u:s0-s0:c0.c1023 Default PAC types: MS-PAC, nfs:NONE IPA masters: srv01.ipa.mydomain.com http://srv01.ipa.mydomain.com, srv02.ipa.mydomain.com http://srv02.ipa.mydomain.com IPA CA servers: srv01.ipa.mydomain.com http://srv01.ipa.mydomain.com, srv02.ipa.mydomain.com http://srv02.ipa.mydomain.com IPA NTP servers: srv01.ipa.mydomain.com http://srv01.ipa.mydomain.com, srv02.ipa.mydomain.com http://srv02.ipa.mydomain.com *IPA CA renewal master: srv01.ipa.mydomain.com http://srv01.ipa.mydomain.com*
[root@srv01 ~]# ipactl status Directory Service: RUNNING krb5kdc Service: RUNNING kadmin Service: RUNNING named Service: RUNNING httpd Service: RUNNING ipa-custodia Service: RUNNING ntpd Service: RUNNING pki-tomcatd Service: STOPPED *smb Service: RUNNING winbind Service: RUNNING* ipa-otpd Service: RUNNING ipa-dnskeysyncd Service: RUNNING ipa: INFO: The ipactl command was successful
[root@srv02 ~]# ipactl status Directory Service: RUNNING krb5kdc Service: RUNNING kadmin Service: RUNNING named Service: RUNNING httpd Service: RUNNING ipa-custodia Service: RUNNING ntpd Service: RUNNING pki-tomcatd Service: STOPPED ipa-otpd Service: RUNNING ipa-dnskeysyncd Service: RUNNING ipa: INFO: The ipactl command was successful
[root@srv01 ~]# certutil -L -d /etc/pki/pki-tomcat/alias Certificate Nickname Trust Attributes SSL,S/MIME,JAR/XPI Server-Cert cert-pki-ca u,u,u subsystemCert cert-pki-ca u,u,u caSigningCert cert-pki-ca *CTu,Cu,Cu* ocspSigningCert cert-pki-ca u,u,u auditSigningCert cert-pki-ca u,u,Pu
[root@srv02 ~]# certutil -L -d /etc/pki/pki-tomcat/alias Certificate Nickname Trust Attributes SSL,S/MIME,JAR/XPI Server-Cert cert-pki-ca u,u,u subsystemCert cert-pki-ca u,u,u caSigningCert cert-pki-ca *CTu,u,u* ocspSigningCert cert-pki-ca u,u,u auditSigningCert cert-pki-ca u,u,Pu/
It seems that AD trust controller role, IPA CA renewal master, smb and windbind are only in the 1st server. And also caSigningCert cert-pki-ca entry is different (CTu,Cu,Cu vs CTu,u,u).
I can see only in the 1st server these DNS records: /_kerberos._tcp.Default-First-Site-Name._sites.dc._msdcs SRV 0 100 88 srv01 _kerberos._tcp.dc._msdcs SRV 0 100 88 srv01 _kerberos._udp.Default-First-Site-Name._sites.dc._msdcs SRV 0 100 88 srv01 _kerberos._udp.dc._msdcs SRV 0 100 88 srv01 _ldap._tcp.Default-First-Site-Name._sites.dc._msdcs 0 100 389 srv01 _ldap._tcp.dc._msdcs 0 100 389 srv01/
Srv01 is the first master, I know, but is the server VM that has clock problems, in both situations. So I want to keep srv02 and install a new one.
What do I have to do to let the 2nd VM be a single server? Could I use these URLs? /https://www.freeipa.org/page/Backup_and_Restore#One_Server_Loss_-_First_Mast... https://www.freeipa.org/page/V4/Server_Roles#Upgrade/
- uninstall ipa-server from the 1st server (srv01) and then powering
off it, assuming that all data into the 2nd one are ok (srv02)
update freeipa and all other RPM packages into the VM srv02
install a new fresh VM, always with 7 release, and create a new replica
Could I use the same old hostname (srv01) and IP address for this new VM? Or is better to use the same IP but a new name, like srv03?
Do you think this is the right way to solve my issue? Or do you have any better idea?
Please let me know, thanks. Bye, Morgan
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
Hi.
I don't have access to RedHat portal :( There are similar articles in a public forum?
Anyway ... could I stop ipa-server, change the value of *nsslapd-ignore-time-skew* into */etc/dirsrv/slapd-IPA-MYDOMAIN-COM/dse.ldif* and start again the server? Or is more complicated to change the configuration?
VMs are local, but the cluster where the 1st server is running is affected by NTP problems ... For this reason I want to remove the First Master and install another replica in the new cluster.
Thanks, bye. Morgan
Il giorno mer 22 apr 2020 alle ore 11:33 thierry bordaz via FreeIPA-users < freeipa-users@lists.fedorahosted.org> ha scritto:
Hi,
CSN generator time skew is a pending issue still under investigation.
At the moment the way your csn generator is messed up looks not fatal. You can allow replication to continue with the setting of nsslapd-ignore-time-skew on all servers. ( https://access.redhat.com/solutions/1162703)
If it does not allow replication to continue there is a recovery procedure but I would recommend to first try ignore-time-skew ( https://access.redhat.com/solutions/3543811)
NTP tuning or specific VMs are suspected to contribute to time skew. What type of VMs are you using (local or cloud (AWS)) ?
best regards thierry
On 4/21/20 5:42 PM, Morgan Marodin via FreeIPA-users wrote:
Hi.
Into my environment I have two IPA server, replicating each other. They are both 7.6 OS systems, ipa-server RPM version is 4.6.4-10.0.1.el7_6.2.x86_64.
The first server installed was srv01 (many years ago), then I installed the replica into srv02 (like a year later the 1st node). When I had a single server I did also a trust with my corporate Active Directory. VMs are running in 2 different hypervisor clusters.
Now the replication doesn't works. Into log files I have this error:
*[16/Apr/2020:12:25:36.856632697 +0200] - ERR - csngen_adjust_time - Adjustment limit exceeded; value - 23221226, limit - 86400 [16/Apr/2020:12:25:36.857909222 +0200] - ERR - NSMMReplicationPlugin - repl5_inc_run - agmt="cn=meTosrv01.ipa.mydomain.com http://meTosrv01.ipa.mydomain.com" (srv01:389): Fatal error - too much time skew between replicas! [16/Apr/2020:12:25:36.862233147 +0200] - ERR - NSMMReplicationPlugin - repl5_inc_run - agmt="cn=meTosrv01.ipa.mydomain.com http://meTosrv01.ipa.mydomain.com" (srv01:389): Incremental update failed and requires administrator action*
I tried to force the replica, but the limit exceeded problem doesn't allow the sync. I know that the problem is that CSN generator has become grossly skewed. Using the external script readNsState.py I found that there was as offset time for about a month, so ... I waited for a month and then the issue disappeared. But now the offset is about 9 months ... I can't wait so much time :)
*[root@srv01 scripts]# ./readNsState.py /etc/dirsrv/slapd-IPA-MYDOMAIN-COM/dse.ldif nsState is BAAAAAAAAACCN/xfAAAAAHbiBAAAAAAABCgAAAAAAAANdQAAAAAAAA== Little Endian For replica cn=replica,cn=dc\3Dipa\2Cdc\3Dmydomain\2Cdc\3Dcom,cn=mapping tree,cn=con fmtstr=[H6x3QH6x] size=40 len of nsstate is 40 CSN generator state: Replica ID : 4 Sampled Time : 1610364802 Gen as csn : 5ffc37822996500040000 Time as str : Mon Jan 11 12:33:22 2021 Local Offset : 320118 Remote Offset : 10244 Seq. num : 29965 System time : Tue Apr 21 15:03:45 2020 Diff in sec. : -22890577 Day:sec diff : -265:5423 nsState is YAAAAAAAAAADLZheAAAAAAAAAAAAAAAAXSgAAAAAAAATAAAAAAAAAA== Little Endian For replica cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config fmtstr=[H6x3QH6x] size=40 len of nsstate is 40 CSN generator state: Replica ID : 96 Sampled Time : 1587031299 Gen as csn : 5e982d03001900960000 Time as str : Thu Apr 16 12:01:39 2020 Local Offset : 0 Remote Offset : 10333 Seq. num : 19 System time : Tue Apr 21 15:03:45 2020 Diff in sec. : 442926 Day:sec diff : 5:10926 [root@srv02 scripts]# ./readNsState.py /etc/dirsrv/slapd-IPA-MYDOMAIN-COM/dse.ldif nsState is AwAAAAAAAABU7p5eAAAAAAAAAAAAAAAAsVNiAQAAAAAAAAAAAAAAAA== Little Endian For replica cn=replica,cn=dc\3Dipa\2Cdc\3Dmydomain\2Cdc\3Dcom,cn=mapping tree,cn=con fmtstr=[H6x3QH6x] size=40 len of nsstate is 40 CSN generator state: Replica ID : 3 Sampled Time : 1587474004 Gen as csn : 5e9eee54000000030000 Time as str : Tue Apr 21 15:00:04 2020 Local Offset : 0 Remote Offset : 23221169 Seq. num : 0 System time : Tue Apr 21 15:02:38 2020 Diff in sec. : 154 Day:sec diff : 0:154 nsState is YQAAAAAAAAAuLZheAAAAAEUBAAAAAAAA7SYAAAAAAAASAAAAAAAAAA== Little Endian For replica cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config fmtstr=[H6x3QH6x] size=40 len of nsstate is 40 CSN generator state: Replica ID : 97 Sampled Time : 1587031342 Gen as csn : 5e982d2e001800970000 Time as str : Thu Apr 16 12:02:22 2020 Local Offset : 325 Remote Offset : 9965 Seq. num : 18 System time : Tue Apr 21 15:02:38 2020 Diff in sec. : 442816 Day:sec diff : 5:10816*
As you can see in the 1st node the Time as str is Jan 11 of 2021. With timedatectl command I see that both VMs use the same Time zone and the clock is correct.
I found this old article to fix my issue: *https://www.redhat.com/archives/freeipa-users/2014-February/msg00007.html https://www.redhat.com/archives/freeipa-users/2014-February/msg00007.html*
But ... I had the same issue in the past, always in the 1st server. So, in my mind I don't want to try to use that fix. I have a new hypervisor cluster, so I would prefer to reinstall the 1st server, using these steps:
- check if all roles (also the CA) is installed in srv02
You can find here some data about the VMs:
*[root@srv01 ~]# ipa server-show srv01.ipa.mydomain.com http://srv01.ipa.mydomain.com Server name: srv01.ipa.mydomain.com http://srv01.ipa.mydomain.com Managed suffixes: domain, ca Min domain level: 0 Max domain level: 1 Enabled server roles: CA server, IPA master, DNS server, NTP server, AD trust controller [root@srv02 ~]# ipa server-show srv02.ipa.mydomain.com http://srv02.ipa.mydomain.com Server name: srv02.ipa.mydomain.com http://srv02.ipa.mydomain.com Managed suffixes: domain, ca Min domain level: 0 Max domain level: 1 Enabled server roles: CA server, IPA master, DNS server, NTP server [root@srv01 ~]# ipa config-show Maximum username length: 32 Home directory base: /home Default shell: /bin/bash Default users group: ipausers Default e-mail domain: ipa.mydomain.com http://ipa.mydomain.com Search time limit: 2 Search size limit: 100 User search fields: uid,givenname,sn,telephonenumber,ou,title Group search fields: cn,description Enable migration mode: FALSE Certificate Subject base: O=IPA.MYDOMAIN.COM http://IPA.MYDOMAIN.COM Password Expiration Notification (days): 4 Password plugin features: AllowNThash SELinux user map order: guest_u:s0$xguest_u:s0$user_u:s0$staff_u:s0-s0:c0.c1023$unconfined_u:s0-s0:c0.c1023 Default SELinux user: unconfined_u:s0-s0:c0.c1023 Default PAC types: MS-PAC, nfs:NONE IPA masters: srv01.ipa.mydomain.com http://srv01.ipa.mydomain.com, srv02.ipa.mydomain.com http://srv02.ipa.mydomain.com IPA CA servers: srv01.ipa.mydomain.com http://srv01.ipa.mydomain.com, srv02.ipa.mydomain.com http://srv02.ipa.mydomain.com IPA NTP servers: srv01.ipa.mydomain.com http://srv01.ipa.mydomain.com, srv02.ipa.mydomain.com http://srv02.ipa.mydomain.com IPA CA renewal master: srv01.ipa.mydomain.com http://srv01.ipa.mydomain.com [root@srv02 ~]# ipa config-show Maximum username length: 32 Home directory base: /home Default shell: /bin/bash Default users group: ipausers Default e-mail domain: ipa.mydomain.com http://ipa.mydomain.com Search time limit: 2 Search size limit: 100 User search fields: uid,givenname,sn,telephonenumber,ou,title Group search fields: cn,description Enable migration mode: FALSE Certificate Subject base: O=IPA.MYDOMAIN.COM http://IPA.MYDOMAIN.COM Password Expiration Notification (days): 4 Password plugin features: AllowNThash SELinux user map order: guest_u:s0$xguest_u:s0$user_u:s0$staff_u:s0-s0:c0.c1023$unconfined_u:s0-s0:c0.c1023 Default SELinux user: unconfined_u:s0-s0:c0.c1023 Default PAC types: MS-PAC, nfs:NONE IPA masters: srv01.ipa.mydomain.com http://srv01.ipa.mydomain.com, srv02.ipa.mydomain.com http://srv02.ipa.mydomain.com IPA CA servers: srv01.ipa.mydomain.com http://srv01.ipa.mydomain.com, srv02.ipa.mydomain.com http://srv02.ipa.mydomain.com IPA NTP servers: srv01.ipa.mydomain.com http://srv01.ipa.mydomain.com, srv02.ipa.mydomain.com http://srv02.ipa.mydomain.com IPA CA renewal master: srv01.ipa.mydomain.com http://srv01.ipa.mydomain.com [root@srv01 ~]# ipactl status Directory Service: RUNNING krb5kdc Service: RUNNING kadmin Service: RUNNING named Service: RUNNING httpd Service: RUNNING ipa-custodia Service: RUNNING ntpd Service: RUNNING pki-tomcatd Service: STOPPED smb Service: RUNNING winbind Service: RUNNING ipa-otpd Service: RUNNING ipa-dnskeysyncd Service: RUNNING ipa: INFO: The ipactl command was successful [root@srv02 ~]# ipactl status Directory Service: RUNNING krb5kdc Service: RUNNING kadmin Service: RUNNING named Service: RUNNING httpd Service: RUNNING ipa-custodia Service: RUNNING ntpd Service: RUNNING pki-tomcatd Service: STOPPED ipa-otpd Service: RUNNING ipa-dnskeysyncd Service: RUNNING ipa: INFO: The ipactl command was successful [root@srv01 ~]# certutil -L -d /etc/pki/pki-tomcat/alias Certificate Nickname Trust Attributes SSL,S/MIME,JAR/XPI Server-Cert cert-pki-ca u,u,u subsystemCert cert-pki-ca u,u,u caSigningCert cert-pki-ca CTu,Cu,Cu ocspSigningCert cert-pki-ca u,u,u auditSigningCert cert-pki-ca u,u,Pu [root@srv02 ~]# certutil -L -d /etc/pki/pki-tomcat/alias Certificate Nickname Trust Attributes SSL,S/MIME,JAR/XPI Server-Cert cert-pki-ca u,u,u subsystemCert cert-pki-ca u,u,u caSigningCert cert-pki-ca CTu,u,u ocspSigningCert cert-pki-ca u,u,u auditSigningCert cert-pki-ca u,u,Pu*
It seems that AD trust controller role, IPA CA renewal master, smb and windbind are only in the 1st server. And also caSigningCert cert-pki-ca entry is different (CTu,Cu,Cu vs CTu,u,u).
I can see only in the 1st server these DNS records:
*_kerberos._tcp.Default-First-Site-Name._sites.dc._msdcs SRV 0 100 88 srv01 _kerberos._tcp.dc._msdcs SRV 0 100 88 srv01 _kerberos._udp.Default-First-Site-Name._sites.dc._msdcs SRV 0 100 88 srv01 _kerberos._udp.dc._msdcs SRV 0 100 88 srv01 _ldap._tcp.Default-First-Site-Name._sites.dc._msdcs 0 100 389 srv01 _ldap._tcp.dc._msdcs 0 100 389 srv01*
Srv01 is the first master, I know, but is the server VM that has clock problems, in both situations. So I want to keep srv02 and install a new one.
What do I have to do to let the 2nd VM be a single server? Could I use these URLs?
*https://www.freeipa.org/page/Backup_and_Restore#One_Server_Loss_-_First_Mast... https://www.freeipa.org/page/Backup_and_Restore#One_Server_Loss_-_First_Master https://www.freeipa.org/page/V4/Server_Roles#Upgrade https://www.freeipa.org/page/V4/Server_Roles#Upgrade*
- uninstall ipa-server from the 1st server (srv01) and then powering off
it, assuming that all data into the 2nd one are ok (srv02)
update freeipa and all other RPM packages into the VM srv02
install a new fresh VM, always with 7 release, and create a new replica
Could I use the same old hostname (srv01) and IP address for this new VM? Or is better to use the same IP but a new name, like srv03?
Do you think this is the right way to solve my issue? Or do you have any better idea?
Please let me know, thanks. Bye, Morgan
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
Hi Morgan,
Sure. The most immediate and safest action is to do
|dn: cn=config changetype: modify replace: nsslapd-ignore-time-skew nsslapd-ignore-time-skew: on |
On all servers in the topology (no need to restart). Then monitor if replication is catching up. Okay NTP issues is likely the RC of your time skew but there is not easy way to prove it if any.
best regards theirry
On 4/22/20 3:16 PM, Morgan Marodin via FreeIPA-users wrote:
Hi.
I don't have access to RedHat portal :( There are similar articles in a public forum?
Anyway ... could I stop ipa-server, change the value of /nsslapd-ignore-time-skew/ into //etc/dirsrv/slapd-IPA-MYDOMAIN-COM/dse.ldif/ and start again the server? Or is more complicated to change the configuration?
VMs are local, but the cluster where the 1st server is running is affected by NTP problems ... For this reason I want to remove the First Master and install another replica in the new cluster.
Thanks, bye. Morgan
Il giorno mer 22 apr 2020 alle ore 11:33 thierry bordaz via FreeIPA-users <freeipa-users@lists.fedorahosted.org mailto:freeipa-users@lists.fedorahosted.org> ha scritto:
Hi, CSN generator time skew is a pending issue still under investigation. At the moment the way your csn generator is messed up looks not fatal. You can allow replication to continue with the setting of nsslapd-ignore-time-skew on all servers. (https://access.redhat.com/solutions/1162703) If it does not allow replication to continue there is a recovery procedure but I would recommend to first try ignore-time-skew (https://access.redhat.com/solutions/3543811) NTP tuning or specific VMs are suspected to contribute to time skew. What type of VMs are you using (local or cloud (AWS)) ? best regards thierry On 4/21/20 5:42 PM, Morgan Marodin via FreeIPA-users wrote:Hi. Into my environment I have two IPA server, replicating each other. They are both 7.6 OS systems, ipa-server RPM version is 4.6.4-10.0.1.el7_6.2.x86_64. The first server installed was srv01 (many years ago), then I installed the replica into srv02 (like a year later the 1st node). When I had a single server I did also a trust with my corporate Active Directory. VMs are running in 2 different hypervisor clusters. Now the replication doesn't works. Into log files I have this error: /[16/Apr/2020:12:25:36.856632697 +0200] - ERR - csngen_adjust_time - Adjustment limit exceeded; value - 23221226, limit - 86400 [16/Apr/2020:12:25:36.857909222 +0200] - ERR - NSMMReplicationPlugin - repl5_inc_run - agmt="cn=meTosrv01.ipa.mydomain.com <http://meTosrv01.ipa.mydomain.com>" (srv01:389): Fatal error - too much time skew between replicas! [16/Apr/2020:12:25:36.862233147 +0200] - ERR - NSMMReplicationPlugin - repl5_inc_run - agmt="cn=meTosrv01.ipa.mydomain.com <http://meTosrv01.ipa.mydomain.com>" (srv01:389): Incremental update failed and requires administrator action/ I tried to force the replica, but the limit exceeded problem doesn't allow the sync. I know that the problem is that CSN generator has become grossly skewed. Using the external script readNsState.py I found that there was as offset time for about a month, so ... I waited for a month and then the issue disappeared. But now the offset is about 9 months ... I can't wait so much time :) /[root@srv01 scripts]# ./readNsState.py /etc/dirsrv/slapd-IPA-MYDOMAIN-COM/dse.ldif nsState is BAAAAAAAAACCN/xfAAAAAHbiBAAAAAAABCgAAAAAAAANdQAAAAAAAA== Little Endian For replica cn=replica,cn=dc\3Dipa\2Cdc\3Dmydomain\2Cdc\3Dcom,cn=mapping tree,cn=con fmtstr=[H6x3QH6x] size=40 len of nsstate is 40 CSN generator state: Replica ID : 4 Sampled Time : 1610364802 Gen as csn : 5ffc37822996500040000 *Time as str : Mon Jan 11 12:33:22 2021* Local Offset : 320118 Remote Offset : 10244 Seq. num : 29965 System time : Tue Apr 21 15:03:45 2020 Diff in sec. : -22890577 Day:sec diff : -265:5423 nsState is YAAAAAAAAAADLZheAAAAAAAAAAAAAAAAXSgAAAAAAAATAAAAAAAAAA== Little Endian For replica cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config fmtstr=[H6x3QH6x] size=40 len of nsstate is 40 CSN generator state: Replica ID : 96 Sampled Time : 1587031299 Gen as csn : 5e982d03001900960000 Time as str : Thu Apr 16 12:01:39 2020 Local Offset : 0 Remote Offset : 10333 Seq. num : 19 System time : Tue Apr 21 15:03:45 2020 Diff in sec. : 442926 Day:sec diff : 5:10926 [root@srv02 scripts]# ./readNsState.py /etc/dirsrv/slapd-IPA-MYDOMAIN-COM/dse.ldif nsState is AwAAAAAAAABU7p5eAAAAAAAAAAAAAAAAsVNiAQAAAAAAAAAAAAAAAA== Little Endian For replica cn=replica,cn=dc\3Dipa\2Cdc\3Dmydomain\2Cdc\3Dcom,cn=mapping tree,cn=con fmtstr=[H6x3QH6x] size=40 len of nsstate is 40 CSN generator state: Replica ID : 3 Sampled Time : 1587474004 Gen as csn : 5e9eee54000000030000 Time as str : Tue Apr 21 15:00:04 2020 Local Offset : 0 Remote Offset : 23221169 Seq. num : 0 System time : Tue Apr 21 15:02:38 2020 Diff in sec. : 154 Day:sec diff : 0:154 nsState is YQAAAAAAAAAuLZheAAAAAEUBAAAAAAAA7SYAAAAAAAASAAAAAAAAAA== Little Endian For replica cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config fmtstr=[H6x3QH6x] size=40 len of nsstate is 40 CSN generator state: Replica ID : 97 Sampled Time : 1587031342 Gen as csn : 5e982d2e001800970000 Time as str : Thu Apr 16 12:02:22 2020 Local Offset : 325 Remote Offset : 9965 Seq. num : 18 System time : Tue Apr 21 15:02:38 2020 Diff in sec. : 442816 Day:sec diff : 5:10816/ As you can see in the 1st node the Time as str is Jan 11 of 2021. With timedatectl command I see that both VMs use the same Time zone and the clock is correct. I found this old article to fix my issue: /https://www.redhat.com/archives/freeipa-users/2014-February/msg00007.html/ But ... I had the same issue in the past, always in the 1st server. So, in my mind I don't want to try to use that fix. I have a new hypervisor cluster, so I would prefer to reinstall the 1st server, using these steps: 1) check if all roles (also the CA) is installed in srv02 You can find here some data about the VMs: /[root@srv01 ~]# ipa server-show srv01.ipa.mydomain.com <http://srv01.ipa.mydomain.com> Server name: srv01.ipa.mydomain.com <http://srv01.ipa.mydomain.com> Managed suffixes: domain, ca Min domain level: 0 Max domain level: 1 Enabled server roles: CA server, IPA master, DNS server, NTP server, *AD trust controller* [root@srv02 ~]# ipa server-show srv02.ipa.mydomain.com <http://srv02.ipa.mydomain.com> Server name: srv02.ipa.mydomain.com <http://srv02.ipa.mydomain.com> Managed suffixes: domain, ca Min domain level: 0 Max domain level: 1 Enabled server roles: CA server, IPA master, DNS server, NTP server [root@srv01 ~]# ipa config-show Maximum username length: 32 Home directory base: /home Default shell: /bin/bash Default users group: ipausers Default e-mail domain: ipa.mydomain.com <http://ipa.mydomain.com> Search time limit: 2 Search size limit: 100 User search fields: uid,givenname,sn,telephonenumber,ou,title Group search fields: cn,description Enable migration mode: FALSE Certificate Subject base: O=IPA.MYDOMAIN.COM <http://IPA.MYDOMAIN.COM> Password Expiration Notification (days): 4 Password plugin features: AllowNThash SELinux user map order: guest_u:s0$xguest_u:s0$user_u:s0$staff_u:s0-s0:c0.c1023$unconfined_u:s0-s0:c0.c1023 Default SELinux user: unconfined_u:s0-s0:c0.c1023 Default PAC types: MS-PAC, nfs:NONE IPA masters: srv01.ipa.mydomain.com <http://srv01.ipa.mydomain.com>, srv02.ipa.mydomain.com <http://srv02.ipa.mydomain.com> IPA CA servers: srv01.ipa.mydomain.com <http://srv01.ipa.mydomain.com>, srv02.ipa.mydomain.com <http://srv02.ipa.mydomain.com> IPA NTP servers: srv01.ipa.mydomain.com <http://srv01.ipa.mydomain.com>, srv02.ipa.mydomain.com <http://srv02.ipa.mydomain.com> IPA CA renewal master: srv01.ipa.mydomain.com <http://srv01.ipa.mydomain.com> [root@srv02 ~]# ipa config-show Maximum username length: 32 Home directory base: /home Default shell: /bin/bash Default users group: ipausers Default e-mail domain: ipa.mydomain.com <http://ipa.mydomain.com> Search time limit: 2 Search size limit: 100 User search fields: uid,givenname,sn,telephonenumber,ou,title Group search fields: cn,description Enable migration mode: FALSE Certificate Subject base: O=IPA.MYDOMAIN.COM <http://IPA.MYDOMAIN.COM> Password Expiration Notification (days): 4 Password plugin features: AllowNThash SELinux user map order: guest_u:s0$xguest_u:s0$user_u:s0$staff_u:s0-s0:c0.c1023$unconfined_u:s0-s0:c0.c1023 Default SELinux user: unconfined_u:s0-s0:c0.c1023 Default PAC types: MS-PAC, nfs:NONE IPA masters: srv01.ipa.mydomain.com <http://srv01.ipa.mydomain.com>, srv02.ipa.mydomain.com <http://srv02.ipa.mydomain.com> IPA CA servers: srv01.ipa.mydomain.com <http://srv01.ipa.mydomain.com>, srv02.ipa.mydomain.com <http://srv02.ipa.mydomain.com> IPA NTP servers: srv01.ipa.mydomain.com <http://srv01.ipa.mydomain.com>, srv02.ipa.mydomain.com <http://srv02.ipa.mydomain.com> *IPA CA renewal master: srv01.ipa.mydomain.com <http://srv01.ipa.mydomain.com>* [root@srv01 ~]# ipactl status Directory Service: RUNNING krb5kdc Service: RUNNING kadmin Service: RUNNING named Service: RUNNING httpd Service: RUNNING ipa-custodia Service: RUNNING ntpd Service: RUNNING pki-tomcatd Service: STOPPED *smb Service: RUNNING winbind Service: RUNNING* ipa-otpd Service: RUNNING ipa-dnskeysyncd Service: RUNNING ipa: INFO: The ipactl command was successful [root@srv02 ~]# ipactl status Directory Service: RUNNING krb5kdc Service: RUNNING kadmin Service: RUNNING named Service: RUNNING httpd Service: RUNNING ipa-custodia Service: RUNNING ntpd Service: RUNNING pki-tomcatd Service: STOPPED ipa-otpd Service: RUNNING ipa-dnskeysyncd Service: RUNNING ipa: INFO: The ipactl command was successful [root@srv01 ~]# certutil -L -d /etc/pki/pki-tomcat/alias Certificate Nickname Trust Attributes SSL,S/MIME,JAR/XPI Server-Cert cert-pki-ca u,u,u subsystemCert cert-pki-ca u,u,u caSigningCert cert-pki-ca *CTu,Cu,Cu* ocspSigningCert cert-pki-ca u,u,u auditSigningCert cert-pki-ca u,u,Pu [root@srv02 ~]# certutil -L -d /etc/pki/pki-tomcat/alias Certificate Nickname Trust Attributes SSL,S/MIME,JAR/XPI Server-Cert cert-pki-ca u,u,u subsystemCert cert-pki-ca u,u,u caSigningCert cert-pki-ca *CTu,u,u* ocspSigningCert cert-pki-ca u,u,u auditSigningCert cert-pki-ca u,u,Pu/ It seems that AD trust controller role, IPA CA renewal master, smb and windbind are only in the 1st server. And also caSigningCert cert-pki-ca entry is different (CTu,Cu,Cu vs CTu,u,u). I can see only in the 1st server these DNS records: /_kerberos._tcp.Default-First-Site-Name._sites.dc._msdcs SRV 0 100 88 srv01 _kerberos._tcp.dc._msdcs SRV 0 100 88 srv01 _kerberos._udp.Default-First-Site-Name._sites.dc._msdcs SRV 0 100 88 srv01 _kerberos._udp.dc._msdcs SRV 0 100 88 srv01 _ldap._tcp.Default-First-Site-Name._sites.dc._msdcs 0 100 389 srv01 _ldap._tcp.dc._msdcs 0 100 389 srv01/ Srv01 is the first master, I know, but is the server VM that has clock problems, in both situations. So I want to keep srv02 and install a new one. What do I have to do to let the 2nd VM be a single server? Could I use these URLs? /https://www.freeipa.org/page/Backup_and_Restore#One_Server_Loss_-_First_Master https://www.freeipa.org/page/V4/Server_Roles#Upgrade/ 2) uninstall ipa-server from the 1st server (srv01) and then powering off it, assuming that all data into the 2nd one are ok (srv02) 3) update freeipa and all other RPM packages into the VM srv02 4) install a new fresh VM, always with 7 release, and create a new replica Could I use the same old hostname (srv01) and IP address for this new VM? Or is better to use the same IP but a new name, like srv03? Do you think this is the right way to solve my issue? Or do you have any better idea? Please let me know, thanks. Bye, Morgan _______________________________________________ FreeIPA-users mailing list --freeipa-users@lists.fedorahosted.org <mailto:freeipa-users@lists.fedorahosted.org> To unsubscribe send an email tofreeipa-users-leave@lists.fedorahosted.org <mailto:freeipa-users-leave@lists.fedorahosted.org> Fedora Code of Conduct:https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines:https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives:https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org_______________________________________________ FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org <mailto:freeipa-users@lists.fedorahosted.org> To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org <mailto:freeipa-users-leave@lists.fedorahosted.org> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
Hi Theirry.
To tell the truth my configuration was already set to on, on both VMs:
*[root@srv01 ~]# ldapsearch -D "cn=Directory Manager" -h localhost -b "cn=config" -w $PASS | grep nsslapd-ignore-time-skewnsslapd-ignore-time-skew: on*
*[root@srv02 ~]# ldapsearch -D "cn=Directory Manager" -h localhost -b "cn=config" -w $PASS | grep nsslapd-ignore-time-skewnsslapd-ignore-time-skew: on*
Anyway, I tried to set up it to *off* and then to *on* again, but now I have a new issue into logs of the 2nd server :(
Srv01 logs are similar as before: *[23/Apr/2020:09:28:45.958922636 +0200] - WARN - csngen_new_csn - Too much time skew (-22777212 secs). Current seqnum=5ca0*
Srv02 logs now are like these:
*[23/Apr/2020:09:32:14.919328803 +0200] - ERR - agmt="cn=meTosrv01.ipa.mydomain.com http://meTosrv01.ipa.mydomain.com" (srv01:389) - clcache_load_buffer - Can't locate CSN 5e7cfe03000400030000 in the changelog (DB rc=-30988). If replication stops, the consumer may need to be reinitialized.[23/Apr/2020:09:32:14.920873489 +0200] - ERR - NSMMReplicationPlugin - changelog program - repl_plugin_name_cl - agmt="cn=meTosrv01.ipa.mydomain.com http://meTosrv01.ipa.mydomain.com" (srv01:389): CSN 5e7cfe03000400030000 not found, we aren't as up to date, or we purged[23/Apr/2020:09:32:14.922161821 +0200] - ERR - NSMMReplicationPlugin - send_updates - agmt="cn=meTosrv01.ipa.mydomain.com http://meTosrv01.ipa.mydomain.com" (srv01:389): Data required to update replica has been purged from the changelog. If the error persists the replica must be reinitialized.*
I have just tried to force a replica on both sides, without success:
*[root@srv01 ~]# ipa-replica-manage force-sync --from srv02.ipa.mydomain.com http://srv02.ipa.mydomain.comNo status yetNo status yetNo status yet[root@srv02 ~]# ipa-replica-manage force-sync --from srv01.ipa.mydomain.com http://srv01.ipa.mydomain.comNo status yetNo status yetNo status yet*
What could I do now? Thanks, bye
Il giorno mer 22 apr 2020 alle ore 17:08 thierry bordaz via FreeIPA-users < freeipa-users@lists.fedorahosted.org> ha scritto:
Hi Morgan,
Sure. The most immediate and safest action is to do
dn: cn=config changetype: modify replace: nsslapd-ignore-time-skew nsslapd-ignore-time-skew: on
On all servers in the topology (no need to restart). Then monitor if replication is catching up. Okay NTP issues is likely the RC of your time skew but there is not easy way to prove it if any.
best regards theirry
On 4/22/20 3:16 PM, Morgan Marodin via FreeIPA-users wrote:
Hi.
I don't have access to RedHat portal :( There are similar articles in a public forum?
Anyway ... could I stop ipa-server, change the value of *nsslapd-ignore-time-skew* into */etc/dirsrv/slapd-IPA-MYDOMAIN-COM/dse.ldif* and start again the server? Or is more complicated to change the configuration?
VMs are local, but the cluster where the 1st server is running is affected by NTP problems ... For this reason I want to remove the First Master and install another replica in the new cluster.
Thanks, bye. Morgan
Il giorno mer 22 apr 2020 alle ore 11:33 thierry bordaz via FreeIPA-users < freeipa-users@lists.fedorahosted.org> ha scritto:
Hi,
CSN generator time skew is a pending issue still under investigation.
At the moment the way your csn generator is messed up looks not fatal. You can allow replication to continue with the setting of nsslapd-ignore-time-skew on all servers. ( https://access.redhat.com/solutions/1162703)
If it does not allow replication to continue there is a recovery procedure but I would recommend to first try ignore-time-skew ( https://access.redhat.com/solutions/3543811)
NTP tuning or specific VMs are suspected to contribute to time skew. What type of VMs are you using (local or cloud (AWS)) ?
best regards thierry
On 4/21/20 5:42 PM, Morgan Marodin via FreeIPA-users wrote:
Hi.
Into my environment I have two IPA server, replicating each other. They are both 7.6 OS systems, ipa-server RPM version is 4.6.4-10.0.1.el7_6.2.x86_64.
The first server installed was srv01 (many years ago), then I installed the replica into srv02 (like a year later the 1st node). When I had a single server I did also a trust with my corporate Active Directory. VMs are running in 2 different hypervisor clusters.
Now the replication doesn't works. Into log files I have this error:
*[16/Apr/2020:12:25:36.856632697 +0200] - ERR - csngen_adjust_time - Adjustment limit exceeded; value - 23221226, limit - 86400 [16/Apr/2020:12:25:36.857909222 +0200] - ERR - NSMMReplicationPlugin - repl5_inc_run - agmt="cn=meTosrv01.ipa.mydomain.com http://meTosrv01.ipa.mydomain.com" (srv01:389): Fatal error - too much time skew between replicas! [16/Apr/2020:12:25:36.862233147 +0200] - ERR - NSMMReplicationPlugin - repl5_inc_run - agmt="cn=meTosrv01.ipa.mydomain.com http://meTosrv01.ipa.mydomain.com" (srv01:389): Incremental update failed and requires administrator action*
I tried to force the replica, but the limit exceeded problem doesn't allow the sync. I know that the problem is that CSN generator has become grossly skewed. Using the external script readNsState.py I found that there was as offset time for about a month, so ... I waited for a month and then the issue disappeared. But now the offset is about 9 months ... I can't wait so much time :)
*[root@srv01 scripts]# ./readNsState.py /etc/dirsrv/slapd-IPA-MYDOMAIN-COM/dse.ldif nsState is BAAAAAAAAACCN/xfAAAAAHbiBAAAAAAABCgAAAAAAAANdQAAAAAAAA== Little Endian For replica cn=replica,cn=dc\3Dipa\2Cdc\3Dmydomain\2Cdc\3Dcom,cn=mapping tree,cn=con fmtstr=[H6x3QH6x] size=40 len of nsstate is 40 CSN generator state: Replica ID : 4 Sampled Time : 1610364802 Gen as csn : 5ffc37822996500040000 Time as str : Mon Jan 11 12:33:22 2021 Local Offset : 320118 Remote Offset : 10244 Seq. num : 29965 System time : Tue Apr 21 15:03:45 2020 Diff in sec. : -22890577 Day:sec diff : -265:5423 nsState is YAAAAAAAAAADLZheAAAAAAAAAAAAAAAAXSgAAAAAAAATAAAAAAAAAA== Little Endian For replica cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config fmtstr=[H6x3QH6x] size=40 len of nsstate is 40 CSN generator state: Replica ID : 96 Sampled Time : 1587031299 Gen as csn : 5e982d03001900960000 Time as str : Thu Apr 16 12:01:39 2020 Local Offset : 0 Remote Offset : 10333 Seq. num : 19 System time : Tue Apr 21 15:03:45 2020 Diff in sec. : 442926 Day:sec diff : 5:10926 [root@srv02 scripts]# ./readNsState.py /etc/dirsrv/slapd-IPA-MYDOMAIN-COM/dse.ldif nsState is AwAAAAAAAABU7p5eAAAAAAAAAAAAAAAAsVNiAQAAAAAAAAAAAAAAAA== Little Endian For replica cn=replica,cn=dc\3Dipa\2Cdc\3Dmydomain\2Cdc\3Dcom,cn=mapping tree,cn=con fmtstr=[H6x3QH6x] size=40 len of nsstate is 40 CSN generator state: Replica ID : 3 Sampled Time : 1587474004 Gen as csn : 5e9eee54000000030000 Time as str : Tue Apr 21 15:00:04 2020 Local Offset : 0 Remote Offset : 23221169 Seq. num : 0 System time : Tue Apr 21 15:02:38 2020 Diff in sec. : 154 Day:sec diff : 0:154 nsState is YQAAAAAAAAAuLZheAAAAAEUBAAAAAAAA7SYAAAAAAAASAAAAAAAAAA== Little Endian For replica cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config fmtstr=[H6x3QH6x] size=40 len of nsstate is 40 CSN generator state: Replica ID : 97 Sampled Time : 1587031342 Gen as csn : 5e982d2e001800970000 Time as str : Thu Apr 16 12:02:22 2020 Local Offset : 325 Remote Offset : 9965 Seq. num : 18 System time : Tue Apr 21 15:02:38 2020 Diff in sec. : 442816 Day:sec diff : 5:10816*
As you can see in the 1st node the Time as str is Jan 11 of 2021. With timedatectl command I see that both VMs use the same Time zone and the clock is correct.
I found this old article to fix my issue: *https://www.redhat.com/archives/freeipa-users/2014-February/msg00007.html https://www.redhat.com/archives/freeipa-users/2014-February/msg00007.html*
But ... I had the same issue in the past, always in the 1st server. So, in my mind I don't want to try to use that fix. I have a new hypervisor cluster, so I would prefer to reinstall the 1st server, using these steps:
- check if all roles (also the CA) is installed in srv02
You can find here some data about the VMs:
*[root@srv01 ~]# ipa server-show srv01.ipa.mydomain.com http://srv01.ipa.mydomain.com Server name: srv01.ipa.mydomain.com http://srv01.ipa.mydomain.com Managed suffixes: domain, ca Min domain level: 0 Max domain level: 1 Enabled server roles: CA server, IPA master, DNS server, NTP server, AD trust controller [root@srv02 ~]# ipa server-show srv02.ipa.mydomain.com http://srv02.ipa.mydomain.com Server name: srv02.ipa.mydomain.com http://srv02.ipa.mydomain.com Managed suffixes: domain, ca Min domain level: 0 Max domain level: 1 Enabled server roles: CA server, IPA master, DNS server, NTP server [root@srv01 ~]# ipa config-show Maximum username length: 32 Home directory base: /home Default shell: /bin/bash Default users group: ipausers Default e-mail domain: ipa.mydomain.com http://ipa.mydomain.com Search time limit: 2 Search size limit: 100 User search fields: uid,givenname,sn,telephonenumber,ou,title Group search fields: cn,description Enable migration mode: FALSE Certificate Subject base: O=IPA.MYDOMAIN.COM http://IPA.MYDOMAIN.COM Password Expiration Notification (days): 4 Password plugin features: AllowNThash SELinux user map order: guest_u:s0$xguest_u:s0$user_u:s0$staff_u:s0-s0:c0.c1023$unconfined_u:s0-s0:c0.c1023 Default SELinux user: unconfined_u:s0-s0:c0.c1023 Default PAC types: MS-PAC, nfs:NONE IPA masters: srv01.ipa.mydomain.com http://srv01.ipa.mydomain.com, srv02.ipa.mydomain.com http://srv02.ipa.mydomain.com IPA CA servers: srv01.ipa.mydomain.com http://srv01.ipa.mydomain.com, srv02.ipa.mydomain.com http://srv02.ipa.mydomain.com IPA NTP servers: srv01.ipa.mydomain.com http://srv01.ipa.mydomain.com, srv02.ipa.mydomain.com http://srv02.ipa.mydomain.com IPA CA renewal master: srv01.ipa.mydomain.com http://srv01.ipa.mydomain.com [root@srv02 ~]# ipa config-show Maximum username length: 32 Home directory base: /home Default shell: /bin/bash Default users group: ipausers Default e-mail domain: ipa.mydomain.com http://ipa.mydomain.com Search time limit: 2 Search size limit: 100 User search fields: uid,givenname,sn,telephonenumber,ou,title Group search fields: cn,description Enable migration mode: FALSE Certificate Subject base: O=IPA.MYDOMAIN.COM http://IPA.MYDOMAIN.COM Password Expiration Notification (days): 4 Password plugin features: AllowNThash SELinux user map order: guest_u:s0$xguest_u:s0$user_u:s0$staff_u:s0-s0:c0.c1023$unconfined_u:s0-s0:c0.c1023 Default SELinux user: unconfined_u:s0-s0:c0.c1023 Default PAC types: MS-PAC, nfs:NONE IPA masters: srv01.ipa.mydomain.com http://srv01.ipa.mydomain.com, srv02.ipa.mydomain.com http://srv02.ipa.mydomain.com IPA CA servers: srv01.ipa.mydomain.com http://srv01.ipa.mydomain.com, srv02.ipa.mydomain.com http://srv02.ipa.mydomain.com IPA NTP servers: srv01.ipa.mydomain.com http://srv01.ipa.mydomain.com, srv02.ipa.mydomain.com http://srv02.ipa.mydomain.com IPA CA renewal master: srv01.ipa.mydomain.com http://srv01.ipa.mydomain.com [root@srv01 ~]# ipactl status Directory Service: RUNNING krb5kdc Service: RUNNING kadmin Service: RUNNING named Service: RUNNING httpd Service: RUNNING ipa-custodia Service: RUNNING ntpd Service: RUNNING pki-tomcatd Service: STOPPED smb Service: RUNNING winbind Service: RUNNING ipa-otpd Service: RUNNING ipa-dnskeysyncd Service: RUNNING ipa: INFO: The ipactl command was successful [root@srv02 ~]# ipactl status Directory Service: RUNNING krb5kdc Service: RUNNING kadmin Service: RUNNING named Service: RUNNING httpd Service: RUNNING ipa-custodia Service: RUNNING ntpd Service: RUNNING pki-tomcatd Service: STOPPED ipa-otpd Service: RUNNING ipa-dnskeysyncd Service: RUNNING ipa: INFO: The ipactl command was successful [root@srv01 ~]# certutil -L -d /etc/pki/pki-tomcat/alias Certificate Nickname Trust Attributes SSL,S/MIME,JAR/XPI Server-Cert cert-pki-ca u,u,u subsystemCert cert-pki-ca u,u,u caSigningCert cert-pki-ca CTu,Cu,Cu ocspSigningCert cert-pki-ca u,u,u auditSigningCert cert-pki-ca u,u,Pu [root@srv02 ~]# certutil -L -d /etc/pki/pki-tomcat/alias Certificate Nickname Trust Attributes SSL,S/MIME,JAR/XPI Server-Cert cert-pki-ca u,u,u subsystemCert cert-pki-ca u,u,u caSigningCert cert-pki-ca CTu,u,u ocspSigningCert cert-pki-ca u,u,u auditSigningCert cert-pki-ca u,u,Pu*
It seems that AD trust controller role, IPA CA renewal master, smb and windbind are only in the 1st server. And also caSigningCert cert-pki-ca entry is different (CTu,Cu,Cu vs CTu,u,u).
I can see only in the 1st server these DNS records:
*_kerberos._tcp.Default-First-Site-Name._sites.dc._msdcs SRV 0 100 88 srv01 _kerberos._tcp.dc._msdcs SRV 0 100 88 srv01 _kerberos._udp.Default-First-Site-Name._sites.dc._msdcs SRV 0 100 88 srv01 _kerberos._udp.dc._msdcs SRV 0 100 88 srv01 _ldap._tcp.Default-First-Site-Name._sites.dc._msdcs 0 100 389 srv01 _ldap._tcp.dc._msdcs 0 100 389 srv01*
Srv01 is the first master, I know, but is the server VM that has clock problems, in both situations. So I want to keep srv02 and install a new one.
What do I have to do to let the 2nd VM be a single server? Could I use these URLs?
*https://www.freeipa.org/page/Backup_and_Restore#One_Server_Loss_-_First_Mast... https://www.freeipa.org/page/Backup_and_Restore#One_Server_Loss_-_First_Master https://www.freeipa.org/page/V4/Server_Roles#Upgrade https://www.freeipa.org/page/V4/Server_Roles#Upgrade*
- uninstall ipa-server from the 1st server (srv01) and then powering off
it, assuming that all data into the 2nd one are ok (srv02)
update freeipa and all other RPM packages into the VM srv02
install a new fresh VM, always with 7 release, and create a new replica
Could I use the same old hostname (srv01) and IP address for this new VM? Or is better to use the same IP but a new name, like srv03?
Do you think this is the right way to solve my issue? Or do you have any better idea?
Please let me know, thanks. Bye, Morgan
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
Sorry Thierry, any news for my issue? Did you see new logs?
Could I use the command *ipa-replica-manage re-initialize --from srv02.ipa.mydomain.com http://srv02.ipa.mydomain.com* in the First Master server? Or is it better to solve the issue in another way?
Please let me know, thanks. Bye
Il giorno gio 23 apr 2020 alle ore 09:54 Morgan Marodin morgan@marodin.it ha scritto:
Hi Theirry.
To tell the truth my configuration was already set to on, on both VMs:
*[root@srv01 ~]# ldapsearch -D "cn=Directory Manager" -h localhost -b "cn=config" -w $PASS | grep nsslapd-ignore-time-skewnsslapd-ignore-time-skew: on*
*[root@srv02 ~]# ldapsearch -D "cn=Directory Manager" -h localhost -b "cn=config" -w $PASS | grep nsslapd-ignore-time-skewnsslapd-ignore-time-skew: on*
Anyway, I tried to set up it to *off* and then to *on* again, but now I have a new issue into logs of the 2nd server :(
Srv01 logs are similar as before: *[23/Apr/2020:09:28:45.958922636 +0200] - WARN - csngen_new_csn - Too much time skew (-22777212 secs). Current seqnum=5ca0*
Srv02 logs now are like these:
*[23/Apr/2020:09:32:14.919328803 +0200] - ERR - agmt="cn=meTosrv01.ipa.mydomain.com http://meTosrv01.ipa.mydomain.com" (srv01:389) - clcache_load_buffer - Can't locate CSN 5e7cfe03000400030000 in the changelog (DB rc=-30988). If replication stops, the consumer may need to be reinitialized.[23/Apr/2020:09:32:14.920873489 +0200] - ERR - NSMMReplicationPlugin - changelog program - repl_plugin_name_cl - agmt="cn=meTosrv01.ipa.mydomain.com http://meTosrv01.ipa.mydomain.com" (srv01:389): CSN 5e7cfe03000400030000 not found, we aren't as up to date, or we purged[23/Apr/2020:09:32:14.922161821 +0200] - ERR - NSMMReplicationPlugin - send_updates - agmt="cn=meTosrv01.ipa.mydomain.com http://meTosrv01.ipa.mydomain.com" (srv01:389): Data required to update replica has been purged from the changelog. If the error persists the replica must be reinitialized.*
I have just tried to force a replica on both sides, without success:
*[root@srv01 ~]# ipa-replica-manage force-sync --from srv02.ipa.mydomain.com http://srv02.ipa.mydomain.comNo status yetNo status yetNo status yet[root@srv02 ~]# ipa-replica-manage force-sync --from srv01.ipa.mydomain.com http://srv01.ipa.mydomain.comNo status yetNo status yetNo status yet*
What could I do now? Thanks, bye
Il giorno mer 22 apr 2020 alle ore 17:08 thierry bordaz via FreeIPA-users < freeipa-users@lists.fedorahosted.org> ha scritto:
Hi Morgan,
Sure. The most immediate and safest action is to do
dn: cn=config changetype: modify replace: nsslapd-ignore-time-skew nsslapd-ignore-time-skew: on
On all servers in the topology (no need to restart). Then monitor if replication is catching up. Okay NTP issues is likely the RC of your time skew but there is not easy way to prove it if any.
best regards theirry
On 4/22/20 3:16 PM, Morgan Marodin via FreeIPA-users wrote:
Hi.
I don't have access to RedHat portal :( There are similar articles in a public forum?
Anyway ... could I stop ipa-server, change the value of *nsslapd-ignore-time-skew* into */etc/dirsrv/slapd-IPA-MYDOMAIN-COM/dse.ldif* and start again the server? Or is more complicated to change the configuration?
VMs are local, but the cluster where the 1st server is running is affected by NTP problems ... For this reason I want to remove the First Master and install another replica in the new cluster.
Thanks, bye. Morgan
Il giorno mer 22 apr 2020 alle ore 11:33 thierry bordaz via FreeIPA-users freeipa-users@lists.fedorahosted.org ha scritto:
Hi,
CSN generator time skew is a pending issue still under investigation.
At the moment the way your csn generator is messed up looks not fatal. You can allow replication to continue with the setting of nsslapd-ignore-time-skew on all servers. ( https://access.redhat.com/solutions/1162703)
If it does not allow replication to continue there is a recovery procedure but I would recommend to first try ignore-time-skew ( https://access.redhat.com/solutions/3543811)
NTP tuning or specific VMs are suspected to contribute to time skew. What type of VMs are you using (local or cloud (AWS)) ?
best regards thierry
On 4/21/20 5:42 PM, Morgan Marodin via FreeIPA-users wrote:
Hi.
Into my environment I have two IPA server, replicating each other. They are both 7.6 OS systems, ipa-server RPM version is 4.6.4-10.0.1.el7_6.2.x86_64.
The first server installed was srv01 (many years ago), then I installed the replica into srv02 (like a year later the 1st node). When I had a single server I did also a trust with my corporate Active Directory. VMs are running in 2 different hypervisor clusters.
Now the replication doesn't works. Into log files I have this error:
*[16/Apr/2020:12:25:36.856632697 +0200] - ERR - csngen_adjust_time - Adjustment limit exceeded; value - 23221226, limit - 86400 [16/Apr/2020:12:25:36.857909222 +0200] - ERR - NSMMReplicationPlugin - repl5_inc_run - agmt="cn=meTosrv01.ipa.mydomain.com http://meTosrv01.ipa.mydomain.com" (srv01:389): Fatal error - too much time skew between replicas! [16/Apr/2020:12:25:36.862233147 +0200] - ERR - NSMMReplicationPlugin - repl5_inc_run - agmt="cn=meTosrv01.ipa.mydomain.com http://meTosrv01.ipa.mydomain.com" (srv01:389): Incremental update failed and requires administrator action*
I tried to force the replica, but the limit exceeded problem doesn't allow the sync. I know that the problem is that CSN generator has become grossly skewed. Using the external script readNsState.py I found that there was as offset time for about a month, so ... I waited for a month and then the issue disappeared. But now the offset is about 9 months ... I can't wait so much time :)
*[root@srv01 scripts]# ./readNsState.py /etc/dirsrv/slapd-IPA-MYDOMAIN-COM/dse.ldif nsState is BAAAAAAAAACCN/xfAAAAAHbiBAAAAAAABCgAAAAAAAANdQAAAAAAAA== Little Endian For replica cn=replica,cn=dc\3Dipa\2Cdc\3Dmydomain\2Cdc\3Dcom,cn=mapping tree,cn=con fmtstr=[H6x3QH6x] size=40 len of nsstate is 40 CSN generator state: Replica ID : 4 Sampled Time : 1610364802 Gen as csn : 5ffc37822996500040000 Time as str : Mon Jan 11 12:33:22 2021 Local Offset : 320118 Remote Offset : 10244 Seq. num : 29965 System time : Tue Apr 21 15:03:45 2020 Diff in sec. : -22890577 Day:sec diff : -265:5423 nsState is YAAAAAAAAAADLZheAAAAAAAAAAAAAAAAXSgAAAAAAAATAAAAAAAAAA== Little Endian For replica cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config fmtstr=[H6x3QH6x] size=40 len of nsstate is 40 CSN generator state: Replica ID : 96 Sampled Time : 1587031299 Gen as csn : 5e982d03001900960000 Time as str : Thu Apr 16 12:01:39 2020 Local Offset : 0 Remote Offset : 10333 Seq. num : 19 System time : Tue Apr 21 15:03:45 2020 Diff in sec. : 442926 Day:sec diff : 5:10926 [root@srv02 scripts]# ./readNsState.py /etc/dirsrv/slapd-IPA-MYDOMAIN-COM/dse.ldif nsState is AwAAAAAAAABU7p5eAAAAAAAAAAAAAAAAsVNiAQAAAAAAAAAAAAAAAA== Little Endian For replica cn=replica,cn=dc\3Dipa\2Cdc\3Dmydomain\2Cdc\3Dcom,cn=mapping tree,cn=con fmtstr=[H6x3QH6x] size=40 len of nsstate is 40 CSN generator state: Replica ID : 3 Sampled Time : 1587474004 Gen as csn : 5e9eee54000000030000 Time as str : Tue Apr 21 15:00:04 2020 Local Offset : 0 Remote Offset : 23221169 Seq. num : 0 System time : Tue Apr 21 15:02:38 2020 Diff in sec. : 154 Day:sec diff : 0:154 nsState is YQAAAAAAAAAuLZheAAAAAEUBAAAAAAAA7SYAAAAAAAASAAAAAAAAAA== Little Endian For replica cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config fmtstr=[H6x3QH6x] size=40 len of nsstate is 40 CSN generator state: Replica ID : 97 Sampled Time : 1587031342 Gen as csn : 5e982d2e001800970000 Time as str : Thu Apr 16 12:02:22 2020 Local Offset : 325 Remote Offset : 9965 Seq. num : 18 System time : Tue Apr 21 15:02:38 2020 Diff in sec. : 442816 Day:sec diff : 5:10816*
As you can see in the 1st node the Time as str is Jan 11 of 2021. With timedatectl command I see that both VMs use the same Time zone and the clock is correct.
I found this old article to fix my issue: *https://www.redhat.com/archives/freeipa-users/2014-February/msg00007.html https://www.redhat.com/archives/freeipa-users/2014-February/msg00007.html*
But ... I had the same issue in the past, always in the 1st server. So, in my mind I don't want to try to use that fix. I have a new hypervisor cluster, so I would prefer to reinstall the 1st server, using these steps:
- check if all roles (also the CA) is installed in srv02
You can find here some data about the VMs:
*[root@srv01 ~]# ipa server-show srv01.ipa.mydomain.com http://srv01.ipa.mydomain.com Server name: srv01.ipa.mydomain.com http://srv01.ipa.mydomain.com Managed suffixes: domain, ca Min domain level: 0 Max domain level: 1 Enabled server roles: CA server, IPA master, DNS server, NTP server, AD trust controller [root@srv02 ~]# ipa server-show srv02.ipa.mydomain.com http://srv02.ipa.mydomain.com Server name: srv02.ipa.mydomain.com http://srv02.ipa.mydomain.com Managed suffixes: domain, ca Min domain level: 0 Max domain level: 1 Enabled server roles: CA server, IPA master, DNS server, NTP server [root@srv01 ~]# ipa config-show Maximum username length: 32 Home directory base: /home Default shell: /bin/bash Default users group: ipausers Default e-mail domain: ipa.mydomain.com http://ipa.mydomain.com Search time limit: 2 Search size limit: 100 User search fields: uid,givenname,sn,telephonenumber,ou,title Group search fields: cn,description Enable migration mode: FALSE Certificate Subject base: O=IPA.MYDOMAIN.COM http://IPA.MYDOMAIN.COM Password Expiration Notification (days): 4 Password plugin features: AllowNThash SELinux user map order: guest_u:s0$xguest_u:s0$user_u:s0$staff_u:s0-s0:c0.c1023$unconfined_u:s0-s0:c0.c1023 Default SELinux user: unconfined_u:s0-s0:c0.c1023 Default PAC types: MS-PAC, nfs:NONE IPA masters: srv01.ipa.mydomain.com http://srv01.ipa.mydomain.com, srv02.ipa.mydomain.com http://srv02.ipa.mydomain.com IPA CA servers: srv01.ipa.mydomain.com http://srv01.ipa.mydomain.com, srv02.ipa.mydomain.com http://srv02.ipa.mydomain.com IPA NTP servers: srv01.ipa.mydomain.com http://srv01.ipa.mydomain.com, srv02.ipa.mydomain.com http://srv02.ipa.mydomain.com IPA CA renewal master: srv01.ipa.mydomain.com http://srv01.ipa.mydomain.com [root@srv02 ~]# ipa config-show Maximum username length: 32 Home directory base: /home Default shell: /bin/bash Default users group: ipausers Default e-mail domain: ipa.mydomain.com http://ipa.mydomain.com Search time limit: 2 Search size limit: 100 User search fields: uid,givenname,sn,telephonenumber,ou,title Group search fields: cn,description Enable migration mode: FALSE Certificate Subject base: O=IPA.MYDOMAIN.COM http://IPA.MYDOMAIN.COM Password Expiration Notification (days): 4 Password plugin features: AllowNThash SELinux user map order: guest_u:s0$xguest_u:s0$user_u:s0$staff_u:s0-s0:c0.c1023$unconfined_u:s0-s0:c0.c1023 Default SELinux user: unconfined_u:s0-s0:c0.c1023 Default PAC types: MS-PAC, nfs:NONE IPA masters: srv01.ipa.mydomain.com http://srv01.ipa.mydomain.com, srv02.ipa.mydomain.com http://srv02.ipa.mydomain.com IPA CA servers: srv01.ipa.mydomain.com http://srv01.ipa.mydomain.com, srv02.ipa.mydomain.com http://srv02.ipa.mydomain.com IPA NTP servers: srv01.ipa.mydomain.com http://srv01.ipa.mydomain.com, srv02.ipa.mydomain.com http://srv02.ipa.mydomain.com IPA CA renewal master: srv01.ipa.mydomain.com http://srv01.ipa.mydomain.com [root@srv01 ~]# ipactl status Directory Service: RUNNING krb5kdc Service: RUNNING kadmin Service: RUNNING named Service: RUNNING httpd Service: RUNNING ipa-custodia Service: RUNNING ntpd Service: RUNNING pki-tomcatd Service: STOPPED smb Service: RUNNING winbind Service: RUNNING ipa-otpd Service: RUNNING ipa-dnskeysyncd Service: RUNNING ipa: INFO: The ipactl command was successful [root@srv02 ~]# ipactl status Directory Service: RUNNING krb5kdc Service: RUNNING kadmin Service: RUNNING named Service: RUNNING httpd Service: RUNNING ipa-custodia Service: RUNNING ntpd Service: RUNNING pki-tomcatd Service: STOPPED ipa-otpd Service: RUNNING ipa-dnskeysyncd Service: RUNNING ipa: INFO: The ipactl command was successful [root@srv01 ~]# certutil -L -d /etc/pki/pki-tomcat/alias Certificate Nickname Trust Attributes SSL,S/MIME,JAR/XPI Server-Cert cert-pki-ca u,u,u subsystemCert cert-pki-ca u,u,u caSigningCert cert-pki-ca CTu,Cu,Cu ocspSigningCert cert-pki-ca u,u,u auditSigningCert cert-pki-ca u,u,Pu [root@srv02 ~]# certutil -L -d /etc/pki/pki-tomcat/alias Certificate Nickname Trust Attributes SSL,S/MIME,JAR/XPI Server-Cert cert-pki-ca u,u,u subsystemCert cert-pki-ca u,u,u caSigningCert cert-pki-ca CTu,u,u ocspSigningCert cert-pki-ca u,u,u auditSigningCert cert-pki-ca u,u,Pu*
It seems that AD trust controller role, IPA CA renewal master, smb and windbind are only in the 1st server. And also caSigningCert cert-pki-ca entry is different (CTu,Cu,Cu vs CTu,u,u).
I can see only in the 1st server these DNS records:
*_kerberos._tcp.Default-First-Site-Name._sites.dc._msdcs SRV 0 100 88 srv01 _kerberos._tcp.dc._msdcs SRV 0 100 88 srv01 _kerberos._udp.Default-First-Site-Name._sites.dc._msdcs SRV 0 100 88 srv01 _kerberos._udp.dc._msdcs SRV 0 100 88 srv01 _ldap._tcp.Default-First-Site-Name._sites.dc._msdcs 0 100 389 srv01 _ldap._tcp.dc._msdcs 0 100 389 srv01*
Srv01 is the first master, I know, but is the server VM that has clock problems, in both situations. So I want to keep srv02 and install a new one.
What do I have to do to let the 2nd VM be a single server? Could I use these URLs?
*https://www.freeipa.org/page/Backup_and_Restore#One_Server_Loss_-_First_Mast... https://www.freeipa.org/page/Backup_and_Restore#One_Server_Loss_-_First_Master https://www.freeipa.org/page/V4/Server_Roles#Upgrade https://www.freeipa.org/page/V4/Server_Roles#Upgrade*
- uninstall ipa-server from the 1st server (srv01) and then powering
off it, assuming that all data into the 2nd one are ok (srv02)
update freeipa and all other RPM packages into the VM srv02
install a new fresh VM, always with 7 release, and create a new
replica Could I use the same old hostname (srv01) and IP address for this new VM? Or is better to use the same IP but a new name, like srv03?
Do you think this is the right way to solve my issue? Or do you have any better idea?
Please let me know, thanks. Bye, Morgan
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
My issue got worse, the certificate has expired on the replica server, and it can't be renewed because it cannot communicate with the master server. I can start the server using the *--ignore-service-failure* parameter, but the *httpd* and the *pki-tomcatd* failed to start ...
Do I have to start up with a fresh install? :(
Do you know any company that can give me support about FreeIPA?
Please let me know, thanks. Bye, Morgan
Il giorno lun 27 apr 2020 alle ore 12:16 Morgan Marodin morgan@marodin.it ha scritto:
Sorry Thierry, any news for my issue? Did you see new logs?
Could I use the command *ipa-replica-manage re-initialize --from srv02.ipa.mydomain.com http://srv02.ipa.mydomain.com* in the First Master server? Or is it better to solve the issue in another way?
Please let me know, thanks. Bye
Il giorno gio 23 apr 2020 alle ore 09:54 Morgan Marodin morgan@marodin.it ha scritto:
Hi Theirry.
To tell the truth my configuration was already set to on, on both VMs:
*[root@srv01 ~]# ldapsearch -D "cn=Directory Manager" -h localhost -b "cn=config" -w $PASS | grep nsslapd-ignore-time-skewnsslapd-ignore-time-skew: on*
*[root@srv02 ~]# ldapsearch -D "cn=Directory Manager" -h localhost -b "cn=config" -w $PASS | grep nsslapd-ignore-time-skewnsslapd-ignore-time-skew: on*
Anyway, I tried to set up it to *off* and then to *on* again, but now I have a new issue into logs of the 2nd server :(
Srv01 logs are similar as before: *[23/Apr/2020:09:28:45.958922636 +0200] - WARN - csngen_new_csn - Too much time skew (-22777212 secs). Current seqnum=5ca0*
Srv02 logs now are like these:
*[23/Apr/2020:09:32:14.919328803 +0200] - ERR - agmt="cn=meTosrv01.ipa.mydomain.com http://meTosrv01.ipa.mydomain.com" (srv01:389) - clcache_load_buffer - Can't locate CSN 5e7cfe03000400030000 in the changelog (DB rc=-30988). If replication stops, the consumer may need to be reinitialized.[23/Apr/2020:09:32:14.920873489 +0200] - ERR - NSMMReplicationPlugin - changelog program - repl_plugin_name_cl - agmt="cn=meTosrv01.ipa.mydomain.com http://meTosrv01.ipa.mydomain.com" (srv01:389): CSN 5e7cfe03000400030000 not found, we aren't as up to date, or we purged[23/Apr/2020:09:32:14.922161821 +0200] - ERR - NSMMReplicationPlugin - send_updates - agmt="cn=meTosrv01.ipa.mydomain.com http://meTosrv01.ipa.mydomain.com" (srv01:389): Data required to update replica has been purged from the changelog. If the error persists the replica must be reinitialized.*
I have just tried to force a replica on both sides, without success:
*[root@srv01 ~]# ipa-replica-manage force-sync --from srv02.ipa.mydomain.com http://srv02.ipa.mydomain.comNo status yetNo status yetNo status yet[root@srv02 ~]# ipa-replica-manage force-sync --from srv01.ipa.mydomain.com http://srv01.ipa.mydomain.comNo status yetNo status yetNo status yet*
What could I do now? Thanks, bye
Il giorno mer 22 apr 2020 alle ore 17:08 thierry bordaz via FreeIPA-users freeipa-users@lists.fedorahosted.org ha scritto:
Hi Morgan,
Sure. The most immediate and safest action is to do
dn: cn=config changetype: modify replace: nsslapd-ignore-time-skew nsslapd-ignore-time-skew: on
On all servers in the topology (no need to restart). Then monitor if replication is catching up. Okay NTP issues is likely the RC of your time skew but there is not easy way to prove it if any.
best regards theirry
On 4/22/20 3:16 PM, Morgan Marodin via FreeIPA-users wrote:
Hi.
I don't have access to RedHat portal :( There are similar articles in a public forum?
Anyway ... could I stop ipa-server, change the value of *nsslapd-ignore-time-skew* into */etc/dirsrv/slapd-IPA-MYDOMAIN-COM/dse.ldif* and start again the server? Or is more complicated to change the configuration?
VMs are local, but the cluster where the 1st server is running is affected by NTP problems ... For this reason I want to remove the First Master and install another replica in the new cluster.
Thanks, bye. Morgan
Il giorno mer 22 apr 2020 alle ore 11:33 thierry bordaz via FreeIPA-users freeipa-users@lists.fedorahosted.org ha scritto:
Hi,
CSN generator time skew is a pending issue still under investigation.
At the moment the way your csn generator is messed up looks not fatal. You can allow replication to continue with the setting of nsslapd-ignore-time-skew on all servers. ( https://access.redhat.com/solutions/1162703)
If it does not allow replication to continue there is a recovery procedure but I would recommend to first try ignore-time-skew ( https://access.redhat.com/solutions/3543811)
NTP tuning or specific VMs are suspected to contribute to time skew. What type of VMs are you using (local or cloud (AWS)) ?
best regards thierry
On 4/21/20 5:42 PM, Morgan Marodin via FreeIPA-users wrote:
Hi.
Into my environment I have two IPA server, replicating each other. They are both 7.6 OS systems, ipa-server RPM version is 4.6.4-10.0.1.el7_6.2.x86_64.
The first server installed was srv01 (many years ago), then I installed the replica into srv02 (like a year later the 1st node). When I had a single server I did also a trust with my corporate Active Directory. VMs are running in 2 different hypervisor clusters.
Now the replication doesn't works. Into log files I have this error:
*[16/Apr/2020:12:25:36.856632697 +0200] - ERR - csngen_adjust_time - Adjustment limit exceeded; value - 23221226, limit - 86400 [16/Apr/2020:12:25:36.857909222 +0200] - ERR - NSMMReplicationPlugin - repl5_inc_run - agmt="cn=meTosrv01.ipa.mydomain.com http://meTosrv01.ipa.mydomain.com" (srv01:389): Fatal error - too much time skew between replicas! [16/Apr/2020:12:25:36.862233147 +0200] - ERR - NSMMReplicationPlugin - repl5_inc_run - agmt="cn=meTosrv01.ipa.mydomain.com http://meTosrv01.ipa.mydomain.com" (srv01:389): Incremental update failed and requires administrator action*
I tried to force the replica, but the limit exceeded problem doesn't allow the sync. I know that the problem is that CSN generator has become grossly skewed. Using the external script readNsState.py I found that there was as offset time for about a month, so ... I waited for a month and then the issue disappeared. But now the offset is about 9 months ... I can't wait so much time :)
*[root@srv01 scripts]# ./readNsState.py /etc/dirsrv/slapd-IPA-MYDOMAIN-COM/dse.ldif nsState is BAAAAAAAAACCN/xfAAAAAHbiBAAAAAAABCgAAAAAAAANdQAAAAAAAA== Little Endian For replica cn=replica,cn=dc\3Dipa\2Cdc\3Dmydomain\2Cdc\3Dcom,cn=mapping tree,cn=con fmtstr=[H6x3QH6x] size=40 len of nsstate is 40 CSN generator state: Replica ID : 4 Sampled Time : 1610364802 Gen as csn : 5ffc37822996500040000 Time as str : Mon Jan 11 12:33:22 2021 Local Offset : 320118 Remote Offset : 10244 Seq. num : 29965 System time : Tue Apr 21 15:03:45 2020 Diff in sec. : -22890577 Day:sec diff : -265:5423 nsState is YAAAAAAAAAADLZheAAAAAAAAAAAAAAAAXSgAAAAAAAATAAAAAAAAAA== Little Endian For replica cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config fmtstr=[H6x3QH6x] size=40 len of nsstate is 40 CSN generator state: Replica ID : 96 Sampled Time : 1587031299 Gen as csn : 5e982d03001900960000 Time as str : Thu Apr 16 12:01:39 2020 Local Offset : 0 Remote Offset : 10333 Seq. num : 19 System time : Tue Apr 21 15:03:45 2020 Diff in sec. : 442926 Day:sec diff : 5:10926 [root@srv02 scripts]# ./readNsState.py /etc/dirsrv/slapd-IPA-MYDOMAIN-COM/dse.ldif nsState is AwAAAAAAAABU7p5eAAAAAAAAAAAAAAAAsVNiAQAAAAAAAAAAAAAAAA== Little Endian For replica cn=replica,cn=dc\3Dipa\2Cdc\3Dmydomain\2Cdc\3Dcom,cn=mapping tree,cn=con fmtstr=[H6x3QH6x] size=40 len of nsstate is 40 CSN generator state: Replica ID : 3 Sampled Time : 1587474004 Gen as csn : 5e9eee54000000030000 Time as str : Tue Apr 21 15:00:04 2020 Local Offset : 0 Remote Offset : 23221169 Seq. num : 0 System time : Tue Apr 21 15:02:38 2020 Diff in sec. : 154 Day:sec diff : 0:154 nsState is YQAAAAAAAAAuLZheAAAAAEUBAAAAAAAA7SYAAAAAAAASAAAAAAAAAA== Little Endian For replica cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config fmtstr=[H6x3QH6x] size=40 len of nsstate is 40 CSN generator state: Replica ID : 97 Sampled Time : 1587031342 Gen as csn : 5e982d2e001800970000 Time as str : Thu Apr 16 12:02:22 2020 Local Offset : 325 Remote Offset : 9965 Seq. num : 18 System time : Tue Apr 21 15:02:38 2020 Diff in sec. : 442816 Day:sec diff : 5:10816*
As you can see in the 1st node the Time as str is Jan 11 of 2021. With timedatectl command I see that both VMs use the same Time zone and the clock is correct.
I found this old article to fix my issue: *https://www.redhat.com/archives/freeipa-users/2014-February/msg00007.html https://www.redhat.com/archives/freeipa-users/2014-February/msg00007.html*
But ... I had the same issue in the past, always in the 1st server. So, in my mind I don't want to try to use that fix. I have a new hypervisor cluster, so I would prefer to reinstall the 1st server, using these steps:
- check if all roles (also the CA) is installed in srv02
You can find here some data about the VMs:
*[root@srv01 ~]# ipa server-show srv01.ipa.mydomain.com http://srv01.ipa.mydomain.com Server name: srv01.ipa.mydomain.com http://srv01.ipa.mydomain.com Managed suffixes: domain, ca Min domain level: 0 Max domain level: 1 Enabled server roles: CA server, IPA master, DNS server, NTP server, AD trust controller [root@srv02 ~]# ipa server-show srv02.ipa.mydomain.com http://srv02.ipa.mydomain.com Server name: srv02.ipa.mydomain.com http://srv02.ipa.mydomain.com Managed suffixes: domain, ca Min domain level: 0 Max domain level: 1 Enabled server roles: CA server, IPA master, DNS server, NTP server [root@srv01 ~]# ipa config-show Maximum username length: 32 Home directory base: /home Default shell: /bin/bash Default users group: ipausers Default e-mail domain: ipa.mydomain.com http://ipa.mydomain.com Search time limit: 2 Search size limit: 100 User search fields: uid,givenname,sn,telephonenumber,ou,title Group search fields: cn,description Enable migration mode: FALSE Certificate Subject base: O=IPA.MYDOMAIN.COM http://IPA.MYDOMAIN.COM Password Expiration Notification (days): 4 Password plugin features: AllowNThash SELinux user map order: guest_u:s0$xguest_u:s0$user_u:s0$staff_u:s0-s0:c0.c1023$unconfined_u:s0-s0:c0.c1023 Default SELinux user: unconfined_u:s0-s0:c0.c1023 Default PAC types: MS-PAC, nfs:NONE IPA masters: srv01.ipa.mydomain.com http://srv01.ipa.mydomain.com, srv02.ipa.mydomain.com http://srv02.ipa.mydomain.com IPA CA servers: srv01.ipa.mydomain.com http://srv01.ipa.mydomain.com, srv02.ipa.mydomain.com http://srv02.ipa.mydomain.com IPA NTP servers: srv01.ipa.mydomain.com http://srv01.ipa.mydomain.com, srv02.ipa.mydomain.com http://srv02.ipa.mydomain.com IPA CA renewal master: srv01.ipa.mydomain.com http://srv01.ipa.mydomain.com [root@srv02 ~]# ipa config-show Maximum username length: 32 Home directory base: /home Default shell: /bin/bash Default users group: ipausers Default e-mail domain: ipa.mydomain.com http://ipa.mydomain.com Search time limit: 2 Search size limit: 100 User search fields: uid,givenname,sn,telephonenumber,ou,title Group search fields: cn,description Enable migration mode: FALSE Certificate Subject base: O=IPA.MYDOMAIN.COM http://IPA.MYDOMAIN.COM Password Expiration Notification (days): 4 Password plugin features: AllowNThash SELinux user map order: guest_u:s0$xguest_u:s0$user_u:s0$staff_u:s0-s0:c0.c1023$unconfined_u:s0-s0:c0.c1023 Default SELinux user: unconfined_u:s0-s0:c0.c1023 Default PAC types: MS-PAC, nfs:NONE IPA masters: srv01.ipa.mydomain.com http://srv01.ipa.mydomain.com, srv02.ipa.mydomain.com http://srv02.ipa.mydomain.com IPA CA servers: srv01.ipa.mydomain.com http://srv01.ipa.mydomain.com, srv02.ipa.mydomain.com http://srv02.ipa.mydomain.com IPA NTP servers: srv01.ipa.mydomain.com http://srv01.ipa.mydomain.com, srv02.ipa.mydomain.com http://srv02.ipa.mydomain.com IPA CA renewal master: srv01.ipa.mydomain.com http://srv01.ipa.mydomain.com [root@srv01 ~]# ipactl status Directory Service: RUNNING krb5kdc Service: RUNNING kadmin Service: RUNNING named Service: RUNNING httpd Service: RUNNING ipa-custodia Service: RUNNING ntpd Service: RUNNING pki-tomcatd Service: STOPPED smb Service: RUNNING winbind Service: RUNNING ipa-otpd Service: RUNNING ipa-dnskeysyncd Service: RUNNING ipa: INFO: The ipactl command was successful [root@srv02 ~]# ipactl status Directory Service: RUNNING krb5kdc Service: RUNNING kadmin Service: RUNNING named Service: RUNNING httpd Service: RUNNING ipa-custodia Service: RUNNING ntpd Service: RUNNING pki-tomcatd Service: STOPPED ipa-otpd Service: RUNNING ipa-dnskeysyncd Service: RUNNING ipa: INFO: The ipactl command was successful [root@srv01 ~]# certutil -L -d /etc/pki/pki-tomcat/alias Certificate Nickname Trust Attributes SSL,S/MIME,JAR/XPI Server-Cert cert-pki-ca u,u,u subsystemCert cert-pki-ca u,u,u caSigningCert cert-pki-ca CTu,Cu,Cu ocspSigningCert cert-pki-ca u,u,u auditSigningCert cert-pki-ca u,u,Pu [root@srv02 ~]# certutil -L -d /etc/pki/pki-tomcat/alias Certificate Nickname Trust Attributes SSL,S/MIME,JAR/XPI Server-Cert cert-pki-ca u,u,u subsystemCert cert-pki-ca u,u,u caSigningCert cert-pki-ca CTu,u,u ocspSigningCert cert-pki-ca u,u,u auditSigningCert cert-pki-ca u,u,Pu*
It seems that AD trust controller role, IPA CA renewal master, smb and windbind are only in the 1st server. And also caSigningCert cert-pki-ca entry is different (CTu,Cu,Cu vs CTu,u,u).
I can see only in the 1st server these DNS records:
*_kerberos._tcp.Default-First-Site-Name._sites.dc._msdcs SRV 0 100 88 srv01 _kerberos._tcp.dc._msdcs SRV 0 100 88 srv01 _kerberos._udp.Default-First-Site-Name._sites.dc._msdcs SRV 0 100 88 srv01 _kerberos._udp.dc._msdcs SRV 0 100 88 srv01 _ldap._tcp.Default-First-Site-Name._sites.dc._msdcs 0 100 389 srv01 _ldap._tcp.dc._msdcs 0 100 389 srv01*
Srv01 is the first master, I know, but is the server VM that has clock problems, in both situations. So I want to keep srv02 and install a new one.
What do I have to do to let the 2nd VM be a single server? Could I use these URLs?
*https://www.freeipa.org/page/Backup_and_Restore#One_Server_Loss_-_First_Mast... https://www.freeipa.org/page/Backup_and_Restore#One_Server_Loss_-_First_Master https://www.freeipa.org/page/V4/Server_Roles#Upgrade https://www.freeipa.org/page/V4/Server_Roles#Upgrade*
- uninstall ipa-server from the 1st server (srv01) and then powering
off it, assuming that all data into the 2nd one are ok (srv02)
update freeipa and all other RPM packages into the VM srv02
install a new fresh VM, always with 7 release, and create a new
replica Could I use the same old hostname (srv01) and IP address for this new VM? Or is better to use the same IP but a new name, like srv03?
Do you think this is the right way to solve my issue? Or do you have any better idea?
Please let me know, thanks. Bye, Morgan
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
On 8/10/20 11:51 AM, Morgan Marodin via FreeIPA-users wrote:
My issue got worse, the certificate has expired on the replica server, and it can't be renewed because it cannot communicate with the master server. I can start the server using the /--ignore-service-failure/ parameter, but the /httpd/ and the /pki-tomcatd/ failed to start ...
Do I have to start up with a fresh install? :(
It's often possible to repair a broken replica but it will be quicker to completely reinstall the replica. On the master, remove the replica: # kinit admin # ipa server-del <replicafqdn> --force
On the replica, uninstall ipa: ipa-server-install --uninstall -U kdestroy -A
On the replica, re-install: ipa-replica-install [options]
HTH, flo
Do you know any company that can give me support about FreeIPA?
Please let me know, thanks. Bye, Morgan
Il giorno lun 27 apr 2020 alle ore 12:16 Morgan Marodin <morgan@marodin.it mailto:morgan@marodin.it> ha scritto:
Sorry Thierry, any news for my issue? Did you see new logs? Could I use the command /ipa-replica-manage re-initialize --from srv02.ipa.mydomain.com <http://srv02.ipa.mydomain.com>/ in the First Master server? Or is it better to solve the issue in another way? Please let me know, thanks. Bye Il giorno gio 23 apr 2020 alle ore 09:54 Morgan Marodin <morgan@marodin.it <mailto:morgan@marodin.it>> ha scritto: Hi Theirry. To tell the truth my configuration was already set to on, on both VMs: /[root@srv01 ~]# ldapsearch -D "cn=Directory Manager" -h localhost -b "cn=config" -w $PASS | grep nsslapd-ignore-time-skew nsslapd-ignore-time-skew: on/ /[root@srv02 ~]# ldapsearch -D "cn=Directory Manager" -h localhost -b "cn=config" -w $PASS | grep nsslapd-ignore-time-skew nsslapd-ignore-time-skew: on/ Anyway, I tried to set up it to /off/ and then to /on/ again, but now I have a new issue into logs of the 2nd server :( Srv01 logs are similar as before: /[23/Apr/2020:09:28:45.958922636 +0200] - WARN - csngen_new_csn - Too much time skew (-22777212 secs). Current seqnum=5ca0/ Srv02 logs now are like these: /[23/Apr/2020:09:32:14.919328803 +0200] - ERR - agmt="cn=meTosrv01.ipa.mydomain.com <http://meTosrv01.ipa.mydomain.com>" (srv01:389) - clcache_load_buffer - Can't locate CSN 5e7cfe03000400030000 in the changelog (DB rc=-30988). If replication stops, the consumer may need to be reinitialized. [23/Apr/2020:09:32:14.920873489 +0200] - ERR - NSMMReplicationPlugin - changelog program - repl_plugin_name_cl - agmt="cn=meTosrv01.ipa.mydomain.com <http://meTosrv01.ipa.mydomain.com>" (srv01:389): CSN 5e7cfe03000400030000 not found, we aren't as up to date, or we purged [23/Apr/2020:09:32:14.922161821 +0200] - ERR - NSMMReplicationPlugin - send_updates - agmt="cn=meTosrv01.ipa.mydomain.com <http://meTosrv01.ipa.mydomain.com>" (srv01:389): Data required to update replica has been purged from the changelog. If the error persists the replica must be reinitialized./ I have just tried to force a replica on both sides, without success: /[root@srv01 ~]# ipa-replica-manage force-sync --from srv02.ipa.mydomain.com <http://srv02.ipa.mydomain.com> No status yet No status yet No status yet [root@srv02 ~]# ipa-replica-manage force-sync --from srv01.ipa.mydomain.com <http://srv01.ipa.mydomain.com> No status yet No status yet No status yet/ What could I do now? Thanks, bye Il giorno mer 22 apr 2020 alle ore 17:08 thierry bordaz via FreeIPA-users <freeipa-users@lists.fedorahosted.org <mailto:freeipa-users@lists.fedorahosted.org>> ha scritto: Hi Morgan, Sure. The most immediate and safest action is to do |dn: cn=config changetype: modify replace: nsslapd-ignore-time-skew nsslapd-ignore-time-skew: on | On all servers in the topology (no need to restart). Then monitor if replication is catching up. Okay NTP issues is likely the RC of your time skew but there is not easy way to prove it if any. best regards theirry On 4/22/20 3:16 PM, Morgan Marodin via FreeIPA-users wrote:Hi. I don't have access to RedHat portal :( There are similar articles in a public forum? Anyway ... could I stop ipa-server, change the value of /nsslapd-ignore-time-skew/ into //etc/dirsrv/slapd-IPA-MYDOMAIN-COM/dse.ldif/ and start again the server? Or is more complicated to change the configuration? VMs are local, but the cluster where the 1st server is running is affected by NTP problems ... For this reason I want to remove the First Master and install another replica in the new cluster. Thanks, bye. Morgan Il giorno mer 22 apr 2020 alle ore 11:33 thierry bordaz via FreeIPA-users <freeipa-users@lists.fedorahosted.org <mailto:freeipa-users@lists.fedorahosted.org>> ha scritto: Hi, CSN generator time skew is a pending issue still under investigation. At the moment the way your csn generator is messed up looks not fatal. You can allow replication to continue with the setting of nsslapd-ignore-time-skew on all servers. (https://access.redhat.com/solutions/1162703) If it does not allow replication to continue there is a recovery procedure but I would recommend to first try ignore-time-skew (https://access.redhat.com/solutions/3543811) NTP tuning or specific VMs are suspected to contribute to time skew. What type of VMs are you using (local or cloud (AWS)) ? best regards thierry On 4/21/20 5:42 PM, Morgan Marodin via FreeIPA-users wrote:Hi. Into my environment I have two IPA server, replicating each other. They are both 7.6 OS systems, ipa-server RPM version is 4.6.4-10.0.1.el7_6.2.x86_64. The first server installed was srv01 (many years ago), then I installed the replica into srv02 (like a year later the 1st node). When I had a single server I did also a trust with my corporate Active Directory. VMs are running in 2 different hypervisor clusters. Now the replication doesn't works. Into log files I have this error: /[16/Apr/2020:12:25:36.856632697 +0200] - ERR - csngen_adjust_time - Adjustment limit exceeded; value - 23221226, limit - 86400 [16/Apr/2020:12:25:36.857909222 +0200] - ERR - NSMMReplicationPlugin - repl5_inc_run - agmt="cn=meTosrv01.ipa.mydomain.com <http://meTosrv01.ipa.mydomain.com>" (srv01:389): Fatal error - too much time skew between replicas! [16/Apr/2020:12:25:36.862233147 +0200] - ERR - NSMMReplicationPlugin - repl5_inc_run - agmt="cn=meTosrv01.ipa.mydomain.com <http://meTosrv01.ipa.mydomain.com>" (srv01:389): Incremental update failed and requires administrator action/ I tried to force the replica, but the limit exceeded problem doesn't allow the sync. I know that the problem is that CSN generator has become grossly skewed. Using the external script readNsState.py I found that there was as offset time for about a month, so ... I waited for a month and then the issue disappeared. But now the offset is about 9 months ... I can't wait so much time :) /[root@srv01 scripts]# ./readNsState.py /etc/dirsrv/slapd-IPA-MYDOMAIN-COM/dse.ldif nsState is BAAAAAAAAACCN/xfAAAAAHbiBAAAAAAABCgAAAAAAAANdQAAAAAAAA== Little Endian For replica cn=replica,cn=dc\3Dipa\2Cdc\3Dmydomain\2Cdc\3Dcom,cn=mapping tree,cn=con fmtstr=[H6x3QH6x] size=40 len of nsstate is 40 CSN generator state: Replica ID : 4 Sampled Time : 1610364802 Gen as csn : 5ffc37822996500040000 *Time as str : Mon Jan 11 12:33:22 2021* Local Offset : 320118 Remote Offset : 10244 Seq. num : 29965 System time : Tue Apr 21 15:03:45 2020 Diff in sec. : -22890577 Day:sec diff : -265:5423 nsState is YAAAAAAAAAADLZheAAAAAAAAAAAAAAAAXSgAAAAAAAATAAAAAAAAAA== Little Endian For replica cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config fmtstr=[H6x3QH6x] size=40 len of nsstate is 40 CSN generator state: Replica ID : 96 Sampled Time : 1587031299 Gen as csn : 5e982d03001900960000 Time as str : Thu Apr 16 12:01:39 2020 Local Offset : 0 Remote Offset : 10333 Seq. num : 19 System time : Tue Apr 21 15:03:45 2020 Diff in sec. : 442926 Day:sec diff : 5:10926 [root@srv02 scripts]# ./readNsState.py /etc/dirsrv/slapd-IPA-MYDOMAIN-COM/dse.ldif nsState is AwAAAAAAAABU7p5eAAAAAAAAAAAAAAAAsVNiAQAAAAAAAAAAAAAAAA== Little Endian For replica cn=replica,cn=dc\3Dipa\2Cdc\3Dmydomain\2Cdc\3Dcom,cn=mapping tree,cn=con fmtstr=[H6x3QH6x] size=40 len of nsstate is 40 CSN generator state: Replica ID : 3 Sampled Time : 1587474004 Gen as csn : 5e9eee54000000030000 Time as str : Tue Apr 21 15:00:04 2020 Local Offset : 0 Remote Offset : 23221169 Seq. num : 0 System time : Tue Apr 21 15:02:38 2020 Diff in sec. : 154 Day:sec diff : 0:154 nsState is YQAAAAAAAAAuLZheAAAAAEUBAAAAAAAA7SYAAAAAAAASAAAAAAAAAA== Little Endian For replica cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config fmtstr=[H6x3QH6x] size=40 len of nsstate is 40 CSN generator state: Replica ID : 97 Sampled Time : 1587031342 Gen as csn : 5e982d2e001800970000 Time as str : Thu Apr 16 12:02:22 2020 Local Offset : 325 Remote Offset : 9965 Seq. num : 18 System time : Tue Apr 21 15:02:38 2020 Diff in sec. : 442816 Day:sec diff : 5:10816/ As you can see in the 1st node the Time as str is Jan 11 of 2021. With timedatectl command I see that both VMs use the same Time zone and the clock is correct. I found this old article to fix my issue: /https://www.redhat.com/archives/freeipa-users/2014-February/msg00007.html/ But ... I had the same issue in the past, always in the 1st server. So, in my mind I don't want to try to use that fix. I have a new hypervisor cluster, so I would prefer to reinstall the 1st server, using these steps: 1) check if all roles (also the CA) is installed in srv02 You can find here some data about the VMs: /[root@srv01 ~]# ipa server-show srv01.ipa.mydomain.com <http://srv01.ipa.mydomain.com> Server name: srv01.ipa.mydomain.com <http://srv01.ipa.mydomain.com> Managed suffixes: domain, ca Min domain level: 0 Max domain level: 1 Enabled server roles: CA server, IPA master, DNS server, NTP server, *AD trust controller* [root@srv02 ~]# ipa server-show srv02.ipa.mydomain.com <http://srv02.ipa.mydomain.com> Server name: srv02.ipa.mydomain.com <http://srv02.ipa.mydomain.com> Managed suffixes: domain, ca Min domain level: 0 Max domain level: 1 Enabled server roles: CA server, IPA master, DNS server, NTP server [root@srv01 ~]# ipa config-show Maximum username length: 32 Home directory base: /home Default shell: /bin/bash Default users group: ipausers Default e-mail domain: ipa.mydomain.com <http://ipa.mydomain.com> Search time limit: 2 Search size limit: 100 User search fields: uid,givenname,sn,telephonenumber,ou,title Group search fields: cn,description Enable migration mode: FALSE Certificate Subject base: O=IPA.MYDOMAIN.COM <http://IPA.MYDOMAIN.COM> Password Expiration Notification (days): 4 Password plugin features: AllowNThash SELinux user map order: guest_u:s0$xguest_u:s0$user_u:s0$staff_u:s0-s0:c0.c1023$unconfined_u:s0-s0:c0.c1023 Default SELinux user: unconfined_u:s0-s0:c0.c1023 Default PAC types: MS-PAC, nfs:NONE IPA masters: srv01.ipa.mydomain.com <http://srv01.ipa.mydomain.com>, srv02.ipa.mydomain.com <http://srv02.ipa.mydomain.com> IPA CA servers: srv01.ipa.mydomain.com <http://srv01.ipa.mydomain.com>, srv02.ipa.mydomain.com <http://srv02.ipa.mydomain.com> IPA NTP servers: srv01.ipa.mydomain.com <http://srv01.ipa.mydomain.com>, srv02.ipa.mydomain.com <http://srv02.ipa.mydomain.com> IPA CA renewal master: srv01.ipa.mydomain.com <http://srv01.ipa.mydomain.com> [root@srv02 ~]# ipa config-show Maximum username length: 32 Home directory base: /home Default shell: /bin/bash Default users group: ipausers Default e-mail domain: ipa.mydomain.com <http://ipa.mydomain.com> Search time limit: 2 Search size limit: 100 User search fields: uid,givenname,sn,telephonenumber,ou,title Group search fields: cn,description Enable migration mode: FALSE Certificate Subject base: O=IPA.MYDOMAIN.COM <http://IPA.MYDOMAIN.COM> Password Expiration Notification (days): 4 Password plugin features: AllowNThash SELinux user map order: guest_u:s0$xguest_u:s0$user_u:s0$staff_u:s0-s0:c0.c1023$unconfined_u:s0-s0:c0.c1023 Default SELinux user: unconfined_u:s0-s0:c0.c1023 Default PAC types: MS-PAC, nfs:NONE IPA masters: srv01.ipa.mydomain.com <http://srv01.ipa.mydomain.com>, srv02.ipa.mydomain.com <http://srv02.ipa.mydomain.com> IPA CA servers: srv01.ipa.mydomain.com <http://srv01.ipa.mydomain.com>, srv02.ipa.mydomain.com <http://srv02.ipa.mydomain.com> IPA NTP servers: srv01.ipa.mydomain.com <http://srv01.ipa.mydomain.com>, srv02.ipa.mydomain.com <http://srv02.ipa.mydomain.com> *IPA CA renewal master: srv01.ipa.mydomain.com <http://srv01.ipa.mydomain.com>* [root@srv01 ~]# ipactl status Directory Service: RUNNING krb5kdc Service: RUNNING kadmin Service: RUNNING named Service: RUNNING httpd Service: RUNNING ipa-custodia Service: RUNNING ntpd Service: RUNNING pki-tomcatd Service: STOPPED *smb Service: RUNNING winbind Service: RUNNING* ipa-otpd Service: RUNNING ipa-dnskeysyncd Service: RUNNING ipa: INFO: The ipactl command was successful [root@srv02 ~]# ipactl status Directory Service: RUNNING krb5kdc Service: RUNNING kadmin Service: RUNNING named Service: RUNNING httpd Service: RUNNING ipa-custodia Service: RUNNING ntpd Service: RUNNING pki-tomcatd Service: STOPPED ipa-otpd Service: RUNNING ipa-dnskeysyncd Service: RUNNING ipa: INFO: The ipactl command was successful [root@srv01 ~]# certutil -L -d /etc/pki/pki-tomcat/alias Certificate Nickname Trust Attributes SSL,S/MIME,JAR/XPI Server-Cert cert-pki-ca u,u,u subsystemCert cert-pki-ca u,u,u caSigningCert cert-pki-ca *CTu,Cu,Cu* ocspSigningCert cert-pki-ca u,u,u auditSigningCert cert-pki-ca u,u,Pu [root@srv02 ~]# certutil -L -d /etc/pki/pki-tomcat/alias Certificate Nickname Trust Attributes SSL,S/MIME,JAR/XPI Server-Cert cert-pki-ca u,u,u subsystemCert cert-pki-ca u,u,u caSigningCert cert-pki-ca *CTu,u,u* ocspSigningCert cert-pki-ca u,u,u auditSigningCert cert-pki-ca u,u,Pu/ It seems that AD trust controller role, IPA CA renewal master, smb and windbind are only in the 1st server. And also caSigningCert cert-pki-ca entry is different (CTu,Cu,Cu vs CTu,u,u). I can see only in the 1st server these DNS records: /_kerberos._tcp.Default-First-Site-Name._sites.dc._msdcs SRV 0 100 88 srv01 _kerberos._tcp.dc._msdcs SRV 0 100 88 srv01 _kerberos._udp.Default-First-Site-Name._sites.dc._msdcs SRV 0 100 88 srv01 _kerberos._udp.dc._msdcs SRV 0 100 88 srv01 _ldap._tcp.Default-First-Site-Name._sites.dc._msdcs 0 100 389 srv01 _ldap._tcp.dc._msdcs 0 100 389 srv01/ Srv01 is the first master, I know, but is the server VM that has clock problems, in both situations. So I want to keep srv02 and install a new one. What do I have to do to let the 2nd VM be a single server? Could I use these URLs? /https://www.freeipa.org/page/Backup_and_Restore#One_Server_Loss_-_First_Master https://www.freeipa.org/page/V4/Server_Roles#Upgrade/ 2) uninstall ipa-server from the 1st server (srv01) and then powering off it, assuming that all data into the 2nd one are ok (srv02) 3) update freeipa and all other RPM packages into the VM srv02 4) install a new fresh VM, always with 7 release, and create a new replica Could I use the same old hostname (srv01) and IP address for this new VM? Or is better to use the same IP but a new name, like srv03? Do you think this is the right way to solve my issue? Or do you have any better idea? Please let me know, thanks. Bye, Morgan _______________________________________________ FreeIPA-users mailing list --freeipa-users@lists.fedorahosted.org <mailto:freeipa-users@lists.fedorahosted.org> To unsubscribe send an email tofreeipa-users-leave@lists.fedorahosted.org <mailto:freeipa-users-leave@lists.fedorahosted.org> Fedora Code of Conduct:https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines:https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives:https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org_______________________________________________ FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org <mailto:freeipa-users@lists.fedorahosted.org> To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org <mailto:freeipa-users-leave@lists.fedorahosted.org> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org _______________________________________________ FreeIPA-users mailing list --freeipa-users@lists.fedorahosted.org <mailto:freeipa-users@lists.fedorahosted.org> To unsubscribe send an email tofreeipa-users-leave@lists.fedorahosted.org <mailto:freeipa-users-leave@lists.fedorahosted.org> Fedora Code of Conduct:https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines:https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives:https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org_______________________________________________ FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org <mailto:freeipa-users@lists.fedorahosted.org> To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org <mailto:freeipa-users-leave@lists.fedorahosted.org> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
Hi Flo.
I know, but the master has the CSN issue (january 2021) and the replica has the expired certificate! :| Maybe it will be good to delete the replica from the master, uninstall the replica, and then maybe the CSN Generator issue will disappear ... without the replica. What do you think about it?
Please let me know, thanks. Morgan
Il giorno mar 11 ago 2020 alle ore 09:02 Florence Blanc-Renaud via FreeIPA-users freeipa-users@lists.fedorahosted.org ha scritto:
On 8/10/20 11:51 AM, Morgan Marodin via FreeIPA-users wrote:
My issue got worse, the certificate has expired on the replica server, and it can't be renewed because it cannot communicate with the master server. I can start the server using the /--ignore-service-failure/ parameter, but the /httpd/ and the /pki-tomcatd/ failed to start ...
Do I have to start up with a fresh install? :(
It's often possible to repair a broken replica but it will be quicker to completely reinstall the replica. On the master, remove the replica: # kinit admin # ipa server-del <replicafqdn> --force
On the replica, uninstall ipa: ipa-server-install --uninstall -U kdestroy -A
On the replica, re-install: ipa-replica-install [options]
HTH, flo
Do you know any company that can give me support about FreeIPA?
Please let me know, thanks. Bye, Morgan
Il giorno lun 27 apr 2020 alle ore 12:16 Morgan Marodin <morgan@marodin.it mailto:morgan@marodin.it> ha scritto:
Sorry Thierry, any news for my issue? Did you see new logs? Could I use the command /ipa-replica-manage re-initialize --from srv02.ipa.mydomain.com <http://srv02.ipa.mydomain.com>/ in the First Master server? Or is it better to solve the issue in another way? Please let me know, thanks. Bye Il giorno gio 23 apr 2020 alle ore 09:54 Morgan Marodin <morgan@marodin.it <mailto:morgan@marodin.it>> ha scritto: Hi Theirry. To tell the truth my configuration was already set to on, on both VMs: /[root@srv01 ~]# ldapsearch -D "cn=Directory Manager" -h localhost -b "cn=config" -w $PASS | grep nsslapd-ignore-time-skew nsslapd-ignore-time-skew: on/ /[root@srv02 ~]# ldapsearch -D "cn=Directory Manager" -h localhost -b "cn=config" -w $PASS | grep nsslapd-ignore-time-skew nsslapd-ignore-time-skew: on/ Anyway, I tried to set up it to /off/ and then to /on/ again, but now I have a new issue into logs of the 2nd server :( Srv01 logs are similar as before: /[23/Apr/2020:09:28:45.958922636 +0200] - WARN - csngen_new_csn - Too much time skew (-22777212 secs). Current seqnum=5ca0/ Srv02 logs now are like these: /[23/Apr/2020:09:32:14.919328803 +0200] - ERR - agmt="cn=meTosrv01.ipa.mydomain.com <http://meTosrv01.ipa.mydomain.com>" (srv01:389) - clcache_load_buffer - Can't locate CSN 5e7cfe03000400030000 in the changelog (DB rc=-30988). If replication stops, the consumer may need to be reinitialized. [23/Apr/2020:09:32:14.920873489 +0200] - ERR - NSMMReplicationPlugin - changelog program - repl_plugin_name_cl - agmt="cn=meTosrv01.ipa.mydomain.com <http://meTosrv01.ipa.mydomain.com>" (srv01:389): CSN 5e7cfe03000400030000 not found, we aren't as up to date, or we purged [23/Apr/2020:09:32:14.922161821 +0200] - ERR - NSMMReplicationPlugin - send_updates - agmt="cn=meTosrv01.ipa.mydomain.com <http://meTosrv01.ipa.mydomain.com>" (srv01:389): Data required to update replica has been purged from the changelog. If the error persists the replica must be reinitialized./ I have just tried to force a replica on both sides, withoutsuccess:
/[root@srv01 ~]# ipa-replica-manage force-sync --from srv02.ipa.mydomain.com <http://srv02.ipa.mydomain.com> No status yet No status yet No status yet [root@srv02 ~]# ipa-replica-manage force-sync --from srv01.ipa.mydomain.com <http://srv01.ipa.mydomain.com> No status yet No status yet No status yet/ What could I do now? Thanks, bye Il giorno mer 22 apr 2020 alle ore 17:08 thierry bordaz via FreeIPA-users <freeipa-users@lists.fedorahosted.org <mailto:freeipa-users@lists.fedorahosted.org>> ha scritto: Hi Morgan, Sure. The most immediate and safest action is to do |dn: cn=config changetype: modify replace: nsslapd-ignore-time-skew nsslapd-ignore-time-skew: on | On all servers in the topology (no need to restart). Then monitor if replication is catching up. Okay NTP issues is likely the RC of your time skew but there is not easy way to prove it if any. best regards theirry On 4/22/20 3:16 PM, Morgan Marodin via FreeIPA-users wrote:Hi. I don't have access to RedHat portal :( There are similar articles in a public forum? Anyway ... could I stop ipa-server, change the value of /nsslapd-ignore-time-skew/ into //etc/dirsrv/slapd-IPA-MYDOMAIN-COM/dse.ldif/ and start again the server? Or is more complicated to change the configuration? VMs are local, but the cluster where the 1st server is running is affected by NTP problems ... For this reason I want to remove the First Master and install another replica in the new cluster. Thanks, bye. Morgan Il giorno mer 22 apr 2020 alle ore 11:33 thierry bordaz via FreeIPA-users <freeipa-users@lists.fedorahosted.org <mailto:freeipa-users@lists.fedorahosted.org>> ha scritto: Hi, CSN generator time skew is a pending issue still under investigation. At the moment the way your csn generator is messed up looks not fatal. You can allow replication to continue with the setting of nsslapd-ignore-time-skew on all servers. (https://access.redhat.com/solutions/1162703) If it does not allow replication to continue there is a recovery procedure but I would recommend to first try ignore-time-skew (https://access.redhat.com/solutions/3543811) NTP tuning or specific VMs are suspected to contribute to time skew. What type of VMs are you using (local or cloud (AWS)) ? best regards thierry On 4/21/20 5:42 PM, Morgan Marodin via FreeIPA-users wrote:Hi. Into my environment I have two IPA server, replicating each other. They are both 7.6 OS systems, ipa-server RPM version is 4.6.4-10.0.1.el7_6.2.x86_64. The first server installed was srv01 (many years ago), then I installed the replica into srv02 (like a year later the 1st node). When I had a single server I did also a trust with my corporate Active Directory. VMs are running in 2 different hypervisor clusters. Now the replication doesn't works. Into log files I have this error: /[16/Apr/2020:12:25:36.856632697 +0200] - ERR - csngen_adjust_time - Adjustment limit exceeded; value - 23221226, limit - 86400 [16/Apr/2020:12:25:36.857909222 +0200] - ERR - NSMMReplicationPlugin - repl5_inc_run - agmt="cn=meTosrv01.ipa.mydomain.com <http://meTosrv01.ipa.mydomain.com>" (srv01:389): Fatal error - too much time skew between replicas! [16/Apr/2020:12:25:36.862233147 +0200] - ERR - NSMMReplicationPlugin - repl5_inc_run - agmt="cn=meTosrv01.ipa.mydomain.com <http://meTosrv01.ipa.mydomain.com>" (srv01:389): Incremental update failed and requires administrator action/ I tried to force the replica, but the limit exceeded problem doesn't allow the sync. I know that the problem is that CSN generator has become grossly skewed. Using the external script readNsState.py I found that there was as offset time for about a month, so ... I waited for a month and then the issue disappeared. But now the offset is about 9 months ... I can't wait so much time :) /[root@srv01 scripts]# ./readNsState.py /etc/dirsrv/slapd-IPA-MYDOMAIN-COM/dse.ldif nsState isBAAAAAAAAACCN/xfAAAAAHbiBAAAAAAABCgAAAAAAAANdQAAAAAAAA==
Little Endian For replicacn=replica,cn=dc\3Dipa\2Cdc\3Dmydomain\2Cdc\3Dcom,cn=mapping
tree,cn=con fmtstr=[H6x3QH6x] size=40 len of nsstate is 40 CSN generator state: Replica ID : 4 Sampled Time : 1610364802 Gen as csn : 5ffc37822996500040000 *Time as str : Mon Jan 11 12:33:22 2021* Local Offset : 320118 Remote Offset : 10244 Seq. num : 29965 System time : Tue Apr 21 15:03:45 2020 Diff in sec. : -22890577 Day:sec diff : -265:5423 nsState isYAAAAAAAAAADLZheAAAAAAAAAAAAAAAAXSgAAAAAAAATAAAAAAAAAA==
Little Endian For replica cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config fmtstr=[H6x3QH6x] size=40 len of nsstate is 40 CSN generator state: Replica ID : 96 Sampled Time : 1587031299 Gen as csn : 5e982d03001900960000 Time as str : Thu Apr 16 12:01:39 2020 Local Offset : 0 Remote Offset : 10333 Seq. num : 19 System time : Tue Apr 21 15:03:45 2020 Diff in sec. : 442926 Day:sec diff : 5:10926 [root@srv02 scripts]# ./readNsState.py /etc/dirsrv/slapd-IPA-MYDOMAIN-COM/dse.ldif nsState isAwAAAAAAAABU7p5eAAAAAAAAAAAAAAAAsVNiAQAAAAAAAAAAAAAAAA==
Little Endian For replicacn=replica,cn=dc\3Dipa\2Cdc\3Dmydomain\2Cdc\3Dcom,cn=mapping
tree,cn=con fmtstr=[H6x3QH6x] size=40 len of nsstate is 40 CSN generator state: Replica ID : 3 Sampled Time : 1587474004 Gen as csn : 5e9eee54000000030000 Time as str : Tue Apr 21 15:00:04 2020 Local Offset : 0 Remote Offset : 23221169 Seq. num : 0 System time : Tue Apr 21 15:02:38 2020 Diff in sec. : 154 Day:sec diff : 0:154 nsState isYQAAAAAAAAAuLZheAAAAAEUBAAAAAAAA7SYAAAAAAAASAAAAAAAAAA==
Little Endian For replica cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config fmtstr=[H6x3QH6x] size=40 len of nsstate is 40 CSN generator state: Replica ID : 97 Sampled Time : 1587031342 Gen as csn : 5e982d2e001800970000 Time as str : Thu Apr 16 12:02:22 2020 Local Offset : 325 Remote Offset : 9965 Seq. num : 18 System time : Tue Apr 21 15:02:38 2020 Diff in sec. : 442816 Day:sec diff : 5:10816/ As you can see in the 1st node the Time as str is Jan 11 of 2021. With timedatectl command I see that both VMs use the same Time zone and the clock is correct. I found this old article to fix my issue: /https://www.redhat.com/archives/freeipa-users/2014-February/msg00007.html/
But ... I had the same issue in the past, always in the 1st server. So, in my mind I don't want to try to use that fix. I have a new hypervisor cluster, so I would prefer to reinstall the 1st server, using these steps: 1) check if all roles (also the CA) is installed insrv02
You can find here some data about the VMs: /[root@srv01 ~]# ipa server-show srv01.ipa.mydomain.com <http://srv01.ipa.mydomain.com> Server name: srv01.ipa.mydomain.com <http://srv01.ipa.mydomain.com> Managed suffixes: domain, ca Min domain level: 0 Max domain level: 1 Enabled server roles: CA server, IPA master, DNS server, NTP server, *AD trust controller* [root@srv02 ~]# ipa server-show srv02.ipa.mydomain.com <http://srv02.ipa.mydomain.com> Server name: srv02.ipa.mydomain.com <http://srv02.ipa.mydomain.com> Managed suffixes: domain, ca Min domain level: 0 Max domain level: 1 Enabled server roles: CA server, IPA master, DNS server, NTP server [root@srv01 ~]# ipa config-show Maximum username length: 32 Home directory base: /home Default shell: /bin/bash Default users group: ipausers Default e-mail domain: ipa.mydomain.com <http://ipa.mydomain.com> Search time limit: 2 Search size limit: 100 User search fields: uid,givenname,sn,telephonenumber,ou,title Group search fields: cn,description Enable migration mode: FALSE Certificate Subject base: O=IPA.MYDOMAIN.COM <http://IPA.MYDOMAIN.COM> Password Expiration Notification (days): 4 Password plugin features: AllowNThash SELinux user map order:guest_u:s0$xguest_u:s0$user_u:s0$staff_u:s0-s0:c0.c1023$unconfined_u:s0-s0:c0.c1023
Default SELinux user: unconfined_u:s0-s0:c0.c1023 Default PAC types: MS-PAC, nfs:NONE IPA masters: srv01.ipa.mydomain.com <http://srv01.ipa.mydomain.com>, srv02.ipa.mydomain.com <http://srv02.ipa.mydomain.com> IPA CA servers: srv01.ipa.mydomain.com <http://srv01.ipa.mydomain.com>, srv02.ipa.mydomain.com <http://srv02.ipa.mydomain.com> IPA NTP servers: srv01.ipa.mydomain.com <http://srv01.ipa.mydomain.com>, srv02.ipa.mydomain.com <http://srv02.ipa.mydomain.com> IPA CA renewal master: srv01.ipa.mydomain.com <http://srv01.ipa.mydomain.com> [root@srv02 ~]# ipa config-show Maximum username length: 32 Home directory base: /home Default shell: /bin/bash Default users group: ipausers Default e-mail domain: ipa.mydomain.com <http://ipa.mydomain.com> Search time limit: 2 Search size limit: 100 User search fields: uid,givenname,sn,telephonenumber,ou,title Group search fields: cn,description Enable migration mode: FALSE Certificate Subject base: O=IPA.MYDOMAIN.COM <http://IPA.MYDOMAIN.COM> Password Expiration Notification (days): 4 Password plugin features: AllowNThash SELinux user map order:guest_u:s0$xguest_u:s0$user_u:s0$staff_u:s0-s0:c0.c1023$unconfined_u:s0-s0:c0.c1023
Default SELinux user: unconfined_u:s0-s0:c0.c1023 Default PAC types: MS-PAC, nfs:NONE IPA masters: srv01.ipa.mydomain.com <http://srv01.ipa.mydomain.com>, srv02.ipa.mydomain.com <http://srv02.ipa.mydomain.com> IPA CA servers: srv01.ipa.mydomain.com <http://srv01.ipa.mydomain.com>, srv02.ipa.mydomain.com <http://srv02.ipa.mydomain.com> IPA NTP servers: srv01.ipa.mydomain.com <http://srv01.ipa.mydomain.com>, srv02.ipa.mydomain.com <http://srv02.ipa.mydomain.com> *IPA CA renewal master: srv01.ipa.mydomain.com <http://srv01.ipa.mydomain.com>* [root@srv01 ~]# ipactl status Directory Service: RUNNING krb5kdc Service: RUNNING kadmin Service: RUNNING named Service: RUNNING httpd Service: RUNNING ipa-custodia Service: RUNNING ntpd Service: RUNNING pki-tomcatd Service: STOPPED *smb Service: RUNNING winbind Service: RUNNING* ipa-otpd Service: RUNNING ipa-dnskeysyncd Service: RUNNING ipa: INFO: The ipactl command was successful [root@srv02 ~]# ipactl status Directory Service: RUNNING krb5kdc Service: RUNNING kadmin Service: RUNNING named Service: RUNNING httpd Service: RUNNING ipa-custodia Service: RUNNING ntpd Service: RUNNING pki-tomcatd Service: STOPPED ipa-otpd Service: RUNNING ipa-dnskeysyncd Service: RUNNING ipa: INFO: The ipactl command was successful [root@srv01 ~]# certutil -L -d/etc/pki/pki-tomcat/alias
Certificate Nickname Trust Attributes SSL,S/MIME,JAR/XPI Server-Cert cert-pki-ca u,u,u subsystemCert cert-pki-ca u,u,u caSigningCert cert-pki-ca *CTu,Cu,Cu* ocspSigningCert cert-pki-ca u,u,u auditSigningCert cert-pki-ca u,u,Pu [root@srv02 ~]# certutil -L -d/etc/pki/pki-tomcat/alias
Certificate Nickname Trust Attributes SSL,S/MIME,JAR/XPI Server-Cert cert-pki-ca u,u,u subsystemCert cert-pki-ca u,u,u caSigningCert cert-pki-ca *CTu,u,u* ocspSigningCert cert-pki-ca u,u,u auditSigningCert cert-pki-ca u,u,Pu/ It seems that AD trust controller role, IPA CA renewal master, smb and windbind are only in the 1st server. And also caSigningCert cert-pki-ca entry is different (CTu,Cu,Cu vs CTu,u,u). I can see only in the 1st server these DNS records:/_kerberos._tcp.Default-First-Site-Name._sites.dc._msdcs
SRV 0 100 88 srv01 _kerberos._tcp.dc._msdcs SRV 0 100 88 srv01 _kerberos._udp.Default-First-Site-Name._sites.dc._msdcs SRV 0 100 88 srv01 _kerberos._udp.dc._msdcs SRV 0 100 88 srv01 _ldap._tcp.Default-First-Site-Name._sites.dc._msdcs 0 100 389 srv01 _ldap._tcp.dc._msdcs 0 100 389 srv01/ Srv01 is the first master, I know, but is the server VM that has clock problems, in both situations. So I want to keep srv02 and install a new one. What do I have to do to let the 2nd VM be a single server? Could I use these URLs? /https://www.freeipa.org/page/Backup_and_Restore#One_Server_Loss_-_First_Mast...
https://www.freeipa.org/page/V4/Server_Roles#Upgrade/ 2) uninstall ipa-server from the 1st server (srv01) and then powering off it, assuming that all data into the 2nd one are ok (srv02) 3) update freeipa and all other RPM packages into the VM srv02 4) install a new fresh VM, always with 7 release, and create a new replica Could I use the same old hostname (srv01) and IP address for this new VM? Or is better to use the same IP but a new name, like srv03? Do you think this is the right way to solve my issue? Or do you have any better idea? Please let me know, thanks. Bye, Morgan _______________________________________________ FreeIPA-users mailing list --freeipa-users@lists.fedorahosted.org mailto: freeipa-users@lists.fedorahosted.org
To unsubscribe send an emailtofreeipa-users-leave@lists.fedorahosted.org mailto: freeipa-users-leave@lists.fedorahosted.org
Fedora Code of Conduct:https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines:https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
_______________________________________________ FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org <mailto:freeipa-users@lists.fedorahosted.org> To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org <mailto:freeipa-users-leave@lists.fedorahosted.org> Fedora Code of Conduct:https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives:https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
_______________________________________________ FreeIPA-users mailing list --freeipa-users@lists.fedorahosted.org mailto: freeipa-users@lists.fedorahosted.org
To unsubscribe send an emailtofreeipa-users-leave@lists.fedorahosted.org mailto: freeipa-users-leave@lists.fedorahosted.org
Fedora Code of Conduct:https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines:https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
_______________________________________________ FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org <mailto:freeipa-users@lists.fedorahosted.org> To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org <mailto:freeipa-users-leave@lists.fedorahosted.org> Fedora Code of Conduct:https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives:https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to
freeipa-users-leave@lists.fedorahosted.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives:
https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
freeipa-users@lists.fedorahosted.org
-
Florence Blanc-Renaud -
Morgan Marodin -
thierry bordaz