Hi German,
On Tue, 12 Feb 2019, German Parente via FreeIPA-users wrote:
well, there's still a possibility to remove it manually. it's
rather
easy.
ldapsearch -D "cn=directory manager" -W -b
"cn=topology,cn=ipa,cn=etc,dc=example,dc=com"
that will show all the entries in the topology subtree. You will find
the one with "left-right" or "right-left" connectivity.
Before deleting it, do this:
ldapmodify -D "cn=directory manager" -W << EOF
dn: cn=IPA Topology Configuration,cn=plugins,cn=config
changetype: modify
replace: nsslapd-pluginEnabled
nsslapd-pluginEnabled: off
EOF
restart services.
Delete the wrong entry with "ldapdelete" command.
then, do this:
ldapmodify -D "cn=directory manager" -W << EOF
dn: cn=IPA Topology Configuration,cn=plugins,cn=config
changetype: modify
replace: nsslapd-pluginEnabled
nsslapd-pluginEnabled: on
EOF
restart services.
Check your segments again.
if you have a subscription, please open a support case, ask for my help
and I will fix that in your machines in a remote session.
thanks again for your hints. I guess, i'm out of the woods now:
---
$ ipa topologysegment-find domain
-----------------
1 segment matched
-----------------
Segment name:
ipa2.example.com-to-ipa1.example.com
Left node:
ipa2.example.com
Right node:
ipa1.example.com
Connectivity: both
----------------------------
Number of entries returned 1
----------------------------
$ ipa topologysegment-find ca
------------------
2 segments matched
------------------
Segment name:
ipa2.example.com-to-ipa1.example.com
Left node:
ipa2.example.com
Right node:
ipa1.example.com
Connectivity: both
Segment name:
ipa1.example.com-to-ipa2.example.com
Left node:
ipa1.example.com
Right node:
ipa2.example.com
Connectivity: both
----------------------------
Number of entries returned 2
----------------------------
$ ipa-replica-manage -v list
ipa1.example.com
ipa2.example.com: replica
last init status: Error (0) Total update succeeded
last init ended: 2019-01-11 20:02:40+00:00
last update status: Error (0) Replica acquired successfully: Incremental update
succeeded
last update ended: 2019-02-14 09:05:01+00:00
$ ipa-replica-manage -v list
ipa2.example.com
ipa1.example.com: replica
last init status: None
last init ended: 1970-01-01 00:00:00+00:00
last update status: Error (0) Replica acquired successfully: Incremental update
succeeded
last update ended: 2019-02-14 09:05:00+00:00
$ ipa-csreplica-manage -v list
ipa1.example.com
ipa2.example.com
last init status: Error (0) Total update succeeded
last init ended: 2019-02-13 13:43:48+00:00
last update status: Error (0) Replica acquired successfully: Incremental update
succeeded
last update ended: 2019-02-14 09:02:30+00:00
$ ipa-csreplica-manage -v list
ipa2.example.com
ipa1.example.com
last init status: None
last init ended: 1970-01-01 00:00:00+00:00
last update status: Error (0) Replica acquired successfully: Incremental update
succeeded
last update ended: 2019-02-14 08:57:30+00:00
---
Identical output on both ipa1 and ipa2. In addition to your advice, I had
to re-create segment
ipa1.example.com-to-ipa2.example.com using
topologysegment-add and I also did a
---
$ ipa-csreplica-manage re-initialize --from
ipa1.example.com
---
on ipa2.
I tried to remove the
ipa2.example.com-to-ipa1.example.com ca segment
using your recipe, but that breaks things, I had to re-create it
afterwards. Error messages in dirsrv logs gone.
It remains unclear to me, why removing the 2nd ca segment didn't work - so
just to make sure: One "Connectivity: both"-segment for each domain and ca
is sufficient (for my two-master-only-topology), right? Having 2 ca
segments won't hurt?
Once again: Your help was invaluable.
On Tue, Feb 12, 2019 at 9:17 AM dbischof--- via FreeIPA-users
<freeipa-users(a)lists.fedorahosted.org> wrote:
> On Mon, 11 Feb 2019, German Parente via FreeIPA-users wrote:
>
>> in fact, there's no sense to have "two segments" one from ipa1
<--> ipa2
>> and other ipa1 --> ipa2.
>>
>> you should delete the segment that is showing "right-left"
connectivity.
>
> that doesn't work, i tried that already:
>
> ---
> $ ipa topologysegment-del ca
ipa1.example.com-to-ipa2.example.com
> ipa: ERROR: Server is unwilling to perform: Removal of Segment disconnects
> topology. Deletion not allowed.
> ---
>
> Tried on both masters.
>
>> On Mon, Feb 11, 2019 at 1:47 PM dbischof--- via FreeIPA-users <
>> freeipa-users(a)lists.fedorahosted.org> wrote:
>>
>>> On Mon, 11 Feb 2019, German Parente via FreeIPA-users wrote:
>>>
>>>> don't forget "-r" to export. If not, replication metadata
will not be
>>>> exported and after the import, the replicas will not be in sync.
>>>
>>> thank you for your hints.
>>>
>>> Unfortunately, the replication/topology problem remains unsolved.
>>>
>>> Here's what i did:
>>>
>>> --- ipa1 (IPA running)
>>> db2ldif.pl -Z EXAMPLE-COM -D "cn=Directory Manager" -r -w - -n
ipaca -a
>>> /tmp/foo.dif
>>> ---
>>>
>>> Copied the file over to ipa2, then
>>>
>>> --- ipa2 (IPA not running)
>>> ldif2db -Z EXAMPLE-COM -n ipaca -i foo.dif
>>> ---
>>>
>>> Started IPA on ipa2, but still
>>>
>>> ---
>>> $ ipa topologysegment-find ca
>>> ------------------
>>> 2 segments matched
>>> ------------------
>>> Segment name:
ipa2.example.com-to-ipa1.example.com
>>> Left node:
ipa2.example.com
>>> Right node:
ipa1.example.com
>>> Connectivity: both
>>>
>>> Segment name:
ipa1.example.com-to-ipa2.example.com
>>> Left node:
ipa1.example.com
>>> Right node:
ipa2.example.com
>>> Connectivity: left-right
>>> ----------------------------
>>> Number of entries returned 2
>>> ----------------------------
>>>
>>> In case there's nothing obvious and easy left to be tried out, I'd
>>> consider to uninstall IPA on ipa2, reinstall as client and promote
>>> ipa2 to master again as described in the docs.
>>>
>>>> On Thu, Feb 7, 2019 at 3:46 PM dbischof--- via FreeIPA-users <
>>>> freeipa-users(a)lists.fedorahosted.org> wrote:
>>>>
>>>>> On Wed, 6 Feb 2019, German Parente via FreeIPA-users wrote:
>>>>>
>>>>>> this is a bug in the product that might have been fixed already:
>>>>>>
>>>>>> Connectivity: left-right
>>>>>>
>>>>>> we cannot have these sort of connectivity.
>>>>>>
>>>>>> In ipa02 there's no replication agreement to ipa01 (for
ipaca
>>>>>> database).
>>>>>>
>>>>>> But as in ipa01 we see that the topology is showing
"both" in the
>>>>>> connectivity, I suggest to do export-import "off line"
of the
>>>>>> database. Then the topology subtree will be set in ipa02, exactly
as
>>>>>> in ipa01, and the topology plugin will create automatically the
>>>>>> replication agreement that is missing now.
>>>>>>
>>>>>> export from ipa01 the backend ipaca and re-import it in ipa02.
Then,
>>>>>> start the server and check if now it's showing
"both" in connectivity
>>>>>> at ipa02 side.
>>>>>
>>>>> thank you for your hints.
>>>>>
>>>>> Unfortunately, I never did something like this before (and I
can't
>>>>> access the article you cited below). According to the Directory
>>>>> Manager docs, it's probably something like
>>>>>
>>>>> ---
>>>>> db2ldif.pl -Z EXAMPLE-COM -D "cn=Directory Manager" -w - -n
ipaca -a /tmp/foo.dif
>>>>> ---
>>>>>
>>>>> to export on running ipa1 and
>>>>>
>>>>> ---
>>>>> ldif2db -Z EXAMPLE-COM -n ipaca -i /tmp/foo.dif
>>>>> ---
>>>>>
>>>>> to import on ipa2 with IPA not running, right? Something else to be
>>>>> taken into account to not break something (these are production
>>>>> servers - my group is small but vigorous ;-)
>>>>>
>>>>>> On Wed, Feb 6, 2019 at 4:57 PM dbischof--- via FreeIPA-users
>>>>>> <freeipa-users(a)lists.fedorahosted.org> wrote:
>>>>>>
>>>>>>> On Wed, 6 Feb 2019, German Parente via FreeIPA-users wrote:
>>>>>>>
>>>>>>>> have you tried to use "ipa-csreplica-manage
re-initialize --from
>>>>>>>> <replica1>" in replica1 ?
>>>>>>>
>>>>>>> Thanks for your answer.
>>>>>>>
>>>>>>> I already tried (on ipa2)
>>>>>>>
>>>>>>> ---
>>>>>>> $ ipa-csreplica-manage re-initialize --from
ipa1.example.com
>>>>>>> ---
>>>>>>>
>>>>>>> which failed.
>>>>>>>
>>>>>>> Interestingly enough, the error message is
>>>>>>>
>>>>>>> ---
>>>>>>> unexpected error: Replication agreement for
ipa1.example.com
>>>>>>> not found
>>>>>>> ---
>>>>>>>
>>>>>>> And indeed:
>>>>>>>
>>>>>>> ---
>>>>>>> $ ipa topologysegment-find ca
>>>>>>> ------------------
>>>>>>> 2 segments matched
>>>>>>> ------------------
>>>>>>> Segment name:
ipa2.example.com-to-ipa1.example.com
>>>>>>> Left node:
ipa2.example.com
>>>>>>> Right node:
ipa1.example.com
>>>>>>> Connectivity: both
>>>>>>>
>>>>>>> Segment name:
ipa1.example.com-to-ipa2.example.com
>>>>>>> Left node:
ipa1.example.com
>>>>>>> Right node:
ipa2.example.com
>>>>>>> Connectivity: left-right
>>>>>>> ----------------------------
>>>>>>> Number of entries returned 2
>>>>>>> ----------------------------
>>>>>>> ---
>>>>>>>
>>>>>>> The Web UI topology graph doesn't reflect this, btw.
>>>>>>>
>>>>>>> Isn't the 2nd segment obsolete and probably causing my CS
>>>>>>> replication issues? Just remove it?
>>>>>>>
>>>>>>>> You could also re-init off line by using this article:
>>>>>>>>
>>>>>>>>
https://access.redhat.com/solutions/140483
>>>>>>>>
>>>>>>>> only for ipaca backend.
>>>>>>>>
>>>>>>>> On Wed, Feb 6, 2019 at 11:31 AM dbischof--- via
FreeIPA-users <freeipa-users(a)lists.fedorahosted.org> wrote:
>>>>>>>>
>>>>>>>>> On Wed, 6 Feb 2019, dbischof--- via FreeIPA-users
wrote:
>>>>>>>>>
>>>>>>>>>> On Wed, 6 Feb 2019, Florence Blanc-Renaud via
FreeIPA-users wrote:
>>>>>>>>>>
>>>>>>>>>>> On 2/5/19 4:17 PM, dbischof--- via
FreeIPA-users wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> my IPA system consists of 2 masters
(ipa1 and ipa2, both on
>>>>>>>>>>>> FreeIPA 4.6.4) with their own
self-signed CAs, one of them
>>>>>>>>>>>> being the certificate renewal master
(ipa1). The system has
>>>>>>>>>>>> been running for years and has been
migrated from an IPA 3
>>>>>>>>>>>> system. Both IPA servers are on domain
level 1.
>>>>>>>>>>>>
>>>>>>>>>>>> Problem: CS replication failed,
probably months ago.
>>>>>>>>>>>>
>>>>>>>>>>>> --- ipa1 ---
>>>>>>>>>>>> $ ipa-csreplica-manage -v list
ipa1.example.com
>>>>>>>>>>>>
>>>>>>>>>>>>
ipa2.example.com
>>>>>>>>>>>> last init status: None
>>>>>>>>>>>> last init ended: 1970-01-01
00:00:00+00:00
>>>>>>>>>>>> last update status: Error (-1)
Problem connecting to replica - LDAP error: Can't contact LDAP server (connection
error)
>>>>>>>>>>>> last update ended: 1970-01-01
00:00:00+00:00
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> $ ipa-csreplica-manage -v list
ipa2.example.com
>>>>>>>>>>>>
>>>>>>>>>>>> [no output]
>>>>>>>>>>>> ----
>>>>>>>>>>>>
>>>>>>>>>>>> Same on ipa2.
>>>>>>>>>>>>
>>>>>>>>>>>> Probably related:
>>>>>>>>>>>>
>>>>>>>>>>>> ---
>>>>>>>>>>>> ERR - slapi_ldap_bind - Error: could
not send startTLS
>>>>>>>>>>>> request: error -1 (Can't contact
LDAP server) errno 107
>>>>>>>>>>>> (Transport endpoint is not connected)
>>>>>>>>>>>> ---
>>>>>>>>>>>>
>>>>>>>>>>>> Every 5 mins in
/var/log/dirsrv/slapd-EXAMPLE-COM/errors.
>>>>>>>>>>>> However, these error messages could
refer to
>>>>>>>>>>>>
ipa3.example.com, a master i deleted
long (> 2 years) ago:
>>>>>>>>>>>>
>>>>>>>>>>>> ---
>>>>>>>>>>>> $ ipa-replica-manage list-ruv
>>>>>>>>>>>>
>>>>>>>>>>>> Replica Update Vectors:
>>>>>>>>>>>> ipa2.example.com:389: 10
>>>>>>>>>>>> ipa1.example.com:389: 9
>>>>>>>>>>>> Certificate Server Replica Update
Vectors:
>>>>>>>>>>>> ipa2.example.com:389: 11
>>>>>>>>>>>> ipa1.example.com:389: 91
>>>>>>>>>>>> ipa2.example.com:7389: 96
>>>>>>>>>>>> ipa3.example.com:7389: 97
>>>>>>>>>>>> ---
>>>>>>>>>>>>
>>>>>>>>>>>> How do i track this down and resolve
the problem?
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>> please find more information re. 389-ds
troubleshooting:
>>>>>>>>>>>
https://www.freeipa.org/page/Troubleshooting/Directory_Server
>>>>>>>>>>
>>>>>>>>>> I checked for the common problems described in
that page already,
>>>>>>>>>> but to no avail. I did, however, successfully
manage to remove
>>>>>>>>>> replication references to ipa3 using
"ipa-replica-manage
>>>>>>>>>> clean-dangling-ruv":
>>>>>>>>>>
>>>>>>>>>> ---
>>>>>>>>>> $ ipa-replica-manage list-ruv
>>>>>>>>>> Replica Update Vectors:
>>>>>>>>>> ipa1.example.com:389: 9
>>>>>>>>>> ipa2.example.com:389: 10
>>>>>>>>>> Certificate Server Replica Update Vectors:
>>>>>>>>>> ipa1.example.com:389: 91
>>>>>>>>>> ipa2.example.com:389: 11
>>>>>>>>>> ---
>>>>>>>>>>
>>>>>>>>>> The error message
>>>>>>>>>>
>>>>>>>>>> ---
>>>>>>>>>> [06/Feb/2019:10:38:52.095489260 +0100] - ERR -
slapi_ldap_bind -
>>>>>>>>>> Error: could not send startTLS request: error -1
(Can't contact LDAP
>>>>>>>>>> server) errno 107 (Transport endpoint is not
connected)
>>>>>>>>>> ---
>>>>>>>>>>
>>>>>>>>>> on ipa1 is still in the logs. Additionally, while
cleaning ruvs:
>>>>>>>>>>
>>>>>>>>>> ---
>>>>>>>>>> [06/Feb/2019:10:32:31.029394375 +0100] - ERR -
NSMMReplicationPlugin
>>>>>>>>>> - bind_and_check_pwp -
>>>>>>>>>>
agmt="cn=cloneAgreement1-ipa1.example.com-pki-tomcat" (ipa2:7389) -
>>>>>>>>>> Replication bind with SIMPLE auth failed: LDAP
error -1 (Can't
>>>>>>>>>> contact LDAP server) ()
>>>>>>>>>> ---
>>>>>>>>>>
>>>>>>>>>> The ldapsearch queries described in the above
page can be carried
>>>>>>>>>> out successfully on both servers:
>>>>>>>>>>
>>>>>>>>>> ---
>>>>>>>>>> [...]
>>>>>>>>>> # search result
>>>>>>>>>> search: 4
>>>>>>>>>> result: 0 Success
>>>>>>>>>>
>>>>>>>>>> # numResponses: 2
>>>>>>>>>> # numEntries: 1
>>>>>>>>>> ---
>>>>>>>>>>
>>>>>>>>>> Also, no DNS issues, wrong entries /etc/hosts,
time
>>>>>>>>>> differences or log messages related to SASL
issues.
>>>>>>>>>>
>>>>>>>>>> Maybe a wrong key or certificate somewhere?
>>>>>>>>>
>>>>>>>>> update: ipa-checkcerts.py shows
>>>>>>>>>
>>>>>>>>> ---
>>>>>>>>> [...]
>>>>>>>>> Failures:
>>>>>>>>> ipa: INFO: Unable to find request for serial
268304391
>>>>>>>>> Unable to find request for serial 268304391
>>>>>>>>> ipa: INFO: Unable to find request for serial
268304394
>>>>>>>>> Unable to find request for serial 268304394
>>>>>>>>> ipa: INFO: Unable to find request for serial
268304393
>>>>>>>>> Unable to find request for serial 268304393
>>>>>>>>> ipa: INFO: Unable to find request for serial
268304392
>>>>>>>>> Unable to find request for serial 268304392
>>>>>>>>> ipa: INFO: Subject
O=EXAMPLE.COM,CN=ipa2.example.com
and template
>>>>>>>>> subject
CN=ipa1.example.com,O=EXAMPLE.COM do not
match for serial 57
>>>>>>>>> Subject
O=EXAMPLE.COM,CN=ipa2.example.com and
template subject CN=
>>>>>>>>>
ipa1.example.com,O=EXAMPLE.COM do not match for
serial 57
>>>>>>>>> ---
>>>>>>>>>
>>>>>>>>> So there is a certificate issue.
Mit freundlichen Gruessen/With best regards,
--Daniel.