How to recover from "split brain"

[Rob Brown]

I have 4 IPA servers, all masters, that were previously configured in a
"full mesh" replication.
2 in "prod", 2 in "preprod".
While trying to fix a replication issue, I accidentally did a:
ipa-replica-manage del
on one of the prod servers for BOTH preprod servers.

Now, the prod servers don't "see" either of the preprod servers, so I
effectively created a "split-brain" between the 2 environments. Preprod
still "knows about" the prod ipa servers, but I can't figure out how to
re-establish the replication agreements.

I was about to just blow away the preprod servers and rebuild them (which i
did before on one of them) but noticed one of them has the "KRA" role, and
it is the only one in the domain that has it.
I don't know what that does, or what the effects would be if it went away.
I'm guessing bad.

I have tried "ipa topologysegment-reinitialize domain" on the segments that
preprod still has, but those segments did not show up in prod.
ipa topologysuffix-verify domain says "in order" everywhere.

At this point I am completely lost on how to proceed.

What details can I provide for any help anyone is willing to provide?

[Rob Brown]

ok, did a little googling, and seems like KRA refers to the "vault" feature?
I didn't originally install this myself, so wasn't sure if it is used for
anything critical.
I ran:
# ipa vault-find
----------------
0 vaults matched
----------------
----------------------------
Number of entries returned 0
----------------------------

So, can I assume it is safe to blow away and rebuild the server that has
this role?

On Wed, Jan 31, 2018 at 3:56 PM, Rob Brown  wrote:
> I have 4 IPA servers, all masters, that were previously configured in a
> "full mesh" replication.
> 2 in "prod", 2 in "preprod".
> While trying to fix a replication issue, I accidentally did a:
> ipa-replica-manage del
> on one of the prod servers for BOTH preprod servers.
>
> Now, the prod servers don't "see" either of the preprod servers, so I
> effectively created a "split-brain" between the 2 environments. Preprod
> still "knows about" the prod ipa servers, but I can't figure out how to
> re-establish the replication agreements.
>
> I was about to just blow away the preprod servers and rebuild them (which
> i did before on one of them) but noticed one of them has the "KRA" role,
> and it is the only one in the domain that has it.
> I don't know what that does, or what the effects would be if it went away.
> I'm guessing bad.
>
> I have tried "ipa topologysegment-reinitialize domain" on the segments
> that preprod still has, but those segments did not show up in prod.
> ipa topologysuffix-verify domain says "in order" everywhere.
>
> At this point I am completely lost on how to proceed.
>
> What details can I provide for any help anyone is willing to provide?

[Andrew Radygin]

Though you can completely rebuild preprod servers, still it would be
interesting how to reconnect prod servers with replicas again.

2018-02-01 8:41 GMT+03:00 Rob Brown via FreeIPA-users <
[e-mail]>:
> ok, did a little googling, and seems like KRA refers to the "vault"
> feature?
> I didn't originally install this myself, so wasn't sure if it is used for
> anything critical.
> I ran:
> # ipa vault-find
> 0 vaults matched
> Number of entries returned 0
>
> So, can I assume it is safe to blow away and rebuild the server that has
> this role?
>
> On Wed, Jan 31, 2018 at 3:56 PM, Rob Brown 
> wrote:
> > I have 4 IPA servers, all masters, that were previously configured in a
> > "full mesh" replication.
> > 2 in "prod", 2 in "preprod".
> > While trying to fix a replication issue, I accidentally did a:
> > ipa-replica-manage del
> > on one of the prod servers for BOTH preprod servers.
> >
> > Now, the prod servers don't "see" either of the preprod servers, so I
> > effectively created a "split-brain" between the 2 environments. Preprod
> > still "knows about" the prod ipa servers, but I can't figure out how to
> > re-establish the replication agreements.
> >
> > I was about to just blow away the preprod servers and rebuild them (which
> > i did before on one of them) but noticed one of them has the "KRA" role,
> > and it is the only one in the domain that has it.
> > I don't know what that does, or what the effects would be if it went
> > away. I'm guessing bad.
> >
> > I have tried "ipa topologysegment-reinitialize domain" on the segments
> > that preprod still has, but those segments did not show up in prod.
> > ipa topologysuffix-verify domain says "in order" everywhere.
> >
> > At this point I am completely lost on how to proceed.
> >
> > What details can I provide for any help anyone is willing to provide?
>
> FreeIPA-users mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]

[Rob Brown]

Agreed! I would love to know if that is possible... seems like it should be.
As mentioned previously, preprod still has the agreements, but prod does
not.
Not really sure how I should proceed. I'm a bit stuck, not wanting to
further break anything. For now, auth is still working in both envs.
---
[root@ipa-preprod-1201]# ipa topologysegment-find domain
------------------
5 segments matched
------------------
  Segment name: ipa-preprod-1201-to-ipa-preprod-1202
  Left node: ipa-preprod-1201
  Right node: ipa-preprod-1202
  Connectivity: both

Segment name: ipa-preprod-1201-to-ipa-prod-1201
  Left node: ipa-preprod-1201
  Right node: ipa-prod-1201
  Connectivity: both

Segment name: ipa-preprod-1202-to-ipa-prod-1201
  Left node: ipa-preprod-1202
  Right node: ipa-prod-1201
  Connectivity: both

Segment name: ipa-prod-1201-to-ipa-prod-1202
  Left node: ipa-prod-1201
  Right node: ipa-prod-1202
  Connectivity: both

Segment name: ipa-prod-1202-to-ipa-preprod-1201
  Left node: ipa-prod-1202
  Right node: ipa-preprod-1201
  Connectivity: both

[root@ipa-prod-1201]# ipa topologysegment-find domain
------------------
2 segments matched
------------------
  Segment name: ipa-preprod-1201-to-ipa-preprod-1202
  Left node: ipa-preprod-1201
  Right node: ipa-preprod-1202
  Connectivity: both

Segment name: ipa-prod-1201-to-ipa-prod-1202
  Left node: ipa-prod-1201
  Right node: ipa-prod-1202
  Connectivity: both
----------------------------
Number of entries returned 2
----------------------------

I think part of the problem is that when I did the ipa-replica-manage del,
it removed the preprod servers:

[root@ipa-prod-1201]# ipa server-find
---------------------
2 IPA servers matched
---------------------
  Server name: ipa-prod-1201
  Min domain level: 0
  Max domain level: 1

Server name: ipa-prod-1202
  Min domain level: 0
  Max domain level: 1
----------------------------
Number of entries returned 2
----------------------------

but they still exist on the preprod side:

[root@ipa-preprod-1201]# ipa server-find
---------------------
4 IPA servers matched
---------------------
  Server name: ipa-preprod-1201
  Min domain level: 0
  Max domain level: 1

Server name: ipa-preprod-1202
  Min domain level: 0
  Max domain level: 1

Server name: ipa-prod-1201
  Min domain level: 0
  Max domain level: 1

Server name: ipa-prod-1202
  Min domain level: 0
  Max domain level: 1
----------------------------
Number of entries returned 4
----------------------------

On Wed, Jan 31, 2018 at 10:52 PM, Andrew Radygin  wrote:
> Though you can completely rebuild preprod servers, still it would be
> interesting how to reconnect prod servers with replicas again.
>
> 2018-02-01 8:41 GMT+03:00 Rob Brown via FreeIPA-users <
> [e-mail]>:
> > ok, did a little googling, and seems like KRA refers to the "vault"
> > feature?
> > I didn't originally install this myself, so wasn't sure if it is used for
> > anything critical.
> > I ran:
> > # ipa vault-find
> > 0 vaults matched
> > Number of entries returned 0
> >
> > So, can I assume it is safe to blow away and rebuild the server that has
> > this role?
> >
> > On Wed, Jan 31, 2018 at 3:56 PM, Rob Brown 
> > wrote:
> > > I have 4 IPA servers, all masters, that were previously configured in a
> > > "full mesh" replication.
> > > 2 in "prod", 2 in "preprod".
> > > While trying to fix a replication issue, I accidentally did a:
> > > ipa-replica-manage del
> > > on one of the prod servers for BOTH preprod servers.
> > >
> > > Now, the prod servers don't "see" either of the preprod servers, so I
> > > effectively created a "split-brain" between the 2 environments. Preprod
> > > still "knows about" the prod ipa servers, but I can't figure out how to
> > > re-establish the replication agreements.
> > >
> > > I was about to just blow away the preprod servers and rebuild them
> > > (which i did before on one of them) but noticed one of them has the "KRA"
> > > role, and it is the only one in the domain that has it.
> > > I don't know what that does, or what the effects would be if it went
> > > away. I'm guessing bad.
> > >
> > > I have tried "ipa topologysegment-reinitialize domain" on the segments
> > > that preprod still has, but those segments did not show up in prod.
> > > ipa topologysuffix-verify domain says "in order" everywhere.
> > >
> > > At this point I am completely lost on how to proceed.
> > >
> > > What details can I provide for any help anyone is willing to provide?
> >
> > FreeIPA-users mailing list -- [e-mail]
> > To unsubscribe send an email to [e-mail]
> > rahosted.org
>
> --
> Best regards, Andrew.

[Rob Brown]

BTW:
[root@ipa-prod-1201]# cat /etc/redhat-release
CentOS Linux release 7.3.1611 (Core)
[root@ipa-prod-1201]# rpm -qa|grep ipa-server-4
ipa-server-4.4.0-14.el7.centos.6.x86_64

On Thu, Feb 1, 2018 at 10:53 AM, Rob Brown  wrote:
> Agreed! I would love to know if that is possible... seems like it should
> be.
> As mentioned previously, preprod still has the agreements, but prod does
> not.
> Not really sure how I should proceed. I'm a bit stuck, not wanting to
> further break anything. For now, auth is still working in both envs.
> [root@ipa-preprod-1201]# ipa topologysegment-find domain
> 5 segments matched
>
> Segment name: ipa-preprod-1201-to-ipa-preprod-1202
>   Left node: ipa-preprod-1201
>   Right node: ipa-preprod-1202
>   Connectivity: both
>
> Segment name: ipa-preprod-1201-to-ipa-prod-1201
>   Left node: ipa-preprod-1201
>   Right node: ipa-prod-1201
>   Connectivity: both
>
> Segment name: ipa-preprod-1202-to-ipa-prod-1201
>   Left node: ipa-preprod-1202
>   Right node: ipa-prod-1201
>   Connectivity: both
>
> Segment name: ipa-prod-1201-to-ipa-prod-1202
>   Left node: ipa-prod-1201
>   Right node: ipa-prod-1202
>   Connectivity: both
>
> Segment name: ipa-prod-1202-to-ipa-preprod-1201
>   Left node: ipa-prod-1202
>   Right node: ipa-preprod-1201
>   Connectivity: both
> [root@ipa-prod-1201]# ipa topologysegment-find domain
> 2 segments matched
>
> Segment name: ipa-preprod-1201-to-ipa-preprod-1202
>   Left node: ipa-preprod-1201
>   Right node: ipa-preprod-1202
>   Connectivity: both
>
> Segment name: ipa-prod-1201-to-ipa-prod-1202
>   Left node: ipa-prod-1201
>   Right node: ipa-prod-1202
>   Connectivity: both
> Number of entries returned 2
>
> I think part of the problem is that when I did the ipa-replica-manage del,
> it removed the preprod servers:
> [root@ipa-prod-1201]# ipa server-find
> 2 IPA servers matched
>
> Server name: ipa-prod-1201
>   Min domain level: 0
>   Max domain level: 1
>
> Server name: ipa-prod-1202
>   Min domain level: 0
>   Max domain level: 1
> Number of entries returned 2
>
> but they still exist on the preprod side:
> [root@ipa-preprod-1201]# ipa server-find
> 4 IPA servers matched
>
> Server name: ipa-preprod-1201
>   Min domain level: 0
>   Max domain level: 1
>
> Server name: ipa-preprod-1202
>   Min domain level: 0
>   Max domain level: 1
>
> Server name: ipa-prod-1201
>   Min domain level: 0
>   Max domain level: 1
>
> Server name: ipa-prod-1202
>   Min domain level: 0
>   Max domain level: 1
> Number of entries returned 4
>
> On Wed, Jan 31, 2018 at 10:52 PM, Andrew Radygin 
> wrote:
> > Though you can completely rebuild preprod servers, still it would be
> > interesting how to reconnect prod servers with replicas again.
> >
> > 2018-02-01 8:41 GMT+03:00 Rob Brown via FreeIPA-users <
> > [e-mail]>:
> > > ok, did a little googling, and seems like KRA refers to the "vault"
> > > feature?
> > > I didn't originally install this myself, so wasn't sure if it is used
> > > for anything critical.
> > > I ran:
> > > # ipa vault-find
> > > 0 vaults matched
> > > Number of entries returned 0
> > >
> > > So, can I assume it is safe to blow away and rebuild the server that has
> > > this role?
> > >
> > > On Wed, Jan 31, 2018 at 3:56 PM, Rob Brown 
> > > wrote:
> > > > I have 4 IPA servers, all masters, that were previously configured in a
> > > > "full mesh" replication.
> > > > 2 in "prod", 2 in "preprod".
> > > > While trying to fix a replication issue, I accidentally did a:
> > > > ipa-replica-manage del
> > > > on one of the prod servers for BOTH preprod servers.
> > > >
> > > > Now, the prod servers don't "see" either of the preprod servers, so I
> > > > effectively created a "split-brain" between the 2 environments. Preprod
> > > > still "knows about" the prod ipa servers, but I can't figure out how to
> > > > re-establish the replication agreements.
> > > >
> > > > I was about to just blow away the preprod servers and rebuild them
> > > > (which i did before on one of them) but noticed one of them has the "KRA"
> > > > role, and it is the only one in the domain that has it.
> > > > I don't know what that does, or what the effects would be if it went
> > > > away. I'm guessing bad.
> > > >
> > > > I have tried "ipa topologysegment-reinitialize domain" on the segments
> > > > that preprod still has, but those segments did not show up in prod.
> > > > ipa topologysuffix-verify domain says "in order" everywhere.
> > > >
> > > > At this point I am completely lost on how to proceed.
> > > >
> > > > What details can I provide for any help anyone is willing to provide?
> > >
> > > FreeIPA-users mailing list -- [e-mail]
> > > To unsubscribe send an email to [e-mail]
> > > rahosted.org
> >
> > --
> > Best regards, Andrew.