Terminating replication agreement

[Randy Morgan]

I have a two year image of one of my IPA servers that I am trying to 
bring live.  Unfortunately all of the certs except the CA are expired.  
I have attempted to follow the instructions for updating the certs, but 
it has failed to update them.  After careful and extensive digging, I 
have found that the issue is two replication agreements from other IPA 
servers that have since been rebuilt. Because of the expired certs I 
can't login to the web UI, so I can't terminate the agreements that way, 
and the IPA commands fail.  Is there a way to terminate these agreements 
manually by removing the references to the two servers?

Randy Morgan

[John Keates]

You could turn the clock back, remove the agreements, renew the certs to a future date, shutdown, reset the clock and renew again to get up and running. Make sure you’re doing it while the system is offline to prevent NTP.
Also: make sure you don’t run in to this again by making regular recovery points (backups, snapshots, periodic master updates). I’m assuming this is a recovery action from total loss of everything? If not: don’t bother with that image, install a fresh master instead.

John
> On 26 Sep 2019, at 23:59, Randy Morgan via FreeIPA-users  wrote:
>
> I have a two year image of one of my IPA servers that I am trying to bring live.  Unfortunately all of the certs except the CA are expired.  I have attempted to follow the instructions for updating the certs, but it has failed to update them.  After careful and extensive digging, I have found that the issue is two replication agreements from other IPA servers that have since been rebuilt. Because of the expired certs I can't login to the web UI, so I can't terminate the agreements that way, and the IPA commands fail.  Is there a way to terminate these agreements manually by removing the references to the two servers?
>
> Randy Morgan
>
> -- 
> Randy Morgan
> CSR
> Department of Chemistry/BioChemistry
> Brigham Young University
> _______________________________________________
> FreeIPA-users mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedorahosted.org/archives/list/[e-mail]...

[Randy Morgan]

Tried everything you just suggested, and it doesn't work.  Yes this is a 
recovery option, our environment has become damaged somehow, uncertain 
of all that happened.  This image gives us a path forward, but I need 
these replication agreements to go away and the image to become a stand 
alone master.  Once that happens I believe I can get the certs to 
update, but right now everything seems to be attempting to talk to IPA2 
which is still running but the server was rebuilt after this image was 
made, so we can't talk with the server.

Randy

On 9/26/2019 4:05 PM, John Keates via FreeIPA-users wrote:
> You could turn the clock back, remove the agreements, renew the certs to a future date, shutdown, reset the clock and renew again to get up and running. Make sure you’re doing it while the system is offline to prevent NTP.
> Also: make sure you don’t run in to this again by making regular recovery points (backups, snapshots, periodic master updates). I’m assuming this is a recovery action from total loss of everything? If not: don’t bother with that image, install a fresh master instead.
>
> John
> > On 26 Sep 2019, at 23:59, Randy Morgan via FreeIPA-users  wrote:
> >
> > I have a two year image of one of my IPA servers that I am trying to bring live.  Unfortunately all of the certs except the CA are expired.  I have attempted to follow the instructions for updating the certs, but it has failed to update them.  After careful and extensive digging, I have found that the issue is two replication agreements from other IPA servers that have since been rebuilt. Because of the expired certs I can't login to the web UI, so I can't terminate the agreements that way, and the IPA commands fail.  Is there a way to terminate these agreements manually by removing the references to the two servers?
> >
> > Randy Morgan
> >
> > -- 
> > Randy Morgan
> > CSR
> > Department of Chemistry/BioChemistry
> > Brigham Young University
> > _______________________________________________
> > FreeIPA-users mailing list -- [e-mail]
> > To unsubscribe send an email to [e-mail]
> > Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives: https://lists.fedorahosted.org/archives/list/[e-mail]...
>
> FreeIPA-users mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedorahosted.org/archives/list/[e-mail]...

[John Keates]

What do you mean by ‘rebuilt’? Also: is that image a CA master and how does it fail when you run it with the clock turned back and network unplugged/firewalled?

John
> On 27 Sep 2019, at 00:10, Randy Morgan via FreeIPA-users  wrote:
>
> Tried everything you just suggested, and it doesn't work.  Yes this is a recovery option, our environment has become damaged somehow, uncertain of all that happened.  This image gives us a path forward, but I need these replication agreements to go away and the image to become a stand alone master.  Once that happens I believe I can get the certs to update, but right now everything seems to be attempting to talk to IPA2 which is still running but the server was rebuilt after this image was made, so we can't talk with the server.
>
> Randy
>
> On 9/26/2019 4:05 PM, John Keates via FreeIPA-users wrote:
> > You could turn the clock back, remove the agreements, renew the certs to a future date, shutdown, reset the clock and renew again to get up and running. Make sure you’re doing it while the system is offline to prevent NTP.
> > Also: make sure you don’t run in to this again by making regular recovery points (backups, snapshots, periodic master updates). I’m assuming this is a recovery action from total loss of everything? If not: don’t bother with that image, install a fresh master instead.
> >
> > John
> > > On 26 Sep 2019, at 23:59, Randy Morgan via FreeIPA-users  wrote:
> > >
> > > I have a two year image of one of my IPA servers that I am trying to bring live.  Unfortunately all of the certs except the CA are expired.  I have attempted to follow the instructions for updating the certs, but it has failed to update them.  After careful and extensive digging, I have found that the issue is two replication agreements from other IPA servers that have since been rebuilt. Because of the expired certs I can't login to the web UI, so I can't terminate the agreements that way, and the IPA commands fail.  Is there a way to terminate these agreements manually by removing the references to the two servers?
> > >
> > > Randy Morgan
> > >
> > > -- 
> > > Randy Morgan
> > > CSR
> > > Department of Chemistry/BioChemistry
> > > Brigham Young University
> > > _______________________________________________
> > > FreeIPA-users mailing list -- [e-mail]
> > > To unsubscribe send an email to [e-mail]
> > > Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > > List Archives: https://lists.fedorahosted.org/archives/list/[e-mail]...
> >
> > FreeIPA-users mailing list -- [e-mail]
> > To unsubscribe send an email to [e-mail]
> > Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives: https://lists.fedorahosted.org/archives/list/[e-mail]...
>
> -- 
> Randy Morgan
> CSR
> Department of Chemistry/BioChemistry
> Brigham Young University
> _______________________________________________
> FreeIPA-users mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedorahosted.org/archives/list/[e-mail]...

[Randy Morgan]

IPA2 was removed from the replica set and reinstalled to correct some OS 
issues (twice).  After that it was brought back in as a replica again.  
IPA2 has become corrupted and we are unable to update the subsystem 
certs.  It is the last standing replica for the original cluster.  The 
image of IPA1 is a virtual server, so turning off the network can be 
done but that means working through the console, yuck.  I can try it and 
see what happens.  Yes this image is a CA master, or at least it was 
supposed to be when the image was made.

Randy

On 9/26/2019 4:14 PM, John Keates via FreeIPA-users wrote:
> What do you mean by ‘rebuilt’? Also: is that image a CA master and how does it fail when you run it with the clock turned back and network unplugged/firewalled?
>
> John
> > On 27 Sep 2019, at 00:10, Randy Morgan via FreeIPA-users  wrote:
> >
> > Tried everything you just suggested, and it doesn't work.  Yes this is a recovery option, our environment has become damaged somehow, uncertain of all that happened.  This image gives us a path forward, but I need these replication agreements to go away and the image to become a stand alone master.  Once that happens I believe I can get the certs to update, but right now everything seems to be attempting to talk to IPA2 which is still running but the server was rebuilt after this image was made, so we can't talk with the server.
> >
> > Randy
> >
> > On 9/26/2019 4:05 PM, John Keates via FreeIPA-users wrote:
> > > You could turn the clock back, remove the agreements, renew the certs to a future date, shutdown, reset the clock and renew again to get up and running. Make sure you’re doing it while the system is offline to prevent NTP.
> > > Also: make sure you don’t run in to this again by making regular recovery points (backups, snapshots, periodic master updates). I’m assuming this is a recovery action from total loss of everything? If not: don’t bother with that image, install a fresh master instead.
> > >
> > > John
> > > > On 26 Sep 2019, at 23:59, Randy Morgan via FreeIPA-users  wrote:
> > > >
> > > > I have a two year image of one of my IPA servers that I am trying to bring live.  Unfortunately all of the certs except the CA are expired.  I have attempted to follow the instructions for updating the certs, but it has failed to update them.  After careful and extensive digging, I have found that the issue is two replication agreements from other IPA servers that have since been rebuilt. Because of the expired certs I can't login to the web UI, so I can't terminate the agreements that way, and the IPA commands fail.  Is there a way to terminate these agreements manually by removing the references to the two servers?
> > > >
> > > > Randy Morgan
> > > >
> > > > -- 
> > > > Randy Morgan
> > > > CSR
> > > > Department of Chemistry/BioChemistry
> > > > Brigham Young University
> > > > _______________________________________________
> > > > FreeIPA-users mailing list -- [e-mail]
> > > > To unsubscribe send an email to [e-mail]
> > > > Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > > > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > > > List Archives: https://lists.fedorahosted.org/archives/list/[e-mail]...
> > >
> > > FreeIPA-users mailing list -- [e-mail]
> > > To unsubscribe send an email to [e-mail]
> > > Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > > List Archives: https://lists.fedorahosted.org/archives/list/[e-mail]...
> >
> > -- 
> > Randy Morgan
> > CSR
> > Department of Chemistry/BioChemistry
> > Brigham Young University
> > _______________________________________________
> > FreeIPA-users mailing list -- [e-mail]
> > To unsubscribe send an email to [e-mail]
> > Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives: https://lists.fedorahosted.org/archives/list/[e-mail]...
>
> FreeIPA-users mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedorahosted.org/archives/list/[e-mail]...

[Rob Crittenden]

Randy Morgan via FreeIPA-users wrote:
> IPA2 was removed from the replica set and reinstalled to correct some OS
> issues (twice).  After that it was brought back in as a replica again. 
> IPA2 has become corrupted and we are unable to update the subsystem
> certs.  It is the last standing replica for the original cluster.  The
> image of IPA1 is a virtual server, so turning off the network can be
> done but that means working through the console, yuck.  I can try it and
> see what happens.  Yes this image is a CA master, or at least it was
> supposed to be when the image was made.

The version you're running could make a difference in how things work.

Only the renewal master, and there should only be one, will actually
renew the certificates. So for your "offline" server I'd make sure it is
by going back in time so ipa can start and do ipa config-show and see if
this master is the CA renewal master. If it isn't then set it as it and
try the renewal again.

The current getcert list output may provide some insights as well.

Generally speaking the agreements probably won't work if the master has
been down for a long time, as it seems. The changelog has limited size
so depending on the amount of time and the number of changes it is
probably exhaused so can't replay history fully over replication so it
is throwing up its hands.

rob
> On 9/26/2019 4:14 PM, John Keates via FreeIPA-users wrote:
> > What do you mean by ‘rebuilt’? Also: is that image a CA master and how
> > does it fail when you run it with the clock turned back and network
> > unplugged/firewalled?
> >
> > John
> > > On 27 Sep 2019, at 00:10, Randy Morgan via FreeIPA-users
> > >  wrote:
> > >
> > > Tried everything you just suggested, and it doesn't work.  Yes this
> > > is a recovery option, our environment has become damaged somehow,
> > > uncertain of all that happened.  This image gives us a path forward,
> > > but I need these replication agreements to go away and the image to
> > > become a stand alone master.  Once that happens I believe I can get
> > > the certs to update, but right now everything seems to be attempting
> > > to talk to IPA2 which is still running but the server was rebuilt
> > > after this image was made, so we can't talk with the server.
> > >
> > > Randy
> > >
> > > On 9/26/2019 4:05 PM, John Keates via FreeIPA-users wrote:
> > > > You could turn the clock back, remove the agreements, renew the
> > > > certs to a future date, shutdown, reset the clock and renew again to
> > > > get up and running. Make sure you’re doing it while the system is
> > > > offline to prevent NTP.
> > > > Also: make sure you don’t run in to this again by making regular
> > > > recovery points (backups, snapshots, periodic master updates). I’m
> > > > assuming this is a recovery action from total loss of everything? If
> > > > not: don’t bother with that image, install a fresh master instead.
> > > >
> > > > John
> > > > > On 26 Sep 2019, at 23:59, Randy Morgan via FreeIPA-users
> > > > >  wrote:
> > > > >
> > > > > I have a two year image of one of my IPA servers that I am trying
> > > > > to bring live.  Unfortunately all of the certs except the CA are
> > > > > expired.  I have attempted to follow the instructions for updating
> > > > > the certs, but it has failed to update them.  After careful and
> > > > > extensive digging, I have found that the issue is two replication
> > > > > agreements from other IPA servers that have since been rebuilt.
> > > > > Because of the expired certs I can't login to the web UI, so I
> > > > > can't terminate the agreements that way, and the IPA commands
> > > > > fail.  Is there a way to terminate these agreements manually by
> > > > > removing the references to the two servers?
> > > > >
> > > > > Randy Morgan
> > > > >
> > > > > -- 
> > > > > Randy Morgan
> > > > > CSR
> > > > > Department of Chemistry/BioChemistry
> > > > > Brigham Young University
> > > > > _______________________________________________
> > > > > FreeIPA-users mailing list -- [e-mail]
> > > > > To unsubscribe send an email to
> > > > > [e-mail]
> > > > > Fedora Code of Conduct:
> > > > > https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > > > > List Guidelines:
> > > > > https://fedoraproject.org/wiki/Mailing_list_guidelines
> > > > > List Archives:
> > > > > https://lists.fedorahosted.org/archives/list/[e-mail]...
> > > >
> > > > FreeIPA-users mailing list -- [e-mail]
> > > > To unsubscribe send an email to
> > > > [e-mail]
> > > > Fedora Code of Conduct:
> > > > https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > > > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > > > List Archives:
> > > > https://lists.fedorahosted.org/archives/list/[e-mail]...
> > >
> > > -- 
> > > Randy Morgan
> > > CSR
> > > Department of Chemistry/BioChemistry
> > > Brigham Young University
> > > _______________________________________________
> > > FreeIPA-users mailing list -- [e-mail]
> > > To unsubscribe send an email to
> > > [e-mail]
> > > Fedora Code of Conduct:
> > > https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > > List Archives:
> > > https://lists.fedorahosted.org/archives/list/[e-mail]...
> >
> > FreeIPA-users mailing list -- [e-mail]
> > To unsubscribe send an email to
> > [e-mail]
> > Fedora Code of Conduct:
> > https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives:
> > https://lists.fedorahosted.org/archives/list/[e-mail]...