cant fix ipa server cert issues

[Justen Long]

Thanks in advance for your replies.. I've spent 7 hours looking through posts here and
trying everything... I'm stuck.

Background: I am a System Administrator in a closed, classified environment.
Unfortunately, I cannot post logging here, but I can refer to them as needed.

I inherited this system from someone who departed the program a year or so ago. Fast
forward to today, the server certs expired yesterday. Admittedly, I'm unfamiliar (or
was) with the certificate update process for IPA servers. On a typical server, we replace
the old cert and restart the httpd services; however, I realize this cannot work with IPA
servers now.

Additionally to all of this, the CA chain updated 6 months ago.

I ran ipa-cacert-manage to update the CA chain. When trying to run ipa-certupdate, I
received errors for an invalid server certificate (it expired on 11 April 2023). It simply
won't connect to the web server. HTTPD failed as well, so I had to add
"NSSEnforceValidCerts off" to the nss.conf file for HTTPD to start. Still, no
dice.

I've ran ipa-server-certinstall for the new cert/key as well, and it fails saying its
not trusted ("Peer's certificate issuer is not trusted [certutil: certificate is
invalid: Peer's Certificate issuer is not recognized] Please run ipa-cacert-manage
install and ipa-certupdate to install the CA certificate.... which, as reported above,
can't complete.

I'm at a total loss here... and really struggling being new to all this and trying my
best to keep it afloat. Any help would be GREATLY appreciated!

[Rob Crittenden]

Justen Long via FreeIPA-users wrote:
>  Thanks in advance for your replies.. I've spent 7 hours looking
> through posts here and trying everything... I'm stuck.
>  
>  Background: I am a System Administrator in a closed, classified environment.
> Unfortunately, I cannot post logging here, but I can refer to them as needed.
>  
>  I inherited this system from someone who departed the program a year or so ago. Fast
> forward to today, the server certs expired yesterday. Admittedly, I'm unfamiliar (or
> was) with the certificate update process for IPA servers. On a typical server, we replace
> the old cert and restart the httpd services; however, I realize this cannot work with IPA
> servers now.
>  
>  Additionally to all of this, the CA chain updated 6 months ago.
>  
>  I ran ipa-cacert-manage to update the CA chain. When trying to run ipa-certupdate, I
> received errors for an invalid server certificate (it expired on 11 April 2023). It simply
> won't connect to the web server. HTTPD failed as well, so I had to add
> "NSSEnforceValidCerts off" to the nss.conf file for HTTPD to start. Still, no
> dice.
>  
>  I've ran ipa-server-certinstall for the new cert/key as well, and it fails saying its
> not trusted ("Peer's certificate issuer is not trusted [certutil: certificate is
> invalid: Peer's Certificate issuer is not recognized] Please run ipa-cacert-manage
> install and ipa-certupdate to install the CA certificate.... which, as reported above,
> can't complete.
>  
>  I'm at a total loss here... and really struggling being new to all this and trying my
> best to keep it afloat. Any help would be GREATLY appreciated! 
Let's gather some information first.

What version of IPA is this, on what distribution?

IPA designates one server to be the "renewal master" which handles the
renewals. The output of `ipa config-show` should tell you (depending on
version). That's the server you want to work on.

How many servers in your topology and how many have a CA installed?

Does `getcert list` show a set of 8-10 tracked certificates? What are
the states?

You mention ipa-server-certinstall. Are you using 3rd party certificates
in addition to IPA CA-issued certificates or was that just an attempt to
get things working again?

rob

[Justen Long]

Rob,

Apologies for the delay in response. Once I'm home, I don't have access to
the information readily available to respond with. Here is the information
you requested:

The version of IPA we are using is 4.6.8, rpm specifically for us is
ipa-server-4.6.8-5.el7.centos.12.x86_64 and we are using CentOS 7.9
currently with plans to move to RHEL9 within the next year or so.

Unfortunately, 'ipa config-show' doesn't work. It populates the same error
stating "ipa: ERROR: cannot connect to 'https://ipaServer/ipa/json': [SSL:
CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:618).

We have ~50 hosts connected via IPA. We have two IPA servers, one as a
replica of the other.

'getcert list' only shows 1 certificate. It's state is "MONITORING"
and
seems related to kerberos.

As far as I know, we don't use IPA CA-issued certificates. I recall seeing
errors yesterday stating CA wasn't enabled on our servers. We have always
used 3rd party CAs to my knowledge.

-justen

On Wed, Apr 12, 2023 at 2:42 PM Rob Crittenden <rcritten(a)redhat.com&gt; wrote:

>  Justen Long via FreeIPA-users wrote:
>  > Thanks in advance for your replies.. I've spent 7 hours looking through
>  posts here and trying everything... I'm stuck.
>  >
>  > Background: I am a System Administrator in a closed, classified
>  environment. Unfortunately, I cannot post logging here, but I can refer to
>  them as needed.
>  >
>  > I inherited this system from someone who departed the program a year or
>  so ago. Fast forward to today, the server certs expired yesterday.
>  Admittedly, I'm unfamiliar (or was) with the certificate update process for
>  IPA servers. On a typical server, we replace the old cert and restart the
>  httpd services; however, I realize this cannot work with IPA servers now.
>  >
>  > Additionally to all of this, the CA chain updated 6 months ago.
>  >
>  > I ran ipa-cacert-manage to update the CA chain. When trying to run
>  ipa-certupdate, I received errors for an invalid server certificate (it
>  expired on 11 April 2023). It simply won't connect to the web server. HTTPD
>  failed as well, so I had to add "NSSEnforceValidCerts off" to the nss.conf
>  file for HTTPD to start. Still, no dice.
>  >
>  > I've ran ipa-server-certinstall for the new cert/key as well, and it
>  fails saying its not trusted ("Peer's certificate issuer is not trusted
>  [certutil: certificate is invalid: Peer's Certificate issuer is not
>  recognized] Please run ipa-cacert-manage install and ipa-certupdate to
>  install the CA certificate.... which, as reported above, can't complete.
>  >
>  > I'm at a total loss here... and really struggling being new to all this
>  and trying my best to keep it afloat. Any help would be GREATLY appreciated!
>
>  Let's gather some information first.
>
>  What version of IPA is this, on what distribution?
>
>  IPA designates one server to be the "renewal master" which handles the
>  renewals. The output of `ipa config-show` should tell you (depending on
>  version). That's the server you want to work on.
>
>  How many servers in your topology and how many have a CA installed?
>
>  Does `getcert list` show a set of 8-10 tracked certificates? What are
>  the states?
>
>  You mention ipa-server-certinstall. Are you using 3rd party certificates
>  in addition to IPA CA-issued certificates or was that just an attempt to
>  get things working again?
>
>  rob
>
>

[Justen Long]

Additionally, is there any way to force the CA cert update to be
recognized? When I run it to update the CA chain, everything is verified..
but /etc/ipa/ca.crt didn't reflect the change.. so I manually populated it
by copying over the guts of the CA bundle to the /etc/ipa/ca.crt before
trying to install the new server cert and it still doesn't recognize it as
trusted although the issuer is the same and within the CA bundle.

On Thu, Apr 13, 2023 at 6:20 AM Justen Long <mr.justenlong(a)gmail.com&gt; wrote:

>  Rob,
>
>  Apologies for the delay in response. Once I'm home, I don't have access to
>  the information readily available to respond with. Here is the information
>  you requested:
>
>  The version of IPA we are using is 4.6.8, rpm specifically for us is
>  ipa-server-4.6.8-5.el7.centos.12.x86_64 and we are using CentOS 7.9
>  currently with plans to move to RHEL9 within the next year or so.
>
>  Unfortunately, 'ipa config-show' doesn't work. It populates the same error
>  stating "ipa: ERROR: cannot connect to 'https://ipaServer/ipa/json':
>  [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:618).
>
>  We have ~50 hosts connected via IPA. We have two IPA servers, one as a
>  replica of the other.
>
>  'getcert list' only shows 1 certificate. It's state is "MONITORING"
> and
>  seems related to kerberos.
>
>  As far as I know, we don't use IPA CA-issued certificates. I recall seeing
>  errors yesterday stating CA wasn't enabled on our servers. We have always
>  used 3rd party CAs to my knowledge.
>
>  -justen
>
>  On Wed, Apr 12, 2023 at 2:42 PM Rob Crittenden <rcritten(a)redhat.com&gt;
>  wrote:
>
> > Justen Long via FreeIPA-users wrote:
> > > Thanks in advance for your replies.. I've spent 7 hours looking through
> > posts here and trying everything... I'm stuck.
> > >
> > > Background: I am a System Administrator in a closed, classified
> > environment. Unfortunately, I cannot post logging here, but I can refer to
> > them as needed.
> > >
> > > I inherited this system from someone who departed the program a year or
> > so ago. Fast forward to today, the server certs expired yesterday.
> > Admittedly, I'm unfamiliar (or was) with the certificate update process for
> > IPA servers. On a typical server, we replace the old cert and restart the
> > httpd services; however, I realize this cannot work with IPA servers now.
> > >
> > > Additionally to all of this, the CA chain updated 6 months ago.
> > >
> > > I ran ipa-cacert-manage to update the CA chain. When trying to run
> > ipa-certupdate, I received errors for an invalid server certificate (it
> > expired on 11 April 2023). It simply won't connect to the web server. HTTPD
> > failed as well, so I had to add "NSSEnforceValidCerts off" to the nss.conf
> > file for HTTPD to start. Still, no dice.
> > >
> > > I've ran ipa-server-certinstall for the new cert/key as well, and it
> > fails saying its not trusted ("Peer's certificate issuer is not trusted
> > [certutil: certificate is invalid: Peer's Certificate issuer is not
> > recognized] Please run ipa-cacert-manage install and ipa-certupdate to
> > install the CA certificate.... which, as reported above, can't complete.
> > >
> > > I'm at a total loss here... and really struggling being new to all this
> > and trying my best to keep it afloat. Any help would be GREATLY appreciated!
> >
> > Let's gather some information first.
> >
> > What version of IPA is this, on what distribution?
> >
> > IPA designates one server to be the "renewal master" which handles the
> > renewals. The output of `ipa config-show` should tell you (depending on
> > version). That's the server you want to work on.
> >
> > How many servers in your topology and how many have a CA installed?
> >
> > Does `getcert list` show a set of 8-10 tracked certificates? What are
> > the states?
> >
> > You mention ipa-server-certinstall. Are you using 3rd party certificates
> > in addition to IPA CA-issued certificates or was that just an attempt to
> > get things working again?
> >
> > rob
> >
> >

[Rob Crittenden]

Justen Long wrote:
>  Additionally, is there any way to force the CA cert update to be
>  recognized? When I run it to update the CA chain, everything is
>  verified.. but /etc/ipa/ca.crt didn't reflect the change.. so I manually
>  populated it by copying over the guts of the CA bundle to the
>  /etc/ipa/ca.crt before trying to install the new server cert and it
>  still doesn't recognize it as trusted although the issuer is the same
>  and within the CA bundle. 
This is going to sound weird, but I'd just go back in time to April 10,
restart all services but ntp (which will reset the time) and then the
commands should work. Once the certs are updated and working, return to
present time.

rob

>  
>  On Thu, Apr 13, 2023 at 6:20 AM Justen Long <mr.justenlong(a)gmail.com
>  <mailto:[e-mail]>> wrote:
>  
>      Rob,
>  
>      Apologies for the delay in response. Once I'm home, I don't have
>      access to the information readily available to respond with. Here is
>      the information you requested:
>  
>      The version of IPA we are using is 4.6.8, rpm specifically for us is
>      ipa-server-4.6.8-5.el7.centos.12.x86_64 and we are using CentOS 7.9
>      currently with plans to move to RHEL9 within the next year or so.
>  
>      Unfortunately, 'ipa config-show' doesn't work. It populates the same
>      error stating "ipa: ERROR: cannot connect to
>      'https://ipaServer/ipa/json': [SSL: CERTIFICATE_VERIFY_FAILED]
>      certificate verify failed (_ssl.c:618). 
The smack heard around the world was my head hitting my desk. Of course
this command failed.

>  
>      We have ~50 hosts connected via IPA. We have two IPA servers, one as
>      a replica of the other.
>  
>      'getcert list' only shows 1 certificate. It's state is
> "MONITORING"
>      and seems related to kerberos.
>  
>      As far as I know, we don't use IPA CA-issued certificates. I recall
>      seeing errors yesterday stating CA wasn't enabled on our servers. We
>      have always used 3rd party CAs to my knowledge.
>  
>      -justen
>  
>      On Wed, Apr 12, 2023 at 2:42 PM Rob Crittenden <rcritten(a)redhat.com
>      <mailto:[e-mail]>> wrote:
>  
>          Justen Long via FreeIPA-users wrote:
>          > Thanks in advance for your replies.. I've spent 7 hours
>          looking through posts here and trying everything... I'm stuck.
>          >
>          > Background: I am a System Administrator in a closed,
>          classified environment. Unfortunately, I cannot post logging
>          here, but I can refer to them as needed.
>          >
>          > I inherited this system from someone who departed the program
>          a year or so ago. Fast forward to today, the server certs
>          expired yesterday. Admittedly, I'm unfamiliar (or was) with the
>          certificate update process for IPA servers. On a typical server,
>          we replace the old cert and restart the httpd services; however,
>          I realize this cannot work with IPA servers now.
>          >
>          > Additionally to all of this, the CA chain updated 6 months ago.
>          >
>          > I ran ipa-cacert-manage to update the CA chain. When trying to
>          run ipa-certupdate, I received errors for an invalid server
>          certificate (it expired on 11 April 2023). It simply won't
>          connect to the web server. HTTPD failed as well, so I had to add
>          "NSSEnforceValidCerts off" to the nss.conf file for HTTPD to
>          start. Still, no dice.
>          >
>          > I've ran ipa-server-certinstall for the new cert/key as well,
>          and it fails saying its not trusted ("Peer's certificate issuer
>          is not trusted [certutil: certificate is invalid: Peer's
>          Certificate issuer is not recognized] Please run
>          ipa-cacert-manage install and ipa-certupdate to install the CA
>          certificate.... which, as reported above, can't complete.
>          >
>          > I'm at a total loss here... and really struggling being new to
>          all this and trying my best to keep it afloat. Any help would be
>          GREATLY appreciated!
>  
>          Let's gather some information first.
>  
>          What version of IPA is this, on what distribution?
>  
>          IPA designates one server to be the "renewal master" which
>          handles the
>          renewals. The output of `ipa config-show` should tell you
>          (depending on
>          version). That's the server you want to work on.
>  
>          How many servers in your topology and how many have a CA installed?
>  
>          Does `getcert list` show a set of 8-10 tracked certificates?
>          What are
>          the states?
>  
>          You mention ipa-server-certinstall. Are you using 3rd party
>          certificates
>          in addition to IPA CA-issued certificates or was that just an
>          attempt to
>          get things working again?
>  
>          rob
>

[Justen Long]

Rob,

I entered 'date --date="7 April 2023", verified it updated the system time
appropriately. Restarted dirsrv, ipa-custodia, ipa-otpd, httpd.. krb5kdc
and kadmin failed. Still, tried to send ipa cert-update, and it popped the
same SSL Certificate Verify Failed error.

On Thu, Apr 13, 2023 at 11:32 AM Rob Crittenden <rcritten(a)redhat.com&gt; wrote:

>  Justen Long wrote:
>  > Additionally, is there any way to force the CA cert update to be
>  > recognized? When I run it to update the CA chain, everything is
>  > verified.. but /etc/ipa/ca.crt didn't reflect the change.. so I manually
>  > populated it by copying over the guts of the CA bundle to the
>  > /etc/ipa/ca.crt before trying to install the new server cert and it
>  > still doesn't recognize it as trusted although the issuer is the same
>  > and within the CA bundle.
>
>  This is going to sound weird, but I'd just go back in time to April 10,
>  restart all services but ntp (which will reset the time) and then the
>  commands should work. Once the certs are updated and working, return to
>  present time.
>
>  rob
>
>  >
>  > On Thu, Apr 13, 2023 at 6:20 AM Justen Long <mr.justenlong(a)gmail.com
>  > <mailto:[e-mail]>> wrote:
>  >
>  >     Rob,
>  >
>  >     Apologies for the delay in response. Once I'm home, I don't have
>  >     access to the information readily available to respond with. Here is
>  >     the information you requested:
>  >
>  >     The version of IPA we are using is 4.6.8, rpm specifically for us is
>  >     ipa-server-4.6.8-5.el7.centos.12.x86_64 and we are using CentOS 7.9
>  >     currently with plans to move to RHEL9 within the next year or so.
>  >
>  >     Unfortunately, 'ipa config-show' doesn't work. It populates the
> same
>  >     error stating "ipa: ERROR: cannot connect to
>  >     'https://ipaServer/ipa/json': [SSL: CERTIFICATE_VERIFY_FAILED]
>  >     certificate verify failed (_ssl.c:618).
>
>  The smack heard around the world was my head hitting my desk. Of course
>  this command failed.
>
>  >
>  >     We have ~50 hosts connected via IPA. We have two IPA servers, one as
>  >     a replica of the other.
>  >
>  >     'getcert list' only shows 1 certificate. It's state is
> "MONITORING"
>  >     and seems related to kerberos.
>  >
>  >     As far as I know, we don't use IPA CA-issued certificates. I recall
>  >     seeing errors yesterday stating CA wasn't enabled on our servers. We
>  >     have always used 3rd party CAs to my knowledge.
>  >
>  >     -justen
>  >
>  >     On Wed, Apr 12, 2023 at 2:42 PM Rob Crittenden <rcritten(a)redhat.com
>  >     <mailto:[e-mail]>> wrote:
>  >
>  >         Justen Long via FreeIPA-users wrote:
>  >         > Thanks in advance for your replies.. I've spent 7 hours
>  >         looking through posts here and trying everything... I'm stuck.
>  >         >
>  >         > Background: I am a System Administrator in a closed,
>  >         classified environment. Unfortunately, I cannot post logging
>  >         here, but I can refer to them as needed.
>  >         >
>  >         > I inherited this system from someone who departed the program
>  >         a year or so ago. Fast forward to today, the server certs
>  >         expired yesterday. Admittedly, I'm unfamiliar (or was) with the
>  >         certificate update process for IPA servers. On a typical server,
>  >         we replace the old cert and restart the httpd services; however,
>  >         I realize this cannot work with IPA servers now.
>  >         >
>  >         > Additionally to all of this, the CA chain updated 6 months ago.
>  >         >
>  >         > I ran ipa-cacert-manage to update the CA chain. When trying to
>  >         run ipa-certupdate, I received errors for an invalid server
>  >         certificate (it expired on 11 April 2023). It simply won't
>  >         connect to the web server. HTTPD failed as well, so I had to add
>  >         "NSSEnforceValidCerts off" to the nss.conf file for HTTPD to
>  >         start. Still, no dice.
>  >         >
>  >         > I've ran ipa-server-certinstall for the new cert/key as well,
>  >         and it fails saying its not trusted ("Peer's certificate issuer
>  >         is not trusted [certutil: certificate is invalid: Peer's
>  >         Certificate issuer is not recognized] Please run
>  >         ipa-cacert-manage install and ipa-certupdate to install the CA
>  >         certificate.... which, as reported above, can't complete.
>  >         >
>  >         > I'm at a total loss here... and really struggling being new to
>  >         all this and trying my best to keep it afloat. Any help would be
>  >         GREATLY appreciated!
>  >
>  >         Let's gather some information first.
>  >
>  >         What version of IPA is this, on what distribution?
>  >
>  >         IPA designates one server to be the "renewal master" which
>  >         handles the
>  >         renewals. The output of `ipa config-show` should tell you
>  >         (depending on
>  >         version). That's the server you want to work on.
>  >
>  >         How many servers in your topology and how many have a CA
>  installed?
>  >
>  >         Does `getcert list` show a set of 8-10 tracked certificates?
>  >         What are
>  >         the states?
>  >
>  >         You mention ipa-server-certinstall. Are you using 3rd party
>  >         certificates
>  >         in addition to IPA CA-issued certificates or was that just an
>  >         attempt to
>  >         get things working again?
>  >
>  >         rob
>  >
>
>

[Justen Long]

Following up, I see the date command just changed it momentarily... using
timedatectl and will report back.

On Thu, Apr 13, 2023 at 12:31 PM Justen Long <mr.justenlong(a)gmail.com&gt;
wrote:

>  Rob,
>
>  I entered 'date --date="7 April 2023", verified it updated the system time
>  appropriately. Restarted dirsrv, ipa-custodia, ipa-otpd, httpd.. krb5kdc
>  and kadmin failed. Still, tried to send ipa cert-update, and it popped the
>  same SSL Certificate Verify Failed error.
>
>  On Thu, Apr 13, 2023 at 11:32 AM Rob Crittenden <rcritten(a)redhat.com&gt;
>  wrote:
>
> > Justen Long wrote:
> > > Additionally, is there any way to force the CA cert update to be
> > > recognized? When I run it to update the CA chain, everything is
> > > verified.. but /etc/ipa/ca.crt didn't reflect the change.. so I manually
> > > populated it by copying over the guts of the CA bundle to the
> > > /etc/ipa/ca.crt before trying to install the new server cert and it
> > > still doesn't recognize it as trusted although the issuer is the same
> > > and within the CA bundle.
> >
> > This is going to sound weird, but I'd just go back in time to April 10,
> > restart all services but ntp (which will reset the time) and then the
> > commands should work. Once the certs are updated and working, return to
> > present time.
> >
> > rob
> >
> > >
> > > On Thu, Apr 13, 2023 at 6:20 AM Justen Long <mr.justenlong(a)gmail.com
> > > <mailto:[e-mail]>> wrote:
> > >
> > >     Rob,
> > >
> > >     Apologies for the delay in response. Once I'm home, I don't have
> > >     access to the information readily available to respond with. Here is
> > >     the information you requested:
> > >
> > >     The version of IPA we are using is 4.6.8, rpm specifically for us is
> > >     ipa-server-4.6.8-5.el7.centos.12.x86_64 and we are using CentOS 7.9
> > >     currently with plans to move to RHEL9 within the next year or so.
> > >
> > >     Unfortunately, 'ipa config-show' doesn't work. It populates the
> same
> > >     error stating "ipa: ERROR: cannot connect to
> > >     'https://ipaServer/ipa/json': [SSL: CERTIFICATE_VERIFY_FAILED]
> > >     certificate verify failed (_ssl.c:618).
> >
> > The smack heard around the world was my head hitting my desk. Of course
> > this command failed.
> >
> > >
> > >     We have ~50 hosts connected via IPA. We have two IPA servers, one as
> > >     a replica of the other.
> > >
> > >     'getcert list' only shows 1 certificate. It's state is
> "MONITORING"
> > >     and seems related to kerberos.
> > >
> > >     As far as I know, we don't use IPA CA-issued certificates. I recall
> > >     seeing errors yesterday stating CA wasn't enabled on our servers. We
> > >     have always used 3rd party CAs to my knowledge.
> > >
> > >     -justen
> > >
> > >     On Wed, Apr 12, 2023 at 2:42 PM Rob Crittenden <rcritten(a)redhat.com
> > >     <mailto:[e-mail]>> wrote:
> > >
> > >         Justen Long via FreeIPA-users wrote:
> > >         > Thanks in advance for your replies.. I've spent 7 hours
> > >         looking through posts here and trying everything... I'm stuck.
> > >         >
> > >         > Background: I am a System Administrator in a closed,
> > >         classified environment. Unfortunately, I cannot post logging
> > >         here, but I can refer to them as needed.
> > >         >
> > >         > I inherited this system from someone who departed the program
> > >         a year or so ago. Fast forward to today, the server certs
> > >         expired yesterday. Admittedly, I'm unfamiliar (or was) with the
> > >         certificate update process for IPA servers. On a typical server,
> > >         we replace the old cert and restart the httpd services; however,
> > >         I realize this cannot work with IPA servers now.
> > >         >
> > >         > Additionally to all of this, the CA chain updated 6 months
> > ago.
> > >         >
> > >         > I ran ipa-cacert-manage to update the CA chain. When trying to
> > >         run ipa-certupdate, I received errors for an invalid server
> > >         certificate (it expired on 11 April 2023). It simply won't
> > >         connect to the web server. HTTPD failed as well, so I had to add
> > >         "NSSEnforceValidCerts off" to the nss.conf file for HTTPD to
> > >         start. Still, no dice.
> > >         >
> > >         > I've ran ipa-server-certinstall for the new cert/key as well,
> > >         and it fails saying its not trusted ("Peer's certificate
> issuer
> > >         is not trusted [certutil: certificate is invalid: Peer's
> > >         Certificate issuer is not recognized] Please run
> > >         ipa-cacert-manage install and ipa-certupdate to install the CA
> > >         certificate.... which, as reported above, can't complete.
> > >         >
> > >         > I'm at a total loss here... and really struggling being new to
> > >         all this and trying my best to keep it afloat. Any help would be
> > >         GREATLY appreciated!
> > >
> > >         Let's gather some information first.
> > >
> > >         What version of IPA is this, on what distribution?
> > >
> > >         IPA designates one server to be the "renewal master" which
> > >         handles the
> > >         renewals. The output of `ipa config-show` should tell you
> > >         (depending on
> > >         version). That's the server you want to work on.
> > >
> > >         How many servers in your topology and how many have a CA
> > installed?
> > >
> > >         Does `getcert list` show a set of 8-10 tracked certificates?
> > >         What are
> > >         the states?
> > >
> > >         You mention ipa-server-certinstall. Are you using 3rd party
> > >         certificates
> > >         in addition to IPA CA-issued certificates or was that just an
> > >         attempt to
> > >         get things working again?
> > >
> > >         rob
> > >
> >
> >

[Justen Long]

I'm getting closer... it's not recognizing my admin password for IPA, or
for my personal account with admin rights now.. but no more SSL errors..
just can't run ipa-certupdate without the proper kerberos creds..

On Thu, Apr 13, 2023 at 12:51 PM Justen Long <mr.justenlong(a)gmail.com&gt;
wrote:

>  Following up, I see the date command just changed it momentarily...
> using
>  timedatectl and will report back.
>
>  On Thu, Apr 13, 2023 at 12:31 PM Justen Long <mr.justenlong(a)gmail.com&gt;
>  wrote:
>
> > Rob,
> >
> > I entered 'date --date="7 April 2023", verified it updated the system
> > time appropriately. Restarted dirsrv, ipa-custodia, ipa-otpd, httpd..
> > krb5kdc and kadmin failed. Still, tried to send ipa cert-update, and it
> > popped the same SSL Certificate Verify Failed error.
> >
> > On Thu, Apr 13, 2023 at 11:32 AM Rob Crittenden <rcritten(a)redhat.com&gt;
> > wrote:
> >
> >> Justen Long wrote:
> >> > Additionally, is there any way to force the CA cert update to be
> >> > recognized? When I run it to update the CA chain, everything is
> >> > verified.. but /etc/ipa/ca.crt didn't reflect the change.. so I
> >> manually
> >> > populated it by copying over the guts of the CA bundle to the
> >> > /etc/ipa/ca.crt before trying to install the new server cert and it
> >> > still doesn't recognize it as trusted although the issuer is the same
> >> > and within the CA bundle.
> >>
> >> This is going to sound weird, but I'd just go back in time to April 10,
> >> restart all services but ntp (which will reset the time) and then the
> >> commands should work. Once the certs are updated and working, return to
> >> present time.
> >>
> >> rob
> >>
> >> >
> >> > On Thu, Apr 13, 2023 at 6:20 AM Justen Long <mr.justenlong(a)gmail.com
> >> > <mailto:[e-mail]>> wrote:
> >> >
> >> >     Rob,
> >> >
> >> >     Apologies for the delay in response. Once I'm home, I don't
> have
> >> >     access to the information readily available to respond with. Here
> >> is
> >> >     the information you requested:
> >> >
> >> >     The version of IPA we are using is 4.6.8, rpm specifically for us
> >> is
> >> >     ipa-server-4.6.8-5.el7.centos.12.x86_64 and we are using CentOS 7.9
> >> >     currently with plans to move to RHEL9 within the next year or so.
> >> >
> >> >     Unfortunately, 'ipa config-show' doesn't work. It populates
> the
> >> same
> >> >     error stating "ipa: ERROR: cannot connect to
> >> >     'https://ipaServer/ipa/json': [SSL: CERTIFICATE_VERIFY_FAILED]
> >> >     certificate verify failed (_ssl.c:618).
> >>
> >> The smack heard around the world was my head hitting my desk. Of course
> >> this command failed.
> >>
> >> >
> >> >     We have ~50 hosts connected via IPA. We have two IPA servers, one
> >> as
> >> >     a replica of the other.
> >> >
> >> >     'getcert list' only shows 1 certificate. It's state is
> "MONITORING"
> >> >     and seems related to kerberos.
> >> >
> >> >     As far as I know, we don't use IPA CA-issued certificates. I recall
> >> >     seeing errors yesterday stating CA wasn't enabled on our servers.
> >> We
> >> >     have always used 3rd party CAs to my knowledge.
> >> >
> >> >     -justen
> >> >
> >> >     On Wed, Apr 12, 2023 at 2:42 PM Rob Crittenden <
> >> rcritten(a)redhat.com
> >> >     <mailto:[e-mail]>> wrote:
> >> >
> >> >         Justen Long via FreeIPA-users wrote:
> >> >         > Thanks in advance for your replies.. I've spent 7 hours
> >> >         looking through posts here and trying everything... I'm stuck.
> >> >         >
> >> >         > Background: I am a System Administrator in a closed,
> >> >         classified environment. Unfortunately, I cannot post logging
> >> >         here, but I can refer to them as needed.
> >> >         >
> >> >         > I inherited this system from someone who departed the program
> >> >         a year or so ago. Fast forward to today, the server certs
> >> >         expired yesterday. Admittedly, I'm unfamiliar (or was) with the
> >> >         certificate update process for IPA servers. On a typical
> >> server,
> >> >         we replace the old cert and restart the httpd services;
> >> however,
> >> >         I realize this cannot work with IPA servers now.
> >> >         >
> >> >         > Additionally to all of this, the CA chain updated 6 months
> >> ago.
> >> >         >
> >> >         > I ran ipa-cacert-manage to update the CA chain. When trying
> >> to
> >> >         run ipa-certupdate, I received errors for an invalid server
> >> >         certificate (it expired on 11 April 2023). It simply won't
> >> >         connect to the web server. HTTPD failed as well, so I had to
> >> add
> >> >         "NSSEnforceValidCerts off" to the nss.conf file for HTTPD
> to
> >> >         start. Still, no dice.
> >> >         >
> >> >         > I've ran ipa-server-certinstall for the new cert/key as
> well,
> >> >         and it fails saying its not trusted ("Peer's certificate
> issuer
> >> >         is not trusted [certutil: certificate is invalid: Peer's
> >> >         Certificate issuer is not recognized] Please run
> >> >         ipa-cacert-manage install and ipa-certupdate to install the CA
> >> >         certificate.... which, as reported above, can't complete.
> >> >         >
> >> >         > I'm at a total loss here... and really struggling being
> new
> >> to
> >> >         all this and trying my best to keep it afloat. Any help would
> >> be
> >> >         GREATLY appreciated!
> >> >
> >> >         Let's gather some information first.
> >> >
> >> >         What version of IPA is this, on what distribution?
> >> >
> >> >         IPA designates one server to be the "renewal master"
> which
> >> >         handles the
> >> >         renewals. The output of `ipa config-show` should tell you
> >> >         (depending on
> >> >         version). That's the server you want to work on.
> >> >
> >> >         How many servers in your topology and how many have a CA
> >> installed?
> >> >
> >> >         Does `getcert list` show a set of 8-10 tracked certificates?
> >> >         What are
> >> >         the states?
> >> >
> >> >         You mention ipa-server-certinstall. Are you using 3rd party
> >> >         certificates
> >> >         in addition to IPA CA-issued certificates or was that just an
> >> >         attempt to
> >> >         get things working again?
> >> >
> >> >         rob
> >> >
> >>
> >>

[Rob Crittenden]

Justen Long wrote:
>  I'm getting closer... it's not recognizing my admin password
> for IPA, or
>  for my personal account with admin rights now.. but no more SSL errors..
>  just can't run ipa-certupdate without the proper kerberos creds.. 
By not recognizing your password I assume you mean kinit is failing? Is
the KDC running? I assume 389-ds is running? All restarted after time
became stable in the past?

rob

>  
>  On Thu, Apr 13, 2023 at 12:51 PM Justen Long <mr.justenlong(a)gmail.com
>  <mailto:[e-mail]>> wrote:
>  
>      Following up, I see the date command just changed it momentarily...
>      using timedatectl and will report back.
>  
>      On Thu, Apr 13, 2023 at 12:31 PM Justen Long
>      <mr.justenlong(a)gmail.com <mailto:[e-mail]>> wrote:
>  
>          Rob,
>  
>          I entered 'date --date="7 April 2023", verified it updated the
>          system time appropriately. Restarted dirsrv, ipa-custodia,
>          ipa-otpd, httpd.. krb5kdc and kadmin failed. Still, tried to
>          send ipa cert-update, and it popped the same SSL Certificate
>          Verify Failed error.
>  
>          On Thu, Apr 13, 2023 at 11:32 AM Rob Crittenden
>          <rcritten(a)redhat.com <mailto:[e-mail]>> wrote:
>  
>              Justen Long wrote:
>              > Additionally, is there any way to force the CA cert update
>              to be
>              > recognized? When I run it to update the CA chain,
>              everything is
>              > verified.. but /etc/ipa/ca.crt didn't reflect the change..
>              so I manually
>              > populated it by copying over the guts of the CA bundle to the
>              > /etc/ipa/ca.crt before trying to install the new server
>              cert and it
>              > still doesn't recognize it as trusted although the issuer
>              is the same
>              > and within the CA bundle.
>  
>              This is going to sound weird, but I'd just go back in time
>              to April 10,
>              restart all services but ntp (which will reset the time) and
>              then the
>              commands should work. Once the certs are updated and
>              working, return to
>              present time.
>  
>              rob
>  
>              >
>              > On Thu, Apr 13, 2023 at 6:20 AM Justen Long
>              <mr.justenlong(a)gmail.com <mailto:[e-mail]>
>              > <mailto:[e-mail]
>              <mailto:[e-mail]>>> wrote:
>              >
>              >     Rob,
>              >
>              >     Apologies for the delay in response. Once I'm home, I
>              don't have
>              >     access to the information readily available to respond
>              with. Here is
>              >     the information you requested:
>              >
>              >     The version of IPA we are using is 4.6.8, rpm
>              specifically for us is
>              >     ipa-server-4.6.8-5.el7.centos.12.x86_64 and we are
>              using CentOS 7.9
>              >     currently with plans to move to RHEL9 within the next
>              year or so.
>              >
>              >     Unfortunately, 'ipa config-show' doesn't work. It
>              populates the same
>              >     error stating "ipa: ERROR: cannot connect to
>              >     'https://ipaServer/ipa/json': [SSL:
>              CERTIFICATE_VERIFY_FAILED]
>              >     certificate verify failed (_ssl.c:618).
>  
>              The smack heard around the world was my head hitting my
>              desk. Of course
>              this command failed.
>  
>              >
>              >     We have ~50 hosts connected via IPA. We have two IPA
>              servers, one as
>              >     a replica of the other.
>              >
>              >     'getcert list' only shows 1 certificate. It's state is
>              "MONITORING"
>              >     and seems related to kerberos.
>              >
>              >     As far as I know, we don't use IPA CA-issued
>              certificates. I recall
>              >     seeing errors yesterday stating CA wasn't enabled on
>              our servers. We
>              >     have always used 3rd party CAs to my knowledge.
>              >
>              >     -justen
>              >
>              >     On Wed, Apr 12, 2023 at 2:42 PM Rob Crittenden
>              <rcritten(a)redhat.com <mailto:[e-mail]>
>              >     <mailto:[e-mail]
>              <mailto:[e-mail]>>> wrote:
>              >
>              >         Justen Long via FreeIPA-users wrote:
>              >         > Thanks in advance for your replies.. I've spent
>              7 hours
>              >         looking through posts here and trying
>              everything... I'm stuck.
>              >         >
>              >         > Background: I am a System Administrator in a closed,
>              >         classified environment. Unfortunately, I cannot
>              post logging
>              >         here, but I can refer to them as needed.
>              >         >
>              >         > I inherited this system from someone who
>              departed the program
>              >         a year or so ago. Fast forward to today, the
>              server certs
>              >         expired yesterday. Admittedly, I'm unfamiliar (or
>              was) with the
>              >         certificate update process for IPA servers. On a
>              typical server,
>              >         we replace the old cert and restart the httpd
>              services; however,
>              >         I realize this cannot work with IPA servers now.
>              >         >
>              >         > Additionally to all of this, the CA chain
>              updated 6 months ago.
>              >         >
>              >         > I ran ipa-cacert-manage to update the CA chain.
>              When trying to
>              >         run ipa-certupdate, I received errors for an
>              invalid server
>              >         certificate (it expired on 11 April 2023). It
>              simply won't
>              >         connect to the web server. HTTPD failed as well,
>              so I had to add
>              >         "NSSEnforceValidCerts off" to the nss.conf file
>              for HTTPD to
>              >         start. Still, no dice.
>              >         >
>              >         > I've ran ipa-server-certinstall for the new
>              cert/key as well,
>              >         and it fails saying its not trusted ("Peer's
>              certificate issuer
>              >         is not trusted [certutil: certificate is invalid:
>              Peer's
>              >         Certificate issuer is not recognized] Please run
>              >         ipa-cacert-manage install and ipa-certupdate to
>              install the CA
>              >         certificate.... which, as reported above, can't
>              complete.
>              >         >
>              >         > I'm at a total loss here... and really
>              struggling being new to
>              >         all this and trying my best to keep it afloat. Any
>              help would be
>              >         GREATLY appreciated!
>              >
>              >         Let's gather some information first.
>              >
>              >         What version of IPA is this, on what distribution?
>              >
>              >         IPA designates one server to be the "renewal
>              master" which
>              >         handles the
>              >         renewals. The output of `ipa config-show` should
>              tell you
>              >         (depending on
>              >         version). That's the server you want to work on.
>              >
>              >         How many servers in your topology and how many
>              have a CA installed?
>              >
>              >         Does `getcert list` show a set of 8-10 tracked
>              certificates?
>              >         What are
>              >         the states?
>              >
>              >         You mention ipa-server-certinstall. Are you using
>              3rd party
>              >         certificates
>              >         in addition to IPA CA-issued certificates or was
>              that just an
>              >         attempt to
>              >         get things working again?
>              >
>              >         rob
>              >
>

[Justen Long]

Quick update and answers to your questions.

We have two IPA servers, both masters, hiipa03 and hiipa04.

hiipa04, I was able to set the time back, run the ipa-ca-cert-manage and
update the CA (minimally, but enough to accept the new cert)..
ipa-certupdate ran fine on it. Then, ran ipa-server-certinstall on it, and
it took. Website loads, can log into it, do some user management stuff..
can't run "ipa-replica-manage list" as it kicks an error for sslv3
handshake failure.. but was trying that to remove hiipa03 and copy 04 to 03
somehow, maybe?

hiipa03 is still giving me grief. Set the time using timedatectl, verified
ntp is off and 'date' reports properly.  I had tried to follow this:
https://computingforgeeks.com/reset-freeipa-admin-password-as-root-user-o...
when it wasn't accepting the admin password.. it failed saying object not
found. So, I try 'kinit admin' again, with the new password I tried.. says
its expired. Type it in, type in a new password.. and then it failed saying
"kinit: Password change failed while getting initial credentials"

On Thu, Apr 13, 2023 at 1:40 PM Rob Crittenden <rcritten(a)redhat.com&gt; wrote:

>  Justen Long wrote:
>  > I'm getting closer... it's not recognizing my admin password for IPA, or
>  > for my personal account with admin rights now.. but no more SSL errors..
>  > just can't run ipa-certupdate without the proper kerberos creds..
>
>  By not recognizing your password I assume you mean kinit is failing? Is
>  the KDC running? I assume 389-ds is running? All restarted after time
>  became stable in the past?
>
>  rob
>
>  >
>  > On Thu, Apr 13, 2023 at 12:51 PM Justen Long <mr.justenlong(a)gmail.com
>  > <mailto:[e-mail]>> wrote:
>  >
>  >     Following up, I see the date command just changed it momentarily...
>  >     using timedatectl and will report back.
>  >
>  >     On Thu, Apr 13, 2023 at 12:31 PM Justen Long
>  >     <mr.justenlong(a)gmail.com <mailto:[e-mail]>> wrote:
>  >
>  >         Rob,
>  >
>  >         I entered 'date --date="7 April 2023", verified it updated
> the
>  >         system time appropriately. Restarted dirsrv, ipa-custodia,
>  >         ipa-otpd, httpd.. krb5kdc and kadmin failed. Still, tried to
>  >         send ipa cert-update, and it popped the same SSL Certificate
>  >         Verify Failed error.
>  >
>  >         On Thu, Apr 13, 2023 at 11:32 AM Rob Crittenden
>  >         <rcritten(a)redhat.com <mailto:[e-mail]>> wrote:
>  >
>  >             Justen Long wrote:
>  >             > Additionally, is there any way to force the CA cert update
>  >             to be
>  >             > recognized? When I run it to update the CA chain,
>  >             everything is
>  >             > verified.. but /etc/ipa/ca.crt didn't reflect the change..
>  >             so I manually
>  >             > populated it by copying over the guts of the CA bundle to
>  the
>  >             > /etc/ipa/ca.crt before trying to install the new server
>  >             cert and it
>  >             > still doesn't recognize it as trusted although the issuer
>  >             is the same
>  >             > and within the CA bundle.
>  >
>  >             This is going to sound weird, but I'd just go back in time
>  >             to April 10,
>  >             restart all services but ntp (which will reset the time) and
>  >             then the
>  >             commands should work. Once the certs are updated and
>  >             working, return to
>  >             present time.
>  >
>  >             rob
>  >
>  >             >
>  >             > On Thu, Apr 13, 2023 at 6:20 AM Justen Long
>  >             <mr.justenlong(a)gmail.com <mailto:[e-mail]>
>  >             > <mailto:[e-mail]
>  >             <mailto:[e-mail]>>> wrote:
>  >             >
>  >             >     Rob,
>  >             >
>  >             >     Apologies for the delay in response. Once I'm home, I
>  >             don't have
>  >             >     access to the information readily available to respond
>  >             with. Here is
>  >             >     the information you requested:
>  >             >
>  >             >     The version of IPA we are using is 4.6.8, rpm
>  >             specifically for us is
>  >             >     ipa-server-4.6.8-5.el7.centos.12.x86_64 and we are
>  >             using CentOS 7.9
>  >             >     currently with plans to move to RHEL9 within the next
>  >             year or so.
>  >             >
>  >             >     Unfortunately, 'ipa config-show' doesn't work. It
>  >             populates the same
>  >             >     error stating "ipa: ERROR: cannot connect to
>  >             >     'https://ipaServer/ipa/json': [SSL:
>  >             CERTIFICATE_VERIFY_FAILED]
>  >             >     certificate verify failed (_ssl.c:618).
>  >
>  >             The smack heard around the world was my head hitting my
>  >             desk. Of course
>  >             this command failed.
>  >
>  >             >
>  >             >     We have ~50 hosts connected via IPA. We have two IPA
>  >             servers, one as
>  >             >     a replica of the other.
>  >             >
>  >             >     'getcert list' only shows 1 certificate. It's state
> is
>  >             "MONITORING"
>  >             >     and seems related to kerberos.
>  >             >
>  >             >     As far as I know, we don't use IPA CA-issued
>  >             certificates. I recall
>  >             >     seeing errors yesterday stating CA wasn't enabled on
>  >             our servers. We
>  >             >     have always used 3rd party CAs to my knowledge.
>  >             >
>  >             >     -justen
>  >             >
>  >             >     On Wed, Apr 12, 2023 at 2:42 PM Rob Crittenden
>  >             <rcritten(a)redhat.com <mailto:[e-mail]>
>  >             >     <mailto:[e-mail]
>  >             <mailto:[e-mail]>>> wrote:
>  >             >
>  >             >         Justen Long via FreeIPA-users wrote:
>  >             >         > Thanks in advance for your replies.. I've spent
>  >             7 hours
>  >             >         looking through posts here and trying
>  >             everything... I'm stuck.
>  >             >         >
>  >             >         > Background: I am a System Administrator in a
>  closed,
>  >             >         classified environment. Unfortunately, I cannot
>  >             post logging
>  >             >         here, but I can refer to them as needed.
>  >             >         >
>  >             >         > I inherited this system from someone who
>  >             departed the program
>  >             >         a year or so ago. Fast forward to today, the
>  >             server certs
>  >             >         expired yesterday. Admittedly, I'm unfamiliar (or
>  >             was) with the
>  >             >         certificate update process for IPA servers. On a
>  >             typical server,
>  >             >         we replace the old cert and restart the httpd
>  >             services; however,
>  >             >         I realize this cannot work with IPA servers now.
>  >             >         >
>  >             >         > Additionally to all of this, the CA chain
>  >             updated 6 months ago.
>  >             >         >
>  >             >         > I ran ipa-cacert-manage to update the CA chain.
>  >             When trying to
>  >             >         run ipa-certupdate, I received errors for an
>  >             invalid server
>  >             >         certificate (it expired on 11 April 2023). It
>  >             simply won't
>  >             >         connect to the web server. HTTPD failed as well,
>  >             so I had to add
>  >             >         "NSSEnforceValidCerts off" to the nss.conf file
>  >             for HTTPD to
>  >             >         start. Still, no dice.
>  >             >         >
>  >             >         > I've ran ipa-server-certinstall for the new
>  >             cert/key as well,
>  >             >         and it fails saying its not trusted ("Peer's
>  >             certificate issuer
>  >             >         is not trusted [certutil: certificate is invalid:
>  >             Peer's
>  >             >         Certificate issuer is not recognized] Please run
>  >             >         ipa-cacert-manage install and ipa-certupdate to
>  >             install the CA
>  >             >         certificate.... which, as reported above, can't
>  >             complete.
>  >             >         >
>  >             >         > I'm at a total loss here... and really
>  >             struggling being new to
>  >             >         all this and trying my best to keep it afloat. Any
>  >             help would be
>  >             >         GREATLY appreciated!
>  >             >
>  >             >         Let's gather some information first.
>  >             >
>  >             >         What version of IPA is this, on what distribution?
>  >             >
>  >             >         IPA designates one server to be the "renewal
>  >             master" which
>  >             >         handles the
>  >             >         renewals. The output of `ipa config-show` should
>  >             tell you
>  >             >         (depending on
>  >             >         version). That's the server you want to work on.
>  >             >
>  >             >         How many servers in your topology and how many
>  >             have a CA installed?
>  >             >
>  >             >         Does `getcert list` show a set of 8-10 tracked
>  >             certificates?
>  >             >         What are
>  >             >         the states?
>  >             >
>  >             >         You mention ipa-server-certinstall. Are you using
>  >             3rd party
>  >             >         certificates
>  >             >         in addition to IPA CA-issued certificates or was
>  >             that just an
>  >             >         attempt to
>  >             >         get things working again?
>  >             >
>  >             >         rob
>  >             >
>  >
>
>

[Justen Long]

One go back.. when I tried to run "ipa-certupdate" on the other hosts
(clients), it points to hiipa03 and fails for SSL still.. so, hoping once I
get the cert updated on THAT server, that all restores to its okay state.

On Thu, Apr 13, 2023 at 1:46 PM Justen Long <mr.justenlong(a)gmail.com&gt; wrote:

>  Quick update and answers to your questions.
>
>  We have two IPA servers, both masters, hiipa03 and hiipa04.
>
>  hiipa04, I was able to set the time back, run the ipa-ca-cert-manage and
>  update the CA (minimally, but enough to accept the new cert)..
>  ipa-certupdate ran fine on it. Then, ran ipa-server-certinstall on it, and
>  it took. Website loads, can log into it, do some user management stuff..
>  can't run "ipa-replica-manage list" as it kicks an error for sslv3
>  handshake failure.. but was trying that to remove hiipa03 and copy 04 to 03
>  somehow, maybe?
>
>  hiipa03 is still giving me grief. Set the time using timedatectl, verified
>  ntp is off and 'date' reports properly.  I had tried to follow this:
>  https://computingforgeeks.com/reset-freeipa-admin-password-as-root-user-o...
>  when it wasn't accepting the admin password.. it failed saying object not
>  found. So, I try 'kinit admin' again, with the new password I tried.. says
>  its expired. Type it in, type in a new password.. and then it failed saying
>  "kinit: Password change failed while getting initial credentials"
>
>  On Thu, Apr 13, 2023 at 1:40 PM Rob Crittenden <rcritten(a)redhat.com&gt;
>  wrote:
>
> > Justen Long wrote:
> > > I'm getting closer... it's not recognizing my admin password for IPA,
> or
> > > for my personal account with admin rights now.. but no more SSL errors..
> > > just can't run ipa-certupdate without the proper kerberos creds..
> >
> > By not recognizing your password I assume you mean kinit is failing? Is
> > the KDC running? I assume 389-ds is running? All restarted after time
> > became stable in the past?
> >
> > rob
> >
> > >
> > > On Thu, Apr 13, 2023 at 12:51 PM Justen Long <mr.justenlong(a)gmail.com
> > > <mailto:[e-mail]>> wrote:
> > >
> > >     Following up, I see the date command just changed it momentarily...
> > >     using timedatectl and will report back.
> > >
> > >     On Thu, Apr 13, 2023 at 12:31 PM Justen Long
> > >     <mr.justenlong(a)gmail.com <mailto:[e-mail]>>
> wrote:
> > >
> > >         Rob,
> > >
> > >         I entered 'date --date="7 April 2023", verified it updated
> the
> > >         system time appropriately. Restarted dirsrv, ipa-custodia,
> > >         ipa-otpd, httpd.. krb5kdc and kadmin failed. Still, tried to
> > >         send ipa cert-update, and it popped the same SSL Certificate
> > >         Verify Failed error.
> > >
> > >         On Thu, Apr 13, 2023 at 11:32 AM Rob Crittenden
> > >         <rcritten(a)redhat.com <mailto:[e-mail]>> wrote:
> > >
> > >             Justen Long wrote:
> > >             > Additionally, is there any way to force the CA cert update
> > >             to be
> > >             > recognized? When I run it to update the CA chain,
> > >             everything is
> > >             > verified.. but /etc/ipa/ca.crt didn't reflect the change..
> > >             so I manually
> > >             > populated it by copying over the guts of the CA bundle to
> > the
> > >             > /etc/ipa/ca.crt before trying to install the new server
> > >             cert and it
> > >             > still doesn't recognize it as trusted although the issuer
> > >             is the same
> > >             > and within the CA bundle.
> > >
> > >             This is going to sound weird, but I'd just go back in time
> > >             to April 10,
> > >             restart all services but ntp (which will reset the time) and
> > >             then the
> > >             commands should work. Once the certs are updated and
> > >             working, return to
> > >             present time.
> > >
> > >             rob
> > >
> > >             >
> > >             > On Thu, Apr 13, 2023 at 6:20 AM Justen Long
> > >             <mr.justenlong(a)gmail.com <mailto:[e-mail]>
> > >             > <mailto:[e-mail]
> > >             <mailto:[e-mail]>>> wrote:
> > >             >
> > >             >     Rob,
> > >             >
> > >             >     Apologies for the delay in response. Once I'm home, I
> > >             don't have
> > >             >     access to the information readily available to respond
> > >             with. Here is
> > >             >     the information you requested:
> > >             >
> > >             >     The version of IPA we are using is 4.6.8, rpm
> > >             specifically for us is
> > >             >     ipa-server-4.6.8-5.el7.centos.12.x86_64 and we are
> > >             using CentOS 7.9
> > >             >     currently with plans to move to RHEL9 within the next
> > >             year or so.
> > >             >
> > >             >     Unfortunately, 'ipa config-show' doesn't work.
> It
> > >             populates the same
> > >             >     error stating "ipa: ERROR: cannot connect to
> > >             >     'https://ipaServer/ipa/json': [SSL:
> > >             CERTIFICATE_VERIFY_FAILED]
> > >             >     certificate verify failed (_ssl.c:618).
> > >
> > >             The smack heard around the world was my head hitting my
> > >             desk. Of course
> > >             this command failed.
> > >
> > >             >
> > >             >     We have ~50 hosts connected via IPA. We have two IPA
> > >             servers, one as
> > >             >     a replica of the other.
> > >             >
> > >             >     'getcert list' only shows 1 certificate. It's
> state is
> > >             "MONITORING"
> > >             >     and seems related to kerberos.
> > >             >
> > >             >     As far as I know, we don't use IPA CA-issued
> > >             certificates. I recall
> > >             >     seeing errors yesterday stating CA wasn't enabled on
> > >             our servers. We
> > >             >     have always used 3rd party CAs to my knowledge.
> > >             >
> > >             >     -justen
> > >             >
> > >             >     On Wed, Apr 12, 2023 at 2:42 PM Rob Crittenden
> > >             <rcritten(a)redhat.com <mailto:[e-mail]>
> > >             >     <mailto:[e-mail]
> > >             <mailto:[e-mail]>>> wrote:
> > >             >
> > >             >         Justen Long via FreeIPA-users wrote:
> > >             >         > Thanks in advance for your replies.. I've
> spent
> > >             7 hours
> > >             >         looking through posts here and trying
> > >             everything... I'm stuck.
> > >             >         >
> > >             >         > Background: I am a System Administrator in a
> > closed,
> > >             >         classified environment. Unfortunately, I cannot
> > >             post logging
> > >             >         here, but I can refer to them as needed.
> > >             >         >
> > >             >         > I inherited this system from someone who
> > >             departed the program
> > >             >         a year or so ago. Fast forward to today, the
> > >             server certs
> > >             >         expired yesterday. Admittedly, I'm unfamiliar (or
> > >             was) with the
> > >             >         certificate update process for IPA servers. On a
> > >             typical server,
> > >             >         we replace the old cert and restart the httpd
> > >             services; however,
> > >             >         I realize this cannot work with IPA servers now.
> > >             >         >
> > >             >         > Additionally to all of this, the CA chain
> > >             updated 6 months ago.
> > >             >         >
> > >             >         > I ran ipa-cacert-manage to update the CA chain.
> > >             When trying to
> > >             >         run ipa-certupdate, I received errors for an
> > >             invalid server
> > >             >         certificate (it expired on 11 April 2023). It
> > >             simply won't
> > >             >         connect to the web server. HTTPD failed as well,
> > >             so I had to add
> > >             >         "NSSEnforceValidCerts off" to the nss.conf
> file
> > >             for HTTPD to
> > >             >         start. Still, no dice.
> > >             >         >
> > >             >         > I've ran ipa-server-certinstall for the new
> > >             cert/key as well,
> > >             >         and it fails saying its not trusted ("Peer's
> > >             certificate issuer
> > >             >         is not trusted [certutil: certificate is invalid:
> > >             Peer's
> > >             >         Certificate issuer is not recognized] Please run
> > >             >         ipa-cacert-manage install and ipa-certupdate to
> > >             install the CA
> > >             >         certificate.... which, as reported above, can't
> > >             complete.
> > >             >         >
> > >             >         > I'm at a total loss here... and really
> > >             struggling being new to
> > >             >         all this and trying my best to keep it afloat. Any
> > >             help would be
> > >             >         GREATLY appreciated!
> > >             >
> > >             >         Let's gather some information first.
> > >             >
> > >             >         What version of IPA is this, on what distribution?
> > >             >
> > >             >         IPA designates one server to be the "renewal
> > >             master" which
> > >             >         handles the
> > >             >         renewals. The output of `ipa config-show` should
> > >             tell you
> > >             >         (depending on
> > >             >         version). That's the server you want to work on.
> > >             >
> > >             >         How many servers in your topology and how many
> > >             have a CA installed?
> > >             >
> > >             >         Does `getcert list` show a set of 8-10 tracked
> > >             certificates?
> > >             >         What are
> > >             >         the states?
> > >             >
> > >             >         You mention ipa-server-certinstall. Are you using
> > >             3rd party
> > >             >         certificates
> > >             >         in addition to IPA CA-issued certificates or was
> > >             that just an
> > >             >         attempt to
> > >             >         get things working again?
> > >             >
> > >             >         rob
> > >             >
> > >
> >
> >

[Justen Long]

Also just tried to change password again.. I see where I missed a dc=
entry.. fixed that.

Enter "new" password (which is what is existing on the other server), enter
it again.. enter dirsrv (LDAP) password.. kicks back Result: Operations
error (1) \ Additional info: Failed to update password

On Thu, Apr 13, 2023 at 1:48 PM Justen Long <mr.justenlong(a)gmail.com&gt; wrote:

>  One go back.. when I tried to run "ipa-certupdate" on the
> other hosts
>  (clients), it points to hiipa03 and fails for SSL still.. so, hoping once I
>  get the cert updated on THAT server, that all restores to its okay state.
>
>  On Thu, Apr 13, 2023 at 1:46 PM Justen Long <mr.justenlong(a)gmail.com&gt;
>  wrote:
>
> > Quick update and answers to your questions.
> >
> > We have two IPA servers, both masters, hiipa03 and hiipa04.
> >
> > hiipa04, I was able to set the time back, run the ipa-ca-cert-manage and
> > update the CA (minimally, but enough to accept the new cert)..
> > ipa-certupdate ran fine on it. Then, ran ipa-server-certinstall on it, and
> > it took. Website loads, can log into it, do some user management stuff..
> > can't run "ipa-replica-manage list" as it kicks an error for sslv3
> > handshake failure.. but was trying that to remove hiipa03 and copy 04 to 03
> > somehow, maybe?
> >
> > hiipa03 is still giving me grief. Set the time using timedatectl,
> > verified ntp is off and 'date' reports properly.  I had tried to follow
> > this:
> > https://computingforgeeks.com/reset-freeipa-admin-password-as-root-user-o...
> > when it wasn't accepting the admin password.. it failed saying object not
> > found. So, I try 'kinit admin' again, with the new password I tried.. says
> > its expired. Type it in, type in a new password.. and then it failed saying
> > "kinit: Password change failed while getting initial credentials"
> >
> > On Thu, Apr 13, 2023 at 1:40 PM Rob Crittenden <rcritten(a)redhat.com&gt;
> > wrote:
> >
> >> Justen Long wrote:
> >> > I'm getting closer... it's not recognizing my admin password for
> IPA,
> >> or
> >> > for my personal account with admin rights now.. but no more SSL
> >> errors..
> >> > just can't run ipa-certupdate without the proper kerberos creds..
> >>
> >> By not recognizing your password I assume you mean kinit is failing? Is
> >> the KDC running? I assume 389-ds is running? All restarted after time
> >> became stable in the past?
> >>
> >> rob
> >>
> >> >
> >> > On Thu, Apr 13, 2023 at 12:51 PM Justen Long <mr.justenlong(a)gmail.com
> >> > <mailto:[e-mail]>> wrote:
> >> >
> >> >     Following up, I see the date command just changed it momentarily...
> >> >     using timedatectl and will report back.
> >> >
> >> >     On Thu, Apr 13, 2023 at 12:31 PM Justen Long
> >> >     <mr.justenlong(a)gmail.com <mailto:[e-mail]>>
> wrote:
> >> >
> >> >         Rob,
> >> >
> >> >         I entered 'date --date="7 April 2023", verified it
> updated the
> >> >         system time appropriately. Restarted dirsrv, ipa-custodia,
> >> >         ipa-otpd, httpd.. krb5kdc and kadmin failed. Still, tried to
> >> >         send ipa cert-update, and it popped the same SSL Certificate
> >> >         Verify Failed error.
> >> >
> >> >         On Thu, Apr 13, 2023 at 11:32 AM Rob Crittenden
> >> >         <rcritten(a)redhat.com <mailto:[e-mail]>>
> wrote:
> >> >
> >> >             Justen Long wrote:
> >> >             > Additionally, is there any way to force the CA cert
> >> update
> >> >             to be
> >> >             > recognized? When I run it to update the CA chain,
> >> >             everything is
> >> >             > verified.. but /etc/ipa/ca.crt didn't reflect the
> >> change..
> >> >             so I manually
> >> >             > populated it by copying over the guts of the CA bundle
> >> to the
> >> >             > /etc/ipa/ca.crt before trying to install the new server
> >> >             cert and it
> >> >             > still doesn't recognize it as trusted although the
> issuer
> >> >             is the same
> >> >             > and within the CA bundle.
> >> >
> >> >             This is going to sound weird, but I'd just go back in time
> >> >             to April 10,
> >> >             restart all services but ntp (which will reset the time)
> >> and
> >> >             then the
> >> >             commands should work. Once the certs are updated and
> >> >             working, return to
> >> >             present time.
> >> >
> >> >             rob
> >> >
> >> >             >
> >> >             > On Thu, Apr 13, 2023 at 6:20 AM Justen Long
> >> >             <mr.justenlong(a)gmail.com
> <mailto:[e-mail]>
> >> >             > <mailto:[e-mail]
> >> >             <mailto:[e-mail]>>> wrote:
> >> >             >
> >> >             >     Rob,
> >> >             >
> >> >             >     Apologies for the delay in response. Once I'm home,
> I
> >> >             don't have
> >> >             >     access to the information readily available to
> >> respond
> >> >             with. Here is
> >> >             >     the information you requested:
> >> >             >
> >> >             >     The version of IPA we are using is 4.6.8, rpm
> >> >             specifically for us is
> >> >             >     ipa-server-4.6.8-5.el7.centos.12.x86_64 and we are
> >> >             using CentOS 7.9
> >> >             >     currently with plans to move to RHEL9 within the next
> >> >             year or so.
> >> >             >
> >> >             >     Unfortunately, 'ipa config-show' doesn't
> work. It
> >> >             populates the same
> >> >             >     error stating "ipa: ERROR: cannot connect to
> >> >             >     'https://ipaServer/ipa/json': [SSL:
> >> >             CERTIFICATE_VERIFY_FAILED]
> >> >             >     certificate verify failed (_ssl.c:618).
> >> >
> >> >             The smack heard around the world was my head hitting my
> >> >             desk. Of course
> >> >             this command failed.
> >> >
> >> >             >
> >> >             >     We have ~50 hosts connected via IPA. We have two IPA
> >> >             servers, one as
> >> >             >     a replica of the other.
> >> >             >
> >> >             >     'getcert list' only shows 1 certificate.
> It's state
> >> is
> >> >             "MONITORING"
> >> >             >     and seems related to kerberos.
> >> >             >
> >> >             >     As far as I know, we don't use IPA CA-issued
> >> >             certificates. I recall
> >> >             >     seeing errors yesterday stating CA wasn't enabled
> on
> >> >             our servers. We
> >> >             >     have always used 3rd party CAs to my knowledge.
> >> >             >
> >> >             >     -justen
> >> >             >
> >> >             >     On Wed, Apr 12, 2023 at 2:42 PM Rob Crittenden
> >> >             <rcritten(a)redhat.com <mailto:[e-mail]>
> >> >             >     <mailto:[e-mail]
> >> >             <mailto:[e-mail]>>> wrote:
> >> >             >
> >> >             >         Justen Long via FreeIPA-users wrote:
> >> >             >         > Thanks in advance for your replies.. I've
> spent
> >> >             7 hours
> >> >             >         looking through posts here and trying
> >> >             everything... I'm stuck.
> >> >             >         >
> >> >             >         > Background: I am a System Administrator in a
> >> closed,
> >> >             >         classified environment. Unfortunately, I cannot
> >> >             post logging
> >> >             >         here, but I can refer to them as needed.
> >> >             >         >
> >> >             >         > I inherited this system from someone who
> >> >             departed the program
> >> >             >         a year or so ago. Fast forward to today, the
> >> >             server certs
> >> >             >         expired yesterday. Admittedly, I'm unfamiliar
> (or
> >> >             was) with the
> >> >             >         certificate update process for IPA servers. On a
> >> >             typical server,
> >> >             >         we replace the old cert and restart the httpd
> >> >             services; however,
> >> >             >         I realize this cannot work with IPA servers now.
> >> >             >         >
> >> >             >         > Additionally to all of this, the CA chain
> >> >             updated 6 months ago.
> >> >             >         >
> >> >             >         > I ran ipa-cacert-manage to update the CA
> chain.
> >> >             When trying to
> >> >             >         run ipa-certupdate, I received errors for an
> >> >             invalid server
> >> >             >         certificate (it expired on 11 April 2023). It
> >> >             simply won't
> >> >             >         connect to the web server. HTTPD failed as well,
> >> >             so I had to add
> >> >             >         "NSSEnforceValidCerts off" to the
> nss.conf file
> >> >             for HTTPD to
> >> >             >         start. Still, no dice.
> >> >             >         >
> >> >             >         > I've ran ipa-server-certinstall for the
> new
> >> >             cert/key as well,
> >> >             >         and it fails saying its not trusted
> ("Peer's
> >> >             certificate issuer
> >> >             >         is not trusted [certutil: certificate is invalid:
> >> >             Peer's
> >> >             >         Certificate issuer is not recognized] Please run
> >> >             >         ipa-cacert-manage install and ipa-certupdate to
> >> >             install the CA
> >> >             >         certificate.... which, as reported above,
> can't
> >> >             complete.
> >> >             >         >
> >> >             >         > I'm at a total loss here... and really
> >> >             struggling being new to
> >> >             >         all this and trying my best to keep it afloat.
> >> Any
> >> >             help would be
> >> >             >         GREATLY appreciated!
> >> >             >
> >> >             >         Let's gather some information first.
> >> >             >
> >> >             >         What version of IPA is this, on what
> >> distribution?
> >> >             >
> >> >             >         IPA designates one server to be the "renewal
> >> >             master" which
> >> >             >         handles the
> >> >             >         renewals. The output of `ipa config-show` should
> >> >             tell you
> >> >             >         (depending on
> >> >             >         version). That's the server you want to work
> on.
> >> >             >
> >> >             >         How many servers in your topology and how many
> >> >             have a CA installed?
> >> >             >
> >> >             >         Does `getcert list` show a set of 8-10 tracked
> >> >             certificates?
> >> >             >         What are
> >> >             >         the states?
> >> >             >
> >> >             >         You mention ipa-server-certinstall. Are you using
> >> >             3rd party
> >> >             >         certificates
> >> >             >         in addition to IPA CA-issued certificates or was
> >> >             that just an
> >> >             >         attempt to
> >> >             >         get things working again?
> >> >             >
> >> >             >         rob
> >> >             >
> >> >
> >>
> >>

[Rob Crittenden]

Justen Long wrote:
>  One go back.. when I tried to run "ipa-certupdate" on the
> other hosts
>  (clients), it points to hiipa03 and fails for SSL still.. so, hoping
>  once I get the cert updated on THAT server, that all restores to its
>  okay state. 
You shouldn't need to go back for clients. Now that the server
certificates are valid they should just work. Unless you also changed
the CA chain in which case you're caught in a catch-22: you need to get
the updated chain from the server but you don't trust the chain of the
server.

rob

>  
>  On Thu, Apr 13, 2023 at 1:46 PM Justen Long <mr.justenlong(a)gmail.com
>  <mailto:[e-mail]>> wrote:
>  
>      Quick update and answers to your questions.
>  
>      We have two IPA servers, both masters, hiipa03 and hiipa04.
>  
>      hiipa04, I was able to set the time back, run the ipa-ca-cert-manage
>      and update the CA (minimally, but enough to accept the new cert)..
>      ipa-certupdate ran fine on it. Then, ran ipa-server-certinstall on
>      it, and it took. Website loads, can log into it, do some user
>      management stuff.. can't run "ipa-replica-manage list" as it kicks
>      an error for sslv3 handshake failure.. but was trying that to remove
>      hiipa03 and copy 04 to 03 somehow, maybe? 
>  
>      hiipa03 is still giving me grief. Set the time using timedatectl,
>      verified ntp is off and 'date' reports properly.  I had tried to
>      follow
>     
> this: https://computingforgeeks.com/reset-freeipa-admin-password-as-root-...
>      when it wasn't accepting the admin password.. it failed saying
>      object not found. So, I try 'kinit admin' again, with the new
>      password I tried.. says its expired. Type it in, type in a new
>      password.. and then it failed saying "kinit: Password change failed
>      while getting initial credentials"
>  
>      On Thu, Apr 13, 2023 at 1:40 PM Rob Crittenden <rcritten(a)redhat.com
>      <mailto:[e-mail]>> wrote:
>  
>          Justen Long wrote:
>          > I'm getting closer... it's not recognizing my admin password
>          for IPA, or
>          > for my personal account with admin rights now.. but no more
>          SSL errors..
>          > just can't run ipa-certupdate without the proper kerberos creds..
>  
>          By not recognizing your password I assume you mean kinit is
>          failing? Is
>          the KDC running? I assume 389-ds is running? All restarted after
>          time
>          became stable in the past?
>  
>          rob
>  
>          >
>          > On Thu, Apr 13, 2023 at 12:51 PM Justen Long
>          <mr.justenlong(a)gmail.com <mailto:[e-mail]>
>          > <mailto:[e-mail]
>          <mailto:[e-mail]>>> wrote:
>          >
>          >     Following up, I see the date command just changed it
>          momentarily...
>          >     using timedatectl and will report back.
>          >
>          >     On Thu, Apr 13, 2023 at 12:31 PM Justen Long
>          >     &lt;mr.justenlong(a)gmail.com <mailto:[e-mail]>
>          <mailto:[e-mail]
>          <mailto:[e-mail]>>> wrote:
>          >
>          >         Rob,
>          >
>          >         I entered 'date --date="7 April 2023", verified it
>          updated the
>          >         system time appropriately. Restarted dirsrv, ipa-custodia,
>          >         ipa-otpd, httpd.. krb5kdc and kadmin failed. Still,
>          tried to
>          >         send ipa cert-update, and it popped the same SSL
>          Certificate
>          >         Verify Failed error.
>          >
>          >         On Thu, Apr 13, 2023 at 11:32 AM Rob Crittenden
>          >         &lt;rcritten(a)redhat.com <mailto:[e-mail]>
>          <mailto:[e-mail] <mailto:[e-mail]>>> wrote:
>          >
>          >             Justen Long wrote:
>          >             > Additionally, is there any way to force the CA
>          cert update
>          >             to be
>          >             > recognized? When I run it to update the CA chain,
>          >             everything is
>          >             > verified.. but /etc/ipa/ca.crt didn't reflect
>          the change..
>          >             so I manually
>          >             > populated it by copying over the guts of the CA
>          bundle to the
>          >             > /etc/ipa/ca.crt before trying to install the new
>          server
>          >             cert and it
>          >             > still doesn't recognize it as trusted although
>          the issuer
>          >             is the same
>          >             > and within the CA bundle.
>          >
>          >             This is going to sound weird, but I'd just go back
>          in time
>          >             to April 10,
>          >             restart all services but ntp (which will reset the
>          time) and
>          >             then the
>          >             commands should work. Once the certs are updated and
>          >             working, return to
>          >             present time.
>          >
>          >             rob
>          >
>          >             >
>          >             > On Thu, Apr 13, 2023 at 6:20 AM Justen Long
>          >             &lt;mr.justenlong(a)gmail.com
>          <mailto:[e-mail]> <mailto:[e-mail]
>          <mailto:[e-mail]>>
>          >             > <mailto:[e-mail]
>          <mailto:[e-mail]>
>          >             <mailto:[e-mail]
>          <mailto:[e-mail]>>>> wrote:
>          >             >
>          >             >     Rob,
>          >             >
>          >             >     Apologies for the delay in response. Once
>          I'm home, I
>          >             don't have
>          >             >     access to the information readily available
>          to respond
>          >             with. Here is
>          >             >     the information you requested:
>          >             >
>          >             >     The version of IPA we are using is 4.6.8, rpm
>          >             specifically for us is
>          >             >     ipa-server-4.6.8-5.el7.centos.12.x86_64 and
>          we are
>          >             using CentOS 7.9
>          >             >     currently with plans to move to RHEL9 within
>          the next
>          >             year or so.
>          >             >
>          >             >     Unfortunately, 'ipa config-show' doesn't
>          work. It
>          >             populates the same
>          >             >     error stating "ipa: ERROR: cannot connect to
>          >             >     'https://ipaServer/ipa/json': [SSL:
>          >             CERTIFICATE_VERIFY_FAILED]
>          >             >     certificate verify failed (_ssl.c:618).
>          >
>          >             The smack heard around the world was my head
>          hitting my
>          >             desk. Of course
>          >             this command failed.
>          >
>          >             >
>          >             >     We have ~50 hosts connected via IPA. We have
>          two IPA
>          >             servers, one as
>          >             >     a replica of the other.
>          >             >
>          >             >     'getcert list' only shows 1 certificate.
>          It's state is
>          >             "MONITORING"
>          >             >     and seems related to kerberos.
>          >             >
>          >             >     As far as I know, we don't use IPA CA-issued
>          >             certificates. I recall
>          >             >     seeing errors yesterday stating CA wasn't
>          enabled on
>          >             our servers. We
>          >             >     have always used 3rd party CAs to my knowledge.
>          >             >
>          >             >     -justen
>          >             >
>          >             >     On Wed, Apr 12, 2023 at 2:42 PM Rob Crittenden
>          >             &lt;rcritten(a)redhat.com <mailto:[e-mail]>
>          <mailto:[e-mail] <mailto:[e-mail]>>
>          >             >     <mailto:[e-mail]
>          <mailto:[e-mail]>
>          >             <mailto:[e-mail]
>          <mailto:[e-mail]>>>> wrote:
>          >             >
>          >             >         Justen Long via FreeIPA-users wrote:
>          >             >         > Thanks in advance for your replies..
>          I've spent
>          >             7 hours
>          >             >         looking through posts here and trying
>          >             everything... I'm stuck.
>          >             >         >
>          >             >         > Background: I am a System
>          Administrator in a closed,
>          >             >         classified environment. Unfortunately, I
>          cannot
>          >             post logging
>          >             >         here, but I can refer to them as needed.
>          >             >         >
>          >             >         > I inherited this system from someone who
>          >             departed the program
>          >             >         a year or so ago. Fast forward to today, the
>          >             server certs
>          >             >         expired yesterday. Admittedly, I'm
>          unfamiliar (or
>          >             was) with the
>          >             >         certificate update process for IPA
>          servers. On a
>          >             typical server,
>          >             >         we replace the old cert and restart the
>          httpd
>          >             services; however,
>          >             >         I realize this cannot work with IPA
>          servers now.
>          >             >         >
>          >             >         > Additionally to all of this, the CA chain
>          >             updated 6 months ago.
>          >             >         >
>          >             >         > I ran ipa-cacert-manage to update the
>          CA chain.
>          >             When trying to
>          >             >         run ipa-certupdate, I received errors for an
>          >             invalid server
>          >             >         certificate (it expired on 11 April
>          2023). It
>          >             simply won't
>          >             >         connect to the web server. HTTPD failed
>          as well,
>          >             so I had to add
>          >             >         "NSSEnforceValidCerts off" to the
>          nss.conf file
>          >             for HTTPD to
>          >             >         start. Still, no dice.
>          >             >         >
>          >             >         > I've ran ipa-server-certinstall for
>          the new
>          >             cert/key as well,
>          >             >         and it fails saying its not trusted
> ("Peer's
>          >             certificate issuer
>          >             >         is not trusted [certutil: certificate is
>          invalid:
>          >             Peer's
>          >             >         Certificate issuer is not recognized]
>          Please run
>          >             >         ipa-cacert-manage install and
>          ipa-certupdate to
>          >             install the CA
>          >             >         certificate.... which, as reported
>          above, can't
>          >             complete.
>          >             >         >
>          >             >         > I'm at a total loss here... and really
>          >             struggling being new to
>          >             >         all this and trying my best to keep it
>          afloat. Any
>          >             help would be
>          >             >         GREATLY appreciated!
>          >             >
>          >             >         Let's gather some information first.
>          >             >
>          >             >         What version of IPA is this, on what
>          distribution?
>          >             >
>          >             >         IPA designates one server to be the "renewal
>          >             master" which
>          >             >         handles the
>          >             >         renewals. The output of `ipa
>          config-show` should
>          >             tell you
>          >             >         (depending on
>          >             >         version). That's the server you want to
>          work on.
>          >             >
>          >             >         How many servers in your topology and
>          how many
>          >             have a CA installed?
>          >             >
>          >             >         Does `getcert list` show a set of 8-10
>          tracked
>          >             certificates?
>          >             >         What are
>          >             >         the states?
>          >             >
>          >             >         You mention ipa-server-certinstall. Are
>          you using
>          >             3rd party
>          >             >         certificates
>          >             >         in addition to IPA CA-issued
>          certificates or was
>          >             that just an
>          >             >         attempt to
>          >             >         get things working again?
>          >             >
>          >             >         rob
>          >             >
>          >
>

[Justen Long]

Rob,

FreeIPA documentation stated that all clients must receive the
ipa-updatecert command before installing the new server cert.. on the
hiipa04 server I installed the cert in a hurry for validity sake. Does that
mess with anything?

Note: hiipa03 seems to be primary but they’re both masters. I can’t get the
ipa-server-certinstall to take on this server stating it’s missing the
whole chain.. but it worked just fine on hiipa04.. so that’s why I was
trying to “kinit admin” on 03, to run ipa-updatecert or the
ipa-cacert-manage there if it didn’t take it from 04, which I’m assuming it
didn’t.

On Thu, Apr 13, 2023 at 2:41 PM Rob Crittenden <rcritten(a)redhat.com&gt; wrote:

>  Justen Long wrote:
>  > One go back.. when I tried to run "ipa-certupdate" on the other hosts
>  > (clients), it points to hiipa03 and fails for SSL still.. so, hoping
>  > once I get the cert updated on THAT server, that all restores to its
>  > okay state.
>
>  You shouldn't need to go back for clients. Now that the server
>  certificates are valid they should just work. Unless you also changed
>  the CA chain in which case you're caught in a catch-22: you need to get
>  the updated chain from the server but you don't trust the chain of the
>  server.
>
>  rob
>
>  >
>  > On Thu, Apr 13, 2023 at 1:46 PM Justen Long <mr.justenlong(a)gmail.com
>  > <mailto:[e-mail]>> wrote:
>  >
>  >     Quick update and answers to your questions.
>  >
>  >     We have two IPA servers, both masters, hiipa03 and hiipa04.
>  >
>  >     hiipa04, I was able to set the time back, run the ipa-ca-cert-manage
>  >     and update the CA (minimally, but enough to accept the new cert)..
>  >     ipa-certupdate ran fine on it. Then, ran ipa-server-certinstall on
>  >     it, and it took. Website loads, can log into it, do some user
>  >     management stuff.. can't run "ipa-replica-manage list" as it
> kicks
>  >     an error for sslv3 handshake failure.. but was trying that to remove
>  >     hiipa03 and copy 04 to 03 somehow, maybe?
>  >
>  >     hiipa03 is still giving me grief. Set the time using timedatectl,
>  >     verified ntp is off and 'date' reports properly.  I had tried to
>  >     follow
>  >     this:
>  https://computingforgeeks.com/reset-freeipa-admin-password-as-root-user-o...
>  >     when it wasn't accepting the admin password.. it failed saying
>  >     object not found. So, I try 'kinit admin' again, with the new
>  >     password I tried.. says its expired. Type it in, type in a new
>  >     password.. and then it failed saying "kinit: Password change failed
>  >     while getting initial credentials"
>  >
>  >     On Thu, Apr 13, 2023 at 1:40 PM Rob Crittenden <rcritten(a)redhat.com
>  >     <mailto:[e-mail]>> wrote:
>  >
>  >         Justen Long wrote:
>  >         > I'm getting closer... it's not recognizing my admin password
>  >         for IPA, or
>  >         > for my personal account with admin rights now.. but no more
>  >         SSL errors..
>  >         > just can't run ipa-certupdate without the proper kerberos
>  creds..
>  >
>  >         By not recognizing your password I assume you mean kinit is
>  >         failing? Is
>  >         the KDC running? I assume 389-ds is running? All restarted after
>  >         time
>  >         became stable in the past?
>  >
>  >         rob
>  >
>  >         >
>  >         > On Thu, Apr 13, 2023 at 12:51 PM Justen Long
>  >         <mr.justenlong(a)gmail.com <mailto:[e-mail]>
>  >         > <mailto:[e-mail]
>  >         <mailto:[e-mail]>>> wrote:
>  >         >
>  >         >     Following up, I see the date command just changed it
>  >         momentarily...
>  >         >     using timedatectl and will report back.
>  >         >
>  >         >     On Thu, Apr 13, 2023 at 12:31 PM Justen Long
>  >         >     <mr.justenlong(a)gmail.com <mailto:[e-mail]>
>  >         <mailto:[e-mail]
>  >         <mailto:[e-mail]>>> wrote:
>  >         >
>  >         >         Rob,
>  >         >
>  >         >         I entered 'date --date="7 April 2023", verified
> it
>  >         updated the
>  >         >         system time appropriately. Restarted dirsrv,
>  ipa-custodia,
>  >         >         ipa-otpd, httpd.. krb5kdc and kadmin failed. Still,
>  >         tried to
>  >         >         send ipa cert-update, and it popped the same SSL
>  >         Certificate
>  >         >         Verify Failed error.
>  >         >
>  >         >         On Thu, Apr 13, 2023 at 11:32 AM Rob Crittenden
>  >         >         <rcritten(a)redhat.com <mailto:[e-mail]>
>  >         <mailto:[e-mail] <mailto:[e-mail]>>>
>  wrote:
>  >         >
>  >         >             Justen Long wrote:
>  >         >             > Additionally, is there any way to force the CA
>  >         cert update
>  >         >             to be
>  >         >             > recognized? When I run it to update the CA chain,
>  >         >             everything is
>  >         >             > verified.. but /etc/ipa/ca.crt didn't reflect
>  >         the change..
>  >         >             so I manually
>  >         >             > populated it by copying over the guts of the CA
>  >         bundle to the
>  >         >             > /etc/ipa/ca.crt before trying to install the new
>  >         server
>  >         >             cert and it
>  >         >             > still doesn't recognize it as trusted although
>  >         the issuer
>  >         >             is the same
>  >         >             > and within the CA bundle.
>  >         >
>  >         >             This is going to sound weird, but I'd just go back
>  >         in time
>  >         >             to April 10,
>  >         >             restart all services but ntp (which will reset the
>  >         time) and
>  >         >             then the
>  >         >             commands should work. Once the certs are updated
>  and
>  >         >             working, return to
>  >         >             present time.
>  >         >
>  >         >             rob
>  >         >
>  >         >             >
>  >         >             > On Thu, Apr 13, 2023 at 6:20 AM Justen Long
>  >         >             <mr.justenlong(a)gmail.com
>  >         <mailto:[e-mail]> <mailto:[e-mail]
>  >         <mailto:[e-mail]>>
>  >         >             > <mailto:[e-mail]
>  >         <mailto:[e-mail]>
>  >         >             <mailto:[e-mail]
>  >         <mailto:[e-mail]>>>> wrote:
>  >         >             >
>  >         >             >     Rob,
>  >         >             >
>  >         >             >     Apologies for the delay in response. Once
>  >         I'm home, I
>  >         >             don't have
>  >         >             >     access to the information readily available
>  >         to respond
>  >         >             with. Here is
>  >         >             >     the information you requested:
>  >         >             >
>  >         >             >     The version of IPA we are using is 4.6.8, rpm
>  >         >             specifically for us is
>  >         >             >     ipa-server-4.6.8-5.el7.centos.12.x86_64 and
>  >         we are
>  >         >             using CentOS 7.9
>  >         >             >     currently with plans to move to RHEL9 within
>  >         the next
>  >         >             year or so.
>  >         >             >
>  >         >             >     Unfortunately, 'ipa config-show'
> doesn't
>  >         work. It
>  >         >             populates the same
>  >         >             >     error stating "ipa: ERROR: cannot connect to
>  >         >             >     'https://ipaServer/ipa/json': [SSL:
>  >         >             CERTIFICATE_VERIFY_FAILED]
>  >         >             >     certificate verify failed (_ssl.c:618).
>  >         >
>  >         >             The smack heard around the world was my head
>  >         hitting my
>  >         >             desk. Of course
>  >         >             this command failed.
>  >         >
>  >         >             >
>  >         >             >     We have ~50 hosts connected via IPA. We have
>  >         two IPA
>  >         >             servers, one as
>  >         >             >     a replica of the other.
>  >         >             >
>  >         >             >     'getcert list' only shows 1 certificate.
>  >         It's state is
>  >         >             "MONITORING"
>  >         >             >     and seems related to kerberos.
>  >         >             >
>  >         >             >     As far as I know, we don't use IPA CA-issued
>  >         >             certificates. I recall
>  >         >             >     seeing errors yesterday stating CA wasn't
>  >         enabled on
>  >         >             our servers. We
>  >         >             >     have always used 3rd party CAs to my
>  knowledge.
>  >         >             >
>  >         >             >     -justen
>  >         >             >
>  >         >             >     On Wed, Apr 12, 2023 at 2:42 PM Rob
>  Crittenden
>  >         >             <rcritten(a)redhat.com <mailto:[e-mail]>
>  >         <mailto:[e-mail] <mailto:[e-mail]>>
>  >         >             >     <mailto:[e-mail]
>  >         <mailto:[e-mail]>
>  >         >             <mailto:[e-mail]
>  >         <mailto:[e-mail]>>>> wrote:
>  >         >             >
>  >         >             >         Justen Long via FreeIPA-users wrote:
>  >         >             >         > Thanks in advance for your replies..
>  >         I've spent
>  >         >             7 hours
>  >         >             >         looking through posts here and trying
>  >         >             everything... I'm stuck.
>  >         >             >         >
>  >         >             >         > Background: I am a System
>  >         Administrator in a closed,
>  >         >             >         classified environment. Unfortunately, I
>  >         cannot
>  >         >             post logging
>  >         >             >         here, but I can refer to them as needed.
>  >         >             >         >
>  >         >             >         > I inherited this system from someone
>  who
>  >         >             departed the program
>  >         >             >         a year or so ago. Fast forward to today,
>  the
>  >         >             server certs
>  >         >             >         expired yesterday. Admittedly, I'm
>  >         unfamiliar (or
>  >         >             was) with the
>  >         >             >         certificate update process for IPA
>  >         servers. On a
>  >         >             typical server,
>  >         >             >         we replace the old cert and restart the
>  >         httpd
>  >         >             services; however,
>  >         >             >         I realize this cannot work with IPA
>  >         servers now.
>  >         >             >         >
>  >         >             >         > Additionally to all of this, the CA
>  chain
>  >         >             updated 6 months ago.
>  >         >             >         >
>  >         >             >         > I ran ipa-cacert-manage to update the
>  >         CA chain.
>  >         >             When trying to
>  >         >             >         run ipa-certupdate, I received errors
>  for an
>  >         >             invalid server
>  >         >             >         certificate (it expired on 11 April
>  >         2023). It
>  >         >             simply won't
>  >         >             >         connect to the web server. HTTPD failed
>  >         as well,
>  >         >             so I had to add
>  >         >             >         "NSSEnforceValidCerts off" to the
>  >         nss.conf file
>  >         >             for HTTPD to
>  >         >             >         start. Still, no dice.
>  >         >             >         >
>  >         >             >         > I've ran ipa-server-certinstall for
>  >         the new
>  >         >             cert/key as well,
>  >         >             >         and it fails saying its not trusted
>  ("Peer's
>  >         >             certificate issuer
>  >         >             >         is not trusted [certutil: certificate is
>  >         invalid:
>  >         >             Peer's
>  >         >             >         Certificate issuer is not recognized]
>  >         Please run
>  >         >             >         ipa-cacert-manage install and
>  >         ipa-certupdate to
>  >         >             install the CA
>  >         >             >         certificate.... which, as reported
>  >         above, can't
>  >         >             complete.
>  >         >             >         >
>  >         >             >         > I'm at a total loss here... and
> really
>  >         >             struggling being new to
>  >         >             >         all this and trying my best to keep it
>  >         afloat. Any
>  >         >             help would be
>  >         >             >         GREATLY appreciated!
>  >         >             >
>  >         >             >         Let's gather some information first.
>  >         >             >
>  >         >             >         What version of IPA is this, on what
>  >         distribution?
>  >         >             >
>  >         >             >         IPA designates one server to be the
>  "renewal
>  >         >             master" which
>  >         >             >         handles the
>  >         >             >         renewals. The output of `ipa
>  >         config-show` should
>  >         >             tell you
>  >         >             >         (depending on
>  >         >             >         version). That's the server you want to
>  >         work on.
>  >         >             >
>  >         >             >         How many servers in your topology and
>  >         how many
>  >         >             have a CA installed?
>  >         >             >
>  >         >             >         Does `getcert list` show a set of 8-10
>  >         tracked
>  >         >             certificates?
>  >         >             >         What are
>  >         >             >         the states?
>  >         >             >
>  >         >             >         You mention ipa-server-certinstall. Are
>  >         you using
>  >         >             3rd party
>  >         >             >         certificates
>  >         >             >         in addition to IPA CA-issued
>  >         certificates or was
>  >         >             that just an
>  >         >             >         attempt to
>  >         >             >         get things working again?
>  >         >             >
>  >         >             >         rob
>  >         >             >
>  >         >
>  >
>
>

[Justen Long]

Additional random question.. if I produced a new certificate now, would I
be able to update with it, or no because it would be dated “in the future?”

On Thu, Apr 13, 2023 at 2:48 PM Justen Long <mr.justenlong(a)gmail.com&gt; wrote:

>  Rob,
>
>  FreeIPA documentation stated that all clients must receive the
>  ipa-updatecert command before installing the new server cert.. on the
>  hiipa04 server I installed the cert in a hurry for validity sake. Does that
>  mess with anything?
>
>  Note: hiipa03 seems to be primary but they’re both masters. I can’t get
>  the ipa-server-certinstall to take on this server stating it’s missing the
>  whole chain.. but it worked just fine on hiipa04.. so that’s why I was
>  trying to “kinit admin” on 03, to run ipa-updatecert or the
>  ipa-cacert-manage there if it didn’t take it from 04, which I’m assuming it
>  didn’t.
>
>  On Thu, Apr 13, 2023 at 2:41 PM Rob Crittenden <rcritten(a)redhat.com&gt;
>  wrote:
>
> > Justen Long wrote:
> > > One go back.. when I tried to run "ipa-certupdate" on the other hosts
> > > (clients), it points to hiipa03 and fails for SSL still.. so, hoping
> > > once I get the cert updated on THAT server, that all restores to its
> > > okay state.
> >
> > You shouldn't need to go back for clients. Now that the server
> > certificates are valid they should just work. Unless you also changed
> > the CA chain in which case you're caught in a catch-22: you need to get
> > the updated chain from the server but you don't trust the chain of the
> > server.
> >
> > rob
> >
> > >
> > > On Thu, Apr 13, 2023 at 1:46 PM Justen Long <mr.justenlong(a)gmail.com
> > > <mailto:[e-mail]>> wrote:
> > >
> > >     Quick update and answers to your questions.
> > >
> > >     We have two IPA servers, both masters, hiipa03 and hiipa04.
> > >
> > >     hiipa04, I was able to set the time back, run the ipa-ca-cert-manage
> > >     and update the CA (minimally, but enough to accept the new cert)..
> > >     ipa-certupdate ran fine on it. Then, ran ipa-server-certinstall on
> > >     it, and it took. Website loads, can log into it, do some user
> > >     management stuff.. can't run "ipa-replica-manage list" as it
> kicks
> > >     an error for sslv3 handshake failure.. but was trying that to remove
> > >     hiipa03 and copy 04 to 03 somehow, maybe?
> > >
> > >     hiipa03 is still giving me grief. Set the time using timedatectl,
> > >     verified ntp is off and 'date' reports properly.  I had tried to
> > >     follow
> > >     this:
> > https://computingforgeeks.com/reset-freeipa-admin-password-as-root-user-o...
> > >     when it wasn't accepting the admin password.. it failed saying
> > >     object not found. So, I try 'kinit admin' again, with the new
> > >     password I tried.. says its expired. Type it in, type in a new
> > >     password.. and then it failed saying "kinit: Password change failed
> > >     while getting initial credentials"
> > >
> > >     On Thu, Apr 13, 2023 at 1:40 PM Rob Crittenden <rcritten(a)redhat.com
> > >     <mailto:[e-mail]>> wrote:
> > >
> > >         Justen Long wrote:
> > >         > I'm getting closer... it's not recognizing my admin
> password
> > >         for IPA, or
> > >         > for my personal account with admin rights now.. but no more
> > >         SSL errors..
> > >         > just can't run ipa-certupdate without the proper kerberos
> > creds..
> > >
> > >         By not recognizing your password I assume you mean kinit is
> > >         failing? Is
> > >         the KDC running? I assume 389-ds is running? All restarted after
> > >         time
> > >         became stable in the past?
> > >
> > >         rob
> > >
> > >         >
> > >         > On Thu, Apr 13, 2023 at 12:51 PM Justen Long
> > >         <mr.justenlong(a)gmail.com <mailto:[e-mail]>
> > >         > <mailto:[e-mail]
> > >         <mailto:[e-mail]>>> wrote:
> > >         >
> > >         >     Following up, I see the date command just changed it
> > >         momentarily...
> > >         >     using timedatectl and will report back.
> > >         >
> > >         >     On Thu, Apr 13, 2023 at 12:31 PM Justen Long
> > >         >     <mr.justenlong(a)gmail.com
> <mailto:[e-mail]>
> > >         <mailto:[e-mail]
> > >         <mailto:[e-mail]>>> wrote:
> > >         >
> > >         >         Rob,
> > >         >
> > >         >         I entered 'date --date="7 April 2023",
> verified it
> > >         updated the
> > >         >         system time appropriately. Restarted dirsrv,
> > ipa-custodia,
> > >         >         ipa-otpd, httpd.. krb5kdc and kadmin failed. Still,
> > >         tried to
> > >         >         send ipa cert-update, and it popped the same SSL
> > >         Certificate
> > >         >         Verify Failed error.
> > >         >
> > >         >         On Thu, Apr 13, 2023 at 11:32 AM Rob Crittenden
> > >         >         <rcritten(a)redhat.com <mailto:[e-mail]>
> > >         <mailto:[e-mail]
> <mailto:[e-mail]>>>
> > wrote:
> > >         >
> > >         >             Justen Long wrote:
> > >         >             > Additionally, is there any way to force the CA
> > >         cert update
> > >         >             to be
> > >         >             > recognized? When I run it to update the CA
> > chain,
> > >         >             everything is
> > >         >             > verified.. but /etc/ipa/ca.crt didn't reflect
> > >         the change..
> > >         >             so I manually
> > >         >             > populated it by copying over the guts of the CA
> > >         bundle to the
> > >         >             > /etc/ipa/ca.crt before trying to install the new
> > >         server
> > >         >             cert and it
> > >         >             > still doesn't recognize it as trusted
> although
> > >         the issuer
> > >         >             is the same
> > >         >             > and within the CA bundle.
> > >         >
> > >         >             This is going to sound weird, but I'd just go back
> > >         in time
> > >         >             to April 10,
> > >         >             restart all services but ntp (which will reset the
> > >         time) and
> > >         >             then the
> > >         >             commands should work. Once the certs are updated
> > and
> > >         >             working, return to
> > >         >             present time.
> > >         >
> > >         >             rob
> > >         >
> > >         >             >
> > >         >             > On Thu, Apr 13, 2023 at 6:20 AM Justen Long
> > >         >             <mr.justenlong(a)gmail.com
> > >         <mailto:[e-mail]> <mailto:
> > mr.justenlong(a)gmail.com
> > >         <mailto:[e-mail]>>
> > >         >             > <mailto:[e-mail]
> > >         <mailto:[e-mail]>
> > >         >             <mailto:[e-mail]
> > >         <mailto:[e-mail]>>>> wrote:
> > >         >             >
> > >         >             >     Rob,
> > >         >             >
> > >         >             >     Apologies for the delay in response. Once
> > >         I'm home, I
> > >         >             don't have
> > >         >             >     access to the information readily available
> > >         to respond
> > >         >             with. Here is
> > >         >             >     the information you requested:
> > >         >             >
> > >         >             >     The version of IPA we are using is 4.6.8,
> > rpm
> > >         >             specifically for us is
> > >         >             >     ipa-server-4.6.8-5.el7.centos.12.x86_64 and
> > >         we are
> > >         >             using CentOS 7.9
> > >         >             >     currently with plans to move to RHEL9 within
> > >         the next
> > >         >             year or so.
> > >         >             >
> > >         >             >     Unfortunately, 'ipa config-show'
> doesn't
> > >         work. It
> > >         >             populates the same
> > >         >             >     error stating "ipa: ERROR: cannot connect
> to
> > >         >             >     'https://ipaServer/ipa/json': [SSL:
> > >         >             CERTIFICATE_VERIFY_FAILED]
> > >         >             >     certificate verify failed (_ssl.c:618).
> > >         >
> > >         >             The smack heard around the world was my head
> > >         hitting my
> > >         >             desk. Of course
> > >         >             this command failed.
> > >         >
> > >         >             >
> > >         >             >     We have ~50 hosts connected via IPA. We have
> > >         two IPA
> > >         >             servers, one as
> > >         >             >     a replica of the other.
> > >         >             >
> > >         >             >     'getcert list' only shows 1
> certificate.
> > >         It's state is
> > >         >             "MONITORING"
> > >         >             >     and seems related to kerberos.
> > >         >             >
> > >         >             >     As far as I know, we don't use IPA
> CA-issued
> > >         >             certificates. I recall
> > >         >             >     seeing errors yesterday stating CA wasn't
> > >         enabled on
> > >         >             our servers. We
> > >         >             >     have always used 3rd party CAs to my
> > knowledge.
> > >         >             >
> > >         >             >     -justen
> > >         >             >
> > >         >             >     On Wed, Apr 12, 2023 at 2:42 PM Rob
> > Crittenden
> > >         >             <rcritten(a)redhat.com
> <mailto:[e-mail]>
> > >         <mailto:[e-mail] <mailto:[e-mail]>>
> > >         >             >     <mailto:[e-mail]
> > >         <mailto:[e-mail]>
> > >         >             <mailto:[e-mail]
> > >         <mailto:[e-mail]>>>> wrote:
> > >         >             >
> > >         >             >         Justen Long via FreeIPA-users wrote:
> > >         >             >         > Thanks in advance for your replies..
> > >         I've spent
> > >         >             7 hours
> > >         >             >         looking through posts here and trying
> > >         >             everything... I'm stuck.
> > >         >             >         >
> > >         >             >         > Background: I am a System
> > >         Administrator in a closed,
> > >         >             >         classified environment. Unfortunately, I
> > >         cannot
> > >         >             post logging
> > >         >             >         here, but I can refer to them as needed.
> > >         >             >         >
> > >         >             >         > I inherited this system from someone
> > who
> > >         >             departed the program
> > >         >             >         a year or so ago. Fast forward to
> > today, the
> > >         >             server certs
> > >         >             >         expired yesterday. Admittedly, I'm
> > >         unfamiliar (or
> > >         >             was) with the
> > >         >             >         certificate update process for IPA
> > >         servers. On a
> > >         >             typical server,
> > >         >             >         we replace the old cert and restart the
> > >         httpd
> > >         >             services; however,
> > >         >             >         I realize this cannot work with IPA
> > >         servers now.
> > >         >             >         >
> > >         >             >         > Additionally to all of this, the CA
> > chain
> > >         >             updated 6 months ago.
> > >         >             >         >
> > >         >             >         > I ran ipa-cacert-manage to update
> the
> > >         CA chain.
> > >         >             When trying to
> > >         >             >         run ipa-certupdate, I received errors
> > for an
> > >         >             invalid server
> > >         >             >         certificate (it expired on 11 April
> > >         2023). It
> > >         >             simply won't
> > >         >             >         connect to the web server. HTTPD failed
> > >         as well,
> > >         >             so I had to add
> > >         >             >         "NSSEnforceValidCerts off" to
> the
> > >         nss.conf file
> > >         >             for HTTPD to
> > >         >             >         start. Still, no dice.
> > >         >             >         >
> > >         >             >         > I've ran ipa-server-certinstall
> for
> > >         the new
> > >         >             cert/key as well,
> > >         >             >         and it fails saying its not trusted
> > ("Peer's
> > >         >             certificate issuer
> > >         >             >         is not trusted [certutil: certificate is
> > >         invalid:
> > >         >             Peer's
> > >         >             >         Certificate issuer is not recognized]
> > >         Please run
> > >         >             >         ipa-cacert-manage install and
> > >         ipa-certupdate to
> > >         >             install the CA
> > >         >             >         certificate.... which, as reported
> > >         above, can't
> > >         >             complete.
> > >         >             >         >
> > >         >             >         > I'm at a total loss here... and
> really
> > >         >             struggling being new to
> > >         >             >         all this and trying my best to keep it
> > >         afloat. Any
> > >         >             help would be
> > >         >             >         GREATLY appreciated!
> > >         >             >
> > >         >             >         Let's gather some information first.
> > >         >             >
> > >         >             >         What version of IPA is this, on what
> > >         distribution?
> > >         >             >
> > >         >             >         IPA designates one server to be the
> > "renewal
> > >         >             master" which
> > >         >             >         handles the
> > >         >             >         renewals. The output of `ipa
> > >         config-show` should
> > >         >             tell you
> > >         >             >         (depending on
> > >         >             >         version). That's the server you want
> to
> > >         work on.
> > >         >             >
> > >         >             >         How many servers in your topology and
> > >         how many
> > >         >             have a CA installed?
> > >         >             >
> > >         >             >         Does `getcert list` show a set of 8-10
> > >         tracked
> > >         >             certificates?
> > >         >             >         What are
> > >         >             >         the states?
> > >         >             >
> > >         >             >         You mention ipa-server-certinstall. Are
> > >         you using
> > >         >             3rd party
> > >         >             >         certificates
> > >         >             >         in addition to IPA CA-issued
> > >         certificates or was
> > >         >             that just an
> > >         >             >         attempt to
> > >         >             >         get things working again?
> > >         >             >
> > >         >             >         rob
> > >         >             >
> > >         >
> > >
> >
> >

[Rob Crittenden]

Justen Long wrote:
>  Rob,
>  
>  FreeIPA documentation stated that all clients must receive the
>  ipa-updatecert command before installing the new server cert.. on the
>  hiipa04 server I installed the cert in a hurry for validity sake. Does
>  that mess with anything? 
ipa-certupdate distributes the CA chain. If the chain didn't change then
it should be fine.

>  
>  Note: hiipa03 seems to be primary but they’re both masters. I can’t get
>  the ipa-server-certinstall to take on this server stating it’s missing
>  the whole chain.. but it worked just fine on hiipa04.. so that’s why I
>  was trying to “kinit admin” on 03, to run ipa-updatecert or the
>  ipa-cacert-manage there if it didn’t take it from 04, which I’m assuming
>  it didn’t.  
You could try going back in time and use ipa-cacert-manage install to
update the chain and then run ipa-server-certinstall.

But like I said before, if you're changing your chain AND replacing the
certs at the same time none of your clients are going to know what to do.

The process is:

1. ipa-cacert-manage install <new chain>
2. ipa-certupdate on ALL enrolled machines
3. Replace the server certs

If you deviate then you're caught in the catch-22 I mentioned. Clients
won't trust the updated chain in order to download the updated chain.
The aren't a lot of great options out of that. Unenrolling and
re-enrolling the client is the best way.

rob
>  
>  On Thu, Apr 13, 2023 at 2:41 PM Rob Crittenden <rcritten(a)redhat.com
>  <mailto:[e-mail]>> wrote:
>  
>      Justen Long wrote:
>      > One go back.. when I tried to run "ipa-certupdate" on the other hosts
>      > (clients), it points to hiipa03 and fails for SSL still.. so, hoping
>      > once I get the cert updated on THAT server, that all restores to its
>      > okay state.
>  
>      You shouldn't need to go back for clients. Now that the server
>      certificates are valid they should just work. Unless you also changed
>      the CA chain in which case you're caught in a catch-22: you need to get
>      the updated chain from the server but you don't trust the chain of the
>      server.
>  
>      rob
>  
>      >
>      > On Thu, Apr 13, 2023 at 1:46 PM Justen Long
>      <mr.justenlong(a)gmail.com <mailto:[e-mail]>
>      > <mailto:[e-mail]
> <mailto:[e-mail]>>>
>      wrote:
>      >
>      >     Quick update and answers to your questions.
>      >
>      >     We have two IPA servers, both masters, hiipa03 and hiipa04.
>      >
>      >     hiipa04, I was able to set the time back, run the
>      ipa-ca-cert-manage
>      >     and update the CA (minimally, but enough to accept the new cert)..
>      >     ipa-certupdate ran fine on it. Then, ran ipa-server-certinstall on
>      >     it, and it took. Website loads, can log into it, do some user
>      >     management stuff.. can't run "ipa-replica-manage list" as it
> kicks
>      >     an error for sslv3 handshake failure.. but was trying that to
>      remove
>      >     hiipa03 and copy 04 to 03 somehow, maybe? 
>      >
>      >     hiipa03 is still giving me grief. Set the time using timedatectl,
>      >     verified ntp is off and 'date' reports properly.  I had tried to
>      >     follow
>      >   
>     
>  this: https://computingforgeeks.com/reset-freeipa-admin-password-as-root-user-on-linux/
>      >     when it wasn't accepting the admin password.. it failed saying
>      >     object not found. So, I try 'kinit admin' again, with the new
>      >     password I tried.. says its expired. Type it in, type in a new
>      >     password.. and then it failed saying "kinit: Password change
>      failed
>      >     while getting initial credentials"
>      >
>      >     On Thu, Apr 13, 2023 at 1:40 PM Rob Crittenden
>      <rcritten(a)redhat.com <mailto:[e-mail]>
>      >     <mailto:[e-mail] <mailto:[e-mail]>>>
> wrote:
>      >
>      >         Justen Long wrote:
>      >         > I'm getting closer... it's not recognizing my admin
> password
>      >         for IPA, or
>      >         > for my personal account with admin rights now.. but no more
>      >         SSL errors..
>      >         > just can't run ipa-certupdate without the proper
>      kerberos creds..
>      >
>      >         By not recognizing your password I assume you mean kinit is
>      >         failing? Is
>      >         the KDC running? I assume 389-ds is running? All restarted
>      after
>      >         time
>      >         became stable in the past?
>      >
>      >         rob
>      >
>      >         >
>      >         > On Thu, Apr 13, 2023 at 12:51 PM Justen Long
>      >         &lt;mr.justenlong(a)gmail.com <mailto:[e-mail]>
>      <mailto:[e-mail] <mailto:[e-mail]>>
>      >         > <mailto:[e-mail]
>      <mailto:[e-mail]>
>      >         <mailto:[e-mail]
>      <mailto:[e-mail]>>>> wrote:
>      >         >
>      >         >     Following up, I see the date command just changed it
>      >         momentarily...
>      >         >     using timedatectl and will report back.
>      >         >
>      >         >     On Thu, Apr 13, 2023 at 12:31 PM Justen Long
>      >         >     &lt;mr.justenlong(a)gmail.com
>      <mailto:[e-mail]> <mailto:[e-mail]
>      <mailto:[e-mail]>>
>      >         <mailto:[e-mail]
>      <mailto:[e-mail]>
>      >         <mailto:[e-mail]
>      <mailto:[e-mail]>>>> wrote:
>      >         >
>      >         >         Rob,
>      >         >
>      >         >         I entered 'date --date="7 April 2023",
> verified it
>      >         updated the
>      >         >         system time appropriately. Restarted dirsrv,
>      ipa-custodia,
>      >         >         ipa-otpd, httpd.. krb5kdc and kadmin failed. Still,
>      >         tried to
>      >         >         send ipa cert-update, and it popped the same SSL
>      >         Certificate
>      >         >         Verify Failed error.
>      >         >
>      >         >         On Thu, Apr 13, 2023 at 11:32 AM Rob Crittenden
>      >         >         &lt;rcritten(a)redhat.com
>      <mailto:[e-mail]> <mailto:[e-mail]
>      <mailto:[e-mail]>>
>      >         <mailto:[e-mail] <mailto:[e-mail]>
>      <mailto:[e-mail] <mailto:[e-mail]>>>> wrote:
>      >         >
>      >         >             Justen Long wrote:
>      >         >             > Additionally, is there any way to force the CA
>      >         cert update
>      >         >             to be
>      >         >             > recognized? When I run it to update the CA
>      chain,
>      >         >             everything is
>      >         >             > verified.. but /etc/ipa/ca.crt didn't reflect
>      >         the change..
>      >         >             so I manually
>      >         >             > populated it by copying over the guts of
>      the CA
>      >         bundle to the
>      >         >             > /etc/ipa/ca.crt before trying to install
>      the new
>      >         server
>      >         >             cert and it
>      >         >             > still doesn't recognize it as trusted
> although
>      >         the issuer
>      >         >             is the same
>      >         >             > and within the CA bundle.
>      >         >
>      >         >             This is going to sound weird, but I'd just
>      go back
>      >         in time
>      >         >             to April 10,
>      >         >             restart all services but ntp (which will
>      reset the
>      >         time) and
>      >         >             then the
>      >         >             commands should work. Once the certs are
>      updated and
>      >         >             working, return to
>      >         >             present time.
>      >         >
>      >         >             rob
>      >         >
>      >         >             >
>      >         >             > On Thu, Apr 13, 2023 at 6:20 AM Justen Long
>      >         >             &lt;mr.justenlong(a)gmail.com
>      <mailto:[e-mail]>
>      >         <mailto:[e-mail]
>      <mailto:[e-mail]>> <mailto:[e-mail]
>      <mailto:[e-mail]>
>      >         <mailto:[e-mail]
>      <mailto:[e-mail]>>>
>      >         >             > <mailto:[e-mail]
>      <mailto:[e-mail]>
>      >         <mailto:[e-mail]
>      <mailto:[e-mail]>>
>      >         >             <mailto:[e-mail]
>      <mailto:[e-mail]>
>      >         <mailto:[e-mail]
>      <mailto:[e-mail]>>>>> wrote:
>      >         >             >
>      >         >             >     Rob,
>      >         >             >
>      >         >             >     Apologies for the delay in response. Once
>      >         I'm home, I
>      >         >             don't have
>      >         >             >     access to the information readily
>      available
>      >         to respond
>      >         >             with. Here is
>      >         >             >     the information you requested:
>      >         >             >
>      >         >             >     The version of IPA we are using is
>      4.6.8, rpm
>      >         >             specifically for us is
>      >         >             >   
>       ipa-server-4.6.8-5.el7.centos.12.x86_64 and
>      >         we are
>      >         >             using CentOS 7.9
>      >         >             >     currently with plans to move to RHEL9
>      within
>      >         the next
>      >         >             year or so.
>      >         >             >
>      >         >             >     Unfortunately, 'ipa config-show'
> doesn't
>      >         work. It
>      >         >             populates the same
>      >         >             >     error stating "ipa: ERROR: cannot
>      connect to
>      >         >             >     'https://ipaServer/ipa/json': [SSL:
>      >         >             CERTIFICATE_VERIFY_FAILED]
>      >         >             >     certificate verify failed (_ssl.c:618).
>      >         >
>      >         >             The smack heard around the world was my head
>      >         hitting my
>      >         >             desk. Of course
>      >         >             this command failed.
>      >         >
>      >         >             >
>      >         >             >     We have ~50 hosts connected via IPA.
>      We have
>      >         two IPA
>      >         >             servers, one as
>      >         >             >     a replica of the other.
>      >         >             >
>      >         >             >     'getcert list' only shows 1
> certificate.
>      >         It's state is
>      >         >             "MONITORING"
>      >         >             >     and seems related to kerberos.
>      >         >             >
>      >         >             >     As far as I know, we don't use IPA
>      CA-issued
>      >         >             certificates. I recall
>      >         >             >     seeing errors yesterday stating CA wasn't
>      >         enabled on
>      >         >             our servers. We
>      >         >             >     have always used 3rd party CAs to my
>      knowledge.
>      >         >             >
>      >         >             >     -justen
>      >         >             >
>      >         >             >     On Wed, Apr 12, 2023 at 2:42 PM Rob
>      Crittenden
>      >         >             &lt;rcritten(a)redhat.com
>      <mailto:[e-mail]> <mailto:[e-mail]
>      <mailto:[e-mail]>>
>      >         <mailto:[e-mail] <mailto:[e-mail]>
>      <mailto:[e-mail] <mailto:[e-mail]>>>
>      >         >             >     <mailto:[e-mail]
>      <mailto:[e-mail]>
>      >         <mailto:[e-mail] <mailto:[e-mail]>>
>      >         >             <mailto:[e-mail]
>      <mailto:[e-mail]>
>      >         <mailto:[e-mail]
>      <mailto:[e-mail]>>>>> wrote:
>      >         >             >
>      >         >             >         Justen Long via FreeIPA-users wrote:
>      >         >             >         > Thanks in advance for your replies..
>      >         I've spent
>      >         >             7 hours
>      >         >             >         looking through posts here and trying
>      >         >             everything... I'm stuck.
>      >         >             >         >
>      >         >             >         > Background: I am a System
>      >         Administrator in a closed,
>      >         >             >         classified environment.
>      Unfortunately, I
>      >         cannot
>      >         >             post logging
>      >         >             >         here, but I can refer to them as
>      needed.
>      >         >             >         >
>      >         >             >         > I inherited this system from
>      someone who
>      >         >             departed the program
>      >         >             >         a year or so ago. Fast forward to
>      today, the
>      >         >             server certs
>      >         >             >         expired yesterday. Admittedly, I'm
>      >         unfamiliar (or
>      >         >             was) with the
>      >         >             >         certificate update process for IPA
>      >         servers. On a
>      >         >             typical server,
>      >         >             >         we replace the old cert and
>      restart the
>      >         httpd
>      >         >             services; however,
>      >         >             >         I realize this cannot work with IPA
>      >         servers now.
>      >         >             >         >
>      >         >             >         > Additionally to all of this, the
>      CA chain
>      >         >             updated 6 months ago.
>      >         >             >         >
>      >         >             >         > I ran ipa-cacert-manage to
>      update the
>      >         CA chain.
>      >         >             When trying to
>      >         >             >         run ipa-certupdate, I received
>      errors for an
>      >         >             invalid server
>      >         >             >         certificate (it expired on 11 April
>      >         2023). It
>      >         >             simply won't
>      >         >             >         connect to the web server. HTTPD
>      failed
>      >         as well,
>      >         >             so I had to add
>      >         >             >         "NSSEnforceValidCerts off" to
> the
>      >         nss.conf file
>      >         >             for HTTPD to
>      >         >             >         start. Still, no dice.
>      >         >             >         >
>      >         >             >         > I've ran ipa-server-certinstall
> for
>      >         the new
>      >         >             cert/key as well,
>      >         >             >         and it fails saying its not
>      trusted ("Peer's
>      >         >             certificate issuer
>      >         >             >         is not trusted [certutil:
>      certificate is
>      >         invalid:
>      >         >             Peer's
>      >         >             >         Certificate issuer is not recognized]
>      >         Please run
>      >         >             >         ipa-cacert-manage install and
>      >         ipa-certupdate to
>      >         >             install the CA
>      >         >             >         certificate.... which, as reported
>      >         above, can't
>      >         >             complete.
>      >         >             >         >
>      >         >             >         > I'm at a total loss here... and
>      really
>      >         >             struggling being new to
>      >         >             >         all this and trying my best to keep it
>      >         afloat. Any
>      >         >             help would be
>      >         >             >         GREATLY appreciated!
>      >         >             >
>      >         >             >         Let's gather some information first.
>      >         >             >
>      >         >             >         What version of IPA is this, on what
>      >         distribution?
>      >         >             >
>      >         >             >         IPA designates one server to be
>      the "renewal
>      >         >             master" which
>      >         >             >         handles the
>      >         >             >         renewals. The output of `ipa
>      >         config-show` should
>      >         >             tell you
>      >         >             >         (depending on
>      >         >             >         version). That's the server you
>      want to
>      >         work on.
>      >         >             >
>      >         >             >         How many servers in your topology and
>      >         how many
>      >         >             have a CA installed?
>      >         >             >
>      >         >             >         Does `getcert list` show a set of 8-10
>      >         tracked
>      >         >             certificates?
>      >         >             >         What are
>      >         >             >         the states?
>      >         >             >
>      >         >             >         You mention
>      ipa-server-certinstall. Are
>      >         you using
>      >         >             3rd party
>      >         >             >         certificates
>      >         >             >         in addition to IPA CA-issued
>      >         certificates or was
>      >         >             that just an
>      >         >             >         attempt to
>      >         >             >         get things working again?
>      >         >             >
>      >         >             >         rob
>      >         >             >
>      >         >
>      >
>

[Justen Long]

Rob,

Currently only hiipa03 and hiipa04 (these are the masters/IPA servers)
would utilize the ipa-server-certinstall commands. The other servers don't
use any HTTPD or anything like that, and have never had issues in the past
simply replacing the cert file in /etc/pki/tls/certs and letting it ride.

So, with regard to the catch-22...

hiipa03 isn't communicating appropriately or something, in my opinion.
When attempting to run "ipa-certupdate" on it, it wants the Kerberos
credentials (assuming 'kinit admin'). I run 'kinit admin" and enter the
admin password that works on hiipa04, and it doesn't like it. I attempted
to change that password using the long 'ldappasswd' command, but forgot to
put dc=hilab,dc=ipa, and instead had put dc=hilab.ipa. This results in a
failure, object not found.. (had a duh moment).. went to try the
'ldappasswd' command again, using dc=hilab,dc=ipa.. it then fails stating
"Result: Operations error (1) \\ Additional info: Failed to update
password".

So, I just, for s&g, run 'kinit admin' again. Prompts for password. Try the
one that failed because of the wrong dc input, and it takes it, stating its
expired. Try to change it, and get the same "Result: Operations error (1)
\\ Additional info:" Failed to update password."

On the majority of the clients, when I enter "ipa-certupdate" it points to
hiipa03, rather than hiipa04.. and fails because hiipa03 is still in
shambles. I think if I can get hiipa03 back to operational, then everything
else would work.

1) Is there some magical way to destroy the admin password (i.e. blank it
out) so it prompts for a new one, or to replicate the admin stuff from
hiipa04 to hiipa03 so that it all works?
2) If not, how can I possibly move forward? The only hold up is the admin
password at this point.

On Fri, Apr 14, 2023 at 7:11 AM Rob Crittenden <rcritten(a)redhat.com&gt; wrote:

>  Justen Long wrote:
>  > Rob,
>  >
>  > FreeIPA documentation stated that all clients must receive the
>  > ipa-updatecert command before installing the new server cert.. on the
>  > hiipa04 server I installed the cert in a hurry for validity sake. Does
>  > that mess with anything?
>
>  ipa-certupdate distributes the CA chain. If the chain didn't change then
>  it should be fine.
>
>  >
>  > Note: hiipa03 seems to be primary but they’re both masters. I can’t get
>  > the ipa-server-certinstall to take on this server stating it’s missing
>  > the whole chain.. but it worked just fine on hiipa04.. so that’s why I
>  > was trying to “kinit admin” on 03, to run ipa-updatecert or the
>  > ipa-cacert-manage there if it didn’t take it from 04, which I’m assuming
>  > it didn’t.
>
>  You could try going back in time and use ipa-cacert-manage install to
>  update the chain and then run ipa-server-certinstall.
>
>  But like I said before, if you're changing your chain AND replacing the
>  certs at the same time none of your clients are going to know what to do.
>
>  The process is:
>
>  1. ipa-cacert-manage install <new chain>
>  2. ipa-certupdate on ALL enrolled machines
>  3. Replace the server certs
>
>  If you deviate then you're caught in the catch-22 I mentioned. Clients
>  won't trust the updated chain in order to download the updated chain.
>  The aren't a lot of great options out of that. Unenrolling and
>  re-enrolling the client is the best way.
>
>  rob
>  >
>  > On Thu, Apr 13, 2023 at 2:41 PM Rob Crittenden <rcritten(a)redhat.com
>  > <mailto:[e-mail]>> wrote:
>  >
>  >     Justen Long wrote:
>  >     > One go back.. when I tried to run "ipa-certupdate" on the other
>  hosts
>  >     > (clients), it points to hiipa03 and fails for SSL still.. so,
>  hoping
>  >     > once I get the cert updated on THAT server, that all restores to
>  its
>  >     > okay state.
>  >
>  >     You shouldn't need to go back for clients. Now that the server
>  >     certificates are valid they should just work. Unless you also changed
>  >     the CA chain in which case you're caught in a catch-22: you need to
>  get
>  >     the updated chain from the server but you don't trust the chain of
>  the
>  >     server.
>  >
>  >     rob
>  >
>  >     >
>  >     > On Thu, Apr 13, 2023 at 1:46 PM Justen Long
>  >     <mr.justenlong(a)gmail.com <mailto:[e-mail]>
>  >     > <mailto:[e-mail]
> <mailto:[e-mail]>>>
>  >     wrote:
>  >     >
>  >     >     Quick update and answers to your questions.
>  >     >
>  >     >     We have two IPA servers, both masters, hiipa03 and hiipa04.
>  >     >
>  >     >     hiipa04, I was able to set the time back, run the
>  >     ipa-ca-cert-manage
>  >     >     and update the CA (minimally, but enough to accept the new
>  cert)..
>  >     >     ipa-certupdate ran fine on it. Then, ran
>  ipa-server-certinstall on
>  >     >     it, and it took. Website loads, can log into it, do some user
>  >     >     management stuff.. can't run "ipa-replica-manage list" as
> it
>  kicks
>  >     >     an error for sslv3 handshake failure.. but was trying that to
>  >     remove
>  >     >     hiipa03 and copy 04 to 03 somehow, maybe?
>  >     >
>  >     >     hiipa03 is still giving me grief. Set the time using
>  timedatectl,
>  >     >     verified ntp is off and 'date' reports properly.  I had tried
>  to
>  >     >     follow
>  >     >
>  >      this:
>  https://computingforgeeks.com/reset-freeipa-admin-password-as-root-user-o...
>  >     >     when it wasn't accepting the admin password.. it failed saying
>  >     >     object not found. So, I try 'kinit admin' again, with the new
>  >     >     password I tried.. says its expired. Type it in, type in a new
>  >     >     password.. and then it failed saying "kinit: Password change
>  >     failed
>  >     >     while getting initial credentials"
>  >     >
>  >     >     On Thu, Apr 13, 2023 at 1:40 PM Rob Crittenden
>  >     <rcritten(a)redhat.com <mailto:[e-mail]>
>  >     >     <mailto:[e-mail]
> <mailto:[e-mail]>>>
>  wrote:
>  >     >
>  >     >         Justen Long wrote:
>  >     >         > I'm getting closer... it's not recognizing my admin
>  password
>  >     >         for IPA, or
>  >     >         > for my personal account with admin rights now.. but no
>  more
>  >     >         SSL errors..
>  >     >         > just can't run ipa-certupdate without the proper
>  >     kerberos creds..
>  >     >
>  >     >         By not recognizing your password I assume you mean kinit is
>  >     >         failing? Is
>  >     >         the KDC running? I assume 389-ds is running? All restarted
>  >     after
>  >     >         time
>  >     >         became stable in the past?
>  >     >
>  >     >         rob
>  >     >
>  >     >         >
>  >     >         > On Thu, Apr 13, 2023 at 12:51 PM Justen Long
>  >     >         <mr.justenlong(a)gmail.com <mailto:[e-mail]>
>  >     <mailto:[e-mail] <mailto:[e-mail]>>
>  >     >         > <mailto:[e-mail]
>  >     <mailto:[e-mail]>
>  >     >         <mailto:[e-mail]
>  >     <mailto:[e-mail]>>>> wrote:
>  >     >         >
>  >     >         >     Following up, I see the date command just changed it
>  >     >         momentarily...
>  >     >         >     using timedatectl and will report back.
>  >     >         >
>  >     >         >     On Thu, Apr 13, 2023 at 12:31 PM Justen Long
>  >     >         >     <mr.justenlong(a)gmail.com
>  >     <mailto:[e-mail]> <mailto:[e-mail]
>  >     <mailto:[e-mail]>>
>  >     >         <mailto:[e-mail]
>  >     <mailto:[e-mail]>
>  >     >         <mailto:[e-mail]
>  >     <mailto:[e-mail]>>>> wrote:
>  >     >         >
>  >     >         >         Rob,
>  >     >         >
>  >     >         >         I entered 'date --date="7 April 2023",
> verified
>  it
>  >     >         updated the
>  >     >         >         system time appropriately. Restarted dirsrv,
>  >     ipa-custodia,
>  >     >         >         ipa-otpd, httpd.. krb5kdc and kadmin failed.
>  Still,
>  >     >         tried to
>  >     >         >         send ipa cert-update, and it popped the same SSL
>  >     >         Certificate
>  >     >         >         Verify Failed error.
>  >     >         >
>  >     >         >         On Thu, Apr 13, 2023 at 11:32 AM Rob Crittenden
>  >     >         >         <rcritten(a)redhat.com
>  >     <mailto:[e-mail]> <mailto:[e-mail]
>  >     <mailto:[e-mail]>>
>  >     >         <mailto:[e-mail] <mailto:[e-mail]>
>  >     <mailto:[e-mail] <mailto:[e-mail]>>>>
> wrote:
>  >     >         >
>  >     >         >             Justen Long wrote:
>  >     >         >             > Additionally, is there any way to force
>  the CA
>  >     >         cert update
>  >     >         >             to be
>  >     >         >             > recognized? When I run it to update the CA
>  >     chain,
>  >     >         >             everything is
>  >     >         >             > verified.. but /etc/ipa/ca.crt didn't
>  reflect
>  >     >         the change..
>  >     >         >             so I manually
>  >     >         >             > populated it by copying over the guts of
>  >     the CA
>  >     >         bundle to the
>  >     >         >             > /etc/ipa/ca.crt before trying to install
>  >     the new
>  >     >         server
>  >     >         >             cert and it
>  >     >         >             > still doesn't recognize it as trusted
>  although
>  >     >         the issuer
>  >     >         >             is the same
>  >     >         >             > and within the CA bundle.
>  >     >         >
>  >     >         >             This is going to sound weird, but I'd just
>  >     go back
>  >     >         in time
>  >     >         >             to April 10,
>  >     >         >             restart all services but ntp (which will
>  >     reset the
>  >     >         time) and
>  >     >         >             then the
>  >     >         >             commands should work. Once the certs are
>  >     updated and
>  >     >         >             working, return to
>  >     >         >             present time.
>  >     >         >
>  >     >         >             rob
>  >     >         >
>  >     >         >             >
>  >     >         >             > On Thu, Apr 13, 2023 at 6:20 AM Justen Long
>  >     >         >             <mr.justenlong(a)gmail.com
>  >     <mailto:[e-mail]>
>  >     >         <mailto:[e-mail]
>  >     <mailto:[e-mail]>> <mailto:[e-mail]
>  >     <mailto:[e-mail]>
>  >     >         <mailto:[e-mail]
>  >     <mailto:[e-mail]>>>
>  >     >         >             > <mailto:[e-mail]
>  >     <mailto:[e-mail]>
>  >     >         <mailto:[e-mail]
>  >     <mailto:[e-mail]>>
>  >     >         >             <mailto:[e-mail]
>  >     <mailto:[e-mail]>
>  >     >         <mailto:[e-mail]
>  >     <mailto:[e-mail]>>>>> wrote:
>  >     >         >             >
>  >     >         >             >     Rob,
>  >     >         >             >
>  >     >         >             >     Apologies for the delay in response.
>  Once
>  >     >         I'm home, I
>  >     >         >             don't have
>  >     >         >             >     access to the information readily
>  >     available
>  >     >         to respond
>  >     >         >             with. Here is
>  >     >         >             >     the information you requested:
>  >     >         >             >
>  >     >         >             >     The version of IPA we are using is
>  >     4.6.8, rpm
>  >     >         >             specifically for us is
>  >     >         >             >
>  >      ipa-server-4.6.8-5.el7.centos.12.x86_64 and
>  >     >         we are
>  >     >         >             using CentOS 7.9
>  >     >         >             >     currently with plans to move to RHEL9
>  >     within
>  >     >         the next
>  >     >         >             year or so.
>  >     >         >             >
>  >     >         >             >     Unfortunately, 'ipa config-show'
>  doesn't
>  >     >         work. It
>  >     >         >             populates the same
>  >     >         >             >     error stating "ipa: ERROR: cannot
>  >     connect to
>  >     >         >             >     'https://ipaServer/ipa/json':
> [SSL:
>  >     >         >             CERTIFICATE_VERIFY_FAILED]
>  >     >         >             >     certificate verify failed (_ssl.c:618).
>  >     >         >
>  >     >         >             The smack heard around the world was my head
>  >     >         hitting my
>  >     >         >             desk. Of course
>  >     >         >             this command failed.
>  >     >         >
>  >     >         >             >
>  >     >         >             >     We have ~50 hosts connected via IPA.
>  >     We have
>  >     >         two IPA
>  >     >         >             servers, one as
>  >     >         >             >     a replica of the other.
>  >     >         >             >
>  >     >         >             >     'getcert list' only shows 1
>  certificate.
>  >     >         It's state is
>  >     >         >             "MONITORING"
>  >     >         >             >     and seems related to kerberos.
>  >     >         >             >
>  >     >         >             >     As far as I know, we don't use IPA
>  >     CA-issued
>  >     >         >             certificates. I recall
>  >     >         >             >     seeing errors yesterday stating CA
>  wasn't
>  >     >         enabled on
>  >     >         >             our servers. We
>  >     >         >             >     have always used 3rd party CAs to my
>  >     knowledge.
>  >     >         >             >
>  >     >         >             >     -justen
>  >     >         >             >
>  >     >         >             >     On Wed, Apr 12, 2023 at 2:42 PM Rob
>  >     Crittenden
>  >     >         >             <rcritten(a)redhat.com
>  >     <mailto:[e-mail]> <mailto:[e-mail]
>  >     <mailto:[e-mail]>>
>  >     >         <mailto:[e-mail] <mailto:[e-mail]>
>  >     <mailto:[e-mail] <mailto:[e-mail]>>>
>  >     >         >             >     <mailto:[e-mail]
>  >     <mailto:[e-mail]>
>  >     >         <mailto:[e-mail]
> <mailto:[e-mail]>>
>  >     >         >             <mailto:[e-mail]
>  >     <mailto:[e-mail]>
>  >     >         <mailto:[e-mail]
>  >     <mailto:[e-mail]>>>>> wrote:
>  >     >         >             >
>  >     >         >             >         Justen Long via FreeIPA-users
>  wrote:
>  >     >         >             >         > Thanks in advance for your
>  replies..
>  >     >         I've spent
>  >     >         >             7 hours
>  >     >         >             >         looking through posts here and
>  trying
>  >     >         >             everything... I'm stuck.
>  >     >         >             >         >
>  >     >         >             >         > Background: I am a System
>  >     >         Administrator in a closed,
>  >     >         >             >         classified environment.
>  >     Unfortunately, I
>  >     >         cannot
>  >     >         >             post logging
>  >     >         >             >         here, but I can refer to them as
>  >     needed.
>  >     >         >             >         >
>  >     >         >             >         > I inherited this system from
>  >     someone who
>  >     >         >             departed the program
>  >     >         >             >         a year or so ago. Fast forward to
>  >     today, the
>  >     >         >             server certs
>  >     >         >             >         expired yesterday. Admittedly,
> I'm
>  >     >         unfamiliar (or
>  >     >         >             was) with the
>  >     >         >             >         certificate update process for IPA
>  >     >         servers. On a
>  >     >         >             typical server,
>  >     >         >             >         we replace the old cert and
>  >     restart the
>  >     >         httpd
>  >     >         >             services; however,
>  >     >         >             >         I realize this cannot work with IPA
>  >     >         servers now.
>  >     >         >             >         >
>  >     >         >             >         > Additionally to all of this,
> the
>  >     CA chain
>  >     >         >             updated 6 months ago.
>  >     >         >             >         >
>  >     >         >             >         > I ran ipa-cacert-manage to
>  >     update the
>  >     >         CA chain.
>  >     >         >             When trying to
>  >     >         >             >         run ipa-certupdate, I received
>  >     errors for an
>  >     >         >             invalid server
>  >     >         >             >         certificate (it expired on 11 April
>  >     >         2023). It
>  >     >         >             simply won't
>  >     >         >             >         connect to the web server. HTTPD
>  >     failed
>  >     >         as well,
>  >     >         >             so I had to add
>  >     >         >             >         "NSSEnforceValidCerts off"
> to the
>  >     >         nss.conf file
>  >     >         >             for HTTPD to
>  >     >         >             >         start. Still, no dice.
>  >     >         >             >         >
>  >     >         >             >         > I've ran
> ipa-server-certinstall
>  for
>  >     >         the new
>  >     >         >             cert/key as well,
>  >     >         >             >         and it fails saying its not
>  >     trusted ("Peer's
>  >     >         >             certificate issuer
>  >     >         >             >         is not trusted [certutil:
>  >     certificate is
>  >     >         invalid:
>  >     >         >             Peer's
>  >     >         >             >         Certificate issuer is not
>  recognized]
>  >     >         Please run
>  >     >         >             >         ipa-cacert-manage install and
>  >     >         ipa-certupdate to
>  >     >         >             install the CA
>  >     >         >             >         certificate.... which, as reported
>  >     >         above, can't
>  >     >         >             complete.
>  >     >         >             >         >
>  >     >         >             >         > I'm at a total loss here...
> and
>  >     really
>  >     >         >             struggling being new to
>  >     >         >             >         all this and trying my best to
>  keep it
>  >     >         afloat. Any
>  >     >         >             help would be
>  >     >         >             >         GREATLY appreciated!
>  >     >         >             >
>  >     >         >             >         Let's gather some information
>  first.
>  >     >         >             >
>  >     >         >             >         What version of IPA is this, on
>  what
>  >     >         distribution?
>  >     >         >             >
>  >     >         >             >         IPA designates one server to be
>  >     the "renewal
>  >     >         >             master" which
>  >     >         >             >         handles the
>  >     >         >             >         renewals. The output of `ipa
>  >     >         config-show` should
>  >     >         >             tell you
>  >     >         >             >         (depending on
>  >     >         >             >         version). That's the server you
>  >     want to
>  >     >         work on.
>  >     >         >             >
>  >     >         >             >         How many servers in your topology
>  and
>  >     >         how many
>  >     >         >             have a CA installed?
>  >     >         >             >
>  >     >         >             >         Does `getcert list` show a set of
>  8-10
>  >     >         tracked
>  >     >         >             certificates?
>  >     >         >             >         What are
>  >     >         >             >         the states?
>  >     >         >             >
>  >     >         >             >         You mention
>  >     ipa-server-certinstall. Are
>  >     >         you using
>  >     >         >             3rd party
>  >     >         >             >         certificates
>  >     >         >             >         in addition to IPA CA-issued
>  >     >         certificates or was
>  >     >         >             that just an
>  >     >         >             >         attempt to
>  >     >         >             >         get things working again?
>  >     >         >             >
>  >     >         >             >         rob
>  >     >         >             >
>  >     >         >
>  >     >
>  >
>
>

[Rob Crittenden]

Justen Long wrote:
>  Rob,
>  
>  Currently only hiipa03 and hiipa04 (these are the masters/IPA servers)
>  would utilize the ipa-server-certinstall commands. The other servers
>  don't use any HTTPD or anything like that, and have never had issues in
>  the past simply replacing the cert file in /etc/pki/tls/certs and
>  letting it ride.
>  
>  So, with regard to the catch-22...
>  
>  hiipa03 isn't communicating appropriately or something, in my opinion. 
>  When attempting to run "ipa-certupdate" on it, it wants the Kerberos
>  credentials (assuming 'kinit admin'). I run 'kinit admin" and enter the
>  admin password that works on hiipa04, and it doesn't like it. I
>  attempted to change that password using the long 'ldappasswd' command,
>  but forgot to put dc=hilab,dc=ipa, and instead had put dc=hilab.ipa.
>  This results in a failure, object not found.. (had a duh moment).. went
>  to try the 'ldappasswd' command again, using dc=hilab,dc=ipa.. it then
>  fails stating "Result: Operations error (1) \\ Additional info: Failed
>  to update password".
>  
>  So, I just, for s&g, run 'kinit admin' again. Prompts for password. Try
>  the one that failed because of the wrong dc input, and it takes it,
>  stating its expired. Try to change it, and get the same "Result:
>  Operations error (1) \\ Additional info:" Failed to update password."
>  
>  On the majority of the clients, when I enter "ipa-certupdate" it points
>  to hiipa03, rather than hiipa04.. and fails because hiipa03 is still in
>  shambles. I think if I can get hiipa03 back to operational, then
>  everything else would work.
>  
>  1) Is there some magical way to destroy the admin password (i.e. blank
>  it out) so it prompts for a new one, or to replicate the admin stuff
>  from hiipa04 to hiipa03 so that it all works?
>  2) If not, how can I possibly move forward? The only hold up is the
>  admin password at this point. 
It's hard to say without any logs. It sure seems like an issue with the
389 server on 03. It isn't clear what you've already done, like
restarting services. I assume you got the certificates replaced there as
well?

But if passwords can't be set that points to something in 389. The
errors log might contain information.

The hardcore option would be to uninstall IPA from 03 and install it as
a new replica from scratch. This assumes that 04 really is working properly.

rob

>  
>  On Fri, Apr 14, 2023 at 7:11 AM Rob Crittenden <rcritten(a)redhat.com
>  <mailto:[e-mail]>> wrote:
>  
>      Justen Long wrote:
>      > Rob,
>      >
>      > FreeIPA documentation stated that all clients must receive the
>      > ipa-updatecert command before installing the new server cert.. on the
>      > hiipa04 server I installed the cert in a hurry for validity sake. Does
>      > that mess with anything?
>  
>      ipa-certupdate distributes the CA chain. If the chain didn't change then
>      it should be fine.
>  
>      >
>      > Note: hiipa03 seems to be primary but they’re both masters. I
>      can’t get
>      > the ipa-server-certinstall to take on this server stating it’s missing
>      > the whole chain.. but it worked just fine on hiipa04.. so that’s why I
>      > was trying to “kinit admin” on 03, to run ipa-updatecert or the
>      > ipa-cacert-manage there if it didn’t take it from 04, which I’m
>      assuming
>      > it didn’t.
>  
>      You could try going back in time and use ipa-cacert-manage install to
>      update the chain and then run ipa-server-certinstall.
>  
>      But like I said before, if you're changing your chain AND replacing the
>      certs at the same time none of your clients are going to know what
>      to do.
>  
>      The process is:
>  
>      1. ipa-cacert-manage install <new chain>
>      2. ipa-certupdate on ALL enrolled machines
>      3. Replace the server certs
>  
>      If you deviate then you're caught in the catch-22 I mentioned. Clients
>      won't trust the updated chain in order to download the updated chain.
>      The aren't a lot of great options out of that. Unenrolling and
>      re-enrolling the client is the best way.
>  
>      rob
>      >
>      > On Thu, Apr 13, 2023 at 2:41 PM Rob Crittenden
>      <rcritten(a)redhat.com <mailto:[e-mail]>
>      > <mailto:[e-mail] <mailto:[e-mail]>>>
> wrote:
>      >
>      >     Justen Long wrote:
>      >     > One go back.. when I tried to run "ipa-certupdate" on the
>      other hosts
>      >     > (clients), it points to hiipa03 and fails for SSL still..
>      so, hoping
>      >     > once I get the cert updated on THAT server, that all
>      restores to its
>      >     > okay state.
>      >
>      >     You shouldn't need to go back for clients. Now that the server
>      >     certificates are valid they should just work. Unless you also
>      changed
>      >     the CA chain in which case you're caught in a catch-22: you
>      need to get
>      >     the updated chain from the server but you don't trust the
>      chain of the
>      >     server.
>      >
>      >     rob
>      >
>      >     >
>      >     > On Thu, Apr 13, 2023 at 1:46 PM Justen Long
>      >     &lt;mr.justenlong(a)gmail.com <mailto:[e-mail]>
>      <mailto:[e-mail] <mailto:[e-mail]>>
>      >     > <mailto:[e-mail]
>      <mailto:[e-mail]> <mailto:[e-mail]
>      <mailto:[e-mail]>>>>
>      >     wrote:
>      >     >
>      >     >     Quick update and answers to your questions.
>      >     >
>      >     >     We have two IPA servers, both masters, hiipa03 and hiipa04.
>      >     >
>      >     >     hiipa04, I was able to set the time back, run the
>      >     ipa-ca-cert-manage
>      >     >     and update the CA (minimally, but enough to accept the
>      new cert)..
>      >     >     ipa-certupdate ran fine on it. Then, ran
>      ipa-server-certinstall on
>      >     >     it, and it took. Website loads, can log into it, do some
>      user
>      >     >     management stuff.. can't run "ipa-replica-manage
> list"
>      as it kicks
>      >     >     an error for sslv3 handshake failure.. but was trying
>      that to
>      >     remove
>      >     >     hiipa03 and copy 04 to 03 somehow, maybe? 
>      >     >
>      >     >     hiipa03 is still giving me grief. Set the time using
>      timedatectl,
>      >     >     verified ntp is off and 'date' reports properly.  I had
>      tried to
>      >     >     follow
>      >     >   
>      >   
>     
>   this: https://computingforgeeks.com/reset-freeipa-admin-password-as-root-user-on-linux/
>      >     >     when it wasn't accepting the admin password.. it failed
>      saying
>      >     >     object not found. So, I try 'kinit admin' again, with
>      the new
>      >     >     password I tried.. says its expired. Type it in, type in
>      a new
>      >     >     password.. and then it failed saying "kinit: Password change
>      >     failed
>      >     >     while getting initial credentials"
>      >     >
>      >     >     On Thu, Apr 13, 2023 at 1:40 PM Rob Crittenden
>      >     &lt;rcritten(a)redhat.com <mailto:[e-mail]>
>      <mailto:[e-mail] <mailto:[e-mail]>>
>      >     >     <mailto:[e-mail] <mailto:[e-mail]>
>      <mailto:[e-mail] <mailto:[e-mail]>>>> wrote:
>      >     >
>      >     >         Justen Long wrote:
>      >     >         > I'm getting closer... it's not recognizing my
>      admin password
>      >     >         for IPA, or
>      >     >         > for my personal account with admin rights now..
>      but no more
>      >     >         SSL errors..
>      >     >         > just can't run ipa-certupdate without the proper
>      >     kerberos creds..
>      >     >
>      >     >         By not recognizing your password I assume you mean
>      kinit is
>      >     >         failing? Is
>      >     >         the KDC running? I assume 389-ds is running? All
>      restarted
>      >     after
>      >     >         time
>      >     >         became stable in the past?
>      >     >
>      >     >         rob
>      >     >
>      >     >         >
>      >     >         > On Thu, Apr 13, 2023 at 12:51 PM Justen Long
>      >     >         &lt;mr.justenlong(a)gmail.com
>      <mailto:[e-mail]> <mailto:[e-mail]
>      <mailto:[e-mail]>>
>      >     <mailto:[e-mail]
>      <mailto:[e-mail]> <mailto:[e-mail]
>      <mailto:[e-mail]>>>
>      >     >         > <mailto:[e-mail]
>      <mailto:[e-mail]>
>      >     <mailto:[e-mail]
> <mailto:[e-mail]>>
>      >     >         <mailto:[e-mail]
>      <mailto:[e-mail]>
>      >     <mailto:[e-mail]
>      <mailto:[e-mail]>>>>> wrote:
>      >     >         >
>      >     >         >     Following up, I see the date command just
>      changed it
>      >     >         momentarily...
>      >     >         >     using timedatectl and will report back.
>      >     >         >
>      >     >         >     On Thu, Apr 13, 2023 at 12:31 PM Justen Long
>      >     >         >     &lt;mr.justenlong(a)gmail.com
>      <mailto:[e-mail]>
>      >     <mailto:[e-mail]
>      <mailto:[e-mail]>> <mailto:[e-mail]
>      <mailto:[e-mail]>
>      >     <mailto:[e-mail]
> <mailto:[e-mail]>>>
>      >     >         <mailto:[e-mail]
>      <mailto:[e-mail]>
>      >     <mailto:[e-mail]
> <mailto:[e-mail]>>
>      >     >         <mailto:[e-mail]
>      <mailto:[e-mail]>
>      >     <mailto:[e-mail]
>      <mailto:[e-mail]>>>>> wrote:
>      >     >         >
>      >     >         >         Rob,
>      >     >         >
>      >     >         >         I entered 'date --date="7 April
> 2023",
>      verified it
>      >     >         updated the
>      >     >         >         system time appropriately. Restarted dirsrv,
>      >     ipa-custodia,
>      >     >         >         ipa-otpd, httpd.. krb5kdc and kadmin
>      failed. Still,
>      >     >         tried to
>      >     >         >         send ipa cert-update, and it popped the
>      same SSL
>      >     >         Certificate
>      >     >         >         Verify Failed error.
>      >     >         >
>      >     >         >         On Thu, Apr 13, 2023 at 11:32 AM Rob
>      Crittenden
>      >     >         >         &lt;rcritten(a)redhat.com
>      <mailto:[e-mail]>
>      >     <mailto:[e-mail] <mailto:[e-mail]>>
>      <mailto:[e-mail] <mailto:[e-mail]>
>      >     <mailto:[e-mail] <mailto:[e-mail]>>>
>      >     >         <mailto:[e-mail]
>      <mailto:[e-mail]> <mailto:[e-mail]
>      <mailto:[e-mail]>>
>      >     <mailto:[e-mail] <mailto:[e-mail]>
>      <mailto:[e-mail] <mailto:[e-mail]>>>>>
> wrote:
>      >     >         >
>      >     >         >             Justen Long wrote:
>      >     >         >             > Additionally, is there any way to
>      force the CA
>      >     >         cert update
>      >     >         >             to be
>      >     >         >             > recognized? When I run it to update
>      the CA
>      >     chain,
>      >     >         >             everything is
>      >     >         >             > verified.. but /etc/ipa/ca.crt
>      didn't reflect
>      >     >         the change..
>      >     >         >             so I manually
>      >     >         >             > populated it by copying over the guts of
>      >     the CA
>      >     >         bundle to the
>      >     >         >             > /etc/ipa/ca.crt before trying to install
>      >     the new
>      >     >         server
>      >     >         >             cert and it
>      >     >         >             > still doesn't recognize it as
>      trusted although
>      >     >         the issuer
>      >     >         >             is the same
>      >     >         >             > and within the CA bundle.
>      >     >         >
>      >     >         >             This is going to sound weird, but I'd
> just
>      >     go back
>      >     >         in time
>      >     >         >             to April 10,
>      >     >         >             restart all services but ntp (which will
>      >     reset the
>      >     >         time) and
>      >     >         >             then the
>      >     >         >             commands should work. Once the certs are
>      >     updated and
>      >     >         >             working, return to
>      >     >         >             present time.
>      >     >         >
>      >     >         >             rob
>      >     >         >
>      >     >         >             >
>      >     >         >             > On Thu, Apr 13, 2023 at 6:20 AM
>      Justen Long
>      >     >         >             &lt;mr.justenlong(a)gmail.com
>      <mailto:[e-mail]>
>      >     <mailto:[e-mail]
> <mailto:[e-mail]>>
>      >     >         <mailto:[e-mail]
>      <mailto:[e-mail]>
>      >     <mailto:[e-mail]
>      <mailto:[e-mail]>>> <mailto:[e-mail]
>      <mailto:[e-mail]>
>      >     <mailto:[e-mail]
> <mailto:[e-mail]>>
>      >     >         <mailto:[e-mail]
>      <mailto:[e-mail]>
>      >     <mailto:[e-mail]
>      <mailto:[e-mail]>>>>
>      >     >         >             > <mailto:[e-mail]
>      <mailto:[e-mail]>
>      >     <mailto:[e-mail]
> <mailto:[e-mail]>>
>      >     >         <mailto:[e-mail]
>      <mailto:[e-mail]>
>      >     <mailto:[e-mail]
> <mailto:[e-mail]>>>
>      >     >         >             <mailto:[e-mail]
>      <mailto:[e-mail]>
>      >     <mailto:[e-mail]
> <mailto:[e-mail]>>
>      >     >         <mailto:[e-mail]
>      <mailto:[e-mail]>
>      >     <mailto:[e-mail]
>      <mailto:[e-mail]>>>>>> wrote:
>      >     >         >             >
>      >     >         >             >     Rob,
>      >     >         >             >
>      >     >         >             >     Apologies for the delay in
>      response. Once
>      >     >         I'm home, I
>      >     >         >             don't have
>      >     >         >             >     access to the information readily
>      >     available
>      >     >         to respond
>      >     >         >             with. Here is
>      >     >         >             >     the information you requested:
>      >     >         >             >
>      >     >         >             >     The version of IPA we are using is
>      >     4.6.8, rpm
>      >     >         >             specifically for us is
>      >     >         >             >   
>      >      ipa-server-4.6.8-5.el7.centos.12.x86_64 and
>      >     >         we are
>      >     >         >             using CentOS 7.9
>      >     >         >             >     currently with plans to move to
>      RHEL9
>      >     within
>      >     >         the next
>      >     >         >             year or so.
>      >     >         >             >
>      >     >         >             >     Unfortunately, 'ipa
> config-show'
>      doesn't
>      >     >         work. It
>      >     >         >             populates the same
>      >     >         >             >     error stating "ipa: ERROR:
> cannot
>      >     connect to
>      >     >         >             >     'https://ipaServer/ipa/json':
> [SSL:
>      >     >         >             CERTIFICATE_VERIFY_FAILED]
>      >     >         >             >     certificate verify failed
>      (_ssl.c:618).
>      >     >         >
>      >     >         >             The smack heard around the world was
>      my head
>      >     >         hitting my
>      >     >         >             desk. Of course
>      >     >         >             this command failed.
>      >     >         >
>      >     >         >             >
>      >     >         >             >     We have ~50 hosts connected via IPA.
>      >     We have
>      >     >         two IPA
>      >     >         >             servers, one as
>      >     >         >             >     a replica of the other.
>      >     >         >             >
>      >     >         >             >     'getcert list' only shows 1
>      certificate.
>      >     >         It's state is
>      >     >         >             "MONITORING"
>      >     >         >             >     and seems related to kerberos.
>      >     >         >             >
>      >     >         >             >     As far as I know, we don't use
> IPA
>      >     CA-issued
>      >     >         >             certificates. I recall
>      >     >         >             >     seeing errors yesterday stating
>      CA wasn't
>      >     >         enabled on
>      >     >         >             our servers. We
>      >     >         >             >     have always used 3rd party CAs to my
>      >     knowledge.
>      >     >         >             >
>      >     >         >             >     -justen
>      >     >         >             >
>      >     >         >             >     On Wed, Apr 12, 2023 at 2:42 PM Rob
>      >     Crittenden
>      >     >         >             &lt;rcritten(a)redhat.com
>      <mailto:[e-mail]>
>      >     <mailto:[e-mail] <mailto:[e-mail]>>
>      <mailto:[e-mail] <mailto:[e-mail]>
>      >     <mailto:[e-mail] <mailto:[e-mail]>>>
>      >     >         <mailto:[e-mail]
>      <mailto:[e-mail]> <mailto:[e-mail]
>      <mailto:[e-mail]>>
>      >     <mailto:[e-mail] <mailto:[e-mail]>
>      <mailto:[e-mail] <mailto:[e-mail]>>>>
>      >     >         >             >     <mailto:[e-mail]
>      <mailto:[e-mail]>
>      >     <mailto:[e-mail] <mailto:[e-mail]>>
>      >     >         <mailto:[e-mail]
>      <mailto:[e-mail]> <mailto:[e-mail]
>      <mailto:[e-mail]>>>
>      >     >         >             <mailto:[e-mail]
>      <mailto:[e-mail]>
>      >     <mailto:[e-mail] <mailto:[e-mail]>>
>      >     >         <mailto:[e-mail]
> <mailto:[e-mail]>
>      >     <mailto:[e-mail]
> <mailto:[e-mail]>>>>>>
>      wrote:
>      >     >         >             >
>      >     >         >             >         Justen Long via
>      FreeIPA-users wrote:
>      >     >         >             >         > Thanks in advance for your
>      replies..
>      >     >         I've spent
>      >     >         >             7 hours
>      >     >         >             >         looking through posts here
>      and trying
>      >     >         >             everything... I'm stuck.
>      >     >         >             >         >
>      >     >         >             >         > Background: I am a System
>      >     >         Administrator in a closed,
>      >     >         >             >         classified environment.
>      >     Unfortunately, I
>      >     >         cannot
>      >     >         >             post logging
>      >     >         >             >         here, but I can refer to them as
>      >     needed.
>      >     >         >             >         >
>      >     >         >             >         > I inherited this system
> from
>      >     someone who
>      >     >         >             departed the program
>      >     >         >             >         a year or so ago. Fast
>      forward to
>      >     today, the
>      >     >         >             server certs
>      >     >         >             >         expired yesterday.
>      Admittedly, I'm
>      >     >         unfamiliar (or
>      >     >         >             was) with the
>      >     >         >             >         certificate update process
>      for IPA
>      >     >         servers. On a
>      >     >         >             typical server,
>      >     >         >             >         we replace the old cert and
>      >     restart the
>      >     >         httpd
>      >     >         >             services; however,
>      >     >         >             >         I realize this cannot work
>      with IPA
>      >     >         servers now.
>      >     >         >             >         >
>      >     >         >             >         > Additionally to all of
>      this, the
>      >     CA chain
>      >     >         >             updated 6 months ago.
>      >     >         >             >         >
>      >     >         >             >         > I ran ipa-cacert-manage to
>      >     update the
>      >     >         CA chain.
>      >     >         >             When trying to
>      >     >         >             >         run ipa-certupdate, I received
>      >     errors for an
>      >     >         >             invalid server
>      >     >         >             >         certificate (it expired on
>      11 April
>      >     >         2023). It
>      >     >         >             simply won't
>      >     >         >             >         connect to the web server. HTTPD
>      >     failed
>      >     >         as well,
>      >     >         >             so I had to add
>      >     >         >             >         "NSSEnforceValidCerts
> off"
>      to the
>      >     >         nss.conf file
>      >     >         >             for HTTPD to
>      >     >         >             >         start. Still, no dice.
>      >     >         >             >         >
>      >     >         >             >         > I've ran
>      ipa-server-certinstall for
>      >     >         the new
>      >     >         >             cert/key as well,
>      >     >         >             >         and it fails saying its not
>      >     trusted ("Peer's
>      >     >         >             certificate issuer
>      >     >         >             >         is not trusted [certutil:
>      >     certificate is
>      >     >         invalid:
>      >     >         >             Peer's
>      >     >         >             >         Certificate issuer is not
>      recognized]
>      >     >         Please run
>      >     >         >             >         ipa-cacert-manage install and
>      >     >         ipa-certupdate to
>      >     >         >             install the CA
>      >     >         >             >         certificate.... which, as
>      reported
>      >     >         above, can't
>      >     >         >             complete.
>      >     >         >             >         >
>      >     >         >             >         > I'm at a total loss
>      here... and
>      >     really
>      >     >         >             struggling being new to
>      >     >         >             >         all this and trying my best
>      to keep it
>      >     >         afloat. Any
>      >     >         >             help would be
>      >     >         >             >         GREATLY appreciated!
>      >     >         >             >
>      >     >         >             >         Let's gather some
>      information first.
>      >     >         >             >
>      >     >         >             >         What version of IPA is this,
>      on what
>      >     >         distribution?
>      >     >         >             >
>      >     >         >             >         IPA designates one server to be
>      >     the "renewal
>      >     >         >             master" which
>      >     >         >             >         handles the
>      >     >         >             >         renewals. The output of `ipa
>      >     >         config-show` should
>      >     >         >             tell you
>      >     >         >             >         (depending on
>      >     >         >             >         version). That's the server
> you
>      >     want to
>      >     >         work on.
>      >     >         >             >
>      >     >         >             >         How many servers in your
>      topology and
>      >     >         how many
>      >     >         >             have a CA installed?
>      >     >         >             >
>      >     >         >             >         Does `getcert list` show a
>      set of 8-10
>      >     >         tracked
>      >     >         >             certificates?
>      >     >         >             >         What are
>      >     >         >             >         the states?
>      >     >         >             >
>      >     >         >             >         You mention
>      >     ipa-server-certinstall. Are
>      >     >         you using
>      >     >         >             3rd party
>      >     >        ��>             >         certificates
>      >     >         >             >         in addition to IPA CA-issued
>      >     >         certificates or was
>      >     >         >             that just an
>      >     >         >             >         attempt to
>      >     >         >             >         get things working again?
>      >     >         >             >
>      >     >         >             >         rob
>      >     >         >             >
>      >     >         >
>      >     >
>      >
>

[Justen Long]

Rob,

I wound up hitting the *nuclear option* button, and as you mentioned,
removed 03 and rebuilt as a replica from 04.

I was still getting errors when attempting it regarding STARTTLS.

Add this to your notes, and then feel free to slap me in the back of the
head, but..

A huge part of our compliance / security team barking, is alllllllllllll
about TLS. After hours of troubleshooting, I went into the "error" log
/var/log/dirsrv/slapd-INSTANCE/error ... and I saw errors about my ciphers
in dse.ldif.

I had (a week or two ago) updated our ciphers listed under nsSSL3Ciphers:
to disallow some ciphers that were considered weak now. My scans for TLS
came back clean, but the error was because I left "default" there, instead
of changing to "+all"

Once I switched it to +all, the ipa-replica-install worked. Now I need to
figure out how to get the webgui working and I'll be golden, I think!

On Fri, Apr 14, 2023 at 11:07 AM Rob Crittenden <rcritten(a)redhat.com&gt; wrote:

>  Justen Long wrote:
>  > Rob,
>  >
>  > Currently only hiipa03 and hiipa04 (these are the masters/IPA servers)
>  > would utilize the ipa-server-certinstall commands. The other servers
>  > don't use any HTTPD or anything like that, and have never had issues in
>  > the past simply replacing the cert file in /etc/pki/tls/certs and
>  > letting it ride.
>  >
>  > So, with regard to the catch-22...
>  >
>  > hiipa03 isn't communicating appropriately or something, in my opinion.
>  > When attempting to run "ipa-certupdate" on it, it wants the Kerberos
>  > credentials (assuming 'kinit admin'). I run 'kinit admin" and enter
> the
>  > admin password that works on hiipa04, and it doesn't like it. I
>  > attempted to change that password using the long 'ldappasswd' command,
>  > but forgot to put dc=hilab,dc=ipa, and instead had put dc=hilab.ipa.
>  > This results in a failure, object not found.. (had a duh moment).. went
>  > to try the 'ldappasswd' command again, using dc=hilab,dc=ipa.. it then
>  > fails stating "Result: Operations error (1) \\ Additional info: Failed
>  > to update password".
>  >
>  > So, I just, for s&g, run 'kinit admin' again. Prompts for password. Try
>  > the one that failed because of the wrong dc input, and it takes it,
>  > stating its expired. Try to change it, and get the same "Result:
>  > Operations error (1) \\ Additional info:" Failed to update password."
>  >
>  > On the majority of the clients, when I enter "ipa-certupdate" it points
>  > to hiipa03, rather than hiipa04.. and fails because hiipa03 is still in
>  > shambles. I think if I can get hiipa03 back to operational, then
>  > everything else would work.
>  >
>  > 1) Is there some magical way to destroy the admin password (i.e. blank
>  > it out) so it prompts for a new one, or to replicate the admin stuff
>  > from hiipa04 to hiipa03 so that it all works?
>  > 2) If not, how can I possibly move forward? The only hold up is the
>  > admin password at this point.
>
>  It's hard to say without any logs. It sure seems like an issue with the
>  389 server on 03. It isn't clear what you've already done, like
>  restarting services. I assume you got the certificates replaced there as
>  well?
>
>  But if passwords can't be set that points to something in 389. The
>  errors log might contain information.
>
>  The hardcore option would be to uninstall IPA from 03 and install it as
>  a new replica from scratch. This assumes that 04 really is working
>  properly.
>
>  rob
>
>  >
>  > On Fri, Apr 14, 2023 at 7:11 AM Rob Crittenden <rcritten(a)redhat.com
>  > <mailto:[e-mail]>> wrote:
>  >
>  >     Justen Long wrote:
>  >     > Rob,
>  >     >
>  >     > FreeIPA documentation stated that all clients must receive the
>  >     > ipa-updatecert command before installing the new server cert.. on
>  the
>  >     > hiipa04 server I installed the cert in a hurry for validity sake.
>  Does
>  >     > that mess with anything?
>  >
>  >     ipa-certupdate distributes the CA chain. If the chain didn't change
>  then
>  >     it should be fine.
>  >
>  >     >
>  >     > Note: hiipa03 seems to be primary but they’re both masters. I
>  >     can’t get
>  >     > the ipa-server-certinstall to take on this server stating it’s
>  missing
>  >     > the whole chain.. but it worked just fine on hiipa04.. so that’s
>  why I
>  >     > was trying to “kinit admin” on 03, to run ipa-updatecert or the
>  >     > ipa-cacert-manage there if it didn’t take it from 04, which I’m
>  >     assuming
>  >     > it didn’t.
>  >
>  >     You could try going back in time and use ipa-cacert-manage install to
>  >     update the chain and then run ipa-server-certinstall.
>  >
>  >     But like I said before, if you're changing your chain AND replacing
>  the
>  >     certs at the same time none of your clients are going to know what
>  >     to do.
>  >
>  >     The process is:
>  >
>  >     1. ipa-cacert-manage install <new chain>
>  >     2. ipa-certupdate on ALL enrolled machines
>  >     3. Replace the server certs
>  >
>  >     If you deviate then you're caught in the catch-22 I mentioned.
>  Clients
>  >     won't trust the updated chain in order to download the updated chain.
>  >     The aren't a lot of great options out of that. Unenrolling and
>  >     re-enrolling the client is the best way.
>  >
>  >     rob
>  >     >
>  >     > On Thu, Apr 13, 2023 at 2:41 PM Rob Crittenden
>  >     <rcritten(a)redhat.com <mailto:[e-mail]>
>  >     > <mailto:[e-mail] <mailto:[e-mail]>>>
> wrote:
>  >     >
>  >     >     Justen Long wrote:
>  >     >     > One go back.. when I tried to run "ipa-certupdate" on
> the
>  >     other hosts
>  >     >     > (clients), it points to hiipa03 and fails for SSL still..
>  >     so, hoping
>  >     >     > once I get the cert updated on THAT server, that all
>  >     restores to its
>  >     >     > okay state.
>  >     >
>  >     >     You shouldn't need to go back for clients. Now that the server
>  >     >     certificates are valid they should just work. Unless you also
>  >     changed
>  >     >     the CA chain in which case you're caught in a catch-22: you
>  >     need to get
>  >     >     the updated chain from the server but you don't trust the
>  >     chain of the
>  >     >     server.
>  >     >
>  >     >     rob
>  >     >
>  >     >     >
>  >     >     > On Thu, Apr 13, 2023 at 1:46 PM Justen Long
>  >     >     <mr.justenlong(a)gmail.com <mailto:[e-mail]>
>  >     <mailto:[e-mail] <mailto:[e-mail]>>
>  >     >     > <mailto:[e-mail]
>  >     <mailto:[e-mail]> <mailto:[e-mail]
>  >     <mailto:[e-mail]>>>>
>  >     >     wrote:
>  >     >     >
>  >     >     >     Quick update and answers to your questions.
>  >     >     >
>  >     >     >     We have two IPA servers, both masters, hiipa03 and
>  hiipa04.
>  >     >     >
>  >     >     >     hiipa04, I was able to set the time back, run the
>  >     >     ipa-ca-cert-manage
>  >     >     >     and update the CA (minimally, but enough to accept the
>  >     new cert)..
>  >     >     >     ipa-certupdate ran fine on it. Then, ran
>  >     ipa-server-certinstall on
>  >     >     >     it, and it took. Website loads, can log into it, do some
>  >     user
>  >     >     >     management stuff.. can't run "ipa-replica-manage
> list"
>  >     as it kicks
>  >     >     >     an error for sslv3 handshake failure.. but was trying
>  >     that to
>  >     >     remove
>  >     >     >     hiipa03 and copy 04 to 03 somehow, maybe?
>  >     >     >
>  >     >     >     hiipa03 is still giving me grief. Set the time using
>  >     timedatectl,
>  >     >     >     verified ntp is off and 'date' reports properly.  I
> had
>  >     tried to
>  >     >     >     follow
>  >     >     >
>  >     >
>  >       this:
>  https://computingforgeeks.com/reset-freeipa-admin-password-as-root-user-o...
>  >     >     >     when it wasn't accepting the admin password.. it failed
>  >     saying
>  >     >     >     object not found. So, I try 'kinit admin' again, with
>  >     the new
>  >     >     >     password I tried.. says its expired. Type it in, type in
>  >     a new
>  >     >     >     password.. and then it failed saying "kinit: Password
>  change
>  >     >     failed
>  >     >     >     while getting initial credentials"
>  >     >     >
>  >     >     >     On Thu, Apr 13, 2023 at 1:40 PM Rob Crittenden
>  >     >     <rcritten(a)redhat.com <mailto:[e-mail]>
>  >     <mailto:[e-mail] <mailto:[e-mail]>>
>  >     >     >     <mailto:[e-mail]
> <mailto:[e-mail]>
>  >     <mailto:[e-mail] <mailto:[e-mail]>>>>
> wrote:
>  >     >     >
>  >     >     >         Justen Long wrote:
>  >     >     >         > I'm getting closer... it's not recognizing
> my
>  >     admin password
>  >     >     >         for IPA, or
>  >     >     >         > for my personal account with admin rights now..
>  >     but no more
>  >     >     >         SSL errors..
>  >     >     >         > just can't run ipa-certupdate without the proper
>  >     >     kerberos creds..
>  >     >     >
>  >     >     >         By not recognizing your password I assume you mean
>  >     kinit is
>  >     >     >         failing? Is
>  >     >     >         the KDC running? I assume 389-ds is running? All
>  >     restarted
>  >     >     after
>  >     >     >         time
>  >     >     >         became stable in the past?
>  >     >     >
>  >     >     >         rob
>  >     >     >
>  >     >     >         >
>  >     >     >         > On Thu, Apr 13, 2023 at 12:51 PM Justen Long
>  >     >     >         <mr.justenlong(a)gmail.com
>  >     <mailto:[e-mail]> <mailto:[e-mail]
>  >     <mailto:[e-mail]>>
>  >     >     <mailto:[e-mail]
>  >     <mailto:[e-mail]> <mailto:[e-mail]
>  >     <mailto:[e-mail]>>>
>  >     >     >         > <mailto:[e-mail]
>  >     <mailto:[e-mail]>
>  >     >     <mailto:[e-mail] <mailto:
>  mr.justenlong(a)gmail.com&gt;&gt;
>  >     >     >         <mailto:[e-mail]
>  >     <mailto:[e-mail]>
>  >     >     <mailto:[e-mail]
>  >     <mailto:[e-mail]>>>>> wrote:
>  >     >     >         >
>  >     >     >         >     Following up, I see the date command just
>  >     changed it
>  >     >     >         momentarily...
>  >     >     >         >     using timedatectl and will report back.
>  >     >     >         >
>  >     >     >         >     On Thu, Apr 13, 2023 at 12:31 PM Justen Long
>  >     >     >         >     <mr.justenlong(a)gmail.com
>  >     <mailto:[e-mail]>
>  >     >     <mailto:[e-mail]
>  >     <mailto:[e-mail]>> <mailto:[e-mail]
>  >     <mailto:[e-mail]>
>  >     >     <mailto:[e-mail] <mailto:
>  mr.justenlong(a)gmail.com&gt;&gt;&gt;
>  >     >     >         <mailto:[e-mail]
>  >     <mailto:[e-mail]>
>  >     >     <mailto:[e-mail] <mailto:
>  mr.justenlong(a)gmail.com&gt;&gt;
>  >     >     >         <mailto:[e-mail]
>  >     <mailto:[e-mail]>
>  >     >     <mailto:[e-mail]
>  >     <mailto:[e-mail]>>>>> wrote:
>  >     >     >         >
>  >     >     >         >         Rob,
>  >     >     >         >
>  >     >     >         >         I entered 'date --date="7 April
> 2023",
>  >     verified it
>  >     >     >         updated the
>  >     >     >         >         system time appropriately. Restarted
>  dirsrv,
>  >     >     ipa-custodia,
>  >     >     >         >         ipa-otpd, httpd.. krb5kdc and kadmin
>  >     failed. Still,
>  >     >     >         tried to
>  >     >     >         >         send ipa cert-update, and it popped the
>  >     same SSL
>  >     >     >         Certificate
>  >     >     >         >         Verify Failed error.
>  >     >     >         >
>  >     >     >         >         On Thu, Apr 13, 2023 at 11:32 AM Rob
>  >     Crittenden
>  >     >     >         >         <rcritten(a)redhat.com
>  >     <mailto:[e-mail]>
>  >     >     <mailto:[e-mail] <mailto:[e-mail]>>
>  >     <mailto:[e-mail] <mailto:[e-mail]>
>  >     >     <mailto:[e-mail]
> <mailto:[e-mail]>>>
>  >     >     >         <mailto:[e-mail]
>  >     <mailto:[e-mail]> <mailto:[e-mail]
>  >     <mailto:[e-mail]>>
>  >     >     <mailto:[e-mail] <mailto:[e-mail]>
>  >     <mailto:[e-mail]
> <mailto:[e-mail]>>>>> wrote:
>  >     >     >         >
>  >     >     >         >             Justen Long wrote:
>  >     >     >         >             > Additionally, is there any way to
>  >     force the CA
>  >     >     >         cert update
>  >     >     >         >             to be
>  >     >     >         >             > recognized? When I run it to update
>  >     the CA
>  >     >     chain,
>  >     >     >         >             everything is
>  >     >     >         >             > verified.. but /etc/ipa/ca.crt
>  >     didn't reflect
>  >     >     >         the change..
>  >     >     >         >             so I manually
>  >     >     >         >             > populated it by copying over the
>  guts of
>  >     >     the CA
>  >     >     >         bundle to the
>  >     >     >         >             > /etc/ipa/ca.crt before trying to
>  install
>  >     >     the new
>  >     >     >         server
>  >     >     >         >             cert and it
>  >     >     >         >             > still doesn't recognize it as
>  >     trusted although
>  >     >     >         the issuer
>  >     >     >         >             is the same
>  >     >     >         >             > and within the CA bundle.
>  >     >     >         >
>  >     >     >         >             This is going to sound weird, but
> I'd
>  just
>  >     >     go back
>  >     >     >         in time
>  >     >     >         >             to April 10,
>  >     >     >         >             restart all services but ntp (which
>  will
>  >     >     reset the
>  >     >     >         time) and
>  >     >     >         >             then the
>  >     >     >         >             commands should work. Once the certs
>  are
>  >     >     updated and
>  >     >     >         >             working, return to
>  >     >     >         >             present time.
>  >     >     >         >
>  >     >     >         >             rob
>  >     >     >         >
>  >     >     >         >             >
>  >     >     >         >             > On Thu, Apr 13, 2023 at 6:20 AM
>  >     Justen Long
>  >     >     >         >             <mr.justenlong(a)gmail.com
>  >     <mailto:[e-mail]>
>  >     >     <mailto:[e-mail] <mailto:
>  mr.justenlong(a)gmail.com&gt;&gt;
>  >     >     >         <mailto:[e-mail]
>  >     <mailto:[e-mail]>
>  >     >     <mailto:[e-mail]
>  >     <mailto:[e-mail]>>>
> <mailto:[e-mail]
>  >     <mailto:[e-mail]>
>  >     >     <mailto:[e-mail] <mailto:
>  mr.justenlong(a)gmail.com&gt;&gt;
>  >     >     >         <mailto:[e-mail]
>  >     <mailto:[e-mail]>
>  >     >     <mailto:[e-mail]
>  >     <mailto:[e-mail]>>>>
>  >     >     >         >             > <mailto:[e-mail]
>  >     <mailto:[e-mail]>
>  >     >     <mailto:[e-mail] <mailto:
>  mr.justenlong(a)gmail.com&gt;&gt;
>  >     >     >         <mailto:[e-mail]
>  >     <mailto:[e-mail]>
>  >     >     <mailto:[e-mail] <mailto:
>  mr.justenlong(a)gmail.com&gt;&gt;&gt;
>  >     >     >         >             <mailto:[e-mail]
>  >     <mailto:[e-mail]>
>  >     >     <mailto:[e-mail] <mailto:
>  mr.justenlong(a)gmail.com&gt;&gt;
>  >     >     >         <mailto:[e-mail]
>  >     <mailto:[e-mail]>
>  >     >     <mailto:[e-mail]
>  >     <mailto:[e-mail]>>>>>> wrote:
>  >     >     >         >             >
>  >     >     >         >             >     Rob,
>  >     >     >         >             >
>  >     >     >         >             >     Apologies for the delay in
>  >     response. Once
>  >     >     >         I'm home, I
>  >     >     >         >             don't have
>  >     >     >         >             >     access to the information
> readily
>  >     >     available
>  >     >     >         to respond
>  >     >     >         >             with. Here is
>  >     >     >         >             >     the information you requested:
>  >     >     >         >             >
>  >     >     >         >             >     The version of IPA we are using
>  is
>  >     >     4.6.8, rpm
>  >     >     >         >             specifically for us is
>  >     >     >         >             >
>  >     >      ipa-server-4.6.8-5.el7.centos.12.x86_64 and
>  >     >     >         we are
>  >     >     >         >             using CentOS 7.9
>  >     >     >         >             >     currently with plans to move to
>  >     RHEL9
>  >     >     within
>  >     >     >         the next
>  >     >     >         >             year or so.
>  >     >     >         >             >
>  >     >     >         >             >     Unfortunately, 'ipa
> config-show'
>  >     doesn't
>  >     >     >         work. It
>  >     >     >         >             populates the same
>  >     >     >         >             >     error stating "ipa: ERROR:
> cannot
>  >     >     connect to
>  >     >     >         >             >    
> 'https://ipaServer/ipa/json':
>  [SSL:
>  >     >     >         >             CERTIFICATE_VERIFY_FAILED]
>  >     >     >         >             >     certificate verify failed
>  >     (_ssl.c:618).
>  >     >     >         >
>  >     >     >         >             The smack heard around the world was
>  >     my head
>  >     >     >         hitting my
>  >     >     >         >             desk. Of course
>  >     >     >         >             this command failed.
>  >     >     >         >
>  >     >     >         >             >
>  >     >     >         >             >     We have ~50 hosts connected via
>  IPA.
>  >     >     We have
>  >     >     >         two IPA
>  >     >     >         >             servers, one as
>  >     >     >         >             >     a replica of the other.
>  >     >     >         >             >
>  >     >     >         >             >     'getcert list' only
> shows 1
>  >     certificate.
>  >     >     >         It's state is
>  >     >     >         >             "MONITORING"
>  >     >     >         >             >     and seems related to kerberos.
>  >     >     >         >             >
>  >     >     >         >             >     As far as I know, we don't
> use
>  IPA
>  >     >     CA-issued
>  >     >     >         >             certificates. I recall
>  >     >     >         >             >     seeing errors yesterday stating
>  >     CA wasn't
>  >     >     >         enabled on
>  >     >     >         >             our servers. We
>  >     >     >         >             >     have always used 3rd party CAs
>  to my
>  >     >     knowledge.
>  >     >     >         >             >
>  >     >     >         >             >     -justen
>  >     >     >         >             >
>  >     >     >         >             >     On Wed, Apr 12, 2023 at 2:42 PM
>  Rob
>  >     >     Crittenden
>  >     >     >         >             <rcritten(a)redhat.com
>  >     <mailto:[e-mail]>
>  >     >     <mailto:[e-mail] <mailto:[e-mail]>>
>  >     <mailto:[e-mail] <mailto:[e-mail]>
>  >     >     <mailto:[e-mail]
> <mailto:[e-mail]>>>
>  >     >     >         <mailto:[e-mail]
>  >     <mailto:[e-mail]> <mailto:[e-mail]
>  >     <mailto:[e-mail]>>
>  >     >     <mailto:[e-mail] <mailto:[e-mail]>
>  >     <mailto:[e-mail] <mailto:[e-mail]>>>>
>  >     >     >         >             >     <mailto:[e-mail]
>  >     <mailto:[e-mail]>
>  >     >     <mailto:[e-mail] <mailto:[e-mail]>>
>  >     >     >         <mailto:[e-mail]
>  >     <mailto:[e-mail]> <mailto:[e-mail]
>  >     <mailto:[e-mail]>>>
>  >     >     >         >             <mailto:[e-mail]
>  >     <mailto:[e-mail]>
>  >     >     <mailto:[e-mail] <mailto:[e-mail]>>
>  >     >     >         <mailto:[e-mail] <mailto:
>  rcritten(a)redhat.com&gt;
>  >     >     <mailto:[e-mail]
> <mailto:[e-mail]>>>>>>
>  >     wrote:
>  >     >     >         >             >
>  >     >     >         >             >         Justen Long via
>  >     FreeIPA-users wrote:
>  >     >     >         >             >         > Thanks in advance for
> your
>  >     replies..
>  >     >     >         I've spent
>  >     >     >         >             7 hours
>  >     >     >         >             >         looking through posts here
>  >     and trying
>  >     >     >         >             everything... I'm stuck.
>  >     >     >         >             >         >
>  >     >     >         >             >         > Background: I am a
> System
>  >     >     >         Administrator in a closed,
>  >     >     >         >             >         classified environment.
>  >     >     Unfortunately, I
>  >     >     >         cannot
>  >     >     >         >             post logging
>  >     >     >         >             >         here, but I can refer to
>  them as
>  >     >     needed.
>  >     >     >         >             >         >
>  >     >     >         >             >         > I inherited this
> system
>  from
>  >     >     someone who
>  >     >     >         >             departed the program
>  >     >     >         >             >         a year or so ago. Fast
>  >     forward to
>  >     >     today, the
>  >     >     >         >             server certs
>  >     >     >         >             >         expired yesterday.
>  >     Admittedly, I'm
>  >     >     >         unfamiliar (or
>  >     >     >         >             was) with the
>  >     >     >         >             >         certificate update process
>  >     for IPA
>  >     >     >         servers. On a
>  >     >     >         >             typical server,
>  >     >     >         >             >         we replace the old cert and
>  >     >     restart the
>  >     >     >         httpd
>  >     >     >         >             services; however,
>  >     >     >         >             >         I realize this cannot work
>  >     with IPA
>  >     >     >         servers now.
>  >     >     >         >             >         >
>  >     >     >         >             >         > Additionally to all of
>  >     this, the
>  >     >     CA chain
>  >     >     >         >             updated 6 months ago.
>  >     >     >         >             >         >
>  >     >     >         >             >         > I ran ipa-cacert-manage
> to
>  >     >     update the
>  >     >     >         CA chain.
>  >     >     >         >             When trying to
>  >     >     >         >             >         run ipa-certupdate, I
>  received
>  >     >     errors for an
>  >     >     >         >             invalid server
>  >     >     >         >             >         certificate (it expired on
>  >     11 April
>  >     >     >         2023). It
>  >     >     >         >             simply won't
>  >     >     >         >             >         connect to the web server.
>  HTTPD
>  >     >     failed
>  >     >     >         as well,
>  >     >     >         >             so I had to add
>  >     >     >         >             >         "NSSEnforceValidCerts
> off"
>  >     to the
>  >     >     >         nss.conf file
>  >     >     >         >             for HTTPD to
>  >     >     >         >             >         start. Still, no dice.
>  >     >     >         >             >         >
>  >     >     >         >             >         > I've ran
>  >     ipa-server-certinstall for
>  >     >     >         the new
>  >     >     >         >             cert/key as well,
>  >     >     >         >             >         and it fails saying its not
>  >     >     trusted ("Peer's
>  >     >     >         >             certificate issuer
>  >     >     >         >             >         is not trusted [certutil:
>  >     >     certificate is
>  >     >     >         invalid:
>  >     >     >         >             Peer's
>  >     >     >         >             >         Certificate issuer is not
>  >     recognized]
>  >     >     >         Please run
>  >     >     >         >             >         ipa-cacert-manage install
> and
>  >     >     >         ipa-certupdate to
>  >     >     >         >             install the CA
>  >     >     >         >             >         certificate.... which, as
>  >     reported
>  >     >     >         above, can't
>  >     >     >         >             complete.
>  >     >     >         >             >         >
>  >     >     >         >             >         > I'm at a total
> loss
>  >     here... and
>  >     >     really
>  >     >     >         >             struggling being new to
>  >     >     >         >             >         all this and trying my best
>  >     to keep it
>  >     >     >         afloat. Any
>  >     >     >         >             help would be
>  >     >     >         >             >         GREATLY appreciated!
>  >     >     >         >             >
>  >     >     >         >             >         Let's gather some
>  >     information first.
>  >     >     >         >             >
>  >     >     >         >             >         What version of IPA is
> this,
>  >     on what
>  >     >     >         distribution?
>  >     >     >         >             >
>  >     >     >         >             >         IPA designates one server
> to
>  be
>  >     >     the "renewal
>  >     >     >         >             master" which
>  >     >     >         >             >         handles the
>  >     >     >         >             >         renewals. The output of
> `ipa
>  >     >     >         config-show` should
>  >     >     >         >             tell you
>  >     >     >         >             >         (depending on
>  >     >     >         >             >         version). That's the
> server
>  you
>  >     >     want to
>  >     >     >         work on.
>  >     >     >         >             >
>  >     >     >         >             >         How many servers in your
>  >     topology and
>  >     >     >         how many
>  >     >     >         >             have a CA installed?
>  >     >     >         >             >
>  >     >     >         >             >         Does `getcert list` show a
>  >     set of 8-10
>  >     >     >         tracked
>  >     >     >         >             certificates?
>  >     >     >         >             >         What are
>  >     >     >         >             >         the states?
>  >     >     >         >             >
>  >     >     >         >             >         You mention
>  >     >     ipa-server-certinstall. Are
>  >     >     >         you using
>  >     >     >         >             3rd party
>  >     >     >        ��>             >         certificates
>  >     >     >         >             >         in addition to IPA
> CA-issued
>  >     >     >         certificates or was
>  >     >     >         >             that just an
>  >     >     >         >             >         attempt to
>  >     >     >         >             >         get things working again?
>  >     >     >         >             >
>  >     >     >         >             >         rob
>  >     >     >         >             >
>  >     >     >         >
>  >     >     >
>  >     >
>  >
>
>