Quick update and answers to your questions.

We have two IPA servers, both masters, hiipa03 and hiipa04.

hiipa04, I was able to set the time back, run the ipa-ca-cert-manage and update the CA (minimally, but enough to accept the new cert).. ipa-certupdate ran fine on it. Then, ran ipa-server-certinstall on it, and it took. Website loads, can log into it, do some user management stuff.. can't run "ipa-replica-manage list" as it kicks an error for sslv3 handshake failure.. but was trying that to remove hiipa03 and copy 04 to 03 somehow, maybe? 

hiipa03 is still giving me grief. Set the time using timedatectl, verified ntp is off and 'date' reports properly.  I had tried to follow this: https://computingforgeeks.com/reset-freeipa-admin-password-as-root-user-on-linux/ when it wasn't accepting the admin password.. it failed saying object not found. So, I try 'kinit admin' again, with the new password I tried.. says its expired. Type it in, type in a new password.. and then it failed saying "kinit: Password change failed while getting initial credentials"

On Thu, Apr 13, 2023 at 1:40 PM Rob Crittenden <rcritten@redhat.com> wrote:
Justen Long wrote:
> I'm getting closer... it's not recognizing my admin password for IPA, or
> for my personal account with admin rights now.. but no more SSL errors..
> just can't run ipa-certupdate without the proper kerberos creds..

By not recognizing your password I assume you mean kinit is failing? Is
the KDC running? I assume 389-ds is running? All restarted after time
became stable in the past?

rob

>
> On Thu, Apr 13, 2023 at 12:51 PM Justen Long <mr.justenlong@gmail.com
> <mailto:mr.justenlong@gmail.com>> wrote:
>
>     Following up, I see the date command just changed it momentarily...
>     using timedatectl and will report back.
>
>     On Thu, Apr 13, 2023 at 12:31 PM Justen Long
>     <mr.justenlong@gmail.com <mailto:mr.justenlong@gmail.com>> wrote:
>
>         Rob,
>
>         I entered 'date --date="7 April 2023", verified it updated the
>         system time appropriately. Restarted dirsrv, ipa-custodia,
>         ipa-otpd, httpd.. krb5kdc and kadmin failed. Still, tried to
>         send ipa cert-update, and it popped the same SSL Certificate
>         Verify Failed error.
>
>         On Thu, Apr 13, 2023 at 11:32 AM Rob Crittenden
>         <rcritten@redhat.com <mailto:rcritten@redhat.com>> wrote:
>
>             Justen Long wrote:
>             > Additionally, is there any way to force the CA cert update
>             to be
>             > recognized? When I run it to update the CA chain,
>             everything is
>             > verified.. but /etc/ipa/ca.crt didn't reflect the change..
>             so I manually
>             > populated it by copying over the guts of the CA bundle to the
>             > /etc/ipa/ca.crt before trying to install the new server
>             cert and it
>             > still doesn't recognize it as trusted although the issuer
>             is the same
>             > and within the CA bundle.
>
>             This is going to sound weird, but I'd just go back in time
>             to April 10,
>             restart all services but ntp (which will reset the time) and
>             then the
>             commands should work. Once the certs are updated and
>             working, return to
>             present time.
>
>             rob
>
>             >
>             > On Thu, Apr 13, 2023 at 6:20 AM Justen Long
>             <mr.justenlong@gmail.com <mailto:mr.justenlong@gmail.com>
>             > <mailto:mr.justenlong@gmail.com
>             <mailto:mr.justenlong@gmail.com>>> wrote:
>             >
>             >     Rob,
>             >
>             >     Apologies for the delay in response. Once I'm home, I
>             don't have
>             >     access to the information readily available to respond
>             with. Here is
>             >     the information you requested:
>             >
>             >     The version of IPA we are using is 4.6.8, rpm
>             specifically for us is
>             >     ipa-server-4.6.8-5.el7.centos.12.x86_64 and we are
>             using CentOS 7.9
>             >     currently with plans to move to RHEL9 within the next
>             year or so.
>             >
>             >     Unfortunately, 'ipa config-show' doesn't work. It
>             populates the same
>             >     error stating "ipa: ERROR: cannot connect to
>             >     'https://ipaServer/ipa/json': [SSL:
>             CERTIFICATE_VERIFY_FAILED]
>             >     certificate verify failed (_ssl.c:618).
>
>             The smack heard around the world was my head hitting my
>             desk. Of course
>             this command failed.
>
>             >
>             >     We have ~50 hosts connected via IPA. We have two IPA
>             servers, one as
>             >     a replica of the other.
>             >
>             >     'getcert list' only shows 1 certificate. It's state is
>             "MONITORING"
>             >     and seems related to kerberos.
>             >
>             >     As far as I know, we don't use IPA CA-issued
>             certificates. I recall
>             >     seeing errors yesterday stating CA wasn't enabled on
>             our servers. We
>             >     have always used 3rd party CAs to my knowledge.
>             >
>             >     -justen
>             >
>             >     On Wed, Apr 12, 2023 at 2:42 PM Rob Crittenden
>             <rcritten@redhat.com <mailto:rcritten@redhat.com>
>             >     <mailto:rcritten@redhat.com
>             <mailto:rcritten@redhat.com>>> wrote:
>             >
>             >         Justen Long via FreeIPA-users wrote:
>             >         > Thanks in advance for your replies.. I've spent
>             7 hours
>             >         looking through posts here and trying
>             everything... I'm stuck.
>             >         >
>             >         > Background: I am a System Administrator in a closed,
>             >         classified environment. Unfortunately, I cannot
>             post logging
>             >         here, but I can refer to them as needed.
>             >         >
>             >         > I inherited this system from someone who
>             departed the program
>             >         a year or so ago. Fast forward to today, the
>             server certs
>             >         expired yesterday. Admittedly, I'm unfamiliar (or
>             was) with the
>             >         certificate update process for IPA servers. On a
>             typical server,
>             >         we replace the old cert and restart the httpd
>             services; however,
>             >         I realize this cannot work with IPA servers now.
>             >         >
>             >         > Additionally to all of this, the CA chain
>             updated 6 months ago.
>             >         >
>             >         > I ran ipa-cacert-manage to update the CA chain.
>             When trying to
>             >         run ipa-certupdate, I received errors for an
>             invalid server
>             >         certificate (it expired on 11 April 2023). It
>             simply won't
>             >         connect to the web server. HTTPD failed as well,
>             so I had to add
>             >         "NSSEnforceValidCerts off" to the nss.conf file
>             for HTTPD to
>             >         start. Still, no dice.
>             >         >
>             >         > I've ran ipa-server-certinstall for the new
>             cert/key as well,
>             >         and it fails saying its not trusted ("Peer's
>             certificate issuer
>             >         is not trusted [certutil: certificate is invalid:
>             Peer's
>             >         Certificate issuer is not recognized] Please run
>             >         ipa-cacert-manage install and ipa-certupdate to
>             install the CA
>             >         certificate.... which, as reported above, can't
>             complete.
>             >         >
>             >         > I'm at a total loss here... and really
>             struggling being new to
>             >         all this and trying my best to keep it afloat. Any
>             help would be
>             >         GREATLY appreciated!
>             >
>             >         Let's gather some information first.
>             >
>             >         What version of IPA is this, on what distribution?
>             >
>             >         IPA designates one server to be the "renewal
>             master" which
>             >         handles the
>             >         renewals. The output of `ipa config-show` should
>             tell you
>             >         (depending on
>             >         version). That's the server you want to work on.
>             >
>             >         How many servers in your topology and how many
>             have a CA installed?
>             >
>             >         Does `getcert list` show a set of 8-10 tracked
>             certificates?
>             >         What are
>             >         the states?
>             >
>             >         You mention ipa-server-certinstall. Are you using
>             3rd party
>             >         certificates
>             >         in addition to IPA CA-issued certificates or was
>             that just an
>             >         attempt to
>             >         get things working again?
>             >
>             >         rob
>             >
>