Rob,

Currently only hiipa03 and hiipa04 (these are the masters/IPA servers) would utilize the ipa-server-certinstall commands. The other servers don't use any HTTPD or anything like that, and have never had issues in the past simply replacing the cert file in /etc/pki/tls/certs and letting it ride.

So, with regard to the catch-22...

hiipa03 isn't communicating appropriately or something, in my opinion.  When attempting to run "ipa-certupdate" on it, it wants the Kerberos credentials (assuming 'kinit admin'). I run 'kinit admin" and enter the admin password that works on hiipa04, and it doesn't like it. I attempted to change that password using the long 'ldappasswd' command, but forgot to put dc=hilab,dc=ipa, and instead had put dc=hilab.ipa. This results in a failure, object not found.. (had a duh moment).. went to try the 'ldappasswd' command again, using dc=hilab,dc=ipa.. it then fails stating "Result: Operations error (1) \\ Additional info: Failed to update password".

So, I just, for s&g, run 'kinit admin' again. Prompts for password. Try the one that failed because of the wrong dc input, and it takes it, stating its expired. Try to change it, and get the same "Result: Operations error (1) \\ Additional info:" Failed to update password."

On the majority of the clients, when I enter "ipa-certupdate" it points to hiipa03, rather than hiipa04.. and fails because hiipa03 is still in shambles. I think if I can get hiipa03 back to operational, then everything else would work.

1) Is there some magical way to destroy the admin password (i.e. blank it out) so it prompts for a new one, or to replicate the admin stuff from hiipa04 to hiipa03 so that it all works?
2) If not, how can I possibly move forward? The only hold up is the admin password at this point.

On Fri, Apr 14, 2023 at 7:11 AM Rob Crittenden <rcritten@redhat.com> wrote:
Justen Long wrote:
> Rob,
>
> FreeIPA documentation stated that all clients must receive the
> ipa-updatecert command before installing the new server cert.. on the
> hiipa04 server I installed the cert in a hurry for validity sake. Does
> that mess with anything?

ipa-certupdate distributes the CA chain. If the chain didn't change then
it should be fine.

>
> Note: hiipa03 seems to be primary but they’re both masters. I can’t get
> the ipa-server-certinstall to take on this server stating it’s missing
> the whole chain.. but it worked just fine on hiipa04.. so that’s why I
> was trying to “kinit admin” on 03, to run ipa-updatecert or the
> ipa-cacert-manage there if it didn’t take it from 04, which I’m assuming
> it didn’t.

You could try going back in time and use ipa-cacert-manage install to
update the chain and then run ipa-server-certinstall.

But like I said before, if you're changing your chain AND replacing the
certs at the same time none of your clients are going to know what to do.

The process is:

1. ipa-cacert-manage install <new chain>
2. ipa-certupdate on ALL enrolled machines
3. Replace the server certs

If you deviate then you're caught in the catch-22 I mentioned. Clients
won't trust the updated chain in order to download the updated chain.
The aren't a lot of great options out of that. Unenrolling and
re-enrolling the client is the best way.

rob
>
> On Thu, Apr 13, 2023 at 2:41 PM Rob Crittenden <rcritten@redhat.com
> <mailto:rcritten@redhat.com>> wrote:
>
>     Justen Long wrote:
>     > One go back.. when I tried to run "ipa-certupdate" on the other hosts
>     > (clients), it points to hiipa03 and fails for SSL still.. so, hoping
>     > once I get the cert updated on THAT server, that all restores to its
>     > okay state.
>
>     You shouldn't need to go back for clients. Now that the server
>     certificates are valid they should just work. Unless you also changed
>     the CA chain in which case you're caught in a catch-22: you need to get
>     the updated chain from the server but you don't trust the chain of the
>     server.
>
>     rob
>
>     >
>     > On Thu, Apr 13, 2023 at 1:46 PM Justen Long
>     <mr.justenlong@gmail.com <mailto:mr.justenlong@gmail.com>
>     > <mailto:mr.justenlong@gmail.com <mailto:mr.justenlong@gmail.com>>>
>     wrote:
>     >
>     >     Quick update and answers to your questions.
>     >
>     >     We have two IPA servers, both masters, hiipa03 and hiipa04.
>     >
>     >     hiipa04, I was able to set the time back, run the
>     ipa-ca-cert-manage
>     >     and update the CA (minimally, but enough to accept the new cert)..
>     >     ipa-certupdate ran fine on it. Then, ran ipa-server-certinstall on
>     >     it, and it took. Website loads, can log into it, do some user
>     >     management stuff.. can't run "ipa-replica-manage list" as it kicks
>     >     an error for sslv3 handshake failure.. but was trying that to
>     remove
>     >     hiipa03 and copy 04 to 03 somehow, maybe? 
>     >
>     >     hiipa03 is still giving me grief. Set the time using timedatectl,
>     >     verified ntp is off and 'date' reports properly.  I had tried to
>     >     follow
>     >   
>      this: https://computingforgeeks.com/reset-freeipa-admin-password-as-root-user-on-linux/
>     >     when it wasn't accepting the admin password.. it failed saying
>     >     object not found. So, I try 'kinit admin' again, with the new
>     >     password I tried.. says its expired. Type it in, type in a new
>     >     password.. and then it failed saying "kinit: Password change
>     failed
>     >     while getting initial credentials"
>     >
>     >     On Thu, Apr 13, 2023 at 1:40 PM Rob Crittenden
>     <rcritten@redhat.com <mailto:rcritten@redhat.com>
>     >     <mailto:rcritten@redhat.com <mailto:rcritten@redhat.com>>> wrote:
>     >
>     >         Justen Long wrote:
>     >         > I'm getting closer... it's not recognizing my admin password
>     >         for IPA, or
>     >         > for my personal account with admin rights now.. but no more
>     >         SSL errors..
>     >         > just can't run ipa-certupdate without the proper
>     kerberos creds..
>     >
>     >         By not recognizing your password I assume you mean kinit is
>     >         failing? Is
>     >         the KDC running? I assume 389-ds is running? All restarted
>     after
>     >         time
>     >         became stable in the past?
>     >
>     >         rob
>     >
>     >         >
>     >         > On Thu, Apr 13, 2023 at 12:51 PM Justen Long
>     >         <mr.justenlong@gmail.com <mailto:mr.justenlong@gmail.com>
>     <mailto:mr.justenlong@gmail.com <mailto:mr.justenlong@gmail.com>>
>     >         > <mailto:mr.justenlong@gmail.com
>     <mailto:mr.justenlong@gmail.com>
>     >         <mailto:mr.justenlong@gmail.com
>     <mailto:mr.justenlong@gmail.com>>>> wrote:
>     >         >
>     >         >     Following up, I see the date command just changed it
>     >         momentarily...
>     >         >     using timedatectl and will report back.
>     >         >
>     >         >     On Thu, Apr 13, 2023 at 12:31 PM Justen Long
>     >         >     <mr.justenlong@gmail.com
>     <mailto:mr.justenlong@gmail.com> <mailto:mr.justenlong@gmail.com
>     <mailto:mr.justenlong@gmail.com>>
>     >         <mailto:mr.justenlong@gmail.com
>     <mailto:mr.justenlong@gmail.com>
>     >         <mailto:mr.justenlong@gmail.com
>     <mailto:mr.justenlong@gmail.com>>>> wrote:
>     >         >
>     >         >         Rob,
>     >         >
>     >         >         I entered 'date --date="7 April 2023", verified it
>     >         updated the
>     >         >         system time appropriately. Restarted dirsrv,
>     ipa-custodia,
>     >         >         ipa-otpd, httpd.. krb5kdc and kadmin failed. Still,
>     >         tried to
>     >         >         send ipa cert-update, and it popped the same SSL
>     >         Certificate
>     >         >         Verify Failed error.
>     >         >
>     >         >         On Thu, Apr 13, 2023 at 11:32 AM Rob Crittenden
>     >         >         <rcritten@redhat.com
>     <mailto:rcritten@redhat.com> <mailto:rcritten@redhat.com
>     <mailto:rcritten@redhat.com>>
>     >         <mailto:rcritten@redhat.com <mailto:rcritten@redhat.com>
>     <mailto:rcritten@redhat.com <mailto:rcritten@redhat.com>>>> wrote:
>     >         >
>     >         >             Justen Long wrote:
>     >         >             > Additionally, is there any way to force the CA
>     >         cert update
>     >         >             to be
>     >         >             > recognized? When I run it to update the CA
>     chain,
>     >         >             everything is
>     >         >             > verified.. but /etc/ipa/ca.crt didn't reflect
>     >         the change..
>     >         >             so I manually
>     >         >             > populated it by copying over the guts of
>     the CA
>     >         bundle to the
>     >         >             > /etc/ipa/ca.crt before trying to install
>     the new
>     >         server
>     >         >             cert and it
>     >         >             > still doesn't recognize it as trusted although
>     >         the issuer
>     >         >             is the same
>     >         >             > and within the CA bundle.
>     >         >
>     >         >             This is going to sound weird, but I'd just
>     go back
>     >         in time
>     >         >             to April 10,
>     >         >             restart all services but ntp (which will
>     reset the
>     >         time) and
>     >         >             then the
>     >         >             commands should work. Once the certs are
>     updated and
>     >         >             working, return to
>     >         >             present time.
>     >         >
>     >         >             rob
>     >         >
>     >         >             >
>     >         >             > On Thu, Apr 13, 2023 at 6:20 AM Justen Long
>     >         >             <mr.justenlong@gmail.com
>     <mailto:mr.justenlong@gmail.com>
>     >         <mailto:mr.justenlong@gmail.com
>     <mailto:mr.justenlong@gmail.com>> <mailto:mr.justenlong@gmail.com
>     <mailto:mr.justenlong@gmail.com>
>     >         <mailto:mr.justenlong@gmail.com
>     <mailto:mr.justenlong@gmail.com>>>
>     >         >             > <mailto:mr.justenlong@gmail.com
>     <mailto:mr.justenlong@gmail.com>
>     >         <mailto:mr.justenlong@gmail.com
>     <mailto:mr.justenlong@gmail.com>>
>     >         >             <mailto:mr.justenlong@gmail.com
>     <mailto:mr.justenlong@gmail.com>
>     >         <mailto:mr.justenlong@gmail.com
>     <mailto:mr.justenlong@gmail.com>>>>> wrote:
>     >         >             >
>     >         >             >     Rob,
>     >         >             >
>     >         >             >     Apologies for the delay in response. Once
>     >         I'm home, I
>     >         >             don't have
>     >         >             >     access to the information readily
>     available
>     >         to respond
>     >         >             with. Here is
>     >         >             >     the information you requested:
>     >         >             >
>     >         >             >     The version of IPA we are using is
>     4.6.8, rpm
>     >         >             specifically for us is
>     >         >             >   
>      ipa-server-4.6.8-5.el7.centos.12.x86_64 and
>     >         we are
>     >         >             using CentOS 7.9
>     >         >             >     currently with plans to move to RHEL9
>     within
>     >         the next
>     >         >             year or so.
>     >         >             >
>     >         >             >     Unfortunately, 'ipa config-show' doesn't
>     >         work. It
>     >         >             populates the same
>     >         >             >     error stating "ipa: ERROR: cannot
>     connect to
>     >         >             >     'https://ipaServer/ipa/json': [SSL:
>     >         >             CERTIFICATE_VERIFY_FAILED]
>     >         >             >     certificate verify failed (_ssl.c:618).
>     >         >
>     >         >             The smack heard around the world was my head
>     >         hitting my
>     >         >             desk. Of course
>     >         >             this command failed.
>     >         >
>     >         >             >
>     >         >             >     We have ~50 hosts connected via IPA.
>     We have
>     >         two IPA
>     >         >             servers, one as
>     >         >             >     a replica of the other.
>     >         >             >
>     >         >             >     'getcert list' only shows 1 certificate.
>     >         It's state is
>     >         >             "MONITORING"
>     >         >             >     and seems related to kerberos.
>     >         >             >
>     >         >             >     As far as I know, we don't use IPA
>     CA-issued
>     >         >             certificates. I recall
>     >         >             >     seeing errors yesterday stating CA wasn't
>     >         enabled on
>     >         >             our servers. We
>     >         >             >     have always used 3rd party CAs to my
>     knowledge.
>     >         >             >
>     >         >             >     -justen
>     >         >             >
>     >         >             >     On Wed, Apr 12, 2023 at 2:42 PM Rob
>     Crittenden
>     >         >             <rcritten@redhat.com
>     <mailto:rcritten@redhat.com> <mailto:rcritten@redhat.com
>     <mailto:rcritten@redhat.com>>
>     >         <mailto:rcritten@redhat.com <mailto:rcritten@redhat.com>
>     <mailto:rcritten@redhat.com <mailto:rcritten@redhat.com>>>
>     >         >             >     <mailto:rcritten@redhat.com
>     <mailto:rcritten@redhat.com>
>     >         <mailto:rcritten@redhat.com <mailto:rcritten@redhat.com>>
>     >         >             <mailto:rcritten@redhat.com
>     <mailto:rcritten@redhat.com>
>     >         <mailto:rcritten@redhat.com
>     <mailto:rcritten@redhat.com>>>>> wrote:
>     >         >             >
>     >         >             >         Justen Long via FreeIPA-users wrote:
>     >         >             >         > Thanks in advance for your replies..
>     >         I've spent
>     >         >             7 hours
>     >         >             >         looking through posts here and trying
>     >         >             everything... I'm stuck.
>     >         >             >         >
>     >         >             >         > Background: I am a System
>     >         Administrator in a closed,
>     >         >             >         classified environment.
>     Unfortunately, I
>     >         cannot
>     >         >             post logging
>     >         >             >         here, but I can refer to them as
>     needed.
>     >         >             >         >
>     >         >             >         > I inherited this system from
>     someone who
>     >         >             departed the program
>     >         >             >         a year or so ago. Fast forward to
>     today, the
>     >         >             server certs
>     >         >             >         expired yesterday. Admittedly, I'm
>     >         unfamiliar (or
>     >         >             was) with the
>     >         >             >         certificate update process for IPA
>     >         servers. On a
>     >         >             typical server,
>     >         >             >         we replace the old cert and
>     restart the
>     >         httpd
>     >         >             services; however,
>     >         >             >         I realize this cannot work with IPA
>     >         servers now.
>     >         >             >         >
>     >         >             >         > Additionally to all of this, the
>     CA chain
>     >         >             updated 6 months ago.
>     >         >             >         >
>     >         >             >         > I ran ipa-cacert-manage to
>     update the
>     >         CA chain.
>     >         >             When trying to
>     >         >             >         run ipa-certupdate, I received
>     errors for an
>     >         >             invalid server
>     >         >             >         certificate (it expired on 11 April
>     >         2023). It
>     >         >             simply won't
>     >         >             >         connect to the web server. HTTPD
>     failed
>     >         as well,
>     >         >             so I had to add
>     >         >             >         "NSSEnforceValidCerts off" to the
>     >         nss.conf file
>     >         >             for HTTPD to
>     >         >             >         start. Still, no dice.
>     >         >             >         >
>     >         >             >         > I've ran ipa-server-certinstall for
>     >         the new
>     >         >             cert/key as well,
>     >         >             >         and it fails saying its not
>     trusted ("Peer's
>     >         >             certificate issuer
>     >         >             >         is not trusted [certutil:
>     certificate is
>     >         invalid:
>     >         >             Peer's
>     >         >             >         Certificate issuer is not recognized]
>     >         Please run
>     >         >             >         ipa-cacert-manage install and
>     >         ipa-certupdate to
>     >         >             install the CA
>     >         >             >         certificate.... which, as reported
>     >         above, can't
>     >         >             complete.
>     >         >             >         >
>     >         >             >         > I'm at a total loss here... and
>     really
>     >         >             struggling being new to
>     >         >             >         all this and trying my best to keep it
>     >         afloat. Any
>     >         >             help would be
>     >         >             >         GREATLY appreciated!
>     >         >             >
>     >         >             >         Let's gather some information first.
>     >         >             >
>     >         >             >         What version of IPA is this, on what
>     >         distribution?
>     >         >             >
>     >         >             >         IPA designates one server to be
>     the "renewal
>     >         >             master" which
>     >         >             >         handles the
>     >         >             >         renewals. The output of `ipa
>     >         config-show` should
>     >         >             tell you
>     >         >             >         (depending on
>     >         >             >         version). That's the server you
>     want to
>     >         work on.
>     >         >             >
>     >         >             >         How many servers in your topology and
>     >         how many
>     >         >             have a CA installed?
>     >         >             >
>     >         >             >         Does `getcert list` show a set of 8-10
>     >         tracked
>     >         >             certificates?
>     >         >             >         What are
>     >         >             >         the states?
>     >         >             >
>     >         >             >         You mention
>     ipa-server-certinstall. Are
>     >         you using
>     >         >             3rd party
>     >         >             >         certificates
>     >         >             >         in addition to IPA CA-issued
>     >         certificates or was
>     >         >             that just an
>     >         >             >         attempt to
>     >         >             >         get things working again?
>     >         >             >
>     >         >             >         rob
>     >         >             >
>     >         >
>     >
>