On Fri, Apr 14, 2023 at 11:07 AM Rob Crittenden <rcritten@redhat.com> wrote:

Justen Long wrote:
> Rob,
>
> Currently only hiipa03 and hiipa04 (these are the masters/IPA servers)
> would utilize the ipa-server-certinstall commands. The other servers
> don't use any HTTPD or anything like that, and have never had issues in
> the past simply replacing the cert file in /etc/pki/tls/certs and
> letting it ride.
>
> So, with regard to the catch-22...
>
> hiipa03 isn't communicating appropriately or something, in my opinion.
> When attempting to run "ipa-certupdate" on it, it wants the Kerberos
> credentials (assuming 'kinit admin'). I run 'kinit admin" and enter the
> admin password that works on hiipa04, and it doesn't like it. I
> attempted to change that password using the long 'ldappasswd' command,
> but forgot to put dc=hilab,dc=ipa, and instead had put dc=hilab.ipa.
> This results in a failure, object not found.. (had a duh moment).. went
> to try the 'ldappasswd' command again, using dc=hilab,dc=ipa.. it then
> fails stating "Result: Operations error (1) \\ Additional info: Failed
> to update password".
>
> So, I just, for s&g, run 'kinit admin' again. Prompts for password. Try
> the one that failed because of the wrong dc input, and it takes it,
> stating its expired. Try to change it, and get the same "Result:
> Operations error (1) \\ Additional info:" Failed to update password."
>
> On the majority of the clients, when I enter "ipa-certupdate" it points
> to hiipa03, rather than hiipa04.. and fails because hiipa03 is still in
> shambles. I think if I can get hiipa03 back to operational, then
> everything else would work.
>
> 1) Is there some magical way to destroy the admin password (i.e. blank
> it out) so it prompts for a new one, or to replicate the admin stuff
> from hiipa04 to hiipa03 so that it all works?
> 2) If not, how can I possibly move forward? The only hold up is the
> admin password at this point.

It's hard to say without any logs. It sure seems like an issue with the
389 server on 03. It isn't clear what you've already done, like
restarting services. I assume you got the certificates replaced there as
well?

But if passwords can't be set that points to something in 389. The
errors log might contain information.

The hardcore option would be to uninstall IPA from 03 and install it as
a new replica from scratch. This assumes that 04 really is working properly.

rob

>
> On Fri, Apr 14, 2023 at 7:11 AM Rob Crittenden <rcritten@redhat.com
> <mailto:rcritten@redhat.com>> wrote:
>
> Justen Long wrote:
> > Rob,
> >
> > FreeIPA documentation stated that all clients must receive the
> > ipa-updatecert command before installing the new server cert.. on the
> > hiipa04 server I installed the cert in a hurry for validity sake. Does
> > that mess with anything?
>
> ipa-certupdate distributes the CA chain. If the chain didn't change then
> it should be fine.
>
> >
> > Note: hiipa03 seems to be primary but they’re both masters. I
> can’t get
> > the ipa-server-certinstall to take on this server stating it’s missing
> > the whole chain.. but it worked just fine on hiipa04.. so that’s why I
> > was trying to “kinit admin” on 03, to run ipa-updatecert or the
> > ipa-cacert-manage there if it didn’t take it from 04, which I’m
> assuming
> > it didn’t.
>
> You could try going back in time and use ipa-cacert-manage install to
> update the chain and then run ipa-server-certinstall.
>
> But like I said before, if you're changing your chain AND replacing the
> certs at the same time none of your clients are going to know what
> to do.
>
> The process is:
>
> 1. ipa-cacert-manage install <new chain>
> 2. ipa-certupdate on ALL enrolled machines
> 3. Replace the server certs
>
> If you deviate then you're caught in the catch-22 I mentioned. Clients
> won't trust the updated chain in order to download the updated chain.
> The aren't a lot of great options out of that. Unenrolling and
> re-enrolling the client is the best way.
>
> rob
> >
> > On Thu, Apr 13, 2023 at 2:41 PM Rob Crittenden
> <rcritten@redhat.com <mailto:rcritten@redhat.com>
> > <mailto:rcritten@redhat.com <mailto:rcritten@redhat.com>>> wrote:
> >
> > Justen Long wrote:
> > > One go back.. when I tried to run "ipa-certupdate" on the
> other hosts
> > > (clients), it points to hiipa03 and fails for SSL still..
> so, hoping
> > > once I get the cert updated on THAT server, that all
> restores to its
> > > okay state.
> >
> > You shouldn't need to go back for clients. Now that the server
> > certificates are valid they should just work. Unless you also
> changed
> > the CA chain in which case you're caught in a catch-22: you
> need to get
> > the updated chain from the server but you don't trust the
> chain of the
> > server.
> >
> > rob
> >
> > >
> > > On Thu, Apr 13, 2023 at 1:46 PM Justen Long
> > <mr.justenlong@gmail.com <mailto:mr.justenlong@gmail.com>
> <mailto:mr.justenlong@gmail.com <mailto:mr.justenlong@gmail.com>>
> > > <mailto:mr.justenlong@gmail.com
> <mailto:mr.justenlong@gmail.com> <mailto:mr.justenlong@gmail.com
> <mailto:mr.justenlong@gmail.com>>>>
> > wrote:
> > >
> > > Quick update and answers to your questions.
> > >
> > > We have two IPA servers, both masters, hiipa03 and hiipa04.
> > >
> > > hiipa04, I was able to set the time back, run the
> > ipa-ca-cert-manage
> > > and update the CA (minimally, but enough to accept the
> new cert)..
> > > ipa-certupdate ran fine on it. Then, ran
> ipa-server-certinstall on
> > > it, and it took. Website loads, can log into it, do some
> user
> > > management stuff.. can't run "ipa-replica-manage list"
> as it kicks
> > > an error for sslv3 handshake failure.. but was trying
> that to
> > remove
> > > hiipa03 and copy 04 to 03 somehow, maybe?
> > >
> > > hiipa03 is still giving me grief. Set the time using
> timedatectl,
> > > verified ntp is off and 'date' reports properly. I had
> tried to
> > > follow
> > >
> >
> this: https://computingforgeeks.com/reset-freeipa-admin-password-as-root-user-on-linux/
> > > when it wasn't accepting the admin password.. it failed
> saying
> > > object not found. So, I try 'kinit admin' again, with
> the new
> > > password I tried.. says its expired. Type it in, type in
> a new
> > > password.. and then it failed saying "kinit: Password change
> > failed
> > > while getting initial credentials"
> > >
> > > On Thu, Apr 13, 2023 at 1:40 PM Rob Crittenden
> > <rcritten@redhat.com <mailto:rcritten@redhat.com>
> <mailto:rcritten@redhat.com <mailto:rcritten@redhat.com>>
> > > <mailto:rcritten@redhat.com <mailto:rcritten@redhat.com>
> <mailto:rcritten@redhat.com <mailto:rcritten@redhat.com>>>> wrote:
> > >
> > > Justen Long wrote:
> > > > I'm getting closer... it's not recognizing my
> admin password
> > > for IPA, or
> > > > for my personal account with admin rights now..
> but no more
> > > SSL errors..
> > > > just can't run ipa-certupdate without the proper
> > kerberos creds..
> > >
> > > By not recognizing your password I assume you mean
> kinit is
> > > failing? Is
> > > the KDC running? I assume 389-ds is running? All
> restarted
> > after
> > > time
> > > became stable in the past?
> > >
> > > rob
> > >
> > > >
> > > > On Thu, Apr 13, 2023 at 12:51 PM Justen Long
> > > <mr.justenlong@gmail.com
> <mailto:mr.justenlong@gmail.com> <mailto:mr.justenlong@gmail.com
> <mailto:mr.justenlong@gmail.com>>
> > <mailto:mr.justenlong@gmail.com
> <mailto:mr.justenlong@gmail.com> <mailto:mr.justenlong@gmail.com
> <mailto:mr.justenlong@gmail.com>>>
> > > > <mailto:mr.justenlong@gmail.com
> <mailto:mr.justenlong@gmail.com>
> > <mailto:mr.justenlong@gmail.com <mailto:mr.justenlong@gmail.com>>
> > > <mailto:mr.justenlong@gmail.com
> <mailto:mr.justenlong@gmail.com>
> > <mailto:mr.justenlong@gmail.com
> <mailto:mr.justenlong@gmail.com>>>>> wrote:
> > > >
> > > > Following up, I see the date command just
> changed it
> > > momentarily...
> > > > using timedatectl and will report back.
> > > >
> > > > On Thu, Apr 13, 2023 at 12:31 PM Justen Long
> > > > <mr.justenlong@gmail.com
> <mailto:mr.justenlong@gmail.com>
> > <mailto:mr.justenlong@gmail.com
> <mailto:mr.justenlong@gmail.com>> <mailto:mr.justenlong@gmail.com
> <mailto:mr.justenlong@gmail.com>
> > <mailto:mr.justenlong@gmail.com <mailto:mr.justenlong@gmail.com>>>
> > > <mailto:mr.justenlong@gmail.com
> <mailto:mr.justenlong@gmail.com>
> > <mailto:mr.justenlong@gmail.com <mailto:mr.justenlong@gmail.com>>
> > > <mailto:mr.justenlong@gmail.com
> <mailto:mr.justenlong@gmail.com>
> > <mailto:mr.justenlong@gmail.com
> <mailto:mr.justenlong@gmail.com>>>>> wrote:
> > > >
> > > > Rob,
> > > >
> > > > I entered 'date --date="7 April 2023",
> verified it
> > > updated the
> > > > system time appropriately. Restarted dirsrv,
> > ipa-custodia,
> > > > ipa-otpd, httpd.. krb5kdc and kadmin
> failed. Still,
> > > tried to
> > > > send ipa cert-update, and it popped the
> same SSL
> > > Certificate
> > > > Verify Failed error.
> > > >
> > > > On Thu, Apr 13, 2023 at 11:32 AM Rob
> Crittenden
> > > > <rcritten@redhat.com
> <mailto:rcritten@redhat.com>
> > <mailto:rcritten@redhat.com <mailto:rcritten@redhat.com>>
> <mailto:rcritten@redhat.com <mailto:rcritten@redhat.com>
> > <mailto:rcritten@redhat.com <mailto:rcritten@redhat.com>>>
> > > <mailto:rcritten@redhat.com
> <mailto:rcritten@redhat.com> <mailto:rcritten@redhat.com
> <mailto:rcritten@redhat.com>>
> > <mailto:rcritten@redhat.com <mailto:rcritten@redhat.com>
> <mailto:rcritten@redhat.com <mailto:rcritten@redhat.com>>>>> wrote:
> > > >
> > > > Justen Long wrote:
> > > > > Additionally, is there any way to
> force the CA
> > > cert update
> > > > to be
> > > > > recognized? When I run it to update
> the CA
> > chain,
> > > > everything is
> > > > > verified.. but /etc/ipa/ca.crt
> didn't reflect
> > > the change..
> > > > so I manually
> > > > > populated it by copying over the guts of
> > the CA
> > > bundle to the
> > > > > /etc/ipa/ca.crt before trying to install
> > the new
> > > server
> > > > cert and it
> > > > > still doesn't recognize it as
> trusted although
> > > the issuer
> > > > is the same
> > > > > and within the CA bundle.
> > > >
> > > > This is going to sound weird, but I'd just
> > go back
> > > in time
> > > > to April 10,
> > > > restart all services but ntp (which will
> > reset the
> > > time) and
> > > > then the
> > > > commands should work. Once the certs are
> > updated and
> > > > working, return to
> > > > present time.
> > > >
> > > > rob
> > > >
> > > > >
> > > > > On Thu, Apr 13, 2023 at 6:20 AM
> Justen Long
> > > > <mr.justenlong@gmail.com
> <mailto:mr.justenlong@gmail.com>
> > <mailto:mr.justenlong@gmail.com <mailto:mr.justenlong@gmail.com>>
> > > <mailto:mr.justenlong@gmail.com
> <mailto:mr.justenlong@gmail.com>
> > <mailto:mr.justenlong@gmail.com
> <mailto:mr.justenlong@gmail.com>>> <mailto:mr.justenlong@gmail.com
> <mailto:mr.justenlong@gmail.com>
> > <mailto:mr.justenlong@gmail.com <mailto:mr.justenlong@gmail.com>>
> > > <mailto:mr.justenlong@gmail.com
> <mailto:mr.justenlong@gmail.com>
> > <mailto:mr.justenlong@gmail.com
> <mailto:mr.justenlong@gmail.com>>>>
> > > > > <mailto:mr.justenlong@gmail.com
> <mailto:mr.justenlong@gmail.com>
> > <mailto:mr.justenlong@gmail.com <mailto:mr.justenlong@gmail.com>>
> > > <mailto:mr.justenlong@gmail.com
> <mailto:mr.justenlong@gmail.com>
> > <mailto:mr.justenlong@gmail.com <mailto:mr.justenlong@gmail.com>>>
> > > > <mailto:mr.justenlong@gmail.com
> <mailto:mr.justenlong@gmail.com>
> > <mailto:mr.justenlong@gmail.com <mailto:mr.justenlong@gmail.com>>
> > > <mailto:mr.justenlong@gmail.com
> <mailto:mr.justenlong@gmail.com>
> > <mailto:mr.justenlong@gmail.com
> <mailto:mr.justenlong@gmail.com>>>>>> wrote:
> > > > >
> > > > > Rob,
> > > > >
> > > > > Apologies for the delay in
> response. Once
> > > I'm home, I
> > > > don't have
> > > > > access to the information readily
> > available
> > > to respond
> > > > with. Here is
> > > > > the information you requested:
> > > > >
> > > > > The version of IPA we are using is
> > 4.6.8, rpm
> > > > specifically for us is
> > > > >
> > ipa-server-4.6.8-5.el7.centos.12.x86_64 and
> > > we are
> > > > using CentOS 7.9
> > > > > currently with plans to move to
> RHEL9
> > within
> > > the next
> > > > year or so.
> > > > >
> > > > > Unfortunately, 'ipa config-show'
> doesn't
> > > work. It
> > > > populates the same
> > > > > error stating "ipa: ERROR: cannot
> > connect to
> > > > > 'https://ipaServer/ipa/json': [SSL:
> > > > CERTIFICATE_VERIFY_FAILED]
> > > > > certificate verify failed
> (_ssl.c:618).
> > > >
> > > > The smack heard around the world was
> my head
> > > hitting my
> > > > desk. Of course
> > > > this command failed.
> > > >
> > > > >
> > > > > We have ~50 hosts connected via IPA.
> > We have
> > > two IPA
> > > > servers, one as
> > > > > a replica of the other.
> > > > >
> > > > > 'getcert list' only shows 1
> certificate.
> > > It's state is
> > > > "MONITORING"
> > > > > and seems related to kerberos.
> > > > >
> > > > > As far as I know, we don't use IPA
> > CA-issued
> > > > certificates. I recall
> > > > > seeing errors yesterday stating
> CA wasn't
> > > enabled on
> > > > our servers. We
> > > > > have always used 3rd party CAs to my
> > knowledge.
> > > > >
> > > > > -justen
> > > > >
> > > > > On Wed, Apr 12, 2023 at 2:42 PM Rob
> > Crittenden
> > > > <rcritten@redhat.com
> <mailto:rcritten@redhat.com>
> > <mailto:rcritten@redhat.com <mailto:rcritten@redhat.com>>
> <mailto:rcritten@redhat.com <mailto:rcritten@redhat.com>
> > <mailto:rcritten@redhat.com <mailto:rcritten@redhat.com>>>
> > > <mailto:rcritten@redhat.com
> <mailto:rcritten@redhat.com> <mailto:rcritten@redhat.com
> <mailto:rcritten@redhat.com>>
> > <mailto:rcritten@redhat.com <mailto:rcritten@redhat.com>
> <mailto:rcritten@redhat.com <mailto:rcritten@redhat.com>>>>
> > > > > <mailto:rcritten@redhat.com
> <mailto:rcritten@redhat.com>
> > <mailto:rcritten@redhat.com <mailto:rcritten@redhat.com>>
> > > <mailto:rcritten@redhat.com
> <mailto:rcritten@redhat.com> <mailto:rcritten@redhat.com
> <mailto:rcritten@redhat.com>>>
> > > > <mailto:rcritten@redhat.com
> <mailto:rcritten@redhat.com>
> > <mailto:rcritten@redhat.com <mailto:rcritten@redhat.com>>
> > > <mailto:rcritten@redhat.com <mailto:rcritten@redhat.com>
> > <mailto:rcritten@redhat.com <mailto:rcritten@redhat.com>>>>>>
> wrote:
> > > > >
> > > > > Justen Long via
> FreeIPA-users wrote:
> > > > > > Thanks in advance for your
> replies..
> > > I've spent
> > > > 7 hours
> > > > > looking through posts here
> and trying
> > > > everything... I'm stuck.
> > > > > >
> > > > > > Background: I am a System
> > > Administrator in a closed,
> > > > > classified environment.
> > Unfortunately, I
> > > cannot
> > > > post logging
> > > > > here, but I can refer to them as
> > needed.
> > > > > >
> > > > > > I inherited this system from
> > someone who
> > > > departed the program
> > > > > a year or so ago. Fast
> forward to
> > today, the
> > > > server certs
> > > > > expired yesterday.
> Admittedly, I'm
> > > unfamiliar (or
> > > > was) with the
> > > > > certificate update process
> for IPA
> > > servers. On a
> > > > typical server,
> > > > > we replace the old cert and
> > restart the
> > > httpd
> > > > services; however,
> > > > > I realize this cannot work
> with IPA
> > > servers now.
> > > > > >
> > > > > > Additionally to all of
> this, the
> > CA chain
> > > > updated 6 months ago.
> > > > > >
> > > > > > I ran ipa-cacert-manage to
> > update the
> > > CA chain.
> > > > When trying to
> > > > > run ipa-certupdate, I received
> > errors for an
> > > > invalid server
> > > > > certificate (it expired on
> 11 April
> > > 2023). It
> > > > simply won't
> > > > > connect to the web server. HTTPD
> > failed
> > > as well,
> > > > so I had to add
> > > > > "NSSEnforceValidCerts off"
> to the
> > > nss.conf file
> > > > for HTTPD to
> > > > > start. Still, no dice.
> > > > > >
> > > > > > I've ran
> ipa-server-certinstall for
> > > the new
> > > > cert/key as well,
> > > > > and it fails saying its not
> > trusted ("Peer's
> > > > certificate issuer
> > > > > is not trusted [certutil:
> > certificate is
> > > invalid:
> > > > Peer's
> > > > > Certificate issuer is not
> recognized]
> > > Please run
> > > > > ipa-cacert-manage install and
> > > ipa-certupdate to
> > > > install the CA
> > > > > certificate.... which, as
> reported
> > > above, can't
> > > > complete.
> > > > > >
> > > > > > I'm at a total loss
> here... and
> > really
> > > > struggling being new to
> > > > > all this and trying my best
> to keep it
> > > afloat. Any
> > > > help would be
> > > > > GREATLY appreciated!
> > > > >
> > > > > Let's gather some
> information first.
> > > > >
> > > > > What version of IPA is this,
> on what
> > > distribution?
> > > > >
> > > > > IPA designates one server to be
> > the "renewal
> > > > master" which
> > > > > handles the
> > > > > renewals. The output of `ipa
> > > config-show` should
> > > > tell you
> > > > > (depending on
> > > > > version). That's the server you
> > want to
> > > work on.
> > > > >
> > > > > How many servers in your
> topology and
> > > how many
> > > > have a CA installed?
> > > > >
> > > > > Does `getcert list` show a
> set of 8-10
> > > tracked
> > > > certificates?
> > > > > What are
> > > > > the states?
> > > > >
> > > > > You mention
> > ipa-server-certinstall. Are
> > > you using
> > > > 3rd party
> > > ��> > certificates
> > > > > in addition to IPA CA-issued
> > > certificates or was
> > > > that just an
> > > > > attempt to
> > > > > get things working again?
> > > > >
> > > > > rob
> > > > >
> > > >
> > >
> >
>