Justen Long wrote:
Rob,
Currently only hiipa03 and hiipa04 (these are the masters/IPA servers)
would utilize the ipa-server-certinstall commands. The other servers
don't use any HTTPD or anything like that, and have never had issues in
the past simply replacing the cert file in /etc/pki/tls/certs and
letting it ride.
So, with regard to the catch-22...
hiipa03 isn't communicating appropriately or something, in my opinion.
When attempting to run "ipa-certupdate" on it, it wants the Kerberos
credentials (assuming 'kinit admin'). I run 'kinit admin" and enter the
admin password that works on hiipa04, and it doesn't like it. I
attempted to change that password using the long 'ldappasswd' command,
but forgot to put dc=hilab,dc=ipa, and instead had put dc=hilab.ipa.
This results in a failure, object not found.. (had a duh moment).. went
to try the 'ldappasswd' command again, using dc=hilab,dc=ipa.. it then
fails stating "Result: Operations error (1) \\ Additional info: Failed
to update password".
So, I just, for s&g, run 'kinit admin' again. Prompts for password. Try
the one that failed because of the wrong dc input, and it takes it,
stating its expired. Try to change it, and get the same "Result:
Operations error (1) \\ Additional info:" Failed to update password."
On the majority of the clients, when I enter "ipa-certupdate" it points
to hiipa03, rather than hiipa04.. and fails because hiipa03 is still in
shambles. I think if I can get hiipa03 back to operational, then
everything else would work.
1) Is there some magical way to destroy the admin password (i.e. blank
it out) so it prompts for a new one, or to replicate the admin stuff
from hiipa04 to hiipa03 so that it all works?
2) If not, how can I possibly move forward? The only hold up is the
admin password at this point.
It's hard to say without any logs. It sure seems like an issue with the
389 server on 03. It isn't clear what you've already done, like
restarting services. I assume you got the certificates replaced there as
well?
But if passwords can't be set that points to something in 389. The
errors log might contain information.
The hardcore option would be to uninstall IPA from 03 and install it as
a new replica from scratch. This assumes that 04 really is working properly.
rob
On Fri, Apr 14, 2023 at 7:11 AM Rob Crittenden <rcritten(a)redhat.com
<mailto:rcritten@redhat.com>> wrote:
Justen Long wrote:
> Rob,
>
> FreeIPA documentation stated that all clients must receive the
> ipa-updatecert command before installing the new server cert.. on the
> hiipa04 server I installed the cert in a hurry for validity sake. Does
> that mess with anything?
ipa-certupdate distributes the CA chain. If the chain didn't change then
it should be fine.
>
> Note: hiipa03 seems to be primary but they’re both masters. I
can’t get
> the ipa-server-certinstall to take on this server stating it’s missing
> the whole chain.. but it worked just fine on hiipa04.. so that’s why I
> was trying to “kinit admin” on 03, to run ipa-updatecert or the
> ipa-cacert-manage there if it didn’t take it from 04, which I’m
assuming
> it didn’t.
You could try going back in time and use ipa-cacert-manage install to
update the chain and then run ipa-server-certinstall.
But like I said before, if you're changing your chain AND replacing the
certs at the same time none of your clients are going to know what
to do.
The process is:
1. ipa-cacert-manage install <new chain>
2. ipa-certupdate on ALL enrolled machines
3. Replace the server certs
If you deviate then you're caught in the catch-22 I mentioned. Clients
won't trust the updated chain in order to download the updated chain.
The aren't a lot of great options out of that. Unenrolling and
re-enrolling the client is the best way.
rob
>
> On Thu, Apr 13, 2023 at 2:41 PM Rob Crittenden
<rcritten(a)redhat.com <mailto:rcritten@redhat.com>
> <mailto:rcritten@redhat.com <mailto:rcritten@redhat.com>>>
wrote:
>
> Justen Long wrote:
> > One go back.. when I tried to run "ipa-certupdate" on the
other hosts
> > (clients), it points to hiipa03 and fails for SSL still..
so, hoping
> > once I get the cert updated on THAT server, that all
restores to its
> > okay state.
>
> You shouldn't need to go back for clients. Now that the server
> certificates are valid they should just work. Unless you also
changed
> the CA chain in which case you're caught in a catch-22: you
need to get
> the updated chain from the server but you don't trust the
chain of the
> server.
>
> rob
>
> >
> > On Thu, Apr 13, 2023 at 1:46 PM Justen Long
> <mr.justenlong(a)gmail.com <mailto:mr.justenlong@gmail.com>
<mailto:mr.justenlong@gmail.com <mailto:mr.justenlong@gmail.com>>
> > <mailto:mr.justenlong@gmail.com
<mailto:mr.justenlong@gmail.com> <mailto:mr.justenlong@gmail.com
<mailto:mr.justenlong@gmail.com>>>>
> wrote:
> >
> > Quick update and answers to your questions.
> >
> > We have two IPA servers, both masters, hiipa03 and hiipa04.
> >
> > hiipa04, I was able to set the time back, run the
> ipa-ca-cert-manage
> > and update the CA (minimally, but enough to accept the
new cert)..
> > ipa-certupdate ran fine on it. Then, ran
ipa-server-certinstall on
> > it, and it took. Website loads, can log into it, do some
user
> > management stuff.. can't run "ipa-replica-manage
list"
as it kicks
> > an error for sslv3 handshake failure.. but was trying
that to
> remove
> > hiipa03 and copy 04 to 03 somehow, maybe?
> >
> > hiipa03 is still giving me grief. Set the time using
timedatectl,
> > verified ntp is off and 'date' reports properly. I had
tried to
> > follow
> >
>
this: https://computingforgeeks.com/reset-freeipa-admin-password-as-root-user-on-linux/
> > when it wasn't accepting the admin password.. it failed
saying
> > object not found. So, I try 'kinit admin' again, with
the new
> > password I tried.. says its expired. Type it in, type in
a new
> > password.. and then it failed saying "kinit: Password change
> failed
> > while getting initial credentials"
> >
> > On Thu, Apr 13, 2023 at 1:40 PM Rob Crittenden
> <rcritten(a)redhat.com <mailto:rcritten@redhat.com>
<mailto:rcritten@redhat.com <mailto:rcritten@redhat.com>>
> > <mailto:rcritten@redhat.com <mailto:rcritten@redhat.com>
<mailto:rcritten@redhat.com <mailto:rcritten@redhat.com>>>> wrote:
> >
> > Justen Long wrote:
> > > I'm getting closer... it's not recognizing my
admin password
> > for IPA, or
> > > for my personal account with admin rights now..
but no more
> > SSL errors..
> > > just can't run ipa-certupdate without the proper
> kerberos creds..
> >
> > By not recognizing your password I assume you mean
kinit is
> > failing? Is
> > the KDC running? I assume 389-ds is running? All
restarted
> after
> > time
> > became stable in the past?
> >
> > rob
> >
> > >
> > > On Thu, Apr 13, 2023 at 12:51 PM Justen Long
> > <mr.justenlong(a)gmail.com
<mailto:mr.justenlong@gmail.com> <mailto:mr.justenlong@gmail.com
<mailto:mr.justenlong@gmail.com>>
> <mailto:mr.justenlong@gmail.com
<mailto:mr.justenlong@gmail.com> <mailto:mr.justenlong@gmail.com
<mailto:mr.justenlong@gmail.com>>>
> > > <mailto:mr.justenlong@gmail.com
<mailto:mr.justenlong@gmail.com>
> <mailto:mr.justenlong@gmail.com
<mailto:mr.justenlong@gmail.com>>
> > <mailto:mr.justenlong@gmail.com
<mailto:mr.justenlong@gmail.com>
> <mailto:mr.justenlong@gmail.com
<mailto:mr.justenlong@gmail.com>>>>> wrote:
> > >
> > > Following up, I see the date command just
changed it
> > momentarily...
> > > using timedatectl and will report back.
> > >
> > > On Thu, Apr 13, 2023 at 12:31 PM Justen Long
> > > <mr.justenlong(a)gmail.com
<mailto:mr.justenlong@gmail.com>
> <mailto:mr.justenlong@gmail.com
<mailto:mr.justenlong@gmail.com>> <mailto:mr.justenlong@gmail.com
<mailto:mr.justenlong@gmail.com>
> <mailto:mr.justenlong@gmail.com
<mailto:mr.justenlong@gmail.com>>>
> > <mailto:mr.justenlong@gmail.com
<mailto:mr.justenlong@gmail.com>
> <mailto:mr.justenlong@gmail.com
<mailto:mr.justenlong@gmail.com>>
> > <mailto:mr.justenlong@gmail.com
<mailto:mr.justenlong@gmail.com>
> <mailto:mr.justenlong@gmail.com
<mailto:mr.justenlong@gmail.com>>>>> wrote:
> > >
> > > Rob,
> > >
> > > I entered 'date --date="7 April
2023",
verified it
> > updated the
> > > system time appropriately. Restarted dirsrv,
> ipa-custodia,
> > > ipa-otpd, httpd.. krb5kdc and kadmin
failed. Still,
> > tried to
> > > send ipa cert-update, and it popped the
same SSL
> > Certificate
> > > Verify Failed error.
> > >
> > > On Thu, Apr 13, 2023 at 11:32 AM Rob
Crittenden
> > > <rcritten(a)redhat.com
<mailto:rcritten@redhat.com>
> <mailto:rcritten@redhat.com <mailto:rcritten@redhat.com>>
<mailto:rcritten@redhat.com <mailto:rcritten@redhat.com>
> <mailto:rcritten@redhat.com <mailto:rcritten@redhat.com>>>
> > <mailto:rcritten@redhat.com
<mailto:rcritten@redhat.com> <mailto:rcritten@redhat.com
<mailto:rcritten@redhat.com>>
> <mailto:rcritten@redhat.com <mailto:rcritten@redhat.com>
<mailto:rcritten@redhat.com <mailto:rcritten@redhat.com>>>>>
wrote:
> > >
> > > Justen Long wrote:
> > > > Additionally, is there any way to
force the CA
> > cert update
> > > to be
> > > > recognized? When I run it to update
the CA
> chain,
> > > everything is
> > > > verified.. but /etc/ipa/ca.crt
didn't reflect
> > the change..
> > > so I manually
> > > > populated it by copying over the guts of
> the CA
> > bundle to the
> > > > /etc/ipa/ca.crt before trying to install
> the new
> > server
> > > cert and it
> > > > still doesn't recognize it as
trusted although
> > the issuer
> > > is the same
> > > > and within the CA bundle.
> > >
> > > This is going to sound weird, but I'd
just
> go back
> > in time
> > > to April 10,
> > > restart all services but ntp (which will
> reset the
> > time) and
> > > then the
> > > commands should work. Once the certs are
> updated and
> > > working, return to
> > > present time.
> > >
> > > rob
> > >
> > > >
> > > > On Thu, Apr 13, 2023 at 6:20 AM
Justen Long
> > > <mr.justenlong(a)gmail.com
<mailto:mr.justenlong@gmail.com>
> <mailto:mr.justenlong@gmail.com
<mailto:mr.justenlong@gmail.com>>
> > <mailto:mr.justenlong@gmail.com
<mailto:mr.justenlong@gmail.com>
> <mailto:mr.justenlong@gmail.com
<mailto:mr.justenlong@gmail.com>>> <mailto:mr.justenlong@gmail.com
<mailto:mr.justenlong@gmail.com>
> <mailto:mr.justenlong@gmail.com
<mailto:mr.justenlong@gmail.com>>
> > <mailto:mr.justenlong@gmail.com
<mailto:mr.justenlong@gmail.com>
> <mailto:mr.justenlong@gmail.com
<mailto:mr.justenlong@gmail.com>>>>
> > > > <mailto:mr.justenlong@gmail.com
<mailto:mr.justenlong@gmail.com>
> <mailto:mr.justenlong@gmail.com
<mailto:mr.justenlong@gmail.com>>
> > <mailto:mr.justenlong@gmail.com
<mailto:mr.justenlong@gmail.com>
> <mailto:mr.justenlong@gmail.com
<mailto:mr.justenlong@gmail.com>>>
> > > <mailto:mr.justenlong@gmail.com
<mailto:mr.justenlong@gmail.com>
> <mailto:mr.justenlong@gmail.com
<mailto:mr.justenlong@gmail.com>>
> > <mailto:mr.justenlong@gmail.com
<mailto:mr.justenlong@gmail.com>
> <mailto:mr.justenlong@gmail.com
<mailto:mr.justenlong@gmail.com>>>>>> wrote:
> > > >
> > > > Rob,
> > > >
> > > > Apologies for the delay in
response. Once
> > I'm home, I
> > > don't have
> > > > access to the information readily
> available
> > to respond
> > > with. Here is
> > > > the information you requested:
> > > >
> > > > The version of IPA we are using is
> 4.6.8, rpm
> > > specifically for us is
> > > >
> ipa-server-4.6.8-5.el7.centos.12.x86_64 and
> > we are
> > > using CentOS 7.9
> > > > currently with plans to move to
RHEL9
> within
> > the next
> > > year or so.
> > > >
> > > > Unfortunately, 'ipa
config-show'
doesn't
> > work. It
> > > populates the same
> > > > error stating "ipa: ERROR:
cannot
> connect to
> > > > 'https://ipaServer/ipa/json':
[SSL:
> > > CERTIFICATE_VERIFY_FAILED]
> > > > certificate verify failed
(_ssl.c:618).
> > >
> > > The smack heard around the world was
my head
> > hitting my
> > > desk. Of course
> > > this command failed.
> > >
> > > >
> > > > We have ~50 hosts connected via IPA.
> We have
> > two IPA
> > > servers, one as
> > > > a replica of the other.
> > > >
> > > > 'getcert list' only shows 1
certificate.
> > It's state is
> > > "MONITORING"
> > > > and seems related to kerberos.
> > > >
> > > > As far as I know, we don't use
IPA
> CA-issued
> > > certificates. I recall
> > > > seeing errors yesterday stating
CA wasn't
> > enabled on
> > > our servers. We
> > > > have always used 3rd party CAs to my
> knowledge.
> > > >
> > > > -justen
> > > >
> > > > On Wed, Apr 12, 2023 at 2:42 PM Rob
> Crittenden
> > > <rcritten(a)redhat.com
<mailto:rcritten@redhat.com>
> <mailto:rcritten@redhat.com <mailto:rcritten@redhat.com>>
<mailto:rcritten@redhat.com <mailto:rcritten@redhat.com>
> <mailto:rcritten@redhat.com <mailto:rcritten@redhat.com>>>
> > <mailto:rcritten@redhat.com
<mailto:rcritten@redhat.com> <mailto:rcritten@redhat.com
<mailto:rcritten@redhat.com>>
> <mailto:rcritten@redhat.com <mailto:rcritten@redhat.com>
<mailto:rcritten@redhat.com <mailto:rcritten@redhat.com>>>>
> > > > <mailto:rcritten@redhat.com
<mailto:rcritten@redhat.com>
> <mailto:rcritten@redhat.com <mailto:rcritten@redhat.com>>
> > <mailto:rcritten@redhat.com
<mailto:rcritten@redhat.com> <mailto:rcritten@redhat.com
<mailto:rcritten@redhat.com>>>
> > > <mailto:rcritten@redhat.com
<mailto:rcritten@redhat.com>
> <mailto:rcritten@redhat.com <mailto:rcritten@redhat.com>>
> > <mailto:rcritten@redhat.com
<mailto:rcritten@redhat.com>
> <mailto:rcritten@redhat.com
<mailto:rcritten@redhat.com>>>>>>
wrote:
> > > >
> > > > Justen Long via
FreeIPA-users wrote:
> > > > > Thanks in advance for your
replies..
> > I've spent
> > > 7 hours
> > > > looking through posts here
and trying
> > > everything... I'm stuck.
> > > > >
> > > > > Background: I am a System
> > Administrator in a closed,
> > > > classified environment.
> Unfortunately, I
> > cannot
> > > post logging
> > > > here, but I can refer to them as
> needed.
> > > > >
> > > > > I inherited this system
from
> someone who
> > > departed the program
> > > > a year or so ago. Fast
forward to
> today, the
> > > server certs
> > > > expired yesterday.
Admittedly, I'm
> > unfamiliar (or
> > > was) with the
> > > > certificate update process
for IPA
> > servers. On a
> > > typical server,
> > > > we replace the old cert and
> restart the
> > httpd
> > > services; however,
> > > > I realize this cannot work
with IPA
> > servers now.
> > > > >
> > > > > Additionally to all of
this, the
> CA chain
> > > updated 6 months ago.
> > > > >
> > > > > I ran ipa-cacert-manage to
> update the
> > CA chain.
> > > When trying to
> > > > run ipa-certupdate, I received
> errors for an
> > > invalid server
> > > > certificate (it expired on
11 April
> > 2023). It
> > > simply won't
> > > > connect to the web server. HTTPD
> failed
> > as well,
> > > so I had to add
> > > > "NSSEnforceValidCerts
off"
to the
> > nss.conf file
> > > for HTTPD to
> > > > start. Still, no dice.
> > > > >
> > > > > I've ran
ipa-server-certinstall for
> > the new
> > > cert/key as well,
> > > > and it fails saying its not
> trusted ("Peer's
> > > certificate issuer
> > > > is not trusted [certutil:
> certificate is
> > invalid:
> > > Peer's
> > > > Certificate issuer is not
recognized]
> > Please run
> > > > ipa-cacert-manage install and
> > ipa-certupdate to
> > > install the CA
> > > > certificate.... which, as
reported
> > above, can't
> > > complete.
> > > > >
> > > > > I'm at a total loss
here... and
> really
> > > struggling being new to
> > > > all this and trying my best
to keep it
> > afloat. Any
> > > help would be
> > > > GREATLY appreciated!
> > > >
> > > > Let's gather some
information first.
> > > >
> > > > What version of IPA is this,
on what
> > distribution?
> > > >
> > > > IPA designates one server to be
> the "renewal
> > > master" which
> > > > handles the
> > > > renewals. The output of `ipa
> > config-show` should
> > > tell you
> > > > (depending on
> > > > version). That's the server
you
> want to
> > work on.
> > > >
> > > > How many servers in your
topology and
> > how many
> > > have a CA installed?
> > > >
> > > > Does `getcert list` show a
set of 8-10
> > tracked
> > > certificates?
> > > > What are
> > > > the states?
> > > >
> > > > You mention
> ipa-server-certinstall. Are
> > you using
> > > 3rd party
> > ��> > certificates
> > > > in addition to IPA CA-issued
> > certificates or was
> > > that just an
> > > > attempt to
> > > > get things working again?
> > > >
> > > > rob
> > > >
> > >
> >
>