Re: ipa-restore and the issues

[Rob Crittenden]

Please keep responses on the list.

Ian Kumlien wrote:
> ipa find-user admin
> ipa: ERROR: No valid Negotiate header in server response
>
> And a lot of krb issues according to the http logs

I think we need to see the logs to diagnose.
> I wasn't expecting this - since all keys should be the same as the one
> installed - which is why i asked about any changes to the ldap data

It could happen, for example, if you had gotten a new keytab for one or
more service and restored old data. Unlikely, but possible.

Comparing the klist output with kvno for all the keytabs and principals
will tell you.

rob
> If there is something more specific you want me to look at, just let me know
>
> On Wed, Feb 5, 2020 at 4:54 PM Rob Crittenden  wrote:
> > Ian Kumlien via FreeIPA-users wrote:
> > > Hi,
> > >
> > > Due to issues, I'm trying to do a partial restore of all the "important bits"
> > >
> > > But if I do ipa-restore --online --data --backend=userRoot $BACKUP
> > >
> > > I end up in a semiworking environment - the webui doen't work - kinit does...
> > >
> > > ipa doesn't etc..
> >
> > It doesn't work how? What have you done to troubleshoot? What do the
> > logs say?
> >
> > rob

ipa-restore and the issues

[Ian Kumlien]

Sorry for the high latency, there has been quite a bit of prio 1
things that needed
fixing that's been delaying this

On Wed, Feb 5, 2020 at 7:13 PM Rob Crittenden  wrote:
> Please keep responses on the list.
>
> Ian Kumlien wrote:
> > ipa find-user admin
> > ipa: ERROR: No valid Negotiate header in server response
> >
> > And a lot of krb issues according to the http logs
>
> I think we need to see the logs to diagnose.

httpd/error_log:
[Tue Mar 17 10:25:19.273326 2020] [auth_gssapi:error] [pid 24047:tid
140398705956608] [client 10.0.0.15:52430] GSS ERROR
gss_acquire_cred[_from]() failed to get server creds: [Unspecified GSS
failure.  Minor code may provide more information ( SPNEGO cannot find
mechanisms to negotiate)]
[Tue Mar 17 10:25:19.277017 2020] [wsgi:error] [pid 24045:tid
140398987495168] [remote 100.94.37.38:34088] ipa: INFO: 401
Unauthorized: No session cookie found
> > I wasn't expecting this - since all keys should be the same as the one
> > installed - which is why i asked about any changes to the ldap data
>
> It could happen, for example, if you had gotten a new keytab for one or
> more service and restored old data. Unlikely, but possible.

Thats exactly whats happened, could I just do a ldap-updater script to
update the keys?
> Comparing the klist output with kvno for all the keytabs and principals
> will tell you.
>
> rob
> > If there is something more specific you want me to look at, just let me know
> >
> > On Wed, Feb 5, 2020 at 4:54 PM Rob Crittenden  wrote:
> > > Ian Kumlien via FreeIPA-users wrote:
> > > > Hi,
> > > >
> > > > Due to issues, I'm trying to do a partial restore of all the "important bits"
> > > >
> > > > But if I do ipa-restore --online --data --backend=userRoot $BACKUP
> > > >
> > > > I end up in a semiworking environment - the webui doen't work - kinit does...
> > > >
> > > > ipa doesn't etc..
> > >
> > > It doesn't work how? What have you done to troubleshoot? What do the
> > > logs say?
> > >
> > > rob

ipa-restore and the issues

[Rob Crittenden]

Ian Kumlien wrote:
> Sorry for the high latency, there has been quite a bit of prio 1
> things that needed
> fixing that's been delaying this
>
> On Wed, Feb 5, 2020 at 7:13 PM Rob Crittenden  wrote:
> > Please keep responses on the list.
> >
> > Ian Kumlien wrote:
> > > ipa find-user admin
> > > ipa: ERROR: No valid Negotiate header in server response
> > >
> > > And a lot of krb issues according to the http logs
> >
> > I think we need to see the logs to diagnose.
>
> httpd/error_log:
> [Tue Mar 17 10:25:19.273326 2020] [auth_gssapi:error] [pid 24047:tid
> 140398705956608] [client 10.0.0.15:52430] GSS ERROR
> gss_acquire_cred[_from]() failed to get server creds: [Unspecified GSS
> failure.  Minor code may provide more information ( SPNEGO cannot find
> mechanisms to negotiate)]
> [Tue Mar 17 10:25:19.277017 2020] [wsgi:error] [pid 24045:tid
> 140398987495168] [remote 100.94.37.38:34088] ipa: INFO: 401
> Unauthorized: No session cookie found
> > > I wasn't expecting this - since all keys should be the same as the one
> > > installed - which is why i asked about any changes to the ldap data
> >
> > It could happen, for example, if you had gotten a new keytab for one or
> > more service and restored old data. Unlikely, but possible.
>
> Thats exactly whats happened, could I just do a ldap-updater script to
> update the keys?

There is no current automation to refresh all kerberos keytabs. You
would need to run ipa-getkeytab on each one individually. I'd only renew
ones that are out-of-sync though.

rob
> > Comparing the klist output with kvno for all the keytabs and principals
> > will tell you.
> >
> > rob
> > > If there is something more specific you want me to look at, just let me know
> > >
> > > On Wed, Feb 5, 2020 at 4:54 PM Rob Crittenden  wrote:
> > > > Ian Kumlien via FreeIPA-users wrote:
> > > > > Hi,
> > > > >
> > > > > Due to issues, I'm trying to do a partial restore of all the "important bits"
> > > > >
> > > > > But if I do ipa-restore --online --data --backend=userRoot $BACKUP
> > > > >
> > > > > I end up in a semiworking environment - the webui doen't work - kinit does...
> > > > >
> > > > > ipa doesn't etc..
> > > >
> > > > It doesn't work how? What have you done to troubleshoot? What do the
> > > > logs say?
> > > >
> > > > rob

ipa-restore and the issues

[Ian Kumlien]

Oh, yes, managed!

But, welcome new issues ;)

I had problems with the CA but managed to fix them by adding the cert.

Now I can't install replicas since there is problems communicating
with the CA and ipa-server-upgrade results in:
...
[Update certmonger certificate renewal configuration]
Introspect error on :1.88:/org/fedorahosted/certmonger:
dbus.exceptions.DBusException: org.freedesktop.DBus.Error.NoReply: Did
not receive a reply. Possible causes include: the remote application
did not send a reply, the message bus security policy blocked the
reply, the reply timeout expired, or the network connection was
broken.
...
[Adding default OCSP URI configuration]
pki-tomcat configuration changed, restart pki-tomcat
[Ensuring CA is using LDAPProfileSubsystem]
[Migrating certificate profiles to LDAP]
IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run
command ipa-server-upgrade manually.
Unexpected error - see /var/log/ipaupgrade.log for details:
RemoteRetrieveError: Failed to authenticate to CA REST API
The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for
more information
---

From the log:
2020-03-18T21:12:15Z INFO [Migrating certificate profiles to LDAP]
2020-03-18T21:12:15Z DEBUG Created connection context.ldap2_140683437055616
2020-03-18T21:12:15Z DEBUG flushing
ldapi://%2Fvar%2Frun%2Fslapd-XERCES-LAN.socket from SchemaCache
2020-03-18T21:12:15Z DEBUG retrieving schema for SchemaCache
url=ldapi://%2Fvar%2Frun%2Fslapd-XERCES-LAN.socket
conn=<ldap.ldapobject.SimpleLDAPObject object at 0x7ff36a2a2fd0>
2020-03-18T21:12:15Z DEBUG Destroyed connection context.ldap2_140683437055616
2020-03-18T21:12:15Z DEBUG request GET
https://freeipa-1.xerces.lan:8443/ca/rest/account/login
2020-03-18T21:12:15Z DEBUG request body ''
2020-03-18T21:12:15Z DEBUG response status 401
2020-03-18T21:12:15Z DEBUG response headers Cache-Control: private
Expires: Thu, 01 Jan 1970 00:00:00 GMT
WWW-Authenticate: Basic realm="Certificate Authority"
Content-Type: text/html;charset=utf-8
Content-Language: en
Content-Length: 1033
Date: Wed, 18 Mar 2020 21:12:15 GMT

2020-03-18T21:12:15Z DEBUG response body (decoded): b'<!doctype
html><html lang="en"><head><title>HTTP Status 401 \xe2\x80\x93
Unauthorized</title><style type="text/css">h1
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;}
h2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;}
h3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;}
body {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;}
b {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;}
p {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}
a {color:black;} a.name {color:black;} .line
{height:1px;background-color:#525D76;border:none;}</style></head><body><h1>HTTP
Status 401 \xe2\x80\x93 Unauthorized</h1><hr class="line"
/><p><b>Type</b> Status Report</p><p><b>Description</b> The request
has not been applied because it lacks valid authentication credentials
for the target resource.</p><hr class="line" /><h3>Apache
Tomcat/9.0.7</h3></body></html>'
2020-03-18T21:12:15Z ERROR IPA server upgrade failed: Inspect
/var/log/ipaupgrade.log and run command ipa-server-upgrade manually.
2020-03-18T21:12:15Z DEBUG   File
"/usr/lib/python3.6/site-packages/ipapython/admintool.py", line 179,
in execute
    return_value = self.run()
  File "/usr/lib/python3.6/site-packages/ipaserver/install/ipa_server_upgrade.py",
line 54, in run
    server.upgrade()
  File "/usr/lib/python3.6/site-packages/ipaserver/install/server/upgrade.py",
line 2227, in upgrade
    upgrade_configuration()
  File "/usr/lib/python3.6/site-packages/ipaserver/install/server/upgrade.py",
line 2097, in upgrade_configuration
    ca_enable_ldap_profile_subsystem(ca)
  File "/usr/lib/python3.6/site-packages/ipaserver/install/server/upgrade.py",
line 412, in ca_enable_ldap_profile_subsystem
    cainstance.migrate_profiles_to_ldap()
  File "/usr/lib/python3.6/site-packages/ipaserver/install/cainstance.py",
line 1922, in migrate_profiles_to_ldap
    _create_dogtag_profile(profile_id, profile_data, overwrite=False)
  File "/usr/lib/python3.6/site-packages/ipaserver/install/cainstance.py",
line 1928, in _create_dogtag_profile
    with api.Backend.ra_certprofile as profile_api:
  File "/usr/lib/python3.6/site-packages/ipaserver/plugins/dogtag.py",
line 1315, in __enter__
    raise errors.RemoteRetrieveError(reason=_('Failed to authenticate
to CA REST API'))

2020-03-18T21:12:15Z DEBUG The ipa-server-upgrade command failed,
exception: RemoteRetrieveError: Failed to authenticate to CA REST API
2020-03-18T21:12:15Z ERROR Unexpected error - see
/var/log/ipaupgrade.log for details:
RemoteRetrieveError: Failed to authenticate to CA REST API
---

This is on centos 8 stream with idm:DL1 (freeipa 4.8.0)