Hidden replica ipa-healthcheck error ADTRUST service is not enabled
by Duncan Mortimer
Hi,
We have a three node Centos 8 IPA domain which we upgraded from Centos 8.3 to 8.4 (IPA version 4.9.2) yesterday. This appeared to succeed without issue, with clients continuing to operate as expected. To give us confidence that all was well we ran ipa-healthcheck on each node post upgrade and this found no issues on our two client facing servers ipa0 and ipa1 but on the third machine, ipa2, which is configured as a hidden replica (and is used to take disaster recovery backups) we received the following error:
[
{
"source": "ipahealthcheck.ipa.trust",
"check": "IPATrustControllerServiceCheck",
"result": "ERROR",
"uuid": "f5d524a3-43c3-4320-92cb-c984727243d9",
"when": "20210624085009Z",
"duration": "0.000512",
"kw": {
"key": "ADTRUST",
"msg": "{key} service is not enabled"
}
}
]
We have Samba services setup for Centos based SMB file sharing - there is no Active Directory install on our network.
Is this error a concern or is it a consequence of being a hidden replica. If it is a problem, how might we go about fixing the issue? As a first step I've tried re-running ipa-adtrust and this didn't seem to need to make any changes:
The log file for this installation can be found in /var/log/ipaserver-adtrust-install.log
==============================================================================
This program will setup components needed to establish trust to AD domains for
the IPA Server.
This includes:
* Configure Samba
* Add trust related objects to IPA LDAP server
To accept the default shown in brackets, press the Enter key.
Configuring cross-realm trusts for IPA server requires password for user 'admin'.
This user is a regular system account used for IPA server administration.
admin password:
IPA generated smb.conf detected.
Overwrite smb.conf? [no]: yes
The following operations may take some minutes to complete.
Please wait until the prompt is returned.
Configuring CIFS
[1/24]: validate server hostname
[2/24]: stopping smbd
[3/24]: creating samba domain object
Samba domain object already exists
[4/24]: retrieve local idmap range
[5/24]: writing samba config file
[6/24]: creating samba config registry
[7/24]: adding cifs Kerberos principal
[8/24]: adding cifs and host Kerberos principals to the adtrust agents group
[9/24]: check for cifs services defined on other replicas
[10/24]: adding cifs principal to S4U2Proxy targets
cifs principal already targeted, nothing to do.
[11/24]: adding admin(group) SIDs
Admin SID already set, nothing to do
Admin group SID already set, nothing to do
[12/24]: adding RID bases
RID bases already set, nothing to do
[13/24]: updating Kerberos config
'dns_lookup_kdc' already set to 'true', nothing to do.
[14/24]: activating CLDAP plugin
CLDAP plugin already configured, nothing to do
[15/24]: activating sidgen task
Sidgen task plugin already configured, nothing to do
[16/24]: map BUILTIN\Guests to nobody group
[17/24]: configuring smbd to start on boot
[18/24]: enabling trusted domains support for older clients via Schema Compatibility plugin
[19/24]: restarting Directory Server to take MS PAC and LDAP plugins changes into account
[20/24]: adding fallback group
Fallback group already set, nothing to do
[21/24]: adding Default Trust View
Default Trust View already exists.
[22/24]: setting SELinux booleans
[23/24]: starting CIFS services
[24/24]: restarting smbd
Done configuring CIFS.
=============================================================================
Setup complete
You must make sure these network ports are open:
TCP Ports:
* 135: epmap
* 138: netbios-dgm
* 139: netbios-ssn
* 445: microsoft-ds
* 1024..1300: epmap listener range
* 3268: msft-gc
UDP Ports:
* 138: netbios-dgm
* 139: netbios-ssn
* 389: (C)LDAP
* 445: microsoft-ds
See the ipa-adtrust-install(1) man page for more details
=============================================================================
But there is no change in ipahealthcheck output.
Regards,
Duncan
--
Duncan Mortimer
2 years, 9 months
FreeIPA w. letsencrypt for HTTPS/LDAP failing to communicate with itself
by Chris Moody
Hello folks.
Hopefully I'm just missing something face-palm level obvious, but I am
running into some trouble when interfacing with my CA functionality on
an IPA server cluster. My attempts at scouring all my saved prior-comms
from the mailing-list as well as several search-engines are not
enchanting me with much clue.
It appears that my need for the LetsEncrypt certs for the user-facing
Web-UI and LDAPs components are causing IPA to dis-trust itself.
===
4-node cluster - Ubuntu19.10
(all nodes currently fully updated/patched via the official Ubuntu repos)
===
ipa --version
VERSION: 4.8.1, API_VERSION: 2.233
===
running letsencrypt certificates successfully for HTTPs & LDAPs connectivity
===
These 4-nodes are all happily running and replicating betwixt each
other. LDAPs is functioning great and many linux systems are able to
all join as freeipa-clients. Users and groups are replicating and being
used elegantly for many LDAP-based authentication/authorization needs.
Overall, for these nodes, life is good.
Where I'm running into trouble is in finally wanting to leverage
certificate issuance on a per-user basis. End goal is integrating
things like yubikeys, user-cert auth, and so on.
In the UI, when I enter a user's account and select Actions->New
Certificate, I am able to successfully issue the couple prompted
'certutil' commands to generate the user's CSR. I then paste in the
contents of the CSR and hit 'Issue' and run into the following error:
==========
IPA Error 907: NetworkError
cannot connect to
'https://REDACTED-1.ipa.REDACTED.com:443/ca/rest/account/login': [SSL:
TLSV1_ALERT_UNKNOWN_CA] tlsv1 alert unknown ca (_ssl.c:2508)
==========
As I then start digging into cli-mode to attempt to understand where
things are unhappy, I run into similar troubles with the server
attempting to talk to itself and not being very happy about it.
==========
chris@REDACTED-1:~$ ipa ca-find
------------
1 CA matched
------------
Name: ipa
Description: IPA CA
Authority ID: 8acca54b-64d7-44bf-b8f7-59316213cfb6
Subject DN: CN=Certificate Authority,O=IPA.REDACTED.COM
Issuer DN: CN=Certificate Authority,O=IPA.REDACTED.COM
----------------------------
Number of entries returned 1
----------------------------
chris@REDACTED-1:~$ ipa ca-show
Name: ipa
ipa: ERROR: cannot connect to
'https://REDACTED-1.ipa.REDACTED.com:443/ca/rest/account/login': [SSL:
TLSV1_ALERT_UNKNOWN_CA] tlsv1 alert unknown ca (_ssl.c:2508)
==========
Verifying with 'openssl s_client' returns the valid and non-expired LE
cert-chain.
==========
chris@REDACTED-1:~$ openssl s_client REDACTED-1.ipa.REDACTED.com:443
CONNECTED(00000003)
depth=2 O = Digital Signature Trust Co., CN = DST Root CA X3
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = REDACTED-1.ipa.REDACTED.com
verify return:1
---
Certificate chain
0 s:CN = REDACTED-1.ipa.REDACTED.com
i:C = US, O = Let's Encrypt, CN = R3
1 s:C = US, O = Let's Encrypt, CN = R3
i:O = Digital Signature Trust Co., CN = DST Root CA X3
---
...<output-truncated>...
---
SSL handshake has read 3046 bytes and written 413 bytes
Verification: OK
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
...<output-truncated>...
==========
Can anyone please hit me with some clue-bat as to where I can read to
understand how to get IPA to love itself? I'm suspecting it's likely
some certificate inclusion/exception that I need to add so that API
calls and the ipa command itself will actually respect the LE cert-chain?
Any hints would be greatly appreciated.
Thanks,
-Chris
--
Node-Nine, Inc.
chris(a)node-nine.com
619.354.6463
2 years, 9 months
compat branch not browseable
by Joseph Fry
I am just curious why the cn=compat,dc=mydomain,dc=org container cannot be reached when I bind to dc=mydomain,dc=org, but I can see it if I bind directly to it. Is there any way to expose it?
2 years, 9 months
Get date user was deleted and preserved
by Tania Hagan
Hi,
Is there a way to get the date and time a user was deleted and preserved (ipa user-del --preserve) and if possible by who?
Many Thanks,
Tania
2 years, 9 months
Re: How to blend IPA server 4.1.4 on F21 with server 4.6.8 on C7?
by Bret Wortman
I cleaned up the contents of our ldap manually, re-created the replica file, and got a lot further than we have before but ipa-replica-install still failed as below:
Configuring certificate server (pki-tomcatd). Estimated time: 3 minutes
[1/30]: configuring certificate server instance
ipaserver.install.dogtaginstance: CRITICAL Failed to configure CA instance: Command '/usr/sbin/pkispawn -s CA -f /tmp/tmphvJyRY' returned non-zero exit status 1
ipaserver.install.dogtaginstance: CRITICAL See the installation logs and the following files/directories for more information:
ipaserver.install.dogtaginstance: CRITICAL /var/log/pki/pki-tomcat
[error] RuntimeError: CA configuration failed.
Your system may be partly configured.
Run /usr/sbin/ipa-server-install --uninstall to clean up.
ipapython.admintool: ERROR CA configuration failed.
ipapython.admintool: ERROR The ipa-replica-install command failed. See /var/log/ipareplica-install.log for more information
[root@ipa2c7 ~]# ipa-server-install --uninstall -U
ipapython.admintool: ERROR Unable to read /etc/httpd/conf.d/ipa-pki-proxy.conf
ipapython.admintool: ERROR The ipa-server-install command failed. See /var/log/ipaserver-uninstall.log for more information
[root@ipa2c7 ~]# touch /etc/httpd/conf.d/ipa-pki-proxy.conf
[root@ipa2c7 ~]# ipa-server-install --uninstall -U
Deleting this server will leave your installation without a CRL generation master.
ipapython.admintool: ERROR Aborting uninstall operation.
ipapython.admintool: ERROR The ipa-server-install command failed. See /var/log/ipaserver-uninstall.log for more information
It looks to have failed in such a way that it doesn't know how to back out again, which I haven't seen before. Thoughts? The error in ipa-uninstall.log looks like a generic admintool.py error:
2021-06-07T12:31:38Z DEBUG retrieving schema for SchemaCache url=ldapi://%2fvar%2frun%2fslapd-OUR-NET.socket conn=<ldap.ldapobject.SimpleLDAPObject instance at 0x7f05289fdf80>
2021-06-07T12:31:38Z DEBUG raw: config_show(version=u'2.237')
2021-06-07T12:31:38Z DEBUG config_show(rights=False, all=False, raw=False, version=u'2.237')
2021-06-07T12:31:38Z DEBUG File "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 178, in execute
return_value = self.run()
File "/usr/lib/python2.7/site-packages/ipapython/install/cli.py", line 319, in run
return cfgr.run()
File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 358, in run
self.validate()
File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 368, in validate
for _nothing in self._validator():
File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 431, in __runner
exc_handler(exc_info)
File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 455, in _handle_validate_exception
self._handle_exception(exc_info)
File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 450, in _handle_exception
six.reraise(*exc_info)
File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 421, in __runner
step()
File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 418, in <lambda>
step = lambda: next(self.__gen)
File "/usr/lib/python2.7/site-packages/ipapython/install/util.py", line 81, in run_generator_with_yield_from
six.reraise(*exc_info)
File "/usr/lib/python2.7/site-packages/ipapython/install/util.py", line 59, in run_generator_with_yield_from
value = gen.send(prev_value)
File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 633, in _configure
next(validator)
File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 431, in __runner
exc_handler(exc_info)
File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 455, in _handle_validate_exception
self._handle_exception(exc_info)
File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 518, in _handle_exception
self.__parent._handle_exception(exc_info)
File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 450, in _handle_exception
six.reraise(*exc_info)
File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 515, in _handle_exception
super(ComponentBase, self)._handle_exception(exc_info)
File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 450, in _handle_exception
six.reraise(*exc_info)
File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 421, in __runner
step()
File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 418, in <lambda>
step = lambda: next(self.__gen)
File "/usr/lib/python2.7/site-packages/ipapython/install/util.py", line 81, in run_generator_with_yield_from
six.reraise(*exc_info)
File "/usr/lib/python2.7/site-packages/ipapython/install/util.py", line 59, in run_generator_with_yield_from
value = gen.send(prev_value)
File "/usr/lib/python2.7/site-packages/ipapython/install/common.py", line 73, in _uninstall
for unused in self._uninstaller(self.parent):
File "/usr/lib/python2.7/site-packages/ipaserver/install/server/__init__.py", line 594, in main
uninstall_check(self)
File "/usr/lib/python2.7/site-packages/ipaserver/install/server/install.py", line 250, in decorated
func(installer)
File "/usr/lib/python2.7/site-packages/ipaserver/install/server/install.py", line 1030, in uninstall_check
ca.uninstall_check(options)
File "/usr/lib/python2.7/site-packages/ipaserver/install/ca.py", line 127, in uninstall_check
raise ScriptError("Aborting uninstall operation.")
2021-06-07T12:31:38Z DEBUG The ipa-server-install command failed, exception: ScriptError: Aborting uninstall operation.
2021-06-07T12:31:38Z ERROR Aborting uninstall operation.
2021-06-07T12:31:38Z ERROR The ipa-server-install command failed. See /var/log/ipaserver-uninstall.log for more information
--
Bret Wortman
bret.wortman(a)damascusgrp.com
On Fri, Jun 4, 2021, at 1:32 PM, Bret Wortman wrote:
> Boom. Looking through the ldifs now. Thanks again, Rob.
>
> --
> Bret Wortman
> bret.wortman(a)damascusgrp.com
>
> On Fri, Jun 4, 2021, at 1:22 PM, Rob Crittenden wrote:
> > Bret Wortman wrote:
> > > What's dsctl? I don't see that anywhere on any of my servers (including the more up-to-date ones). My 389 instance is v1.3.3, if that makes a difference...
> > >
> > >
> >
> > Right, really old release.
> >
> > Try: db2ldif -n userRoot -Z EXAMPLE-TEST -a /path/to/ldif/file
> >
> > To get the CA ldif replace userRoot with ipaca.
> >
> > rob
> >
> >
>
2 years, 10 months
Re: Join command 500 errors, timeouts
by Alfred Victor
Hi FreeIPA list,
I did verify UDP is making it but for some reason ntpd commands fail in the
join anyway (seemingly by timeout), I'm not sure why yet but for now we use
--no-ntp after setting time just after boot so this isn't an issue. Curious
if anyone can spot anything helpful in those logs above please, or how we
can debug further to find out why our joins fail? The 401 not authorized
seems to be really pretty mysterious, and I'm not sure if that "Matching
credential not found" log message is a clue. Likewise, the "No permission
to join" and "Cannot contact any KDC" are troubling, and I'm not sure if
this is a clue below. So far I haven't managed to identify a network
problem and log messages seem few and far between for the nodes which fail
joins. Shared previously:
./krb5kdc.log:389805:Jun 21 14:12:07 hipa0002.redac.com krb5kdc[4746](info):
DISPATCH: repeated (*retransmit*ted?) request from 10.2.11.17, resending
previous response
Alfred
On Tue, Jun 22, 2021 at 1:12 AM Angus Clarke <angus(a)charworth.com> wrote:
> Hello
>
> I've not been following the whole topic but you can replace your UDP
> listener with netcat and indeed use netcat to simulate a client initiation
> too. This could help tell you whether your network is permitting that
> traffic or not.
>
> Duckduckgo found me this which looks suitable
>
> https://fsferrara.github.io/2020/08/02/sending-and-receiving-packets-usin...
>
> Regards
> Angus
>
> ------------------------------
> *From:* Alfred Victor via FreeIPA-users <
> freeipa-users(a)lists.fedorahosted.org>
> *Sent:* Tuesday, June 22, 2021 12:24:32 AM
> *To:* Rob Crittenden <rcritten(a)redhat.com>
> *Cc:* FreeIPA users list <freeipa-users(a)lists.fedorahosted.org>; Alfred
> Victor <alvic266(a)gmail.com>
> *Subject:* [Freeipa-users] Re: Join command 500 errors, timeouts
>
> Thanks again. I've enabled trace and will see what I find. I do know there
> may be an issue with UDP because it would log ntp failures until I did
> --no-ntp, but the port is open on firewall both sides so I believe the
> network is failing to deliver UDP.
>
> Alfred
>
> On Mon, Jun 21, 2021 at 4:39 PM Rob Crittenden <rcritten(a)redhat.com>
> wrote:
>
> Alfred Victor wrote:
> > Hi Rob and FreeIPA list,
> >
> > I wish I was able to say we are seeing success but aside from no longer
> > having the krb5kdc.log log spam, we still experience a lot of join
> > failures. However rather than seeing a ton of timeouts (still a few but
> > much less), we get a lot of errors like below, which totally puzzle us
> > as the ipactl services all report up, and mach_join account should have
> > sufficient privilege. I will note we have our own non-IPA DNS solution
> > where we have added all the records IPA should need but I can't see why
> > this change would cause our errors (and indeed all the output of "ipa
> > *dns*-update-system-records --dry-run" looks ok, not that below errors
> > appear DNS related but its the only change I can think of aside from
> > having replaced our IPA hosts with re-replicated new builds.)
> >
> > This works fine on rerun of install, and I don't see any evidence of
> > service failure on the system this happened against, but neither am I
> > aware of any intermittent network problem (perhaps there is a network
> > problem on the very first connection to IPA we have not identified? but
> > I would expect different errors than the 401 under this for instance)
> >
> > 2021-06-21T19:12:05Z DEBUG Initializing principal
> > mach_join(a)redac.COM using password
> > 2021-06-21T19:12:05Z DEBUG Starting external process
> > 2021-06-21T19:12:05Z DEBUG args=/usr/bin/kinit mach_join(a)redac.COM
> > -c /tmp/krbcc_xp2E2/ccache
> > 2021-06-21T19:12:24Z DEBUG Process finished, return code=1
> > 2021-06-21T19:12:24Z DEBUG stdout=
> > 2021-06-21T19:12:24Z DEBUG stderr=kinit: Cannot contact any KDC for
> > realm 'redac.COM' while getting initial credentials
> > 2021-06-21T19:12:24Z INFO Please make sure the following ports are
> > opened in the firewall settings:
> > TCP: 80, 88, 389
> > UDP: 88 (at least one of TCP/UDP ports 88 has to be open)
> > Also note that following ports are necessary for ipa-client working
> > properly after enrollment:
> > TCP: 464
> > UDP: 464, 123 (if NTP enabled)
> > 2021-06-21T19:12:24Z ERROR Installation failed. Rolling back changes.
> >
> >
> >
> > But then we also see things like this predominantly, in fact this seems
> > now to be the largest category of failure we observe at the highest rate
> > and makes no sense to me, because the mach_join account privileges
> > should not be changing:
> >
> > 2021-06-17T19:29:11Z DEBUG args=/usr/sbin/ipa-join -s
> > hipa0003.redac.com
> <https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fhipa000...>
> <http://hipa0003.redac.com
> <https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fhipa000...>>
> -b dc=redac,dc=com -h
> > hred2-4-18.redac.com
> <https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fhred2-4...>
> <http://hred2-4-18.redac.com
> <https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fhred2-4...>>
> -f
> > 2021-06-17T19:29:31Z DEBUG Process finished, return code=17
> > 2021-06-17T19:29:31Z DEBUG stdout=
> > 2021-06-17T19:29:31Z DEBUG stderr=HTTP response code is 401, not 200
> > 2021-06-17T19:29:31Z ERROR Joining realm failed: HTTP response code
> > is 401, not 200
> >
> >
> >
> > Some more like this, but not quite the same error:
> >
> >
> > 2021-06-17T19:29:32Z DEBUG get_ca_certs_from_ldap() error:
> > Insufficient access: SASL(-1): generic failure: GSSAPI Error:
> > Unspecified GSS failure. Minor code may provide more information
> > (Cannot contact any KDC for realm 'redac.COM')
> > 2021-06-17T19:29:32Z DEBUG Insufficient access: SASL(-1): generic
> > failure: GSSAPI Error: Unspecified GSS failure. Minor code may
> > provide more information (Cannot contact any KDC for realm
> 'redac.COM')
> >
> >
> > And still another example:
> >
> >
> >
> >
> > 2021-06-17T22:25:37Z DEBUG args=/usr/sbin/ipa-join -s
> > hipa0001.redac.com
> <https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fhipa000...>
> <http://hipa0001.redac.com
> <https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fhipa000...>>
> -b dc=redac,dc=com -h
> > hred2-4-18.redac.com
> <https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fhred2-4...>
> <http://hred2-4-18.redac.com
> <https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fhred2-4...>>
> -f
> > 2021-06-17T22:25:41Z DEBUG Process finished, return code=17
> > 2021-06-17T22:25:41Z DEBUG stdout=
> > 2021-06-17T22:25:41Z DEBUG stderr=No permission to join this host to
> > the IPA domain.
> > 2021-06-17T22:25:41Z ERROR Joining realm failed: No permission to
> > join this host to the IPA domain.
> > 2021-06-17T22:25:41Z ERROR Installation failed. Rolling back changes.
> >
> > Any thoughts or ideas please?
>
> Some of these could be network issues. I'd run ipa-client-install with
> Kerberos tracing:
>
> KRB5_TRACE=/path/to/some/log ipa-client-install <options>
>
> So that on failure you can see what KDCs were available, whether one was
> contacted, etc.
>
> rob
>
> >
> > Alfred
> >
> >
> > On Tue, Jun 15, 2021 at 3:19 PM Rob Crittenden <rcritten(a)redhat.com
> > <mailto:rcritten@redhat.com>> wrote:
> >
> > Alfred Victor wrote:
> > > I don't see a directive equivalent of SECURE_NFS to add to
> > nfs.conf (all
> > > documentation seems to still refer to the sysconfig path), or is
> > it the
> > > same? Can I just disable rpcgssd? We have no nfs mounts which are
> > > kerberized yet, and disabling rpcgssd seems to solve our problem,
> > and I
> > > can kinit after disabling rpcgssd. It also does not seem that
> > disabling
> > > rpcgssd hurts running tasks or node, but would like to confirm it's
> > > limited to nfs in function. I still have to wonder if there's a
> > better way.
> >
> > I suspect that SECURE_NFS is a no-op these days. It was necessary in
> > RHEL 5/6 for sure.
> >
> > Yes, I think you can safely disable the service. In some versions of
> > RHEL/Fedora IIRC this is symlinked to nfs-secure-server so mask that
> as
> > well.
> >
> > rob
> >
> > >
> > > Alfred
> > >
> > > On Tue, Jun 15, 2021 at 10:31 AM Alexander Bokovoy
> > <abokovoy(a)redhat.com <mailto:abokovoy@redhat.com>
> > > <mailto:abokovoy@redhat.com <mailto:abokovoy@redhat.com>>> wrote:
> > >
> > > On ti, 15 kesä 2021, Alfred Victor via FreeIPA-users wrote:
> > > >Hi Rob,
> > > >
> > > >We attempted setting sec=sys on the mount, however to our
> > surprise
> > > found
> > > >this didn't work. We then figured out that IPA install is
> adding
> > > this to
> > > >/etc/sysconfig/nfs:
> > > >
> > > >SECURE_NFS=yes
> > > >
> > > >
> > > >We tried removing this to no avail and restarting all the
> related
> > > sytstemd
> > > >units (rpcgssd, nfs, etc). Any idea why sec=sys is being
> ignored?
> > > Should we
> > > >need to set SECURE_NFS=no? On non-IPA nodes this directive
> > does not
> > > exist
> > > >at all. For now, I have also totally disabled rpcgssd as I
> think
> > > this unit
> > > >may be responsible ( it seems that it does the upcall in
> > > >https://access.redhat.com/solutions/225783
> <https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Faccess...>
> ) so I will hope
> > that this
> > > >solves it, I don't believe anything else depends on rpcgssd
> but
> > > will soon
> > > >find out. Any suggestions please? :)
> > >
> > > Depends on nfs-utils version? I remember there had been a
> > change in
> > > configuration in upstream nfs-utils in 2019:
> > >
> > > commit c69875c8afdd877baf7139c0cd5241f70105cbd4
> > > Author: François Cami <fcami(a)redhat.com
> > <mailto:fcami@redhat.com> <mailto:fcami@redhat.com
> > <mailto:fcami@redhat.com>>>
> > > Date: Tue Feb 26 13:59:06 2019 +0100
> > >
> > > ipa-client-automount: handle NFS configuration file
> changes
> > >
> > > nfs-utils in Fedora 30 and later switched its
> configuration
> > > file from /etc/sysconfig/nfs to /etc/nfs.conf, providing a
> > > conversion service (nfs-convert.service) for upgrades.
> > > However, for new installs the original configuration file
> > > is missing. This change:
> > > * adds a tuple-based osinfo.version_number method to
> handle
> > > more kinds of OS versioning schemes
> > > * detects RHEL and Fedora versions with the the new
> nfs-utils
> > > behavior
> > > * avoids backing up the new NFS configuration file as we
> do
> > > not have to modify it.
> > >
> > > See: https://bugzilla.redhat.com/show_bug.cgi?id=1676981
> <https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugzil...>
> > >
> > > Fixes: https://pagure.io/freeipa/issue/7868
> <https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpagure...>
> > >
> > >
> > > >
> > > >Alfred
> > > >
> > > >On Thu, Jun 10, 2021 at 2:17 PM Rob Crittenden
> > <rcritten(a)redhat.com <mailto:rcritten@redhat.com>
> > > <mailto:rcritten@redhat.com <mailto:rcritten@redhat.com>>>
> wrote:
> > > >
> > > >> Alfred Victor wrote:
> > > >> > Thanks very much Rob et al! I believe we have found our
> root
> > > cause and
> > > >> > the fix. If you like I'll provide some more details after
> > we're
> > > done
> > > >> > with everything.
> > > >>
> > > >> Yes, knowing the cause would be great and could be helpful
> > to others!
> > > >>
> > > >> cheers
> > > >>
> > > >> rob
> > > >>
> > > >> >
> > > >> > Alfred
> > > >> >
> > > >> > On Thu, Jun 10, 2021 at 11:02 AM Rob Crittenden
> > > <rcritten(a)redhat.com <mailto:rcritten@redhat.com>
> > <mailto:rcritten@redhat.com <mailto:rcritten@redhat.com>>
> > > >> > <mailto:rcritten@redhat.com <mailto:rcritten@redhat.com>
> > <mailto:rcritten@redhat.com <mailto:rcritten@redhat.com>>>> wrote:
> > > >> >
> > > >> > Alfred Victor wrote:
> > > >> > > Hi all,
> > > >> > >
> > > >> > > Just curious if anyone has suggestions about that
> > please
> > > before I
> > > >> get
> > > >> > > going in a couple of hours with conversions to IPA
> > again?
> > > I did
> > > >> > the math
> > > >> > > and 1,097,471 log messages in 5 hours is about 60
> > times per
> > > >> second, so
> > > >> > > I'm gradually becoming more certain this is why we
> can
> > > only boot
> > > >> 20-30
> > > >> > > nodes at a time when we used to boot hundreds.
> However,
> > > this is
> > > >> still
> > > >> > > just a guess as I don't know the mechanism behind
> > why this
> > > >> interferes
> > > >> > > with IPA joins, some bottleneck with the KDC?
> > > >> >
> > > >> > It sure seems like this is a kerberized NFS request
> for a
> > > host that
> > > >> > doesn't provide it, or doesn't have an nfs principal.
> I
> > > think you'd
> > > >> need
> > > >> > to monitor the offending client to see what it is
> doing.
> > > >> >
> > > >> > If the KDC is being dramatically slowed down by this
> then
> > > yes, you
> > > >> could
> > > >> > see slower Apache performance because it needs to
> obtain
> > > tickets on
> > > >> > behalf of the user doing the join. Whether that would
> > represent
> > > >> itself
> > > >> > as a read timeout I don't know.
> > > >> >
> > > >> > rob
> > > >> >
> > > >> > >
> > > >> > > Alfred
> > > >> > >
> > > >> > > On Wed, Jun 9, 2021 at 3:49 PM Alfred Victor
> > > <alvic266(a)gmail.com <mailto:alvic266@gmail.com>
> > <mailto:alvic266@gmail.com <mailto:alvic266@gmail.com>>
> > > >> > <mailto:alvic266@gmail.com
> > <mailto:alvic266@gmail.com> <mailto:alvic266@gmail.com
> > <mailto:alvic266@gmail.com>>>
> > > >> > > <mailto:alvic266@gmail.com
> > <mailto:alvic266@gmail.com> <mailto:alvic266@gmail.com
> > <mailto:alvic266@gmail.com>>
> > > <mailto:alvic266@gmail.com <mailto:alvic266@gmail.com>
> > <mailto:alvic266@gmail.com <mailto:alvic266@gmail.com>>>>> wrote:
> > > >> > >
> > > >> > > Hi Rob,
> > > >> > >
> > > >> > > I have reduced that timeout and will tune it
> > further.
> > > >> > Regarding ISE
> > > >> > > errors, I think we can make the assumption that
> > this is
> > > >> > entirely an
> > > >> > > issue of the web timeouts, I haven't seen any
> > evidence
> > > >> > otherwise and
> > > >> > > will have another attempt at converting nodes
> > > tomorrow, and
> > > >> with a
> > > >> > > keener eye of what to look for I can make a
> better
> > > >> determination
> > > >> > > then. I am most concerned over what the
> underlying
> > > cause might
> > > >> be
> > > >> > > causing it to take too long and hit the
> > timeout, and
> > > don't
> > > >> want to
> > > >> > > engineer around this by changing Apache
> timeouts if
> > > we can
> > > >> instead
> > > >> > > address the root cause. I am suspicious of krb
> log
> > > messages
> > > >> > flooding
> > > >> > > our IPA systems about a service principal like
> so,
> > > but not
> > > >> sure if
> > > >> > > this is gumming up the works (or even why this
> > > message has
> > > >> started
> > > >> > > appearing since the rebuild):
> > > >> > >
> > > >> > > ./krb5kdc.log:Jun 09 10:38:51
> > redacted.redacted.com
> <https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fredacte...>
> <http://redacted.redacted.com
> <https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fredacte...>
> >
> > > <http://redacted.redacted.com
> <https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fredacte...>
> >
> > > >> > <http://redacted.redacted.com
> <https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fredacte...>
> >
> > <http://redacted.redacted.com
> <https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fredacte...>
> >
> > > >> > krb5kdc[31187](info): TGS_REQ (4 etypes {18 17 16 23})
> > > 10.1.1.27
> > > >> > <http://10.1.1.27
> <https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2F10.1.1....>>:
> LOOKING_UP_SERVER: authtime 0,
> > > >> > host/redacted.redacted.com(a)redacted.COM for
> > > >> > nfs/nfsserver.redacted.com(a)redacted.COM, Server not
> > found in
> > > >> > Kerberos database
> > > >> > >
> > > >> > >
> > > >> > > Just in the last 5 hours alone, this log
> > message and
> > > others
> > > >> > like it (main difference is just the nodename it
> > > references) has
> > > >> > appeared 1,097,471 times. Conceivably there is also
> some
> > > log write
> > > >> > locking or something going on that could be slowing
> IPA
> > > down and
> > > >> > leading to our symptom here?
> > > >> > >
> > > >> > > Alfred
> > > >> > >
> > > >> > >
> > > >> > > On Wed, Jun 9, 2021 at 9:19 AM Rob Crittenden
> > > >> > <rcritten(a)redhat.com <mailto:rcritten@redhat.com>
> > <mailto:rcritten@redhat.com <mailto:rcritten@redhat.com>>
> > > <mailto:rcritten@redhat.com <mailto:rcritten@redhat.com>
> > <mailto:rcritten@redhat.com <mailto:rcritten@redhat.com>>>
> > > >> > > <mailto:rcritten@redhat.com
> > <mailto:rcritten@redhat.com>
> > > <mailto:rcritten@redhat.com <mailto:rcritten@redhat.com>>
> > <mailto:rcritten@redhat.com <mailto:rcritten@redhat.com>
> > > <mailto:rcritten@redhat.com <mailto:rcritten@redhat.com>>>>>
> > > >> wrote:
> > > >> > >
> > > >> > > Alfred Victor wrote:
> > > >> > > > Hi Rob,
> > > >> > > >
> > > >> > > > We did revert to 60s - I seem to remember
> > some
> > > ldapsearch
> > > >> > > timing out
> > > >> > > > previously but maybe we could still
> greatly
> > > reduce this
> > > >> with
> > > >> > > no ill
> > > >> > > > effect. However, we saw no change in join
> > > success either
> > > >> way
> > > >> > > and I have
> > > >> > > > not changed anything in Apache as I would
> > need
> > > to find
> > > >> the
> > > >> > > exact values
> > > >> > > > in question and I think these directive
> > changes
> > > may get
> > > >> lost
> > > >> > > with an
> > > >> > > > update? The timeout/ISE issues are new -
> we
> > > previously
> > > >> > booted
> > > >> > > many nodes
> > > >> > > > concurrently (400+) without problems but
> > now it
> > > happens
> > > >> even
> > > >> > > booting as
> > > >> > > > few as 50 results in 10-20 timeouts or
> ISE.
> > > This has
> > > >> > been the
> > > >> > > case at
> > > >> > > > least since we rebuilt and re-replicated
> the
> > > environment
> > > >> by
> > > >> > > swapping IPA
> > > >> > > > members out of the old one, but maybe
> this is
> > > unrelated.
> > > >> > Is it
> > > >> > > possible
> > > >> > > > we landed on a newer version with
> > different timeout
> > > >> > values? Is it
> > > >> > > > possible that IPA is under slightly more
> load
> > > from the
> > > >> > higher
> > > >> > > number of
> > > >> > > > nodes and has some bottleneck since we
> > last did
> > > >> conversions?
> > > >> > > We have not
> > > >> > > > been able to substantiate the latter by
> > looking at
> > > >> > > CPU/memory/io trends.
> > > >> > > > What might we investigate next to see if
> > we may
> > > have
> > > >> missed
> > > >> > > some ongoing
> > > >> > > > issue that could for instance cause
> locking
> > > problems or
> > > >> > > something else
> > > >> > > > internally in IPA to explain our symptoms?
> > > Also, I have
> > > >> > > observed some
> > > >> > > > nodes which report a successful
> > installation of IPA
> > > >> client,
> > > >> > > but have in
> > > >> > > > fact a lot of failures, for instance with
> > > mounts not
> > > >> > working from
> > > >> > > > automount setup. We will need to try to
> > > reproduce this to
> > > >> > > understand
> > > >> > > > what happened I think as we have already
> > sorted
> > > those
> > > >> nodes
> > > >> > > elsewhere
> > > >> > > > and they have got going outside of IPA.
> > > >> > > >
> > > >> > > > I am interested that you suggested "at
> least
> > > some of the
> > > >> > > clients aren't
> > > >> > > > connecting at all and increasing the
> timeout
> > > could make
> > > >> this
> > > >> > > worse" -
> > > >> > > > this might indicate some sort of network
> > problem at
> > > >> > play, but
> > > >> > > as far as
> > > >> > > > I am aware, everything is working absent
> > IPA so
> > > I do not
> > > >> > > suspect it
> > > >> > > > presently.
> > > >> > >
> > > >> > > A 60 second LDAP timeout is still way too
> big.
> > > The default
> > > >> > is 2.
> > > >> > > Unless
> > > >> > > you are seeing timeouts I'd suggest
> > lowering it.
> > > >> > >
> > > >> > > I'm only seeing hints to what you're seeing
> and
> > > the scope
> > > >> so
> > > >> > > it's hard
> > > >> > > to make further suggestions. Are all the
> > internal
> > > errors
> > > >> the
> > > >> > > same, for
> > > >> > > example? Are some failing due to LDAP
> > timeouts or
> > > are they
> > > >> all
> > > >> > > wsgi read
> > > >> > > timeouts?
> > > >> > >
> > > >> > > The Apache request timeout can be
> configured in
> > > httpd.conf
> > > >> > which is
> > > >> > > independent of the config files that IPA
> > > currently writes
> > > >> > so it
> > > >> > > should
> > > >> > > survive upgrades.
> > > >> > >
> > > >> > > rob
> > > >> > > >
> > > >> > > > Alfred
> > > >> > > >
> > > >> > > > On Mon, Jun 7, 2021 at 4:42 PM Rob
> Crittenden
> > > >> > > <rcritten(a)redhat.com
> > <mailto:rcritten@redhat.com> <mailto:rcritten@redhat.com
> > <mailto:rcritten@redhat.com>>
> > > <mailto:rcritten@redhat.com <mailto:rcritten@redhat.com>
> > <mailto:rcritten@redhat.com <mailto:rcritten@redhat.com>>>
> > > >> > <mailto:rcritten@redhat.com
> > <mailto:rcritten@redhat.com> <mailto:rcritten@redhat.com
> > <mailto:rcritten@redhat.com>>
> > > <mailto:rcritten@redhat.com <mailto:rcritten@redhat.com>
> > <mailto:rcritten@redhat.com <mailto:rcritten@redhat.com>>>>
> > > >> > > > <mailto:rcritten@redhat.com
> > <mailto:rcritten@redhat.com>
> > > <mailto:rcritten@redhat.com <mailto:rcritten@redhat.com>>
> > <mailto:rcritten@redhat.com <mailto:rcritten@redhat.com>
> > > <mailto:rcritten@redhat.com <mailto:rcritten@redhat.com>>>
> > > >> > <mailto:rcritten@redhat.com
> > <mailto:rcritten@redhat.com> <mailto:rcritten@redhat.com
> > <mailto:rcritten@redhat.com>>
> > > <mailto:rcritten@redhat.com <mailto:rcritten@redhat.com>
> > <mailto:rcritten@redhat.com <mailto:rcritten@redhat.com>>>>>> wrote:
> > > >> > > >
> > > >> > > > Alfred Victor wrote:
> > > >> > > > > Actually, no change happened from
> > 300-> 600
> > > >> > timeout, the
> > > >> > > web portal
> > > >> > > > > itself gave me an ISE I hadn't
> noticed
> > > when I tried
> > > >> > > clicking save!
> > > >> > > >
> > > >> > > > I wasn't clear which log to look in.
> > You'll see
> > > >> details
> > > >> > > about where the
> > > >> > > > error is caught in IPA in the Apache
> > log.
> > > To see
> > > >> LDAP
> > > >> > > timeouts you look
> > > >> > > > for err=3 in the 389-ds access log.
> > > >> > > >
> > > >> > > > But since you posted a traceback this
> > time it's
> > > >> > clear that
> > > >> > > this is just
> > > >> > > > Apache waiting for client data to
> read so
> > > any tuning
> > > >> you
> > > >> > > do needs to be
> > > >> > > > in Apache. You could try tuning
> > Timeout which
> > > >> > defaults to
> > > >> > > 60 but this
> > > >> > > > doesn't seem likely to help since at
> > least
> > > some of
> > > >> the
> > > >> > > clients aren't
> > > >> > > > connecting at all and increasing the
> > > timeout could
> > > >> make
> > > >> > > this worse.
> > > >> > > >
> > > >> > > > Please revert the searchtimeout. 300
> > seconds is
> > > >> > orders of
> > > >> > > magnitude
> > > >> > > > too big.
> > > >> > > >
> > > >> > > > rob
> > > >> > > >
> > > >> > > > >
> > > >> > > > > Alfred
> > > >> > > > >
> > > >> > > > > On Mon, Jun 7, 2021 at 3:57 PM
> > Alfred Victor
> > > >> > > <alvic266(a)gmail.com
> > <mailto:alvic266@gmail.com> <mailto:alvic266@gmail.com
> > <mailto:alvic266@gmail.com>>
> > > <mailto:alvic266@gmail.com <mailto:alvic266@gmail.com>
> > <mailto:alvic266@gmail.com <mailto:alvic266@gmail.com>>>
> > > >> > <mailto:alvic266@gmail.com
> > <mailto:alvic266@gmail.com> <mailto:alvic266@gmail.com
> > <mailto:alvic266@gmail.com>>
> > > <mailto:alvic266@gmail.com <mailto:alvic266@gmail.com>
> > <mailto:alvic266@gmail.com <mailto:alvic266@gmail.com>>>>
> > > >> > > > <mailto:alvic266@gmail.com
> > <mailto:alvic266@gmail.com>
> > > <mailto:alvic266@gmail.com <mailto:alvic266@gmail.com>>
> > > >> > <mailto:alvic266@gmail.com
> > <mailto:alvic266@gmail.com> <mailto:alvic266@gmail.com
> > <mailto:alvic266@gmail.com>>>
> > > <mailto:alvic266@gmail.com <mailto:alvic266@gmail.com>
> > <mailto:alvic266@gmail.com <mailto:alvic266@gmail.com>>
> > > >> > <mailto:alvic266@gmail.com
> > <mailto:alvic266@gmail.com> <mailto:alvic266@gmail.com
> > <mailto:alvic266@gmail.com>>>>>
> > > >> > > > > <mailto:alvic266@gmail.com
> > <mailto:alvic266@gmail.com>
> > > <mailto:alvic266@gmail.com <mailto:alvic266@gmail.com>>
> > > >> > <mailto:alvic266@gmail.com
> > <mailto:alvic266@gmail.com> <mailto:alvic266@gmail.com
> > <mailto:alvic266@gmail.com>>>
> > > <mailto:alvic266@gmail.com <mailto:alvic266@gmail.com>
> > <mailto:alvic266@gmail.com <mailto:alvic266@gmail.com>>
> > > >> > <mailto:alvic266@gmail.com
> > <mailto:alvic266@gmail.com> <mailto:alvic266@gmail.com
> > <mailto:alvic266@gmail.com>>>>
> > > >> > > <mailto:alvic266@gmail.com
> > <mailto:alvic266@gmail.com>
> > > <mailto:alvic266@gmail.com <mailto:alvic266@gmail.com>>
> > <mailto:alvic266@gmail.com <mailto:alvic266@gmail.com>
> > > <mailto:alvic266@gmail.com <mailto:alvic266@gmail.com>>>
> > > >> > <mailto:alvic266@gmail.com
> > <mailto:alvic266@gmail.com> <mailto:alvic266@gmail.com
> > <mailto:alvic266@gmail.com>>
> > > <mailto:alvic266@gmail.com <mailto:alvic266@gmail.com>
> > <mailto:alvic266@gmail.com <mailto:alvic266@gmail.com>>>>>>> wrote:
> > > >> > > > >
> > > >> > > > > Hi FreeIPA list,
> > > >> > > > >
> > > >> > > > > I don't see any in error log
> that
> > > match `grep
> > > >> > -i "err=3"
> > > >> > > > > /var/log/httpd/error_log`. We
> > have tried
> > > >> raising
> > > >> > > > searchtimelimit as
> > > >> > > > > high as 120, then 300 (now are
> > trying
> > > 600) but
> > > >> > > observed no
> > > >> > > > > difference in the rate at which
> > nodes
> > > >> succeeded or
> > > >> > > failed in IPA
> > > >> > > > > joins. We are somewhat puzzled
> by
> > > this, as
> > > >> none of
> > > >> > > the other
> > > >> > > > values
> > > >> > > > > we are aware of might have
> changed,
> > > though it
> > > >> is
> > > >> > > possible that the
> > > >> > > > > IPA systems are under a little
> > higher
> > > demand
> > > >> > from client
> > > >> > > > systems, we
> > > >> > > > > have tried to mitigate this by
> > > shutting down
> > > >> some
> > > >> > > workflows and
> > > >> > > > > aren't sure whether we've seen
> any
> > > improvement.
> > > >> > > Short of adjusting
> > > >> > > > > apache/resource/process
> > timeouts it is
> > > >> > difficult to
> > > >> > > say what might
> > > >> > > > > be wrong. To give an example,
> > out of
> > > 250 nodes
> > > >> > > rebooted, only 112
> > > >> > > > > joined IPA successfully. Here
> > is some
> > > output
> > > >> from
> > > >> > > the error log,
> > > >> > > > > following what this looks like
> > in the ipa
> > > >> client
> > > >> > > install log
> > > >> > > > (error
> > > >> > > > > log output will match the node
> > attempt):
> > > >> > > > >
> > > >> > > > >
> > > >> > > > > 2021-06-07T18:25:30Z DEBUG The
> > > >> ipa-client-install
> > > >> > > command
> > > >> > > > failed, exception: NetworkError:
> cannot
> > > connect to
> > > >> > > > 'https://redactednode.com/ipa/json
> <https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fredact...>
> > > >> > > >
> > <https://hauth0004.dug.com/ipa/json
> <https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fhauth0...>>':
> Internal
> > > >> > Server Error
> > > >> > > > >
> > > >> > > > > [Mon Jun 07 13:25:06.198259
> 2021]
> > > [core:error]
> > > >> > [pid
> > > >> > > 25020]
> > > >> > > > [client 10.1.24.48:47808
> <https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2F10.1.24...>
> > <http://10.1.24.48:47808
> <https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2F10.1.24...>
> >
> > > <http://10.1.24.48:47808
> <https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2F10.1.24...>>
> <http://10.1.24.48:47808
> <https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2F10.1.24...>
> >
> > > >> > <http://10.1.24.48:47808
> <https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2F10.1.24...>
> >
> > > >> > > <http://10.1.24.48:47808
> <https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2F10.1.24...>
> >
> > > >> > > > <http://10.1.24.48:47808
> <https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2F10.1.24...>>]
> Script
> > timed out
> > > before
> > > >> > returning
> > > >> > > > headers: wsgi.py, referer:
> > > >> > > https://redacted.redacted.com/ipa/xml
> <https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fredact...>
> > > >> > > > <https://hauth0004.dug.com/ipa/xml
> <https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fhauth0...>
> >
> > > >> > > > >
> > > >> > > > > Different node, same time
> period:
> > > >> > > > >
> > > >> > > > >
> > > >> > > > > [Mon Jun 07 13:24:02.178092
> 2021]
> > > [:error] [pid
> > > >> > > 25725] ipa:
> > > >> > > > INFO: [xmlserver]
> mach_join(a)redacted.COM:
> > > >> > > join(u'redacted.node.com
> <https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fredacte...>
> > <http://redacted.node.com
> <https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fredacte...>
> >
> > > <http://redacted.node.com
> <https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fredacte...>>
> <http://redacted.node.com
> <https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fredacte...>
> >
> > > >> > <http://redacted.node.com
> <https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fredacte...>
> >
> > > >> > > > <http://redacted.node.com
> <https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fredacte...>
> >
> > > <http://redacted.node.com
> <https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fredacte...>
> > > >> >',
> > > >> > > > nshardwareplatform=u'x86_64',
> > > >> > > >
> > > >> nsosversion=u'3.10.0-1062.18.1.1.el7.redacted.x86_64',
> > > >> > > > version=u'2.51'): TimeLimitExceeded
> > > >> > > > >
> > > >> > > > > I also saw this:
> > > >> > > > >
> > > >> > > > > [Mon Jun 07 13:25:07.103503
> 2021]
> > > [:error] [pid
> > > >> > > 25725] ipa:
> > > >> > > > ERROR: non-public: IOError: request
> data
> > > read error
> > > >> > > > > [Mon Jun 07 13:25:07.103529
> 2021]
> > > [:error]
> > > >> > [pid 25725]
> > > >> > > > Traceback (most recent call last):
> > > >> > > > > [Mon Jun 07 13:25:07.103536
> 2021]
> > > [:error] [pid
> > > >> > > 25725] File
> > > >> > > >
> > > >> >
> > "/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py",
> > > >> > > line 360,
> > > >> > > > in wsgi_execute
> > > >> > > > > [Mon Jun 07 13:25:07.103542
> 2021]
> > > [:error] [pid
> > > >> > > 25725]
> > > >> > > > data = read_input(environ)
> > > >> > > > > [Mon Jun 07 13:25:07.103548
> 2021]
> > > [:error] [pid
> > > >> > > 25725] File
> > > >> > > >
> > > >> >
> > "/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py",
> > > >> > > line 200,
> > > >> > > > in read_input
> > > >> > > > > [Mon Jun 07 13:25:07.103553
> 2021]
> > > [:error] [pid
> > > >> > > 25725]
> > > >> > > > return
> > > >> > environ['wsgi.input'].read(length).decode('utf-8')
> > > >> > > > > [Mon Jun 07 13:25:07.103559
> 2021]
> > > [:error]
> > > >> > [pid 25725]
> > > >> > > > IOError: request data read error
> > > >> > > > > [Mon Jun 07 13:25:07.103826
> 2021]
> > > [:error] [pid
> > > >> > > 25725] ipa:
> > > >> > > > INFO: [xmlserver]
> > mach_join(a)redacted.COM: None:
> > > >> > InternalError
> > > >> > > > > [Mon Jun 07 13:25:07.149962
> 2021]
> > > [:error] [pid
> > > >> > > 25726] ipa:
> > > >> > > > ERROR: non-public: IOError: request
> data
> > > read error
> > > >> > > > > [Mon Jun 07 13:25:07.149984
> 2021]
> > > [:error]
> > > >> > [pid 25726]
> > > >> > > > Traceback (most recent call last):
> > > >> > > > > [Mon Jun 07 13:25:07.149991
> 2021]
> > > [:error] [pid
> > > >> > > 25726] File
> > > >> > > >
> > > >> >
> > "/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py",
> > > >> > > line 360,
> > > >> > > > in wsgi_execute
> > > >> > > > > [Mon Jun 07 13:25:07.149997
> 2021]
> > > [:error] [pid
> > > >> > > 25726]
> > > >> > > > data = read_input(environ)
> > > >> > > > > [Mon Jun 07 13:25:07.150002
> 2021]
> > > [:error] [pid
> > > >> > > 25726] File
> > > >> > > >
> > > >> >
> > "/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py",
> > > >> > > line 200,
> > > >> > > > in read_input
> > > >> > > > > [Mon Jun 07 13:25:07.150008
> 2021]
> > > [:error] [pid
> > > >> > > 25726]
> > > >> > > > return
> > > >> > environ['wsgi.input'].read(length).decode('utf-8')
> > > >> > > > > [Mon Jun 07 13:25:07.150013
> 2021]
> > > [:error]
> > > >> > [pid 25726]
> > > >> > > > IOError: request data read error
> > > >> > > > >
> > > >> > > > >
> > > >> > > > >
> > > >> > > > > After setting the timeout to
> > 600 and
> > > rebooting
> > > >> the
> > > >> > > remaining
> > > >> > > > 139 nodes from the initial set of
> 250, 83
> > > joined of
> > > >> the
> > > >> > > 139 and we
> > > >> > > > still had ISE occurring. In some
> > cases, it
> > > would ISE
> > > >> on
> > > >> > > the first
> > > >> > > > attempt, try another IPA system, and
> > > succeed. I'm
> > > >> > not sure
> > > >> > > that even
> > > >> > > > such a long timeout as 600 has helped.
> > > >> > > > >
> > > >> > > > > Alfred
> > > >> > > > >
> > > >> > > > >
> > > >> > > > >
> > > >> > > > >
> > > >> > > > > On Thu, Jun 3, 2021 at 7:51 PM
> Rob
> > > Crittenden
> > > >> > > > <rcritten(a)redhat.com
> > <mailto:rcritten@redhat.com>
> > > <mailto:rcritten@redhat.com <mailto:rcritten@redhat.com>>
> > <mailto:rcritten@redhat.com <mailto:rcritten@redhat.com>
> > > <mailto:rcritten@redhat.com <mailto:rcritten@redhat.com>>>
> > > >> > <mailto:rcritten@redhat.com
> > <mailto:rcritten@redhat.com> <mailto:rcritten@redhat.com
> > <mailto:rcritten@redhat.com>>
> > > <mailto:rcritten@redhat.com <mailto:rcritten@redhat.com>
> > <mailto:rcritten@redhat.com <mailto:rcritten@redhat.com>>>>
> > > >> > > <mailto:rcritten@redhat.com
> > <mailto:rcritten@redhat.com>
> > > <mailto:rcritten@redhat.com <mailto:rcritten@redhat.com>>
> > <mailto:rcritten@redhat.com <mailto:rcritten@redhat.com>
> > > <mailto:rcritten@redhat.com <mailto:rcritten@redhat.com>>>
> > > >> > <mailto:rcritten@redhat.com
> > <mailto:rcritten@redhat.com> <mailto:rcritten@redhat.com
> > <mailto:rcritten@redhat.com>>
> > > <mailto:rcritten@redhat.com <mailto:rcritten@redhat.com>
> > <mailto:rcritten@redhat.com <mailto:rcritten@redhat.com>>>>>
> > > >> > > > > <mailto:rcritten@redhat.com
> > <mailto:rcritten@redhat.com>
> > > <mailto:rcritten@redhat.com <mailto:rcritten@redhat.com>>
> > > >> > <mailto:rcritten@redhat.com
> > <mailto:rcritten@redhat.com> <mailto:rcritten@redhat.com
> > <mailto:rcritten@redhat.com>>>
> > > >> > > <mailto:rcritten@redhat.com
> > <mailto:rcritten@redhat.com>
> > > <mailto:rcritten@redhat.com <mailto:rcritten@redhat.com>>
> > <mailto:rcritten@redhat.com <mailto:rcritten@redhat.com>
> > > <mailto:rcritten@redhat.com <mailto:rcritten@redhat.com>>>>
> > > >> > <mailto:rcritten@redhat.com
> > <mailto:rcritten@redhat.com> <mailto:rcritten@redhat.com
> > <mailto:rcritten@redhat.com>>
> > > <mailto:rcritten@redhat.com <mailto:rcritten@redhat.com>
> > <mailto:rcritten@redhat.com <mailto:rcritten@redhat.com>>>
> > > >> > > <mailto:rcritten@redhat.com
> > <mailto:rcritten@redhat.com>
> > > <mailto:rcritten@redhat.com <mailto:rcritten@redhat.com>>
> > > >> > <mailto:rcritten@redhat.com <mailto:
> rcritten(a)redhat.com>
> > > <mailto:rcritten@redhat.com <mailto:rcritten@redhat.com>>>>>>>
> > wrote:
> > > >> > > > >
> > > >> > > > > Alfred Victor via
> FreeIPA-users
> > > wrote:
> > > >> > > > > > Hi FreeIPA list,
> > > >> > > > > >
> > > >> > > > > > We are having an issue
> > with our IPA
> > > >> > > environment of 4
> > > >> > > > > replicated FreeIPA
> > > >> > > > > > systems serving linux
> compute
> > > clients
> > > >> > > which join from a
> > > >> > > > command in
> > > >> > > > > > rc.local after boot. This
> > > worked in the
> > > >> > past,
> > > >> > > but the system
> > > >> > > > > has been
> > > >> > > > > > rebuilt since and the
> > join command
> > > >> changed
> > > >> > > > slightly. Unfortunately
> > > >> > > > > > booting a few dozen nodes
> > at a
> > > time,
> > > >> though
> > > >> > > they each
> > > >> > > > talk to a
> > > >> > > > > > different IPA system by
> > design,
> > > leads to
> > > >> > > problems such as
> > > >> > > > > these - though
> > > >> > > > > > 40-100 nodes can boot ok
> at a
> > > time there
> > > >> are
> > > >> > > always many
> > > >> > > > > stragglers, and
> > > >> > > > > > the more we attempt to
> > boot at
> > > once the
> > > >> more
> > > >> > > fail to
> > > >> > > > join IPA
> > > >> > > > > (if we try
> > > >> > > > > > to boot 500 nodes, we are
> > lucky
> > > if we
> > > >> get a
> > > >> > > fifth of that
> > > >> > > > > joining IPA).
> > > >> > > > > > Can you please advise on
> this
> > > output?
> > > >> > Here is
> > > >> > > our join
> > > >> > > > command in
> > > >> > > > > > compute node rc.local:
> > > >> > > > > >
> > > >> > > > > >
> > > >> > > > > >
> > > >> > > > > > ipa-client-install -U
> > -q -p
> > > >> mach_join \
> > > >> > > > > > -w <redacted> \
> > > >> > > > > > --force-join \
> > > >> > > > > > --no-dns-sshfp \
> > > >> > > > > >
> > > --automount-location=redacted-node
> > > >> > > > > >
> > > >> > > > > >
> > > >> > > > > >
> > > >> > > > > >
> > > >> > > > > > And here is some log
> > output of
> > > the 500
> > > >> > error:
> > > >> > > > > >
> > > >> > > > > > ProtocolError:
> > > <ProtocolError for
> > > >> > > > >
> > redacted.redacted.com/ipa/json
> <https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fredacte...>
> <http://redacted.redacted.com/ipa/json
> <https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fredacte...>
> >
> > > <http://redacted.redacted.com/ipa/json
> <https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fredacte...>
> >
> > > >> > <http://redacted.redacted.com/ipa/json
> <https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fredacte...>
> >
> > > >> > > <http://redacted.redacted.com/ipa/json
> <https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fredacte...>
> >
> > > >> > > > <
> http://redacted.redacted.com/ipa/json
> <https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fredacte...>
> >
> > > >> > > > >
> > > <http://redacted.redacted.com/ipa/json
> <https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fredacte...>
> >
> > > >> > > > >
> > > <http://redacted.redacted.com/ipa/json
> <https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fredacte...>
> >:
> > > >> 500
> > > >> > > Internal
> > > >> > > > Server Error>
> > > >> > > > > > Cannot connect to the
> > > server due to
> > > >> > > generic error:
> > > >> > > > cannot
> > > >> > > > > connect to
> > > >> > 'https://redacted.redacted.com/ipa/json
> <https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fredact...>
> > > >> > > > >
> > > <https://hauth0003.dug.com/ipa/json
> <https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fhauth0...>
> >':
> > > >> > Internal
> > > >> > > Server Error
> > > >> > > > > >
> > > >> > > > > >
> > > >> > > > > > As well as:
> > > >> > > > > >
> > > >> > > > > > 2021-06-02T21:39:11Z
> > DEBUG
> > > Starting
> > > >> > > external process
> > > >> > > > > > 2021-06-02T21:39:11Z
> > DEBUG
> > > >> > > args=/usr/sbin/ipa-join -s
> > > >> > > > > > redacted.redacted.com
> <https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fredacte...>
> > <http://redacted.redacted.com
> <https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fredacte...>
> >
> > > <http://redacted.redacted.com
> <https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fredacte...>
> >
> > > >> > <http://redacted.redacted.com
> <https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fredacte...>
> >
> > > >> > > <http://redacted.redacted.com
> <https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fredacte...>>
> <
> > > >> http://redacted.redacted.com
> <https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fredacte...>
> >
> > > >> > > > <http://redacted.redacted.com
> <https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fredacte...>
> >
> > > >> > > > >
> > <http://redacted.redacted.com
> <https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fredacte...>>
> -b
> > > >> > > > > > dc=redacted,dc=com -h
> > > >> > > redactednode.redacted.com
> <https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fredacte...>
> > <http://redactednode.redacted.com
> <https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fredacte...>
> >
> > > <http://redactednode.redacted.com
> <https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fredacte...>
> >
> > > >> > <http://redactednode.redacted.com
> <https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fredacte...>
> >
> > > <http://redactednode.redacted.com
> <https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fredacte...>
> > > >> >
> > > >> > > > <http://redactednode.redacted.com
> <https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fredacte...>
> >
> > > >> > > > >
> > <http://redactednode.redacted.com
> <https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fredacte...>
> >
> > > >> > > > > >
> > > <http://redactednode.redacted.com
> <https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fredacte...>
> >
> > > >> -f
> > > >> > > > > > 2021-06-02T21:40:13Z
> > DEBUG
> > > Process
> > > >> > > finished, return
> > > >> > > > code=17
> > > >> > > > > > 2021-06-02T21:40:13Z
> > DEBUG
> > > stdout=
> > > >> > > > > > 2021-06-02T21:40:13Z
> > DEBUG
> > > >> stderr=HTTP
> > > >> > > response code is
> > > >> > > > > 500, not 200
> > > >> > > > > > 2021-06-02T21:40:13Z
> > ERROR
> > > Joining
> > > >> realm
> > > >> > > failed: HTTP
> > > >> > > > > response code
> > > >> > > > > > is 500, not 200
> > > >> > > > > >
> > > >> > > > > > And we also see timeouts
> > happen:
> > > >> > > > > >
> > > >> > > > > >
> > > >> > > > > > 2021-06-02T22:08:50Z
> > DEBUG
> > > >> > > args=/usr/sbin/ipa-join -s
> > > >> > > > > > redacted.redacted.com
> <https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fredacte...>
> > <http://redacted.redacted.com
> <https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fredacte...>
> >
> > > <http://redacted.redacted.com
> <https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fredacte...>
> >
> > > >> > <http://redacted.redacted.com
> <https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fredacte...>
> >
> > > >> > > <http://redacted.redacted.com
> <https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fredacte...>>
> <
> > > >> http://redacted.redacted.com
> <https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fredacte...>
> >
> > > >> > > > <http://redacted.redacted.com
> <https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fredacte...>
> >
> > > >> > > > >
> > <http://redacted.redacted.com
> <https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fredacte...>>
> -b
> > > >> > > > > > dc=redacted,dc=com -h
> > > >> > > redactednode.redacted.com
> <https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fredacte...>
> > <http://redactednode.redacted.com
> <https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fredacte...>
> >
> > > <http://redactednode.redacted.com
> <https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fredacte...>
> >
> > > >> > <http://redactednode.redacted.com
> <https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fredacte...>
> >
> > > <http://redactednode.redacted.com
> <https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fredacte...>
> > > >> >
> > > >> > > > <http://redactednode.redacted.com
> <https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fredacte...>
> >
> > > >> > > > >
> > <http://redactednode.redacted.com
> <https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fredacte...>
> >
> > > >> > > > > >
> > > <http://redactednode.redacted.com
> <https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fredacte...>
> >
> > > >> -f
> > > >> > > > > > 2021-06-02T22:09:01Z
> > DEBUG
> > > Process
> > > >> > > finished, return
> > > >> > > > code=17
> > > >> > > > > > 2021-06-02T22:09:01Z
> > DEBUG
> > > stdout=
> > > >> > > > > > 2021-06-02T22:09:01Z
> > DEBUG
> > > stderr=RPC
> > > >> > > failed at server.
> > > >> > > > > Configured
> > > >> > > > > > time limit exceeded
> > > >> > > > > > 2021-06-02T22:09:01Z
> > ERROR
> > > Joining
> > > >> realm
> > > >> > > failed: RPC
> > > >> > > > failed at
> > > >> > > > > > server. Configured
> > time limit
> > > >> exceeded
> > > >> > > > > >
> > > >> > > > > >
> > > >> > > > > > And we also see later
> > timeouts
> > > near the
> > > >> > end of
> > > >> > > the log
> > > >> > > > in some
> > > >> > > > �� > cases though are able to
> > > authenticate and
> > > >> it
> > > >> > > didn't back
> > > >> > > > out the
> > > >> > > > > install, but never got
> > going healthy
> > > >> either:
> > > >> > > > > >
> > > >> > > > > >
> > > >> > > > > >
> > > >> > > > > > 2021-06-03T19:20:13Z
> > DEBUG The
> > > >> > > ipa-client-install
> > > >> > > > command
> > > >> > > > > failed,
> > > >> > > > > > exception:
> > TimeLimitExceeded:
> > > >> Configured
> > > >> > > time limit
> > > >> > > > exceeded
> > > >> > > > >
> > > >> > > > > When you see Internal Error
> > look
> > > to the
> > > >> Apache
> > > >> > > error log
> > > >> > > > on the
> > > >> > > > > server
> > > >> > > > > for more information.
> > > >> > > > >
> > > >> > > > > In this case an LDAP search
> is
> > > failing
> > > >> because
> > > >> > > the server
> > > >> > > > is too
> > > >> > > > > busy.
> > > >> > > > > Look for queries failing
> with
> > > err=3 to get
> > > >> an
> > > >> > > idea of how long
> > > >> > > > > it is taking.
> > > >> > > > >
> > > >> > > > > To increase the timeout
> > use: ipa
> > > config-mod
> > > >> > > > --searchtimelimit=INT
> > > >> > > > >
> > > >> > > > > The default is 2 seconds.
> > > >> > > > >
> > > >> > > > > You can pick a time at
> > random but
> > > could see
> > > >> > > failures again.
> > > >> > > > >
> > > >> > > > > rob
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > > >>
> > >
> > >
> > >
> > >
> > > --
> > > / Alexander Bokovoy
> > > Sr. Principal Software Engineer
> > > Security / Identity Management Engineering
> > > Red Hat Limited, Finland
> > >
> >
>
>
2 years, 10 months
Simulate the AD 'Computer' Object
by Joseph Fry
I support a product that natively supports ingesting computers from Active Directory via an LDAP query.
We are able to ingest users from FreeIPA (actually Redhat IDM) without a problem, but hosts cannot be ingested because the objectClass and attributes are different than what is used by ActiveDirectory.
Is there a simple solution in FreeIPA to have the host objects simulate an AD 'computer' object?
2 years, 10 months
Join command 500 errors, timeouts
by Alfred Victor
Hi FreeIPA list,
We are having an issue with our IPA environment of 4 replicated FreeIPA
systems serving linux compute clients which join from a command in rc.local
after boot. This worked in the past, but the system has been rebuilt since
and the join command changed slightly. Unfortunately booting a few dozen
nodes at a time, though they each talk to a different IPA system by design,
leads to problems such as these - though 40-100 nodes can boot ok at a time
there are always many stragglers, and the more we attempt to boot at once
the more fail to join IPA (if we try to boot 500 nodes, we are lucky if we
get a fifth of that joining IPA). Can you please advise on this output?
Here is our join command in compute node rc.local:
ipa-client-install -U -q -p mach_join \
> -w <redacted> \
> --force-join \
> --no-dns-sshfp \
> --automount-location=redacted-node
And here is some log output of the 500 error:
ProtocolError: <ProtocolError for redacted.redacted.com/ipa/json: 500
Internal Server Error>
> Cannot connect to the server due to generic error: cannot connect to 'https://redacted.redacted.com/ipa/json <https://hauth0003.dug.com/ipa/json>': Internal Server Error
>
>
As well as:
2021-06-02T21:39:11Z DEBUG Starting external process
> 2021-06-02T21:39:11Z DEBUG args=/usr/sbin/ipa-join -s
> redacted.redacted.com -b dc=redacted,dc=com -h redactednode.redacted.com
> -f
> 2021-06-02T21:40:13Z DEBUG Process finished, return code=17
> 2021-06-02T21:40:13Z DEBUG stdout=
> 2021-06-02T21:40:13Z DEBUG stderr=HTTP response code is 500, not 200
> 2021-06-02T21:40:13Z ERROR Joining realm failed: HTTP response code is
> 500, not 200
And we also see timeouts happen:
>
> 2021-06-02T22:08:50Z DEBUG args=/usr/sbin/ipa-join -s redacted.redacted.com -b dc=redacted,dc=com -h redactednode.redacted.com -f
> 2021-06-02T22:09:01Z DEBUG Process finished, return code=17
> 2021-06-02T22:09:01Z DEBUG stdout=
> 2021-06-02T22:09:01Z DEBUG stderr=RPC failed at server. Configured time limit exceeded
> 2021-06-02T22:09:01Z ERROR Joining realm failed: RPC failed at server. Configured time limit exceeded
And we also see later timeouts near the end of the log in some cases though
are able to authenticate and it didn't back out the install, but never got
going healthy either:
>
> 2021-06-03T19:20:13Z DEBUG The ipa-client-install command failed,
> exception: TimeLimitExceeded: Configured time limit exceeded
Donald
2 years, 10 months
ipactl status hangs
by Ronald Wimmer
On some servers I can see that ipactl status hangs from time to time. I
can see SSSD reporting the backend to be offline and dirsrv not running
for some reason. ipactl -d status reveals that it hangs when issuing
systemctl is-active dirsrv. Instead of hanging I would expect ipcactl
status to report a problem (= dirsrv not running).
Cheers,
Ronald
2 years, 10 months
