Debugging hint re: replica 'rndc sign' fail -> ipa-dnskeysyncd fail

[Harry G. Coin]

I was going to ask for help for a very perplexing problem.  The symptoms
seemed to have very little to do with the solution, so searching online
led nowhere.  Hopefully, by posting this the next person to hit this
will find this answer.

In short, the answer is, on the replica:

dsconf -D "cn=Directory Manager" ldap://localhost config replace
nsslapd-ioblocktimeout=40000

And here is the shaggy dog story description of the problem, mostly for
the benefit of search engines.

I need(ed) help with where to look next for a strange replica-dns
blocking freeipa problem.

Using fedora 34 latest with a fresh simple master/replica install of
ipa-server with dns: I'm getting normal operations on the master and

ipa-dnskeysyncd.service: Main process exited, code=exited, status=1/FAILURE

on the replica.  The master has good dnssec operation on a few zones as
reported on the master (you can check for example quietfountain.com
yourself).  But on the replica it fails to reply at all.

ipa-healthcheck reports the same thing on both the slave an the replica,
what appears to be a known and mostly ignored problem to do with KRA not
working on the replica.

[root@registry1 etc]# ipa-healthcheck
Internal error testing KRA clone. KRA clone problem detected  Host:
registry2.1.quietfountain.com Port: 443
[
  {
    "source": "pki.server.healthcheck.clones.connectivity_and_data",
    "check": "ClonesConnectivyAndDataCheck",
    "result": "ERROR",
    "uuid": "7cd8344e-1fff-45c8-a613-893a1baf5cf9",
    "when": "20210827172132Z",
    "duration": "11.141044",
    "kw": {
      "status": "ERROR:  pki-tomcat : Internal error testing KRA clone.
Host: registry2.1.quietfountain.com Port: 443"
    }
  }
]
[root@registry1 etc]#

On both the master and the replica, all services are running:

*[root@registry1 ~]# systemctl is-system-running
running
*

[root@registry2 ~]# systemctl is-system-running
running

Now, here's the clue I need help with where to go from here:


On the master we have this normal set of two commands:

[root@registry1 ~]# rndc status
version: BIND 9.16.20-RH (Extended Support Version) <id:26db37f>
running on registry1.1.quietfountain.com: Linux x86_64
5.13.12-200.fc34.x86_64 #1 SMP Wed Aug 18 13:27:18 UTC 2021
boot time: Fri, 27 Aug 2021 17:55:31 GMT
last configured: Fri, 27 Aug 2021 17:55:33 GMT
configuration file: /etc/named.conf
CPUs found: 4
worker threads: 4
UDP listeners per interface: 4
number of zones: 301 (94 automatic)
debug level: 0
xfers running: 0
xfers deferred: 0
soa queries in progress: 0
query logging is ON
recursive clients: 0/900/1000
tcp clients: 0/150
TCP high-water: 11
server is up and running
[root@registry1 ~]# rndc zonestatus quietfountain.com.
name: quietfountain.com.
type: primary
files: dyndb-ldap/ipa/master/quietfountain.com/raw
serial: 2021082484
signed serial: 2021082494
nodes: 29
last loaded: Fri, 27 Aug 2021 17:57:06 GMT
secure: yes
inline signing: yes
key maintenance: automatic
next key event: Fri, 27 Aug 2021 18:59:22 GMT
next resign node: quietfountain.com/NS
next resign time: Sat, 28 Aug 2021 04:20:04 GMT
dynamic: yes
frozen: no
reconfigurable via modzone: no
[root@registry1 ~]#

But on the replica... it fails really oddly, and the failure breaks
ipa-dnskeysyncd

[root@registry2 ~]# rndc status
version: BIND 9.16.20-RH (Extended Support Version) <id:26db37f>
running on registry2.1.quietfountain.com: Linux x86_64
5.13.12-200.fc34.x86_64 #1 SMP Wed Aug 18 13:27:18 UTC 2021
boot time: Fri, 27 Aug 2021 17:43:59 GMT
last configured: Fri, 27 Aug 2021 17:44:01 GMT
configuration file: /etc/named.conf
CPUs found: 4
worker threads: 4
UDP listeners per interface: 4
number of zones: 302 (95 automatic)
debug level: 0
xfers running: 0
xfers deferred: 0
soa queries in progress: 0
query logging is ON
recursive clients: 9/900/1000
tcp clients: 0/150
TCP high-water: 5
server is up and running
[root@registry2 ~]# rndc zonestatus quietfountain.com
rndc: 'zonestatus' failed: not found
no matching zone 'quietfountain.com' in any view
[root@registry2 ~]#


What really gets me is, if you point a dig at each named daemon for the
'allegedly missing zone' -- both the master and the replica report
correctly. 

So how is it possible for rndc to fail to find a domain when that same
named process quite plainly can give details about it? 

The only other clue I have is: the master is quite obviously the
authoritative name server for greekdishesbynausica.com (among others,
just ask dnssec analzers that report all good from the master but can't
reach the replica...) -- but the logs on the replica are filled with
"not authoritative"..  ??

Aug 27 12:59:33 registry2.1.quietfountain.com named[1036]: client
@0x7fee142ac528 10.12.127.252#42708: received notify for zone
'greekdishesbynausica.com': not authoritative



Here are the two 'digs':

[root@registry2 ~]# dig @10.12.112.3 DS quietfountain.com.

; <<>> DiG 9.16.20-RH <<>> @10.12.112.3 DS quietfountain.com.
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 65463
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
; COOKIE: bbe4c713019a8d940100000061292addc6ca238f90f7831f (good)
;; QUESTION SECTION:
;quietfountain.com.             IN      DS

;; ANSWER SECTION:
quietfountain.com.      86337   IN      DS      13711 8 2
89F5FD8758CB880C6DE8CCFF9971DEDAEC81BC33FA3CCE092781AD15 6F305F50

;; Query time: 1 msec
;; SERVER: 10.12.112.3#53(10.12.112.3)
;; WHEN: Fri Aug 27 13:11:41 CDT 2021
;; MSG SIZE  rcvd: 122

[root@registry2 ~]# dig @10.12.112.2 DS quietfountain.com.

; <<>> DiG 9.16.20-RH <<>> @10.12.112.2 DS quietfountain.com.
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 14869
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
; COOKIE: fe7fe619a7f0fb2b0100000061292ae263bec72b087ad2ce (good)
;; QUESTION SECTION:
;quietfountain.com.             IN      DS

;; ANSWER SECTION:
quietfountain.com.      86153   IN      DS      13711 8 2
89F5FD8758CB880C6DE8CCFF9971DEDAEC81BC33FA3CCE092781AD15 6F305F50

;; Query time: 2 msec
;; SERVER: 10.12.112.2#53(10.12.112.2)
;; WHEN: Fri Aug 27 13:11:46 CDT 2021
;; MSG SIZE  rcvd: 122

[root@registry2 ~]#