broken trust chain resolving using 8.8.8.8 as forwarder

[Rob van Halteren]

Hi,
I have trouble resolving some addresses with my freeipa server . in the log there are lots of "broken trust chain" lines. like:

validating gew4-spclient.spotify.com/CNAME: bad cache hit (com/DS)
May  3 14:36:11 myserver named-pkcs11[30906]: validating gew4-spclient.spotify.com/CNAME: bad cache hit (com/DS)
May  3 14:36:11 myserver named-pkcs11[30906]: broken trust chain resolving 'gew4-spclient.spotify.com/A/IN': 8.8.8.8#53
May  3 14:36:11 myserver named-pkcs11[30906]: broken trust chain resolving 'gew4-spclient.spotify.com/TYPE65/IN': 8.8.8.8#53

I setup a global forward to 8.8.8.8 and forward only setting in the web gui.

I tried to change the dnssec settings in /etc/named.conf :  dnssec-enable no;      dnssec-validation no;
That did not help.

I run freeipa 4.6.8. Release: 5.el7.centos.12 on centos7.9

When I change forwarding to: forward disabled in the webgui, i get lots of "network unreachable resolving" in the logs.
I then can resolve most addresses but not all

To me looks like dns is not resolving as expected, but have no clue in where to look for a solution.

Any help appreciated.

Thanks,

Rob.

[Alexander Bokovoy]

On ke, 03 touko 2023, Rob van Halteren via FreeIPA-users wrote:
> Hi,
> I have trouble resolving some addresses with my freeipa server . in the log there are lots of "broken trust chain" lines. like:
>
> validating gew4-spclient.spotify.com/CNAME: bad cache hit (com/DS)
> May  3 14:36:11 myserver named-pkcs11[30906]: validating gew4-spclient.spotify.com/CNAME: bad cache hit (com/DS)
> May  3 14:36:11 myserver named-pkcs11[30906]: broken trust chain resolving 'gew4-spclient.spotify.com/A/IN': 8.8.8.8#53
> May  3 14:36:11 myserver named-pkcs11[30906]: broken trust chain resolving 'gew4-spclient.spotify.com/TYPE65/IN': 8.8.8.8#53
>
> I setup a global forward to 8.8.8.8 and forward only setting in the web gui.
>
> I tried to change the dnssec settings in /etc/named.conf :  dnssec-enable no;      dnssec-validation no;
> That did not help.
>
> I run freeipa 4.6.8. Release: 5.el7.centos.12 on centos7.9
>
> When I change forwarding to: forward disabled in the webgui, i get lots of "network unreachable resolving" in the logs.
> I then can resolve most addresses but not all
>
> To me looks like dns is not resolving as expected, but have no clue in where to look for a solution.

spotify.com isn't signed correctly. You can see this with 'delv'
utility: https://kb.isc.org/docs/aa-01152

$ delv @8.8.8.8 gew4-spclient.spotify.com +vtrust +multi
;; fetch: gew4-spclient.spotify.com/A
;; validating gew4-spclient.spotify.com/CNAME: starting
;; validating gew4-spclient.spotify.com/CNAME: attempting insecurity proof
;; validating gew4-spclient.spotify.com/CNAME: checking existence of DS at 'com'
;; fetch: com/DS
;; validating com/DS: starting
;; validating com/DS: attempting positive response validation
;; fetch: ./DNSKEY
;; validating ./DNSKEY: starting
;; validating ./DNSKEY: attempting positive response validation
;; validating ./DNSKEY: verify rdataset (keyid=20326): success
;; validating ./DNSKEY: marking as secure (DS)
;; validating com/DS: in fetch_callback_dnskey
;; validating com/DS: keyset with trust secure
;; validating com/DS: resuming validate
;; validating com/DS: verify rdataset (keyid=60955): success
;; validating com/DS: marking as secure, noqname proof not needed
;; validating gew4-spclient.spotify.com/CNAME: in fetch_callback_ds
;; validating gew4-spclient.spotify.com/CNAME: resuming proveunsecure
;; validating gew4-spclient.spotify.com/CNAME: checking existence of DS at 'spotify.com'
;; fetch: spotify.com/DS
;; validating spotify.com/DS: starting
;; validating spotify.com/DS: attempting negative response validation from message
;;   validating CK0POJMG874LJREF7EFN8430QVIT8BSM.com/NSEC3: starting
;;   validating CK0POJMG874LJREF7EFN8430QVIT8BSM.com/NSEC3: attempting positive response validation
;; fetch: com/DNSKEY
;; validating com/DNSKEY: starting
;; validating com/DNSKEY: attempting positive response validation
;; validating com/DNSKEY: verify rdataset (keyid=30909): success
;; validating com/DNSKEY: marking as secure (DS)
;;   validating CK0POJMG874LJREF7EFN8430QVIT8BSM.com/NSEC3: in fetch_callback_dnskey
;;   validating CK0POJMG874LJREF7EFN8430QVIT8BSM.com/NSEC3: keyset with trust secure
;;   validating CK0POJMG874LJREF7EFN8430QVIT8BSM.com/NSEC3: resuming validate
;;   validating CK0POJMG874LJREF7EFN8430QVIT8BSM.com/NSEC3: verify rdataset (keyid=46551): success
;;   validating CK0POJMG874LJREF7EFN8430QVIT8BSM.com/NSEC3: marking as secure, noqname proof not needed
;; validating spotify.com/DS: in validator_callback_nsec
;; validating spotify.com/DS: resuming validate_nx
;;   validating com/SOA: starting
;;   validating com/SOA: attempting positive response validation
;;   validating com/SOA: keyset with trust secure
;;   validating com/SOA: verify rdataset (keyid=46551): success
;;   validating com/SOA: marking as secure, noqname proof not needed
;; validating spotify.com/DS: in validator_callback_nsec
;; validating spotify.com/DS: resuming validate_nx
;;   validating VB2HUP9O0DVURL5ULM1QVE6079MMRS5P.com/NSEC3: starting
;;   validating VB2HUP9O0DVURL5ULM1QVE6079MMRS5P.com/NSEC3: attempting positive response validation
;;   validating VB2HUP9O0DVURL5ULM1QVE6079MMRS5P.com/NSEC3: keyset with trust secure
;;   validating VB2HUP9O0DVURL5ULM1QVE6079MMRS5P.com/NSEC3: verify rdataset (keyid=46551): success
;;   validating VB2HUP9O0DVURL5ULM1QVE6079MMRS5P.com/NSEC3: marking as secure, noqname proof not needed
;; validating spotify.com/DS: in validator_callback_nsec
;; validating spotify.com/DS: resuming validate_nx
;; validating spotify.com/DS: looking for relevant NSEC3
;; validating spotify.com/DS: looking for relevant NSEC3
;; validating spotify.com/DS: looking for relevant NSEC3
;; validating spotify.com/DS: NSEC3 indicates potential closest encloser: 'com'
;; validating spotify.com/DS: NSEC3 at super-domain com
;; validating spotify.com/DS: looking for relevant NSEC3
;; validating spotify.com/DS: NSEC3 proves name does not exist: 'spotify.com'
;; validating spotify.com/DS: NSEC3 indicates optout
;; validating spotify.com/DS: in checkwildcard: *.com
;; validating spotify.com/DS: looking for relevant NSEC3
;; validating spotify.com/DS: NSEC3 at super-domain com
;; validating spotify.com/DS: looking for relevant NSEC3
;; validating spotify.com/DS: in checkwildcard: *.com
;; validating spotify.com/DS: nonexistence proof(s) found
;; validating gew4-spclient.spotify.com/CNAME: in fetch_callback_ds
;; validating gew4-spclient.spotify.com/CNAME: marking as answer (fetch_callback_ds)
;; fetch: edge-web-gew4.dual-gslb.spotify.com/A
;; validating edge-web-gew4.dual-gslb.spotify.com/A: starting
;; validating edge-web-gew4.dual-gslb.spotify.com/A: attempting insecurity proof
;; validating edge-web-gew4.dual-gslb.spotify.com/A: checking existence of DS at 'com'
;; validating edge-web-gew4.dual-gslb.spotify.com/A: checking existence of DS at 'spotify.com'
;; validating edge-web-gew4.dual-gslb.spotify.com/A: marking as answer (proveunsecure (4))
; unsigned answer
gew4-spclient.spotify.com. 31	IN	CNAME	edge-web-gew4.dual-gslb.spotify.com.
edge-web-gew4.dual-gslb.spotify.com. 58	IN A	35.186.224.17

[Rob van Halteren]

Hi Alexander,

Do you mean that forwarding is actually working correct but that addresses with log entry “broken trust chain resolving ‘addres’ are most likely sites that have dnssec issues ?
I have lots of entry’s like that in my log.

Regards,

ROB VAN HALTEREN
AV | IT System Engineer
Entrepotdok 66
NL-1018 AD Amsterdam
T: [phone]

Out of office on Monday's
www.filmmore.eu http://www.filmmore.eu/
 http://www.filmmore.eu/
 http://www.imdb.com/company/co0190130/ http://twitter.com/filmmore http://www.facebook.com/FilmmoreINT/ http://www.linkedin.com/company/filmmore-amsterdam/
> On 3 May 2023, at 16:55, Alexander Bokovoy  wrote:
>
> On ke, 03 touko 2023, Rob van Halteren via FreeIPA-users wrote:
> > Hi,
> > I have trouble resolving some addresses with my freeipa server . in the log there are lots of "broken trust chain" lines. like:
> >
> > validating gew4-spclient.spotify.com/CNAME: bad cache hit (com/DS)
> > May  3 14:36:11 myserver named-pkcs11[30906]: validating gew4-spclient.spotify.com/CNAME: bad cache hit (com/DS)
> > May  3 14:36:11 myserver named-pkcs11[30906]: broken trust chain resolving 'gew4-spclient.spotify.com/A/IN': 8.8.8.8#53
> > May  3 14:36:11 myserver named-pkcs11[30906]: broken trust chain resolving 'gew4-spclient.spotify.com/TYPE65/IN': 8.8.8.8#53
> >
> > I setup a global forward to 8.8.8.8 and forward only setting in the web gui.
> >
> > I tried to change the dnssec settings in /etc/named.conf :  dnssec-enable no;      dnssec-validation no;
> > That did not help.
> >
> > I run freeipa 4.6.8. Release: 5.el7.centos.12 on centos7.9
> >
> > When I change forwarding to: forward disabled in the webgui, i get lots of "network unreachable resolving" in the logs.
> > I then can resolve most addresses but not all
> >
> > To me looks like dns is not resolving as expected, but have no clue in where to look for a solution.
>
> spotify.com isn't signed correctly. You can see this with 'delv'
> utility: https://kb.isc.org/docs/aa-01152
>
> $ delv @8.8.8.8 gew4-spclient.spotify.com +vtrust +multi
> ;; fetch: gew4-spclient.spotify.com/A
> ;; validating gew4-spclient.spotify.com/CNAME: starting
> ;; validating gew4-spclient.spotify.com/CNAME: attempting insecurity proof
> ;; validating gew4-spclient.spotify.com/CNAME: checking existence of DS at 'com'
> ;; fetch: com/DS
> ;; validating com/DS: starting
> ;; validating com/DS: attempting positive response validation
> ;; fetch: ./DNSKEY
> ;; validating ./DNSKEY: starting
> ;; validating ./DNSKEY: attempting positive response validation
> ;; validating ./DNSKEY: verify rdataset (keyid=20326): success
> ;; validating ./DNSKEY: marking as secure (DS)
> ;; validating com/DS: in fetch_callback_dnskey
> ;; validating com/DS: keyset with trust secure
> ;; validating com/DS: resuming validate
> ;; validating com/DS: verify rdataset (keyid=60955): success
> ;; validating com/DS: marking as secure, noqname proof not needed
> ;; validating gew4-spclient.spotify.com/CNAME: in fetch_callback_ds
> ;; validating gew4-spclient.spotify.com/CNAME: resuming proveunsecure
> ;; validating gew4-spclient.spotify.com/CNAME: checking existence of DS at 'spotify.com'
> ;; fetch: spotify.com/DS
> ;; validating spotify.com/DS: starting
> ;; validating spotify.com/DS: attempting negative response validation from message
> ;;   validating CK0POJMG874LJREF7EFN8430QVIT8BSM.com/NSEC3: starting
> ;;   validating CK0POJMG874LJREF7EFN8430QVIT8BSM.com/NSEC3: attempting positive response validation
> ;; fetch: com/DNSKEY
> ;; validating com/DNSKEY: starting
> ;; validating com/DNSKEY: attempting positive response validation
> ;; validating com/DNSKEY: verify rdataset (keyid=30909): success
> ;; validating com/DNSKEY: marking as secure (DS)
> ;;   validating CK0POJMG874LJREF7EFN8430QVIT8BSM.com/NSEC3: in fetch_callback_dnskey
> ;;   validating CK0POJMG874LJREF7EFN8430QVIT8BSM.com/NSEC3: keyset with trust secure
> ;;   validating CK0POJMG874LJREF7EFN8430QVIT8BSM.com/NSEC3: resuming validate
> ;;   validating CK0POJMG874LJREF7EFN8430QVIT8BSM.com/NSEC3: verify rdataset (keyid=46551): success
> ;;   validating CK0POJMG874LJREF7EFN8430QVIT8BSM.com/NSEC3: marking as secure, noqname proof not needed
> ;; validating spotify.com/DS: in validator_callback_nsec
> ;; validating spotify.com/DS: resuming validate_nx
> ;;   validating com/SOA: starting
> ;;   validating com/SOA: attempting positive response validation
> ;;   validating com/SOA: keyset with trust secure
> ;;   validating com/SOA: verify rdataset (keyid=46551): success
> ;;   validating com/SOA: marking as secure, noqname proof not needed
> ;; validating spotify.com/DS: in validator_callback_nsec
> ;; validating spotify.com/DS: resuming validate_nx
> ;;   validating VB2HUP9O0DVURL5ULM1QVE6079MMRS5P.com/NSEC3: starting
> ;;   validating VB2HUP9O0DVURL5ULM1QVE6079MMRS5P.com/NSEC3: attempting positive response validation
> ;;   validating VB2HUP9O0DVURL5ULM1QVE6079MMRS5P.com/NSEC3: keyset with trust secure
> ;;   validating VB2HUP9O0DVURL5ULM1QVE6079MMRS5P.com/NSEC3: verify rdataset (keyid=46551): success
> ;;   validating VB2HUP9O0DVURL5ULM1QVE6079MMRS5P.com/NSEC3: marking as secure, noqname proof not needed
> ;; validating spotify.com/DS: in validator_callback_nsec
> ;; validating spotify.com/DS: resuming validate_nx
> ;; validating spotify.com/DS: looking for relevant NSEC3
> ;; validating spotify.com/DS: looking for relevant NSEC3
> ;; validating spotify.com/DS: looking for relevant NSEC3
> ;; validating spotify.com/DS: NSEC3 indicates potential closest encloser: 'com'
> ;; validating spotify.com/DS: NSEC3 at super-domain com
> ;; validating spotify.com/DS: looking for relevant NSEC3
> ;; validating spotify.com/DS: NSEC3 proves name does not exist: 'spotify.com'
> ;; validating spotify.com/DS: NSEC3 indicates optout
> ;; validating spotify.com/DS: in checkwildcard: *.com
> ;; validating spotify.com/DS: looking for relevant NSEC3
> ;; validating spotify.com/DS: NSEC3 at super-domain com
> ;; validating spotify.com/DS: looking for relevant NSEC3
> ;; validating spotify.com/DS: in checkwildcard: *.com
> ;; validating spotify.com/DS: nonexistence proof(s) found
> ;; validating gew4-spclient.spotify.com/CNAME: in fetch_callback_ds
> ;; validating gew4-spclient.spotify.com/CNAME: marking as answer (fetch_callback_ds)
> ;; fetch: edge-web-gew4.dual-gslb.spotify.com/A
> ;; validating edge-web-gew4.dual-gslb.spotify.com/A: starting
> ;; validating edge-web-gew4.dual-gslb.spotify.com/A: attempting insecurity proof
> ;; validating edge-web-gew4.dual-gslb.spotify.com/A: checking existence of DS at 'com'
> ;; validating edge-web-gew4.dual-gslb.spotify.com/A: checking existence of DS at 'spotify.com'
> ;; validating edge-web-gew4.dual-gslb.spotify.com/A: marking as answer (proveunsecure (4))
> ; unsigned answer
> gew4-spclient.spotify.com. 31	IN	CNAME	edge-web-gew4.dual-gslb.spotify.com.
> edge-web-gew4.dual-gslb.spotify.com. 58	IN A	35.186.224.17
>
> -- 
> / Alexander Bokovoy
> Sr. Principal Software Engineer
> Security / Identity Management Engineering
> Red Hat Limited, Finland

[Alexander Bokovoy]

On ke, 03 touko 2023, Rob van Halteren wrote:
> Hi Alexander,
>
> Do you mean that forwarding is actually working correct but that
> addresses with log entry “broken trust chain resolving ‘addres’ are
> most likely sites that have dnssec issues ?  I have lots of entry’s
> like that in my log.

Correct. DNSSEC support across multiple DNS zones on the Internet is
patchy, so to say.  DNSSEC validation is often failing due to this or
that zone intermediaries or misconfigurations. That's why BIND has
separate options to enable/disable DNSSEC validation.

Spotify is simply not providing DNSSEC signatures for its own zone:
https://dnsviz.net/d/spotify.com/dnssec/