DNS resolution failures

[Tania Hagan]

Hi Freeipa-users,

We are currently running Freeipa version 4.9.11 on Rocky 8.8.  

We have noticed over the last few months that external name resolution e.g. google.com
fails to resolve on multiple Freeipa replicas even though the service named-pkcs11 remains
up and running and journalctl or logs aren’t showing up any obvious errors to why this
might be happening.  We temporarily fix this by restarting the service, but the problem
comes back at random times.  

We currently have 39 DNS Zones 

Our DNS Global Configuration has a forward policy of forward only, though the individual
zones are set to forward first. 

I’ve read a few articles that say maybe changing the forward policy might fix it, but
nothing that mentions how to double check if changing the policy will fix it.  

Are there any useful troubleshooting checks I could run to either help explain why our
service keeps failing at random intervals or confirm any changes would fix the issue
without the risk of potential downtime of our DNS service?

Many Thanks, 
Tania

[David Harvey]

Just checking if there are any suggestions as to how to debug this
effectively. The lack of smoking barrel log entries we've seen with it have
left us a little stumped!
Thanks as always,
David

On Wed, 17 Jan 2024 at 10:54, Tania Hagan via FreeIPA-users <
freeipa-users(a)lists.fedorahosted.org&gt; wrote:

>  Hi Freeipa-users,
>
>  We are currently running Freeipa version 4.9.11 on Rocky 8.8.
>
>  We have noticed over the last few months that external name resolution
>  e.g. google.com fails to resolve on multiple Freeipa replicas even though
>  the service named-pkcs11 remains up and running and journalctl or logs
>  aren’t showing up any obvious errors to why this might be happening.  We
>  temporarily fix this by restarting the service, but the problem comes back
>  at random times.
>
>  We currently have 39 DNS Zones
>
>  Our DNS Global Configuration has a forward policy of forward only, though
>  the individual zones are set to forward first.
>
>  I’ve read a few articles that say maybe changing the forward policy might
>  fix it, but nothing that mentions how to double check if changing the
>  policy will fix it.
>
>  Are there any useful troubleshooting checks I could run to either help
>  explain why our service keeps failing at random intervals or confirm any
>  changes would fix the issue without the risk of potential downtime of our
>  DNS service?
>
>  Many Thanks,
>  Tania
>  --
>  _______________________________________________
>  FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
>  To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org
>  Fedora Code of Conduct:
>  https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
>  https://lists.fedorahosted.org/archives/list/[e-mail]...
>  Do not reply to spam, report it:
>  https://pagure.io/fedora-infrastructure/new_issue
>

[Natxo Asenjo]

hi,

a bit late, but you should check the forwarding logs (maybe enable them,
bit unsure if it is enabled per default on named).

Without any proof, my gut feeling is on dnssec :-), I have had to turn it
off a few times.

Regards,

Natxo Asenjo

On Tue, Jan 30, 2024 at 5:11 PM David Harvey via FreeIPA-users <
freeipa-users(a)lists.fedorahosted.org&gt; wrote:

>  Just checking if there are any suggestions as to how to debug this
>  effectively. The lack of smoking barrel log entries we've seen with it have
>  left us a little stumped!
>  Thanks as always,
>  David
>
>  On Wed, 17 Jan 2024 at 10:54, Tania Hagan via FreeIPA-users <
>  freeipa-users(a)lists.fedorahosted.org&gt; wrote:
>
> > Hi Freeipa-users,
> >
> > We are currently running Freeipa version 4.9.11 on Rocky 8.8.
> >
> > We have noticed over the last few months that external name resolution
> > e.g. google.com fails to resolve on multiple Freeipa replicas even
> > though the service named-pkcs11 remains up and running and journalctl or
> > logs aren’t showing up any obvious errors to why this might be happening.
> > We temporarily fix this by restarting the service, but the problem comes
> > back at random times.
> >
> > We currently have 39 DNS Zones
> >
> > Our DNS Global Configuration has a forward policy of forward only, though
> > the individual zones are set to forward first.
> >
> > I’ve read a few articles that say maybe changing the forward policy might
> > fix it, but nothing that mentions how to double check if changing the
> > policy will fix it.
> >
> > Are there any useful troubleshooting checks I could run to either help
> > explain why our service keeps failing at random intervals or confirm any
> > changes would fix the issue without the risk of potential downtime of our
> > DNS service?
> >
> > Many Thanks,
> > Tania
> > --
> > _______________________________________________
> > FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
> > To unsubscribe send an email to
> > freeipa-users-leave(a)lists.fedorahosted.org
> > Fedora Code of Conduct:
> > https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives:
> > https://lists.fedorahosted.org/archives/list/[e-mail]...
> > Do not reply to spam, report it:
> > https://pagure.io/fedora-infrastructure/new_issue
> >
>  --
>  _______________________________________________
>  FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
>  To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org
>  Fedora Code of Conduct:
>  https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
>  https://lists.fedorahosted.org/archives/list/[e-mail]...
>  Do not reply to spam, report it:
>  https://pagure.io/fedora-infrastructure/new_issue
>  

-- 
--
Groeten,
natxo

[Harry G Coin]

On 2/17/24 00:54, Natxo Asenjo via FreeIPA-users wrote:
>  hi,
>
>  a bit late, but you should check the forwarding logs (maybe enable 
>  them, bit unsure if it is enabled per default on named).
>
>  Without any proof, my gut feeling is on dnssec :-), I have had to turn 
>  it off a few times.
>
>  Regards,
>
>  Natxo Asenjo
>
>  On Tue, Jan 30, 2024 at 5:11 PM David Harvey via FreeIPA-users 
>  <freeipa-users(a)lists.fedorahosted.org&gt; wrote:
>
>      Just checking if there are any suggestions as to how to debug this
>      effectively. The lack of smoking barrel log entries we've seen
>      with it have left us a little stumped!
>      Thanks as always,
>      David
>
>      On Wed, 17 Jan 2024 at 10:54, Tania Hagan via FreeIPA-users
>      <freeipa-users(a)lists.fedorahosted.org&gt; wrote:
>
>          Hi Freeipa-users,
>
>          We are currently running Freeipa version 4.9.11 on Rocky 8.8.
>
>          We have noticed over the last few months that external name
>          resolution e.g. google.com <http://google.com> fails to
>          resolve on multiple Freeipa replicas even though the service
>          named-pkcs11 remains up and running and journalctl or logs
>          aren’t showing up any obvious errors to why this might be
>          happening.  We temporarily fix this by restarting the service,
>          but the problem comes back at random times.
>
>          We currently have 39 DNS Zones
>
>          Our DNS Global Configuration has a forward policy of forward
>          only, though the individual zones are set to forward first.
>
>          I’ve read a few articles that say maybe changing the forward
>          policy might fix it, but nothing that mentions how to double
>          check if changing the policy will fix it.
>
>          Are there any useful troubleshooting checks I could run to
>          either help explain why our service keeps failing at random
>          intervals or confirm any changes would fix the issue without
>          the risk of potential downtime of our DNS service?
>
>          Many Thanks,
>          Tania
>          --
>  
If, after booting you wait an hour, and it seems to be normal for no 
good reason, maybe we're hitting the same bug.   I notice, each boot, 
the logs are filled with entries about 'notify' messages during which 
resolution requests time out.  My hunch is it's either some overlap of 
ip4/ip6 causing a race condition, or DNSSEC rr records holding some lock 
until they're all updated.  But that's just a hunch why turning dnssec 
off leads to normal resolution services.

Harry

[Harry G Coin]

I've had the same experience.  Adding memory and processing power seems 
to have avoided the bug.  Before I did that I found I had to restart 
bind9/named.  And yes the logs are entirely silent about the cause, the 
system reported free memory available even before I added it.   My guess 
is some unresolved race condition or peak memory use left dnydb-ldap 
crippled.  I generally find it has to do with replication issues 
generating side effects elsewhere.


On 1/17/24 04:54, Tania Hagan via FreeIPA-users wrote:
> Hi Freeipa-users,
>
> We are currently running Freeipa version 4.9.11 on Rocky 8.8.
>
> We have noticed over the last few months that external name resolution e.g.
google.com fails to resolve on multiple Freeipa replicas even though the service
named-pkcs11 remains up and running and journalctl or logs aren’t showing up any obvious
errors to why this might be happening.  We temporarily fix this by restarting the service,
but the problem comes back at random times.
>
> We currently have 39 DNS Zones
>
> Our DNS Global Configuration has a forward policy of forward only, though the
individual zones are set to forward first.
>
> I’ve read a few articles that say maybe changing the forward policy might fix it, but
nothing that mentions how to double check if changing the policy will fix it.
>
> Are there any useful troubleshooting checks I could run to either help explain why
our service keeps failing at random intervals or confirm any changes would fix the issue
without the risk of potential downtime of our DNS service?
>
> Many Thanks,
> Tania
> --
> _______________________________________________
> FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
> To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org
> Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
https://lists.fedorahosted.org/archives/list/[e-mail]...
> Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue