9:23 a.m.
https://fedorahosted.org/sssd/ticket/1966 (SSSD failover doesn't work if
the first DNS server in resolv.conf is unavailable).
The problem here is, that if first nameserver in resolv.conf is down,
the resolution is too slow and SSSD will not wait for the result of
ares_search and go offline. In my case the resolution was sometimes more
than a minute, because all search domains in resolv.conf were searched
inside ares_search() call using the first (not working) nameserver in
the first place and then with the working nameserver (and before that,
SSSD tried to figure out the domain name from my incorrectly set
hostname, which added more unnecessary DNS lookups).
To avoid this problem, the option dns_discovery_domain must be set
properly, so that only the correct domain is searched, but even that is
not enough, because the default timeout for dns resolver operation in
sssd is too low. This patch rises the default value to 15 seconds
(instead of 5 seconds).
Another option might be to lower the amount of time ares waits for a
nameserver to respond (currently it is 5 seconds, that is why 5 second
for the entire dns resolution is not sufficient), but I do not want to
do this.
These patches also change man pages, so probably master only (string
freeze)? Even if this is a really small change.
I was also thinking, would it make sense to write a warning to the logs
if the dns_discovery_domain option is not set? It seem to be important
to set it properly for cases like this one.
Thanks
Michal
12:45 p.m.
On Thu, 2013-07-04 at 16:23 +0200, Michal Židek wrote:
https://fedorahosted.org/sssd/ticket/1966 (SSSD failover doesn't
work if
the first DNS server in resolv.conf is unavailable).
The problem here is, that if first nameserver in resolv.conf is down,
the resolution is too slow and SSSD will not wait for the result of
ares_search and go offline. In my case the resolution was sometimes more
than a minute, because all search domains in resolv.conf were searched
inside ares_search() call using the first (not working) nameserver in
the first place and then with the working nameserver (and before that,
SSSD tried to figure out the domain name from my incorrectly set
hostname, which added more unnecessary DNS lookups).
To avoid this problem, the option dns_discovery_domain must be set
properly, so that only the correct domain is searched, but even that is
not enough, because the default timeout for dns resolver operation in
sssd is too low. This patch rises the default value to 15 seconds
(instead of 5 seconds).
Another option might be to lower the amount of time ares waits for a
nameserver to respond (currently it is 5 seconds, that is why 5 second
for the entire dns resolution is not sufficient), but I do not want to
do this.
Why not ?
5 seconds these days is quite awfully high all in all.
I think 1 second should be plenty in most cases, so maybe a default of 2
seconds should be considered.
These patches also change man pages, so probably master only (string
freeze)? Even if this is a really small change.
I was also thinking, would it make sense to write a warning to the logs
if the dns_discovery_domain option is not set? It seem to be important
to set it properly for cases like this one.
I wonder, should we use the machine name to automatically set the
discovery domain, and only if that fails fall back to the ones defined
in resolv.conf ?
Simo.
--
Simo Sorce * Red Hat, Inc * New York
2:43 p.m.
On 07/04/2013 07:45 PM, Simo Sorce wrote:
On Thu, 2013-07-04 at 16:23 +0200, Michal Židek wrote:
> https://fedorahosted.org/sssd/ticket/1966 (SSSD failover doesn't work if
> the first DNS server in resolv.conf is unavailable).
>
> The problem here is, that if first nameserver in resolv.conf is down,
> the resolution is too slow and SSSD will not wait for the result of
> ares_search and go offline. In my case the resolution was sometimes more
> than a minute, because all search domains in resolv.conf were searched
> inside ares_search() call using the first (not working) nameserver in
> the first place and then with the working nameserver (and before that,
> SSSD tried to figure out the domain name from my incorrectly set
> hostname, which added more unnecessary DNS lookups).
>
> To avoid this problem, the option dns_discovery_domain must be set
> properly, so that only the correct domain is searched, but even that is
> not enough, because the default timeout for dns resolver operation in
> sssd is too low. This patch rises the default value to 15 seconds
> (instead of 5 seconds).
>
> Another option might be to lower the amount of time ares waits for a
> nameserver to respond (currently it is 5 seconds, that is why 5 second
> for the entire dns resolution is not sufficient), but I do not want to
> do this.
Why not ?
5 seconds these days is quite awfully high all in all.
I think 1 second should be plenty in most cases, so maybe a default of 2
seconds should be considered.
I also think that 5 seconds is high value, but in general, I have bad
feeling when lowering not configurable timeouts (theoretically it could
cause some DNS servers that are reachable now to become unreachable).
Another possibility would be to make this value configurable. Then I
would have no problem lowering the default value. But, should this be
configurable? We already have one option to configure timeout for the
entire DNS resolution, adding another timeout for subtaks of this
operation does not sound nice to me. But I do not have strong opinion
on this.
Btw, once the correct data is returned from DNS servers, we no longer
require any DNS operation, so it only happens first time the service
is required (for example the first 'getent' call will be slow, but
after that the service is fast). This is not so bad IMO.
> These patches also change man pages, so probably master only
(string
> freeze)? Even if this is a really small change.
>
> I was also thinking, would it make sense to write a warning to the logs
> if the dns_discovery_domain option is not set? It seem to be important
> to set it properly for cases like this one.
I wonder, should we use the machine name to automatically set the
discovery domain, and only if that fails fall back to the ones defined
in resolv.conf ?
Actually, we do this. Wireshark shows that the first srv lookup was
constructed from my hostname. But the whole process is unclear to me,
for example I do not know why we call the resolv_gethostbyname_send()
inside of resolv_get_domain_send(). It adds another DNS lookups
for A records (which causes another slow-down in the case of bad
nameserver searched first). In wireshark I also see searches for A/AAA
records for names like:
mzidek.example.com
mzidek.example.com.searchdom1.com
mzidek.example.com.searchdom2.com
mzidek.example.com.searchdom3.com
where mzidek.example.com is my hostname and searchdom1/2/3 are search
domains specified in resolv.conf. Why do we even try to resolve these
names? It seems like waste of time to me. If the dns_discovery_domain is
properly set, these lookups do not appear.
Thanks
Michal
PS: Patches are the same as before.
Simo.
5:15 a.m.
On Mon, Jul 08, 2013 at 09:43:42PM +0200, Michal Židek wrote:
On 07/04/2013 07:45 PM, Simo Sorce wrote:
>On Thu, 2013-07-04 at 16:23 +0200, Michal Židek wrote:
>>https://fedorahosted.org/sssd/ticket/1966 (SSSD failover doesn't work if
>>the first DNS server in resolv.conf is unavailable).
>>
>>The problem here is, that if first nameserver in resolv.conf is down,
>>the resolution is too slow and SSSD will not wait for the result of
>>ares_search and go offline. In my case the resolution was sometimes more
>>than a minute, because all search domains in resolv.conf were searched
>>inside ares_search() call using the first (not working) nameserver in
>>the first place and then with the working nameserver (and before that,
>>SSSD tried to figure out the domain name from my incorrectly set
>>hostname, which added more unnecessary DNS lookups).
>>
>>To avoid this problem, the option dns_discovery_domain must be set
>>properly, so that only the correct domain is searched, but even that is
>>not enough, because the default timeout for dns resolver operation in
>>sssd is too low. This patch rises the default value to 15 seconds
>>(instead of 5 seconds).
>>
>>Another option might be to lower the amount of time ares waits for a
>>nameserver to respond (currently it is 5 seconds, that is why 5 second
>>for the entire dns resolution is not sufficient), but I do not want to
>>do this.
>
>Why not ?
>
>5 seconds these days is quite awfully high all in all.
>I think 1 second should be plenty in most cases, so maybe a default of 2
>seconds should be considered.
>
I also think that 5 seconds is high value, but in general, I have bad
feeling when lowering not configurable timeouts (theoretically it could
cause some DNS servers that are reachable now to become unreachable).
Right, but in practice I can't think of a valid network configuration
where the name server would take so long to reply. I mean, if the
network was so slow that just resolving name would take 5 seconds, then
chances are we wouldn't be able to download the entry from LDAP at all.
And if the nameserver is slow, then let's rather skip it and fail over
to another one.
Another possibility would be to make this value configurable. Then I
would have no problem lowering the default value. But, should this be
configurable? We already have one option to configure timeout for the
entire DNS resolution, adding another timeout for subtaks of this
operation does not sound nice to me. But I do not have strong opinion
on this.
Here is what we discussed with Michal today:
* make the per-nameserver timeout lower a little. Now it's 5
seconds, which is too long especially considering the failover
timeout is 5 seconds as well. We could change it to 2 or 3
seconds.
* the glibc allows 3 nameservers. The failover request timeut could
then default to 3*nameserver_timeout to give the resolver a chance
to iterate over all the configured name servers.
Btw, once the correct data is returned from DNS servers, we no
longer
require any DNS operation, so it only happens first time the service
is required (for example the first 'getent' call will be slow, but
after that the service is fast). This is not so bad IMO.
>>These patches also change man pages, so probably master only (string
>>freeze)? Even if this is a really small change.
We accept changes again now that we've released 1.10.1.
But this change would be OK even for a string-frozen release, because it
only changes the number, not translatable strings.
>>
>>I was also thinking, would it make sense to write a warning to the logs
>>if the dns_discovery_domain option is not set? It seem to be important
>>to set it properly for cases like this one.
>
>I wonder, should we use the machine name to automatically set the
>discovery domain, and only if that fails fall back to the ones defined
>in resolv.conf ?
Actually, we do this. Wireshark shows that the first srv lookup was
constructed from my hostname. But the whole process is unclear to me,
for example I do not know why we call the resolv_gethostbyname_send()
inside of resolv_get_domain_send(). It adds another DNS lookups
for A records (which causes another slow-down in the case of bad
nameserver searched first). In wireshark I also see searches for A/AAA
records for names like:
mzidek.example.com
mzidek.example.com.searchdom1.com
mzidek.example.com.searchdom2.com
mzidek.example.com.searchdom3.com
IIRC we try to canonicalize the host name before chopping of the first
part and using the rest as domain name.
Would using gethostname() be safe in real world environments?
I know that host names are often set to complete garbage and can't be
relied on..but is it OK to just say "fix your environment" in that case?
Personally I would argue the current canonicalization doesn't add much,
because if the host name is set to complete garbage then the canonicalization
wouldn't magically fix it..
where mzidek.example.com is my hostname and searchdom1/2/3 are search
domains specified in resolv.conf. Why do we even try to resolve these
names? It seems like waste of time to me. If the dns_discovery_domain is
properly set, these lookups do not appear.
Thanks
Michal
PS: Patches are the same as before.
>
>Simo.
>
From 1bb9dc2d17ec10c48860578ebec4dbe19cb166e6 Mon Sep 17 00:00:00
2001
From: Michal Zidek <mzidek(a)redhat.com>
Date: Thu, 4 Jul 2013 13:32:07 +0200
Subject: [PATCH 1/2] Set default DNS resolution timeout to 15 seconds.
Partially solves ticket:
https://fedorahosted.org/sssd/ticket/1966
To avoid the problem mentioned in the ticket above, option
dns_discovery_domain must be set properly.
---
src/providers/data_provider_fo.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/src/providers/data_provider_fo.c b/src/providers/data_provider_fo.c
index 83294b5..3d1de26 100644
--- a/src/providers/data_provider_fo.c
+++ b/src/providers/data_provider_fo.c
@@ -774,8 +774,8 @@ void be_fo_set_port_status(struct be_ctx *ctx,
/* Resolver back end interface */
static struct dp_option dp_res_default_opts[] = {
{ "lookup_family_order", DP_OPT_STRING, { "ipv4_first" },
NULL_STRING },
- { "dns_resolver_timeout", DP_OPT_NUMBER, { .number = 5 }, NULL_NUMBER },
- { "dns_resolver_op_timeout", DP_OPT_NUMBER, { .number = 5 }, NULL_NUMBER
},
+ { "dns_resolver_timeout", DP_OPT_NUMBER, { .number = 15 }, NULL_NUMBER },
+ { "dns_resolver_op_timeout", DP_OPT_NUMBER, { .number = 15 }, NULL_NUMBER
},
{ "dns_discovery_domain", DP_OPT_STRING, NULL_STRING, NULL_STRING },
DP_OPTION_TERMINATOR
};
--
1.7.11.2
From 97cf4e830d59ee04c1bd77cb73241d846144c681 Mon Sep 17 00:00:00
2001
From: Michal Zidek <mzidek(a)redhat.com>
Date: Thu, 4 Jul 2013 14:51:52 +0200
Subject: [PATCH 2/2] MAN: Changed default for dns_resolver_timeout
Changed default value of dns_resolver_timeout
from 5 to 15 seconds.
---
src/man/sssd.conf.5.xml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/src/man/sssd.conf.5.xml b/src/man/sssd.conf.5.xml
index 8df2bd9..a8b71ae 100644
--- a/src/man/sssd.conf.5.xml
+++ b/src/man/sssd.conf.5.xml
@@ -1656,7 +1656,7 @@ override_homedir = /home/%u
offline mode.
</para>
<para>
- Default: 5
+ Default: 15
</para>
</listitem>
</varlistentry>
--
1.7.11.2
_______________________________________________
sssd-devel mailing list
sssd-devel(a)lists.fedorahosted.org
https://lists.fedorahosted.org/mailman/listinfo/sssd-devel
11:06 a.m.
On 07/19/2013 12:15 PM, Jakub Hrozek wrote:
On Mon, Jul 08, 2013 at 09:43:42PM +0200, Michal Židek wrote:
> On 07/04/2013 07:45 PM, Simo Sorce wrote:
>> On Thu, 2013-07-04 at 16:23 +0200, Michal Židek wrote:
>>> https://fedorahosted.org/sssd/ticket/1966 (SSSD failover doesn't work if
>>> the first DNS server in resolv.conf is unavailable).
>>>
>>> The problem here is, that if first nameserver in resolv.conf is down,
>>> the resolution is too slow and SSSD will not wait for the result of
>>> ares_search and go offline. In my case the resolution was sometimes more
>>> than a minute, because all search domains in resolv.conf were searched
>>> inside ares_search() call using the first (not working) nameserver in
>>> the first place and then with the working nameserver (and before that,
>>> SSSD tried to figure out the domain name from my incorrectly set
>>> hostname, which added more unnecessary DNS lookups).
>>>
>>> To avoid this problem, the option dns_discovery_domain must be set
>>> properly, so that only the correct domain is searched, but even that is
>>> not enough, because the default timeout for dns resolver operation in
>>> sssd is too low. This patch rises the default value to 15 seconds
>>> (instead of 5 seconds).
>>>
>>> Another option might be to lower the amount of time ares waits for a
>>> nameserver to respond (currently it is 5 seconds, that is why 5 second
>>> for the entire dns resolution is not sufficient), but I do not want to
>>> do this.
>>
>> Why not ?
>>
>> 5 seconds these days is quite awfully high all in all.
>> I think 1 second should be plenty in most cases, so maybe a default of 2
>> seconds should be considered.
>>
>
> I also think that 5 seconds is high value, but in general, I have bad
> feeling when lowering not configurable timeouts (theoretically it could
> cause some DNS servers that are reachable now to become unreachable).
>
Right, but in practice I can't think of a valid network configuration
where the name server would take so long to reply. I mean, if the
network was so slow that just resolving name would take 5 seconds, then
chances are we wouldn't be able to download the entry from LDAP at all.
And if the nameserver is slow, then let's rather skip it and fail over
to another one.
> Another possibility would be to make this value configurable. Then I
> would have no problem lowering the default value. But, should this be
> configurable? We already have one option to configure timeout for the
> entire DNS resolution, adding another timeout for subtaks of this
> operation does not sound nice to me. But I do not have strong opinion
> on this.
>
Here is what we discussed with Michal today:
* make the per-nameserver timeout lower a little. Now it's 5
seconds, which is too long especially considering the failover
timeout is 5 seconds as well. We could change it to 2 or 3
seconds.
* the glibc allows 3 nameservers. The failover request timeut could
then default to 3*nameserver_timeout to give the resolver a chance
to iterate over all the configured name servers.
> Btw, once the correct data is returned from DNS servers, we no longer
> require any DNS operation, so it only happens first time the service
> is required (for example the first 'getent' call will be slow, but
> after that the service is fast). This is not so bad IMO.
>
>>> These patches also change man pages, so probably master only (string
>>> freeze)? Even if this is a really small change.
We accept changes again now that we've released 1.10.1.
But this change would be OK even for a string-frozen release, because it
only changes the number, not translatable strings.
>>>
>>> I was also thinking, would it make sense to write a warning to the logs
>>> if the dns_discovery_domain option is not set? It seem to be important
>>> to set it properly for cases like this one.
>>
>> I wonder, should we use the machine name to automatically set the
>> discovery domain, and only if that fails fall back to the ones defined
>> in resolv.conf ?
>
> Actually, we do this. Wireshark shows that the first srv lookup was
> constructed from my hostname. But the whole process is unclear to me,
> for example I do not know why we call the resolv_gethostbyname_send()
> inside of resolv_get_domain_send(). It adds another DNS lookups
> for A records (which causes another slow-down in the case of bad
> nameserver searched first). In wireshark I also see searches for A/AAA
> records for names like:
> mzidek.example.com
> mzidek.example.com.searchdom1.com
> mzidek.example.com.searchdom2.com
> mzidek.example.com.searchdom3.com
IIRC we try to canonicalize the host name before chopping of the first
part and using the rest as domain name.
Would using gethostname() be safe in real world environments?
I know that host names are often set to complete garbage and can't be
relied on..but is it OK to just say "fix your environment" in that case?
Personally I would argue the current canonicalization doesn't add much,
because if the host name is set to complete garbage then the canonicalization
wouldn't magically fix it..
>
> where mzidek.example.com is my hostname and searchdom1/2/3 are search
> domains specified in resolv.conf. Why do we even try to resolve these
> names? It seems like waste of time to me. If the dns_discovery_domain is
> properly set, these lookups do not appear.
>
> Thanks
> Michal
>
> PS: Patches are the same as before.
>
>>
>> Simo.
>>
>
New patches attached. As discussed above, I changed the timeout to
contact DNS server from 5s to 2s, and timeout for the entire DNS
resolution from 5 to 6 seconds (instead of 15s as proposed originally)
(so it has enough time to contact 3 nameservers).
I tested it with one and two unreachable nameservers.
Thanks
Michal
11:57 a.m.
On Mon, Jul 22, 2013 at 06:06:39PM +0200, Michal Židek wrote:
On 07/19/2013 12:15 PM, Jakub Hrozek wrote:
>On Mon, Jul 08, 2013 at 09:43:42PM +0200, Michal Židek wrote:
>>On 07/04/2013 07:45 PM, Simo Sorce wrote:
>>>On Thu, 2013-07-04 at 16:23 +0200, Michal Židek wrote:
>>>>https://fedorahosted.org/sssd/ticket/1966 (SSSD failover doesn't work
if
>>>>the first DNS server in resolv.conf is unavailable).
>>>>
>>>>The problem here is, that if first nameserver in resolv.conf is down,
>>>>the resolution is too slow and SSSD will not wait for the result of
>>>>ares_search and go offline. In my case the resolution was sometimes more
>>>>than a minute, because all search domains in resolv.conf were searched
>>>>inside ares_search() call using the first (not working) nameserver in
>>>>the first place and then with the working nameserver (and before that,
>>>>SSSD tried to figure out the domain name from my incorrectly set
>>>>hostname, which added more unnecessary DNS lookups).
>>>>
>>>>To avoid this problem, the option dns_discovery_domain must be set
>>>>properly, so that only the correct domain is searched, but even that is
>>>>not enough, because the default timeout for dns resolver operation in
>>>>sssd is too low. This patch rises the default value to 15 seconds
>>>>(instead of 5 seconds).
>>>>
>>>>Another option might be to lower the amount of time ares waits for a
>>>>nameserver to respond (currently it is 5 seconds, that is why 5 second
>>>>for the entire dns resolution is not sufficient), but I do not want to
>>>>do this.
>>>
>>>Why not ?
>>>
>>>5 seconds these days is quite awfully high all in all.
>>>I think 1 second should be plenty in most cases, so maybe a default of 2
>>>seconds should be considered.
>>>
>>
>>I also think that 5 seconds is high value, but in general, I have bad
>>feeling when lowering not configurable timeouts (theoretically it could
>>cause some DNS servers that are reachable now to become unreachable).
>>
>
>Right, but in practice I can't think of a valid network configuration
>where the name server would take so long to reply. I mean, if the
>network was so slow that just resolving name would take 5 seconds, then
>chances are we wouldn't be able to download the entry from LDAP at all.
>And if the nameserver is slow, then let's rather skip it and fail over
>to another one.
>
>>Another possibility would be to make this value configurable. Then I
>>would have no problem lowering the default value. But, should this be
>>configurable? We already have one option to configure timeout for the
>>entire DNS resolution, adding another timeout for subtaks of this
>>operation does not sound nice to me. But I do not have strong opinion
>>on this.
>>
>
>Here is what we discussed with Michal today:
> * make the per-nameserver timeout lower a little. Now it's 5
> seconds, which is too long especially considering the failover
> timeout is 5 seconds as well. We could change it to 2 or 3
> seconds.
> * the glibc allows 3 nameservers. The failover request timeut could
> then default to 3*nameserver_timeout to give the resolver a chance
> to iterate over all the configured name servers.
>
>>Btw, once the correct data is returned from DNS servers, we no longer
>>require any DNS operation, so it only happens first time the service
>>is required (for example the first 'getent' call will be slow, but
>>after that the service is fast). This is not so bad IMO.
>>
>>>>These patches also change man pages, so probably master only (string
>>>>freeze)? Even if this is a really small change.
>
>We accept changes again now that we've released 1.10.1.
>
>But this change would be OK even for a string-frozen release, because it
>only changes the number, not translatable strings.
>
>>>>
>>>>I was also thinking, would it make sense to write a warning to the logs
>>>>if the dns_discovery_domain option is not set? It seem to be important
>>>>to set it properly for cases like this one.
>>>
>>>I wonder, should we use the machine name to automatically set the
>>>discovery domain, and only if that fails fall back to the ones defined
>>>in resolv.conf ?
>>
>>Actually, we do this. Wireshark shows that the first srv lookup was
>>constructed from my hostname. But the whole process is unclear to me,
>>for example I do not know why we call the resolv_gethostbyname_send()
>>inside of resolv_get_domain_send(). It adds another DNS lookups
>>for A records (which causes another slow-down in the case of bad
>>nameserver searched first). In wireshark I also see searches for A/AAA
>>records for names like:
>>mzidek.example.com
>>mzidek.example.com.searchdom1.com
>>mzidek.example.com.searchdom2.com
>>mzidek.example.com.searchdom3.com
>
>IIRC we try to canonicalize the host name before chopping of the first
>part and using the rest as domain name.
>
>Would using gethostname() be safe in real world environments?
>I know that host names are often set to complete garbage and can't be
>relied on..but is it OK to just say "fix your environment" in that case?
>
>Personally I would argue the current canonicalization doesn't add much,
>because if the host name is set to complete garbage then the canonicalization
>wouldn't magically fix it..
>
>>
>>where mzidek.example.com is my hostname and searchdom1/2/3 are search
>>domains specified in resolv.conf. Why do we even try to resolve these
>>names? It seems like waste of time to me. If the dns_discovery_domain is
>>properly set, these lookups do not appear.
>>
>>Thanks
>>Michal
>>
>>PS: Patches are the same as before.
>>
>>>
>>>Simo.
>>>
>>
>
New patches attached. As discussed above, I changed the timeout to
contact DNS server from 5s to 2s, and timeout for the entire DNS resolution
from 5 to 6 seconds (instead of 15s as proposed originally)
(so it has enough time to contact 3 nameservers).
I tested it with one and two unreachable nameservers.
Thanks
Michal
Works for me, too, so I'll ACK the patches. But I'll leave them on the
list for a day longer in case somebody else would like to voice their
concern about changing the default timeout value. I personally think
it's OK.
7 a.m.
On Tue, Jul 23, 2013 at 06:57:08PM +0200, Jakub Hrozek wrote:
On Mon, Jul 22, 2013 at 06:06:39PM +0200, Michal Židek wrote:
> On 07/19/2013 12:15 PM, Jakub Hrozek wrote:
> >On Mon, Jul 08, 2013 at 09:43:42PM +0200, Michal Židek wrote:
> >>On 07/04/2013 07:45 PM, Simo Sorce wrote:
> >>>On Thu, 2013-07-04 at 16:23 +0200, Michal Židek wrote:
> >>>>https://fedorahosted.org/sssd/ticket/1966 (SSSD failover doesn't
work if
> >>>>the first DNS server in resolv.conf is unavailable).
> >>>>
> >>>>The problem here is, that if first nameserver in resolv.conf is
down,
> >>>>the resolution is too slow and SSSD will not wait for the result of
> >>>>ares_search and go offline. In my case the resolution was sometimes
more
> >>>>than a minute, because all search domains in resolv.conf were
searched
> >>>>inside ares_search() call using the first (not working) nameserver
in
> >>>>the first place and then with the working nameserver (and before
that,
> >>>>SSSD tried to figure out the domain name from my incorrectly set
> >>>>hostname, which added more unnecessary DNS lookups).
> >>>>
> >>>>To avoid this problem, the option dns_discovery_domain must be set
> >>>>properly, so that only the correct domain is searched, but even that
is
> >>>>not enough, because the default timeout for dns resolver operation
in
> >>>>sssd is too low. This patch rises the default value to 15 seconds
> >>>>(instead of 5 seconds).
> >>>>
> >>>>Another option might be to lower the amount of time ares waits for
a
> >>>>nameserver to respond (currently it is 5 seconds, that is why 5
second
> >>>>for the entire dns resolution is not sufficient), but I do not want
to
> >>>>do this.
> >>>
> >>>Why not ?
> >>>
> >>>5 seconds these days is quite awfully high all in all.
> >>>I think 1 second should be plenty in most cases, so maybe a default of
2
> >>>seconds should be considered.
> >>>
> >>
> >>I also think that 5 seconds is high value, but in general, I have bad
> >>feeling when lowering not configurable timeouts (theoretically it could
> >>cause some DNS servers that are reachable now to become unreachable).
> >>
> >
> >Right, but in practice I can't think of a valid network configuration
> >where the name server would take so long to reply. I mean, if the
> >network was so slow that just resolving name would take 5 seconds, then
> >chances are we wouldn't be able to download the entry from LDAP at all.
> >And if the nameserver is slow, then let's rather skip it and fail over
> >to another one.
> >
> >>Another possibility would be to make this value configurable. Then I
> >>would have no problem lowering the default value. But, should this be
> >>configurable? We already have one option to configure timeout for the
> >>entire DNS resolution, adding another timeout for subtaks of this
> >>operation does not sound nice to me. But I do not have strong opinion
> >>on this.
> >>
> >
> >Here is what we discussed with Michal today:
> > * make the per-nameserver timeout lower a little. Now it's 5
> > seconds, which is too long especially considering the failover
> > timeout is 5 seconds as well. We could change it to 2 or 3
> > seconds.
> > * the glibc allows 3 nameservers. The failover request timeut could
> > then default to 3*nameserver_timeout to give the resolver a chance
> > to iterate over all the configured name servers.
> >
> >>Btw, once the correct data is returned from DNS servers, we no longer
> >>require any DNS operation, so it only happens first time the service
> >>is required (for example the first 'getent' call will be slow, but
> >>after that the service is fast). This is not so bad IMO.
> >>
> >>>>These patches also change man pages, so probably master only
(string
> >>>>freeze)? Even if this is a really small change.
> >
> >We accept changes again now that we've released 1.10.1.
> >
> >But this change would be OK even for a string-frozen release, because it
> >only changes the number, not translatable strings.
> >
> >>>>
> >>>>I was also thinking, would it make sense to write a warning to the
logs
> >>>>if the dns_discovery_domain option is not set? It seem to be
important
> >>>>to set it properly for cases like this one.
> >>>
> >>>I wonder, should we use the machine name to automatically set the
> >>>discovery domain, and only if that fails fall back to the ones defined
> >>>in resolv.conf ?
> >>
> >>Actually, we do this. Wireshark shows that the first srv lookup was
> >>constructed from my hostname. But the whole process is unclear to me,
> >>for example I do not know why we call the resolv_gethostbyname_send()
> >>inside of resolv_get_domain_send(). It adds another DNS lookups
> >>for A records (which causes another slow-down in the case of bad
> >>nameserver searched first). In wireshark I also see searches for A/AAA
> >>records for names like:
> >>mzidek.example.com
> >>mzidek.example.com.searchdom1.com
> >>mzidek.example.com.searchdom2.com
> >>mzidek.example.com.searchdom3.com
> >
> >IIRC we try to canonicalize the host name before chopping of the first
> >part and using the rest as domain name.
> >
> >Would using gethostname() be safe in real world environments?
> >I know that host names are often set to complete garbage and can't be
> >relied on..but is it OK to just say "fix your environment" in that
case?
> >
> >Personally I would argue the current canonicalization doesn't add much,
> >because if the host name is set to complete garbage then the canonicalization
> >wouldn't magically fix it..
> >
> >>
> >>where mzidek.example.com is my hostname and searchdom1/2/3 are search
> >>domains specified in resolv.conf. Why do we even try to resolve these
> >>names? It seems like waste of time to me. If the dns_discovery_domain is
> >>properly set, these lookups do not appear.
> >>
> >>Thanks
> >>Michal
> >>
> >>PS: Patches are the same as before.
> >>
> >>>
> >>>Simo.
> >>>
> >>
> >
>
> New patches attached. As discussed above, I changed the timeout to
> contact DNS server from 5s to 2s, and timeout for the entire DNS resolution
> from 5 to 6 seconds (instead of 15s as proposed originally)
> (so it has enough time to contact 3 nameservers).
>
> I tested it with one and two unreachable nameservers.
>
> Thanks
> Michal
Works for me, too, so I'll ACK the patches. But I'll leave them on the
list for a day longer in case somebody else would like to voice their
concern about changing the default timeout value. I personally think
it's OK.
Pushed to master and sssd-1-10
4:51 a.m.
On Wed, Jul 24, 2013 at 02:00:38PM +0200, Jakub Hrozek wrote:
On Tue, Jul 23, 2013 at 06:57:08PM +0200, Jakub Hrozek wrote:
> On Mon, Jul 22, 2013 at 06:06:39PM +0200, Michal Židek wrote:
> > On 07/19/2013 12:15 PM, Jakub Hrozek wrote:
> > >On Mon, Jul 08, 2013 at 09:43:42PM +0200, Michal Židek wrote:
> > >>On 07/04/2013 07:45 PM, Simo Sorce wrote:
> > >>>On Thu, 2013-07-04 at 16:23 +0200, Michal Židek wrote:
> > >>>>https://fedorahosted.org/sssd/ticket/1966 (SSSD failover
doesn't work if
> > >>>>the first DNS server in resolv.conf is unavailable).
> > >>>>
> > >>>>The problem here is, that if first nameserver in resolv.conf is
down,
> > >>>>the resolution is too slow and SSSD will not wait for the
result of
> > >>>>ares_search and go offline. In my case the resolution was
sometimes more
> > >>>>than a minute, because all search domains in resolv.conf were
searched
> > >>>>inside ares_search() call using the first (not working)
nameserver in
> > >>>>the first place and then with the working nameserver (and
before that,
> > >>>>SSSD tried to figure out the domain name from my incorrectly
set
> > >>>>hostname, which added more unnecessary DNS lookups).
> > >>>>
> > >>>>To avoid this problem, the option dns_discovery_domain must be
set
> > >>>>properly, so that only the correct domain is searched, but even
that is
> > >>>>not enough, because the default timeout for dns resolver
operation in
> > >>>>sssd is too low. This patch rises the default value to 15
seconds
> > >>>>(instead of 5 seconds).
> > >>>>
> > >>>>Another option might be to lower the amount of time ares waits
for a
> > >>>>nameserver to respond (currently it is 5 seconds, that is why 5
second
> > >>>>for the entire dns resolution is not sufficient), but I do not
want to
> > >>>>do this.
> > >>>
> > >>>Why not ?
> > >>>
> > >>>5 seconds these days is quite awfully high all in all.
> > >>>I think 1 second should be plenty in most cases, so maybe a default
of 2
> > >>>seconds should be considered.
> > >>>
> > >>
> > >>I also think that 5 seconds is high value, but in general, I have bad
> > >>feeling when lowering not configurable timeouts (theoretically it
could
> > >>cause some DNS servers that are reachable now to become unreachable).
> > >>
> > >
> > >Right, but in practice I can't think of a valid network configuration
> > >where the name server would take so long to reply. I mean, if the
> > >network was so slow that just resolving name would take 5 seconds, then
> > >chances are we wouldn't be able to download the entry from LDAP at
all.
> > >And if the nameserver is slow, then let's rather skip it and fail over
> > >to another one.
> > >
> > >>Another possibility would be to make this value configurable. Then I
> > >>would have no problem lowering the default value. But, should this be
> > >>configurable? We already have one option to configure timeout for the
> > >>entire DNS resolution, adding another timeout for subtaks of this
> > >>operation does not sound nice to me. But I do not have strong opinion
> > >>on this.
> > >>
> > >
> > >Here is what we discussed with Michal today:
> > > * make the per-nameserver timeout lower a little. Now it's 5
> > > seconds, which is too long especially considering the failover
> > > timeout is 5 seconds as well. We could change it to 2 or 3
> > > seconds.
> > > * the glibc allows 3 nameservers. The failover request timeut could
> > > then default to 3*nameserver_timeout to give the resolver a chance
> > > to iterate over all the configured name servers.
> > >
> > >>Btw, once the correct data is returned from DNS servers, we no longer
> > >>require any DNS operation, so it only happens first time the service
> > >>is required (for example the first 'getent' call will be slow,
but
> > >>after that the service is fast). This is not so bad IMO.
> > >>
> > >>>>These patches also change man pages, so probably master only
(string
> > >>>>freeze)? Even if this is a really small change.
> > >
> > >We accept changes again now that we've released 1.10.1.
> > >
> > >But this change would be OK even for a string-frozen release, because it
> > >only changes the number, not translatable strings.
> > >
> > >>>>
> > >>>>I was also thinking, would it make sense to write a warning to
the logs
> > >>>>if the dns_discovery_domain option is not set? It seem to be
important
> > >>>>to set it properly for cases like this one.
> > >>>
> > >>>I wonder, should we use the machine name to automatically set the
> > >>>discovery domain, and only if that fails fall back to the ones
defined
> > >>>in resolv.conf ?
> > >>
> > >>Actually, we do this. Wireshark shows that the first srv lookup was
> > >>constructed from my hostname. But the whole process is unclear to me,
> > >>for example I do not know why we call the resolv_gethostbyname_send()
> > >>inside of resolv_get_domain_send(). It adds another DNS lookups
> > >>for A records (which causes another slow-down in the case of bad
> > >>nameserver searched first). In wireshark I also see searches for A/AAA
> > >>records for names like:
> > >>mzidek.example.com
> > >>mzidek.example.com.searchdom1.com
> > >>mzidek.example.com.searchdom2.com
> > >>mzidek.example.com.searchdom3.com
> > >
> > >IIRC we try to canonicalize the host name before chopping of the first
> > >part and using the rest as domain name.
> > >
> > >Would using gethostname() be safe in real world environments?
> > >I know that host names are often set to complete garbage and can't be
> > >relied on..but is it OK to just say "fix your environment" in
that case?
> > >
> > >Personally I would argue the current canonicalization doesn't add
much,
> > >because if the host name is set to complete garbage then the
canonicalization
> > >wouldn't magically fix it..
> > >
> > >>
> > >>where mzidek.example.com is my hostname and searchdom1/2/3 are search
> > >>domains specified in resolv.conf. Why do we even try to resolve these
> > >>names? It seems like waste of time to me. If the dns_discovery_domain
is
> > >>properly set, these lookups do not appear.
> > >>
> > >>Thanks
> > >>Michal
> > >>
> > >>PS: Patches are the same as before.
> > >>
> > >>>
> > >>>Simo.
> > >>>
> > >>
> > >
> >
> > New patches attached. As discussed above, I changed the timeout to
> > contact DNS server from 5s to 2s, and timeout for the entire DNS resolution
> > from 5 to 6 seconds (instead of 15s as proposed originally)
> > (so it has enough time to contact 3 nameservers).
> >
> > I tested it with one and two unreachable nameservers.
> >
> > Thanks
> > Michal
>
> Works for me, too, so I'll ACK the patches. But I'll leave them on the
> list for a day longer in case somebody else would like to voice their
> concern about changing the default timeout value. I personally think
> it's OK.
Pushed to master and sssd-1-10
Hi,
attached is a 1.9 bacport of the patches. Michal, do you think we need
to lower the FO_DEFAULT_SVC_TIMEOUT as well? I guess so, but you know
the details better..
9:33 a.m.
On 08/09/2013 11:51 AM, Jakub Hrozek wrote:
On Wed, Jul 24, 2013 at 02:00:38PM +0200, Jakub Hrozek wrote:
> On Tue, Jul 23, 2013 at 06:57:08PM +0200, Jakub Hrozek wrote:
>> On Mon, Jul 22, 2013 at 06:06:39PM +0200, Michal Židek wrote:
>>> On 07/19/2013 12:15 PM, Jakub Hrozek wrote:
>>>> On Mon, Jul 08, 2013 at 09:43:42PM +0200, Michal Židek wrote:
>>>>> On 07/04/2013 07:45 PM, Simo Sorce wrote:
>>>>>> On Thu, 2013-07-04 at 16:23 +0200, Michal Židek wrote:
>>>>>>> https://fedorahosted.org/sssd/ticket/1966 (SSSD failover
doesn't work if
>>>>>>> the first DNS server in resolv.conf is unavailable).
>>>>>>>
>>>>>>> The problem here is, that if first nameserver in resolv.conf
is down,
>>>>>>> the resolution is too slow and SSSD will not wait for the
result of
>>>>>>> ares_search and go offline. In my case the resolution was
sometimes more
>>>>>>> than a minute, because all search domains in resolv.conf were
searched
>>>>>>> inside ares_search() call using the first (not working)
nameserver in
>>>>>>> the first place and then with the working nameserver (and
before that,
>>>>>>> SSSD tried to figure out the domain name from my incorrectly
set
>>>>>>> hostname, which added more unnecessary DNS lookups).
>>>>>>>
>>>>>>> To avoid this problem, the option dns_discovery_domain must
be set
>>>>>>> properly, so that only the correct domain is searched, but
even that is
>>>>>>> not enough, because the default timeout for dns resolver
operation in
>>>>>>> sssd is too low. This patch rises the default value to 15
seconds
>>>>>>> (instead of 5 seconds).
>>>>>>>
>>>>>>> Another option might be to lower the amount of time ares
waits for a
>>>>>>> nameserver to respond (currently it is 5 seconds, that is why
5 second
>>>>>>> for the entire dns resolution is not sufficient), but I do
not want to
>>>>>>> do this.
>>>>>>
>>>>>> Why not ?
>>>>>>
>>>>>> 5 seconds these days is quite awfully high all in all.
>>>>>> I think 1 second should be plenty in most cases, so maybe a
default of 2
>>>>>> seconds should be considered.
>>>>>>
>>>>>
>>>>> I also think that 5 seconds is high value, but in general, I have
bad
>>>>> feeling when lowering not configurable timeouts (theoretically it
could
>>>>> cause some DNS servers that are reachable now to become
unreachable).
>>>>>
>>>>
>>>> Right, but in practice I can't think of a valid network
configuration
>>>> where the name server would take so long to reply. I mean, if the
>>>> network was so slow that just resolving name would take 5 seconds, then
>>>> chances are we wouldn't be able to download the entry from LDAP at
all.
>>>> And if the nameserver is slow, then let's rather skip it and fail
over
>>>> to another one.
>>>>
>>>>> Another possibility would be to make this value configurable. Then I
>>>>> would have no problem lowering the default value. But, should this
be
>>>>> configurable? We already have one option to configure timeout for
the
>>>>> entire DNS resolution, adding another timeout for subtaks of this
>>>>> operation does not sound nice to me. But I do not have strong
opinion
>>>>> on this.
>>>>>
>>>>
>>>> Here is what we discussed with Michal today:
>>>> * make the per-nameserver timeout lower a little. Now it's 5
>>>> seconds, which is too long especially considering the failover
>>>> timeout is 5 seconds as well. We could change it to 2 or 3
>>>> seconds.
>>>> * the glibc allows 3 nameservers. The failover request timeut could
>>>> then default to 3*nameserver_timeout to give the resolver a
chance
>>>> to iterate over all the configured name servers.
>>>>
>>>>> Btw, once the correct data is returned from DNS servers, we no
longer
>>>>> require any DNS operation, so it only happens first time the service
>>>>> is required (for example the first 'getent' call will be
slow, but
>>>>> after that the service is fast). This is not so bad IMO.
>>>>>
>>>>>>> These patches also change man pages, so probably master only
(string
>>>>>>> freeze)? Even if this is a really small change.
>>>>
>>>> We accept changes again now that we've released 1.10.1.
>>>>
>>>> But this change would be OK even for a string-frozen release, because it
>>>> only changes the number, not translatable strings.
>>>>
>>>>>>>
>>>>>>> I was also thinking, would it make sense to write a warning
to the logs
>>>>>>> if the dns_discovery_domain option is not set? It seem to be
important
>>>>>>> to set it properly for cases like this one.
>>>>>>
>>>>>> I wonder, should we use the machine name to automatically set
the
>>>>>> discovery domain, and only if that fails fall back to the ones
defined
>>>>>> in resolv.conf ?
>>>>>
>>>>> Actually, we do this. Wireshark shows that the first srv lookup was
>>>>> constructed from my hostname. But the whole process is unclear to
me,
>>>>> for example I do not know why we call the
resolv_gethostbyname_send()
>>>>> inside of resolv_get_domain_send(). It adds another DNS lookups
>>>>> for A records (which causes another slow-down in the case of bad
>>>>> nameserver searched first). In wireshark I also see searches for
A/AAA
>>>>> records for names like:
>>>>> mzidek.example.com
>>>>> mzidek.example.com.searchdom1.com
>>>>> mzidek.example.com.searchdom2.com
>>>>> mzidek.example.com.searchdom3.com
>>>>
>>>> IIRC we try to canonicalize the host name before chopping of the first
>>>> part and using the rest as domain name.
>>>>
>>>> Would using gethostname() be safe in real world environments?
>>>> I know that host names are often set to complete garbage and can't
be
>>>> relied on..but is it OK to just say "fix your environment" in
that case?
>>>>
>>>> Personally I would argue the current canonicalization doesn't add
much,
>>>> because if the host name is set to complete garbage then the
canonicalization
>>>> wouldn't magically fix it..
>>>>
>>>>>
>>>>> where mzidek.example.com is my hostname and searchdom1/2/3 are
search
>>>>> domains specified in resolv.conf. Why do we even try to resolve
these
>>>>> names? It seems like waste of time to me. If the dns_discovery_domain
is
>>>>> properly set, these lookups do not appear.
>>>>>
>>>>> Thanks
>>>>> Michal
>>>>>
>>>>> PS: Patches are the same as before.
>>>>>
>>>>>>
>>>>>> Simo.
>>>>>>
>>>>>
>>>>
>>>
>>> New patches attached. As discussed above, I changed the timeout to
>>> contact DNS server from 5s to 2s, and timeout for the entire DNS resolution
>>> from 5 to 6 seconds (instead of 15s as proposed originally)
>>> (so it has enough time to contact 3 nameservers).
>>>
>>> I tested it with one and two unreachable nameservers.
>>>
>>> Thanks
>>> Michal
>>
>> Works for me, too, so I'll ACK the patches. But I'll leave them on the
>> list for a day longer in case somebody else would like to voice their
>> concern about changing the default timeout value. I personally think
>> it's OK.
>
> Pushed to master and sssd-1-10
Hi,
attached is a 1.9 bacport of the patches. Michal, do you think we need
to lower the FO_DEFAULT_SVC_TIMEOUT as well? I guess so, but you know
the details better..
FO_DEFAULT_SVC_TIMEOUT is not used anywhere. It is actually removed in
the two minor fixes you posted earlier .
Ack to both patches.
Michal
9:51 a.m.
On Fri, Aug 09, 2013 at 04:33:43PM +0200, Michal Židek wrote:
On 08/09/2013 11:51 AM, Jakub Hrozek wrote:
>On Wed, Jul 24, 2013 at 02:00:38PM +0200, Jakub Hrozek wrote:
>>On Tue, Jul 23, 2013 at 06:57:08PM +0200, Jakub Hrozek wrote:
>>>On Mon, Jul 22, 2013 at 06:06:39PM +0200, Michal Židek wrote:
>>>>On 07/19/2013 12:15 PM, Jakub Hrozek wrote:
>>>>>On Mon, Jul 08, 2013 at 09:43:42PM +0200, Michal Židek wrote:
>>>>>>On 07/04/2013 07:45 PM, Simo Sorce wrote:
>>>>>>>On Thu, 2013-07-04 at 16:23 +0200, Michal Židek wrote:
>>>>>>>>https://fedorahosted.org/sssd/ticket/1966 (SSSD failover
doesn't work if
>>>>>>>>the first DNS server in resolv.conf is unavailable).
>>>>>>>>
>>>>>>>>The problem here is, that if first nameserver in
resolv.conf is down,
>>>>>>>>the resolution is too slow and SSSD will not wait for the
result of
>>>>>>>>ares_search and go offline. In my case the resolution was
sometimes more
>>>>>>>>than a minute, because all search domains in resolv.conf
were searched
>>>>>>>>inside ares_search() call using the first (not working)
nameserver in
>>>>>>>>the first place and then with the working nameserver (and
before that,
>>>>>>>>SSSD tried to figure out the domain name from my
incorrectly set
>>>>>>>>hostname, which added more unnecessary DNS lookups).
>>>>>>>>
>>>>>>>>To avoid this problem, the option dns_discovery_domain
must be set
>>>>>>>>properly, so that only the correct domain is searched,
but even that is
>>>>>>>>not enough, because the default timeout for dns resolver
operation in
>>>>>>>>sssd is too low. This patch rises the default value to 15
seconds
>>>>>>>>(instead of 5 seconds).
>>>>>>>>
>>>>>>>>Another option might be to lower the amount of time ares
waits for a
>>>>>>>>nameserver to respond (currently it is 5 seconds, that is
why 5 second
>>>>>>>>for the entire dns resolution is not sufficient), but I
do not want to
>>>>>>>>do this.
>>>>>>>
>>>>>>>Why not ?
>>>>>>>
>>>>>>>5 seconds these days is quite awfully high all in all.
>>>>>>>I think 1 second should be plenty in most cases, so maybe a
default of 2
>>>>>>>seconds should be considered.
>>>>>>>
>>>>>>
>>>>>>I also think that 5 seconds is high value, but in general, I have
bad
>>>>>>feeling when lowering not configurable timeouts (theoretically it
could
>>>>>>cause some DNS servers that are reachable now to become
unreachable).
>>>>>>
>>>>>
>>>>>Right, but in practice I can't think of a valid network
configuration
>>>>>where the name server would take so long to reply. I mean, if the
>>>>>network was so slow that just resolving name would take 5 seconds,
then
>>>>>chances are we wouldn't be able to download the entry from LDAP
at all.
>>>>>And if the nameserver is slow, then let's rather skip it and fail
over
>>>>>to another one.
>>>>>
>>>>>>Another possibility would be to make this value configurable.
Then I
>>>>>>would have no problem lowering the default value. But, should
this be
>>>>>>configurable? We already have one option to configure timeout for
the
>>>>>>entire DNS resolution, adding another timeout for subtaks of
this
>>>>>>operation does not sound nice to me. But I do not have strong
opinion
>>>>>>on this.
>>>>>>
>>>>>
>>>>>Here is what we discussed with Michal today:
>>>>> * make the per-nameserver timeout lower a little. Now it's
5
>>>>> seconds, which is too long especially considering the
failover
>>>>> timeout is 5 seconds as well. We could change it to 2 or 3
>>>>> seconds.
>>>>> * the glibc allows 3 nameservers. The failover request timeut
could
>>>>> then default to 3*nameserver_timeout to give the resolver a
chance
>>>>> to iterate over all the configured name servers.
>>>>>
>>>>>>Btw, once the correct data is returned from DNS servers, we no
longer
>>>>>>require any DNS operation, so it only happens first time the
service
>>>>>>is required (for example the first 'getent' call will be
slow, but
>>>>>>after that the service is fast). This is not so bad IMO.
>>>>>>
>>>>>>>>These patches also change man pages, so probably master
only (string
>>>>>>>>freeze)? Even if this is a really small change.
>>>>>
>>>>>We accept changes again now that we've released 1.10.1.
>>>>>
>>>>>But this change would be OK even for a string-frozen release, because
it
>>>>>only changes the number, not translatable strings.
>>>>>
>>>>>>>>
>>>>>>>>I was also thinking, would it make sense to write a
warning to the logs
>>>>>>>>if the dns_discovery_domain option is not set? It seem to
be important
>>>>>>>>to set it properly for cases like this one.
>>>>>>>
>>>>>>>I wonder, should we use the machine name to automatically set
the
>>>>>>>discovery domain, and only if that fails fall back to the
ones defined
>>>>>>>in resolv.conf ?
>>>>>>
>>>>>>Actually, we do this. Wireshark shows that the first srv lookup
was
>>>>>>constructed from my hostname. But the whole process is unclear to
me,
>>>>>>for example I do not know why we call the
resolv_gethostbyname_send()
>>>>>>inside of resolv_get_domain_send(). It adds another DNS lookups
>>>>>>for A records (which causes another slow-down in the case of bad
>>>>>>nameserver searched first). In wireshark I also see searches for
A/AAA
>>>>>>records for names like:
>>>>>>mzidek.example.com
>>>>>>mzidek.example.com.searchdom1.com
>>>>>>mzidek.example.com.searchdom2.com
>>>>>>mzidek.example.com.searchdom3.com
>>>>>
>>>>>IIRC we try to canonicalize the host name before chopping of the
first
>>>>>part and using the rest as domain name.
>>>>>
>>>>>Would using gethostname() be safe in real world environments?
>>>>>I know that host names are often set to complete garbage and
can't be
>>>>>relied on..but is it OK to just say "fix your environment"
in that case?
>>>>>
>>>>>Personally I would argue the current canonicalization doesn't add
much,
>>>>>because if the host name is set to complete garbage then the
canonicalization
>>>>>wouldn't magically fix it..
>>>>>
>>>>>>
>>>>>>where mzidek.example.com is my hostname and searchdom1/2/3 are
search
>>>>>>domains specified in resolv.conf. Why do we even try to resolve
these
>>>>>>names? It seems like waste of time to me. If the
dns_discovery_domain is
>>>>>>properly set, these lookups do not appear.
>>>>>>
>>>>>>Thanks
>>>>>>Michal
>>>>>>
>>>>>>PS: Patches are the same as before.
>>>>>>
>>>>>>>
>>>>>>>Simo.
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>>New patches attached. As discussed above, I changed the timeout to
>>>>contact DNS server from 5s to 2s, and timeout for the entire DNS
resolution
>>>>from 5 to 6 seconds (instead of 15s as proposed originally)
>>>>(so it has enough time to contact 3 nameservers).
>>>>
>>>>I tested it with one and two unreachable nameservers.
>>>>
>>>>Thanks
>>>>Michal
>>>
>>>Works for me, too, so I'll ACK the patches. But I'll leave them on
the
>>>list for a day longer in case somebody else would like to voice their
>>>concern about changing the default timeout value. I personally think
>>>it's OK.
>>
>>Pushed to master and sssd-1-10
>
>Hi,
>
>attached is a 1.9 bacport of the patches. Michal, do you think we need
>to lower the FO_DEFAULT_SVC_TIMEOUT as well? I guess so, but you know
>the details better..
>
FO_DEFAULT_SVC_TIMEOUT is not used anywhere. It is actually removed
in the two minor fixes you posted earlier .
Ack to both patches.
Michal
Actually these two patches are only intended for the 1.9 branch and the
other two patches I sent are for master. In 1.9, FO_DEFAULT_SVC_TIMEOUT
is still used:
From src/providers/data_provider_fo.c:
67 static int be_fo_get_options(struct be_ctx *ctx,
68 struct fo_options *opts)
69 {
70 errno_t ret;
71
72 ret = confdb_get_int(ctx->cdb, ctx->conf_path,
73 CONFDB_DOMAIN_RESOLV_TIMEOUT,
74 FO_DEFAULT_SVC_TIMEOUT,
75 &opts->service_resolv_timeout);
76 if (ret != EOK) {
77 return ret;
78 }
So FO_DEFAULT_SVC_TIMEOUT is used as the per-service (aka per-request)
timeout and in 1.9 it's set to 10 seconds. I remember that the your patches
set both the c-ares timeout and the per-service timeout to the same value,
so I was wondering if in 1.9 we should do the same.
10:13 a.m.
On 08/09/2013 04:51 PM, Jakub Hrozek wrote:
On Fri, Aug 09, 2013 at 04:33:43PM +0200, Michal Židek wrote:
> On 08/09/2013 11:51 AM, Jakub Hrozek wrote:
>> On Wed, Jul 24, 2013 at 02:00:38PM +0200, Jakub Hrozek wrote:
>>> On Tue, Jul 23, 2013 at 06:57:08PM +0200, Jakub Hrozek wrote:
>>>> On Mon, Jul 22, 2013 at 06:06:39PM +0200, Michal Židek wrote:
>>>>> On 07/19/2013 12:15 PM, Jakub Hrozek wrote:
>>>>>> On Mon, Jul 08, 2013 at 09:43:42PM +0200, Michal Židek wrote:
>>>>>>> On 07/04/2013 07:45 PM, Simo Sorce wrote:
>>>>>>>> On Thu, 2013-07-04 at 16:23 +0200, Michal Židek wrote:
>>>>>>>>> https://fedorahosted.org/sssd/ticket/1966 (SSSD
failover doesn't work if
>>>>>>>>> the first DNS server in resolv.conf is unavailable).
>>>>>>>>>
>>>>>>>>> The problem here is, that if first nameserver in
resolv.conf is down,
>>>>>>>>> the resolution is too slow and SSSD will not wait for
the result of
>>>>>>>>> ares_search and go offline. In my case the resolution
was sometimes more
>>>>>>>>> than a minute, because all search domains in
resolv.conf were searched
>>>>>>>>> inside ares_search() call using the first (not
working) nameserver in
>>>>>>>>> the first place and then with the working nameserver
(and before that,
>>>>>>>>> SSSD tried to figure out the domain name from my
incorrectly set
>>>>>>>>> hostname, which added more unnecessary DNS lookups).
>>>>>>>>>
>>>>>>>>> To avoid this problem, the option
dns_discovery_domain must be set
>>>>>>>>> properly, so that only the correct domain is
searched, but even that is
>>>>>>>>> not enough, because the default timeout for dns
resolver operation in
>>>>>>>>> sssd is too low. This patch rises the default value
to 15 seconds
>>>>>>>>> (instead of 5 seconds).
>>>>>>>>>
>>>>>>>>> Another option might be to lower the amount of time
ares waits for a
>>>>>>>>> nameserver to respond (currently it is 5 seconds,
that is why 5 second
>>>>>>>>> for the entire dns resolution is not sufficient), but
I do not want to
>>>>>>>>> do this.
>>>>>>>>
>>>>>>>> Why not ?
>>>>>>>>
>>>>>>>> 5 seconds these days is quite awfully high all in all.
>>>>>>>> I think 1 second should be plenty in most cases, so maybe
a default of 2
>>>>>>>> seconds should be considered.
>>>>>>>>
>>>>>>>
>>>>>>> I also think that 5 seconds is high value, but in general, I
have bad
>>>>>>> feeling when lowering not configurable timeouts
(theoretically it could
>>>>>>> cause some DNS servers that are reachable now to become
unreachable).
>>>>>>>
>>>>>>
>>>>>> Right, but in practice I can't think of a valid network
configuration
>>>>>> where the name server would take so long to reply. I mean, if
the
>>>>>> network was so slow that just resolving name would take 5
seconds, then
>>>>>> chances are we wouldn't be able to download the entry from
LDAP at all.
>>>>>> And if the nameserver is slow, then let's rather skip it and
fail over
>>>>>> to another one.
>>>>>>
>>>>>>> Another possibility would be to make this value configurable.
Then I
>>>>>>> would have no problem lowering the default value. But, should
this be
>>>>>>> configurable? We already have one option to configure timeout
for the
>>>>>>> entire DNS resolution, adding another timeout for subtaks of
this
>>>>>>> operation does not sound nice to me. But I do not have strong
opinion
>>>>>>> on this.
>>>>>>>
>>>>>>
>>>>>> Here is what we discussed with Michal today:
>>>>>> * make the per-nameserver timeout lower a little. Now
it's 5
>>>>>> seconds, which is too long especially considering the
failover
>>>>>> timeout is 5 seconds as well. We could change it to 2 or
3
>>>>>> seconds.
>>>>>> * the glibc allows 3 nameservers. The failover request
timeut could
>>>>>> then default to 3*nameserver_timeout to give the resolver
a chance
>>>>>> to iterate over all the configured name servers.
>>>>>>
>>>>>>> Btw, once the correct data is returned from DNS servers, we
no longer
>>>>>>> require any DNS operation, so it only happens first time the
service
>>>>>>> is required (for example the first 'getent' call will
be slow, but
>>>>>>> after that the service is fast). This is not so bad IMO.
>>>>>>>
>>>>>>>>> These patches also change man pages, so probably
master only (string
>>>>>>>>> freeze)? Even if this is a really small change.
>>>>>>
>>>>>> We accept changes again now that we've released 1.10.1.
>>>>>>
>>>>>> But this change would be OK even for a string-frozen release,
because it
>>>>>> only changes the number, not translatable strings.
>>>>>>
>>>>>>>>>
>>>>>>>>> I was also thinking, would it make sense to write a
warning to the logs
>>>>>>>>> if the dns_discovery_domain option is not set? It
seem to be important
>>>>>>>>> to set it properly for cases like this one.
>>>>>>>>
>>>>>>>> I wonder, should we use the machine name to automatically
set the
>>>>>>>> discovery domain, and only if that fails fall back to the
ones defined
>>>>>>>> in resolv.conf ?
>>>>>>>
>>>>>>> Actually, we do this. Wireshark shows that the first srv
lookup was
>>>>>>> constructed from my hostname. But the whole process is
unclear to me,
>>>>>>> for example I do not know why we call the
resolv_gethostbyname_send()
>>>>>>> inside of resolv_get_domain_send(). It adds another DNS
lookups
>>>>>>> for A records (which causes another slow-down in the case of
bad
>>>>>>> nameserver searched first). In wireshark I also see searches
for A/AAA
>>>>>>> records for names like:
>>>>>>> mzidek.example.com
>>>>>>> mzidek.example.com.searchdom1.com
>>>>>>> mzidek.example.com.searchdom2.com
>>>>>>> mzidek.example.com.searchdom3.com
>>>>>>
>>>>>> IIRC we try to canonicalize the host name before chopping of the
first
>>>>>> part and using the rest as domain name.
>>>>>>
>>>>>> Would using gethostname() be safe in real world environments?
>>>>>> I know that host names are often set to complete garbage and
can't be
>>>>>> relied on..but is it OK to just say "fix your
environment" in that case?
>>>>>>
>>>>>> Personally I would argue the current canonicalization doesn't
add much,
>>>>>> because if the host name is set to complete garbage then the
canonicalization
>>>>>> wouldn't magically fix it..
>>>>>>
>>>>>>>
>>>>>>> where mzidek.example.com is my hostname and searchdom1/2/3
are search
>>>>>>> domains specified in resolv.conf. Why do we even try to
resolve these
>>>>>>> names? It seems like waste of time to me. If the
dns_discovery_domain is
>>>>>>> properly set, these lookups do not appear.
>>>>>>>
>>>>>>> Thanks
>>>>>>> Michal
>>>>>>>
>>>>>>> PS: Patches are the same as before.
>>>>>>>
>>>>>>>>
>>>>>>>> Simo.
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>> New patches attached. As discussed above, I changed the timeout to
>>>>> contact DNS server from 5s to 2s, and timeout for the entire DNS
resolution
>>>> >from 5 to 6 seconds (instead of 15s as proposed originally)
>>>>> (so it has enough time to contact 3 nameservers).
>>>>>
>>>>> I tested it with one and two unreachable nameservers.
>>>>>
>>>>> Thanks
>>>>> Michal
>>>>
>>>> Works for me, too, so I'll ACK the patches. But I'll leave them
on the
>>>> list for a day longer in case somebody else would like to voice their
>>>> concern about changing the default timeout value. I personally think
>>>> it's OK.
>>>
>>> Pushed to master and sssd-1-10
>>
>> Hi,
>>
>> attached is a 1.9 bacport of the patches. Michal, do you think we need
>> to lower the FO_DEFAULT_SVC_TIMEOUT as well? I guess so, but you know
>> the details better..
>>
>
>
> FO_DEFAULT_SVC_TIMEOUT is not used anywhere. It is actually removed
> in the two minor fixes you posted earlier .
>
> Ack to both patches.
>
> Michal
Actually these two patches are only intended for the 1.9 branch and the
other two patches I sent are for master. In 1.9, FO_DEFAULT_SVC_TIMEOUT
is still used:
From src/providers/data_provider_fo.c:
67 static int be_fo_get_options(struct be_ctx *ctx,
68 struct fo_options *opts)
69 {
70 errno_t ret;
71
72 ret = confdb_get_int(ctx->cdb, ctx->conf_path,
73 CONFDB_DOMAIN_RESOLV_TIMEOUT,
74 FO_DEFAULT_SVC_TIMEOUT,
75 &opts->service_resolv_timeout);
76 if (ret != EOK) {
77 return ret;
78 }
So FO_DEFAULT_SVC_TIMEOUT is used as the per-service (aka per-request)
timeout and in 1.9 it's set to 10 seconds. I remember that the your patches
set both the c-ares timeout and the per-service timeout to the same value,
so I was wondering if in 1.9 we should do the same.
Ah, I see it now. Well, 10s is OK. Anything less than 6 seconds would be
a problem. But yes, we can change it to 6 seconds to have the same
defaults in master and 1.9.
10:14 a.m.
On 08/09/2013 04:33 PM, Michal Židek wrote:
On 08/09/2013 11:51 AM, Jakub Hrozek wrote:
> On Wed, Jul 24, 2013 at 02:00:38PM +0200, Jakub Hrozek wrote:
>> On Tue, Jul 23, 2013 at 06:57:08PM +0200, Jakub Hrozek wrote:
>>> On Mon, Jul 22, 2013 at 06:06:39PM +0200, Michal Židek wrote:
>>>> On 07/19/2013 12:15 PM, Jakub Hrozek wrote:
>>>>> On Mon, Jul 08, 2013 at 09:43:42PM +0200, Michal Židek wrote:
>>>>>> On 07/04/2013 07:45 PM, Simo Sorce wrote:
>>>>>>> On Thu, 2013-07-04 at 16:23 +0200, Michal Židek wrote:
>>>>>>>> https://fedorahosted.org/sssd/ticket/1966 (SSSD failover
>>>>>>>> doesn't work if
>>>>>>>> the first DNS server in resolv.conf is unavailable).
>>>>>>>>
>>>>>>>> The problem here is, that if first nameserver in
resolv.conf is
>>>>>>>> down,
>>>>>>>> the resolution is too slow and SSSD will not wait for
the
>>>>>>>> result of
>>>>>>>> ares_search and go offline. In my case the resolution
was
>>>>>>>> sometimes more
>>>>>>>> than a minute, because all search domains in resolv.conf
were
>>>>>>>> searched
>>>>>>>> inside ares_search() call using the first (not working)
>>>>>>>> nameserver in
>>>>>>>> the first place and then with the working nameserver
(and
>>>>>>>> before that,
>>>>>>>> SSSD tried to figure out the domain name from my
incorrectly set
>>>>>>>> hostname, which added more unnecessary DNS lookups).
>>>>>>>>
>>>>>>>> To avoid this problem, the option dns_discovery_domain
must be set
>>>>>>>> properly, so that only the correct domain is searched,
but even
>>>>>>>> that is
>>>>>>>> not enough, because the default timeout for dns resolver
>>>>>>>> operation in
>>>>>>>> sssd is too low. This patch rises the default value to 15
seconds
>>>>>>>> (instead of 5 seconds).
>>>>>>>>
>>>>>>>> Another option might be to lower the amount of time ares
waits
>>>>>>>> for a
>>>>>>>> nameserver to respond (currently it is 5 seconds, that is
why 5
>>>>>>>> second
>>>>>>>> for the entire dns resolution is not sufficient), but I
do not
>>>>>>>> want to
>>>>>>>> do this.
>>>>>>>
>>>>>>> Why not ?
>>>>>>>
>>>>>>> 5 seconds these days is quite awfully high all in all.
>>>>>>> I think 1 second should be plenty in most cases, so maybe a
>>>>>>> default of 2
>>>>>>> seconds should be considered.
>>>>>>>
>>>>>>
>>>>>> I also think that 5 seconds is high value, but in general, I
have
>>>>>> bad
>>>>>> feeling when lowering not configurable timeouts (theoretically
it
>>>>>> could
>>>>>> cause some DNS servers that are reachable now to become
>>>>>> unreachable).
>>>>>>
>>>>>
>>>>> Right, but in practice I can't think of a valid network
configuration
>>>>> where the name server would take so long to reply. I mean, if the
>>>>> network was so slow that just resolving name would take 5 seconds,
>>>>> then
>>>>> chances are we wouldn't be able to download the entry from LDAP
at
>>>>> all.
>>>>> And if the nameserver is slow, then let's rather skip it and
fail
>>>>> over
>>>>> to another one.
>>>>>
>>>>>> Another possibility would be to make this value configurable.
Then I
>>>>>> would have no problem lowering the default value. But, should
>>>>>> this be
>>>>>> configurable? We already have one option to configure timeout
for
>>>>>> the
>>>>>> entire DNS resolution, adding another timeout for subtaks of
this
>>>>>> operation does not sound nice to me. But I do not have strong
>>>>>> opinion
>>>>>> on this.
>>>>>>
>>>>>
>>>>> Here is what we discussed with Michal today:
>>>>> * make the per-nameserver timeout lower a little. Now it's
5
>>>>> seconds, which is too long especially considering the
failover
>>>>> timeout is 5 seconds as well. We could change it to 2 or 3
>>>>> seconds.
>>>>> * the glibc allows 3 nameservers. The failover request timeut
>>>>> could
>>>>> then default to 3*nameserver_timeout to give the resolver a
>>>>> chance
>>>>> to iterate over all the configured name servers.
>>>>>
>>>>>> Btw, once the correct data is returned from DNS servers, we no
>>>>>> longer
>>>>>> require any DNS operation, so it only happens first time the
service
>>>>>> is required (for example the first 'getent' call will be
slow, but
>>>>>> after that the service is fast). This is not so bad IMO.
>>>>>>
>>>>>>>> These patches also change man pages, so probably master
only
>>>>>>>> (string
>>>>>>>> freeze)? Even if this is a really small change.
>>>>>
>>>>> We accept changes again now that we've released 1.10.1.
>>>>>
>>>>> But this change would be OK even for a string-frozen release,
>>>>> because it
>>>>> only changes the number, not translatable strings.
>>>>>
>>>>>>>>
>>>>>>>> I was also thinking, would it make sense to write a
warning to
>>>>>>>> the logs
>>>>>>>> if the dns_discovery_domain option is not set? It seem to
be
>>>>>>>> important
>>>>>>>> to set it properly for cases like this one.
>>>>>>>
>>>>>>> I wonder, should we use the machine name to automatically set
the
>>>>>>> discovery domain, and only if that fails fall back to the
ones
>>>>>>> defined
>>>>>>> in resolv.conf ?
>>>>>>
>>>>>> Actually, we do this. Wireshark shows that the first srv lookup
was
>>>>>> constructed from my hostname. But the whole process is unclear
to
>>>>>> me,
>>>>>> for example I do not know why we call the
>>>>>> resolv_gethostbyname_send()
>>>>>> inside of resolv_get_domain_send(). It adds another DNS lookups
>>>>>> for A records (which causes another slow-down in the case of bad
>>>>>> nameserver searched first). In wireshark I also see searches for
>>>>>> A/AAA
>>>>>> records for names like:
>>>>>> mzidek.example.com
>>>>>> mzidek.example.com.searchdom1.com
>>>>>> mzidek.example.com.searchdom2.com
>>>>>> mzidek.example.com.searchdom3.com
>>>>>
>>>>> IIRC we try to canonicalize the host name before chopping of the
>>>>> first
>>>>> part and using the rest as domain name.
>>>>>
>>>>> Would using gethostname() be safe in real world environments?
>>>>> I know that host names are often set to complete garbage and
can't be
>>>>> relied on..but is it OK to just say "fix your environment"
in that
>>>>> case?
>>>>>
>>>>> Personally I would argue the current canonicalization doesn't
add
>>>>> much,
>>>>> because if the host name is set to complete garbage then the
>>>>> canonicalization
>>>>> wouldn't magically fix it..
>>>>>
>>>>>>
>>>>>> where mzidek.example.com is my hostname and searchdom1/2/3 are
>>>>>> search
>>>>>> domains specified in resolv.conf. Why do we even try to resolve
>>>>>> these
>>>>>> names? It seems like waste of time to me. If the
>>>>>> dns_discovery_domain is
>>>>>> properly set, these lookups do not appear.
>>>>>>
>>>>>> Thanks
>>>>>> Michal
>>>>>>
>>>>>> PS: Patches are the same as before.
>>>>>>
>>>>>>>
>>>>>>> Simo.
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>> New patches attached. As discussed above, I changed the timeout to
>>>> contact DNS server from 5s to 2s, and timeout for the entire DNS
>>>> resolution
>>>> from 5 to 6 seconds (instead of 15s as proposed originally)
>>>> (so it has enough time to contact 3 nameservers).
>>>>
>>>> I tested it with one and two unreachable nameservers.
>>>>
>>>> Thanks
>>>> Michal
>>>
>>> Works for me, too, so I'll ACK the patches. But I'll leave them on
the
>>> list for a day longer in case somebody else would like to voice their
>>> concern about changing the default timeout value. I personally think
>>> it's OK.
>>
>> Pushed to master and sssd-1-10
>
> Hi,
>
> attached is a 1.9 bacport of the patches. Michal, do you think we need
> to lower the FO_DEFAULT_SVC_TIMEOUT as well? I guess so, but you know
> the details better..
>
FO_DEFAULT_SVC_TIMEOUT is not used anywhere. It is actually removed in
the two minor fixes you posted earlier .
Ack to both patches.
Michal
Just remembered. If you backport these patches, we may consider
backporting d910b4ebaa8e68bd24ad22b8f65aed7c6812aaf1 as well.
Backported patch is attached.
Thanks
Michal
10:48 a.m.
On Fri, Aug 09, 2013 at 05:14:32PM +0200, Michal Židek wrote:
On 08/09/2013 04:33 PM, Michal Židek wrote:
>On 08/09/2013 11:51 AM, Jakub Hrozek wrote:
>>On Wed, Jul 24, 2013 at 02:00:38PM +0200, Jakub Hrozek wrote:
>>>On Tue, Jul 23, 2013 at 06:57:08PM +0200, Jakub Hrozek wrote:
>>>>On Mon, Jul 22, 2013 at 06:06:39PM +0200, Michal Židek wrote:
>>>>>On 07/19/2013 12:15 PM, Jakub Hrozek wrote:
>>>>>>On Mon, Jul 08, 2013 at 09:43:42PM +0200, Michal Židek wrote:
>>>>>>>On 07/04/2013 07:45 PM, Simo Sorce wrote:
>>>>>>>>On Thu, 2013-07-04 at 16:23 +0200, Michal Židek wrote:
>>>>>>>>>https://fedorahosted.org/sssd/ticket/1966 (SSSD
failover
>>>>>>>>>doesn't work if
>>>>>>>>>the first DNS server in resolv.conf is unavailable).
>>>>>>>>>
>>>>>>>>>The problem here is, that if first nameserver in
resolv.conf is
>>>>>>>>>down,
>>>>>>>>>the resolution is too slow and SSSD will not wait for
the
>>>>>>>>>result of
>>>>>>>>>ares_search and go offline. In my case the resolution
was
>>>>>>>>>sometimes more
>>>>>>>>>than a minute, because all search domains in
resolv.conf were
>>>>>>>>>searched
>>>>>>>>>inside ares_search() call using the first (not
working)
>>>>>>>>>nameserver in
>>>>>>>>>the first place and then with the working nameserver
(and
>>>>>>>>>before that,
>>>>>>>>>SSSD tried to figure out the domain name from my
incorrectly set
>>>>>>>>>hostname, which added more unnecessary DNS lookups).
>>>>>>>>>
>>>>>>>>>To avoid this problem, the option
dns_discovery_domain must be set
>>>>>>>>>properly, so that only the correct domain is
searched, but even
>>>>>>>>>that is
>>>>>>>>>not enough, because the default timeout for dns
resolver
>>>>>>>>>operation in
>>>>>>>>>sssd is too low. This patch rises the default value
to 15 seconds
>>>>>>>>>(instead of 5 seconds).
>>>>>>>>>
>>>>>>>>>Another option might be to lower the amount of time
ares waits
>>>>>>>>>for a
>>>>>>>>>nameserver to respond (currently it is 5 seconds,
that is why 5
>>>>>>>>>second
>>>>>>>>>for the entire dns resolution is not sufficient), but
I do not
>>>>>>>>>want to
>>>>>>>>>do this.
>>>>>>>>
>>>>>>>>Why not ?
>>>>>>>>
>>>>>>>>5 seconds these days is quite awfully high all in all.
>>>>>>>>I think 1 second should be plenty in most cases, so maybe
a
>>>>>>>>default of 2
>>>>>>>>seconds should be considered.
>>>>>>>>
>>>>>>>
>>>>>>>I also think that 5 seconds is high value, but in general, I
have
>>>>>>>bad
>>>>>>>feeling when lowering not configurable timeouts
(theoretically it
>>>>>>>could
>>>>>>>cause some DNS servers that are reachable now to become
>>>>>>>unreachable).
>>>>>>>
>>>>>>
>>>>>>Right, but in practice I can't think of a valid network
configuration
>>>>>>where the name server would take so long to reply. I mean, if
the
>>>>>>network was so slow that just resolving name would take 5
seconds,
>>>>>>then
>>>>>>chances are we wouldn't be able to download the entry from
LDAP at
>>>>>>all.
>>>>>>And if the nameserver is slow, then let's rather skip it and
fail
>>>>>>over
>>>>>>to another one.
>>>>>>
>>>>>>>Another possibility would be to make this value configurable.
Then I
>>>>>>>would have no problem lowering the default value. But,
should
>>>>>>>this be
>>>>>>>configurable? We already have one option to configure timeout
for
>>>>>>>the
>>>>>>>entire DNS resolution, adding another timeout for subtaks of
this
>>>>>>>operation does not sound nice to me. But I do not have
strong
>>>>>>>opinion
>>>>>>>on this.
>>>>>>>
>>>>>>
>>>>>>Here is what we discussed with Michal today:
>>>>>> * make the per-nameserver timeout lower a little. Now
it's 5
>>>>>> seconds, which is too long especially considering the
failover
>>>>>> timeout is 5 seconds as well. We could change it to 2 or
3
>>>>>> seconds.
>>>>>> * the glibc allows 3 nameservers. The failover request
timeut
>>>>>>could
>>>>>> then default to 3*nameserver_timeout to give the resolver
a
>>>>>>chance
>>>>>> to iterate over all the configured name servers.
>>>>>>
>>>>>>>Btw, once the correct data is returned from DNS servers, we
no
>>>>>>>longer
>>>>>>>require any DNS operation, so it only happens first time the
service
>>>>>>>is required (for example the first 'getent' call will
be slow, but
>>>>>>>after that the service is fast). This is not so bad IMO.
>>>>>>>
>>>>>>>>>These patches also change man pages, so probably
master only
>>>>>>>>>(string
>>>>>>>>>freeze)? Even if this is a really small change.
>>>>>>
>>>>>>We accept changes again now that we've released 1.10.1.
>>>>>>
>>>>>>But this change would be OK even for a string-frozen release,
>>>>>>because it
>>>>>>only changes the number, not translatable strings.
>>>>>>
>>>>>>>>>
>>>>>>>>>I was also thinking, would it make sense to write a
warning to
>>>>>>>>>the logs
>>>>>>>>>if the dns_discovery_domain option is not set? It
seem to be
>>>>>>>>>important
>>>>>>>>>to set it properly for cases like this one.
>>>>>>>>
>>>>>>>>I wonder, should we use the machine name to automatically
set the
>>>>>>>>discovery domain, and only if that fails fall back to the
ones
>>>>>>>>defined
>>>>>>>>in resolv.conf ?
>>>>>>>
>>>>>>>Actually, we do this. Wireshark shows that the first srv
lookup was
>>>>>>>constructed from my hostname. But the whole process is
unclear to
>>>>>>>me,
>>>>>>>for example I do not know why we call the
>>>>>>>resolv_gethostbyname_send()
>>>>>>>inside of resolv_get_domain_send(). It adds another DNS
lookups
>>>>>>>for A records (which causes another slow-down in the case of
bad
>>>>>>>nameserver searched first). In wireshark I also see searches
for
>>>>>>>A/AAA
>>>>>>>records for names like:
>>>>>>>mzidek.example.com
>>>>>>>mzidek.example.com.searchdom1.com
>>>>>>>mzidek.example.com.searchdom2.com
>>>>>>>mzidek.example.com.searchdom3.com
>>>>>>
>>>>>>IIRC we try to canonicalize the host name before chopping of the
>>>>>>first
>>>>>>part and using the rest as domain name.
>>>>>>
>>>>>>Would using gethostname() be safe in real world environments?
>>>>>>I know that host names are often set to complete garbage and
can't be
>>>>>>relied on..but is it OK to just say "fix your
environment" in that
>>>>>>case?
>>>>>>
>>>>>>Personally I would argue the current canonicalization doesn't
add
>>>>>>much,
>>>>>>because if the host name is set to complete garbage then the
>>>>>>canonicalization
>>>>>>wouldn't magically fix it..
>>>>>>
>>>>>>>
>>>>>>>where mzidek.example.com is my hostname and searchdom1/2/3
are
>>>>>>>search
>>>>>>>domains specified in resolv.conf. Why do we even try to
resolve
>>>>>>>these
>>>>>>>names? It seems like waste of time to me. If the
>>>>>>>dns_discovery_domain is
>>>>>>>properly set, these lookups do not appear.
>>>>>>>
>>>>>>>Thanks
>>>>>>>Michal
>>>>>>>
>>>>>>>PS: Patches are the same as before.
>>>>>>>
>>>>>>>>
>>>>>>>>Simo.
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>New patches attached. As discussed above, I changed the timeout to
>>>>>contact DNS server from 5s to 2s, and timeout for the entire DNS
>>>>>resolution
>>>>>from 5 to 6 seconds (instead of 15s as proposed originally)
>>>>>(so it has enough time to contact 3 nameservers).
>>>>>
>>>>>I tested it with one and two unreachable nameservers.
>>>>>
>>>>>Thanks
>>>>>Michal
>>>>
>>>>Works for me, too, so I'll ACK the patches. But I'll leave them
on the
>>>>list for a day longer in case somebody else would like to voice their
>>>>concern about changing the default timeout value. I personally think
>>>>it's OK.
>>>
>>>Pushed to master and sssd-1-10
>>
>>Hi,
>>
>>attached is a 1.9 bacport of the patches. Michal, do you think we need
>>to lower the FO_DEFAULT_SVC_TIMEOUT as well? I guess so, but you know
>>the details better..
>>
>
>
>FO_DEFAULT_SVC_TIMEOUT is not used anywhere. It is actually removed in
>the two minor fixes you posted earlier .
>
>Ack to both patches.
>
>Michal
>
Just remembered. If you backport these patches, we may consider
backporting d910b4ebaa8e68bd24ad22b8f65aed7c6812aaf1 as well.
Backported patch is attached.
Thanks
Michal
OK, so attached are three patches I'd like to push to 1.9
11:07 a.m.
On 08/09/2013 05:48 PM, Jakub Hrozek wrote:
On Fri, Aug 09, 2013 at 05:14:32PM +0200, Michal Židek wrote:
> On 08/09/2013 04:33 PM, Michal Židek wrote:
>> On 08/09/2013 11:51 AM, Jakub Hrozek wrote:
>>> On Wed, Jul 24, 2013 at 02:00:38PM +0200, Jakub Hrozek wrote:
>>>> On Tue, Jul 23, 2013 at 06:57:08PM +0200, Jakub Hrozek wrote:
>>>>> On Mon, Jul 22, 2013 at 06:06:39PM +0200, Michal Židek wrote:
>>>>>> On 07/19/2013 12:15 PM, Jakub Hrozek wrote:
>>>>>>> On Mon, Jul 08, 2013 at 09:43:42PM +0200, Michal Židek
wrote:
>>>>>>>> On 07/04/2013 07:45 PM, Simo Sorce wrote:
>>>>>>>>> On Thu, 2013-07-04 at 16:23 +0200, Michal Židek
wrote:
>>>>>>>>>> https://fedorahosted.org/sssd/ticket/1966 (SSSD
failover
>>>>>>>>>> doesn't work if
>>>>>>>>>> the first DNS server in resolv.conf is
unavailable).
>>>>>>>>>>
>>>>>>>>>> The problem here is, that if first nameserver in
resolv.conf is
>>>>>>>>>> down,
>>>>>>>>>> the resolution is too slow and SSSD will not wait
for the
>>>>>>>>>> result of
>>>>>>>>>> ares_search and go offline. In my case the
resolution was
>>>>>>>>>> sometimes more
>>>>>>>>>> than a minute, because all search domains in
resolv.conf were
>>>>>>>>>> searched
>>>>>>>>>> inside ares_search() call using the first (not
working)
>>>>>>>>>> nameserver in
>>>>>>>>>> the first place and then with the working
nameserver (and
>>>>>>>>>> before that,
>>>>>>>>>> SSSD tried to figure out the domain name from my
incorrectly set
>>>>>>>>>> hostname, which added more unnecessary DNS
lookups).
>>>>>>>>>>
>>>>>>>>>> To avoid this problem, the option
dns_discovery_domain must be set
>>>>>>>>>> properly, so that only the correct domain is
searched, but even
>>>>>>>>>> that is
>>>>>>>>>> not enough, because the default timeout for dns
resolver
>>>>>>>>>> operation in
>>>>>>>>>> sssd is too low. This patch rises the default
value to 15 seconds
>>>>>>>>>> (instead of 5 seconds).
>>>>>>>>>>
>>>>>>>>>> Another option might be to lower the amount of
time ares waits
>>>>>>>>>> for a
>>>>>>>>>> nameserver to respond (currently it is 5 seconds,
that is why 5
>>>>>>>>>> second
>>>>>>>>>> for the entire dns resolution is not sufficient),
but I do not
>>>>>>>>>> want to
>>>>>>>>>> do this.
>>>>>>>>>
>>>>>>>>> Why not ?
>>>>>>>>>
>>>>>>>>> 5 seconds these days is quite awfully high all in
all.
>>>>>>>>> I think 1 second should be plenty in most cases, so
maybe a
>>>>>>>>> default of 2
>>>>>>>>> seconds should be considered.
>>>>>>>>>
>>>>>>>>
>>>>>>>> I also think that 5 seconds is high value, but in
general, I have
>>>>>>>> bad
>>>>>>>> feeling when lowering not configurable timeouts
(theoretically it
>>>>>>>> could
>>>>>>>> cause some DNS servers that are reachable now to become
>>>>>>>> unreachable).
>>>>>>>>
>>>>>>>
>>>>>>> Right, but in practice I can't think of a valid network
configuration
>>>>>>> where the name server would take so long to reply. I mean, if
the
>>>>>>> network was so slow that just resolving name would take 5
seconds,
>>>>>>> then
>>>>>>> chances are we wouldn't be able to download the entry
from LDAP at
>>>>>>> all.
>>>>>>> And if the nameserver is slow, then let's rather skip it
and fail
>>>>>>> over
>>>>>>> to another one.
>>>>>>>
>>>>>>>> Another possibility would be to make this value
configurable. Then I
>>>>>>>> would have no problem lowering the default value. But,
should
>>>>>>>> this be
>>>>>>>> configurable? We already have one option to configure
timeout for
>>>>>>>> the
>>>>>>>> entire DNS resolution, adding another timeout for subtaks
of this
>>>>>>>> operation does not sound nice to me. But I do not have
strong
>>>>>>>> opinion
>>>>>>>> on this.
>>>>>>>>
>>>>>>>
>>>>>>> Here is what we discussed with Michal today:
>>>>>>> * make the per-nameserver timeout lower a little. Now
it's 5
>>>>>>> seconds, which is too long especially considering the
failover
>>>>>>> timeout is 5 seconds as well. We could change it to 2
or 3
>>>>>>> seconds.
>>>>>>> * the glibc allows 3 nameservers. The failover request
timeut
>>>>>>> could
>>>>>>> then default to 3*nameserver_timeout to give the
resolver a
>>>>>>> chance
>>>>>>> to iterate over all the configured name servers.
>>>>>>>
>>>>>>>> Btw, once the correct data is returned from DNS servers,
we no
>>>>>>>> longer
>>>>>>>> require any DNS operation, so it only happens first time
the service
>>>>>>>> is required (for example the first 'getent' call
will be slow, but
>>>>>>>> after that the service is fast). This is not so bad IMO.
>>>>>>>>
>>>>>>>>>> These patches also change man pages, so probably
master only
>>>>>>>>>> (string
>>>>>>>>>> freeze)? Even if this is a really small change.
>>>>>>>
>>>>>>> We accept changes again now that we've released 1.10.1.
>>>>>>>
>>>>>>> But this change would be OK even for a string-frozen
release,
>>>>>>> because it
>>>>>>> only changes the number, not translatable strings.
>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I was also thinking, would it make sense to write
a warning to
>>>>>>>>>> the logs
>>>>>>>>>> if the dns_discovery_domain option is not set? It
seem to be
>>>>>>>>>> important
>>>>>>>>>> to set it properly for cases like this one.
>>>>>>>>>
>>>>>>>>> I wonder, should we use the machine name to
automatically set the
>>>>>>>>> discovery domain, and only if that fails fall back to
the ones
>>>>>>>>> defined
>>>>>>>>> in resolv.conf ?
>>>>>>>>
>>>>>>>> Actually, we do this. Wireshark shows that the first srv
lookup was
>>>>>>>> constructed from my hostname. But the whole process is
unclear to
>>>>>>>> me,
>>>>>>>> for example I do not know why we call the
>>>>>>>> resolv_gethostbyname_send()
>>>>>>>> inside of resolv_get_domain_send(). It adds another DNS
lookups
>>>>>>>> for A records (which causes another slow-down in the case
of bad
>>>>>>>> nameserver searched first). In wireshark I also see
searches for
>>>>>>>> A/AAA
>>>>>>>> records for names like:
>>>>>>>> mzidek.example.com
>>>>>>>> mzidek.example.com.searchdom1.com
>>>>>>>> mzidek.example.com.searchdom2.com
>>>>>>>> mzidek.example.com.searchdom3.com
>>>>>>>
>>>>>>> IIRC we try to canonicalize the host name before chopping of
the
>>>>>>> first
>>>>>>> part and using the rest as domain name.
>>>>>>>
>>>>>>> Would using gethostname() be safe in real world
environments?
>>>>>>> I know that host names are often set to complete garbage and
can't be
>>>>>>> relied on..but is it OK to just say "fix your
environment" in that
>>>>>>> case?
>>>>>>>
>>>>>>> Personally I would argue the current canonicalization
doesn't add
>>>>>>> much,
>>>>>>> because if the host name is set to complete garbage then the
>>>>>>> canonicalization
>>>>>>> wouldn't magically fix it..
>>>>>>>
>>>>>>>>
>>>>>>>> where mzidek.example.com is my hostname and
searchdom1/2/3 are
>>>>>>>> search
>>>>>>>> domains specified in resolv.conf. Why do we even try to
resolve
>>>>>>>> these
>>>>>>>> names? It seems like waste of time to me. If the
>>>>>>>> dns_discovery_domain is
>>>>>>>> properly set, these lookups do not appear.
>>>>>>>>
>>>>>>>> Thanks
>>>>>>>> Michal
>>>>>>>>
>>>>>>>> PS: Patches are the same as before.
>>>>>>>>
>>>>>>>>>
>>>>>>>>> Simo.
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>> New patches attached. As discussed above, I changed the timeout
to
>>>>>> contact DNS server from 5s to 2s, and timeout for the entire DNS
>>>>>> resolution
>>>>> >from 5 to 6 seconds (instead of 15s as proposed originally)
>>>>>> (so it has enough time to contact 3 nameservers).
>>>>>>
>>>>>> I tested it with one and two unreachable nameservers.
>>>>>>
>>>>>> Thanks
>>>>>> Michal
>>>>>
>>>>> Works for me, too, so I'll ACK the patches. But I'll leave
them on the
>>>>> list for a day longer in case somebody else would like to voice
their
>>>>> concern about changing the default timeout value. I personally think
>>>>> it's OK.
>>>>
>>>> Pushed to master and sssd-1-10
>>>
>>> Hi,
>>>
>>> attached is a 1.9 bacport of the patches. Michal, do you think we need
>>> to lower the FO_DEFAULT_SVC_TIMEOUT as well? I guess so, but you know
>>> the details better..
>>>
>>
>>
>> FO_DEFAULT_SVC_TIMEOUT is not used anywhere. It is actually removed in
>> the two minor fixes you posted earlier .
>>
>> Ack to both patches.
>>
>> Michal
>>
>
> Just remembered. If you backport these patches, we may consider
> backporting d910b4ebaa8e68bd24ad22b8f65aed7c6812aaf1 as well.
>
> Backported patch is attached.
>
> Thanks
> Michal
OK, so attached are three patches I'd like to push to 1.9
Ack to all 3.
Michal
11:15 a.m.
On Fri, Aug 09, 2013 at 06:07:46PM +0200, Michal Židek wrote:
>>Just remembered. If you backport these patches, we may
consider
>>backporting d910b4ebaa8e68bd24ad22b8f65aed7c6812aaf1 as well.
>>
>>Backported patch is attached.
>>
>>Thanks
>>Michal
>
>OK, so attached are three patches I'd like to push to 1.9
>
Ack to all 3.
Michal
Pushed all three to sssd-1-9
3913
days inactive
3949
days old
sssd-devel@lists.fedorahosted.org
14 comments
3 participants
participants (3)
-
Jakub Hrozek -
Michal Židek -
Simo Sorce