<html>
<head>
<meta content="text/html; charset=UTF-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<div class="moz-cite-prefix">On 09/10/2013 04:35 PM, Ludwig Krispenz
wrote:<br>
</div>
<blockquote cite="mid:522F2E1B.9010600@redhat.com" type="cite">
<br>
On 09/10/2013 04:29 PM, Rich Megginson wrote:
<br>
<blockquote type="cite">On 09/10/2013 01:47 AM, Ludwig Krispenz
wrote:
<br>
<blockquote type="cite">
<br>
On 09/09/2013 07:19 PM, Rich Megginson wrote:
<br>
<blockquote type="cite">On 09/09/2013 02:27 AM, Ludwig
Krispenz wrote:
<br>
<blockquote type="cite">
<br>
On 09/07/2013 05:02 AM, David Boreham wrote:
<br>
<blockquote type="cite">On 9/6/2013 8:49 PM, Nathan Kinder
wrote:
<br>
<blockquote type="cite">This is a good idea, and it is
something that we discussed briefly off-list. The
only downside is that we need to change the index
format to keep a count of ids for each key.
Implementing this isn't a big problem, but it does
mean that the existing indexes need to be updated to
populate the count based off of the contents (as you
mention above).
<br>
</blockquote>
<br>
I don't think you need to do this (I certainly wasn't
advocating doing so). The "statistics" state is much the
same as that proposed in Rich's design. In fact you
could probably just use that same information. My idea
is more about where and how you use the information. All
you need is something associated with each index that
says "not much point looking here if you're after
something specific, move along, look somewhere else
instead". This is much the same information as "don't
use a high scan limit here".
<br>
<br>
<blockquote type="cite">
<br>
In the short term, we are looking for a way to be able
to improve performance for specific search filters
that are not possible to modify on the client side
(for whatever reason) while leaving the index file
format exactly as it is. I still feel that there is
potentially great value in keeping a count of ids per
key so we can optimize things on the server side
automatically without the need for complex index
configuration on the administrator's part. I think we
should consider this for an additional future
enhancement.
<br>
</blockquote>
<br>
I'm saying the same thing. Keeping a cardinality count
per key is way more than I'm proposing, and I'm not sure
how useful that would be anyway, unless you want to do
OLAP in the DS ;)
<br>
</blockquote>
we have the cardinality of the key in old-idl and this
makes some searches where parts of the filter are allids
fast.
<br>
<br>
I'm late in the discussion, but I think Rich's proposal is
very promising to address all the problems related to
allids in new-idl.
<br>
<br>
We could then eventually rework filter ordering based on
these configurations. Right now we only have a filter
ordering based on index type and try to postpone "<="
or similar filter as they are known to be costly, but this
could be more elaborate.
<br>
<br>
An alternative would be to have some kind of index lookup
caching. In the example in ticket 47474 the filter is
(&(|(objectClass=organizationalPerson)(objectClass=inetOrgPerson)(objectClass=organization)(objectClass=organizationalUnit)(objectClass=groupOf<br>
Names)(objectClass=groupOfUniqueNames)(objectClass=group))(c3sUserID=EndUser0000078458))"
and probably only the "c3sUserID=xxxxx" part will change,
if we cache the result for the (&(|(objectClass=...
part, even if it is expensive, it would be done only once.
<br>
</blockquote>
<br>
Thanks everyone for the comments. I have added Noriko's
suggestion:
<br>
<a class="moz-txt-link-freetext" href="http://port389.org/wiki/Design/Fine_Grained_ID_List_Size">http://port389.org/wiki/Design/Fine_Grained_ID_List_Size</a>
<br>
<br>
David, Ludwig: Does the current design address your
concerns, and/or provide the necessary first step for
further refinements?
<br>
</blockquote>
yes, the topic of filter reordering or caching could be looked
at independently.
<br>
<br>
Just one concern abou the syntax:
<br>
<br>
nsIndexIDListScanLimit:
maxsize[:indextype][:flag[,flag...]][:value[,value...]]
<br>
<br>
since everything is optional, how do you decide if in
nsIndexIDListScanLimit: 6:eq:AND "AND" is a value or a flag ?
<br>
and as it defines limits for specific keys, could the
attributname reflect this, eg nsIndexKeyIDListScanLimit or
nsIndexKeyScanLimit or ... ?
<br>
</blockquote>
<br>
Thanks, yes, it is ambiguous.
<br>
I think it may have to use keyword=value, so something like
this:
<br>
<br>
nsIndexIDListScanLimit: limit=NNN [type=eq[,sub]]
[flags=ADD[,OR]] [values=val[,val...]]
<br>
<br>
That should be easy to parse for both humans and machines.
<br>
For values, will have to figure out a way to have escapes (e.g.
if a value contains a comma or an escape character). Was
thinking of using LDAP escapes (e.g. \, or \032)
<br>
</blockquote>
they should be treated as in filters and normalized, in the config
it should be the string representation according to the
attributetype
<br>
</blockquote>
<br>
Hi,<br>
<br>
<blockquote>I was wondering if this configuration attribute at the
index level, could not also be implemented at the bind-base level.<br>
If an application use to bind with a given entry, it could use its
own limitations put for example into operational attribute in the
bound entry itself.<br>
So that two applications, using the same filter component could
have their specific idlist size.<br>
Anyway if it makes sense it could be added later.<br>
</blockquote>
best regards<br>
thierry<br>
<blockquote cite="mid:522F2E1B.9010600@redhat.com" type="cite">
<blockquote type="cite">
<br>
<blockquote type="cite">
<blockquote type="cite">
<br>
<blockquote type="cite">
<blockquote type="cite">
<br>
<br>
-- <br>
389-devel mailing list
<br>
<a class="moz-txt-link-abbreviated" href="mailto:389-devel@lists.fedoraproject.org">389-devel@lists.fedoraproject.org</a>
<br>
<a class="moz-txt-link-freetext" href="https://admin.fedoraproject.org/mailman/listinfo/389-devel">https://admin.fedoraproject.org/mailman/listinfo/389-devel</a>
<br>
</blockquote>
<br>
-- <br>
389-devel mailing list
<br>
<a class="moz-txt-link-abbreviated" href="mailto:389-devel@lists.fedoraproject.org">389-devel@lists.fedoraproject.org</a>
<br>
<a class="moz-txt-link-freetext" href="https://admin.fedoraproject.org/mailman/listinfo/389-devel">https://admin.fedoraproject.org/mailman/listinfo/389-devel</a>
<br>
</blockquote>
<br>
</blockquote>
<br>
</blockquote>
<br>
</blockquote>
<br>
--
<br>
389-devel mailing list
<br>
<a class="moz-txt-link-abbreviated" href="mailto:389-devel@lists.fedoraproject.org">389-devel@lists.fedoraproject.org</a>
<br>
<a class="moz-txt-link-freetext" href="https://admin.fedoraproject.org/mailman/listinfo/389-devel">https://admin.fedoraproject.org/mailman/listinfo/389-devel</a><br>
</blockquote>
<br>
</body>
</html>