<html>
<head>
<meta content="text/html; charset=UTF-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<div class="moz-cite-prefix">On 09/12/2013 07:39 AM, thierry bordaz
wrote:<br>
</div>
<blockquote cite="mid:5231C408.6070009@redhat.com" type="cite">
<meta content="text/html; charset=UTF-8" http-equiv="Content-Type">
<div class="moz-cite-prefix">On 09/10/2013 04:35 PM, Ludwig
Krispenz wrote:<br>
</div>
<blockquote cite="mid:522F2E1B.9010600@redhat.com" type="cite"> <br>
On 09/10/2013 04:29 PM, Rich Megginson wrote: <br>
<blockquote type="cite">On 09/10/2013 01:47 AM, Ludwig Krispenz
wrote: <br>
<blockquote type="cite"> <br>
On 09/09/2013 07:19 PM, Rich Megginson wrote: <br>
<blockquote type="cite">On 09/09/2013 02:27 AM, Ludwig
Krispenz wrote: <br>
<blockquote type="cite"> <br>
On 09/07/2013 05:02 AM, David Boreham wrote: <br>
<blockquote type="cite">On 9/6/2013 8:49 PM, Nathan
Kinder wrote: <br>
<blockquote type="cite">This is a good idea, and it is
something that we discussed briefly off-list. The
only downside is that we need to change the index
format to keep a count of ids for each key.
Implementing this isn't a big problem, but it does
mean that the existing indexes need to be updated to
populate the count based off of the contents (as you
mention above). <br>
</blockquote>
<br>
I don't think you need to do this (I certainly wasn't
advocating doing so). The "statistics" state is much
the same as that proposed in Rich's design. In fact
you could probably just use that same information. My
idea is more about where and how you use the
information. All you need is something associated with
each index that says "not much point looking here if
you're after something specific, move along, look
somewhere else instead". This is much the same
information as "don't use a high scan limit here". <br>
<br>
<blockquote type="cite"> <br>
In the short term, we are looking for a way to be
able to improve performance for specific search
filters that are not possible to modify on the
client side (for whatever reason) while leaving the
index file format exactly as it is. I still feel
that there is potentially great value in keeping a
count of ids per key so we can optimize things on
the server side automatically without the need for
complex index configuration on the administrator's
part. I think we should consider this for an
additional future enhancement. <br>
</blockquote>
<br>
I'm saying the same thing. Keeping a cardinality count
per key is way more than I'm proposing, and I'm not
sure how useful that would be anyway, unless you want
to do OLAP in the DS ;) <br>
</blockquote>
we have the cardinality of the key in old-idl and this
makes some searches where parts of the filter are allids
fast. <br>
<br>
I'm late in the discussion, but I think Rich's proposal
is very promising to address all the problems related to
allids in new-idl. <br>
<br>
We could then eventually rework filter ordering based on
these configurations. Right now we only have a filter
ordering based on index type and try to postpone "<="
or similar filter as they are known to be costly, but
this could be more elaborate. <br>
<br>
An alternative would be to have some kind of index
lookup caching. In the example in ticket 47474 the
filter is
(&(|(objectClass=organizationalPerson)(objectClass=inetOrgPerson)(objectClass=organization)(objectClass=organizationalUnit)(objectClass=groupOf<br>
Names)(objectClass=groupOfUniqueNames)(objectClass=group))(c3sUserID=EndUser0000078458))"
and probably only the "c3sUserID=xxxxx" part will
change, if we cache the result for the
(&(|(objectClass=... part, even if it is expensive,
it would be done only once. <br>
</blockquote>
<br>
Thanks everyone for the comments. I have added Noriko's
suggestion: <br>
<a moz-do-not-send="true" class="moz-txt-link-freetext"
href="http://port389.org/wiki/Design/Fine_Grained_ID_List_Size">http://port389.org/wiki/Design/Fine_Grained_ID_List_Size</a>
<br>
<br>
David, Ludwig: Does the current design address your
concerns, and/or provide the necessary first step for
further refinements? <br>
</blockquote>
yes, the topic of filter reordering or caching could be
looked at independently. <br>
<br>
Just one concern abou the syntax: <br>
<br>
nsIndexIDListScanLimit:
maxsize[:indextype][:flag[,flag...]][:value[,value...]] <br>
<br>
since everything is optional, how do you decide if in
nsIndexIDListScanLimit: 6:eq:AND "AND" is a value or a flag
? <br>
and as it defines limits for specific keys, could the
attributname reflect this, eg nsIndexKeyIDListScanLimit or
nsIndexKeyScanLimit or ... ? <br>
</blockquote>
<br>
Thanks, yes, it is ambiguous. <br>
I think it may have to use keyword=value, so something like
this: <br>
<br>
nsIndexIDListScanLimit: limit=NNN [type=eq[,sub]]
[flags=ADD[,OR]] [values=val[,val...]] <br>
<br>
That should be easy to parse for both humans and machines. <br>
For values, will have to figure out a way to have escapes
(e.g. if a value contains a comma or an escape character).
Was thinking of using LDAP escapes (e.g. \, or \032) <br>
</blockquote>
they should be treated as in filters and normalized, in the
config it should be the string representation according to the
attributetype <br>
</blockquote>
<br>
Hi,<br>
<br>
<blockquote>I was wondering if this configuration attribute at the
index level, could not also be implemented at the bind-base
level.<br>
</blockquote>
</blockquote>
<br>
It could be - it would be more difficult to do - you would have to
have the nsIndexIDListScanLimit attribute specified in the user
entry, and it would have to specify the attribute type e.g. <br>
<br>
dn: uid=admin,....<br>
nsIndexIDListScanLimit: limit=xxxx attr=objectclass type=eq
value=inetOrgPerson<br>
<br>
Or perhaps a new attribute - nsIndexIDListScanLimit should be not
operational for use in nsIndex, but should be operational for use in
a user entry.<br>
<br>
<blockquote cite="mid:5231C408.6070009@redhat.com" type="cite">
<blockquote> If an application use to bind with a given entry, it
could use its own limitations put for example into operational
attribute in the bound entry itself.<br>
</blockquote>
</blockquote>
Yes, and we already do this for other limits.<br>
<blockquote cite="mid:5231C408.6070009@redhat.com" type="cite">
<blockquote> So that two applications, using the same filter
component could have their specific idlist size.<br>
Anyway if it makes sense it could be added later.<br>
</blockquote>
</blockquote>
<br>
Yes, thanks.<br>
<br>
<blockquote cite="mid:5231C408.6070009@redhat.com" type="cite">
<blockquote> </blockquote>
best regards<br>
thierry<br>
<blockquote cite="mid:522F2E1B.9010600@redhat.com" type="cite">
<blockquote type="cite"> <br>
<blockquote type="cite">
<blockquote type="cite"> <br>
<blockquote type="cite">
<blockquote type="cite"> <br>
<br>
-- <br>
389-devel mailing list <br>
<a moz-do-not-send="true"
class="moz-txt-link-abbreviated"
href="mailto:389-devel@lists.fedoraproject.org">389-devel@lists.fedoraproject.org</a>
<br>
<a moz-do-not-send="true"
class="moz-txt-link-freetext"
href="https://admin.fedoraproject.org/mailman/listinfo/389-devel">https://admin.fedoraproject.org/mailman/listinfo/389-devel</a>
<br>
</blockquote>
<br>
-- <br>
389-devel mailing list <br>
<a moz-do-not-send="true"
class="moz-txt-link-abbreviated"
href="mailto:389-devel@lists.fedoraproject.org">389-devel@lists.fedoraproject.org</a>
<br>
<a moz-do-not-send="true" class="moz-txt-link-freetext"
href="https://admin.fedoraproject.org/mailman/listinfo/389-devel">https://admin.fedoraproject.org/mailman/listinfo/389-devel</a>
<br>
</blockquote>
<br>
</blockquote>
<br>
</blockquote>
<br>
</blockquote>
<br>
-- <br>
389-devel mailing list <br>
<a moz-do-not-send="true" class="moz-txt-link-abbreviated"
href="mailto:389-devel@lists.fedoraproject.org">389-devel@lists.fedoraproject.org</a>
<br>
<a moz-do-not-send="true" class="moz-txt-link-freetext"
href="https://admin.fedoraproject.org/mailman/listinfo/389-devel">https://admin.fedoraproject.org/mailman/listinfo/389-devel</a><br>
</blockquote>
<br>
</blockquote>
<br>
</body>
</html>