<html>
<head>
<meta content="text/html; charset=UTF-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<br>
<div class="moz-cite-prefix">On 09/12/2013 04:40 PM, Rich Megginson
wrote:<br>
</div>
<blockquote cite="mid:5231D25A.1060600@redhat.com" type="cite">
<meta content="text/html; charset=UTF-8" http-equiv="Content-Type">
<div class="moz-cite-prefix">On 09/12/2013 07:39 AM, thierry
bordaz wrote:<br>
</div>
<blockquote cite="mid:5231C408.6070009@redhat.com" type="cite">
<meta content="text/html; charset=UTF-8"
http-equiv="Content-Type">
<div class="moz-cite-prefix">On 09/10/2013 04:35 PM, Ludwig
Krispenz wrote:<br>
</div>
<blockquote cite="mid:522F2E1B.9010600@redhat.com" type="cite">
<br>
On 09/10/2013 04:29 PM, Rich Megginson wrote: <br>
<blockquote type="cite">On 09/10/2013 01:47 AM, Ludwig
Krispenz wrote: <br>
<blockquote type="cite"> <br>
On 09/09/2013 07:19 PM, Rich Megginson wrote: <br>
<blockquote type="cite">On 09/09/2013 02:27 AM, Ludwig
Krispenz wrote: <br>
<blockquote type="cite"> <br>
On 09/07/2013 05:02 AM, David Boreham wrote: <br>
<blockquote type="cite">On 9/6/2013 8:49 PM, Nathan
Kinder wrote: <br>
<blockquote type="cite">This is a good idea, and it
is something that we discussed briefly off-list.
The only downside is that we need to change the
index format to keep a count of ids for each key.
Implementing this isn't a big problem, but it does
mean that the existing indexes need to be updated
to populate the count based off of the contents
(as you mention above). <br>
</blockquote>
<br>
I don't think you need to do this (I certainly
wasn't advocating doing so). The "statistics" state
is much the same as that proposed in Rich's design.
In fact you could probably just use that same
information. My idea is more about where and how you
use the information. All you need is something
associated with each index that says "not much point
looking here if you're after something specific,
move along, look somewhere else instead". This is
much the same information as "don't use a high scan
limit here". <br>
<br>
<blockquote type="cite"> <br>
In the short term, we are looking for a way to be
able to improve performance for specific search
filters that are not possible to modify on the
client side (for whatever reason) while leaving
the index file format exactly as it is. I still
feel that there is potentially great value in
keeping a count of ids per key so we can optimize
things on the server side automatically without
the need for complex index configuration on the
administrator's part. I think we should consider
this for an additional future enhancement. <br>
</blockquote>
<br>
I'm saying the same thing. Keeping a cardinality
count per key is way more than I'm proposing, and
I'm not sure how useful that would be anyway, unless
you want to do OLAP in the DS ;) <br>
</blockquote>
we have the cardinality of the key in old-idl and this
makes some searches where parts of the filter are
allids fast. <br>
<br>
I'm late in the discussion, but I think Rich's
proposal is very promising to address all the problems
related to allids in new-idl. <br>
<br>
We could then eventually rework filter ordering based
on these configurations. Right now we only have a
filter ordering based on index type and try to
postpone "<=" or similar filter as they are known
to be costly, but this could be more elaborate. <br>
<br>
An alternative would be to have some kind of index
lookup caching. In the example in ticket 47474 the
filter is
(&(|(objectClass=organizationalPerson)(objectClass=inetOrgPerson)(objectClass=organization)(objectClass=organizationalUnit)(objectClass=groupOf<br>
Names)(objectClass=groupOfUniqueNames)(objectClass=group))(c3sUserID=EndUser0000078458))"
and probably only the "c3sUserID=xxxxx" part will
change, if we cache the result for the
(&(|(objectClass=... part, even if it is
expensive, it would be done only once. <br>
</blockquote>
<br>
Thanks everyone for the comments. I have added Noriko's
suggestion: <br>
<a moz-do-not-send="true" class="moz-txt-link-freetext"
href="http://port389.org/wiki/Design/Fine_Grained_ID_List_Size">http://port389.org/wiki/Design/Fine_Grained_ID_List_Size</a>
<br>
<br>
David, Ludwig: Does the current design address your
concerns, and/or provide the necessary first step for
further refinements? <br>
</blockquote>
yes, the topic of filter reordering or caching could be
looked at independently. <br>
<br>
Just one concern abou the syntax: <br>
<br>
nsIndexIDListScanLimit:
maxsize[:indextype][:flag[,flag...]][:value[,value...]] <br>
<br>
since everything is optional, how do you decide if in
nsIndexIDListScanLimit: 6:eq:AND "AND" is a value or a
flag ? <br>
and as it defines limits for specific keys, could the
attributname reflect this, eg nsIndexKeyIDListScanLimit or
nsIndexKeyScanLimit or ... ? <br>
</blockquote>
<br>
Thanks, yes, it is ambiguous. <br>
I think it may have to use keyword=value, so something like
this: <br>
<br>
nsIndexIDListScanLimit: limit=NNN [type=eq[,sub]]
[flags=ADD[,OR]] [values=val[,val...]] <br>
<br>
That should be easy to parse for both humans and machines. <br>
For values, will have to figure out a way to have escapes
(e.g. if a value contains a comma or an escape character).
Was thinking of using LDAP escapes (e.g. \, or \032) <br>
</blockquote>
they should be treated as in filters and normalized, in the
config it should be the string representation according to the
attributetype <br>
</blockquote>
<br>
Hi,<br>
<br>
<blockquote>I was wondering if this configuration attribute at
the index level, could not also be implemented at the
bind-base level.<br>
</blockquote>
</blockquote>
<br>
It could be - it would be more difficult to do - you would have to
have the nsIndexIDListScanLimit attribute specified in the user
entry, and it would have to specify the attribute type e.g. <br>
<br>
dn: uid=admin,....<br>
nsIndexIDListScanLimit: limit=xxxx attr=objectclass type=eq
value=inetOrgPerson<br>
<br>
Or perhaps a new attribute - nsIndexIDListScanLimit should be not
operational for use in nsIndex, but should be operational for use
in a user entry.<br>
</blockquote>
Or it could be handled as a policy, like password policy, have a
default one and the possibility to assign a specific one at the bind<br>
<blockquote cite="mid:5231D25A.1060600@redhat.com" type="cite"> <br>
<blockquote cite="mid:5231C408.6070009@redhat.com" type="cite">
<blockquote> If an application use to bind with a given entry,
it could use its own limitations put for example into
operational attribute in the bound entry itself.<br>
</blockquote>
</blockquote>
Yes, and we already do this for other limits.<br>
<blockquote cite="mid:5231C408.6070009@redhat.com" type="cite">
<blockquote> So that two applications, using the same filter
component could have their specific idlist size.<br>
Anyway if it makes sense it could be added later.<br>
</blockquote>
</blockquote>
<br>
Yes, thanks.<br>
<br>
<blockquote cite="mid:5231C408.6070009@redhat.com" type="cite">
<blockquote> </blockquote>
best regards<br>
thierry<br>
<blockquote cite="mid:522F2E1B.9010600@redhat.com" type="cite">
<blockquote type="cite"> <br>
<blockquote type="cite">
<blockquote type="cite"> <br>
<blockquote type="cite">
<blockquote type="cite"> <br>
<br>
-- <br>
389-devel mailing list <br>
<a moz-do-not-send="true"
class="moz-txt-link-abbreviated"
href="mailto:389-devel@lists.fedoraproject.org">389-devel@lists.fedoraproject.org</a>
<br>
<a moz-do-not-send="true"
class="moz-txt-link-freetext"
href="https://admin.fedoraproject.org/mailman/listinfo/389-devel">https://admin.fedoraproject.org/mailman/listinfo/389-devel</a>
<br>
</blockquote>
<br>
-- <br>
389-devel mailing list <br>
<a moz-do-not-send="true"
class="moz-txt-link-abbreviated"
href="mailto:389-devel@lists.fedoraproject.org">389-devel@lists.fedoraproject.org</a>
<br>
<a moz-do-not-send="true"
class="moz-txt-link-freetext"
href="https://admin.fedoraproject.org/mailman/listinfo/389-devel">https://admin.fedoraproject.org/mailman/listinfo/389-devel</a>
<br>
</blockquote>
<br>
</blockquote>
<br>
</blockquote>
<br>
</blockquote>
<br>
-- <br>
389-devel mailing list <br>
<a moz-do-not-send="true" class="moz-txt-link-abbreviated"
href="mailto:389-devel@lists.fedoraproject.org">389-devel@lists.fedoraproject.org</a>
<br>
<a moz-do-not-send="true" class="moz-txt-link-freetext"
href="https://admin.fedoraproject.org/mailman/listinfo/389-devel">https://admin.fedoraproject.org/mailman/listinfo/389-devel</a><br>
</blockquote>
<br>
</blockquote>
<br>
</blockquote>
<br>
</body>
</html>