On Thu, 2009-08-13 at 13:17 -0400, Dmitri Pal wrote:
Simo Sorce wrote:
> Hello all,
> during this month I have been slowly working on a set of patches to move
> from storing information in 2 different formats (legacy and
> member/memberOf based) to just one format (member/memberOf based).
> While doing this I had to address some problems that come up when you
> want to store a group and its members have not been stored yet, and
> cases like this.
> All the while I have been testing doing enumerations against a server
> that has more than 3k users and 3k groups.
> This is a medium sized database, and yet getting groups from scratch
> (startup after deleting the .ldb database) could take up to a minute;
> granted the operation is quite a bit faster if the database just needs
> updating and not creation from scratch, but I still think it's too much.
> I've been thinking hard about how to address this problem and solve the
> few hacks we have in the code when it comes to enumeration caching and
> retrieval. We always said that enumerations are evil (and they are
> indeed) and in fact we even have options that disable enumerations by
> default. Yet I think this is not necessarily the right way to go.
> I think we have 2 major problems in our current architecture when it
> comes to enumerations.
> 1) we try to hit the wire when an enumeration request comes in from a
> process and a (small) timeout for the previous enumeration has been
May be then we should as I quick fix have a separate timeout for the
We already have, but a timeout is not going to make any difference to
the call that comes in when it is expired.
> 2) We run the enumeration in a single transaction (and yes I
> recently introduced this), which means any other operation is blocked
> until the enumeration is finished.
Can we create a special back end for enumerations and separate it from
Not sure what you mean here.
> The problem I actually see is that user space apps may have to
> too much, and this *will* turn out to be a problem.
> Even if we give the
> option to turn off enumeration I think that for apps that needs it the
> penalty has become simply too big. Also I think the way we have to
> perform updates using this model is largely inefficient, as we basically
> perform a full new search potentially every few minutes.
Agree, though I think that we can separate these enhancements and do
them as a separate effort (may be later, after F12 if we do not have time).
It has structural implications for the driver, but I am not forcing it
up the schedule, I am interested in the technical discussion at the
> After some hard thinking I wrote down a few points I'd like
> opinion on. If people agree I will start acting on them.
> * stop performing enumerations on demand, and perform them in background
> if enumerations are activated (change the enumeration parameter from a
> bitfield to a true/flase boolean)
Is a separate back end may be?
Don't see any help in having a separate backend, just a lot more
> * perform a full user+group enumeration at startup (possibly
> paged or vlv search)
Yes, I agree, but there is a concern (see below).
> * when possible request the modifyTimestamp attribute and save
> highest modifyTimestamp into the domain entry as originalMaxTimestamp
> * on a tunable interval run a task that refreshes all users and groups
> in the background using a search filter that includes
Okey... Steven explain it in more details since i was concerned about
the time stamp being also updated
on individual refreshes but he said that this time stamp will be touched
only by enumerations.
the originalMaxTimestamp can only be updated on enumerations refreshes,
not when resolving individual entries, or we could miss changes.
Hm I see how that would work.
> * still do a full refresh every X minutes/hours
> * disable using a single huge transaction for enumerations (we
> ok doing a transaction for each page search if pages are small,
> otherwise just revert to the previous behavior of having a transaction
> per stored object)
Can you do page - individual request - page -individual request?
That's my hope, the ldap protocol allows that although I am not sure if
all ldap servers actually allow it. I have a backup plan where we have
an option that makes the ldap driver use a separate connection for the
enumeration refreshes if the remote server does not cope well with
multiple requests being sent at the same time.
If yes then it makes sense to have transaction per page.
when you use pages each page request is actually a new ldap search with
attached a cookie, I don't think there is any problem in intermixing
regular operations between each search request in a page search, but see
the backup plan above if it turns out some ldap server has problems with
If you have to do pages one after another and can't do other
the middle I do not see how changing transaction scope would help.
Still using the trick above I could open 2 connections and perform
enumerations and other searches in parallel, but the single transaction
would block every write access to the db until the slowest request is
finished. That is why I think I will break again the enumeration into a
transaction per page.
A transaction per entry is also possible, it is more expensive (lots of
fsyncs) but if enumerations are done in the background that's a bit less
critical in term of waiting time.
> * Every time we update an entry we store the
> it as a copy of the remote modifyTimestamp, this allows us to know if we
> actually need to touch the cached entry at all upon refresh (like when a
> getpwnam() is called, speeding up operations for entries that need no
> refresh (we will avoid data transformation and a writing to ldb).
Is this only for enumerations or on individual updates too?
It can be used to test if the retrieved entry is newer or not so that we
can save on storing on disk again (write is expensive). But it is
primarily for enumerations.
How it is related to the currently designed and being implemented
This is backend side.
Can you please explain how this would affect the cache logic?
It depends on what cache you are referring to, if you are referring to
refreshes performed by the frontend this may make them unnecessary for
the ldap driver.
> * Every time we run the general refresh task or we save a
> we store a LastUpdatedTimestamp
> * When the refresh task is completed successfully we run another cleanup
> task that searches our LDB for any entry that has a too old
> LastUpdatedTimestamp. If any is found, we double check against the
> remote server if the entry still exists (and update it if it does), and
> otherwise we delete it.
Makes sense. I actually makes me think that with out of band periodic
enumerations and cleanups it becomes more and more appealing
to have it in a separate back end.
Not sure what you mean by that or why it would be appealing, can you
> NOTE: this means that until the first background enumeration is
> complete, a getent passwd or a getent group call may return incomplete
> results. I think this is acceptable as it will really happen only at
> startup, when the daemon caches are empty.
Here is my concern I mentioned above: How many services and daemons at
startup rely on the enumeration?
I think none, only very bad apps really rely on enumeration.
Do we know? Is there any way to know? Should we ask the communities
about those to make sure we meet the expectations?
In my experience generally if ldap or nis are not available the machine
still comes up fine. Posix even says explicitly that enumeration request
can returning nothing IIRC.
Also experience with samba's winbindd with enumeration turned off tells
me this is not going to be a problem.
What about things line network manager, HAL, System bus, auditd, GDE
many other processes that start at boot?
They all just need at most system accounts (which are local), they could
rightly care less a bout other users.
If we block them it might be a show stopper if we provide partial
it might be Ok might be not.
I guess we need more input on the matter. Do you agree?
No I think I have enough knowledge to establish that enumerations are
not critical at all (and that's why we have them disabled by default in
the current released code).
> NOTE2: Off course the scheduled refreshes and cleanup tasks are
> rescheduled if we are offline or if a fatal error occurs during the
IMO this is yet another reason to have a separate back end.
And I still don't get what's the point :)
> NOTE3: I am proposing to change only the way enumerations are
> single user or group lookups will remain unchanged for now and will be
> dealt with later if needed.
Sure. I think the only impact is that the entry might be refreshed with
the enumeration and we need to factor a new time stamp
in the cache refresh logic. Other than that I do not see any impact.
We always store the timestamp when saving entries, so that bit will be
changed also for single user/groups lookups of course.