[devel] cobblerd performance (this works!)
New simpler plan, for which I have a partial implementation already.
Have cobblerd keep a cache of all objects that it loads on startup.
When an edit is made in the API, the API will signal cobblerd to update
it's cache for just that object. The result is that the API only has to
reload that /one/ config file, not all of them.
The API will also signal deletes.
Old behavior with 10k systems:
-- slow cobblerd startup (60 seconds)
-- slow CLI startup (60 seconds)
-- fast inserts
-- very slow webapp and XMLRPC
New behavior
-- slow cobblerd startup
-- slow CLI startup
-- slightly slower inserts due to XMLRPC signaling, most users should
not notice (maybe 2x slower, they were fast enough before)
-- very fast webapp and XMLRPC as if it didn't matter how many systems
were in it (win!)
The result is that a full reload of all data in cobblerd is therefore
/never/ done and everything is zippy, if a bit of a minor RAM hog. It
also makes 'inserts' marginally slower, but I'm mainly concerned with
API load time.
I also note that there is no reason to have a seperate fork for cobblerd
(read-only) and cobblerd (read-write), as you should be able to use the
public methods against the RW XMLRPC instance just fine. This saves us
a process and will be step two. I suspect this might even cut our
startup time down some as they won't be trying to access the filesystem
at the same time (even though they are loading in parallel).
Step three is that with a very large number of systems, opening up a API
handle for the first time is still going to be expensive. The answer
here is probably do a cobbler shell that pays for the API penalty once
and then lets you send commands rapidly as you see fit. I think this
would be easy to do.
All of this can be done without changes to the serializers, and doesn't
require anything as intrusive as inserting a database. Some of this is
working now on my machine and work will be checked in to the devel
branch as it goes along.
I can also work on deferring the cobblerd startup penalty to after the
service is started so it can be backgrounded.
Bottom line: we have a /simple/ solution that fits requirements, and
I'm happy.
This is also tunable so we don't have to stop here.
Largely I expect folks with 10k+ systems to being using cobbler as a
service, not a command line app, so I intend to approach scaling
problems in the context of making sure cobblerd (as an example client)
and koan remain performant. The CLI is a lesser priority.
These are good problems to have as they mean more people are interested
in using cobbler in larger configs -- I would rather have this
challenges than not have them, and this is looking like we are moving in
a very good direction.
--Michael