4:15 p.m.
Michael DeHaan wrote:
Michael DeHaan wrote:
> New simpler plan, for which I have a partial implementation already.
>
> Have cobblerd keep a cache of all objects that it loads on startup.
>
> When an edit is made in the API, the API will signal cobblerd to update
> it's cache for just that object. The result is that the API only has to
> reload that /one/ config file, not all of them.
> The API will also signal deletes.
>
> Old behavior with 10k systems:
>
> -- slow cobblerd startup (60 seconds)
> -- slow CLI startup (60 seconds)
> -- fast inserts
> -- very slow webapp and XMLRPC
>
> New behavior
>
> -- slow cobblerd startup
> -- slow CLI startup
> -- slightly slower inserts due to XMLRPC signaling, most users should
> not notice (maybe 2x slower, they were fast enough before)
> -- very fast webapp and XMLRPC as if it didn't matter how many systems
> were in it (win!)
>
> The result is that a full reload of all data in cobblerd is therefore
> /never/ done and everything is zippy, if a bit of a minor RAM hog. It
> also makes 'inserts' marginally slower, but I'm mainly concerned with
> API load time.
>
> I also note that there is no reason to have a seperate fork for cobblerd
> (read-only) and cobblerd (read-write), as you should be able to use the
> public methods against the RW XMLRPC instance just fine. This saves us
> a process and will be step two. I suspect this might even cut our
> startup time down some as they won't be trying to access the filesystem
> at the same time (even though they are loading in parallel).
>
> Step three is that with a very large number of systems, opening up a API
> handle for the first time is still going to be expensive. The answer
> here is probably do a cobbler shell that pays for the API penalty once
> and then lets you send commands rapidly as you see fit. I think this
> would be easy to do.
>
> All of this can be done without changes to the serializers, and doesn't
> require anything as intrusive as inserting a database. Some of this is
> working now on my machine and work will be checked in to the devel
> branch as it goes along.
>
> I can also work on deferring the cobblerd startup penalty to after the
> service is started so it can be backgrounded.
>
> Bottom line: we have a /simple/ solution that fits requirements, and
> I'm happy.
>
> This is also tunable so we don't have to stop here.
>
> Largely I expect folks with 10k+ systems to being using cobbler as a
> service, not a command line app, so I intend to approach scaling
> problems in the context of making sure cobblerd (as an example client)
> and koan remain performant. The CLI is a lesser priority.
>
> These are good problems to have as they mean more people are interested
> in using cobbler in larger configs -- I would rather have this
> challenges than not have them, and this is looking like we are moving in
> a very good direction.
>
> --Michael
> _______________________________________________
> cobbler mailing list
> cobbler(a)lists.fedorahosted.org
> https://fedorahosted.org/mailman/listinfo/cobbler
>
>
We've talked about this before, but another way CLI speedup can be
gained with a very large number of systems is by having the CLI use
XMLRPC for most operations unless an API handle is needed.
Add/edit/list type operations really wouldn't need to open a new API
handle provided there was a way to identify a valid local connection
(and there is...)
--Michael
_______________________________________________
cobbler mailing list
cobbler(a)lists.fedorahosted.org
https://fedorahosted.org/mailman/listinfo/cobbler
Also scratch the 'memory hog' comment. We can update the cobbler remote
API objects without keeping extra data around and without reloading the
entire config.
So, no slightly slower inserts and no further penalties.
As it was pointed out on IRC, the cobblerd<-> CLI communication will be
faster over a Unix domain socket, so we'll explore that shortly as well
as removing the extra process from cobblerd, which /will/ make IO much
faster.
It's looking good, folks.
--Michael