On Tue, Mar 24, 2009 at 4:11 PM, Michael DeHaan
<mdehaan(a)redhat.com> wrote:
> Michael DeHaan wrote:
>
>> Ok, recently Scott Henson, John Eckersberg, and I were trying to speed
>> up cobbler ... a lot. We got a lot of good things done, including
>> cutting the sync time in half. One thing we also achieved is we found
>> out, that using the python profiler, even though the new yaml
>> implementation is more robust (good, that's the most important thing!),
>> it's also slow... even with libyaml installed. That all being said,
>> the old implementation was also slow. We want /fast/.
>>
>> Some quick benchmarks on my end show that saving objects using
>> simplejson instead of PyYAML result in 10-12 second times to edit 200
>> objects versus 28 seconds.
>>
>> As a result of this, I'm going to go ahead and add
"serializer_catalog2"
>> to the distribution, though I should warn you ... there's no upgrade path.
>>
>> The way to replicate from catalog to catalog2 is to edit modules.conf on
>> a /new/ cobbler machine and cobbler replicate the data to it (since the
>> wire protocol is exactl y the same).
>>
>> Simplejson offers two output modes -- one is a "pretty printed" mode,
>> this mode is maybe 30% slower. It depends how cleanly readable we need
>> the output to be, but the answer is "not really", since we can always
>> get a pretty printed version of a data structure someone pastebins using
>> ipython if absolutely needed. I am in favor of not using the pretty
>> printed version.
>>
>> Just for kicks, I think I'll benchmark cPickle again -- shelve (which
>> uses things like berkley DB) tends to have issues.
>>
>> We may make the JSON implementation the new default in 1.8, we'll see.
>>
>> --Michael
>> _______________________________________________
>> cobbler mailing list
>> cobbler(a)lists.fedorahosted.org
>>
https://fedorahosted.org/mailman/listinfo/cobbler
>>
>>
> Further data... it appears, and I don't really have the data to back
> this up just yet, that with, say 3000 systems in cobbler's
> configuration, "cobbler list" and "cobbler report" on these
systems are
> basically instantaneous.
>
> sync took 30 seconds with 3000 systems (using hardlinks), where before
> it could take 15 seconds with 300. Roughly, this seems like it's
> infinitely faster for loading. Almost seems too good to be true.
>
> Writing is maybe 2x fast.
>
> It would be trivial to write a seperate (not shipped) python script for
> migrating folks, if this proves as fast as I think it is in further tests.
>
> The last time we tried this it was fed very large files (catalog style)
> -- now with the smaller ones it seems very very happy.
>
And this is why folks like us using cobbler in moderate to large
environments love how fast you release. Great work Michael and crew.
FYI -- I've decided to just teach catalog to update folks automatically,
saving new files with an extension showing the type.
I can do this seamlessly, so why not :)
--Michael