On Tue, Mar 24, 2009 at 4:11 PM, Michael DeHaan mdehaan@redhat.com wrote:
Michael DeHaan wrote:
Ok, recently Scott Henson, John Eckersberg, and I were trying to speed up cobbler ... a lot. We got a lot of good things done, including cutting the sync time in half. One thing we also achieved is we found out, that using the python profiler, even though the new yaml implementation is more robust (good, that's the most important thing!), it's also slow... even with libyaml installed. That all being said, the old implementation was also slow. We want /fast/.
Some quick benchmarks on my end show that saving objects using simplejson instead of PyYAML result in 10-12 second times to edit 200 objects versus 28 seconds.
As a result of this, I'm going to go ahead and add "serializer_catalog2" to the distribution, though I should warn you ... there's no upgrade path.
The way to replicate from catalog to catalog2 is to edit modules.conf on a /new/ cobbler machine and cobbler replicate the data to it (since the wire protocol is exactl y the same).
Simplejson offers two output modes -- one is a "pretty printed" mode, this mode is maybe 30% slower. It depends how cleanly readable we need the output to be, but the answer is "not really", since we can always get a pretty printed version of a data structure someone pastebins using ipython if absolutely needed. I am in favor of not using the pretty printed version.
Just for kicks, I think I'll benchmark cPickle again -- shelve (which uses things like berkley DB) tends to have issues.
We may make the JSON implementation the new default in 1.8, we'll see.
--Michael _______________________________________________ cobbler mailing list cobbler@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/cobbler
Further data... it appears, and I don't really have the data to back this up just yet, that with, say 3000 systems in cobbler's configuration, "cobbler list" and "cobbler report" on these systems are basically instantaneous.
sync took 30 seconds with 3000 systems (using hardlinks), where before it could take 15 seconds with 300. Roughly, this seems like it's infinitely faster for loading. Almost seems too good to be true.
Writing is maybe 2x fast.
It would be trivial to write a seperate (not shipped) python script for migrating folks, if this proves as fast as I think it is in further tests.
The last time we tried this it was fed very large files (catalog style) -- now with the smaller ones it seems very very happy.
And this is why folks like us using cobbler in moderate to large environments love how fast you release. Great work Michael and crew.