I'm not sure what benefit this would give - it seems like a step back to when the CLI accessed the API directly, causing all sorts of race conditions, etc. Moving away from that made things a LOT easier to manage (no more serialize crap after every edit...).
This would be for development and testing only, in which case you get local tracebacks and the ability to mock all sorts of things much more easily than at the XMLRPC layer. It would have nothing to do with production usage.
Imagine an XMLRPC library shim being ducked typed, more or less, so the underlying code doesn't care, it just uses what it gets.
If it doesn't work, you won't see it, and if it does, it will not be scary.
cobblerd has to exist because of background jobs and our persistance layer, which I do not wish to rewrite everything to be databasey -- as much as I'd like to, it's way too much work, and really not a field problem. In many cases, it's actually useful, and it's one of a few things that makes working on Cobbler not feel like working on a webapp, and that's not a feeling I'd want to encourage anyway :)
One thing I think we need to change here, is to remove the caching of objects or to at least make it an option that could be switched off. This really interferes with trying to use any other backend (see mongo). In my opinion, the move the FIELDS method for objects was one of the greatest changes we made. It makes extending cobbler so much easier than it was before. You address that in one of your other points, but that change alone eliminated 90% of the boilerplate code that was in there.
(sorry, this is long)
Here's the tricky thing about NoSQL that I don't like. To do it right, you have to embrace it throughout the app and not just at a persistance layer. If you write code as if it were a filesystem, or a database, that's not right... especially for Mongo, where you have to babysit the indexer and design your data "schema" in ways that make it easy to search. In many cases, you want to denormalize things, like have profiles have a list of the systems it has, or other weirdness. Things we don't do today and things that wouldn't make sense in other storage methods.
IMHO, picking mongo is not so good as a NoSQL choice (danger, flamebait!) because it has known reliability and manageability issues -- and while it does have some search capabilities it's indexing and such is quite less capable than a database when you want to use those things. Riak is really strong in terms of adding more nodes and takes a harder line on being reliable first, and doesn't have the sharding management problems. However search there (in Riak) is weaker as it's still evolving. So nothing really has the capabilities I'd want to commit to that as a main datastore, and I feel you really do need to tailor the code to the datastore to get the most out of it. The Mongo guys say in one sentence "it's designed to scale", and then they show you this horrible sharding hack of a mongo of mongos... and it's clear that was never really true... and then the reliability is right out the window in many configurations to, unless you want to learn how to do it right... which I don't want to force everyone to do. I think making those kind of choices can cut into your adoption.
Anyway, with NoSQL you would have the exact same problem you have in cobbler now in which the code needs to know how to migrate stale data -- which is something you'd also like to push into a database if possible. A lot of this code gets pretty rough, in particular, item_system (IIRC) had lots of things to upconvert network information into a list at one point -- this is the kind of thing you *don't* have to do in the main tree with a database, and the code can get more streamlined -- actual DB migrations are simpler. Ultimately while it sounds cool, in the end, I think real databases still have a lot of advantages.... (though we probably don't need one, and our development pace was faster for not having one).
Knowing the sizes of several existing deployments -- and the fact that these deployments are pretty much fine -- I think it's a non-issue. There are decided advantages to the way it works now, namely ease of management, ability to rsync files, ability to version control files, etc.
Another thing to consider is people are moving more and more towards cloud type infrastructures, and the numbers of profiles you are going to have is going to be manageable, your hypervisors are going to be relatively homogenous, and you shouldn't need 45k system records. And if you do... well, we can fight that battle if we want ... but I think it's a lot of regression bait for not a lot of gain. Having 45k system records is a sign you have a real mess. Profiles mapping to something analogous to puppet's external_nodes is good. Having lots of variance is bad. This is why you can use a system record to define a subnet instead -- to set the PXE profile for that whole network. There's probably a whole modeling discussion to be had there.
Another reason I don't think this is a major concern -- many cobbler deployments are very very distributed -- the idea that the profiles get migrated and the systems live in proximity to where the cobbler server is seems to make sense. Now, in that kind of environment, I don't think the webui makes sense at all for managing that -- but that's another story -- I also don't really care much about the webui anymore, but its great for adoption and if people like it, great. I'm not going to try to kill it :)
Bringing this full circle.. The caching thing is dovetailed in. If you remove it, you have some pretty slow command times, which is the reason everything *does* go through cobblerd -- it is the one copy of everything. Now, if you want to remove the caching properly, you pick the database, and you update the code to make the appropriate search calls -- which I don't think is neccessary for reasons above. The idea was if cobblerd was the only copy of the true knowledge, and you only talked to cobblerd, there were no caching problems. Since the XMLRPC move, it wasn't intended that you would access things from pure Python ever, just the master copy.
It's all sort of a self-fulfilling prophesy, strange loop, etc.
Maybe if I set a environment variable like COBBLERTEST=1 and then run cobbler commands or the test suite, it could work that way (locally) and give local exceptions, etc. This would be totally unsuitable for production use, but could be quite useful for us.
One of the things I've had on my list for a long time is that I REALLY hate that cobbler prints stack dumps when it bombs out of code. They are not useful to users and would love to see every stack dump go away unless the log level was set to DEBUG or some other verbose setting.
Stripping the error message out of the stack trace is all that it should print, I'd agree.
I think the original intent was to do this automatically if things were CobblerExceptions, i.e. expected error handling, but any real tracebacks outside of that were code errors in the software that needed to be fixed.
Beyond this, I agree with just about everything else you have on there with one exception - I'm not sure how much time we'd want to devote to getting cobbler running on OSX, for the simple reason that I don't expect anyone to run cobblerd on it. Can you even PXE OSX? Is that a demographic we'd aim for?
It's something I'm going to get running and see how much of the core you can run on it, because that's a piece of hardware I frequently have with me. As a group, we wouldn't really have to do anything.
If it does mean anything, there are various places in cobbler -- all over the place, really, where it makes system calls and tends to be imperative. I'm probably going to want to abstract those out better eventually. This is good to do anyway for testing reasons.
(Do you have pykickstart on a Debian system, etc?...basically something like a resource/provider system seems logical, different repo types are more or less plugins, etc)
I don't really want to PXE OS X itself because I don't have the hardware I can afford to brick. FWIW, the Mac stores do PXE OS X if you need a system reimaged, but that's not what I was talking about here. Cobbler wouldn't even get a --breed=osx option, I'd just be getting the server to work.
Basically if you can get around with cobbler system add, serve kickstart files, and basically just use the core, I'd call that success.
cobbler-devel mailing list cobbler-devel@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/cobbler-devel