Random thoughts on things I'd like to tweak (architecturally, internals, not feature wise)
FWIW, random thoughts on things I'd like to pound on:
** cobblerd **
Making it a little easier to debug/work on.
For testing, I think we should be able to turn cobblerd off and use the XMLRPC library directly without a socket connection, for testing and also local usage (with cobblerd stopped, probably). This could help out things pretty quickly. So the same API syntax we could use in the tests without spawning up the daemon, and that could work exceptionally nicely.
cobblerd has to exist because of background jobs and our persistance layer, which I do not wish to rewrite everything to be databasey -- as much as I'd like to, it's way too much work, and really not a field problem. In many cases, it's actually useful, and it's one of a few things that makes working on Cobbler not feel like working on a webapp, and that's not a feeling I'd want to encourage anyway :)
Maybe if I set a environment variable like COBBLERTEST=1 and then run cobbler commands or the test suite, it could work that way (locally) and give local exceptions, etc. This would be totally unsuitable for production use, but could be quite useful for us.
** more testing coverage **
I've gotten alot better at testing... cobbler has lots of side effects and the need to provision to fully test (in some cases) and deal with a lot of ISOs and a lot of storage means things are difficult. More abstraction layers probably need to exist... I don't know, but I'd like to establish a decent test framework that requires less manual testing. The existing test directory is not enough. I think for most things we can require tests for commits and can mock out parts of the app that are side effecty, and also configure it in such a way where we could have a "install root" where the tests build out their filesystem operations in "/tmp", etc. Just having that concept of a configurable FS root could be huge.
Not sure what to do about auto-testing import, but there's quite a bit that we can test the heck out of (PXE trees, maybe even minimal yum repos, etc).
** cross distro support **
For my own personal reasons, pretty much for mobile development setups, I'd like to see how much of the server plays nice on a mac, and if we can make it to play *nicer*.
** templates **
I kind of hate Cheetah and would like to provide other simpler options. Ideally have something like the bash shebang to specify a template type. This seems actually pretty safe to do, but it would be nice to port all of things away from Cheetah as well. Should not be difficult.
**object system **
Over the years I've had lots of random ideas on cleaning up the item_* system to reduce boilerplate, there's probably more to do here. Not a major priority, requires the test infrastructure first.
** development environment **
"make webtest" is pretty annoying, get cobbler tools to run better out of local checkout with PYTHONPATH. Might be almost free. See also the idea of having a way to configure cobbler's filesystem root. Maybe this only works in local XMLRPC mode as well.
** python versions **
require python2.4 as a minimum in cobbler & koan both, older OS support *gone*, they've had long enough, and it's nice to be able to have decent language capabilities
--Michael
** templates **
I kind of hate Cheetah and would like to provide other simpler options. Ideally have something like the bash shebang to specify a template type. This seems actually pretty safe to do, but it would be nice to port all of things away from Cheetah as well. Should not be difficult.
Already done :) Unfortunately I don't think this was merged into master, so it may take some work to fold it back in. It's in the "new-template" branch on my github.
On Mon, Nov 28, 2011 at 8:07 PM, James Cammarata jimi@sngx.net wrote:
** templates **
I kind of hate Cheetah and would like to provide other simpler options. Ideally have something like the bash shebang to specify a template type. This seems actually pretty safe to do, but it would be nice to port all of things away from Cheetah as well. Should not be difficult.
Already done :) Unfortunately I don't think this was merged into master, so it may take some work to fold it back in. It's in the "new-template" branch on my github.
Ok, maybe not so hard afterall:
# git merge --no-ff new-template -m "Merge in new-template branch, which allows for the use of new template types" Auto-merging config/settings Merge made by recursive. cobbler/templar.py | 110 +++++++++++++++++++++++++++++++++++++++++----------- config/settings | 9 ++++ 2 files changed, 96 insertions(+), 23 deletions(-)
w00t. very nice, push to the main repo?
(FYI -- you may want to refork your github from the master github at some point, it doesn't show up in list of forks, and it's obviously an important one. I retired the People With Git Branches page since it's mostly self managing now)
--Michael
On Mon, Nov 28, 2011 at 9:11 PM, James Cammarata jimi@sngx.net wrote:
On Mon, Nov 28, 2011 at 8:07 PM, James Cammarata jimi@sngx.net wrote:
** templates **
I kind of hate Cheetah and would like to provide other simpler options. Ideally have something like the bash shebang to specify a template type. This seems actually pretty safe to do, but it would be nice to port all of things away from Cheetah as well. Should not be difficult.
Already done :) Unfortunately I don't think this was merged into master, so it may take some work to fold it back in. It's in the "new-template" branch on my github.
Ok, maybe not so hard afterall:
# git merge --no-ff new-template -m "Merge in new-template branch, which allows for the use of new template types" Auto-merging config/settings Merge made by recursive. cobbler/templar.py | 110 +++++++++++++++++++++++++++++++++++++++++----------- config/settings | 9 ++++ 2 files changed, 96 insertions(+), 23 deletions(-) _______________________________________________ cobbler-devel mailing list cobbler-devel@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/cobbler-devel
On Mon, Nov 28, 2011 at 8:40 PM, Michael DeHaan michael.dehaan@gmail.com wrote:
w00t. very nice, push to the main repo?
Done. I'm going to have to get back in the habit of doing a pull before I push... I've been the only one pushing upstream for so long I'm not used to it having diverged :)
(FYI -- you may want to refork your github from the master github at some point, it doesn't show up in list of forks, and it's obviously an important one. I retired the People With Git Branches page since it's mostly self managing now)
Yeah at some point I'll straighten things out, I've just got so many feature branches locally I really don't feel like dealing with having to re-fork and adjust all my remotes again (already went through that once with the move to github).
Is jinja2 supported yet? That is a really popular ten plating language.
Sent from my iPhone
On Nov 28, 2011, at 19:06, James Cammarata jimi@sngx.net wrote:
On Mon, Nov 28, 2011 at 8:40 PM, Michael DeHaan michael.dehaan@gmail.com wrote:
w00t. very nice, push to the main repo?
Done. I'm going to have to get back in the habit of doing a pull before I push... I've been the only one pushing upstream for so long I'm not used to it having diverged :)
(FYI -- you may want to refork your github from the master github at some point, it doesn't show up in list of forks, and it's obviously an important one. I retired the People With Git Branches page since it's mostly self managing now)
Yeah at some point I'll straighten things out, I've just got so many feature branches locally I really don't feel like dealing with having to re-fork and adjust all my remotes again (already went through that once with the move to github). _______________________________________________ cobbler-devel mailing list cobbler-devel@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/cobbler-devel
On Mon, Nov 28, 2011 at 10:13 PM, Jeff Schroeder jeffschroed@gmail.com wrote:
Is jinja2 supported yet? That is a really popular ten plating language.
It is in master as of that push. However, none of the shipped templates are written in it, so you'll have to rewrite anything you want to use. The good thing is you can mix and match, so we don't have to port everything over at once.
On Mon, Nov 28, 2011 at 7:59 PM, Michael DeHaan michael.dehaan@gmail.com wrote:
FWIW, random thoughts on things I'd like to pound on:
** cobblerd **
Making it a little easier to debug/work on.
For testing, I think we should be able to turn cobblerd off and use the XMLRPC library directly without a socket connection, for testing and also local usage (with cobblerd stopped, probably). This could help out things pretty quickly. So the same API syntax we could use in the tests without spawning up the daemon, and that could work exceptionally nicely.
I'm not sure what benefit this would give - it seems like a step back to when the CLI accessed the API directly, causing all sorts of race conditions, etc. Moving away from that made things a LOT easier to manage (no more serialize crap after every edit...).
cobblerd has to exist because of background jobs and our persistance layer, which I do not wish to rewrite everything to be databasey -- as much as I'd like to, it's way too much work, and really not a field problem. In many cases, it's actually useful, and it's one of a few things that makes working on Cobbler not feel like working on a webapp, and that's not a feeling I'd want to encourage anyway :)
One thing I think we need to change here, is to remove the caching of objects or to at least make it an option that could be switched off. This really interferes with trying to use any other backend (see mongo). In my opinion, the move the FIELDS method for objects was one of the greatest changes we made. It makes extending cobbler so much easier than it was before. You address that in one of your other points, but that change alone eliminated 90% of the boilerplate code that was in there.
Maybe if I set a environment variable like COBBLERTEST=1 and then run cobbler commands or the test suite, it could work that way (locally) and give local exceptions, etc. This would be totally unsuitable for production use, but could be quite useful for us.
One of the things I've had on my list for a long time is that I REALLY hate that cobbler prints stack dumps when it bombs out of code. They are not useful to users and would love to see every stack dump go away unless the log level was set to DEBUG or some other verbose setting.
Beyond this, I agree with just about everything else you have on there with one exception - I'm not sure how much time we'd want to devote to getting cobbler running on OSX, for the simple reason that I don't expect anyone to run cobblerd on it. Can you even PXE OSX? Is that a demographic we'd aim for?
I'm not sure what benefit this would give - it seems like a step back to when the CLI accessed the API directly, causing all sorts of race conditions, etc. Moving away from that made things a LOT easier to manage (no more serialize crap after every edit...).
This would be for development and testing only, in which case you get local tracebacks and the ability to mock all sorts of things much more easily than at the XMLRPC layer. It would have nothing to do with production usage.
Imagine an XMLRPC library shim being ducked typed, more or less, so the underlying code doesn't care, it just uses what it gets.
If it doesn't work, you won't see it, and if it does, it will not be scary.
cobblerd has to exist because of background jobs and our persistance layer, which I do not wish to rewrite everything to be databasey -- as much as I'd like to, it's way too much work, and really not a field problem. In many cases, it's actually useful, and it's one of a few things that makes working on Cobbler not feel like working on a webapp, and that's not a feeling I'd want to encourage anyway :)
One thing I think we need to change here, is to remove the caching of objects or to at least make it an option that could be switched off. This really interferes with trying to use any other backend (see mongo). In my opinion, the move the FIELDS method for objects was one of the greatest changes we made. It makes extending cobbler so much easier than it was before. You address that in one of your other points, but that change alone eliminated 90% of the boilerplate code that was in there.
(sorry, this is long)
Here's the tricky thing about NoSQL that I don't like. To do it right, you have to embrace it throughout the app and not just at a persistance layer. If you write code as if it were a filesystem, or a database, that's not right... especially for Mongo, where you have to babysit the indexer and design your data "schema" in ways that make it easy to search. In many cases, you want to denormalize things, like have profiles have a list of the systems it has, or other weirdness. Things we don't do today and things that wouldn't make sense in other storage methods.
IMHO, picking mongo is not so good as a NoSQL choice (danger, flamebait!) because it has known reliability and manageability issues -- and while it does have some search capabilities it's indexing and such is quite less capable than a database when you want to use those things. Riak is really strong in terms of adding more nodes and takes a harder line on being reliable first, and doesn't have the sharding management problems. However search there (in Riak) is weaker as it's still evolving. So nothing really has the capabilities I'd want to commit to that as a main datastore, and I feel you really do need to tailor the code to the datastore to get the most out of it. The Mongo guys say in one sentence "it's designed to scale", and then they show you this horrible sharding hack of a mongo of mongos... and it's clear that was never really true... and then the reliability is right out the window in many configurations to, unless you want to learn how to do it right... which I don't want to force everyone to do. I think making those kind of choices can cut into your adoption.
Anyway, with NoSQL you would have the exact same problem you have in cobbler now in which the code needs to know how to migrate stale data -- which is something you'd also like to push into a database if possible. A lot of this code gets pretty rough, in particular, item_system (IIRC) had lots of things to upconvert network information into a list at one point -- this is the kind of thing you *don't* have to do in the main tree with a database, and the code can get more streamlined -- actual DB migrations are simpler. Ultimately while it sounds cool, in the end, I think real databases still have a lot of advantages.... (though we probably don't need one, and our development pace was faster for not having one).
Knowing the sizes of several existing deployments -- and the fact that these deployments are pretty much fine -- I think it's a non-issue. There are decided advantages to the way it works now, namely ease of management, ability to rsync files, ability to version control files, etc.
Another thing to consider is people are moving more and more towards cloud type infrastructures, and the numbers of profiles you are going to have is going to be manageable, your hypervisors are going to be relatively homogenous, and you shouldn't need 45k system records. And if you do... well, we can fight that battle if we want ... but I think it's a lot of regression bait for not a lot of gain. Having 45k system records is a sign you have a real mess. Profiles mapping to something analogous to puppet's external_nodes is good. Having lots of variance is bad. This is why you can use a system record to define a subnet instead -- to set the PXE profile for that whole network. There's probably a whole modeling discussion to be had there.
Another reason I don't think this is a major concern -- many cobbler deployments are very very distributed -- the idea that the profiles get migrated and the systems live in proximity to where the cobbler server is seems to make sense. Now, in that kind of environment, I don't think the webui makes sense at all for managing that -- but that's another story -- I also don't really care much about the webui anymore, but its great for adoption and if people like it, great. I'm not going to try to kill it :)
Bringing this full circle.. The caching thing is dovetailed in. If you remove it, you have some pretty slow command times, which is the reason everything *does* go through cobblerd -- it is the one copy of everything. Now, if you want to remove the caching properly, you pick the database, and you update the code to make the appropriate search calls -- which I don't think is neccessary for reasons above. The idea was if cobblerd was the only copy of the true knowledge, and you only talked to cobblerd, there were no caching problems. Since the XMLRPC move, it wasn't intended that you would access things from pure Python ever, just the master copy.
It's all sort of a self-fulfilling prophesy, strange loop, etc.
Maybe if I set a environment variable like COBBLERTEST=1 and then run cobbler commands or the test suite, it could work that way (locally) and give local exceptions, etc. This would be totally unsuitable for production use, but could be quite useful for us.
One of the things I've had on my list for a long time is that I REALLY hate that cobbler prints stack dumps when it bombs out of code. They are not useful to users and would love to see every stack dump go away unless the log level was set to DEBUG or some other verbose setting.
Stripping the error message out of the stack trace is all that it should print, I'd agree.
I think the original intent was to do this automatically if things were CobblerExceptions, i.e. expected error handling, but any real tracebacks outside of that were code errors in the software that needed to be fixed.
Beyond this, I agree with just about everything else you have on there with one exception - I'm not sure how much time we'd want to devote to getting cobbler running on OSX, for the simple reason that I don't expect anyone to run cobblerd on it. Can you even PXE OSX? Is that a demographic we'd aim for?
It's something I'm going to get running and see how much of the core you can run on it, because that's a piece of hardware I frequently have with me. As a group, we wouldn't really have to do anything.
If it does mean anything, there are various places in cobbler -- all over the place, really, where it makes system calls and tends to be imperative. I'm probably going to want to abstract those out better eventually. This is good to do anyway for testing reasons.
(Do you have pykickstart on a Debian system, etc?...basically something like a resource/provider system seems logical, different repo types are more or less plugins, etc)
I don't really want to PXE OS X itself because I don't have the hardware I can afford to brick. FWIW, the Mac stores do PXE OS X if you need a system reimaged, but that's not what I was talking about here. Cobbler wouldn't even get a --breed=osx option, I'd just be getting the server to work.
Basically if you can get around with cobbler system add, serve kickstart files, and basically just use the core, I'd call that success.
cobbler-devel mailing list cobbler-devel@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/cobbler-devel
cobbler-devel@lists.fedorahosted.org
-
James Cammarata -
Jeff Schroeder -
Michael DeHaan