-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Hello all,
It's been some number of months (six+) since we decided to change the way mock works. We (Michael Brown and I, with input from skvidal and others) came up with a change in how mock launches, manages permissions and runs privileged commands.
Old mock (mock-0.6 and previous) is a python script that does some sanity checking (insures that the person running mock is not root and is in the mock group) and then processes the input commands. Whenever it needed to do something that required root privileges, old mock ran a setuid root program named mock-helper. Mock-helper is a C program that knows how to do a limited number of things (mount/unmount, run chroot, etc.).
The new mock (mock-0.7 and beyond) turns things around a bit. In new mock, the program /usr/bin/mock is a setuid:root, setgid:mock C program that does one thing only: launches the command "python /usr/libexec/mock.py <args>" in it's own kernel namespace. The mock.py logic still sanity checks that the user is in the mock group and drops privileges to the actual uid while keeping the gid of the process the mock group. As long as we're careful to maintain proper group ownership and permissions of created file and directories, this should go a long way toward fixing the issues we're having with multiple users on a single machine.
New mock will no longer use mock-helper. When it needs to do something that requires root privileges, it will elevate it's privilege level to root (using os.setreuid()), execute the command and then drop privileges back to the normal user.
All of this is working, although it has not been extensively testing (hello rawhide!). I've merged the BZ bugfixes from the mock-0.6 branch of CVS into the head (which is the new mock) and would like to push the new mock out to rawhide for testing.
What I'm looking for from the readership of this list is:
1. Review of strategy and code for security issues 2. Help in formulating an SELinux plan/policy for mock
We had some discussion on this issue back in June 2006, but I'd like to look at it one more time before inflicting the new mock on the rawhide faithful.
With regard to SELinux, I'm not sure where we need to go with mock. I want mock to function properly and securely on a system running SELinux, but I'm just not sure how to go about that. I've looked at the steps mentioned on:
http://fedoraproject.org/wiki/Extras/MockTricks
But I'm too SELinux ignorant to be able to make an informed judgment on whether that's the right thing to do. Help on this would be greatly appreciated.
Thanks, Clark
On Thu, 2007-01-04 at 10:37 -0600, Clark Williams wrote:
What I'm looking for from the readership of this list is:
- Review of strategy and code for security issues
- Help in formulating an SELinux plan/policy for mock
We had some discussion on this issue back in June 2006, but I'd like to look at it one more time before inflicting the new mock on the rawhide faithful.
With regard to SELinux, I'm not sure where we need to go with mock. I want mock to function properly and securely on a system running SELinux, but I'm just not sure how to go about that. I've looked at the steps mentioned on:
http://fedoraproject.org/wiki/Extras/MockTricks
But I'm too SELinux ignorant to be able to make an informed judgment on whether that's the right thing to do. Help on this would be greatly appreciated.
The SELinux policy on the wiki does two main things:
1. Label everything under /var/lib/mock with a special SELinux context type that all processes running under mock can do as they like with.
2. Run the mock process with the ability to execute code from writable memory. This is necessary in order to support technologies such as java and mono, which are sometimes used during package builds.
So basically it's the minimum necessary to get mock usable under the targeted policy. Neither of these things should be affected by the internal restructuring of mock itself.
If the new mock will run OK on FC6, I can test it quite easily as both of my buildsystem machines are running FC6 with SELinux on.
Paul.
On Thu, Jan 04, 2007 at 10:37:03AM -0600, Clark Williams wrote:
New mock will no longer use mock-helper. When it needs to do something that requires root privileges, it will elevate it's privilege level to root (using os.setreuid()), execute the command and then drop privileges back to the normal user.
But isn't this a security regression towards the previous model? Previously all elevation procedures were confined and well controlled.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Axel Thimm wrote:
On Thu, Jan 04, 2007 at 10:37:03AM -0600, Clark Williams wrote:
New mock will no longer use mock-helper. When it needs to do something that requires root privileges, it will elevate it's privilege level to root (using os.setreuid()), execute the command and then drop privileges back to the normal user.
But isn't this a security regression towards the previous model? Previously all elevation procedures were confined and well controlled.
One of the first thing that the __init__() method for class Root does in mock.py is to call self.drop() to lower the privilege level. Thereafter, any command that new mock does as root is done via the do_elevated() method of the Root class, and any time the actual python code needs root access (e.g. the rpm library routines), it's bracketed by elevate() and drop() calls. This makes it easy to audit how the commands are used and in what context code is executed.
The main reason we wanted to get rid of mock-helper is that it was non-trivial C code and the thought was to limit the amount of work that's done at the C level. Yeah, I realize that it's easy to write bad code in Python too, but it's harder to inadvertently set up a buffer overflow situation in Python than in C.
Clark
On Thu, Jan 04, 2007 at 01:13:25PM -0600, Clark Williams wrote:
Axel Thimm wrote:
On Thu, Jan 04, 2007 at 10:37:03AM -0600, Clark Williams wrote:
New mock will no longer use mock-helper. When it needs to do something that requires root privileges, it will elevate it's privilege level to root (using os.setreuid()), execute the command and then drop privileges back to the normal user.
But isn't this a security regression towards the previous model? Previously all elevation procedures were confined and well controlled.
One of the first thing that the __init__() method for class Root does in mock.py is to call self.drop() to lower the privilege level. Thereafter, any command that new mock does as root is done via the do_elevated() method of the Root class, and any time the actual python code needs root access (e.g. the rpm library routines), it's bracketed by elevate() and drop() calls. This makes it easy to audit how the commands are used and in what context code is executed.
I understand the mechanism, but what if a security issue elsewhere in mock allows one to inject code and elevate privildeges? Until now any rogue mock takeover would only be able to do what the confined C helper program would allow, now everything is possible.
The main reason we wanted to get rid of mock-helper is that it was non-trivial C code and the thought was to limit the amount of work that's done at the C level. Yeah, I realize that it's easy to write bad code in Python too, but it's harder to inadvertently set up a buffer overflow situation in Python than in C.
But now you have several times more code that can lead to priviledge escalations compared to before - in fact it sounds like all of the python code including all used non-mock python modules could run setreuid or not?
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Axel Thimm wrote:
On Thu, Jan 04, 2007 at 01:13:25PM -0600, Clark Williams wrote:
One of the first thing that the __init__() method for class Root does in mock.py is to call self.drop() to lower the privilege level. Thereafter, any command that new mock does as root is done via the do_elevated() method of the Root class, and any time the actual python code needs root access (e.g. the rpm library routines), it's bracketed by elevate() and drop() calls. This makes it easy to audit how the commands are used and in what context code is executed.
I understand the mechanism, but what if a security issue elsewhere in mock allows one to inject code and elevate privildeges? Until now any rogue mock takeover would only be able to do what the confined C helper program would allow, now everything is possible.
I'm trying to figure out how someone would inject code into mock and not already be root (which to me means Game Over). To successfully exploit something, an attacker must cause some code to be loaded that either:
1. runs between an elevate and a drop or 2. does it's own os.setreuid(0, ...)
To do #2, they would have to gain control of one of the 15 the files in /usr/lib/python2.X/* that we import. I'm not sure how someone would be able to gain control of a running instance of mock and be able to insert code arbitrarily.
Please don't misunderstand me, I'm not dismissing your concerns, but I don't really see how this is any less secure than what we had previously. To me it's just a matter of where we focus our efforts in making mock secure.
The main reason we wanted to get rid of mock-helper is that it was non-trivial C code and the thought was to limit the amount of work that's done at the C level. Yeah, I realize that it's easy to write bad code in Python too, but it's harder to inadvertently set up a buffer overflow situation in Python than in C.
But now you have several times more code that can lead to priviledge escalations compared to before - in fact it sounds like all of the python code including all used non-mock python modules could run setreuid or not?
It's the same amount of privilege escalations as before, just under the control of the python code.
You might want to take a look at this list thread:
http://www.redhat.com/archives/fedora-buildsys-list/2006-June/msg00018.html
which is what prompted this change.
Clark
On Thu, Jan 04, 2007 at 03:11:23PM -0600, Clark Williams wrote:
Axel Thimm wrote:
On Thu, Jan 04, 2007 at 01:13:25PM -0600, Clark Williams wrote:
One of the first thing that the __init__() method for class Root does in mock.py is to call self.drop() to lower the privilege level. Thereafter, any command that new mock does as root is done via the do_elevated() method of the Root class, and any time the actual python code needs root access (e.g. the rpm library routines), it's bracketed by elevate() and drop() calls. This makes it easy to audit how the commands are used and in what context code is executed.
I understand the mechanism, but what if a security issue elsewhere in mock allows one to inject code and elevate privildeges? Until now any rogue mock takeover would only be able to do what the confined C helper program would allow, now everything is possible.
I'm trying to figure out how someone would inject code into mock and not already be root (which to me means Game Over). To successfully exploit something, an attacker must cause some code to be loaded that either:
- runs between an elevate and a drop or
- does it's own os.setreuid(0, ...)
To do #2, they would have to gain control of one of the 15 the files in /usr/lib/python2.X/* that we import. I'm not sure how someone would be able to gain control of a running instance of mock and be able to insert code arbitrarily.
Well, security is about not creating chances and about having several lines of defenses. We shouldn't argue that any breach is a total breach, so why bother. In this argumentation line, why bother dropping root privileges at all? (That's rhetorical - of course - just trying to make my point).
The point is that you open the vulnerabilty to mock python code, any other python code called by mock, as well as python itself. I think it was easier to deal with the few lines of C code.
The main reason we wanted to get rid of mock-helper is that it was non-trivial C code and the thought was to limit the amount of work that's done at the C level. Yeah, I realize that it's easy to write bad code in Python too, but it's harder to inadvertently set up a buffer overflow situation in Python than in C.
But now you have several times more code that can lead to priviledge escalations compared to before - in fact it sounds like all of the python code including all used non-mock python modules could run setreuid or not?
It's the same amount of privilege escalations as before, just under the control of the python code.
The helper C setuid wrapper would allow only certain operations, i.e. any priviledge escalation is confined to the defined operations in this helper.
This also reduces the amount of code to be reviewed wrt priviledge escalation to just this wrapper. Now you have all of the python code exposed to possible priviledge escalation issues.
In a nutshell: you now carry much more unlimited root power throughout all of mock's invocation cycle in comparison to a confined set of priviledges that the helper was giving.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Axel Thimm wrote:
In a nutshell: you now carry much more unlimited root power throughout all of mock's invocation cycle in comparison to a confined set of priviledges that the helper was giving.
Good point. I still think it's easier to audit python code than C code, but you're talking 500 lines of C versus 1000 lines of python. So, I may just reconsider this change.
One of the reasons I liked moving to a setuid/setgid launcher was that we could move the process into the mock group and fix a bunch of chroot sharing problems with appropriate group permissions. Oh, and we actually kick off the python process in a separate namespace, which means we won't dirty up the mount table if for some reason we exit unexpectedly.
If we just made the launcher setgid:mock and kept mock-helper for rootiness things, would that still trigger your security alarms? Hmmm, now that I think about it, we probably have to be root to create a new namespace, so the launcher might have to stay setuid:root and drop privileges before exec'ing python.
Thoughts?
Clark
On Fri, Jan 05, 2007 at 10:52:04AM -0600, Clark Williams wrote:
Axel Thimm wrote:
In a nutshell: you now carry much more unlimited root power throughout all of mock's invocation cycle in comparison to a confined set of priviledges that the helper was giving.
Good point. I still think it's easier to audit python code than C code, but you're talking 500 lines of C versus 1000 lines of python. So, I may just reconsider this change.
One of the reasons I liked moving to a setuid/setgid launcher was that we could move the process into the mock group and fix a bunch of chroot sharing problems with appropriate group permissions. Oh, and we actually kick off the python process in a separate namespace, which means we won't dirty up the mount table if for some reason we exit unexpectedly.
If we just made the launcher setgid:mock and kept mock-helper for rootiness things, would that still trigger your security alarms? Hmmm, now that I think about it, we probably have to be root to create a new namespace, so the launcher might have to stay setuid:root and drop privileges before exec'ing python.
Thoughts?
How about a very radical approach: Removing the neccessity to have root priviledges at the first place anywhere. The benefits are clear security-wise, and you get the added bonus that you can have people install mock under their $HOME w/o being root on these system (imagine students working on campus/lab PCs). One could even create a Fedora SDK that runs on non-rpm Linux platforms - mock infiltrates everything ;)
The question is whether that is technically possible - for what I use at ATrpms, an ancient bunch of shell scripts being the equivalent of mock, I use fakechroot/fakeroot to maintain chroots as a simple user. I think that will work with eliminating the need for the chroot part in the mock helper as well. If we check the remaining parts in mock requiring root priviledges (perhaps just for mounting?) perhaps we can eliminate these, too, and end up with a pure non-root working mock.
I submitted fakeroot/fakechroot shortly before 2006 phased out, once they are through one can start playing with them (I think fakechroot is there, still waiting for fakeroot+dependency reviews to complete).
BTW the Debian/Ubuntu build systems make extensive use of fakeroot/fakechroot for exactly the same scenarios.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Axel Thimm wrote:
On Fri, Jan 05, 2007 at 10:52:04AM -0600, Clark Williams wrote:
Axel Thimm wrote:
In a nutshell: you now carry much more unlimited root power throughout all of mock's invocation cycle in comparison to a confined set of priviledges that the helper was giving.
Good point. I still think it's easier to audit python code than C code, but you're talking 500 lines of C versus 1000 lines of python. So, I may just reconsider this change.
One of the reasons I liked moving to a setuid/setgid launcher was that we could move the process into the mock group and fix a bunch of chroot sharing problems with appropriate group permissions. Oh, and we actually kick off the python process in a separate namespace, which means we won't dirty up the mount table if for some reason we exit unexpectedly.
If we just made the launcher setgid:mock and kept mock-helper for rootiness things, would that still trigger your security alarms? Hmmm, now that I think about it, we probably have to be root to create a new namespace, so the launcher might have to stay setuid:root and drop privileges before exec'ing python.
Thoughts?
How about a very radical approach: Removing the neccessity to have root priviledges at the first place anywhere. The benefits are clear security-wise, and you get the added bonus that you can have people install mock under their $HOME w/o being root on these system (imagine students working on campus/lab PCs). One could even create a Fedora SDK that runs on non-rpm Linux platforms - mock infiltrates everything ;)
The question is whether that is technically possible - for what I use at ATrpms, an ancient bunch of shell scripts being the equivalent of mock, I use fakechroot/fakeroot to maintain chroots as a simple user. I think that will work with eliminating the need for the chroot part in the mock helper as well. If we check the remaining parts in mock requiring root priviledges (perhaps just for mounting?) perhaps we can eliminate these, too, and end up with a pure non-root working mock.
I submitted fakeroot/fakechroot shortly before 2006 phased out, once they are through one can start playing with them (I think fakechroot is there, still waiting for fakeroot+dependency reviews to complete).
BTW the Debian/Ubuntu build systems make extensive use of fakeroot/fakechroot for exactly the same scenarios.
I've used fakeroot/fakechroot in the past, but ran into RPM problems (where RPM wanted to take a lock that required root). I'm not positive, but I think that particular RPM problem has been fixed.
I'm not adverse to trying it in the long run, but I don't really want to try and put it in for the next release. I want to get three things working for the next release:
1. Simplified uid/gid scheme so that multiple users can work properly 2. Running in our own namespace. 3. Proper SELinux integration
What I'd like for #1 is for the process to setgid:mock once the user has been verified. At that point every file created has gid mock and so there shouldn't be any ownership conflicts in chroots. As far as I can tell the only way to setgid group is to run from a setgid program, so that means the launcher.
I've seen two ways we can do namespaces. The first is a couple of patches that modify mock-helper. The other is to do it in the launcher. I'd prefer the launcher since it's less code than the patches to mock-helper.
If we keep the launcher *and* mock-helper, and have the launcher do a setreuid(getuid(), getuid()) before exec'ing python, I /think/ that means we lose the ability to switch back to root (the euid). I'll need to go dig into my copy of Advanced Programming in the Unix Environment and probably write a test case to be sure.
So again I ask: if we modify the launcher to setgid:mock and to drop root privilege after it's created the new namespace and before it exec's python, keeping mock-helper for root access, will that satisfy your security concerns for now? If so, we can get the three above things in, push it to rawhide and then discuss moving to fakeroot/fakechroot.
Clark
On Fri, Jan 05, 2007 at 01:04:20PM -0600, Clark Williams wrote:
Axel Thimm wrote:
On Fri, Jan 05, 2007 at 10:52:04AM -0600, Clark Williams wrote:
Axel Thimm wrote:
In a nutshell: you now carry much more unlimited root power throughout all of mock's invocation cycle in comparison to a confined set of priviledges that the helper was giving.
Good point. I still think it's easier to audit python code than C code, but you're talking 500 lines of C versus 1000 lines of python. So, I may just reconsider this change.
How about a very radical approach: Removing the neccessity to have root priviledges at the first place anywhere. [...] The question is whether that is technically possible [...] I use fakechroot/fakeroot to maintain chroots as a simple user. I think that will work with eliminating the need for the chroot part in the mock helper as well. If we check the remaining parts in mock requiring root priviledges (perhaps just for mounting?) perhaps we can eliminate these, too, and end up with a pure non-root working mock.
I'm not adverse to trying it in the long run, but I don't really want to try and put it in for the next release. I want to get three things working for the next release:
Indeed, I wasn't suggesting this as a short-term maneuvering.
So again I ask: if we modify the launcher to setgid:mock and to drop root privilege after it's created the new namespace and before it exec's python, keeping mock-helper for root access, will that satisfy your security concerns for now?
Yes, in that scenario it seems like the max possible damage is done by mock-group priviledges (if the helper itself doesn't come up with any issues), and the mock group cannot damage the host other than with resource draining, e.g. no further priviledge escalation.
If so, we can get the three above things in, push it to rawhide and then discuss moving to fakeroot/fakechroot.
On Friday 05 January 2007 12:46, Axel Thimm wrote:
The question is whether that is technically possible - for what I use at ATrpms, an ancient bunch of shell scripts being the equivalent of mock, I use fakechroot/fakeroot to maintain chroots as a simple user. I think that will work with eliminating the need for the chroot part in the mock helper as well. If we check the remaining parts in mock requiring root priviledges (perhaps just for mounting?) perhaps we can eliminate these, too, and end up with a pure non-root working mock.
Would one still be able to do things IN the chroot like create loopback images and write to them? I need mock to be able to compose trees, which requires these types of things.
On Fri, Jan 05, 2007 at 02:42:51PM -0500, Jesse Keating wrote:
On Friday 05 January 2007 12:46, Axel Thimm wrote:
The question is whether that is technically possible - for what I use at ATrpms, an ancient bunch of shell scripts being the equivalent of mock, I use fakechroot/fakeroot to maintain chroots as a simple user. I think that will work with eliminating the need for the chroot part in the mock helper as well. If we check the remaining parts in mock requiring root priviledges (perhaps just for mounting?) perhaps we can eliminate these, too, and end up with a pure non-root working mock.
Would one still be able to do things IN the chroot like create loopback images and write to them? I need mock to be able to compose trees, which requires these types of things.
I don't know, I haven't had the need to do so in chroots, but I would think that not, since fakeroot/fakechroot only emulates some library calls accociated with file access/manipulation and chrooting, and mounting requires kernel cooperation (even for fuse).
So, if the kernel does not allow user mounts, and that's the only way to write/manipulate images then most likely no.
Axel Thimm wrote:
But isn't this a security regression towards the previous model? Previously all elevation procedures were confined and well controlled.
I'm sorry for replying so late on this thread. I'd like to respond to some of the criticisms of the new model and explain some of the reasons I had for suggesting it.
The mock-helper model is flawed because mock-helper does not have enough context to make informed security decisions. Any user in the mock group can run mock-helper directly and get it to perform any action that it would allow mock to do. While these actions are limited, they are not so limited that you can't do anything dangerous. In order to give mock-helper enough context to really know what it should allow, you would have to move large portions of the logic from mock.py to mock-helper, which would negate all the arguments about the size and complexity of the privileged code (note that mock-helper already accounts for about 30% of the lines of code).
The current mock-helper security is swiss cheese anyway. A user in the mock group can become root inside the chroot with a simple mock or mock-helper command. As root, it is possible to escape from the chroot. Game Over.
Suppose that we were to modify mock.py so that the shell and chroot commands run as a non-root user. Unfortunately mock-helper would still have to allow chroot as root, since mock.py needs this functionality itself. Users would still be able to run mock-helper directly. Even if we were able to plug that hole, the user would still be able to use mock-helper to install a suid binary into a chroot and then simply execute it.
With the new model, we can at least address aspects of this because users in the mock group cannot access arbitrary mock-helper features. They can only take the actions that mock offers through the command line. Mock has all the context it needs to make such decisions. It knows what it should be doing. If we don't allow users to point to arbitrary mock configs, they will not be able to install arbitrary suid code in the chroots. If we don't allow users to chroot as root, they won't be able to get around it by calling mock-helper.
I understand the mechanism, but what if a security issue elsewhere in mock allows one to inject code and elevate privildeges? Until now any rogue mock takeover would only be able to do what the confined C helper program would allow, now everything is possible.
I'm not sure what type of exploit you're worried about here. As a python app, mock should be very resistant to buffer overflow exploits. Furthermore I'm not sure what interface the exploit would come through .. the command line?
Clark's plan for dropping privilege and only elevating for key operations is a good one and will provide some protection against coding errors doing too much damage.
I don't mean to say that the new model will be perfect. I think we will always need to caution admins to only add trusted users to the mock group. However, I think mock will be at least as secure as before and much more maintainable.
On Thu, Jan 11, 2007 at 06:15:17PM -0500, Mike McLean wrote:
I understand the mechanism, but what if a security issue elsewhere in mock allows one to inject code and elevate privildeges? Until now any rogue mock takeover would only be able to do what the confined C helper program would allow, now everything is possible.
I'm not sure what type of exploit you're worried about here. As a python app, mock should be very resistant to buffer overflow exploits.
Check out for example CVE-2006-1542 and CVE-2006-4980.
Furthermore I'm not sure what interface the exploit would come through .. the command line?
Anything that mock takes as an input from command line to submitted srpms/spec files. One of the cve's was triggered by specially crafted UTF-32, next exploit could be with UTF-8 found in specfiles. If you run with possible root priviledge elevation capabilities all the time anything mock calls directly or indirectly becomes vulnerable, be it cpython itself or a python module used by mock.
Axel Thimm wrote:
Check out for example CVE-2006-1542 and CVE-2006-4980.
The first points out that CWD is a possible malicious input. If we treat that data as untrusted, then there are ways we could address such a python bug. For the second, there are patches for python that address it, and it appears the major vendors took notice..
But of course, your point was not so much these particular issues as it was to demonstrate that python has suffered buffer overflows. The fact is that mock's basic model requires that python code be executed as root. Even if mock itself were written entirely in C (not that I'm suggesting this), we must run /yum/ as root. To get around that, you'd have to either not use yum or use a substantially different yum, both of which are really far from where we are now. To me, such an approach would be a new project altogether.
Exploits in the python interpreter are going to affect much more than mock. There are plenty of people out there using python in ways (as root or not) where exploits matter. As such, I think it is reasonable to expect that significant python exploits will be dealt with in a reasonable time frame by vendors. Also, a CVE search shows very few exploits that come from the python interpreter itself. In fact, you pretty much listed them all. Most of the rest come from the code written in python. This is something we can control at least.
Anything that mock takes as an input from command line to submitted srpms/spec files. One of the cve's was triggered by specially crafted UTF-32, next exploit could be with UTF-8 found in specfiles. If you run with possible root priviledge elevation capabilities all the time anything mock calls directly or indirectly becomes vulnerable, be it cpython itself or a python module used by mock.
Such a vulnerability could just as well affect yum, which mock has to run as root. Given to relative code sizes and install bases, I would expect it is much more likely that yum would be the target of this sort of exploit, and there's not much we can do about that (without a massive change in direction).
So much of what mock does needs to be done as root that the mock-helper model creates a significant hurdle in clean design. The exec interface is slow and not very expressive. Also note the flaws I pointed out in my previous post.
If the group is really worried about these sorts of exploits, then I think we can work on accomplishing stronger privilege separation within the python code (I have some ideas on this that I can talk about later if folks are interested). I do not think that mock-helper is the right answer. Getting rid of it is a good first step.
buildsys@lists.fedoraproject.org