Hello,
I work on RHEL security problems. I have been looking into a number of exploits and I think we have a problem that has an easy fix. We are not using the CONFIG_STATIC_USERMODEHELPER_PATH kernel config option. There are a number of exploits that overwrite the path to modprobe and then pass something weird that causes modprobe to be invoked. But instead of modprobe, it's their reverse shell.
If we make the assigment CONFIG_STATIC_USERMODEHELPER_PATH="/usr/" and we change /proc/sys/kernel/modprobe to sbin/modprobe and /proc/sys/kernel/ core_pattern to lib/systemd/systemd-coredump %P %u %g %s %t %c %h, then it limits any exploits to programs that are in /usr. Only root can write here, therefore no escalation. Typically, an exploit changes modprobe path to /tmp/ foo which is shorter than /usr/sbin/modprobe and an area the attacker can control.
For this mitigation, we'd need to:
1) set the config option in the kernel build 2) update /proc/sys/kernel/modprobe however it's set (CONFIG_MODPROBE_PATH) 3) update /proc/sys/kernel/core_pattern however it's set
If we fix the modprobe path issue, there are a couple other areas that call usermode helper such as handle_initrd, fork_usermode_driver, CONFIG_UEVENT_HELPER, and sbin/request-key which would need some touch ups.
The benefit is a lot of privilege escalation attacks are taken away.
Does this sound worthwhile? Would people support this? Does this need to be filed as a system wide change? Who could help make this happen if approved?
Thanks, -Steve
On 6/1/23 09:17, Steve Grubb wrote:
Hello,
I work on RHEL security problems. I have been looking into a number of exploits and I think we have a problem that has an easy fix. We are not using the CONFIG_STATIC_USERMODEHELPER_PATH kernel config option. There are a number of exploits that overwrite the path to modprobe and then pass something weird that causes modprobe to be invoked. But instead of modprobe, it's their reverse shell.
If we make the assigment CONFIG_STATIC_USERMODEHELPER_PATH="/usr/" and we change /proc/sys/kernel/modprobe to sbin/modprobe and /proc/sys/kernel/ core_pattern to lib/systemd/systemd-coredump %P %u %g %s %t %c %h, then it limits any exploits to programs that are in /usr. Only root can write here, therefore no escalation. Typically, an exploit changes modprobe path to /tmp/ foo which is shorter than /usr/sbin/modprobe and an area the attacker can control.
For this mitigation, we'd need to:
- set the config option in the kernel build
- update /proc/sys/kernel/modprobe however it's set (CONFIG_MODPROBE_PATH)
- update /proc/sys/kernel/core_pattern however it's set
If we fix the modprobe path issue, there are a couple other areas that call usermode helper such as handle_initrd, fork_usermode_driver, CONFIG_UEVENT_HELPER, and sbin/request-key which would need some touch ups.
The benefit is a lot of privilege escalation attacks are taken away.
Does this sound worthwhile? Would people support this? Does this need to be filed as a system wide change? Who could help make this happen if approved?
It sounds worth while to me, ;)
I'd be up for helping with it.
As much as I hate working in the proc file system I can try
and work out what needs to be done for the proc file system
bits.
Ian
Hello,
I want to add some missing information...
On Thursday, January 5, 2023 8:43:34 PM EST Ian Kent wrote:
On 6/1/23 09:17, Steve Grubb wrote:
I work on RHEL security problems. I have been looking into a number of exploits and I think we have a problem that has an easy fix.
Here's some examples of what I'm look in at:
https://twitter.com/ETenal7/status/1506708119757328385 https://lkmidas.github.io/posts/20210223-linux-kernel-pwn-modprobe/ https://etenal.me/archives/1825 https://twitter.com/ky1ebot/status/1539820418479009792 https://github.com/theori-io/CVE-2022-32250-exploit https://seclists.org/oss-sec/2022/q3/154
etc
We are not using the CONFIG_STATIC_USERMODEHELPER_PATH kernel config option. There are a number of exploits that overwrite the path to modprobe and then pass something weird that causes modprobe to be invoked. But instead of modprobe, it's their reverse shell.
If we make the assigment CONFIG_STATIC_USERMODEHELPER_PATH="/usr/" and we change /proc/sys/kernel/modprobe to sbin/modprobe and /proc/sys/ kernel/core_pattern to lib/systemd/systemd-coredump %P %u %g %s %t %c %h, then it limits any exploits to programs that are in /usr.
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/ kernel/umh.c?h=v5.15#n371
Only root can write here, therefore no escalation. Typically, an exploit changes modprobe path to /tmp/ foo which is shorter than /usr/sbin/modprobe and an area the attacker can control.
For this mitigation, we'd need to:
- set the config option in the kernel build
- update /proc/sys/kernel/modprobe however it's set
(CONFIG_MODPROBE_PATH) 3) update /proc/sys/kernel/core_pattern however it's set
If we fix the modprobe path issue, there are a couple other areas that call usermode helper such as handle_initrd, fork_usermode_driver, CONFIG_UEVENT_HELPER, and sbin/request-key which would need some touch ups.
The benefit is a lot of privilege escalation attacks are taken away.
Does this sound worthwhile? Would people support this? Does this need to be filed as a system wide change? Who could help make this happen if approved?
It sounds worth while to me, ;)
Thanks for the initial vote of confidence. It's 3 am in Europe, so I'll wait a bit for feedback.
-Steve
I'd be up for helping with it.
As much as I hate working in the proc file system I can try and work out what needs to be done for the proc file system bits.
On 6/1/23 10:12, Steve Grubb wrote:
Hello,
I want to add some missing information...
On Thursday, January 5, 2023 8:43:34 PM EST Ian Kent wrote:
On 6/1/23 09:17, Steve Grubb wrote:
I work on RHEL security problems. I have been looking into a number of exploits and I think we have a problem that has an easy fix.
Here's some examples of what I'm look in at:
https://twitter.com/ETenal7/status/1506708119757328385 https://lkmidas.github.io/posts/20210223-linux-kernel-pwn-modprobe/ https://etenal.me/archives/1825 https://twitter.com/ky1ebot/status/1539820418479009792 https://github.com/theori-io/CVE-2022-32250-exploit https://seclists.org/oss-sec/2022/q3/154
etc
We are not using the CONFIG_STATIC_USERMODEHELPER_PATH kernel config option. There are a number of exploits that overwrite the path to modprobe and then pass something weird that causes modprobe to be invoked. But instead of modprobe, it's their reverse shell.
If we make the assigment CONFIG_STATIC_USERMODEHELPER_PATH="/usr/" and we change /proc/sys/kernel/modprobe to sbin/modprobe and /proc/sys/ kernel/core_pattern to lib/systemd/systemd-coredump %P %u %g %s %t %c %h, then it limits any exploits to programs that are in /usr.
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/ kernel/umh.c?h=v5.15#n371
Right, but the above description isn't quite how these config
options are meant to work.
To be maintainable in Fedora the way in which we utilize the bits
that are setup to do this type of thing would need to be inline
with the way they are intended to be used.
CONFIG_STATIC_USERMODEHELPER enables this, CONFIG_STATIC_USERMODEHELPER_PATH allows you to specify the path to the binary through with "all" usermode helper invocations are done.
So we would need to write something that validates its own path and the path of the thing to be invoked and, if all is ok, execute the program.
The problem I see with this is that modprobe isn't the only thing used with umh, there are a number of other things that use this so some care is going to be needed in what is done and how it's done.
The idea of only allowing specific path prefixes sounds ok since it's simple, fairly safe, umh kernel users really should be using programs in "safe" locations, so this all sounds doable ... I'm sure there will be a bunch of gotchas ...
Ian
Only root can write here, therefore no escalation. Typically, an exploit changes modprobe path to /tmp/ foo which is shorter than /usr/sbin/modprobe and an area the attacker can control.
For this mitigation, we'd need to:
- set the config option in the kernel build
- update /proc/sys/kernel/modprobe however it's set
(CONFIG_MODPROBE_PATH) 3) update /proc/sys/kernel/core_pattern however it's set
If we fix the modprobe path issue, there are a couple other areas that call usermode helper such as handle_initrd, fork_usermode_driver, CONFIG_UEVENT_HELPER, and sbin/request-key which would need some touch ups.
The benefit is a lot of privilege escalation attacks are taken away.
Does this sound worthwhile? Would people support this? Does this need to be filed as a system wide change? Who could help make this happen if approved?
It sounds worth while to me, ;)
Thanks for the initial vote of confidence. It's 3 am in Europe, so I'll wait a bit for feedback.
-Steve
I'd be up for helping with it.
As much as I hate working in the proc file system I can try and work out what needs to be done for the proc file system bits.
On Do, 05.01.23 20:17, Steve Grubb (sgrubb@redhat.com) wrote:
Hello,
I work on RHEL security problems. I have been looking into a number of exploits and I think we have a problem that has an easy fix. We are not using the CONFIG_STATIC_USERMODEHELPER_PATH kernel config option. There are a number of exploits that overwrite the path to modprobe and then pass something weird that causes modprobe to be invoked. But instead of modprobe, it's their reverse shell.
umh is such a problematic interface. The processes forked off that way live in a weird netherworld besides the init system, in the root cgroup (and that even though inner cgroups in cgroupsv2 are not supposed to have processes in them) without any of the resource or security restrictions we otherwise make on all of userspace. It's stuff that runs in userspace but is entirely unmanaged by userspace security and resource wise. (And it's also frickin slow)
I wished we could get rid of umh entirely. For example, by having some maybe netlink based protocol or so that userspace could use to subscribe to umh events somehow and then gets the chance to fork off the processes itself. The init system could then hook into that and dispatch things in proper services with sandboxing, resource controls and everything applied, in a sensible cgroup.
Lennart
-- Lennart Poettering, Berlin
Hello,
On Friday, January 6, 2023 9:33:12 AM EST Lennart Poettering wrote:
On Do, 05.01.23 20:17, Steve Grubb (sgrubb@redhat.com) wrote:
I work on RHEL security problems. I have been looking into a number of exploits and I think we have a problem that has an easy fix. We are not using the CONFIG_STATIC_USERMODEHELPER_PATH kernel config option. There are a number of exploits that overwrite the path to modprobe and then pass something weird that causes modprobe to be invoked. But instead of modprobe, it's their reverse shell.
umh is such a problematic interface. The processes forked off that way live in a weird netherworld besides the init system, in the root cgroup (and that even though inner cgroups in cgroupsv2 are not supposed to have processes in them) without any of the resource or security restrictions we otherwise make on all of userspace.
One approach to solving this is to use selinux policy. I was informed overnight that policy 38.2-1 should now enforce kernel transitions to specific helper applications. So, maybe this is solved well enough? I suppose this is easy enough to test by changing core_pattern to something else and see if we get the AVC.
It's stuff that runs in userspace but is entirely unmanaged by userspace security and resource wise. (And it's also frickin slow)
I wished we could get rid of umh entirely. For example, by having some maybe netlink based protocol or so that userspace could use to subscribe to umh events somehow and then gets the chance to fork off the processes itself. The init system could then hook into that and dispatch things in proper services with sandboxing, resource controls and everything applied, in a sensible cgroup.
Yeah, that's another approach that may have merit. But with the asynchronous nature of that approach, I don't know how the kernel would know it can now make calls into the module it needed to have loaded.
The other area that needs a good looking at the the msg_msg attack vector.
-Steve
Hello,
On Friday, January 6, 2023 10:10:21 AM EST Steve Grubb wrote:
One approach to solving this is to use selinux policy. I was informed overnight that policy 38.2-1 should now enforce kernel transitions to specific helper applications. So, maybe this is solved well enough?
I can verify the selinux approach works on rawhide. What this does is copy wall to the admin directory where it gets the home_t label, then we change core_pattern to point to it, then run a program that will dump core, and then check the audit logs.
[root@ospp ~]# cp /usr/bin/wall /home/admin/ [root@ospp ~]# ls -Z /home/admin/wall unconfined_u:object_r:user_home_t:s0 /home/admin/wall [root@ospp ~]# echo "|/home/admin/wall" > /proc/sys/kernel/core_pattern [root@ospp ~]# cat dump.c #include <stdio.h> #include <string.h> #include <stddef.h>
int main(int argc, char *argv[]) { if (strcmp(*argv[2], argv[3]) == 0) printf("foo\n"); else printf("bar\n"); return 0; } [root@ospp ~]# gcc -o dump dump.c 2>/dev/null [root@ospp ~]# ./dump Segmentation fault [root@ospp ~]# ausearch --start recent ---- time->Fri Jan 6 11:24:48 2023 node=ospp type=ANOM_ABEND msg=audit(1673022288.639:253): auid=1000 uid=0 gid=0 ses=1 subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 pid=1078 comm="dump" exe="/root/dump" sig=11 res=1 ---- time->Fri Jan 6 11:24:48 2023 node=ospp type=AVC msg=audit(1673022288.639:254): avc: denied { execute } for pid=1079 comm="kworker/u8:2" name="wall" dev="dm-7" ino=17051899 scontext=system_u:system_r:kernel_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=file permissive=0
I think this closes the modprobe path escalation if you have selinux enabled. This should be publicized. I'd still like to do something for those not using selinux for whatever reason.
-Steve
On Fr, 06.01.23 10:10, Steve Grubb (sgrubb@redhat.com) wrote:
Hello,
On Friday, January 6, 2023 9:33:12 AM EST Lennart Poettering wrote:
On Do, 05.01.23 20:17, Steve Grubb (sgrubb@redhat.com) wrote:
I work on RHEL security problems. I have been looking into a number of exploits and I think we have a problem that has an easy fix. We are not using the CONFIG_STATIC_USERMODEHELPER_PATH kernel config option. There are a number of exploits that overwrite the path to modprobe and then pass something weird that causes modprobe to be invoked. But instead of modprobe, it's their reverse shell.
umh is such a problematic interface. The processes forked off that way live in a weird netherworld besides the init system, in the root cgroup (and that even though inner cgroups in cgroupsv2 are not supposed to have processes in them) without any of the resource or security restrictions we otherwise make on all of userspace.
One approach to solving this is to use selinux policy.
selinux cannot apply resource controls, not can it arrange processes properly in the cgroup tree, nor can it apply seccomp filters, namespacing and so on.
selinux can do some things, sure, but an init system is not a MAC, it does a lot more (and also a lot less).
Yeah, that's another approach that may have merit. But with the asynchronous nature of that approach, I don't know how the kernel would know it can now make calls into the module it needed to have loaded.
Well, umh is async too in a way, kernel must wait for the userspace process to finish. There's not much of a difference to say "fork off + wait for process exit" and "send netlink message to userspace + wait for reply".
Lennart
-- Lennart Poettering, Berlin
On Fri, 2023-01-06 at 18:21 +0100, Lennart Poettering wrote:
On Fr, 06.01.23 10:10, Steve Grubb (sgrubb@redhat.com) wrote:
Hello,
On Friday, January 6, 2023 9:33:12 AM EST Lennart Poettering wrote:
On Do, 05.01.23 20:17, Steve Grubb (sgrubb@redhat.com) wrote:
I work on RHEL security problems. I have been looking into a number of exploits and I think we have a problem that has an easy fix. We are not using the CONFIG_STATIC_USERMODEHELPER_PATH kernel config option. There are a number of exploits that overwrite the path to modprobe and then pass something weird that causes modprobe to be invoked. But instead of modprobe, it's their reverse shell.
umh is such a problematic interface. The processes forked off that way live in a weird netherworld besides the init system, in the root cgroup (and that even though inner cgroups in cgroupsv2 are not supposed to have processes in them) without any of the resource or security restrictions we otherwise make on all of userspace.
One approach to solving this is to use selinux policy.
selinux cannot apply resource controls, not can it arrange processes properly in the cgroup tree, nor can it apply seccomp filters, namespacing and so on.
selinux can do some things, sure, but an init system is not a MAC, it does a lot more (and also a lot less).
Yeah, that's another approach that may have merit. But with the asynchronous nature of that approach, I don't know how the kernel would know it can now make calls into the module it needed to have loaded.
Well, umh is async too in a way, kernel must wait for the userspace process to finish. There's not much of a difference to say "fork off + wait for process exit" and "send netlink message to userspace + wait for reply".
If I remember correctly the claim was that umh is robust if the user space fails and just terminates. As then the kernel know user space is gone, whether it got the data it needed or not and can stop waiting.
While messages may never get replied to and require handling timeouts and then handling the case a user space process was slow and ignoring late replies.
Not sure this is really a good point given waiting indefinitely for a user space program that hangs for some reason seems worse to me.
When I had to code a call from knfsd to user space for GSS-Proxy I used unix sockets. I think that is better than netlink in some cases as sockets are simpler to handle from user space programs and are also easily namespaced...
Simo.
On Mo, 09.01.23 15:18, Simo Sorce (simo@redhat.com) wrote:
If I remember correctly the claim was that umh is robust if the user space fails and just terminates. As then the kernel know user space is gone, whether it got the data it needed or not and can stop waiting.
While messages may never get replied to and require handling timeouts and then handling the case a user space process was slow and ignoring late replies.
Not sure this is really a good point given waiting indefinitely for a user space program that hangs for some reason seems worse to me.
When I had to code a call from knfsd to user space for GSS-Proxy I used unix sockets. I think that is better than netlink in some cases as sockets are simpler to handle from user space programs and are also easily namespaced...
Well, the requests would come from the kernel, and the kernel typically speaks netlink, not AF_UNIX. But quite frankly, I really don't care what transport would be used. I'd just be happy if we could get rid of the kernel forking its own stuff.
Lennart
-- Lennart Poettering, Berlin
-
Ian Kent -
Lennart Poettering -
Simo Sorce -
Steve Grubb