static USERMODEHELPER_PATH

[Steve Grubb]

Hello,

I work on RHEL security problems. I have been looking into a number of 
exploits and I think we have a problem that has an easy fix. We are not using 
the CONFIG_STATIC_USERMODEHELPER_PATH kernel config option. There are a number 
of exploits that overwrite the path to modprobe and then pass something weird 
that causes modprobe to be invoked. But instead of modprobe, it's their 
reverse shell.

If we make the assigment CONFIG_STATIC_USERMODEHELPER_PATH="/usr/" and we 
change /proc/sys/kernel/modprobe to sbin/modprobe and /proc/sys/kernel/
core_pattern to lib/systemd/systemd-coredump %P %u %g %s %t %c %h, then it 
limits any exploits to programs that are in /usr. Only root can write here, 
therefore no escalation. Typically, an exploit changes modprobe path to /tmp/
foo which is shorter than /usr/sbin/modprobe and an area the attacker can 
control.

For this mitigation, we'd need to:

1) set the config option in the kernel build
2) update /proc/sys/kernel/modprobe however it's set (CONFIG_MODPROBE_PATH)
3) update /proc/sys/kernel/core_pattern however it's set

If we fix the modprobe path issue, there are a couple other areas that call 
usermode helper such as handle_initrd, fork_usermode_driver, 
CONFIG_UEVENT_HELPER, and sbin/request-key which would need some touch ups.

The benefit is a lot of privilege escalation attacks are taken away.

Does this sound worthwhile? Would people support this? Does this need to be 
filed as a system wide change? Who could help make this happen if approved?

Thanks,
-Steve

[Ian Kent]

On 6/1/23 09:17, Steve Grubb wrote:
>  Hello,
>
>  I work on RHEL security problems. I have been looking into a number of
>  exploits and I think we have a problem that has an easy fix. We are not using
>  the CONFIG_STATIC_USERMODEHELPER_PATH kernel config option. There are a number
>  of exploits that overwrite the path to modprobe and then pass something weird
>  that causes modprobe to be invoked. But instead of modprobe, it's their
>  reverse shell.
>
>  If we make the assigment CONFIG_STATIC_USERMODEHELPER_PATH="/usr/" and we
>  change /proc/sys/kernel/modprobe to sbin/modprobe and /proc/sys/kernel/
>  core_pattern to lib/systemd/systemd-coredump %P %u %g %s %t %c %h, then it
>  limits any exploits to programs that are in /usr. Only root can write here,
>  therefore no escalation. Typically, an exploit changes modprobe path to /tmp/
>  foo which is shorter than /usr/sbin/modprobe and an area the attacker can
>  control.
>
>  For this mitigation, we'd need to:
>
>  1) set the config option in the kernel build
>  2) update /proc/sys/kernel/modprobe however it's set (CONFIG_MODPROBE_PATH)
>  3) update /proc/sys/kernel/core_pattern however it's set
>
>  If we fix the modprobe path issue, there are a couple other areas that call
>  usermode helper such as handle_initrd, fork_usermode_driver,
>  CONFIG_UEVENT_HELPER, and sbin/request-key which would need some touch ups.
>
>  The benefit is a lot of privilege escalation attacks are taken away.
>
>  Does this sound worthwhile? Would people support this? Does this need to be
>  filed as a system wide change? Who could help make this happen if approved? 
It sounds worth while to me, ;)


I'd be up for helping with it.

As much as I hate working in the proc file system I can try

and work out what needs to be done for the proc file system

bits.


Ian

[Steve Grubb]

Hello,

I want to add some missing information...

On Thursday, January 5, 2023 8:43:34 PM EST Ian Kent wrote:
>  On 6/1/23 09:17, Steve Grubb wrote:
>  > I work on RHEL security problems. I have been looking into a number of
>  > exploits and I think we have a problem that has an easy fix.  
Here's some examples of what I'm look in at:

https://twitter.com/ETenal7/status/1506708119757328385
https://lkmidas.github.io/posts/20210223-linux-kernel-pwn-modprobe/
https://etenal.me/archives/1825
https://twitter.com/ky1ebot/status/1539820418479009792
https://github.com/theori-io/CVE-2022-32250-exploit
https://seclists.org/oss-sec/2022/q3/154

etc

>  > We are not
>  > using the CONFIG_STATIC_USERMODEHELPER_PATH kernel config option. There
>  > are a number of exploits that overwrite the path to modprobe and then
>  > pass something weird that causes modprobe to be invoked. But instead of
>  > modprobe, it's their reverse shell.
>  >
>  > If we make the assigment CONFIG_STATIC_USERMODEHELPER_PATH="/usr/" and
>  > we change /proc/sys/kernel/modprobe to sbin/modprobe and /proc/sys/
>  > kernel/core_pattern to lib/systemd/systemd-coredump %P %u %g %s %t %c %h,
>  > then it limits any exploits to programs that are in /usr. 
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/
kernel/umh.c?h=v5.15#n371

>  > Only root can
>  > write here, therefore no escalation. Typically, an exploit changes
>  > modprobe path to /tmp/ foo which is shorter than /usr/sbin/modprobe and
>  > an area the attacker can control.
>  >
>  > For this mitigation, we'd need to:
>  >
>  > 1) set the config option in the kernel build
>  > 2) update /proc/sys/kernel/modprobe however it's set
>  > (CONFIG_MODPROBE_PATH)
>  > 3) update /proc/sys/kernel/core_pattern however it's set
>  >
>  > If we fix the modprobe path issue, there are a couple other areas that
>  > call usermode helper such as handle_initrd, fork_usermode_driver,
>  > CONFIG_UEVENT_HELPER, and sbin/request-key which would need some touch
>  > ups.
>  >
>  > The benefit is a lot of privilege escalation attacks are taken away.
>  >
>  > Does this sound worthwhile? Would people support this? Does this need to
>  > be filed as a system wide change? Who could help make this happen if
>  > approved?
>  
>  It sounds worth while to me, ;) 
Thanks for the initial vote of confidence. It's 3 am in Europe, so I'll wait a 
bit for feedback.

-Steve
 
>  I'd be up for helping with it.
>  
>  As much as I hate working in the proc file system I can try
>  and work out what needs to be done for the proc file system
>  bits.

[Ian Kent]

On 6/1/23 10:12, Steve Grubb wrote:
>  Hello,
>
>  I want to add some missing information...
>
>  On Thursday, January 5, 2023 8:43:34 PM EST Ian Kent wrote:
> > On 6/1/23 09:17, Steve Grubb wrote:
> >> I work on RHEL security problems. I have been looking into a number of
> >> exploits and I think we have a problem that has an easy fix.
>  Here's some examples of what I'm look in at:
>
>  https://twitter.com/ETenal7/status/1506708119757328385
>  https://lkmidas.github.io/posts/20210223-linux-kernel-pwn-modprobe/
>  https://etenal.me/archives/1825
>  https://twitter.com/ky1ebot/status/1539820418479009792
>  https://github.com/theori-io/CVE-2022-32250-exploit
>  https://seclists.org/oss-sec/2022/q3/154
>
>  etc
>
> >> We are not
> >> using the CONFIG_STATIC_USERMODEHELPER_PATH kernel config option. There
> >> are a number of exploits that overwrite the path to modprobe and then
> >> pass something weird that causes modprobe to be invoked. But instead of
> >> modprobe, it's their reverse shell.
> >>
> >> If we make the assigment CONFIG_STATIC_USERMODEHELPER_PATH="/usr/" and
> >> we change /proc/sys/kernel/modprobe to sbin/modprobe and /proc/sys/
> >> kernel/core_pattern to lib/systemd/systemd-coredump %P %u %g %s %t %c %h,
> >> then it limits any exploits to programs that are in /usr.
>  https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/
>  kernel/umh.c?h=v5.15#n371 
Right, but the above description isn't quite how these config

options are meant to work.


To be maintainable in Fedora the way in which we utilize the bits

that are setup to do this type of thing would need to be inline

with the way they are intended to be used.

CONFIG_STATIC_USERMODEHELPER enables this,
CONFIG_STATIC_USERMODEHELPER_PATH allows you to specify the path
to the binary through with "all" usermode helper invocations are
done.

So we would need to write something that validates its own path
and the path of the thing to be invoked and, if all is ok, execute
the program.

The problem I see with this is that modprobe isn't the only thing
used with umh, there are a number of other things that use this so
some care is going to be needed in what is done and how it's done.

The idea of only allowing specific path prefixes sounds ok since
it's simple, fairly safe, umh kernel users really should be using
programs in "safe" locations, so this all sounds doable ... I'm
sure there will be a bunch of gotchas ...

Ian

>>> Only root can
>>> write here, therefore no escalation. Typically, an exploit changes
>>> modprobe path to /tmp/ foo which is shorter than /usr/sbin/modprobe and
>>> an area the attacker can control.
>>>
>>> For this mitigation, we'd need to:
>>>
>>> 1) set the config option in the kernel build
>>> 2) update /proc/sys/kernel/modprobe however it's set
>>> (CONFIG_MODPROBE_PATH)
>>> 3) update /proc/sys/kernel/core_pattern however it's set
>>>
>>> If we fix the modprobe path issue, there are a couple other areas that
>>> call usermode helper such as handle_initrd, fork_usermode_driver,
>>> CONFIG_UEVENT_HELPER, and sbin/request-key which would need some touch
>>> ups.
>>>
>>> The benefit is a lot of privilege escalation attacks are taken away.
>>>
>>> Does this sound worthwhile? Would people support this? Does this need to
>>> be filed as a system wide change? Who could help make this happen if
>>> approved?
>> It sounds worth while to me, ;)
> Thanks for the initial vote of confidence. It's 3 am in Europe, so I'll wait
a
> bit for feedback.
>
> -Steve
>   
>> I'd be up for helping with it.
>>
>> As much as I hate working in the proc file system I can try
>> and work out what needs to be done for the proc file system
>> bits.
>
>

[Lennart Poettering]

On Do, 05.01.23 20:17, Steve Grubb (sgrubb(a)redhat.com) wrote:

>  Hello,
>
>  I work on RHEL security problems. I have been looking into a number of
>  exploits and I think we have a problem that has an easy fix. We are not using
>  the CONFIG_STATIC_USERMODEHELPER_PATH kernel config option. There are a number
>  of exploits that overwrite the path to modprobe and then pass something weird
>  that causes modprobe to be invoked. But instead of modprobe, it's their
>  reverse shell. 
umh is such a problematic interface. The processes forked off that way
live in a weird netherworld besides the init system, in the root
cgroup (and that even though inner cgroups in cgroupsv2 are not
supposed to have processes in them) without any of the resource or
security restrictions we otherwise make on all of userspace. It's
stuff that runs in userspace but is entirely unmanaged by userspace
security and resource wise. (And it's also frickin slow)

I wished we could get rid of umh entirely. For example, by having some
maybe netlink based protocol or so that userspace could use to
subscribe to umh events somehow and then gets the chance to fork off
the processes itself. The init system could then hook into that and
dispatch things in proper services with sandboxing, resource controls
and everything applied, in a sensible cgroup.

Lennart

--
Lennart Poettering, Berlin

[Steve Grubb]

Hello,

On Friday, January 6, 2023 9:33:12 AM EST Lennart Poettering wrote:
>  On Do, 05.01.23 20:17, Steve Grubb (sgrubb(a)redhat.com) wrote:
>  > I work on RHEL security problems. I have been looking into a number of
>  > exploits and I think we have a problem that has an easy fix. We are not
>  > using the CONFIG_STATIC_USERMODEHELPER_PATH kernel config option. There
>  > are a number of exploits that overwrite the path to modprobe and then
>  > pass something weird that causes modprobe to be invoked. But instead of
>  > modprobe, it's their reverse shell.
>  
>  umh is such a problematic interface. The processes forked off that way
>  live in a weird netherworld besides the init system, in the root
>  cgroup (and that even though inner cgroups in cgroupsv2 are not
>  supposed to have processes in them) without any of the resource or
>  security restrictions we otherwise make on all of userspace.  
One approach to solving this is to use selinux policy. I was informed 
overnight that policy 38.2-1 should now enforce kernel transitions to specific 
helper applications. So, maybe this is solved well enough? I suppose this is 
easy enough to test by changing core_pattern to something else and see if we 
get the AVC.

>  It's stuff that runs in userspace but is entirely unmanaged by
> userspace
>  security and resource wise. (And it's also frickin slow)
>  
>  I wished we could get rid of umh entirely. For example, by having some
>  maybe netlink based protocol or so that userspace could use to
>  subscribe to umh events somehow and then gets the chance to fork off
>  the processes itself. The init system could then hook into that and
>  dispatch things in proper services with sandboxing, resource controls
>  and everything applied, in a sensible cgroup. 
Yeah, that's another approach that may have merit. But with the asynchronous 
nature of that approach, I don't know how the kernel would know it can now 
make calls into the module it needed to have loaded.

The other area that needs a good looking at the the msg_msg attack vector.

-Steve

[Steve Grubb]

Hello,

On Friday, January 6, 2023 10:10:21 AM EST Steve Grubb wrote:
>  One approach to solving this is to use selinux policy. I was
> informed
>  overnight that policy 38.2-1 should now enforce kernel transitions to
>  specific helper applications. So, maybe this is solved well enough? 
I can verify the selinux approach works on rawhide. What this does is copy
wall to the admin directory where it gets the home_t label, then we change 
core_pattern to point to it, then run a program that will dump core, and  then
check the audit logs. 

[root@ospp ~]# cp /usr/bin/wall /home/admin/
[root@ospp ~]# ls -Z /home/admin/wall
unconfined_u:object_r:user_home_t:s0 /home/admin/wall
[root@ospp ~]# echo "|/home/admin/wall" > /proc/sys/kernel/core_pattern 
[root@ospp ~]# cat dump.c
#include <stdio.h>
#include <string.h>
#include <stddef.h>

int main(int argc, char *argv[])
{
	if (strcmp(*argv[2], argv[3]) == 0)
		printf("foo\n");
	else
		printf("bar\n");
	return 0;
}
[root@ospp ~]# gcc -o dump dump.c 2>/dev/null
[root@ospp ~]# ./dump 
Segmentation fault
[root@ospp ~]# ausearch --start recent
----
time->Fri Jan  6 11:24:48 2023
node=ospp type=ANOM_ABEND msg=audit(1673022288.639:253): auid=1000 uid=0 gid=0
ses=1 subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 pid=1078 
comm="dump" exe="/root/dump" sig=11 res=1
----
time->Fri Jan  6 11:24:48 2023
node=ospp type=AVC msg=audit(1673022288.639:254): avc:  denied  { execute } 
for  pid=1079 comm="kworker/u8:2" name="wall" dev="dm-7"
ino=17051899 
scontext=system_u:system_r:kernel_t:s0 
tcontext=unconfined_u:object_r:user_home_t:s0 tclass=file permissive=0


I think this closes the modprobe path escalation if you have selinux enabled.
This should be publicized. I'd still like to do something for those not using 
selinux for whatever reason.

-Steve

[Lennart Poettering]

On Fr, 06.01.23 10:10, Steve Grubb (sgrubb(a)redhat.com) wrote:

>  Hello,
>
>  On Friday, January 6, 2023 9:33:12 AM EST Lennart Poettering wrote:
>  > On Do, 05.01.23 20:17, Steve Grubb (sgrubb(a)redhat.com) wrote:
>  > > I work on RHEL security problems. I have been looking into a number of
>  > > exploits and I think we have a problem that has an easy fix. We are not
>  > > using the CONFIG_STATIC_USERMODEHELPER_PATH kernel config option. There
>  > > are a number of exploits that overwrite the path to modprobe and then
>  > > pass something weird that causes modprobe to be invoked. But instead of
>  > > modprobe, it's their reverse shell.
>  >
>  > umh is such a problematic interface. The processes forked off that way
>  > live in a weird netherworld besides the init system, in the root
>  > cgroup (and that even though inner cgroups in cgroupsv2 are not
>  > supposed to have processes in them) without any of the resource or
>  > security restrictions we otherwise make on all of userspace.
>
>  One approach to solving this is to use selinux policy. 
selinux cannot apply resource controls, not can it arrange processes
properly in the cgroup tree, nor can it apply seccomp filters,
namespacing and so on.

selinux can do some things, sure, but an init system is not a MAC, it
does a lot more (and also a lot less).

>  Yeah, that's another approach that may have merit. But with the
> asynchronous
>  nature of that approach, I don't know how the kernel would know it can now
>  make calls into the module it needed to have loaded. 
Well, umh is async too in a way, kernel must wait for the userspace
process to finish. There's not much of a difference to say "fork off +
wait for process exit" and "send netlink message to userspace + wait
for reply".

Lennart

--
Lennart Poettering, Berlin

[Simo Sorce]

On Fri, 2023-01-06 at 18:21 +0100, Lennart Poettering wrote:
>  On Fr, 06.01.23 10:10, Steve Grubb (sgrubb(a)redhat.com) wrote:
>  
>  > Hello,
>  > 
>  > On Friday, January 6, 2023 9:33:12 AM EST Lennart Poettering wrote:
>  > > On Do, 05.01.23 20:17, Steve Grubb (sgrubb(a)redhat.com) wrote:
>  > > > I work on RHEL security problems. I have been looking into a number of
>  > > > exploits and I think we have a problem that has an easy fix. We are not
>  > > > using the CONFIG_STATIC_USERMODEHELPER_PATH kernel config option. There
>  > > > are a number of exploits that overwrite the path to modprobe and then
>  > > > pass something weird that causes modprobe to be invoked. But instead of
>  > > > modprobe, it's their reverse shell.
>  > > 
>  > > umh is such a problematic interface. The processes forked off that way
>  > > live in a weird netherworld besides the init system, in the root
>  > > cgroup (and that even though inner cgroups in cgroupsv2 are not
>  > > supposed to have processes in them) without any of the resource or
>  > > security restrictions we otherwise make on all of userspace.
>  > 
>  > One approach to solving this is to use selinux policy.
>  
>  selinux cannot apply resource controls, not can it arrange processes
>  properly in the cgroup tree, nor can it apply seccomp filters,
>  namespacing and so on.
>  
>  selinux can do some things, sure, but an init system is not a MAC, it
>  does a lot more (and also a lot less).
>  
>  > Yeah, that's another approach that may have merit. But with the asynchronous
>  > nature of that approach, I don't know how the kernel would know it can now
>  > make calls into the module it needed to have loaded.
>  
>  Well, umh is async too in a way, kernel must wait for the userspace
>  process to finish. There's not much of a difference to say "fork off +
>  wait for process exit" and "send netlink message to userspace + wait
>  for reply". 
If I remember correctly the claim was that umh is robust if the user
space fails and just terminates. As then the kernel know user space is
gone, whether it got the data it needed or not and can stop waiting.

While messages may never get replied to and require handling timeouts
and then handling the case a user space process was slow and ignoring
late replies.

Not sure this is really a good point given waiting indefinitely for a
user space program that hangs for some reason seems worse to me.

When I had to code a call from knfsd to user space for GSS-Proxy I used
unix sockets. I think that is better than netlink in some cases as
sockets are simpler to handle from user space programs and are also
easily namespaced...

Simo.

-- 
Simo Sorce
RHEL Crypto Team
Red Hat, Inc

[Lennart Poettering]

On Mo, 09.01.23 15:18, Simo Sorce (simo(a)redhat.com) wrote:

>  If I remember correctly the claim was that umh is robust if the user
>  space fails and just terminates. As then the kernel know user space is
>  gone, whether it got the data it needed or not and can stop waiting.
>
>  While messages may never get replied to and require handling timeouts
>  and then handling the case a user space process was slow and ignoring
>  late replies.
>
>  Not sure this is really a good point given waiting indefinitely for a
>  user space program that hangs for some reason seems worse to me.
>
>  When I had to code a call from knfsd to user space for GSS-Proxy I used
>  unix sockets. I think that is better than netlink in some cases as
>  sockets are simpler to handle from user space programs and are also
>  easily namespaced... 
Well, the requests would come from the kernel, and the kernel
typically speaks netlink, not AF_UNIX. But quite frankly, I really
don't care what transport would be used. I'd just be happy if we could
get rid of the kernel forking its own stuff.

Lennart

--
Lennart Poettering, Berlin