leaving setfcap in docker containers

[Matthew Miller]

Quick backstory: unless run in privledged mode, Docker drops a bunch of
capabilities when launching a container. One of these is setfcap. This
breaks of binary RPMs like httpd where the daemon is installed with file
capabilities instead.

We're considering removing setfcap from the list of dropped capabilities. It
seems safe to me (note that you run as root inside the container), but I'd
like some security-minded review. Could this be used for evil?

https://bugzilla.redhat.com/show_bug.cgi?id=1012952

Thanks!

-- 
Matthew Miller  ☁☁☁  Fedora Cloud Architect  ☁☁☁  <mattdm(a)fedoraproject.org&gt;

[Jóhann B. Guðmundsson]

On 09/27/2013 05:28 PM, Matthew Miller wrote:
>  We're considering removing setfcap from the list of dropped capabilities. It
>  seems safe to me 
I dont have any security degrees nor do I consider myself an evil man 
and probably Steve and Dan would be better suited to answer this 
question since I'm far from being any expert on the subject but 
hypothetically would not someone being able to do something like this in 
this educational sample I'm providing

cd ~user

vi bd.c

#include <unistd.h>
#include <fcntl.h>
main()
{
setuid(0);
char *name[2];
name[0] = "/bin/sh";
name[1] = 0x0;
execve(name[0], name, 0x0);
return 0;
}

gcc bd.c -o .b
chown user:user .b
chmod 750 .b
setcap cap_setuid=ep
rm bd.c

./.b

if you did?

I personally would recommend we kept it on after all Dan did push for 
that feature for a reason but as I said I'm no expert on the topic.

JBG

[Daniel J Walsh]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 09/27/2013 03:33 PM, "Jóhann B. Guðmundsson" wrote:
>  On 09/27/2013 05:28 PM, Matthew Miller wrote:
> > 
> > We're considering removing setfcap from the list of dropped capabilities.
> > It seems safe to me
>  
>  I dont have any security degrees nor do I consider myself an evil man and 
>  probably Steve and Dan would be better suited to answer this question since
>  I'm far from being any expert on the subject but hypothetically would not
>  someone being able to do something like this in this educational sample I'm
>  providing
>  
>  cd ~user
>  
>  vi bd.c
>  
>  #include <unistd.h> #include <fcntl.h> main() { setuid(0); char *name[2]; 
>  name[0] = "/bin/sh"; name[1] = 0x0; execve(name[0], name, 0x0); return 0; 
>  }
>  
>  gcc bd.c -o .b chown user:user .b chmod 750 .b setcap cap_setuid=ep rm
>  bd.c
>  
>  ./.b
>  
>  if you did?
>  
>  I personally would recommend we kept it on after all Dan did push for that 
>  feature for a reason but as I said I'm no expert on the topic.
>  
>  JBG -- security mailing list security(a)lists.fedoraproject.org 
>  https://admin.fedoraproject.org/mailman/listinfo/security 
Well currently without
setfcap you can do the same thing with

gcc bd.c -o .b
>  chown user:user .b chmod 4750 .b rm bd.c
>  
>  ./.b 
Meaning that eliminating setfcap gives the container no additional security,
just breaks things.
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.14 (GNU/Linux)
> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>
> iEYEARECAAYFAlJIDIEACgkQrlYvE4MpobPJXgCgiENLuXzXzp0Mjukbb5L9DR2q
> ItgAn3pUJ15qATkVQEgUy2SuHqpGNX8y
> =pPRa
> -----END PGP SIGNATURE-----

[Matthew Miller]

On Fri, Sep 27, 2013 at 07:33:28PM +0000, "Jóhann B. Guðmundsson" wrote:
>  I dont have any security degrees nor do I consider myself an evil
>  man and probably Steve and Dan would be better suited to answer this
>  question since I'm far from being any expert on the subject but
>  hypothetically would not someone being able to do something like
>  this in this educational sample I'm providing 
So, to cut out the code, what you're saying is that someone could use this
to create a binary which executes as effective root. This is true, but a)
one is actually running as root inside the container anyway and b) one can
just use full setuid. Additionally, this wouldn't let someone _not_ root in
the container set filesystem capabilities.

-- 
Matthew Miller           mattdm(a)mattdm.org          <http://mattdm.org/>

[Jóhann B. Guðmundsson]

On 10/01/2013 01:27 PM, Matthew Miller wrote:
>  On Fri, Sep 27, 2013 at 07:33:28PM +0000, "Jóhann B.
> Guðmundsson" wrote:
> > I dont have any security degrees nor do I consider myself an evil
> > man and probably Steve and Dan would be better suited to answer this
> > question since I'm far from being any expert on the subject but
> > hypothetically would not someone being able to do something like
> > this in this educational sample I'm providing
>  So, to cut out the code, what you're saying is that someone could use this
>  to create a binary which executes as effective root. This is true, but a)
>  one is actually running as root inside the container anyway and b) one can
>  just use full setuid. Additionally, this wouldn't let someone _not_ root in
>  the container set filesystem capabilities.
>  
Actually the code I posted creates backdoor to give an user who runs it 
the ability to gain root privileges via setcap ( setcap cap_setuid=ep 
.b  ).

I intentionally left out the part how you gain superuser, big 
capabilities, etc to insert it in the first place ( let's not give nsa 
any more bright ideas )

JBG

[Matthew Miller]

On Tue, Oct 01, 2013 at 02:41:53PM +0000, "Jóhann B. Guðmundsson" wrote:
>  Actually the code I posted creates backdoor to give an user who runs
>  it the ability to gain root privileges via setcap ( setcap
>  cap_setuid=ep .b  ). 
Right, but the key is that you _already have_ root privileges in the
container.

However, certain capabilities have been dropped from the _permitted_ set;
once dropped, you can't get them back even by execing a binary with
filesystem capabilities set. Therefore, it seems fairly harmless to allow
them to be set (eg don't drop that particular capability) -- unless I'm
missing something.


-- 
Matthew Miller           mattdm(a)mattdm.org          <http://mattdm.org/>

[Daniel P. Berrange]

On Fri, Sep 27, 2013 at 01:28:29PM -0400, Matthew Miller wrote:
>  Quick backstory: unless run in privledged mode, Docker drops a bunch
> of
>  capabilities when launching a container. One of these is setfcap. This
>  breaks of binary RPMs like httpd where the daemon is installed with file
>  capabilities instead.
>  
>  We're considering removing setfcap from the list of dropped capabilities. It
>  seems safe to me (note that you run as root inside the container), but I'd
>  like some security-minded review. Could this be used for evil?
>  
>  https://bugzilla.redhat.com/show_bug.cgi?id=1012952 
Docker with the its sf.net LXC backend does not utilize any kind of
MAC driver, nor does it utilizer user namespaces, so even with those
capabilities dropped it is still insecure if the container app runs
as the 'root' user. As such allowing CAP_FCAP does not make the
situation worse AFAICT

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|

[Daniel J Walsh]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 09/30/2013 04:23 AM, Daniel P. Berrange wrote:
>  On Fri, Sep 27, 2013 at 01:28:29PM -0400, Matthew Miller wrote:
> > Quick backstory: unless run in privledged mode, Docker drops a bunch of 
> > capabilities when launching a container. One of these is setfcap. This 
> > breaks of binary RPMs like httpd where the daemon is installed with file 
> > capabilities instead.
> > 
> > We're considering removing setfcap from the list of dropped capabilities.
> > It seems safe to me (note that you run as root inside the container), but
> > I'd like some security-minded review. Could this be used for evil?
> > 
> > https://bugzilla.redhat.com/show_bug.cgi?id=1012952
>  
>  Docker with the its sf.net LXC backend does not utilize any kind of MAC
>  driver, nor does it utilizer user namespaces, so even with those 
>  capabilities dropped it is still insecure if the container app runs as the
>  'root' user. As such allowing CAP_FCAP does not make the situation worse
>  AFAICT
>  
>  Regards, Daniel
>   
Yes lets eliminate the idea that running as root within a container without
something like SELinux or User Namespace, is going to be much more secure then
running processes as root outside the container.

I plan on working on adding SELinux to wrap the docker container as we have
done for the virt-sandbox containers, but we still allow a lot of privs to a
privledged process within the container.
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.14 (GNU/Linux)
> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>
> iEYEARECAAYFAlJJbFAACgkQrlYvE4MpobOW2QCfceDBC39gAGkOICNe8NJz2/Ov
> RrgAoJfN6ci+gg8qLvqGTdh32e9szbI7
> =sbRH
> -----END PGP SIGNATURE-----

[Daniel P. Berrange]

On Mon, Sep 30, 2013 at 08:19:28AM -0400, Daniel J Walsh wrote:
>  -----BEGIN PGP SIGNED MESSAGE-----
>  Hash: SHA1
>  
>  On 09/30/2013 04:23 AM, Daniel P. Berrange wrote:
>  > On Fri, Sep 27, 2013 at 01:28:29PM -0400, Matthew Miller wrote:
>  >> Quick backstory: unless run in privledged mode, Docker drops a bunch of 
>  >> capabilities when launching a container. One of these is setfcap. This 
>  >> breaks of binary RPMs like httpd where the daemon is installed with file 
>  >> capabilities instead.
>  >> 
>  >> We're considering removing setfcap from the list of dropped capabilities.
>  >> It seems safe to me (note that you run as root inside the container), but
>  >> I'd like some security-minded review. Could this be used for evil?
>  >> 
>  >> https://bugzilla.redhat.com/show_bug.cgi?id=1012952
>  > 
>  > Docker with the its sf.net LXC backend does not utilize any kind of MAC
>  > driver, nor does it utilizer user namespaces, so even with those 
>  > capabilities dropped it is still insecure if the container app runs as the
>  > 'root' user. As such allowing CAP_FCAP does not make the situation worse
>  > AFAICT
>  > 
>  > Regards, Daniel
>  > 
>  Yes lets eliminate the idea that running as root within a container without
>  something like SELinux or User Namespace, is going to be much more secure then
>  running processes as root outside the container.
>  
>  I plan on working on adding SELinux to wrap the docker container as we have
>  done for the virt-sandbox containers, but we still allow a lot of privs to a
>  privledged process within the container. 
I have an RFE open to get user namespaces enabled in Fedora rawhide

  https://bugzilla.redhat.com/show_bug.cgi?id=917708

Both Libvirt & sf.net lxc have support for user namespaces, so the key
missing piece is getting Docker to make use of this & setup its filesystems
with suitable ownership. This would allow the DAC security model to fully
confine docker instances, even if SELinux is not enforcing it. Having both
DAC & MAC confinement for this is valuable long term.

Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|

[Matthew Miller]

On Mon, Sep 30, 2013 at 08:19:28AM -0400, Daniel J Walsh wrote:
>  I plan on working on adding SELinux to wrap the docker container as
> we have
>  done for the virt-sandbox containers, but we still allow a lot of privs to a
>  privledged process within the container. 
Do you have (or want?) a bugzilla entry to track that? Because I kind of see
that as a killer feature for our Docker implementation.

-- 
Matthew Miller  ☁☁☁  Fedora Cloud Architect  ☁☁☁  <mattdm(a)fedoraproject.org&gt;

[Daniel J Walsh]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 10/01/2013 09:28 AM, Matthew Miller wrote:
>  On Mon, Sep 30, 2013 at 08:19:28AM -0400, Daniel J Walsh wrote:
> > I plan on working on adding SELinux to wrap the docker container as we
> > have done for the virt-sandbox containers, but we still allow a lot of
> > privs to a privledged process within the container.
>  
>  Do you have (or want?) a bugzilla entry to track that? Because I kind of
>  see that as a killer feature for our Docker implementation.
>   
Sure open a bugzilla on it.
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.14 (GNU/Linux)
> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>
> iEYEARECAAYFAlJMKkgACgkQrlYvE4MpobNI7ACgsIv5rV5RTA/pwkzcOb0NMHwc
> xCsAmgLhzAxJIF3h1LInGDEVLO1uy1/F
> =2XvP
> -----END PGP SIGNATURE-----

[Matthew Miller]

On Wed, Oct 02, 2013 at 10:14:32AM -0400, Daniel J Walsh wrote:
>  > Do you have (or want?) a bugzilla entry to track that? Because I
> kind of
>  > see that as a killer feature for our Docker implementation.
>  Sure open a bugzilla on it. 
https://bugzilla.redhat.com/show_bug.cgi?id=1014707

-- 
Matthew Miller  ☁☁☁  Fedora Cloud Architect  ☁☁☁  <mattdm(a)fedoraproject.org&gt;

[Matthew Miller]

On Mon, Sep 30, 2013 at 08:19:28AM -0400, Daniel J Walsh wrote:
>  I plan on working on adding SELinux to wrap the docker container as
> we have
>  done for the virt-sandbox containers, but we still allow a lot of privs to a
>  privledged process within the container. 
Another question, probably a dumb one. Will this work with the lxc-tools
approach or just with libvirt-lxc?


-- 
Matthew Miller  ☁☁☁  Fedora Cloud Architect  ☁☁☁  <mattdm(a)fedoraproject.org&gt;

[Daniel J Walsh]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 10/04/2013 04:27 PM, Matthew Miller wrote:
>  On Mon, Sep 30, 2013 at 08:19:28AM -0400, Daniel J Walsh wrote:
> > I plan on working on adding SELinux to wrap the docker container as we
> > have done for the virt-sandbox containers, but we still allow a lot of
> > privs to a privledged process within the container.
>  
>  Another question, probably a dumb one. Will this work with the lxc-tools 
>  approach or just with libvirt-lxc?
>  
>   
We can work with it on the lxc version, but I am not sure if it will work easily.
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.14 (GNU/Linux)
> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>
> iEYEARECAAYFAlJPPjIACgkQrlYvE4MpobPn3wCgzB7ZbCdYsz3kZI+0pXPlue/a
> DeMAoMrZ5xYuRW3XPZebUA/2o/RM/Ezm
> =NRmo
> -----END PGP SIGNATURE-----

[Matthew Miller]

On Fri, Oct 04, 2013 at 06:16:18PM -0400, Daniel J Walsh wrote:
>  > Another question, probably a dumb one. Will this work with the
> lxc-tools 
>  > approach or just with libvirt-lxc?
>  We can work with it on the lxc version, but I am not sure if it will work
>  easily. 
But libvirt _does_ make it easy? Again, sorry if these are silly questions.
:)

-- 
Matthew Miller  ☁☁☁  Fedora Cloud Architect  ☁☁☁  <mattdm(a)fedoraproject.org&gt;

[Daniel P. Berrange]

On Fri, Oct 04, 2013 at 08:22:10PM -0400, Matthew Miller wrote:
>  On Fri, Oct 04, 2013 at 06:16:18PM -0400, Daniel J Walsh wrote:
>  > > Another question, probably a dumb one. Will this work with the lxc-tools 
>  > > approach or just with libvirt-lxc?
>  > We can work with it on the lxc version, but I am not sure if it will work
>  > easily.
>  
>  But libvirt _does_ make it easy? Again, sorry if these are silly questions.
>  :) 
I don't really know what you're asking here. Libvirt has long supported
SELinux confinement of guests though, and this is used by libvirt-sandbox

Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|

[Kashyap Chamarthy]

On 10/05/2013 05:52 AM, Matthew Miller wrote:
>  On Fri, Oct 04, 2013 at 06:16:18PM -0400, Daniel J Walsh wrote:
> >> Another question, probably a dumb one. Will this work with the lxc-tools 
> >> approach or just with libvirt-lxc?
> > We can work with it on the lxc version, but I am not sure if it will work
> > easily.
>  
>  But libvirt _does_ make it easy? Again, sorry if these are silly questions.
>  :) 
I haven't tried containers yet, as Dan stated libvirt has always supported confinement
of
guests via sVirt.

If you want to do a quick test:

  $ ls -lZ /var/lib/libvirt/images/fed18.qcow2
  -rw-r--r--. qemu qemu system_u:object_r:svirt_image_t:s0:c920,c980
  /var/lib/libvirt/images/fed18.qcow2

  $ ps -eZ | grep qemu
  system_u:system_r:svirt_t:s0:c920,c980 30017 ? 00:00:16 qemu-system-x86


As you can notice above, QEMU process and its associated disk image have a
*unique* SELinux label. So, even if the QEMU process is compromised, it cannot
spill over to other processes.


/kashyap

[Daniel J Walsh]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 10/04/2013 08:22 PM, Matthew Miller wrote:
>  On Fri, Oct 04, 2013 at 06:16:18PM -0400, Daniel J Walsh wrote:
> >> Another question, probably a dumb one. Will this work with the
> >> lxc-tools approach or just with libvirt-lxc?
> > We can work with it on the lxc version, but I am not sure if it will
> > work easily.
>  
>  But libvirt _does_ make it easy? Again, sorry if these are silly
>  questions. :)
>   
Yes libvirt makes it much easier because it is built in.  To get this to work
with lxc tool set we need to add a patch to lxc to launch that applications
that run within a container with a particular SELinux label.  This means we
need to patch docker to take an SELinux label or to pick a default, and then
pass it to lxc which will tell the kernel what label to launch.  We already do
this with the libvirt-sandbox tools, and libvirt-lxc does the setup and launch
with the correct label.
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.14 (GNU/Linux)
> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>
> iEYEARECAAYFAlJSoTMACgkQrlYvE4MpobPkrQCgvf9uYoTWHU0tBtdBG6mIshfI
> 6LsAn3GReKJ2DlHE+qmdtWQINXZpo+1E
> =vgoA
> -----END PGP SIGNATURE-----