New subject: [Bug 1096816] useradd within EL6 container fails: failure while writing changes to /etc/passwd

12 May 2014


      https://bugzilla.redhat.com/show_bug.cgi?id=1096816
Bug ID: 1096816
           Summary: useradd within EL6 container fails: failure while
                    writing changes to /etc/passwd
           Product: Red Hat Enterprise Linux 6
           Version: 6.5
         Component: libselinux
          Assignee: dwalsh@redhat.com
          Reporter: dwalsh@redhat.com
        QA Contact: qe-baseos-security@redhat.com
                CC: admiller@redhat.com, dcleal@redhat.com,
                    dwalsh@redhat.com, golang@lists.fedoraproject.org,
                    jpazdziora@redhat.com, jumanjiman@gmail.com,
                    lsm5@redhat.com, mattdm@redhat.com,
                    mgoldman@redhat.com, s@shk.io, vbatts@redhat.com
        Depends On: 1096123
+++ This bug was initially created as a clone of Bug #1096123 +++
Description of problem:
Between docker-io-0.10.0-2.fc20 and docker-io-0.11.1-1.fc20, the following has
started failing:
$ docker run -t centos /usr/sbin/useradd test
useradd: failure while writing changes to /etc/passwd
'centos' is the official CentOS 6 image (0b443ba03958).
The Fedora 20 host has SELinux enforcing, and the same issue occurs when set to
permissive.  No AVCs are seen.
Version-Release number of selected component (if applicable):
docker-io-0.11.1-1.fc20.x86_64
kernel-3.14.2-200.fc20.x86_64
How reproducible:
Always
Steps to Reproduce:
1. docker pull centos
2. docker run -t centos /usr/sbin/useradd test
Actual results:
useradd: failure while writing changes to /etc/passwd
Expected results:
no output
Additional info:
On 0.10.0, an strace of useradd shows:
open("/etc/group", O_RDONLY|O_CLOEXEC)  = 11
fstat(11, {st_mode=S_IFREG|0644, st_size=379, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) =
0x7feb5efe5000
read(11, "root:x:0:\nbin:x:1:bin,daemon\ndae"..., 4096) = 379
close(11)                               = 0
munmap(0x7feb5efe5000, 4096)            = 0
fchown(10, 500, 12)                     = 0
fchmod(10, 0660)                        = 0
fsync(10)                               = 0
close(10)                               = 0
fstat(6, {st_mode=S_IFREG|0644, st_size=670, ...}) = 0
gettid()                                = 14
open("/proc/self/task/14/attr/fscreate", O_RDONLY) = 10
read(10, "", 4095)                      = 0
close(10)                               = 0
gettid()                                = 14
open("/proc/self/task/14/attr/fscreate", O_RDWR) = 10
write(10, "system_u:object_r:file_t:s0\0", 28) = 28
close(10)                               = 0
fstat(6, {st_mode=S_IFREG|0644, st_size=670, ...}) = 0
umask(077)                              = 022
open("/etc/passwd-", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 10
umask(022)                              = 077
lseek(6, 0, SEEK_SET)                   = 0
read(6, "root:x:0:0:root:/root:/bin/bash\n"..., 4096) = 670
fstat(10, {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) =
0x7feb5efe5000
read(6, "", 4096)                       = 0
write(10, "root:x:0:0:root:/root:/bin/bash\n"..., 670) = 670
While on 0.11.1, strace shows:
open("/etc/group", O_RDONLY|O_CLOEXEC)  = 10
fstat(10, {st_mode=S_IFREG|0644, st_size=379, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) =
0x7f2905a38000
read(10, "root:x:0:\nbin:x:1:bin,daemon\ndae"..., 4096) = 379
close(10)                               = 0
munmap(0x7f2905a38000, 4096)            = 0
fchown(9, 500, 12)                      = 0
fchmod(9, 0660)                         = 0
fsync(9)                                = 0
close(9)                                = 0
fstat(5, {st_mode=S_IFREG|0644, st_size=675, ...}) = 0
gettid()                                = 30
open("/proc/self/task/30/attr/fscreate", O_RDONLY) = 9
read(9, "", 4095)                       = 0
close(9)                                = 0
gettid()                                = 30
open("/proc/self/task/30/attr/fscreate", O_RDWR) = -1 EROFS (Read-only file
system)
write(2, "useradd: failure while writing c"..., 54useradd: failure while
writing changes to /etc/passwd
) = 54
--- Additional comment from Dominic Cleal on 2014-05-09 07:13:07 EDT ---
useradd is just calling libselinux's setfscreatecon, which is being blocked. 
On 0.11.1, this library call fails, on 0.10.0/0.9 it works.
--- Additional comment from Paul Morgan on 2014-05-09 09:55:15 EDT ---
when selinux=disabled, i cannot reproduce the bug.
when selinux is enabled (permissive), the bug is always reproducible.
i reproduced using two docker service configs:
* first, use default systemd unit
* second, add `--selinux-enabled` as described at
  http://blog.docker.io/2014/05/docker-0-11-release-candidate-for-1-0/
override default systemd unit...
# cp /usr/lib/systemd/system/docker.service /etc/systemd/system/
# vim /etc/systemd/system/docker.service
# cat !$ 
[Unit]
Description=Docker Application Container Engine
Documentation=http://docs.docker.io
After=network.target
[Service]
ExecStart=/usr/bin/docker -d --selinux-enabled
Restart=on-failure
LimitNOFILE=1048576
LimitNPROC=1048576
[Install]
WantedBy=multi-user.target
# systemctl restart docker.service 
# systemctl status docker.service 
docker.service - Docker Application Container Engine
   Loaded: loaded (/etc/systemd/system/docker.service; enabled)
   Active: active (running) since Fri 2014-05-09 09:52:25 EDT; 4s ago
     Docs: http://docs.docker.io
 Main PID: 997 (docker)
   CGroup: /system.slice/docker.service
           └─997 /usr/bin/docker -d --selinux-enabled
other system info:
$ rpm -q selinux-policy-targeted
selinux-policy-targeted-3.12.1-158.fc20.noarch
$ uname -a
Linux f20-01.example.com 3.13.9-200.fc20.x86_64 #1 SMP Fri Apr 4 12:13:05 UTC
2014 x86_64 x86_64 x86_64 GNU/Linux
$ rpm -q docker-io
docker-io-0.11.1-1.fc20.x86_64
$ docker version
Client version: 0.11.1
Client API version: 1.11
Go version (client): go1.2.1
Git commit (client): fb99f99/0.11.1
Server version: 0.11.1
Server API version: 1.11
Git commit (server): fb99f99/0.11.1
Go version (server): go1.2.1
Last stable version: 0.11.1
$ docker images | grep '^centos'
centos                centos6             0b443ba03958        3 weeks ago      
  297.6 MB
centos                latest              0b443ba03958        3 weeks ago      
  297.6 MB
centos                6.4                 539c0211cd76        13 months ago    
  300.6 MB
--- Additional comment from Lokesh Mandvekar on 2014-05-09 13:41:51 EDT ---
I have a new scratch build
http://kojipkgs.fedoraproject.org//work/tasks/2230/6832230/docker-io-0.11.1-...
with this build, this error doesn't occur with fedora:20, but still does with
centos
dwalsh, comments?
--- Additional comment from Dominic Cleal on 2014-05-09 13:45:57 EDT ---
Comparing the straces between el6 and fedora:20, I don't see any of the same
accesses to attr/fscreate on f20 that are in the bug description.  The source
of shadow-utils between el6 & f20 looks very different too, I can't see any
setfscreatecon calls in the useradd code path.
--- Additional comment from Daniel Walsh on 2014-05-09 16:51:06 EDT ---
The problem is inside the container it sees SELinux as being enabled, which is
the bug.
If you do id -Z, does it complain inside the container?
docker run --rm -t -i fedora sh
sh-4.2# id -Z
id: --context (-Z) works only on an SELinux-enabled kernel
sh-4.2# mount | grep /sys
sysfs on /sys type sysfs (ro,relatime,seclabel)
SELinux sees the container as being disabled since /sys/fs/selinux is mounted
as read/only, this will tell useradd NOT to try to do any SELinux stuff while
in the container.
--- Additional comment from Fedora Update System on 2014-05-10 00:04:23 EDT ---
docker-io-0.11.1-3.fc20 has been submitted as an update for Fedora 20.
https://admin.fedoraproject.org/updates/docker-io-0.11.1-3.fc20
--- Additional comment from Dominic Cleal on 2014-05-10 08:01:14 EDT ---
(In reply to Daniel Walsh from comment #5)
...
The problem is inside the container it sees SELinux as being enabled, which
is the bug.
If you do id -Z, does it complain inside the container?
No, it runs and reports a context.
...
docker run --rm -t -i fedora sh
sh-4.2# id -Z
id: --context (-Z) works only on an SELinux-enabled kernel
sh-4.2# mount | grep /sys
sysfs on /sys type sysfs (ro,relatime,seclabel)
$ rpm -q docker-io
docker-io-0.9.1-1.fc20.x86_64
$ docker run -i -t centos /bin/bash
bash-4.1# id -Z
system_u:system_r:docker_t:s0
bash-4.1# mount | grep sys
sysfs on /sys type sysfs (rw,seclabel,nosuid,nodev,noexec,relatime)
$ rpm -q docker-io
docker-io-0.11.1-3.fc20.x86_64
$ docker run -i -t centos /bin/bash
bash-4.1# id -Z
system_u:system_r:svirt_lxc_net_t:s0:c231,c400
bash-4.1# mount | grep /sys
sysfs on /sys type sysfs (ro,seclabel,relatime)
...
SELinux sees the container as being disabled since /sys/fs/selinux is
mounted as read/only, this will tell useradd NOT to try to do any SELinux
stuff while in the container.
/sys is correctly read-only as you expected, but it seems useradd's still doing
SELinux stuff then.  These packages are installed inside the EL6 container:
libselinux-2.0.94-5.3.el6_4.1.x86_64
libselinux-utils-2.0.94-5.3.el6_4.1.x86_64
shadow-utils-4.1.4.2-13.el6.x86_64
Calling is_selinux_enabled() on Fedora is returning 0, while on EL6 it's
returning 1.  Another difference - on Fedora, getenforce returns "Disabled" but
on EL6 it prints:
# getenforce
getenforce:  getenforce() failedbash-4.1#
/selinux exists within the container, but nothing is actually mounted there. 
It appears to be simply a directory on the root filesystem (/selinux/booleans
exists as an empty dir).  No other SELinux mounts are visible.
Looking at libselinux-2.0.94, I think it's seeing selinuxfs listed in
/proc/filesystems and assuming SELinux is enabled because of this. 
libselinux-2.2.1 on F20 doesn't seem to have this code.
libselinux-2.0.94/src/enabled.c:
        /* Drop back to detecting it the long way. */
        fp = fopen("/proc/filesystems", "r");
        if (!fp)
                return -1;
__fsetlocking(fp, FSETLOCKING_BYCALLER);
        while ((num = getline(&buf, &len, fp)) != -1) {
                if (strstr(buf, "selinuxfs")) {
                        enabled = 1;
                        break;
                }
        }
# grep selinux /proc/filesystems 
nodev    selinuxfs
(All the above was tested with docker-io-0.11.1-3.fc20)
--- Additional comment from Fedora Update System on 2014-05-12 01:28:00 EDT ---
Package docker-io-0.11.1-3.fc20:
* should fix your issue,
* was pushed to the Fedora 20 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing docker-io-0.11.1-3.fc20'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2014-6281/docker-io-0.11.1-3....
then log in and leave karma (feedback).
--- Additional comment from Dominic Cleal on 2014-05-12 06:34:11 EDT ---
(In reply to Dominic Cleal from comment #7)
...
Looking at libselinux-2.0.94, I think it's seeing selinuxfs listed in
/proc/filesystems and assuming SELinux is enabled because of this. 
libselinux-2.2.1 on F20 doesn't seem to have this code.
Bug #835146 (against EL6) seems to confirm this, suggesting a backport of the
patch that removes the /proc/filesystems based check.
Referenced Bugs:
https://bugzilla.redhat.com/show_bug.cgi?id=1096123
[Bug 1096123] useradd within EL6 container fails: failure while writing
changes to /etc/passwd
-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug 1096816] New: useradd within EL6 container fails: failure while writing changes to /etc/passwd