[Bug 1186669] New: Failed to start Docker Application Container Engine.

[Red Hat Bugzilla]

https://bugzilla.redhat.com/show_bug.cgi?id=1186669

Bug ID: 1186669
           Summary: Failed to start Docker Application Container Engine.
           Product: Fedora
           Version: 21
         Component: docker-io
          Severity: high
          Assignee: [e-mail]
          Reporter: [e-mail]
        QA Contact: [e-mail]
                CC: [e-mail], [e-mail],
                    [e-mail], [e-mail],
                    [e-mail], [e-mail],
                    [e-mail], [e-mail],
                    [e-mail], [e-mail], [e-mail],
                    [e-mail], [e-mail]

Created attachment 985059
  --> https://bugzilla.redhat.com/attachment.cgi?id=985059&action=edit
grep -i 'docker' /var/log/audit/audit.log | tail

Description of problem:

I yum-updated to F21 latest and now docker service fails to start on boot.
This update included reboot to new kernel 3.18

[root@vm117 ~]# uname -a
Linux vm117 3.18.3-201.fc21.x86_64 #1 SMP Mon Jan 19 15:59:31 UTC 2015 x86_64
x86_64 x86_64 GNU/Linux

Maybe this could be related to the other new warning that I see in 
joutnalctl -b

SELinux is preventing sm-notify from write access on the file nlm_end_grace.
For complete SELinux messages. run sealert

and
Dependency failed for Network Manager Wait Online.

Which I assume is new with 3.18 since web search found this:
https://www.kernel.org/pub/linux/utils/nfs-utils/1.3.1/1.3.1-Changelog

I don't see anything notable in /var/log/messages

Will attach a grep of /var/log/audit/audit.log

Version-Release number of selected component (if applicable):

[root@vm117 ~]# yum info docker-io.x86_64
Loaded plugins: langpacks
Installed Packages
Name        : docker-io
Arch        : x86_64
Version     : 1.4.1
Release     : 5.fc21
Size        : 19 M
Repo        : installed
> From repo   : updates

How reproducible:

100% Service Fails to start on every boot.

Steps to Reproduce:
1. Reboot Fedora
2. systemctl status docker.service
3.

Actual results:

Expected results:

Additional info:

[Bug 1186669] Failed to start Docker Application Container Engine.

[Red Hat Bugzilla]

https://bugzilla.redhat.com/show_bug.cgi?id=1186669

colin  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |bugzilla.redhat.com@trancec
                   |                            |ode.co.uk

--- Comment #1 from colin  ---

I yum updated to latest 
docker-io.x86_64 0:1.4.1-8.fc21

(and updated SELinux for RHBZ1181338) 
see: https://bugzilla.redhat.com/show_bug.cgi?id=1181338

But docker still fails to start on boot.

Is there anything that I can do to investigate this issue further?

thanks
Colin.

[Bug 1186669] Failed to start Docker Application Container Engine.

[Red Hat Bugzilla]

https://bugzilla.redhat.com/show_bug.cgi?id=1186669

Lokesh Mandvekar  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |ASSIGNED

--- Comment #2 from Lokesh Mandvekar  ---
Hi Colin,

The docker rpm doesn't use systemd socket activation anymore.

Please check if your /etc/sysconfig/docker has '-H fd://' included as part of
OPTIONS. If yes, please remove it and restart docker. If it still fails, please
post here what 'systemctl status docker.service' spits out.

Also, docker group has been gotten rid of in the defaults. But you can still
manually create docker group and add users to it if you prefer to avoid sudo
(though it'd be a security risk).

[Bug 1186669] Failed to start Docker Application Container Engine.

[Red Hat Bugzilla]

https://bugzilla.redhat.com/show_bug.cgi?id=1186669

Lokesh Mandvekar  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
           Severity|high                        |unspecified

--- Comment #3 from Lokesh Mandvekar  ---
Colin, does this work for you now?

[Bug 1186669] Failed to start Docker Application Container Engine.

[Red Hat Bugzilla]

https://bugzilla.redhat.com/show_bug.cgi?id=1186669

--- Comment #4 from colin  ---
Hi Lokesh.

So I did a yum-update and rebooted the VM

upon login to the (SPICE) RemoteViewer MATE desktop
I see 2 SELinux Alerts for openvswitch.

[root@vm117 yum]# docker images 
FATA[0000] Cannot connect to the Docker daemon. Is 'docker -d' running on this
host?

[root@vm117 yum]# systemctl status -l docker.service
● docker.service - Docker Application Container Engine
   Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled)
   Active: failed (Result: timeout) since Mon 2015-02-23 09:38:27 GMT; 22min
ago
     Docs: http://docs.docker.com
  Process: 1612 ExecStart=/usr/bin/docker -d $OPTIONS $DOCKER_STORAGE_OPTIONS
$DOCKER_NETWORK_OPTIONS $INSECURE_REGISTRY (code=exited, status=0/SUCCESS)
 Main PID: 1612 (code=exited, status=0/SUCCESS)

Feb 23 09:36:59 vm117 docker[1612]: time="2015-02-23T09:36:59Z" level="info"
msg="+job serveapi(unix:///var/run/docker.sock)"
Feb 23 09:36:59 vm117 docker[1612]: time="2015-02-23T09:36:59Z" level="info"
msg="Listening for HTTP on unix (/var/run/docker.sock)"
Feb 23 09:38:04 vm117 python[2253]: SELinux is preventing docker from getattr
access on the directory /run/openvswitch.

*****  Plugin catchall (100. confidence)
suggests   **************************

If you believe that docker should be
allowed getattr access on the openvswitch directory by default.
                                    Then you should report this as a bug.
                                    You can generate a local policy module to
allow this access.
                                    Do
                                    allow this access for now by executing:
                                    # grep docker /var/log/audit/audit.log |
audit2allow -M mypol
                                    # semodule -i mypol.pp

Feb 23 09:38:04 vm117 python[2253]: SELinux is preventing docker from getattr
access on the sock_file /run/openvswitch/db.sock.

*****  Plugin catchall (100. confidence)
suggests   **************************

If you believe that docker should be
allowed getattr access on the db.sock sock_file by default.
                                    Then you should report this as a bug.
                                    You can generate a local policy module to
allow this access.
                                    Do
                                    allow this access for now by executing:
                                    # grep docker /var/log/audit/audit.log |
audit2allow -M mypol
                                    # semodule -i mypol.pp

Feb 23 09:38:17 vm117 python[2253]: SELinux is preventing docker from getattr
access on the directory /run/openvswitch.

*****  Plugin catchall (100. confidence)
suggests   **************************

If you believe that docker should be
allowed getattr access on the openvswitch directory by default.
                                    Then you should report this as a bug.
                                    You can generate a local policy module to
allow this access.
                                    Do
                                    allow this access for now by executing:
                                    # grep docker /var/log/audit/audit.log |
audit2allow -M mypol
                                    # semodule -i mypol.pp

Feb 23 09:38:27 vm117 systemd[1]: docker.service start operation timed out.
Terminating.
Feb 23 09:38:27 vm117 docker[1612]: time="2015-02-23T09:38:27Z" level="info"
msg="Received signal 'terminated', starting shutdown of docker..."
Feb 23 09:38:27 vm117 systemd[1]: Failed to start Docker Application Container
Engine.
Feb 23 09:38:27 vm117 systemd[1]: Unit docker.service entered failed state.
Feb 23 09:38:27 vm117 systemd[1]: docker.service failed.

I do a manual docker start now and this does work.

[root@vm117 yum]# systemctl stop docker.service
[root@vm117 yum]# systemctl start docker.service
[root@vm117 yum]# systemctl status -l docker.service
● docker.service - Docker Application Container Engine
   Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled)
   Active: active (running) since Mon 2015-02-23 10:03:28 GMT; 5s ago
     Docs: http://docs.docker.com
 Main PID: 2550 (docker)
   CGroup: /system.slice/docker.service
           └─2550 /usr/bin/docker -d --selinux-enabled

Feb 23 10:03:27 vm117 python[2507]: SELinux is preventing docker from getattr
access on the sock_file /run/openvswitch/db.sock.

*****  Plugin catchall (100. confidence)
suggests   **************************

If you believe that docker should be
allowed getattr access on the db.sock sock_file by default.
                                    Then you should report this as a bug.
                                    You can generate a local policy module to
allow this access.
                                    Do
                                    allow this access for now by executing:
                                    # grep docker /var/log/audit/audit.log |
audit2allow -M mypol
                                    # semodule -i mypol.pp

Feb 23 10:03:27 vm117 docker[2550]: time="2015-02-23T10:03:27Z" level="info"
msg="-job init_networkdriver() = OK (0)"
Feb 23 10:03:27 vm117 docker[2550]: time="2015-02-23T10:03:27Z" level="info"
msg="Loading containers: start."
Feb 23 10:03:28 vm117 docker[2550]: ........time="2015-02-23T10:03:28Z"
level="error" msg="Warning: error unmounting device
66c4719685e9ca574429ebed6c16816efd3bba70b9cb19dd81209003a6f8d7b5:
UnmountDevice: device not-mounted id
66c4719685e9ca574429ebed6c16816efd3bba70b9cb19dd81209003a6f8d7b5"
Feb 23 10:03:28 vm117 docker[2550]: .time="2015-02-23T10:03:28Z" level="error"
msg="Warning: error unmounting device
7d0bef932cf04222a6417685a64f8ed87b1af9a2ce962d4c7439937a671f00fd:
UnmountDevice: device not-mounted id
7d0bef932cf04222a6417685a64f8ed87b1af9a2ce962d4c7439937a671f00fd"
Feb 23 10:03:28 vm117 docker[2550]: .....time="2015-02-23T10:03:28Z"
level="error" msg="Warning: error unmounting device
d3123de348b17df04fc08607bf826ff1d18b3b4fe2c31f521e846749f2624c2f:
UnmountDevice: device not-mounted id
d3123de348b17df04fc08607bf826ff1d18b3b4fe2c31f521e846749f2624c2f"
Feb 23 10:03:28 vm117 docker[2550]: time="2015-02-23T10:03:28Z" level="info"
msg="Loading containers: done."
Feb 23 10:03:28 vm117 docker[2550]: time="2015-02-23T10:03:28Z" level="info"
msg="docker daemon: 1.5.0 a8a31ef/1.5.0; execdriver: native-0.2; graphdriver:
devicemapper"
Feb 23 10:03:28 vm117 docker[2550]: time="2015-02-23T10:03:28Z" level="info"
msg="+job acceptconnections()"
Feb 23 10:03:28 vm117 docker[2550]: time="2015-02-23T10:03:28Z" level="info"
msg="-job acceptconnections() = OK (0)"
[root@vm117 yum]#

[Bug 1186669] Failed to start Docker Application Container Engine.

[Red Hat Bugzilla]

https://bugzilla.redhat.com/show_bug.cgi?id=1186669

--- Comment #5 from colin  ---
Re: SELinux alert from Comment #4
https://bugzilla.redhat.com/show_bug.cgi?id=1195159

[Bug 1186669] Failed to start Docker Application Container Engine.

[Red Hat Bugzilla]

https://bugzilla.redhat.com/show_bug.cgi?id=1186669

--- Comment #6 from colin  ---
Re: comment #2
> Please check if your /etc/sysconfig/docker has '-H fd://' included as part of > OPTIONS. If yes, please remove it and restart docker. If it still fails, 
> please post here what 'systemctl status docker.service' spits out.

[root@vm117 yum]# cat /etc/sysconfig/docker
OPTIONS=--selinux-enabled
DOCKER_CERT_PATH=/etc/docker

# Location used for temporary files, such as those created by
# docker load and build operations. Default is /var/lib/docker/tmp
# Can be overriden by setting the following environment variable.
# DOCKER_TMPDIR=/var/tmp
[root@vm117 yum]#

[Bug 1186669] Failed to start Docker Application Container Engine.

[Red Hat Bugzilla]

https://bugzilla.redhat.com/show_bug.cgi?id=1186669

Daniel Walsh  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |[e-mail]

--- Comment #7 from Daniel Walsh  ---
0a70c020b334a2fd2c0f92a3528087ba5bbcae59 fixes this in the selinux-policy git

We need to get this backported to f22 and f21.

[Bug 1186669] Failed to start Docker Application Container Engine.

[Red Hat Bugzilla]

https://bugzilla.redhat.com/show_bug.cgi?id=1186669

Daniel Walsh  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |[e-mail],
                   |                            |[e-mail],
                   |                            |[e-mail],
                   |                            |[e-mail]
          Component|docker-io                   |selinux-policy
           Assignee|[e-mail]             |[e-mail]

[Bug 1186669] Failed to start Docker Application Container Engine.

[Red Hat Bugzilla]

https://bugzilla.redhat.com/show_bug.cgi?id=1186669

Lukas Vrabec  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|ASSIGNED                    |MODIFIED

--- Comment #8 from Lukas Vrabec  ---
commit bfed9799e203a94b0c7d0ebd19ec7f9844c8b675
Author: Dan Walsh 
Date:   Mon Mar 9 14:43:56 2015 -0400

Allow docker to relablefrom/to sockets and docker_log_t

Added in F22 and F21

[Bug 1186669] Failed to start Docker Application Container Engine.

[Red Hat Bugzilla]

https://bugzilla.redhat.com/show_bug.cgi?id=1186669

Lukas Vrabec  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
   Fixed In Version|                            |selinux-policy-3.13.1-105.8
                   |                            |.fc21

[Bug 1186669] Failed to start Docker Application Container Engine.

[Red Hat Bugzilla]

https://bugzilla.redhat.com/show_bug.cgi?id=1186669

--- Comment #9 from Fedora Update System  ---
selinux-policy-3.13.1-105.9.fc21 has been submitted as an update for Fedora 21.
https://admin.fedoraproject.org/updates/selinux-policy-3.13.1-105.9.fc21

[Bug 1186669] Failed to start Docker Application Container Engine.

[Red Hat Bugzilla]

https://bugzilla.redhat.com/show_bug.cgi?id=1186669

--- Comment #10 from colin  ---
Hi Lukas

So I went ahead and downloaded

selinux-policy-3.13.1-105.9.fc21.noarch.rpm (info) (download)
    selinux-policy-devel-3.13.1-105.9.fc21.noarch.rpm (info) (download)
    selinux-policy-doc-3.13.1-105.9.fc21.noarch.rpm (info) (download)
    selinux-policy-minimum-3.13.1-105.9.fc21.noarch.rpm (info) (download)
    selinux-policy-mls-3.13.1-105.9.fc21.noarch.rpm (info) (download)
    selinux-policy-sandbox-3.13.1-105.9.fc21.noarch.rpm (info) (download)
    selinux-policy-targeted-3.13.1-105.9.fc21.noarch.rpm (info) (download)

from your Koji test build and installed them with:
'yum localinstall selinux-policy-* | tee
yum_localinstall_koji-selinux-policy-3.13.1-105.9'

There were 3 complaints from yum so I will upload the log as attachment.

I rebooted and am happy to report that this time No New AVC warnings have been
reported. :-)

I can now manually start docker and run my scripts that start openvpn and
openvswitch without causing SELinux warnings.
Brilliant!

(The only minor niggle is that docker.service still won't start from boot- but
I want to move onto a fresh VM now so will wait for all your changes to land
for F22)

Thankyou for the support and fixes. :-)
Colin

[Bug 1186669] Failed to start Docker Application Container Engine.

[Red Hat Bugzilla]

https://bugzilla.redhat.com/show_bug.cgi?id=1186669

--- Comment #11 from colin  ---
Created attachment 1005909
  --> https://bugzilla.redhat.com/attachment.cgi?id=1005909&action=edit
'yum localinstall selinux-policy-* | tee
yum_localinstall_koji-selinux-policy-3.13.1-105.9'

installation log for manual install of koji test build

http://koji.fedoraproject.org/koji/buildinfo?buildID=622719

selinux-policy-3.13.1-105.9.fc21

https://admin.fedoraproject.org/updates/selinux-policy-3.13.1-105.9.fc21?_cs...

works for me. :-)
+1 krama left

[Bug 1186669] Failed to start Docker Application Container Engine.

[Red Hat Bugzilla]

https://bugzilla.redhat.com/show_bug.cgi?id=1186669

Fedora Update System  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|MODIFIED                    |ON_QA

--- Comment #12 from Fedora Update System  ---
Package selinux-policy-3.13.1-105.9.fc21:
* should fix your issue,
* was pushed to the Fedora 21 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing
selinux-policy-3.13.1-105.9.fc21'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2015-4492/selinux-policy-3.13...
then log in and leave karma (feedback).

[Bug 1186669] Failed to start Docker Application Container Engine.

[Red Hat Bugzilla]

https://bugzilla.redhat.com/show_bug.cgi?id=1186669

Fedora Update System  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|ON_QA                       |CLOSED
   Fixed In Version|selinux-policy-3.13.1-105.8 |selinux-policy-3.13.1-105.9
                   |.fc21                       |.fc21
         Resolution|---                         |ERRATA
        Last Closed|                            |2015-03-31 17:46:27

--- Comment #13 from Fedora Update System  ---
selinux-policy-3.13.1-105.9.fc21 has been pushed to the Fedora 21 stable
repository.  If problems still persist, please make note of it in this bug
report.