[Bug 1046469] New: docker privileged mode with cmd /sbin/init - agetty & high cpu

[Red Hat Bugzilla]

https://bugzilla.redhat.com/show_bug.cgi?id=1046469

Bug ID: 1046469
           Summary: docker privileged mode with cmd /sbin/init - agetty &
                    high cpu
           Product: Fedora
           Version: 20
         Component: docker-io
          Assignee: [e-mail]
          Reporter: [e-mail]
        QA Contact: [e-mail]
                CC: [e-mail], [e-mail],
                    [e-mail], [e-mail],
                    [e-mail], [e-mail]

Description of problem:
docker containers in privileged mode mess with the host agetty and produce very
hicht cpu load

Version-Release number of selected component (if applicable):
Fedora 20
Docker 0.7.1, 0.7.2

How reproducible:
docker run -privileged -d mattdm/fedora:latest /sbin/init

Actual results:
docker container runs
cpu load is very high - agetty
stopping the container & the docker daemon do not free the cpu resources
one have to kill the agetty process

Expected results:
docker container runs

Additional info:
In my real case i run rhel containers with puppet & ssh daemons for a small
puppet lab.

The reason i met this bug is that running docker container with /sbin/init cmd
under RHEL 6.5 require priviliged mode.

[Bug 1046469] docker privileged mode with cmd /sbin/init - agetty & high cpu

[Red Hat Bugzilla]

https://bugzilla.redhat.com/show_bug.cgi?id=1046469

Daniel Walsh  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |CLOSED
         Resolution|---                         |CURRENTRELEASE
        Last Closed|                            |2014-05-28 13:55:23

--- Comment #1 from Daniel Walsh  ---
If you try this with the latest code does it still happen.  Killing a container
should stop all processes.

Closing as fixed in current release.  Reopen if it still happens.

[Bug 1046469] docker privileged mode with cmd /sbin/init - agetty & high cpu

[Red Hat Bugzilla]

https://bugzilla.redhat.com/show_bug.cgi?id=1046469

[e-mail] changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |[e-mail]

--- Comment #2 from [e-mail] ---
Still see it in RHEL7.

To reproduce, you need to start two Fedora based container in privileged mode.
If you start only one in privileged mode, there will be no problem.

Dockerfile

FROM centos:centos7
RUN yum -y swap -- remove fakesystemd -- install systemd systemd-libs
CMD ["/usr/sbin/init"]

Script

docker build -t test .
docker run -d --privileged test
docker run -d --privileged test

[Bug 1046469] docker privileged mode with cmd /sbin/init - agetty & high cpu

[Red Hat Bugzilla]

https://bugzilla.redhat.com/show_bug.cgi?id=1046469

Joe Mario  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|CLOSED                      |POST
                 CC|                            |[e-mail]
         Resolution|CURRENTRELEASE              |---
           Keywords|                            |Reopened

--- Comment #3 from Joe Mario  ---
> Closing as fixed in current release.  Reopen if it still happens.

Re-opening.  It's still happening with a RHEL7.2 host, a RHEL7.2 container and
a docker-1.8.2-2.el7.  The Dockerfile contains systemd, if that matters.

Steps to reproduce:
1) Create two containers using:
   docker run -d -v /sys/fs/cgroup:/sys/fs/cgroup:ro --privileged test
/sbin/init
   docker run -d -v /sys/fs/cgroup:/sys/fs/cgroup:ro --privileged test
/sbin/init

2) Start two containers:
   docker exec -it <container-1-id> bash
   docker exec -it <container-2-id> bash

3) Run top on the bare metal host.  You should see an agetty process burning up
one cpu:

PID   USER  PR  NI    VIRT  RES  SHR  S  %CPU %MEM  TIME+    COMMAND
73529   root  20   0  110004  680  660  R  99.7  0.0  10:53.98  agetty

Note, this problem does not occur if, when creating the containers, I replace
the "--privileged" flag with "--security-opt label:disable --cap-add SYS_ADMIN"

Joe

[Bug 1046469] docker privileged mode with cmd /sbin/init - agetty & high cpu

[Red Hat Bugzilla]

https://bugzilla.redhat.com/show_bug.cgi?id=1046469

Daniel Walsh  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |[e-mail]

--- Comment #4 from Daniel Walsh  ---
Not sure this can be fixed.  systemd would have to figure out it is running in
a container and then do something different.

[Bug 1046469] docker privileged mode with cmd /sbin/init - agetty & high cpu

[Red Hat Bugzilla]

https://bugzilla.redhat.com/show_bug.cgi?id=1046469

Daniel Walsh  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |[e-mail],
                   |                            |[e-mail],
                   |                            |[e-mail],
                   |                            |[e-mail],
                   |                            |[e-mail],
                   |                            |[e-mail]
          Component|docker-io                   |docker
            Version|20                          |22

[Bug 1046469] docker privileged mode with cmd /sbin/init - agetty & high cpu

[Red Hat Bugzilla]

https://bugzilla.redhat.com/show_bug.cgi?id=1046469

--- Comment #5 from Daniel Walsh  ---
Potentially being caused by multiple udevs running?

[Bug 1046469] docker privileged mode with cmd /sbin/init - agetty & high cpu

[Red Hat Bugzilla]

https://bugzilla.redhat.com/show_bug.cgi?id=1046469

Jeremy Eder  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |[e-mail]

[Bug 1046469] docker privileged mode with cmd /sbin/init - agetty & high cpu

[Red Hat Bugzilla]

https://bugzilla.redhat.com/show_bug.cgi?id=1046469

Jeremy Eder  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
      Comment #7 is|1                           |0
            private|                            |
      Comment #6 is|1                           |0
            private|                            |

--- Comment #8 from Jeremy Eder  ---
No

...
read(0, "", 1)                          = 0
read(0, "", 1)                          = 0
read(0, "", 1)                          = 0
read(0, "", 1)                          = 0
...

[Bug 1046469] docker privileged mode with cmd /sbin/init - agetty & high cpu

[Red Hat Bugzilla]

https://bugzilla.redhat.com/show_bug.cgi?id=1046469

--- Comment #9 from Joe Mario  ---
Here's a little more info, but likely doesn't help.  (It didn't help me).
But I share it anyways.

The agetty process continuously loops through calls from main (agetty.c:372) to
get_logname().

(gdb) bac
#0  get_logname (cp=<synthetic pointer>, tp=0x7fff062f3520, op=0x7fff062f3870)
at term-utils/agetty.c:1553
#1  main (argc=<optimized out>, argv=<optimized out>) at
term-utils/agetty.c:372

The line read at line 1533 is:
Breakpoint 1, get_logname (cp=<synthetic pointer>, tp=0x7fff062f3520,
op=0x7fff062f3870) at term-utils/agetty.c:1533
1533                if (read(STDIN_FILENO, &c, 1) < 1) {

After it completes, errno is not set, and the value stored into the variable
"c" is:
(gdb) p c
$15 = 3 '\003'

After a bunch of checks, it gets down to the switch stmt at line 1574:
1573                /* Do erase, kill and end-of-line processing. */
1574                switch (key) {

Unfortunately the value of "key" is optimized away.

It falls through to the "default:" case at line 1604, and then executes line
1605:
1602                case CTL('D'):
1603                    exit(EXIT_SUCCESS);
1604                default:
1605                    if (!isascii(ascval) || !isprint(ascval))
1606                        break;

On the call to isprint(), we get to the read() of 1 byte from __fd=0:
(gdb) bac
#0  0x0000000000401fd0 in read@plt ()
#1  0x0000000000403752 in read (__nbytes=1, __buf=0x7fff062f3510, __fd=0) at
/usr/include/bits/unistd.h:44
#2  get_logname (cp=<synthetic pointer>, tp=0x7fff062f3520, op=0x7fff062f3870)
at term-utils/agetty.c:1533
#3  main (argc=<optimized out>, argv=<optimized out>) at
term-utils/agetty.c:372

Unfortunately, even though I'm stepping by instruction, in the optimized
binary, I can't stop between the isprint() and the upper level read() at line
1533 (where we started above).

Joe

[Bug 1046469] docker privileged mode with cmd /sbin/init - agetty & high cpu

[Red Hat Bugzilla]

https://bugzilla.redhat.com/show_bug.cgi?id=1046469

--- Comment #10 from Joe Mario  ---
One more note, wrt to the earlier comment about multiple udevs running:

During the moment that the two privileged containers are started, the number of
systemd-udevd processes jumps from 2 to 146.  After about 5 seconds that number
drops to 4.

If the same two container invocations are done again, but this time without
--privileged, then only 15 systemd-udevd processes are created (again for about
4-5 seconds).

[Bug 1046469] docker privileged mode with cmd /sbin/init - agetty & high cpu

[Red Hat Bugzilla]

https://bugzilla.redhat.com/show_bug.cgi?id=1046469

--- Comment #11 from Joe Mario  ---
Here's one workaround for this.  Sharing what I've learned.

Adding the following to the docker file

rm -f /lib/systemd/system/systemd*udev* ; \
   rm -f /lib/systemd/system/getty.target;

causes both the runnaway agetty and the spike in systemd-udevd processes to go
away.  I understand there is no need for udev or getty in containers.

I did also try:
   RUN systemctl disable getty.target
   RUN systemctl disable systemd-udevd.service

and although the Dockerfile built fine, all the getty and udev services were
still running as they were previously.  So those two "systemctl disable" appear
to be no-ops.  Perhaps they get run before /sbin/init (systemd) is invoked.

[Bug 1046469] docker privileged mode with cmd /sbin/init - agetty & high cpu

[Red Hat Bugzilla]

https://bugzilla.redhat.com/show_bug.cgi?id=1046469

Sat6QE Jenkins  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|POST                        |MODIFIED

[Bug 1046469] docker privileged mode with cmd /sbin/init - agetty & high cpu

[Red Hat Bugzilla]

https://bugzilla.redhat.com/show_bug.cgi?id=1046469

Mike McCune  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|MODIFIED                    |POST

--- Comment #14 from Mike McCune  ---
This bug was accidentally moved from POST to MODIFIED via an error in
automation, please see [e-mail] with any questions

[Bug 1046469] docker privileged mode with cmd /sbin/ init - agetty & high cpu

[Red Hat Bugzilla]

https://bugzilla.redhat.com/show_bug.cgi?id=1046469

Fedora Admin XMLRPC Client  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
           Assignee|[e-mail]             |[e-mail]

--- Comment #15 from Fedora Admin XMLRPC Client  ---
This package has changed ownership in the Fedora Package Database.  Reassigning
to the new owner of this component.