https://bugzilla.redhat.com/show_bug.cgi?id=1096273
Bug ID: 1096273 Summary: repetition of signal 21 with --sig-proxy and SIGSTOP of docker Product: Red Hat Enterprise Linux 7 Version: 7.1 Component: docker Severity: low Assignee: lsm5@redhat.com Reporter: ldoktor@redhat.com QA Contact: virt-bugs@redhat.com CC: admiller@redhat.com, golang@lists.fedoraproject.org, lsm5@redhat.com, mattdm@redhat.com, mgoldman@redhat.com, skottler@redhat.com, vbatts@redhat.com Depends On: 1087704
+++ This bug was initially created as a clone of Bug #1087704 +++ Description of problem: I noticed weird reaction to SIGSTOP signal. When you start container with --sig-proxy, ignore all signals in container and from OS use kill -SIGSTOP to the `docker run` process, it's put into background. (so far correct behavior) Than when you use kill -SIGCONT, it's resumed (still in background) and it's continuing to send signal 21 to the container (100% CPU utilization). This disappears once you attach the process with fg.
Version-Release number of selected component (if applicable): docker-0.10.0-8.el7.x86_64 docker-io-0.9.1-1.fc21.x86_64
How reproducible: always
Steps to Reproduce: 1. /usr/bin/docker -D run --tty=false --rm -i --name test_eoly localhost:5000/ldoktor/fedora:latest bash -c 'for NUM in `seq 1 64`; do trap "echo Received $NUM, ignoring..." $NUM; done; while :; do sleep 1; done' 2. ps ax |grep docker 3. kill -SIGSTOP $PID 4. kill -SIGCONT $PID
Actual results: [1]+ Pozastavena /usr/bin/docker -D run --tty=false --rm -i --name test_eoly localhost:5000/ldoktor/fedora:latest bash -c 'for NUM in `seq 1 64`; do trap "echo Received $NUM, ignoring..." $NUM; done; while :; do sleep 1; done' [root@t530 ~]# Received 21, ignoring... Received 21, ignoring... Received 21, ignoring... Received 21, ignoring... Received 21, ignoring... Received 21, ignoring... ... (100% CPU utilization, when you use fg it stops and you can use the container again...)
Expected results: [1]+ Pozastavena /usr/bin/docker -D run --tty=false --rm -i --name test_eoly localhost:5000/ldoktor/fedora:latest bash -c 'for NUM in `seq 1 64`; do trap "echo Received $NUM, ignoring..." $NUM; done; while :; do sleep 1; done' [root@t530 ~]# Received 21, ignoring...
Additional info: I'm not definitely sure if this is really required as it's not a common usage. But as there is no way to detach the process apart from kill -SIGSTOP people might use it and get to this problem...
--- Additional comment from Lukas Doktor on 2014-05-05 03:59:30 EDT ---
I retested this with the upstream dc9c28f/0.10.0 with the same results. The daemon output is: ... [/home/medic/Work/Projekty/Docker/root|fa3816b6] -job kill(b01a849cb45ebe94c3a61fa021a5464186345d5b159faee4ea9d5da39fb36de5, TTIN) = OK (0) 2014/05/05 09:58:13 POST /v1.10/containers/b01a849cb45ebe94c3a61fa021a5464186345d5b159faee4ea9d5da39fb36de5/kill?signal=TTIN [/home/medic/Work/Projekty/Docker/root|fa3816b6] +job kill(b01a849cb45ebe94c3a61fa021a5464186345d5b159faee4ea9d5da39fb36de5, TTIN) [/home/medic/Work/Projekty/Docker/root|fa3816b6] -job kill(b01a849cb45ebe94c3a61fa021a5464186345d5b159faee4ea9d5da39fb36de5, TTIN) = OK (0) 2014/05/05 09:58:13 POST /v1.10/containers/b01a849cb45ebe94c3a61fa021a5464186345d5b159faee4ea9d5da39fb36de5/kill?signal=TTIN [/home/medic/Work/Projekty/Docker/root|fa3816b6] +job kill(b01a849cb45ebe94c3a61fa021a5464186345d5b159faee4ea9d5da39fb36de5, TTIN) [/home/medic/Work/Projekty/Docker/root|fa3816b6] -job kill(b01a849cb45ebe94c3a61fa021a5464186345d5b159faee4ea9d5da39fb36de5, TTIN) = OK (0) 2014/05/05 09:58:13 POST /v1.10/containers/b01a849cb45ebe94c3a61fa021a5464186345d5b159faee4ea9d5da39fb36de5/kill?signal=TTIN [/home/medic/Work/Projekty/Docker/root|fa3816b6] +job kill(b01a849cb45ebe94c3a61fa021a5464186345d5b159faee4ea9d5da39fb36de5, TTIN) ...
Referenced Bugs:
https://bugzilla.redhat.com/show_bug.cgi?id=1087704 [Bug 1087704] repetition of signal 21 with --sig-proxy and SIGSTOP of docker
https://bugzilla.redhat.com/show_bug.cgi?id=1096273
--- Comment #1 from Lukas Doktor ldoktor@redhat.com --- There is a slight difference between fedora and RHEL behavior in this situation.
[RHEL results] Actual results: [1]+ Pozastavena /usr/bin/docker -D run --tty=false --rm -i --name test_eoly localhost:5000/ldoktor/fedora:latest bash -c 'for NUM in `seq 1 64`; do trap "echo Received $NUM, ignoring..." $NUM; done; while :; do sleep 1; done' [root@t530 eclipse_links 147]$ [debug] stdcopy.go:111 framesize: 25 Received 21, ignoring... [debug] stdcopy.go:111 framesize: 25 Received 21, ignoring...
No new lines, no output.
Output of docker daemon: [31257f3e] -job kill(d09b358f4e33b1bb1e49d7fa6e3b7c52fe618a2d36a8424b0cd4697e009f7688, TTIN) = OK (0) 2014/05/09 16:35:38 POST /v1.11/containers/d09b358f4e33b1bb1e49d7fa6e3b7c52fe618a2d36a8424b0cd4697e009f7688/kill?signal=TTIN [31257f3e] +job kill(d09b358f4e33b1bb1e49d7fa6e3b7c52fe618a2d36a8424b0cd4697e009f7688, TTIN) [31257f3e] -job kill(d09b358f4e33b1bb1e49d7fa6e3b7c52fe618a2d36a8424b0cd4697e009f7688, TTIN) = OK (0) 2014/05/09 16:35:38 http: Accept error: accept unix /var/run/docker.sock: too many open files; retrying in 5ms 2014/05/09 16:35:38 POST /v1.11/containers/d09b358f4e33b1bb1e49d7fa6e3b7c52fe618a2d36a8424b0cd4697e009f7688/kill?signal=TTIN [31257f3e] +job kill(d09b358f4e33b1bb1e49d7fa6e3b7c52fe618a2d36a8424b0cd4697e009f7688, TTIN) [31257f3e] -job kill(d09b358f4e33b1bb1e49d7fa6e3b7c52fe618a2d36a8424b0cd4697e009f7688, TTIN) = OK (0) 2014/05/09 16:35:38 http: Accept error: accept unix /var/run/docker.sock: too many open files; retrying in 10ms 2014/05/09 16:35:38 http: Accept error: accept unix /var/run/docker.sock: too many open files; retrying in 20ms 2014/05/09 16:35:38 http: Accept error: accept unix /var/run/docker.sock: too many open files; retrying in 40ms 2014/05/09 16:35:38 http: Accept error: accept unix /var/run/docker.sock: too many open files; retrying in 80ms 2014/05/09 16:35:38 http: Accept error: accept unix /var/run/docker.sock: too many open files; retrying in 160ms 2014/05/09 16:35:38 http: Accept error: accept unix /var/run/docker.sock: too many open files; retrying in 320ms 2014/05/09 16:35:39 http: Accept error: accept unix /var/run/docker.sock: too many open files; retrying in 640ms 2014/05/09 16:35:39 http: Accept error: accept unix /var/run/docker.sock: too many open files; retrying in 1s 2014/05/09 16:35:40 http: Accept error: accept unix /var/run/docker.sock: too many open files; retrying in 1s 2014/05/09 16:35:41 http: Accept error: accept unix /var/run/docker.sock: too many open files; retrying in 1s 2014/05/09 16:35:42 http: Accept error: accept unix /var/run/docker.sock: too many open files; retrying in 1s ...
docker stops printing the "Received signal..." message. I'm able to use fg to join, but the container is not responding. When I press ctrl+c in docker daemon it prints: ^C2014/05/09 16:41:26 Received signal 'interrupt', starting shutdown of docker... ^C2014/05/09 16:41:27 Received signal 'interrupt', starting shutdown of docker... 2014/05/09 16:41:27 http: Accept error: accept unix /var/run/docker.sock: too many open files; retrying in 1s ^C2014/05/09 16:41:27 Received signal 'interrupt', starting shutdown of docker...
after another ctrl+c: 2014/05/09 16:41:27 Force shutdown of docker, interrupting cleanup
and the process finishes.
When I execute the daemon again, it's unable to remove the containers: docker rm output: Error: Cannot destroy container d09b358f4e33: Driver devicemapper failed to remove root filesystem d09b358f4e33b1bb1e49d7fa6e3b7c52fe618a2d36a8424b0cd4697e009f7688: Error running removeDevice 6de66f25bd2a Error: Cannot destroy container 7a57a8a67336: Driver devicemapper failed to remove root filesystem 7a57a8a6733621bf0b043f4d121221f20853256764fd0762ec49f7f595e87ac7: Error running removeDevice 2014/05/09 16:44:29 Error: failed to remove one or more containers
Docker daemon output: 2014/05/09 16:44:07 DELETE /v1.11/containers/d09b358f4e33?force=1&v=1 [e6f019e8] +job container_delete(d09b358f4e33) Cannot destroy container d09b358f4e33: Driver devicemapper failed to remove root filesystem d09b358f4e33b1bb1e49d7fa6e3b7c52fe618a2d36a8424b0cd4697e009f7688: Error running removeDevice [e6f019e8] -job container_delete(d09b358f4e33) = ERR (1) [error] server.go:1014 Error: Cannot destroy container d09b358f4e33: Driver devicemapper failed to remove root filesystem d09b358f4e33b1bb1e49d7fa6e3b7c52fe618a2d36a8424b0cd4697e009f7688: Error running removeDevice [error] server.go:90 HTTP Error: statusCode=500 Cannot destroy container d09b358f4e33: Driver devicemapper failed to remove root filesystem d09b358f4e33b1bb1e49d7fa6e3b7c52fe618a2d36a8424b0cd4697e009f7688: Error running removeDevice 2014/05/09 16:44:17 DELETE /v1.11/containers/6de66f25bd2a?force=1&v=1 [e6f019e8] +job container_delete(6de66f25bd2a) [e6f019e8] -job container_delete(6de66f25bd2a) = OK (0) 2014/05/09 16:44:18 DELETE /v1.11/containers/7a57a8a67336?force=1&v=1 [e6f019e8] +job container_delete(7a57a8a67336) Cannot destroy container 7a57a8a67336: Driver devicemapper failed to remove root filesystem 7a57a8a6733621bf0b043f4d121221f20853256764fd0762ec49f7f595e87ac7: Error running removeDevice [e6f019e8] -job container_delete(7a57a8a67336) = ERR (1) [error] server.go:1014 Error: Cannot destroy container 7a57a8a67336: Driver devicemapper failed to remove root filesystem 7a57a8a6733621bf0b043f4d121221f20853256764fd0762ec49f7f595e87ac7: Error running removeDevice [error] server.go:90 HTTP Error: statusCode=500 Cannot destroy container 7a57a8a67336: Driver devicemapper failed to remove root filesystem 7a57a8a6733621bf0b043f4d121221f20853256764fd0762ec49f7f595e87ac7: Error running removeDevice
mount output: ... /dev/mapper/docker-253:3-4065163-7a57a8a6733621bf0b043f4d121221f20853256764fd0762ec49f7f595e87ac7 on /home/medic/Work/Projekty/Docker/root/devicemapper/mnt/7a57a8a6733621bf0b043f4d121221f20853256764fd0762ec49f7f595e87ac7 type ext4 (rw,relatime,seclabel,discard,stripe=16,data=ordered) /dev/mapper/docker-253:3-4065163-d09b358f4e33b1bb1e49d7fa6e3b7c52fe618a2d36a8424b0cd4697e009f7688 on /home/medic/Work/Projekty/Docker/root/devicemapper/mnt/d09b358f4e33b1bb1e49d7fa6e3b7c52fe618a2d36a8424b0cd4697e009f7688 type ext4 (rw,relatime,seclabel,discard,stripe=16,data=ordered)
docker ps -a output: CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
When I try to create the container again it fails with: [root@t530 eclipse_links 2]$ /usr/bin/docker -D run --tty=false --rm -i --name test_eoly localhost:5000/ldoktor/fedora:latest bash -c 'for NUM in `seq 1 64`; do trap "echo Received $NUM, ignoring..." $NUM; done; while :; do sleep 1; done' [debug] hijack.go:96 [hijack] End of stdout [debug] commands.go:1958 End of CmdRun(), Waiting for hijack to finish. 2014/05/09 16:46:06 Error: Cannot start container 399677e64c6b8414d39bf316b99f667cb26c76b2d87266124998726acda391d5: Cannot find child for /test_eoly
Docker daemon output: 2014/05/09 16:46:06 POST /v1.11/containers/create?name=test_eoly [e6f019e8] +job create(test_eoly) -----> setting mount label discard [e6f019e8] -job create(test_eoly) = OK (0) 2014/05/09 16:46:06 POST /v1.11/containers/399677e64c6b8414d39bf316b99f667cb26c76b2d87266124998726acda391d5/attach?stderr=1&stdin=1&stdout=1&stream=1 [e6f019e8] +job inspect(399677e64c6b8414d39bf316b99f667cb26c76b2d87266124998726acda391d5, container) [e6f019e8] -job inspect(399677e64c6b8414d39bf316b99f667cb26c76b2d87266124998726acda391d5, container) = OK (0) [e6f019e8] +job attach(399677e64c6b8414d39bf316b99f667cb26c76b2d87266124998726acda391d5) 2014/05/09 16:46:06 POST /v1.11/containers/399677e64c6b8414d39bf316b99f667cb26c76b2d87266124998726acda391d5/start [e6f019e8] +job start(399677e64c6b8414d39bf316b99f667cb26c76b2d87266124998726acda391d5) -----> setting mount label discard [e6f019e8] +job allocate_interface(399677e64c6b8414d39bf316b99f667cb26c76b2d87266124998726acda391d5) [e6f019e8] -job allocate_interface(399677e64c6b8414d39bf316b99f667cb26c76b2d87266124998726acda391d5) = OK (0) [e6f019e8] +job release_interface(399677e64c6b8414d39bf316b99f667cb26c76b2d87266124998726acda391d5) [e6f019e8] -job release_interface(399677e64c6b8414d39bf316b99f667cb26c76b2d87266124998726acda391d5) = OK (0) [e6f019e8] -job attach(399677e64c6b8414d39bf316b99f667cb26c76b2d87266124998726acda391d5) = OK (0) Cannot start container 399677e64c6b8414d39bf316b99f667cb26c76b2d87266124998726acda391d5: Cannot find child for /test_eoly [e6f019e8] -job start(399677e64c6b8414d39bf316b99f667cb26c76b2d87266124998726acda391d5) = ERR (1) [error] server.go:1014 Error: Cannot start container 399677e64c6b8414d39bf316b99f667cb26c76b2d87266124998726acda391d5: Cannot find child for /test_eoly [error] server.go:90 HTTP Error: statusCode=500 Cannot start container 399677e64c6b8414d39bf316b99f667cb26c76b2d87266124998726acda391d5: Cannot find child for /test_eoly
When I repeat the exact same command once again (without restarting the daemon) it passes: 2014/05/09 16:48:33 POST /v1.11/containers/create?name=test_eoly [e6f019e8] +job create(test_eoly) -----> setting mount label discard [e6f019e8] -job create(test_eoly) = OK (0) 2014/05/09 16:48:33 POST /v1.11/containers/4aff2c82307724735a2a1c3b8723022b01642142836f9fd57c244f494eb5e234/attach?stderr=1&stdin=1&stdout=1&stream=1 [e6f019e8] +job inspect(4aff2c82307724735a2a1c3b8723022b01642142836f9fd57c244f494eb5e234, container) [e6f019e8] -job inspect(4aff2c82307724735a2a1c3b8723022b01642142836f9fd57c244f494eb5e234, container) = OK (0) [e6f019e8] +job attach(4aff2c82307724735a2a1c3b8723022b01642142836f9fd57c244f494eb5e234) 2014/05/09 16:48:33 POST /v1.11/containers/4aff2c82307724735a2a1c3b8723022b01642142836f9fd57c244f494eb5e234/start [e6f019e8] +job start(4aff2c82307724735a2a1c3b8723022b01642142836f9fd57c244f494eb5e234) -----> setting mount label discard [e6f019e8] +job allocate_interface(4aff2c82307724735a2a1c3b8723022b01642142836f9fd57c244f494eb5e234) [e6f019e8] -job allocate_interface(4aff2c82307724735a2a1c3b8723022b01642142836f9fd57c244f494eb5e234) = OK (0) [e6f019e8] -job start(4aff2c82307724735a2a1c3b8723022b01642142836f9fd57c244f494eb5e234) = OK (0)
https://bugzilla.redhat.com/show_bug.cgi?id=1096273
Chris Evich cevich@redhat.com changed:
What |Removed |Added ---------------------------------------------------------------------------- CC| |cevich@redhat.com Version|7.1 |7.0 Severity|low |high
https://bugzilla.redhat.com/show_bug.cgi?id=1096273
Daniel Walsh dwalsh@redhat.com changed:
What |Removed |Added ---------------------------------------------------------------------------- CC| |dwalsh@redhat.com
--- Comment #4 from Daniel Walsh dwalsh@redhat.com --- I am not seeing this in the current 0.11 package
https://bugzilla.redhat.com/show_bug.cgi?id=1096273
--- Comment #5 from Lukas Doktor ldoktor@redhat.com --- I retested this with the latest docker-0.11.1-2.el7.x86_64 on RHEL7 with the same results as in the bugzilla:
[root@localhost client]# docker -D run --tty=false --rm -i --name test_eoly fedora bash -c 'for NUM in `seq 1 64`; do trap "echo Received $NUM, ignoring..." $NUM; done; while :; do sleep 1; done'
[1]+ Pozastavena docker -D run --tty=false --rm -i --name test_eoly fedora bash -c 'for NUM in `seq 1 64`; do trap "echo Received $NUM, ignoring..." $NUM; done; while :; do sleep 1; done' [root@localhost client]# [debug] stdcopy.go:111 framesize: 50 Received 21, ignoring... Received 21, ignoring... [debug] stdcopy.go:111 framesize: 25 Received 21, ignoring... [debug] stdcopy.go:111 framesize: 25 Received 21, ignoring... [debug] stdcopy.go:111 framesize: 25 ... # I used fg+enter
fg docker -D run --tty=false --rm -i --name test_eoly fedora bash -c 'for NUM in `seq 1 64`; do trap "echo Received $NUM, ignoring..." $NUM; done; while :; do sleep 1; done' [debug] stdcopy.go:111 framesize: 25 Received 21, ignoring...
Docker daemon output: ... [debug] server.go:981 Calling POST /containers/{name:.*}/kill 2014/05/20 07:16:53 POST /v1.11/containers/90e0ed8723e8b0a8ddfdc8376b7d26465894bb5c4f19cca7fdae1bdf130871f3/kill?signal=TTIN [a8b38a24] +job kill(90e0ed8723e8b0a8ddfdc8376b7d26465894bb5c4f19cca7fdae1bdf130871f3, TTIN) [a8b38a24] -job kill(90e0ed8723e8b0a8ddfdc8376b7d26465894bb5c4f19cca7fdae1bdf130871f3, TTIN) = OK (0) [debug] server.go:981 Calling POST /containers/{name:.*}/kill 2014/05/20 07:16:53 POST /v1.11/containers/90e0ed8723e8b0a8ddfdc8376b7d26465894bb5c4f19cca7fdae1bdf130871f3/kill?signal=TTIN [a8b38a24] +job kill(90e0ed8723e8b0a8ddfdc8376b7d26465894bb5c4f19cca7fdae1bdf130871f3, TTIN) [a8b38a24] -job kill(90e0ed8723e8b0a8ddfdc8376b7d26465894bb5c4f19cca7fdae1bdf130871f3, TTIN) = OK (0) [debug] server.go:981 Calling POST /containers/{name:.*}/kill 2014/05/20 07:16:53 POST /v1.11/containers/90e0ed8723e8b0a8ddfdc8376b7d26465894bb5c4f19cca7fdae1bdf130871f3/kill?signal=TTIN [a8b38a24] +job kill(90e0ed8723e8b0a8ddfdc8376b7d26465894bb5c4f19cca7fdae1bdf130871f3, TTIN) [a8b38a24] -job kill(90e0ed8723e8b0a8ddfdc8376b7d26465894bb5c4f19cca7fdae1bdf130871f3, TTIN) = OK (0)
https://bugzilla.redhat.com/show_bug.cgi?id=1096273
Daniel Walsh dwalsh@redhat.com changed:
What |Removed |Added ---------------------------------------------------------------------------- Assignee|lsm5@redhat.com |mheon@redhat.com
https://bugzilla.redhat.com/show_bug.cgi?id=1096273
--- Comment #6 from Matthew Heon mheon@redhat.com --- So, you're not actually forwarding the signal to the container with --sig-proxy.
Doing a ps -ax | grep docker does not actually find the PID you use to forward signals to the container using --sig-proxy. Docker furnishes the PID of the container itself, via the docker inspect command (docker inspect $CONTAINER_ID | grep Pid). The given PID will not match the one found via ps.
I believe you're stopping and restarting some internal Docker process responsible for managing the container. This is not something that is intended to happen, or something we should support - I imagine Apache would also start throwing errors if we began sending signals to its forked processes.
(For reference, I can reproduce perfectly with your instructions. However, by changing ps -ax to docker inspect, the issue no longer occurs)
This could likely use some documentation to make it clear that ps is not a supported method of finding a Docker container's PID (at least, with any degree of reliability). I'll investigate existing docs to see if they address this.
https://bugzilla.redhat.com/show_bug.cgi?id=1096273
Daniel Walsh dwalsh@redhat.com changed:
What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |ASSIGNED
--- Comment #7 from Daniel Walsh dwalsh@redhat.com --- Should we close this bug, or do we need to update documentation?
https://bugzilla.redhat.com/show_bug.cgi?id=1096273
--- Comment #8 from Matthew Heon mheon@redhat.com --- I think this does need to be documented somewhere, though I'm not sure where best to do it - it doesn't clearly fit into any of the manpages. Might be something for a Docker user's guide, if we have one?
https://bugzilla.redhat.com/show_bug.cgi?id=1096273
--- Comment #9 from Matthew Heon mheon@redhat.com --- It appears I was confused about how to use SIGPROXY - apologies, your original report was valid.
I can reproduce this on RHEL7 with -SIGSTOP and -SIGCONT, but not with -SIGSTOP and fg.
This mainly has to do with the fact that SIGSTOP is not proxied through --sig-proxy, I believe. SIGSTOP is unmaskable, so the Golang signal handler can't catch it - the foreground "docker run" process pauses, the background container does not.
Documentation should be updated to note that Signal 19 is not proxied. Additionally, I'm attempting to diagnose the issue with Signal 21 being repeated forever on resume.
https://bugzilla.redhat.com/show_bug.cgi?id=1096273
--- Comment #10 from Daniel Walsh dwalsh@redhat.com --- Is this fixed in docker-1.0.0-10.el7.x86_64?
https://bugzilla.redhat.com/show_bug.cgi?id=1096273
--- Comment #11 from Matthew Heon mheon@redhat.com --- No, it is not. I can only reproduce this intermittently - sometimes sending SIGCONT to a container stopped with SIGSTOP does nothing at all (the container does not even resume). I'll look into this more.
https://bugzilla.redhat.com/show_bug.cgi?id=1096273
--- Comment #12 from Lukas Doktor ldoktor@redhat.com --- That's weird. I can reproduce this always on booth, RHEL and Fedora. Today I tried it with the latest docker-io-1.0.0-6.fc20.x86_64 and the results are:
1) docker -D run --tty=false --rm -i --name test_eoly localhost:5000/ldoktor/fedora:latest bash -c 'for NUM in `seq 1 64`; do trap "echo Received $NUM, ignoring..." $NUM; done; while :; do sleep 1; done' 2) TERM2: ps -ax |grep docker 3) TERM2: kill -SIGSTOP 10197 generates output in TERM1: [1]+ Pozastavena docker -D run --tty=false --rm -i --name test_eoly localhost:5000/ldoktor/fedora:latest bash -c 'for NUM in `seq 1 64`; do trap "echo Received $NUM, ignoring..." $NUM; done; while :; do sleep 1; done'
4) TERM2: kill -SIGCONT 10197 generates output in TERM1: [medic@t530 ~ 147]$ fg[debug] stdcopy.go:111 framesize: 25 Received 21, ignoring... [debug] commands.go:556 Error sending signal: Post http:///var/run/docker.sock/v1.12/containers/673424bd1e6e95ae83641e20bdc1806...: dial unix /var/run/docker.sock: too many open files [debug] commands.go:556 Error sending signal: Post http:///var/run/docker.sock/v1.12/containers/673424bd1e6e95ae83641e20bdc1806...: dial unix /var/run/docker.sock: too many open files [debug] commands.go:556 Error sending signal: Post http:///var/run/docker.sock/v1.12/containers/673424bd1e6e95ae83641e20bdc1806...: dial unix /var/run/docker.sock: too many open files [debug] commands.go:556 Error sending signal: Post http:///var/run/docker.sock/v1.12/containers/673424bd1e6e95ae83641e20bdc1806...: dial unix /var/run/docker.sock: too many open files [debug] commands.go:556 Error sending signal: Post http:///var/run/docker.sock/v1.12/containers/673424bd1e6e95ae83641e20bdc1806...: dial unix /var/run/docker.sock: too many open files [debug] commands.go:556 Error sending signal: Post http:///var/run/docker.sock/v1.12/containers/673424bd1e6e95ae83641e20bdc1806...: dial unix /var/run/docker.sock: too many open files [debug] commands.go:556 Error sending signal: Post http:///var/run/docker.sock/v1.12/containers/673424bd1e6e95ae83641e20bdc1806...: dial unix /var/run/docker.sock: too many open files [debug] commands.go:556 Error sending signal: Post http:///var/run/docker.sock/v1.12/containers/673424bd1e6e95ae83641e20bdc1806...: dial unix /var/run/docker.sock: too many open files ...
until I execute fg (in TERM1): docker -D run --tty=false --rm -i --name test_eoly localhost:5000/ldoktor/fedora:latest bash -c 'for NUM in `seq 1 64`; do trap "echo Received $NUM, ignoring..." $NUM; done; while :; do sleep 1; done' [debug] commands.go:556 Error sending signal: Post http:///var/run/docker.sock/v1.12/containers/673424bd1e6e95ae83641e20bdc1806...: dial unix /var/run/docker.sock: too many open files [debug] stdcopy.go:111 framesize: 25 Received 21, ignoring...
and waits. When I pres ctrl+c it complains about too many open files: ^C[debug] commands.go:556 Error sending signal: Post http:///var/run/docker.sock/v1.12/containers/673424bd1e6e95ae83641e20bdc1806...: dial unix /var/run/docker.sock: too many open files
The only way to kill it is `kill -9 ...`
For fedora tests I'm executing docker in xterm, for RHEL I ssh from Fedora xterm to virtual RHEL7 kvm guest. Today I tried that with gnome-terminal just to be sure it's not related to some xterm specific behavior and it works 100% of time.
https://bugzilla.redhat.com/show_bug.cgi?id=1096273
--- Comment #13 from Matthew Heon mheon@redhat.com --- I was actually running a build of the latest upstream Docker when attempting to reproduce, which appears to be the source of the inconsistency.
The good news is that this build doesn't express this bug fully - SIG21 is sent only 3 times, and there's no warning of too many open files.
The bad news is that 100% CPU utilization still persists despite this.
I think this likely has to do with SIGSTOP not being proxied (SIGSTOP and SIGKILL can't be caught, so the signal proxy can't forward them - they apply to the docker client process, not the container. I'll write up a patch to clarify this in the docs). It seems like something is timing out on the server during this, and the client is attempting to reconnect.
Worth noting that, if you send both signals near-simultaneously (kill -SIGSTOP $PID; kill -SIGCONT $PID) this does not happen - definitely seems to be a timeout.
I'll investigate further.
https://bugzilla.redhat.com/show_bug.cgi?id=1096273
--- Comment #14 from Daniel Walsh dwalsh@redhat.com --- BTW
Docker pause and docker unpause use CGroups to "FREEZE" containers.
https://bugzilla.redhat.com/show_bug.cgi?id=1096273
--- Comment #15 from Matthew Heon mheon@redhat.com --- Appears to be an issue with Golang proper (https://code.google.com/p/go/issues/detail?id=4391&q=signal&colspec=...)
Unknown when a fix will be released - there hasn't been any movement on the bug since February.
https://bugzilla.redhat.com/show_bug.cgi?id=1096273
Daniel Walsh dwalsh@redhat.com changed:
What |Removed |Added ---------------------------------------------------------------------------- Component|docker |golang QA Contact|virt-bugs@redhat.com |qe-baseos-tools@redhat.com
https://bugzilla.redhat.com/show_bug.cgi?id=1096273
John Skeoch jskeoch@redhat.com changed:
What |Removed |Added ---------------------------------------------------------------------------- Assignee|mheon@redhat.com |jrieden@redhat.com
https://bugzilla.redhat.com/show_bug.cgi?id=1096273 Bug 1096273 depends on bug 1087704, which changed state.
Bug 1087704 Summary: repetition of signal 21 with --sig-proxy and SIGSTOP of docker https://bugzilla.redhat.com/show_bug.cgi?id=1087704
What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |CLOSED Resolution|--- |CANTFIX
https://bugzilla.redhat.com/show_bug.cgi?id=1096273
Min Wang minwang@redhat.com changed:
What |Removed |Added ---------------------------------------------------------------------------- CC| |jrieden@redhat.com, | |minwang@redhat.com Assignee|jrieden@redhat.com |jcajka@redhat.com
https://bugzilla.redhat.com/show_bug.cgi?id=1096273
Matt Newsome mnewsome@redhat.com changed:
What |Removed |Added ---------------------------------------------------------------------------- CC| |mnewsome@redhat.com
https://bugzilla.redhat.com/show_bug.cgi?id=1096273
Martin Cermak mcermak@redhat.com changed:
What |Removed |Added ---------------------------------------------------------------------------- CC| |mcermak@redhat.com
https://bugzilla.redhat.com/show_bug.cgi?id=1096273
Martin Cermak mcermak@redhat.com changed:
What |Removed |Added ---------------------------------------------------------------------------- CC| |mcermak@redhat.com
https://bugzilla.redhat.com/show_bug.cgi?id=1096273
Jakub Čajka jcajka@redhat.com changed:
What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |MODIFIED Fixed In Version| |golang-1.6-1.el7
https://bugzilla.redhat.com/show_bug.cgi?id=1096273
errata-xmlrpc errata-xmlrpc@redhat.com changed:
What |Removed |Added ---------------------------------------------------------------------------- Status|MODIFIED |ON_QA
https://bugzilla.redhat.com/show_bug.cgi?id=1096273
Jakub Čajka jcajka@redhat.com changed:
What |Removed |Added ---------------------------------------------------------------------------- CC|golang@lists.fedoraproject. | |org |
golang@lists.fedoraproject.org
-
Red Hat Bugzilla