https://bugzilla.redhat.com/show_bug.cgi?id=1189028
Bug ID: 1189028 Summary: docker 1.4.1-3.el6 fd leak Product: Fedora EPEL Version: el6 Component: docker-io Assignee: lsm5@redhat.com Reporter: b@wtnb.mydns.jp QA Contact: extras-qa@fedoraproject.org CC: adimania@gmail.com, admiller@redhat.com, golang@lists.fedoraproject.org, hushan.jia@gmail.com, jchaloup@redhat.com, jperrin@centos.org, lsm5@redhat.com, mattdm@redhat.com, mgoldman@redhat.com, miminar@redhat.com, s@shk.io, thrcka@redhat.com, vbatts@redhat.com
Description of problem:
docker daemon doesn't close file descriptor.
Version-Release number of selected component (if applicable): docker-io-1.4.1-3.el6.x86_64
How reproducible:
Steps to Reproduce: 1. while true; do docker run -t -i --rm busybox /bin/true ; done 2. (in other terminal) DEBUG=1 docker info 3. check values at Fds: and Goroutines: lines
Actual results: Fds and Goroutines values are increased forever
Expected results:
Additional info:
- It doesn't happen on RHEL7 + docker 1.4.1 - It doesn't happen on RHEL6 + docker-io-1.3.2-2.el6.x86_64
# lsof -p $(pidof docker) <snip> docker 7771 root 14u REG 0,9 0 3791 [eventfd] docker 7771 root 15u REG 0,9 0 3791 [eventfd] docker 7771 root 16r REG 0,21 0 20709 /cgroup/memory/docker/98b529e398bec4568c58ea96088aef1535eca504bed6453bab8f6f7803d31cfb/memory.oom_control (deleted) docker 7771 root 17u REG 0,9 0 3791 [eventfd] docker 7771 root 18r REG 0,21 0 21176 /cgroup/memory/docker/bc213d3e80815021780064874665fe80801c5a1010d1c0bc4bd5b118ca45d11f/memory.oom_control (deleted) docker 7771 root 19u REG 0,9 0 3791 [eventfd] docker 7771 root 20r REG 0,21 0 21626 /cgroup/memory/docker/7b8d59ec4f23c14e2732726594460ccf89fe31c06e60fabbdc4e358f794af102/memory.oom_control (deleted) docker 7771 root 21w REG 0,9 0 3791 [eventfd] docker 7771 root 22r REG 0,21 0 22112 /cgroup/memory/docker/99ac44fcb18a3e1aea00a5e7cfb040942f2c4f2c673c291105c367788e49d0f2/memory.oom_control (deleted) docker 7771 root 24r REG 0,21 0 22672 /cgroup/memory/docker/b1804ae48289df81d3a5c54167baf729612e12c552ef98917df90d8645d235ad/memory.oom_control (deleted) <snip>
# ps $(pidof docker) m PID TTY STAT TIME COMMAND 7771 pts/0 - 0:09 /usr/bin/docker -d - - Sl 0:02 - - - Sl 0:00 - - - Sl 0:00 - - - Sl 0:00 - - - Sl 0:00 - - - Sl 0:00 - - - Sl 0:00 - - - Sl 0:01 - - - Sl 0:00 - <snip>
https://bugzilla.redhat.com/show_bug.cgi?id=1189028
Daniel Walsh dwalsh@redhat.com changed:
What |Removed |Added ---------------------------------------------------------------------------- CC| |dwalsh@redhat.com
--- Comment #1 from Daniel Walsh dwalsh@redhat.com --- Is this still a bug in docker-1.5?
https://bugzilla.redhat.com/show_bug.cgi?id=1189028
Daniel Walsh dwalsh@redhat.com changed:
What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |ASSIGNED
https://bugzilla.redhat.com/show_bug.cgi?id=1189028
--- Comment #2 from Watanabe Takashi b@wtnb.mydns.jp --- (In reply to Daniel Walsh from comment #1)
Is this still a bug in docker-1.5?
I tried kernel-2.6.32-504.8.1.el6.x86_64, docker-io-1.5.0-1.el6.x86_64(built from http://pkgs.fedoraproject.org/cgit/docker-io.git/log/?h=el6). This problem was reproduced.
[vagrant@localhost ~]$ uname -r 2.6.32-504.8.1.el6.x86_64 [vagrant@localhost ~]$ docker version Client version: 1.5.0 Client API version: 1.17 Go version (client): go1.3.3 Git commit (client): a8a31ef/1.5.0 OS/Arch (client): linux/amd64 Server version: 1.5.0 Server API version: 1.17 Go version (server): go1.3.3 Git commit (server): a8a31ef/1.5.0 [vagrant@localhost ~]$ for i in $(seq 10); do docker run -t -i --rm busybox /bin/true ; done [vagrant@localhost ~]$ DEBUG=1 docker info Containers: 0 Images: 4 Storage Driver: devicemapper Pool Name: docker-253:0-396434-pool Pool Blocksize: 65.54 kB Backing Filesystem: extfs Data file: /dev/loop0 Metadata file: /dev/loop1 Data Space Used: 311 MB Data Space Total: 107.4 GB Metadata Space Used: 823.3 kB Metadata Space Total: 2.147 GB Udev Sync Supported: true Data loop file: /var/lib/docker/devicemapper/devicemapper/data Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata Library Version: 1.02.89-RHEL6 (2014-09-01) Execution Driver: native-0.2 Kernel Version: 2.6.32-504.8.1.el6.x86_64 Operating System: <unknown> CPUs: 2 Total Memory: 490.2 MiB Name: localhost.localdomain ID: 7ZND:FDGS:G34R:CSG5:X5PZ:U27F:PX2D:PDSC:X6U5:SYSW:CHE6:IMD3 Debug mode (server): false Debug mode (client): true Fds: 31 Goroutines: 24 EventsListeners: 0 Init SHA1: 085c9e5c44d5bab44e695bb55892ba6852307cf5 Init Path: /usr/libexec/docker/dockerinit Docker Root Dir: /var/lib/docker [vagrant@localhost ~]$ sudo lsof -p $(pidof docker) COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME docker 5644 root cwd DIR 253,0 4096 2 / docker 5644 root rtd DIR 253,0 4096 2 / docker 5644 root txt REG 253,0 12925591 270035 /usr/bin/docker docker 5644 root mem REG 253,0 55184 130655 /lib64/libudev.so.0.5.1 docker 5644 root mem REG 253,0 248680 130642 /lib64/libsepol.so.1 docker 5644 root mem REG 253,0 122040 130643 /lib64/libselinux.so.1 docker 5644 root mem REG 253,0 19536 130601 /lib64/libdl-2.12.so docker 5644 root mem REG 253,0 1921176 130573 /lib64/libc-2.12.so docker 5644 root mem REG 253,0 287096 130611 /lib64/libdevmapper.so.1.02 docker 5644 root mem REG 253,0 583552 262963 /usr/lib64/libsqlite3.so.0.8.6 docker 5644 root mem REG 253,0 142640 130597 /lib64/libpthread-2.12.so docker 5644 root mem REG 253,0 154624 130579 /lib64/ld-2.12.so docker 5644 root 0u CHR 136,2 0t0 5 /dev/pts/2 docker 5644 root 1w REG 253,0 198319 396408 /var/log/docker docker 5644 root 2w REG 253,0 198319 396408 /var/log/docker docker 5644 root 3r CHR 1,9 0t0 3798 /dev/urandom docker 5644 root 4u CHR 10,58 0t0 5485 /dev/mapper/control docker 5644 root 5u unix 0xffff88001d95aa80 0t0 28908 /var/run/docker.sock docker 5644 root 6u REG 253,0 5120 396446 /var/lib/docker/linkgraph.db docker 5644 root 7r DIR 0,10 0 1 inotify docker 5644 root 8u REG 0,9 0 3791 [eventpoll] docker 5644 root 14r REG 0,21 0 29248 /cgroup/memory/docker/bd14586f8a62117ed2429c6182314be87eed3ae897c6505b41b9ec147b6ac5df/memory.oom_control (deleted) docker 5644 root 15r REG 0,21 0 29651 /cgroup/memory/docker/942b863aacaad28e58b2c88e9b9f445f961e4c29d727b2480fdcee09851e2755/memory.oom_control (deleted) docker 5644 root 16u REG 0,9 0 3791 [eventfd] docker 5644 root 17r REG 0,21 0 30114 /cgroup/memory/docker/1578a0311b931a960e2e9a9b43e2a8c7e593eab36796b03647aa11b99734e745/memory.oom_control (deleted) docker 5644 root 18u REG 0,9 0 3791 [eventfd] docker 5644 root 19r REG 0,21 0 30543 /cgroup/memory/docker/3afaa7caba99c43ad5f75ec73d135314d741c3ef086bf52a762317165bb2d78b/memory.oom_control (deleted) docker 5644 root 20u REG 0,9 0 3791 [eventfd] docker 5644 root 21r REG 0,21 0 30992 /cgroup/memory/docker/c472c181f7c09719b287a861a23267b93764cd5e58e8ecfe128243f81a7ff442/memory.oom_control (deleted) docker 5644 root 22u REG 0,9 0 3791 [eventfd] docker 5644 root 23r REG 0,21 0 31352 /cgroup/memory/docker/5fc09cd4d2866ddf5d64678888d6731476001e6caff2391adbccf7ca2a52654d/memory.oom_control (deleted) docker 5644 root 24u REG 0,9 0 3791 [eventfd] docker 5644 root 25r REG 0,21 0 31806 /cgroup/memory/docker/370318c2632bb34e0333ae7fea71eef55b90869374b88ab094112dbb6742c995/memory.oom_control (deleted) docker 5644 root 26u REG 0,9 0 3791 [eventfd] docker 5644 root 27r REG 0,21 0 32255 /cgroup/memory/docker/f0df7e43de3291370a96073b8cef933aeb05d78edc8c8b367462f010da5fa181/memory.oom_control (deleted) docker 5644 root 28u REG 0,9 0 3791 [eventfd] docker 5644 root 29r REG 0,21 0 32657 /cgroup/memory/docker/58a5fde3ed7c223d530e9896d587a3d90c1938f94cb3b8a72056d98cf938b94a/memory.oom_control (deleted) docker 5644 root 30u REG 0,9 0 3791 [eventfd] docker 5644 root 31r REG 0,21 0 33053 /cgroup/memory/docker/1b139ed3beecf8566aaec445d42438432c8d9e492020da26c0b8bb3dcffa4173/memory.oom_control (deleted) docker 5644 root 32u REG 0,9 0 3791 [eventfd] docker 5644 root 34u REG 0,9 0 3791 [eventfd] [vagrant@localhost ~]$ ps m $(pidof docker) PID TTY STAT TIME COMMAND 5644 pts/2 - 0:04 /usr/bin/docker -d - - Sl 0:00 - - - Sl 0:00 - - - Sl 0:00 - - - Sl 0:00 - - - Sl 0:00 - - - Sl 0:00 - - - Sl 0:00 - - - Sl 0:00 - - - Sl 0:00 - - - Sl 0:01 - - - Sl 0:00 - - - Sl 0:00 - - - Sl 0:00 - - - Sl 0:00 - - - Sl 0:00 - - - Sl 0:00 - - - Sl 0:00 - - - Sl 0:00 -
https://bugzilla.redhat.com/show_bug.cgi?id=1189028
Jeremy Eder jeder@redhat.com changed:
What |Removed |Added ---------------------------------------------------------------------------- Summary|docker 1.4.1-3.el6 fd leak |docker fd leak
https://bugzilla.redhat.com/show_bug.cgi?id=1189028
--- Comment #4 from Timothy St. Clair tstclair@redhat.com --- Is there an upstream issue around *this?
https://bugzilla.redhat.com/show_bug.cgi?id=1189028
--- Comment #6 from Timothy St. Clair tstclair@redhat.com --- xref - https://github.com/docker/docker/issues/8693
https://bugzilla.redhat.com/show_bug.cgi?id=1189028
--- Comment #7 from Joe Mario jmario@redhat.com --- An update to my comment 5 above. I'm now wondering if I'm hitting a different problem.
In my testing, if aio-max-nr is lower than what Oracle needs, then it triggers the file desc usage to grow rapidly till the Oracle processes eventually die/error because the system's /proc/sys/fs/file-max-nr value has been hit.
However, once those Oracle processes exit, the file desc usage count drops accordingly.
Joe
https://bugzilla.redhat.com/show_bug.cgi?id=1189028
Vivek Goyal vgoyal@redhat.com changed:
What |Removed |Added ---------------------------------------------------------------------------- CC| |vgoyal@redhat.com
https://bugzilla.redhat.com/show_bug.cgi?id=1189028
--- Comment #11 from Daniel Walsh dwalsh@redhat.com --- There is no unregistermachine. I don't see a nice way to call conn.Close. I have changed the code to only do conn.InitConnection once, which will at least only open one descriptor for all of the daemon run.
https://bugzilla.redhat.com/show_bug.cgi?id=1189028
--- Comment #12 from Daniel Walsh dwalsh@redhat.com --- I have checked it into the docker-1.5.0 branch.
https://bugzilla.redhat.com/show_bug.cgi?id=1189028
Daniel Walsh dwalsh@redhat.com changed:
What |Removed |Added ---------------------------------------------------------------------------- Blocks| |1202517
Referenced Bugs:
https://bugzilla.redhat.com/show_bug.cgi?id=1202517 [Bug 1202517] docker fd leak
https://bugzilla.redhat.com/show_bug.cgi?id=1189028
Bill C. Riemers briemers@redhat.com changed:
What |Removed |Added ---------------------------------------------------------------------------- CC| |briemers@redhat.com
--- Comment #13 from Bill C. Riemers briemers@redhat.com --- *** Bug 1210472 has been marked as a duplicate of this bug. ***
https://bugzilla.redhat.com/show_bug.cgi?id=1189028
--- Comment #14 from Bill C. Riemers briemers@redhat.com --- (In reply to Daniel Walsh from comment #12)
I have checked it into the docker-1.5.0 branch.
How does that help RHEL 6 users? The current version is:
docker-io-1.4.1-3.el6.x86_64
?
I saw a bug for swarm in github. See the bug I just marked as a duplicate for more information. I'm not sure if that is the same issue, but given than initially I see a TSL error, it probably is.
https://bugzilla.redhat.com/show_bug.cgi?id=1189028
--- Comment #15 from Daniel Walsh dwalsh@redhat.com --- Well hopefully docker-io-1.5 will have the fix in the EPEL version.
https://bugzilla.redhat.com/show_bug.cgi?id=1189028
Daniel Walsh dwalsh@redhat.com changed:
What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |MODIFIED
--- Comment #16 from Daniel Walsh dwalsh@redhat.com --- Fixed in docker-1.6
https://bugzilla.redhat.com/show_bug.cgi?id=1189028
John Morales jmorales@gmail.com changed:
What |Removed |Added ---------------------------------------------------------------------------- CC| |jmorales@gmail.com
--- Comment #17 from John Morales jmorales@gmail.com --- I don't believe this is fixed, I'm seeing this still happening using docker-1.6: Installed Packages Name : docker-io Arch : x86_64 Version : 1.6.2 Release : 1.el6 Size : 21 M Repo : installed
From repo : epel-testing
still seeing: lr-x------ 1 root root 64 1434541872 25 -> /cgroup/memory/docker/.../memory.oom_control (deleted) lrwx------ 1 root root 64 1434541872 26 -> [eventfd] lr-x------ 1 root root 64 1434542118 27 -> /cgroup/memory/docker/.../memory.oom_control (deleted) lrwx------ 1 root root 64 1434542118 28 -> [eventfd]
https://bugzilla.redhat.com/show_bug.cgi?id=1189028
--- Comment #18 from Daniel Walsh dwalsh@redhat.com --- If there is a leak this is a separate new problem. Please open a new bugzilla.
https://bugzilla.redhat.com/show_bug.cgi?id=1189028
Lokesh Mandvekar lsm5@redhat.com changed:
What |Removed |Added ---------------------------------------------------------------------------- Status|MODIFIED |CLOSED Resolution|--- |WONTFIX Last Closed| |2016-04-02 02:37:02
--- Comment #19 from Lokesh Mandvekar lsm5@redhat.com --- Closing as docker-io on epel6 is a dead-end..
golang@lists.fedoraproject.org
-
Red Hat Bugzilla