[Bug 1026976] New: "Error starting container, Error adding device, can't find device for parent ''" when starting fedora docker image

[Red Hat Bugzilla]

https://bugzilla.redhat.com/show_bug.cgi?id=1026976

Bug ID: 1026976
           Summary: "Error starting container, Error adding device, can't
                    find device for parent ''" when starting fedora docker
                    image
           Product: Fedora
           Version: 19
         Component: docker-io
          Assignee: [e-mail]
          Reporter: [e-mail]
        QA Contact: [e-mail]
                CC: [e-mail], [e-mail],
                    [e-mail], [e-mail],
                    [e-mail]

Description of problem:

Running container fails with:

Error: Error starting container 43e36bc82ecd: Error adding device for
'0f3e92b4e94d643339d14b096ade73b42b28ff6f9be99064696a97f248c1a651': can't find
device for parent ''

Additionally /var/log/messages has this:
Nov  5 10:20:20 docker kernel: [   85.389079] device-mapper: space map common:
index_check failed: blocknr 0 != wanted 314
Nov  5 10:20:20 docker kernel: [   85.389102] device-mapper: block manager:
index validator check failed for block 314
Nov  5 10:20:20 docker kernel: [   85.389284] device-mapper: transaction
manager: couldn't open metadata space map
Nov  5 10:20:20 docker kernel: [   85.389295] device-mapper: thin metadata:
tm_open_with_sm failed
Nov  5 10:20:20 docker kernel: [   85.390964] device-mapper: table: 253:1:
thin-pool: Error creating metadata object
Nov  5 10:20:20 docker kernel: [   85.391095] device-mapper: ioctl: error
adding target to table

Version-Release number of selected component (if applicable):
# rpm -qa docker-io
docker-io-0.7-0.13.dm.fc19.x86_64

# uname -a
Linux ... 3.11.6-200.fc19.x86_64 #1 SMP Fri Oct 18 22:34:18 UTC 2013 x86_64
x86_64 x86_64 GNU/Linux

How reproducible:
Every time

Steps to Reproduce:
1. Install docker-io-0.7-0.13.dm.fc19.x86_64.rpm
2. Start docker daemon (docker -d &)
3. Download mattdm/fedora images (docker pull mattdm/fedora)
4. Try to start a fedora image (docker run -i -t mattdm/fedora /bin/bash)

Actual results:
Error: Error starting container 96c6d2706331: Error adding device for
'0f3e92b4e94d643339d14b096ade73b42b28ff6f9be99064696a97f248c1a651': can't find
device for parent ''

Expected results:
Shell access to fedora container

Additional info:

My rootfs is on an LV fedora-root in VG fedora

# dmsetup ls --tree
fedora-root (253:0)
 └─ (252:2)

Not sure if that would make a difference.

[Bug 1026976] "Error starting container, Error adding device, can't find device for parent ''" when starting fedora docker image

[Red Hat Bugzilla]

https://bugzilla.redhat.com/show_bug.cgi?id=1026976

--- Comment #1 from Alexander Larsson  ---
The device mapper errors seem very bad, I have no idea what exactly goes wrong,
but it seems like the devicemapper block devices are bad and it just fails to
set up the devmapper devices, and then later fails due to them not existing.

Its a bit hard to debug this with the upstream dm work in flux though.

[Bug 1026976] "Error starting container, Error adding device, can't find device for parent ''" when starting fedora docker image

[Red Hat Bugzilla]

https://bugzilla.redhat.com/show_bug.cgi?id=1026976

--- Comment #2 from Lokesh Mandvekar  ---
Seth, could you check if this issue persists with the latest rpm?

[Bug 1026976] "Error starting container, Error adding device, can't find device for parent ''" when starting fedora docker image

[Red Hat Bugzilla]

https://bugzilla.redhat.com/show_bug.cgi?id=1026976

--- Comment #3 from Seth Jennings  ---
I don't observe this with docker-io-0.7-0.20.rc7.fc19.x86_64:

[root@localhost ~]# docker -d &
[1] 915
[root@localhost ~]# [/var/lib/docker|72633d2e] +job initapi()
[/var/lib/docker|72633d2e.initapi()] Creating server
Loading containers: done.
[/var/lib/docker|72633d2e.initapi()] Creating pidfile
[/var/lib/docker|72633d2e.initapi()] Setting up signal traps
[/var/lib/docker|72633d2e] Register(create) (handlers=map[initapi:0x491200])
[/var/lib/docker|72633d2e] Register(start) (handlers=map[initapi:0x491200
create:0x4aec70])
[/var/lib/docker|72633d2e] Register(serveapi) (handlers=map[initapi:0x491200
create:0x4aec70 start:0x4aecd0])
[/var/lib/docker|72633d2e] -job initapi() = OK (0)
[/var/lib/docker|72633d2e] +job serveapi(unix:///var/run/docker.sock)
2013/11/25 11:02:17 Listening for HTTP on /var/run/docker.sock (unix)

[root@localhost ~]# rpm -qa docker-io
docker-io-0.7-0.20.rc7.fc19.x86_64

[root@localhost ~]# docker version
Client version: 0.7
Go version (client): go1.1.2
Git commit (client): ea7811c/0.20.rc7.fc19
2013/11/25 11:08:02 GET /v1.7/version
Server version: 0.7
Git commit (server): ea7811c/0.20.rc7.fc19
Go version (server): go1.1.2
Last stable version: 0.6.7, please update docker

[root@localhost ~]# docker images
2013/11/25 11:08:09 GET /v1.7/images/json
REPOSITORY          TAG                 IMAGE ID            CREATED            
SIZE
mattdm/fedora       f20                 1bdd590de956        4 weeks ago        
348 MB (virtual 348 MB)
mattdm/fedora       latest              0f3e92b4e94d        4 weeks ago        
348 MB (virtual 348 MB)
mattdm/fedora       f19                 50f374c05c2c        9 weeks ago        
349 MB (virtual 349 MB)
mattdm/fedora       f20rc3.small        97fc5bf7f8d4        9 weeks ago        
198.4 MB (virtual 198.4 MB)

[root@localhost ~]# uname -a
Linux localhost.localdomain 3.11.9-200.fc19.x86_64 #1 SMP Wed Nov 20 21:22:24
UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

[root@localhost ~]# docker run -i -t mattdm/fedora /bin/bash
2013/11/25 11:08:23 POST /v1.7/containers/create
[/var/lib/docker|72633d2e] +job create()
e9d7ea6c6c99a87afb7de83c1c89e11df1f2429909836c2f151474a263e40610
[/var/lib/docker|72633d2e] -job create() = OK (0)
2013/11/25 11:08:23 POST
/v1.7/containers/e9d7ea6c6c99a87afb7de83c1c89e11df1f2429909836c2f151474a263e40610/attach?stderr=1&stdin=1&stdout=1&stream=1
2013/11/25 11:08:23 POST
/v1.7/containers/e9d7ea6c6c99a87afb7de83c1c89e11df1f2429909836c2f151474a263e40610/start

[/var/lib/docker|72633d2e] +job
start(e9d7ea6c6c99a87afb7de83c1c89e11df1f2429909836c2f151474a263e40610)
                                                              bash-4.2#
[/var/lib/docker|72633d2e] -job
start(e9d7ea6c6c99a87afb7de83c1c89e11df1f2429909836c2f151474a263e40610) = OK
(0)
                               2013/11/25 11:08:23 POST
/v1.7/containers/e9d7ea6c6c99a87afb7de83c1c89e11df1f2429909836c2f151474a263e40610/resize?h=37&w=153
bash-4.2# exit
exit
2013/11/25 11:08:27 GET
/v1.7/containers/e9d7ea6c6c99a87afb7de83c1c89e11df1f2429909836c2f151474a263e40610/json

[root@localhost ~]# dmsetup ls --tree
docker-252:2-307531-1bdd590de956281c7a5b1394cc0801ee879bc220be52e399eb61a9c3f2810c0d
(253:2)
 └─docker-252:2-307531-pool (253:0)
    ├─ (7:0)
    └─ (7:1)
docker-252:2-307531-base (253:1)
 └─docker-252:2-307531-pool (253:0)
    ├─ (7:0)
    └─ (7:1)
docker-252:2-307531-e9d7ea6c6c99a87afb7de83c1c89e11df1f2429909836c2f151474a263e40610-init
(253:8)
 └─docker-252:2-307531-pool (253:0)
    ├─ (7:0)
    └─ (7:1)
docker-252:2-307531-7c46893c7790547d9d6e604cc327f246fd670c872378a6a203e1bf7ae4204c3c
(253:7)
 └─docker-252:2-307531-pool (253:0)
    ├─ (7:0)
    └─ (7:1)
docker-252:2-307531-0f3e92b4e94d643339d14b096ade73b42b28ff6f9be99064696a97f248c1a651
(253:5)
 └─docker-252:2-307531-pool (253:0)
    ├─ (7:0)
    └─ (7:1)
docker-252:2-307531-7c46893c7790547d9d6e604cc327f246fd670c872378a6a203e1bf7ae4204c3c-init
(253:6)
 └─docker-252:2-307531-pool (253:0)
    ├─ (7:0)
    └─ (7:1)
docker-252:2-307531-50f374c05c2c07d28620824a6f8427d50879d92f5be6b318d3340f4f0f6120a2
(253:4)
 └─docker-252:2-307531-pool (253:0)
    ├─ (7:0)
    └─ (7:1)
docker-252:2-307531-e9d7ea6c6c99a87afb7de83c1c89e11df1f2429909836c2f151474a263e40610
(253:9)
 └─docker-252:2-307531-pool (253:0)
    ├─ (7:0)
    └─ (7:1)
docker-252:2-307531-97fc5bf7f8d42606fa896e1d391a0b882f78322ce0ff77c03fbd7f8e3b7a73ed
(253:3)
 └─docker-252:2-307531-pool (253:0)
    ├─ (7:0)
    └─ (7:1)

[Bug 1026976] "Error starting container, Error adding device, can't find device for parent ''" when starting fedora docker image

[Red Hat Bugzilla]

https://bugzilla.redhat.com/show_bug.cgi?id=1026976

Marek Goldmann  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |CLOSED
         Resolution|---                         |WORKSFORME
        Last Closed|                            |2013-12-03 08:17:07

--- Comment #4 from Marek Goldmann  ---
I'll close this issue, feel free to reopen it, but I'm pretty sure it was fixed
with the 0.7.0 release - I don't see this issue.