[Bug 1042786] New: Docker can't talk to the API as certificate can't be verfified

[Red Hat Bugzilla]

https://bugzilla.redhat.com/show_bug.cgi?id=1042786

            Bug ID: 1042786
           Summary: Docker can't talk to the API as certificate can't be
                    verfified
           Product: Fedora
           Version: 19
         Component: docker-io
          Assignee: lsm5(a)redhat.com
          Reporter: peter.meier(a)immerda.ch
        QA Contact: extras-qa(a)fedoraproject.org
                CC: golang(a)lists.fedoraproject.org, lsm5(a)redhat.com,
                    mattdm(a)redhat.com, mgoldman(a)redhat.com,
                    vbatts(a)redhat.com



Description of problem:

I can't use docker, as it can't talk to the API as api.go is not able to
verifiy the certificate of docker's API.

Version-Release number of selected component (if applicable):

# rpm -qi docker-io
Name        : docker-io
Version     : 0.7.0
Release     : 14.fc19
Architecture: x86_64
Install Date: Tue 10 Dec 2013 07:12:16 PM CET
Group       : Unspecified
Size        : 12003115
License     : ASL 2.0
Signature   : RSA/SHA256, Tue 03 Dec 2013 01:17:40 AM CET, Key ID
07477e65fb4b18e6
Source RPM  : docker-io-0.7.0-14.fc19.src.rpm
Build Date  : Mon 02 Dec 2013 05:06:54 PM CET
Build Host  : buildvm-12.phx2.fedoraproject.org


How reproducible:

Install docker-io, try to run a search -> fail


Steps to Reproduce:
1. yum install docker-io
2. systemctl start docker
3. docker search fedora

Actual results:

$ docker search fedora 
2013/12/13 12:41:35 Error: Get https://index.docker.io/v1/search?q=fedora:
x509: certificate signed by unknown authority

/var/log/messages
Dec 13 12:51:10 foo docker[14359]: 2013/12/13 12:51:10 GET
/v1.7/images/search?term=fedora
Dec 13 12:51:10 foo docker[14359]: [error] api.go:1034 Error: Get
https://index.docker.io/v1/search?q=fedora: x509: certificate signed by unknown
authority
Dec 13 12:51:10 foo docker[14359]: [error] api.go:82 HTTP Error: statusCode=500
Get https://index.docker.io/v1/search?q=fedora: x509: certificate signed by
unknown authority


Expected results:

Givme the fedora images


Additional info:

The is no problem to query this URL from curl nor from wget:

# curl https://index.docker.io/v1/search?q=fedora
{"query": "fedora", "num_results": 11, "results":
[{"name": "mattdm/fedora",
"description": "A basic Fedora image corresponding roughly to a minimal
install, minus some things which don't make sense in a container. Use tag `f19`
for Fedora 19."}, {"name": "alexl/fedora-19",
"description": "Minimal base
images based on Fedora 19"}, {"name": "simoncadman/fedora-20",
"description":
"Updated fedora 20, based on goldmann/f20 , includes packages for building
rpms"}, {"name": "mattdm/fedora-small", "description":
"A small Fedora image on
which to build. Contains just enough that you'll be able to run `yum install`
in your dockerfiles to create something useful. Use tag `f19` for Fedora 19."},
{"name": "philips/fedora", "description": ""},
{"name": "kraman/fedora_cfn",
"description": "mattdm/fedora:f19 + cloud-Init\n(SSH key setting has been
disabled)"}, {"name": "dgarcia/fedora18base",
"description": ""}, {"name":
"goldmann/f20", "description": "Fedora 20 repository"},
{"name":
"philips/riak-base", "description": "Base Fedora box with Riak
installed."},
{"name": "jumanjiman/eiffelstudio", "description":
"EiffelStudio IDE in a
Docker container (fedora at the moment)"}, {"name":
"svendowideit/publican",
"description": "Publish DocBook documentation using publican (running in a
fedora container)\n\ndocker run -t -i -v $(pwd):/mnt svendowideit/publican
build"}]}


# wget -O /dev/stdout https://index.docker.io/v1/search?q=fedora
--2013-12-13 12:52:21--  https://index.docker.io/v1/search?q=fedora
Resolving index.docker.io (index.docker.io)... 54.224.119.89, 54.234.135.251
Connecting to index.docker.io (index.docker.io)|54.224.119.89|:443...
connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [application/json]
Saving to: ‘/dev/stdout’

    [<=>                                                                       
                     ] 0           --.-K/s              {"query":
"fedora",
"num_results": 11, "results": [{"name":
"mattdm/fedora", "description": "A
basic Fedora image corresponding roughly to a minimal install, minus some
things which don't make sense in a container. Use tag `f19` for Fedora 19."},
{"name": "alexl/fedora-19", "description": "Minimal
base images based on Fedora
19"}, {"name": "simoncadman/fedora-20", "description":
"Updated fedora 20,
based on goldmann/f20 , includes packages for building rpms"}, {"name":
"mattdm/fedora-small", "description": "A small Fedora image on
which to build.
Contains just enough that you'll be able to run `yum install` in your
dockerfiles to create something useful. Use tag `f19` for Fedora 19."},
{"name": "philips/fedora", "description": ""},
{"name": "kraman/fedora_cfn",
"description": "mattdm/fedora:f19 + cloud-Init\n(SSH key setting has been
disabled)"}, {"name": "dgarcia/fedora18base",
"description": ""}, {"name":
"goldmann/f20", "description": "Fedora 20 repository"},
{"name":
"philips/riak-base", "description": "Base Fedora box with Riak
installed."},
{"name": "jumanjiman/eiffelstudio", "description":
"EiffelStudio IDE in a
Docker container (fedora at the moment)"}, {"name":
"svendowideit/publican",
"description": "Publish DocBook documentation using publican (running in a
fedora container)\n\ndocker run -t -i -v $(pwd):/mnt svendowideit/publican
build"}    [ <=>                                                              

                            ] 1,373       --.-K/s   in 0.001s  

2013-12-13 12:52:21 (1.24 MB/s) - ‘/dev/stdout’ saved [1373]

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug 1042786] Docker can't talk to the API as certificate can't be verfified

[Red Hat Bugzilla]

https://bugzilla.redhat.com/show_bug.cgi?id=1042786



--- Comment #1 from Lokesh Mandvekar <lsm5(a)redhat.com&gt; ---
Hi Peter, 

Can you check if this occurs with 0.7.1-1 (currently in testing repo)? If yes,
can you also check with upstream released binary?
http://docs.docker.io/en/latest/installation/binaries/

I'm seeing something similar to this (Bug 1041400) on rawhide with 0.7.1-1 and
also with the upstream binary.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug 1042786] Docker can't talk to the API as certificate can't be verfified

[Red Hat Bugzilla]

https://bugzilla.redhat.com/show_bug.cgi?id=1042786

Peter Meier <peter.meier(a)immerda.ch&gt; changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |CLOSED
         Resolution|---                         |CURRENTRELEASE
        Last Closed|                            |2013-12-16 17:34:39



--- Comment #2 from Peter Meier <peter.meier(a)immerda.ch&gt; ---
The certificate verification works with 0.7.1-1 (in updates-testing).

However docker seems now to use another api-endpoint which does not work:

$ docker -v
Docker version 0.7.1, build e39d35d/0.7.1
$ rpm -qa | grep docker-io
docker-io-0.7.1-1.fc19.x86_64

$ docker search fedora
2013/12/16 23:31:02 Error: Not Found

Dec 16 23:27:04 foo docker[14359]: 2013/12/16 23:27:04 GET
/v1.8/images/search?term=fedora

And indeed this api calls gives a 404:


$ curl -I https://index.docker.io/v1.8/search?q=fedora
HTTP/1.1 404 Not Found
server: nginx/1.2.1
date: Mon, 16 Dec 2013 22:31:47 GMT
content-type: text/html; charset=utf-8
connection: close
vary: Cookie

Looks like a different problem, so closing this bug.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug 1042786] Docker can't talk to the API as certificate can't be verfified

[Red Hat Bugzilla]

https://bugzilla.redhat.com/show_bug.cgi?id=1042786

Peter Meier <peter.meier(a)immerda.ch&gt; changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|CLOSED                      |NEW
         Resolution|CURRENTRELEASE              |---
           Keywords|                            |Reopened



--- Comment #3 from Peter Meier <peter.meier(a)immerda.ch&gt; ---
Interesting due to #1038329 I restarted the daemon and now it fails again:

Dec 16 23:31:02 foo docker[14359]: 2013/12/16 23:31:02 GET
/v1.8/images/search?term=fedora
Dec 16 23:40:29 foo docker[14359]: 2013/12/16 23:40:29 Received signal
'terminated', exiting
Dec 16 23:40:29 foo docker[23426]: [/var/lib/docker|b832a237] +job initapi()
Dec 16 23:40:29 foo docker[23426]: [/var/lib/docker|b832a237.initapi()]
Creating server
Dec 16 23:40:29 foo docker[23426]: Loading containers:  #010done.
Dec 16 23:40:29 foo docker[23426]: [/var/lib/docker|b832a237.initapi()]
Creating pidfile
Dec 16 23:40:29 foo docker[23426]: [/var/lib/docker|b832a237.initapi()] Setting
up signal traps
Dec 16 23:40:29 foo docker[23426]: [/var/lib/docker|b832a237] Register(create)
(handlers=map[initapi:0x496300])
Dec 16 23:40:29 foo docker[23426]: [/var/lib/docker|b832a237] Register(start)
(handlers=map[initapi:0x496300 create:0x4b4160])
Dec 16 23:40:29 foo docker[23426]: [/var/lib/docker|b832a237]
Register(serveapi) (handlers=map[initapi:0x496300 create:0x4b4160
start:0x4b41a0])
Dec 16 23:40:29 foo docker[23426]: [/var/lib/docker|b832a237] -job initapi() =
OK (0)
Dec 16 23:40:29 foo docker[23426]: [/var/lib/docker|b832a237] +job
serveapi(unix:///var/run/docker.sock) 
Dec 16 23:40:29 foo docker[23426]: 2013/12/16 23:40:29 Listening for HTTP on
/var/run/docker.sock (unix) 
Dec 16 23:40:54 foo docker[23426]: 2013/12/16 23:40:54 GET
/v1.8/images/search?term=fedora
Dec 16 23:40:54 foo docker[23426]: [error] api.go:1065 Error: Get
https://index.docker.io/v1/search?q=fedora: x509: certificate signed by unknown
authority 
Dec 16 23:40:54 foo docker[23426]: [error] api.go:87 HTTP Error: statusCode=500
Get https://index.docker.io/v1/search?q=fedora: x509: certificate signed by
unknown authority
Dec 16 23:41:15 foo docker[23426]: 2013/12/16 23:41:15 GET
/v1.8/images/search?term=fedora
Dec 16 23:41:16 foo docker[23426]: [error] api.go:1065 Error: Get
https://index.docker.io/v1/search?q=fedora: x509: certificate signed by unknown
authority 
Dec 16 23:41:16 foo docker[23426]: [error] api.go:87 HTTP Error: statusCode=500
Get https://index.docker.io/v1/search?q=fedora: x509: certificate signed by
unknown authority


So still no luck :(

The interesting part is that it looks like it is first trying v1.8 and then
then v1 API and only barfs on the last one.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug 1042786] Docker can't talk to the API as certificate can't be verfified

[Red Hat Bugzilla]

https://bugzilla.redhat.com/show_bug.cgi?id=1042786



--- Comment #4 from Lokesh Mandvekar <lsm5(a)redhat.com&gt; ---
Peter, could you check with 0.7.2 ?

this should be going into updates-testing repo for f19 and f20 soon. get the
f19 rpm from here:
http://kojipkgs.fedoraproject.org//packages/docker-io/0.7.2/1.fc19/x86_64...

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug 1042786] Docker can't talk to the API as certificate can't be verfified

[Red Hat Bugzilla]

https://bugzilla.redhat.com/show_bug.cgi?id=1042786



--- Comment #5 from Peter Meier <peter.meier(a)immerda.ch&gt; ---
Unfortunately not:

$ rpm -Uvh
http://kojipkgs.fedoraproject.org//packages/docker-io/0.7.2/1.fc19/x86_64...
Retrieving
http://kojipkgs.fedoraproject.org//packages/docker-io/0.7.2/1.fc19/x86_64...
Preparing...                          ################################# [100%]
Updating / installing...
   1:docker-io-0.7.2-1.fc19           ################################# [ 50%]
Cleaning up / removing...
   2:docker-io-0.7.1-1.fc19           ################################# [100%]

$ service docker restart
Redirecting to /bin/systemctl restart  docker.service

$ docker search fedora
2013/12/19 21:05:32 Error: Get https://index.docker.io/v1/search?q=fedora:
x509: certificate signed by unknown authority

$ docker -v
Docker version 0.7.2, build 28b162e/0.7.2

$ rpm -qi docker-io
Name        : docker-io
Version     : 0.7.2
Release     : 1.fc19
Architecture: x86_64
Install Date: Thu 19 Dec 2013 09:05:18 PM CET
Group       : Unspecified
Size        : 12776659
License     : ASL 2.0
Signature   : (none)
Source RPM  : docker-io-0.7.2-1.fc19.src.rpm
Build Date  : Wed 18 Dec 2013 08:09:48 PM CET
Build Host  : buildvm-24.phx2.fedoraproject.org
Relocations : (not relocatable)
Packager    : Fedora Project
Vendor      : Fedora Project
URL         : http://www.docker.io
Summary     : Automates deployment of containerized applications
Description :
Docker is an open-source engine that automates the deployment of any
application as a lightweight, portable, self-sufficient container that will
run virtually anywhere.

Docker containers can encapsulate any payload, and will run consistently on
and between virtually any server. The same container that a developer builds
and tests on a laptop will run at scale, in production*, on VMs, bare-metal
servers, OpenStack clusters, public instances, or combinations of the above.


$ curl https://index.docker.io/v1/search?q=fedora 
{"query": "fedora", "num_results": 14, "results":
[{"name": "mattdm/fedora",
"description": "A basic Fedora image corresponding roughly to a minimal
install, minus some things which don't make sense in a container. Use tag `f20`
for Fedora 20 or `f19` for Fedora 19."}, {"name":
"alexl/fedora-19",
"description": "Minimal base images based on Fedora 19"},
{"name":
"simoncadman/fedora-20", "description": "Updated fedora 20, based
on
goldmann/f20 , includes packages for building rpms"}, {"name":
"mattdm/fedora-small", "description": "A small Fedora image on
which to build.
Contains just enough that you'll be able to run `yum install` in your
dockerfiles to create something useful. Use tag `f19` for Fedora 19."},
{"name": "philips/fedora", "description": ""},
{"name":
"lsm5/fedora-imagebuilder", "description": ""},
{"name":
"lzap/fedora-foreman-git-base", "description": ""},
{"name":
"lzap/fedora-foreman-git-stable", "description": "Foreman stable
installation
from Git\n\nhttps://github.com/lzap/foreman-docker"}, {"name":
"kraman/fedora_cfn", "description": "mattdm/fedora:f19 +
cloud-Init\n(SSH key
setting has been disabled)"}, {"name": "dgarcia/fedora18base",
"description":
""}, {"name": "goldmann/f20", "description":
"Fedora 20 repository"}, {"name":
"philips/riak-base", "description": "Base Fedora box with Riak
installed."},
{"name": "jumanjiman/eiffelstudio", "description":
"EiffelStudio IDE in a
Docker container (fedora at the moment)"}, {"name":
"svendowideit/publican",
"description": "Publish DocBook documentation using publican (running in a
fedora container)\n\ndocker run -t -i -v $(pwd):/mnt svendowideit/publican
build"}]}


$ grep docker /var/log/messages | tail
Dec 19 21:05:24 gasteiz docker[7093]: [/var/lib/docker|403644c2] Register(tag)
(handlers=map[export:0x4b69c0 stop:0x4b6a40 serveapi:0x4b6b00 initapi:0x496520
version:0x4b1720 create:0x4b6a00 start:0x4b6a80 kill:0x4b6ac0 wait:0x4b6b40])
Dec 19 21:05:24 gasteiz docker[7093]: [/var/lib/docker|403644c2]
Register(resize) (handlers=map[initapi:0x496520 version:0x4b1720
create:0x4b6a00 start:0x4b6a80 kill:0x4b6ac0 wait:0x4b6b40 tag:0x4b6b80
export:0x4b69c0 stop:0x4b6a40 serveapi:0x4b6b00])
Dec 19 21:05:24 gasteiz docker[7093]: [/var/lib/docker|403644c2]
Register(commit) (handlers=map[export:0x4b69c0 stop:0x4b6a40 serveapi:0x4b6b00
resize:0x4b6bc0 initapi:0x496520 version:0x4b1720 create:0x4b6a00
start:0x4b6a80 kill:0x4b6ac0 wait:0x4b6b40 tag:0x4b6b80])
Dec 19 21:05:24 gasteiz docker[7093]: [/var/lib/docker|403644c2] Register(info)
(handlers=map[initapi:0x496520 version:0x4b1720 create:0x4b6a00 start:0x4b6a80
kill:0x4b6ac0 wait:0x4b6b40 tag:0x4b6b80 commit:0x4b6c00 export:0x4b69c0
stop:0x4b6a40 serveapi:0x4b6b00 resize:0x4b6bc0])
Dec 19 21:05:24 gasteiz docker[7093]: [/var/lib/docker|403644c2] -job initapi()
= OK (0)
Dec 19 21:05:24 gasteiz docker[7093]: [/var/lib/docker|403644c2] +job
serveapi(unix:///var/run/docker.sock)
Dec 19 21:05:24 gasteiz docker[7093]: 2013/12/19 21:05:24 Listening for HTTP on
/var/run/docker.sock (unix)
Dec 19 21:05:32 gasteiz docker[7093]: 2013/12/19 21:05:32 GET
/v1.8/images/search?term=fedora
Dec 19 21:05:32 gasteiz docker[7093]: [error] api.go:1062 Error: Get
https://index.docker.io/v1/search?q=fedora: x509: certificate signed by unknown
authority
Dec 19 21:05:32 gasteiz docker[7093]: [error] api.go:87 HTTP Error:
statusCode=500 Get https://index.docker.io/v1/search?q=fedora: x509:
certificate signed by unknown authority

I don't understand why it fails with docker, while curl works...

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug 1042786] Docker can't talk to the API as certificate can't be verfified

[Red Hat Bugzilla]

https://bugzilla.redhat.com/show_bug.cgi?id=1042786



--- Comment #6 from Lokesh Mandvekar <lsm5(a)redhat.com&gt; ---
hmm, I can't seem to replicate it, perhaps you might wanna check if this helps:
https://groups.google.com/d/msg/golang-nuts/vWewH0Wum90/4A4SNmdlb8gJ

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug 1042786] Docker can't talk to the API as certificate can't be verfified

[Red Hat Bugzilla]

https://bugzilla.redhat.com/show_bug.cgi?id=1042786

Peter Meier <peter.meier(a)immerda.ch&gt; changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |CLOSED
         Resolution|---                         |NOTABUG
        Last Closed|2013-12-16 17:34:39         |2014-01-15 15:42:55



--- Comment #7 from Peter Meier <peter.meier(a)immerda.ch&gt; ---
Finally, I found the solution. And it's totally my own fault.

> From previous tinkering with extending the certificate chain I had a
> link 
/etc/pki/tls/certs/ca-certificates.crt pointing to
/etc/pki/tls/certs/ca-bundle.trust.crt which caused these failures.

Removing that faulty link fixed the isuee.

-- 
You are receiving this mail because:
You are on the CC list for the bug.