[Bug 1274854] New: kubectl exec: handshake did not verify certificate chain

[Red Hat Bugzilla]

https://bugzilla.redhat.com/show_bug.cgi?id=1274854

Bug ID: 1274854
           Summary: kubectl exec: handshake did not verify certificate
                    chain
           Product: Fedora
           Version: 23
         Component: kubernetes
          Assignee: [e-mail]
          Reporter: [e-mail]
        QA Contact: [e-mail]
                CC: [e-mail], [e-mail],
                    [e-mail], [e-mail],
                    [e-mail], [e-mail]

Description of problem:

kubectl exec fails like this on Fedora 23:

# kubectl exec mock-ooycy /bin/sh -i
error: Unable to upgrade connection: {
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "tls: handshake did not verify certificate chain",
  "code": 500
}

Version-Release number of selected component (if applicable):

kubernetes-1.1.0-0.16.git388061f.fc23.x86_64

How reproducible:

Every time

Steps to Reproduce:
1. See above

[Bug 1274854] kubectl exec: handshake did not verify certificate chain

[Red Hat Bugzilla]

https://bugzilla.redhat.com/show_bug.cgi?id=1274854

Jan Chaloupka  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |ASSIGNED

[Bug 1274854] kubectl exec: handshake did not verify certificate chain

[Red Hat Bugzilla]

https://bugzilla.redhat.com/show_bug.cgi?id=1274854

--- Comment #1 from Jan Chaloupka  ---
I am able to reproduce it.

Steps:
1) install the latest k8s and start all services
2) update apiserver config and remove ServiceAccount from admission control
3) restart kube-apiserver
3) create mock.yaml file:# cat mock.yaml 
apiVersion: v1
kind: Pod
metadata:
  name: mock
spec:
  containers:
  - name: mock
    image: rusintez/mock

4) kubectl create -f mock.yaml

Running command:
# kubectl exec --v=9 mock /bin/sh -i
I1027 09:11:04.821317    9945 debugging.go:101] curl -k -v -XGET  -H
"User-Agent: kubectl/v1.0.6 (linux/amd64) kubernetes/388061f"
http://localhost:8080/api
I1027 09:11:04.822881    9945 debugging.go:120] GET http://localhost:8080/api
200 OK in 1 milliseconds
I1027 09:11:04.822904    9945 debugging.go:126] Response Headers:
I1027 09:11:04.822915    9945 debugging.go:129]     Content-Length: 32
I1027 09:11:04.822927    9945 debugging.go:129]     Content-Type:
application/json
I1027 09:11:04.822937    9945 debugging.go:129]     Date: Tue, 27 Oct 2015
08:11:04 GMT
I1027 09:11:04.822968    9945 request.go:779] Response Body: {
  "versions": [
    "v1"
  ]
}
I1027 09:11:04.823272    9945 debugging.go:101] curl -k -v -XGET  -H
"User-Agent: kubectl/v1.0.6 (linux/amd64) kubernetes/388061f"
http://localhost:8080/api/v1/namespaces/default/pods/mock
I1027 09:11:04.826041    9945 debugging.go:120] GET
http://localhost:8080/api/v1/namespaces/default/pods/mock 200 OK in 2
milliseconds
I1027 09:11:04.826065    9945 debugging.go:126] Response Headers:
I1027 09:11:04.826077    9945 debugging.go:129]     Content-Type:
application/json
I1027 09:11:04.826087    9945 debugging.go:129]     Date: Tue, 27 Oct 2015
08:11:04 GMT
I1027 09:11:04.826098    9945 debugging.go:129]     Content-Length: 973
I1027 09:11:04.826143    9945 request.go:779] Response Body:
{"kind":"Pod","apiVersion":"v1","metadata":{"name":"mock","namespace":"default","selfLink":"/api/v1/namespaces/default/pods/mock","uid":"33334b8c-7c81-11e5-a55a-525400e11373","resourceVersion":"2196","creationTimestamp":"2015-10-27T08:03:25Z"},"spec":{"containers":[{"name":"mock","image":"rusintez/mock","resources":{},"terminationMessagePath":"/dev/termination-log","imagePullPolicy":"IfNotPresent"}],"restartPolicy":"Always","dnsPolicy":"ClusterFirst","nodeName":"127.0.0.1"},"status":{"phase":"Running","conditions":[{"type":"Ready","status":"True"}],"hostIP":"127.0.0.1","podIP":"172.17.0.1","startTime":"2015-10-27T08:03:25Z","containerStatuses":[{"name":"mock","state":{"running":{"startedAt":"2015-10-27T08:05:41Z"}},"lastState":{},"ready":true,"restartCount":0,"image":"rusintez/mock","imageID":"docker://ca5d0df8796cc6bc30808508084d3ed31b229aec6678ceaf52b5fe295adacac9","containerID":"docker://4f96a3af5476d8cd5d80269976dfa0defb4081fd5b57cc5d2bdcc3704985d951"}]}}
I1027 09:11:04.826794    9945 exec.go:127] defaulting container name to mock
F1027 09:11:04.861602    9945 helpers.go:72] error: Unable to upgrade
connection: {
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "tls: handshake did not verify certificate chain",
  "code": 500
}

[Bug 1274854] kubectl exec: handshake did not verify certificate chain

[Red Hat Bugzilla]

https://bugzilla.redhat.com/show_bug.cgi?id=1274854

--- Comment #2 from Jan Chaloupka  ---
The same happends with different pod:

# cat ubuntu.yaml
apiVersion: v1
kind: Pod
metadata:
  name: counter
spec:
  containers:
  - name: count
    image: ubuntu:14.04

# kubectl exec counter /bin/sh 
error: Unable to upgrade connection: {
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "tls: handshake did not verify certificate chain",
  "code": 500
}

[Bug 1274854] kubectl exec: handshake did not verify certificate chain

[Red Hat Bugzilla]

https://bugzilla.redhat.com/show_bug.cgi?id=1274854

--- Comment #3 from Jan Chaloupka  ---
Based on [1] I have tried to run the scenario with f22 builds (build with
golang-1.4.2-3.fc22.x86_64).

# kubectl exec counter /bin/sh -i
error: Error executing remote command: Error executing command in container:
container not found ("count")

tls error is gone.

[1]
http://stackoverflow.com/questions/32449316/kubernetes-on-mesos-tls-error-on...

[Bug 1274854] kubectl exec: handshake did not verify certificate chain

[Red Hat Bugzilla]

https://bugzilla.redhat.com/show_bug.cgi?id=1274854

--- Comment #4 from Jan Chaloupka  ---
When running the following:

# cat counter.yaml 
apiVersion: v1
kind: Pod
metadata:
  name: counter
spec:
  containers:
  - name: count
    image: ubuntu:14.04
    args: [bash, -c, 
           'for ((i = 0; ; i++)); do echo "$i: $(date)"; sleep 1; done']

# kubectl exec counter /bin/bash -i
uname
Linux
^C^C

Confirming exec command is working is supposed to.

This is another issue with go-1.5 compiler. I can try to rebuilt f23 with the
latest golang-1.5 which is golang-1.5.1-1.fc23. The f23 kubernetes is built
with golang-1.5-8.fc23.x86_64

[Bug 1274854] kubectl exec: handshake did not verify certificate chain

[Red Hat Bugzilla]

https://bugzilla.redhat.com/show_bug.cgi?id=1274854

--- Comment #5 from Jan Chaloupka  ---
k8s built with golang-1.5.1-1.fc23 suffers the same issue.

[Bug 1274854] kubectl exec: handshake did not verify certificate chain

[Red Hat Bugzilla]

https://bugzilla.redhat.com/show_bug.cgi?id=1274854

Jan Chaloupka  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
              Flags|                            |needinfo?([e-mail]
                   |                            |)

--- Comment #6 from Jan Chaloupka  ---
Jakub, Vincent,

what is the current state of golang in f23? How are we doing with TLS support?

[Bug 1274854] kubectl exec: handshake did not verify certificate chain

[Red Hat Bugzilla]

https://bugzilla.redhat.com/show_bug.cgi?id=1274854

Jan Chaloupka  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
              Flags|                            |needinfo?([e-mail]
                   |                            |)

[Bug 1274854] kubectl exec: handshake did not verify certificate chain

[Red Hat Bugzilla]

https://bugzilla.redhat.com/show_bug.cgi?id=1274854

Jan Chaloupka  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
         Depends On|                            |1276255

Referenced Bugs:

https://bugzilla.redhat.com/show_bug.cgi?id=1276255
[Bug 1276255] golang-1.5 breaks tls handshake certificate chain
verification

[Bug 1274854] kubectl exec: handshake did not verify certificate chain

[Red Hat Bugzilla]

https://bugzilla.redhat.com/show_bug.cgi?id=1274854

--- Comment #7 from Jan Chaloupka  ---
The issue is fixed in 1.2.0 version of kubernetes.

[Bug 1274854] kubectl exec: handshake did not verify certificate chain

[Red Hat Bugzilla]

https://bugzilla.redhat.com/show_bug.cgi?id=1274854

Jan Chaloupka  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
              Flags|needinfo?([e-mail] |
                   |)                           |
                   |needinfo?([e-mail] |
                   |)                           |

[Bug 1274854] kubectl exec: handshake did not verify certificate chain

[Red Hat Bugzilla]

https://bugzilla.redhat.com/show_bug.cgi?id=1274854
Bug 1274854 depends on bug 1276255, which changed state.

Bug 1276255 Summary: golang-1.5 breaks tls handshake certificate chain verification
https://bugzilla.redhat.com/show_bug.cgi?id=1276255

What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |CLOSED
         Resolution|---                         |NOTABUG

[Bug 1274854] kubectl exec: handshake did not verify certificate chain

[Red Hat Bugzilla]

https://bugzilla.redhat.com/show_bug.cgi?id=1274854

Jan Chaloupka  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|ASSIGNED                    |MODIFIED
   Fixed In Version|                            |kubernetes-1.1.0-0.17.git38
                   |                            |8061f.fc23

[Bug 1274854] kubectl exec: handshake did not verify certificate chain

[Red Hat Bugzilla]

https://bugzilla.redhat.com/show_bug.cgi?id=1274854

--- Comment #8 from Jan Chaloupka  ---
backported code from 1.2.0:

http://pkgs.fedoraproject.org/cgit/kubernetes.git/commit/?h=f23&id=b0821...

[Bug 1274854] kubectl exec: handshake did not verify certificate chain

[Red Hat Bugzilla]

https://bugzilla.redhat.com/show_bug.cgi?id=1274854

--- Comment #9 from Fedora Update System  ---
kubernetes-1.1.0-0.17.git388061f.fc23 has been submitted as an update to Fedora
23. https://bodhi.fedoraproject.org/updates/FEDORA-2015-c7fa818f3f

[Bug 1274854] kubectl exec: handshake did not verify certificate chain

[Red Hat Bugzilla]

https://bugzilla.redhat.com/show_bug.cgi?id=1274854

Fedora Update System  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|MODIFIED                    |ON_QA

--- Comment #10 from Fedora Update System  ---
kubernetes-1.1.0-0.17.git388061f.fc23 has been pushed to the Fedora 23 testing
repository. If problems still persist, please make note of it in this bug
report.
If you want to test the update, you can install it with
$ su -c 'dnf --enablerepo=updates-testing update kubernetes'
You can provide feedback for this update here:
https://bodhi.fedoraproject.org/updates/FEDORA-2015-c7fa818f3f

[Bug 1274854] kubectl exec: handshake did not verify certificate chain

[Red Hat Bugzilla]

https://bugzilla.redhat.com/show_bug.cgi?id=1274854

--- Comment #11 from Fedora Update System  ---
kubernetes-1.1.0-0.17.git388061f.fc23 has been pushed to the Fedora 23 stable
repository. If problems still persist, please make note of it in this bug
report.

[Bug 1274854] kubectl exec: handshake did not verify certificate chain

[Red Hat Bugzilla]

https://bugzilla.redhat.com/show_bug.cgi?id=1274854

Fedora Update System  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|ON_QA                       |CLOSED
         Resolution|---                         |ERRATA
        Last Closed|                            |2015-11-12 00:23:01