[atomic-wg] Issue #510: Openshift Origin 3.9 fails with error message "Console install failed" on F29 Atomic Host nightly
sinnykumari reported a new issue against the project: `atomic-wg` that you are following: `` I tried to set-up basic 3 node OpenShift cluster using playbook playbooks/deploy_cluster.yml from openshift-ansible/openshift-ansible-3.9.41-1 on Fedora 29 Atomic Host latest nightly AMIs (Fedora-AtomicHost-28-20180902.0.x86_64).
It fails during task run [openshift_web_console : Verify that the console is running] with error message "Console install failed."
After doing ssh into master node and running`oc get nodes` says: ``` $ oc get nodes NAME STATUS ROLES AGE VERSION node1 NotReady master 2h v1.9.1+a0ce1bc657 node2 NotReady compute 2h v1.9.1+a0ce1bc657 node3 NotReady compute 2h v1.9.1+a0ce1bc657 ``` Running kubectl on master node gives following failure message:
``` $ kubectl describe node node1 ... Ready False Wed, 05 Sep 2018 12:11:39 +0000 Wed, 05 Sep 2018 10:08:02 +0000 KubeletNotReady Failed to start ContainerManager Delegation not available for unit type ... ```
Following package versions are available on F29 AH nightly: runc-1.0.0-50.dev.git20aff4f.fc29.x86_64 systemd-239-3.fc29.x86_64 docker-1.13.1-62.git9cb56fd.fc29.x86_64
`Note:` Same set-up works fine with latest F28 Atomic Host
``
To reply, visit the link below or just reply to this email https://pagure.io/atomic-wg/issue/510
dustymabe added a new comment to an issue you are following: ``
Note we had the [same problem in f28](https://pagure.io/atomic-wg/issue/452) and the systemd change was [reverted](https://pagure.io/atomic-wg/issue/452#comment-506779) in order to allow the upstreams (runc/kube) to catch up.
There are three questions here: - Is `runc-1.0.0-50.dev.git20aff4f.fc29` new enough? - Is origin 3.9 new enough? - is origin 3.10 new enough?
cc @mpatel @gscrivano ``
To reply, visit the link below or just reply to this email https://pagure.io/atomic-wg/issue/510
mpatel added a new comment to an issue you are following: `` runc is new enough. I am not sure about origin version. You would need to go and check if they vendor the libcontainer with the runc patch. ``
To reply, visit the link below or just reply to this email https://pagure.io/atomic-wg/issue/510
dustymabe added a new comment to an issue you are following: `` ok so I just looked. [origin v3.10.0](https://github.com/openshift/origin/blob/v3.10.0/glide.yaml#L79-L82) and [origin v3.11.0-alpha.0](https://github.com/openshift/origin/blob/v3.11.0-alpha.0/glide.yaml#L79-L82) are currently using [openshift-runc@da78c1f](https://github.com/openshift/opencontainers-runc/commit/da78c1f) which does not include [the fix](https://github.com/opencontainers/runc/commit/59c60d1) mentioned in https://github.com/opencontainers/runc/pull/1776
@sjenning - is there anything we can do to get that fix pulled in? ``
To reply, visit the link below or just reply to this email https://pagure.io/atomic-wg/issue/510
sjenning added a new comment to an issue you are following: `` v3.11 is in code freeze right now so we'd need to wait for a 3.11 z-stream. Is this to enable something that is different on Fedora vs RHEL? ``
To reply, visit the link below or just reply to this email https://pagure.io/atomic-wg/issue/510
dustymabe added a new comment to an issue you are following: `` it turns out the [release-3.11 branch](https://github.com/openshift/origin/blob/release-3.11/glide.yaml#L70-L73) has a newer commit which includes [the fix](https://github.com/opencontainers/runc/commit/59c60d1) so we should be good for 3.11 once it is released. I'll confirm this with a recent build of origin 3.11. ``
To reply, visit the link below or just reply to this email https://pagure.io/atomic-wg/issue/510
dustymabe added a new comment to an issue you are following: `` @jcajka FYI ^^ maybe we need to jump straight to 3.11 in f29 origin rpm. ``
To reply, visit the link below or just reply to this email https://pagure.io/atomic-wg/issue/510
dustymabe added a new comment to an issue you are following: `` FYI for now you can try oc cluster up with: `oc cluster up --image='quay.io/openshift/origin-${component}:v3.11'` ``
To reply, visit the link below or just reply to this email https://pagure.io/atomic-wg/issue/510
jcajka added a new comment to an issue you are following: `` @dustymabe awesome I have been pulling my hairs over it for past months. I'm not sure how masochistic I want to be and just stay on 3.10 with backports or just stop caring an rebase to 3.11 or even beyond(as I havnen't even done the container images as upstream have renamed bunch of them, finishing 3.6/3.9 f27/f28 respectively now with old names). ``
To reply, visit the link below or just reply to this email https://pagure.io/atomic-wg/issue/510
jcajka added a new comment to an issue you are following: `` @dustymabe maybe best question is what will you/we be missing if we stay with 3.10(patched) in f29? ``
To reply, visit the link below or just reply to this email https://pagure.io/atomic-wg/issue/510
jcajka added a new comment to an issue you are following: `` Oh, and other issue is if upstream will ever backport the fix and rebuild the images for the 3.10...(I don't plan to switch to fedora built image yet). ``
To reply, visit the link below or just reply to this email https://pagure.io/atomic-wg/issue/510
jcajka added a new comment to an issue you are following: `` Pondering it more I feel that calling 3.10 fubar and moving to the 3.11 will be probably best course of actions. Any comments? ``
To reply, visit the link below or just reply to this email https://pagure.io/atomic-wg/issue/510
sinnykumari added a new comment to an issue you are following: ``
FYI for now you can try oc cluster up with: oc cluster up --image='quay.io/openshift/origin-${component}:v3.11'
Ran it on F29 AH beta 1.5 compose and it gives error `panic: assignment to entry in nil map`. Does it need some additional stuff to do?
Complete logs are available below:
``` [fedora@ip-10-0-0-162 ~]$ sudo oc cluster up --image='quay.io/openshift/origin-${component}:v3.11' Getting a Docker client ... Checking if image quay.io/openshift/origin-control-plane:v3.11 is available ... Pulling image quay.io/openshift/origin-control-plane:v3.11 Pulled 1/5 layers, 21% complete Pulled 2/5 layers, 42% complete Pulled 3/5 layers, 72% complete Pulled 4/5 layers, 84% complete Pulled 4/5 layers, 94% complete Pulled 5/5 layers, 100% complete Extracting Image pull complete Pulling image quay.io/openshift/origin-cli:v3.11 Image pull complete Pulling image quay.io/openshift/origin-node:v3.11 Pulled 5/6 layers, 85% complete Pulled 6/6 layers, 100% complete Extracting Image pull complete Checking type of volume mount ... Determining server IP ... Checking if OpenShift is already running ... Checking for supported Docker version (=>1.22) ... Checking if insecured registry is configured properly in Docker ... Checking if required ports are available ... Checking if OpenShift client is configured properly ... Checking if image quay.io/openshift/origin-control-plane:v3.11 is available ... Starting OpenShift using quay.io/openshift/origin-control-plane:v3.11 ... I0924 10:41:54.773567 1487 config.go:42] Running "create-master-config" panic: assignment to entry in nil map
goroutine 1 [running]: github.com/openshift/origin/pkg/oc/clusterup/coreinstall/kubeapiserver.KubeAPIServerStartConfig.MakeMasterConfig(0xc000b6c570, 0x2c, 0xc000765e60, 0x6, 0x6, 0x2f32360, 0xc000446480, 0xc000e179b0, 0x2a, 0x0, ...) /builddir/build/BUILD/origin-dd10d172758d4d02f6d2e24869234fac6c7841a7/_output/local/go/src/github.com/openshift/origin/pkg/oc/clusterup/coreinstall/kubeapiserver/config.go:91 +0xba6 github.com/openshift/origin/pkg/oc/clusterup.(*ClusterUpConfig).makeMasterConfig(0xc000229900, 0xc0005de4e0, 0x1, 0x0, 0x2eca6a0) /builddir/build/BUILD/origin-dd10d172758d4d02f6d2e24869234fac6c7841a7/_output/local/go/src/github.com/openshift/origin/pkg/oc/clusterup/run_self_hosted.go:365 +0x4dd github.com/openshift/origin/pkg/oc/clusterup.(*ClusterUpConfig).BuildConfig(0xc000229900, 0xc000b6c4e0, 0x2c, 0xd) /builddir/build/BUILD/origin-dd10d172758d4d02f6d2e24869234fac6c7841a7/_output/local/go/src/github.com/openshift/origin/pkg/oc/clusterup/run_self_hosted.go:278 +0xe9b github.com/openshift/origin/pkg/oc/clusterup.(*ClusterUpConfig).StartSelfHosted(0xc000229900, 0x2eca660, 0xc00000e018, 0x20, 0xc000c9db30) /builddir/build/BUILD/origin-dd10d172758d4d02f6d2e24869234fac6c7841a7/_output/local/go/src/github.com/openshift/origin/pkg/oc/clusterup/run_self_hosted.go:127 +0x43 github.com/openshift/origin/pkg/oc/clusterup.(*ClusterUpConfig).Start(0xc000229900, 0x2eca660, 0xc00000e018, 0x0, 0x4) /builddir/build/BUILD/origin-dd10d172758d4d02f6d2e24869234fac6c7841a7/_output/local/go/src/github.com/openshift/origin/pkg/oc/clusterup/up.go:473 +0x128 github.com/openshift/origin/pkg/oc/clusterup.NewCmdUp.func1(0xc000229b80, 0xc000254710, 0x0, 0x1) /builddir/build/BUILD/origin-dd10d172758d4d02f6d2e24869234fac6c7841a7/_output/local/go/src/github.com/openshift/origin/pkg/oc/clusterup/up.go:112 +0xe7 github.com/openshift/origin/vendor/github.com/spf13/cobra.(*Command).execute(0xc000229b80, 0xc0002546f0, 0x1, 0x1, 0xc000229b80, 0xc0002546f0) /builddir/build/BUILD/origin-dd10d172758d4d02f6d2e24869234fac6c7841a7/_output/local/go/src/github.com/openshift/origin/vendor/github.com/spf13/cobra/command.go:757 +0x2cc github.com/openshift/origin/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xc0006a1900, 0x17717f1, 0xc00029a200, 0xc0006a1900) /builddir/build/BUILD/origin-dd10d172758d4d02f6d2e24869234fac6c7841a7/_output/local/go/src/github.com/openshift/origin/vendor/github.com/spf13/cobra/command.go:843 +0x2fd github.com/openshift/origin/vendor/github.com/spf13/cobra.(*Command).Execute(0xc0006a1900, 0x2, 0xc0006a1900) /builddir/build/BUILD/origin-dd10d172758d4d02f6d2e24869234fac6c7841a7/_output/local/go/src/github.com/openshift/origin/vendor/github.com/spf13/cobra/command.go:791 +0x2b main.main() /builddir/build/BUILD/origin-dd10d172758d4d02f6d2e24869234fac6c7841a7/_output/local/go/src/github.com/openshift/origin/cmd/oc/oc.go:42 +0x2e5 [fedora@ip-10-0-0-162 ~]$ echo $? 2 ``` ``
To reply, visit the link below or just reply to this email https://pagure.io/atomic-wg/issue/510
dustymabe added a new comment to an issue you are following: ``
@jcajka Pondering it more I feel that calling 3.10 fubar and moving to the 3.11 will be probably best course of actions. Any comments?
:thumbsup: ``
To reply, visit the link below or just reply to this email https://pagure.io/atomic-wg/issue/510
dustymabe added a new comment to an issue you are following: ``
@sinnykumari Ran it on F29 AH beta 1.5 compose and it gives error panic: assignment to entry in nil map. Does it need some additional stuff to do?
other than setting up the insecure registry (which it looks like you've already done) I don't think I had to do anything else. The command I gave you is operating against the release branch for 3.11 (i.e. moving target) so it could have changed since I ran it. I'll try today and see if I get the same results ``
To reply, visit the link below or just reply to this email https://pagure.io/atomic-wg/issue/510
jcajka added a new comment to an issue you are following: `` @sinnykumari I'm getting the same with 3.10 binaries and 3.11 images. As this is not supported configuration by upstream. I would guess it is caused by too old binaries compared to the images. I'm atm re-basing(finding what have been changed, split off) rawhide origin to 3.11, please stand by. ``
To reply, visit the link below or just reply to this email https://pagure.io/atomic-wg/issue/510
sinnykumari added a new comment to an issue you are following: ``
@sinnykumari I'm getting the same with 3.10 binaries and 3.11 images. As this is not supported configuration by upstream. I would guess it is caused by too old binaries compared to the images. I'm atm re-basing(finding what have been changed, split off) rawhide origin to 3.11, please stand by.
+1 ``
To reply, visit the link below or just reply to this email https://pagure.io/atomic-wg/issue/510
dustymabe added a new comment to an issue you are following: `` @jcajka - 3.11.alpha1 isn't new enough :( but the active 3.11 branch is new enough (i.e. we have to wait for the next 3.11 tag to get created).
read carefully my [first comment](https://pagure.io/atomic-wg/issue/510#comment-531340) and [followup comment](https://pagure.io/atomic-wg/issue/510#comment-531542) that explains this. ``
To reply, visit the link below or just reply to this email https://pagure.io/atomic-wg/issue/510
dustymabe added a new comment to an issue you are following: ``
@dustymabe So I have re-tested in clean enviroment with the upstream images and everything seems to work. I would like to have it confirm by you before I will move forward with f29.
how did you test? when I did it before I just ran `oc cluster up --image='quay.io/openshift/origin-${component}:v3.11'` ``
To reply, visit the link below or just reply to this email https://pagure.io/atomic-wg/issue/510
dustymabe added a new comment to an issue you are following: ``
how did you test? when I did it before I just ran oc cluster up --image='quay.io/openshift/origin-${component}:v3.11'
Note that when I did this I also pulled the `quay.io/openshift/origin-node:v3.11` container first and copied the oc binary out of it and into `/usr/local/bin` on the host and used that to do the `oc cluster up` to make sure i got the same version ``
To reply, visit the link below or just reply to this email https://pagure.io/atomic-wg/issue/510
sinnykumari added a new comment to an issue you are following: `` I tested in following ways on a F29 AH machine (ostree version - 29.20180926.n.0) and it seems to work fine. Installed [origin-clients from F30](https://koji.fedoraproject.org/koji/buildinfo?buildID=1147868) which @jcajka built. ``` $ sudo rpm-ostree install https://kojipkgs.fedoraproject.org//packages/origin/3.11.0/0.alpha1.0.fc30/x... ``` Updated insecure registries in /etc/containers/registries.conf to 172.30.0.0/16 and then ` sudo oc cluster up`.
@dustymabe origin source tar used in origin-clients-3.11.0-0.alpha1.0.fc30 package is from [commit 777c966dcc84fba1d32ec928831869ad56fe3e38](https://src.fedoraproject.org/rpms/origin/blob/master/f/origin.spec#_24), which contains [opencontainer-runc with fix](https://github.com/openshift/origin/blob/777c966dcc84fba1d32ec928831869ad56f...). The same opencontainer-runc version is included in [origin release 3.11 branch](https://github.com/openshift/origin/blob/release-3.11/glide.yaml#L70-L73)
``
To reply, visit the link below or just reply to this email https://pagure.io/atomic-wg/issue/510
dustymabe added a new comment to an issue you are following: ``
@dustymabe origin source tar used in origin-clients-3.11.0-0.alpha1.0.fc30 package is from commit 777c966dcc84fba1d32ec928831869ad56fe3e38, which contains opencontainer-runc with fix. The same opencontainer-runc version is included in origin release 3.11 branch
cool, didn't know that. Thanks @sinnykumari. @jcajka can you trust that sinny's testing? I don't know if I'll have time to run through this soon. ``
To reply, visit the link below or just reply to this email https://pagure.io/atomic-wg/issue/510
sinnykumari added a new comment to an issue you are following: `` @jcajka @dustymabe F29 origin-3.11.0-0.alpha1.0.fc29 build works as expected. Added [karma](https://bodhi.fedoraproject.org/updates/origin-3.11.0-0.alpha1.0.fc29) for same. ``
To reply, visit the link below or just reply to this email https://pagure.io/atomic-wg/issue/510
The status of the issue: `Openshift Origin 3.9 fails with error message "Console install failed" on F29 Atomic Host nightly` of project: `atomic-wg` has been updated to: Closed as Fixed by sinnykumari.
sinnykumari added a new comment to an issue you are following: `` origin-3.11.0-0.alpha1.0.fc29 has been pushed to f29. openshift cluster set-up works as expected with openshift-ansible-3.11.21-1-2-g8d402991for 3.11on latest Fedora 29 Atomic Host (ostree version: 29.20181008.n.0).
Thanks all! ``
To reply, visit the link below or just reply to this email https://pagure.io/atomic-wg/issue/510
dustymabe added a new comment to an issue you are following: `` thanks @sinnykumari @jcajka ! ``
To reply, visit the link below or just reply to this email https://pagure.io/atomic-wg/issue/510
atomic@lists.fedoraproject.org
-
Dusty Mabe -
Jakub Čajka
-
Mrunal Patel
-
Seth Jennings
-
Sinny Kumari