[atomic-wg] Issue #176 `status of kubernetes on fedora atomic 25`

[Pagure]

jasonbrooks reported a new issue against the project: `atomic-wg` that you are following:
``
We removed kubernetes, etcd, flannel, gluster and ceph from the atomic host for Fedora 25:
https://pagure.io/fedora-atomic/c/219c9bb26426811a5f32188c59682ad70c3283e...

The idea is that these pkgs can/should be run as docker containers or system containers. I
wrote a post about running kube in containers on CentOS Atomic, but things work the same
for Fedora Atomic:
http://www.projectatomic.io/blog/2016/09/running-kubernetes-in-containers...

However, we don't currently have fedora containers for kubernetes. There's a
year-old PR for this: https://github.com/fedora-cloud/Fedora-Dockerfiles/pull/112. I used
this PR as the basis for kube containers for CentOS:
https://github.com/CentOS/CentOS-Dockerfiles/tree/master/kubernetes. 

Another option is using package layering to install the removed rpms if desired:
http://www.projectatomic.io/blog/2016/07/hacking-and-extending-atomic-host/. This works,
mostly, but the kube-apiserver that comes with kubernetes-master is granted
CAP_NET_BIND_SERVICE so it can use port 443, and package layering won't support this:
https://github.com/projectatomic/rpm-ostree/issues/462.

Whether we install kubernetes via package layering or containers, another issue is that
the kube we have for fedora 25 is very old, 1.2, and current is 1.4.5. There's a kube
1.4.5 in koji for fc26 (http://koji.fedoraproject.org/koji/buildinfo?buildID=816481), and
I've asked Jan about making it available for fedora 25. I built it for f25 and el7 in
this copr: https://copr.fedorainfracloud.org/coprs/jasonbrooks/kubernetes/.

kubeadm (http://kubernetes.io/docs/getting-started-guides/kubeadm/) is a slick way to run
an up-to-date kube cluster, and the kubernetes project offers rpms for kubeadm, but the
kubernetes-cni package they provide doesn't work with atomic. I made a patched package
in a copr for f25 and el7
(https://copr.fedorainfracloud.org/coprs/jasonbrooks/kube-release/) and wrote about it
here:
https://jebpages.com/2016/11/01/installing-kubernetes-on-centos-atomic-ho....
The rpm grabs the binaries from the upstream project, if fedora were to provide this
package, we'd probably want to build the binaries ourselves. kubeadm runs kubernetes
and etcd in containers, and these containers aren't fedora-based, and they come from
upstream. They appear to be based on busybox. Also, kubeadm is considered alpha at this
point.

We should:

* try to move to a more recent kubernetes version
* get kubernetes containers in place for fedora
* get fedora versions of the flannel and etcd system containers in place
* document how to install kubernetes and how to use package layering on fedora
* look further into kubeadm 

``

To reply, visit the link below or just reply to this email
https://pagure.io/atomic-wg/issue/176

[Pagure]

dustymabe added a new comment to an issue you are following:
``

>  There's a kube 1.4.5 in koji for fc26
> (http://koji.fedoraproject.org/koji/buildinfo?buildID=816481), and I've asked Jan
> about making it available for fedora 25.
>  
Can you keep us in the loop here on the progress of this one? 
``

To reply, visit the link below or just reply to this email
https://pagure.io/atomic-wg/issue/176

[Pagure]

jasonbrooks added a new comment to an issue you are following:
``
>  
>  There's a kube 1.4.5 in koji for fc26
> (http://koji.fedoraproject.org/koji/buildinfo?buildID=816481), and I've asked Jan
> about making it available for fedora 25.
>  
>  Can you keep us in the loop here on the progress of this one?
>  
Will do
``

To reply, visit the link below or just reply to this email
https://pagure.io/atomic-wg/issue/176

[Pagure]

dwalsh added a new comment to an issue you are following:
``
I have asked Giuseppe Scrivano <gscrivan(a)redhat.com&gt; to move forward on system
containers to implement kubernetes workflow on atomic host.  He currently has most of
services available as system containers and is moving them into
github.com/projectatomic/atomic-system-containers

We need to get these containers built for Fedora 25.

He even has an experimental system container that runs docker in it.
``

To reply, visit the link below or just reply to this email
https://pagure.io/atomic-wg/issue/176

[Pagure]

gscrivano added a new comment to an issue you are following:
``
There are already two system containers for etcd and flannel, already available on Docker
hub as gscrivano/etcd and gscrivano/flannel.  I'll move them to Fedora as soon as you
tell me how to proceed for doing that.   There is no system container for Kubernetes yet,
so we will need to run it in Docker
``

To reply, visit the link below or just reply to this email
https://pagure.io/atomic-wg/issue/176

[Pagure]

dustymabe added a new comment to an issue you are following:
``
as an alternative for now I have been working on getting the openshift-ansible installer
to work against F25 atomic host. I have opened up several issuses:

https://github.com/openshift/openshift-ansible/issues/2808
https://github.com/openshift/openshift-ansible/issues/2809
https://github.com/openshift/openshift-ansible/issues/2811
``

To reply, visit the link below or just reply to this email
https://pagure.io/atomic-wg/issue/176

[Pagure]

jberkus added a new comment to an issue you are following:
``
Well, OpenShift is its own thing rather than an alternative to Kubernetes.  Some people
want Kube, some want OpenShift.

So I've been doing a bunch with Kubeadm on AH.  Jason's version does actually
work, with some caveats:

1) it requires package layering, which really pushes up the idea of having a way for users
to build their own OStree servers.

2) it doesn't work with setenforce=1.  Even relabelling a bunch of directories, I have
to setenforce 0 to get kubernetes to work reliably.

If we can get over those humps, though, I 'd like to push forward with something based
on Kubeadm.  I think it's the way the kubernetes project is headed, and it means that
we can track them for anyone who wants to use kube-latest on Atomic.  Importantly, the
kubernetes community will continue to add features to kubeadm (like let's encrypt
support), which won't happen with solutions we devise.

One of the things I was wondering about is maybe installing the kubeadm packages as part
of Atomic.  My reasoning is this: kubeadm is an installer rather than kubernetes itself. 
You can, in fact, use it to install an older/stable version of Kubernetes (back to 1.4.0,
though), so we could track upstream Kubeadm without breaking people's stuff.  Also, if
not enabled in systemd (which it wouldn't be, by default), it doesn't interfere
with installing something else like OpenShift.



``

To reply, visit the link below or just reply to this email
https://pagure.io/atomic-wg/issue/176

[Pagure]

jasonbrooks added a new comment to an issue you are following:
``

>  1) it requires package layering, which really pushes up the idea of
> having a way for users to build their own OStree servers.
>  
What do you mean by that? Package layering doesn't require someone to build their own
ostree server, it does require installing the layered pkgs separately on each host,
though, and rebooting in between, although Colin had thoughts on cutting out the reboot
step, which would make it nicer.

We could also offer multiple trees, some with fewer things rolled in by default, some with
more.

And we could run these bits as system containers, like the etcd and flannel ones. I
don't understand exactly what's involved in making those, in particular the bits
about creating the config.json.template. It seems that this project could help w/ that
part: https://github.com/jessfraz/riddler but I haven't been able to figure out how to
build it to use it.
``

To reply, visit the link below or just reply to this email
https://pagure.io/atomic-wg/issue/176

[Pagure]

jberkus added a new comment to an issue you are following:
``
Well, the whole point of Atomic is to be immutable: "configure once, deploy many
times".  Having a key piece of infrastructure depend on per-server package layering
kinda breaks that.  It makes it hard to explain why people should use Atomic in the first
place.

Offering multiple trees would definitely help, but do we have the ability to do that?  I
mean, we have issues with the one tree on a fairly regular basis.

Rebuilding kubeadm as system containers would be *awesome* if we could really do that. 
Weren't there technical issues with running Kubelet in a container though?
``

To reply, visit the link below or just reply to this email
https://pagure.io/atomic-wg/issue/176

[Pagure]

jasonbrooks added a new comment to an issue you are following:
``
kube 1.4.5 for f25 in bodhi:
https://bodhi.fedoraproject.org/updates/FEDORA-2016-fffea4b1c3

I've been working on getting it running in rpm-installed node pkgs / master pkgs in
containers. Notes on that here:
https://gist.github.com/jasonbrooks/bdbb0b142fc82bbd7b33759b3d934ba0
``

To reply, visit the link below or just reply to this email
https://pagure.io/atomic-wg/issue/176

[Pagure]

jasonbrooks added a new comment to an issue you are following:
``
Some updates:

* the issue of fedora having an aged kube is resolved, v1.4.5 is in f25 stable
* the issue of not being able to just `rpm-ostree install kubernetes` and proceed as if we
never removed kube from the image is open. This won't work because [rpm-ostree
can't deal](: https://github.com/projectatomic/rpm-ostree/issues/462) w/ the
kube-apiserver binary in kubernetes-master
* the issue of running kube in containers on fedora atomic is in progress:
    * I've been working w/ these [kube
containers](https://github.com/jasonbrooks/k8s-images/tree/f25), and these
[etcd](https://github.com/projectatomic/atomic-system-containers/tree/mast... and
[flannel](https://github.com/projectatomic/atomic-system-containers/tree/m...
containers. They work (but need [this
PR](https://github.com/projectatomic/atomic-system-containers/pulls)), and I'm
adapting the [upstream ansible
scripts](https://github.com/kubernetes/contrib/tree/master/ansible) to use them.
    * I need to get those kube, etcd and flannel containers into the new fedora build
system, and could use help with that.
    * I need to get the ansible changes finished and upstreamed, and that ought to be
straightforward, but I could use feedback on my PR once I get it together.
``

To reply, visit the link below or just reply to this email
https://pagure.io/atomic-wg/issue/176

[Pagure]

dustymabe added a new comment to an issue you are following:
``
kubernetes rpms are in the install tree now: 
https://pagure.io/fedora-atomic/pull-request/39

the images produced tonight should have the content in there. Let's close this issue
if testing is successful there. 
``

To reply, visit the link below or just reply to this email
https://pagure.io/atomic-wg/issue/176

[Pagure]

dustymabe added a new comment to an issue you are following:
``
new release is out with k8s rpms in it:
https://lists.fedoraproject.org/archives/list/[e-mail].o...
``

To reply, visit the link below or just reply to this email
https://pagure.io/atomic-wg/issue/176