11:03 p.m.
Hi all,
We have put a number of SOPs together, related to Openshift 4, installation
and configuration on Fedora Infra, we are hoping to get some feedback!
If you get a minute please check the following:
https://pagure.io/infra-docs-fpo/pull-request/8
--
David Kirwan
Software Engineer
Community Platform Engineering @ Red Hat
T: +(353) 86-8624108 IM: @dkirwan
6:12 p.m.
On Wed, Sep 22, 2021 at 01:03:52PM +0900, David Kirwan wrote:
Hi all,
We have put a number of SOPs together, related to Openshift 4, installation
and configuration on Fedora Infra, we are hoping to get some feedback!
If you get a minute please check the following:
https://pagure.io/infra-docs-fpo/pull-request/8
Looks pretty nice to me. Thanks for the detailed docs. ;)
I do have some other questions, might as well throw them here:
* On the storage, are we ok if a node goes down? ie, does it spread it
over all the storage nodes/raid? Or is it just in one place and you are
dead if that node dies?
* Is there any way to backup volumes?
* should we make a playbooks/manual/ocp.yml playbook for things like
- list of clusteradmins
- list of clustermoniting
- anything else we want to manage post install
* Have we tried a upgrade of the clusters yet? Did everything go ok?
Do we need any docs on upgrades?
* Since the control plane are vm's I assume we need to drain them one at
a time to reboot the virthosts they are on?
* Should we now delete the kubeadmin user? In 3.x I know they advise to
do that after auth is setup.
* Right now the api is only internal. Is it worth getting a forward
setup to allow folks to use oc locally on their machines? It would
expose that api to the world, but of course it would still need auth.
* Do we want to try and enable http/2 ingress?
https://docs.openshift.com/container-platform/4.5/networking/ingress-oper...
* We will want to enable kubevirt/whatever it's called...
Thanks for the great docs. +1 on them.
kevin
7:55 p.m.
On Wed, Sep 22, 2021 at 7:12 PM Kevin Fenzi <kevin(a)scrye.com> wrote:
On Wed, Sep 22, 2021 at 01:03:52PM +0900, David Kirwan wrote:
> Hi all,
>
> We have put a number of SOPs together, related to Openshift 4, installation
> and configuration on Fedora Infra, we are hoping to get some feedback!
>
> If you get a minute please check the following:
> https://pagure.io/infra-docs-fpo/pull-request/8
Looks pretty nice to me. Thanks for the detailed docs. ;)
I do have some other questions, might as well throw them here:
* On the storage, are we ok if a node goes down? ie, does it spread it
over all the storage nodes/raid? Or is it just in one place and you are
dead if that node dies?
* Is there any way to backup volumes?
* should we make a playbooks/manual/ocp.yml playbook for things like
- list of clusteradmins
- list of clustermoniting
- anything else we want to manage post install
* Have we tried a upgrade of the clusters yet? Did everything go ok?
Do we need any docs on upgrades?
* Since the control plane are vm's I assume we need to drain them one at
a time to reboot the virthosts they are on?
* Should we now delete the kubeadmin user? In 3.x I know they advise to
do that after auth is setup.
I'm not sure that's a good idea. I'm not even certain that was a good
idea in the OCP 3.x days, because eliminating the kubeadmin user means
you lose your failsafe login if all else fails.
* Right now the api is only internal. Is it worth getting a forward
setup to allow folks to use oc locally on their machines? It would
expose that api to the world, but of course it would still need auth.
* Do we want to try and enable http/2 ingress?
https://docs.openshift.com/container-platform/4.5/networking/ingress-oper...
Maybe? I think if we want to have more HA-deployed services, it would
make sense to do so.
* We will want to enable kubevirt/whatever it's called...
I think it's formally called OpenShift Virtualization, but yeah, it's KubeVirt.
Thanks for the great docs. +1 on them.
These are awesome, indeed!
--
真実はいつも一つ！/ Always, there's only one truth!
12:44 a.m.
On 23/09/2021 02:55, Neal Gompa wrote:
On Wed, Sep 22, 2021 at 7:12 PM Kevin Fenzi <kevin(a)scrye.com>
wrote:
<snip>
>
> * Since the control plane are vm's I assume we need to drain them one at
> a time to reboot the virthosts they are on?
Correct
>
> * Should we now delete the kubeadmin user? In 3.x I know they advise to
> do that after auth is setup.
>
I'm not sure that's a good idea. I'm not even certain that was a good
idea in the OCP 3.x days, because eliminating the kubeadmin user means
you lose your failsafe login if all else fails.
+1 here : the reason why we decided to still keep kubeadmin on the other
OCP clusters used for CentOS CI and Stream is exactly for that reason :
still be able to login, if there is a problem with the oauth setup, and
troubleshoot issues if (for example) ipsilon or IPA have troubles ... :-)
> * Right now the api is only internal. Is it worth getting a forward
> setup to allow folks to use oc locally on their machines? It would
> expose that api to the world, but of course it would still need auth.
That's what we decided to do for the CentOS CI ocp setup, and so CI
tenants can use oc from their laptop/infra. As long as cert exposed for
default ingress has it added in the SAN, it works fine :
X509v3 Subject Alternative Name:
DNS:*.apps.ocp.ci.centos.org, DNS:api.ocp.ci.centos.org,
DNS:apps.ocp.ci.centos.org
--
Fabian Arrotin
gpg key: 17F3B7A1 | twitter: @arrfab
2:22 a.m.
On 2021-09-23 07:44, Fabian Arrotin wrote:
On 23/09/2021 02:55, Neal Gompa wrote:
> On Wed, Sep 22, 2021 at 7:12 PM Kevin Fenzi <kevin(a)scrye.com> wrote:
>>
>> * Should we now delete the kubeadmin user? In 3.x I know they advise
>> to
>> do that after auth is setup.
>>
>
> I'm not sure that's a good idea. I'm not even certain that was a good
> idea in the OCP 3.x days, because eliminating the kubeadmin user means
> you lose your failsafe login if all else fails.
+1 here : the reason why we decided to still keep kubeadmin on the
other
OCP clusters used for CentOS CI and Stream is exactly for that reason :
still be able to login, if there is a problem with the oauth setup, and
troubleshoot issues if (for example) ipsilon or IPA have troubles ...
:-)
You can still use the kubeconfig from the installer (if you kept it) to
login with system:admin in case everything else fail.
5:23 p.m.
On Thu, Sep 23, 2021 at 07:44:49AM +0200, Fabian Arrotin wrote:
On 23/09/2021 02:55, Neal Gompa wrote:
> On Wed, Sep 22, 2021 at 7:12 PM Kevin Fenzi <kevin(a)scrye.com> wrote:
<snip>
>>
>> * Since the control plane are vm's I assume we need to drain them one at
>> a time to reboot the virthosts they are on?
Correct
>>
>> * Should we now delete the kubeadmin user? In 3.x I know they advise to
>> do that after auth is setup.
>>
>
> I'm not sure that's a good idea. I'm not even certain that was a good
> idea in the OCP 3.x days, because eliminating the kubeadmin user means
> you lose your failsafe login if all else fails.
+1 here : the reason why we decided to still keep kubeadmin on the other
OCP clusters used for CentOS CI and Stream is exactly for that reason :
still be able to login, if there is a problem with the oauth setup, and
troubleshoot issues if (for example) ipsilon or IPA have troubles ... :-)
We can keep it if folks like. I'd really prefer we don't use it except
for emergency though. Having people do things as their user will make it
way easier to see who did what. ;)
>> * Right now the api is only internal. Is it worth getting a
forward
>> setup to allow folks to use oc locally on their machines? It would
>> expose that api to the world, but of course it would still need auth.
That's what we decided to do for the CentOS CI ocp setup, and so CI
tenants can use oc from their laptop/infra. As long as cert exposed for
default ingress has it added in the SAN, it works fine :
X509v3 Subject Alternative Name:
DNS:*.apps.ocp.ci.centos.org, DNS:api.ocp.ci.centos.org,
DNS:apps.ocp.ci.centos.org
Yeah, thats all fine, but to make it work for our setup, I would need to
get RHIT to nat in port 6443 to proxy01/10 from the internet. At least I
think thats the case. Openshift 3 could just use https, but alas, I fear
OCP4 needs that 6443 port.
kevin
10:01 p.m.
On the storage, are we ok if a node goes down? ie, does it spread it
over all the storage nodes/raid? Or is it just in one place and you are
dead if that node dies?
For storage, we maintain 3 replicas for data, spread across
3 nodes.
However, not all nodes are equally resourced, we have 2 large nodes and 1
much smaller, therefore, more replicas will be spread over these 2 larger
nodes. We can likely afford to lose one of the large physical nodes while
still maintaining data integrity.
Is there any way to backup volumes?
There are ways to clone,
extend, take snapshots etc of these volumes. We've
never done it, so it'll be a learning process for us all ;). We should sync
to get a better handle on the requirements for backups. In CentOS CI we've
set up backups to S3, we can certainly use some of that, eg: backup of
etcd, but may need further investigation to backup the volumes managed by
OCS. Will need to do some research here.
should we make a playbooks/manual/ocp.yml playbook for things like
- list of clusteradmins
- list of clustermoniting
- anything else we want to manage post install
Sure yep, as we're finishing up
soonish, I'd imagine the next few weeks
we'll all be back focused on the Infra/Releng tasks and will be focusing on
tying up any loose ends like this, and starting migration of apps.
Have we tried a upgrade of the clusters yet? Did everything go ok?
Do we need any docs on upgrades?
Yes, we've already completed a number of
upgrades, latest is to 4.8.11. We
have SOPs for upgrades which we can copy over from the CentOS CI infra, and
will make any updates required in the process.
Since the control plane are vm's I assume we need to drain them
one at
a time to reboot the virthosts they are on?
If we are rebooting a single
vmhost/control plane VM at a time, yes that
should be good. If we are doing more than 1 at the same time, we should do
a full graceful cluster shutdown, and then a graceful cluster startup. We
have SOPs for this in CentOS CI also, we'll get those added here and any
content updates made.
* Should we now delete the kubeadmin user? In 3.x I know they advise
to
do that after auth is setup.
We can delete it, as we have system:admin available
from the os-control01
node. Best practices might suggest we do. We can also give cluster-admin
role to all users in the sysadmin-main and sysadmin-openshift groups.
I'm in two minds about deleting it, I was hoping to wait until we get a
solution that syncs IPA groups/users to Openshift. There is an official
supported solution for syncing LDAP (think that will work?).
* Right now the api is only internal. Is it worth getting a forward
setup to allow folks to use oc locally on their machines? It would
expose that api to the world, but of course it would still need auth.
We'd love
to expose it, but.. all interaction with the clusters upto this
point have also only been done via Ansible, so if it turns out we can't
expose the API like this we're ok with that. With minor changes to the
playbook we should be able to at least replicate the current 3.11
experience.
> That's what we decided to do for the CentOS CI ocp setup, and
so CI
> tenants can use oc from their laptop/infra. As long as cert exposed for
> default ingress has it added in the SAN, it works fine :
>
> X509v3 Subject Alternative Name:
> DNS:*.apps.ocp.ci.centos.org, DNS:api.ocp.ci.centos.org,
> DNS:apps.ocp.ci.centos.org
Yeah, thats all fine, but to make it work for our setup, I would need
to
get RHIT to nat in port 6443 to proxy01/10 from the internet. At least I
think thats the case. Openshift 3 could just use https, but alas, I fear
OCP4 needs that 6443 port.
Yep think you're right on that.
Do we want to try and enable http/2 ingress?
https://docs.openshift.com/container-platform/4.5/networking/ingress-oper...
We can take a look and see if we can figure it out!
We will want to enable kubevirt/whatever it's called...
We
definitely want to make this available, but we will have to set quotas
on usage. We should enable on staging, but should we enable on production?
On CentOS CI OCP4 cluster, we have Openshift Virtualization / kubevirt
installed, but I don't think anyone is actually using *it*. We have several
tenants which have elevated permissions, and are then accessing KVM
directly to bring up VMs on the Openshift nodes, this is something we want
to avoid, as we can't effectively set quotas on this type of usage.
On Fri, 24 Sept 2021 at 07:24, Kevin Fenzi <kevin(a)scrye.com> wrote:
> On Thu, Sep 23, 2021 at 07:44:49AM +0200, Fabian Arrotin wrote:
> > On 23/09/2021 02:55, Neal Gompa wrote:
> > > On Wed, Sep 22, 2021 at 7:12 PM Kevin Fenzi <kevin(a)scrye.com> wrote:
> > <snip>
> >
> > >>
> > >> * Since the control plane are vm's I assume we need to drain them
one
> at
> > >> a time to reboot the virthosts they are on?
> >
> > Correct
> >
> > >>
> > >> * Should we now delete the kubeadmin user? In 3.x I know they advise
> to
> > >> do that after auth is setup.
> > >>
> > >
> > > I'm not sure that's a good idea. I'm not even certain that was
a good
> > > idea in the OCP 3.x days, because eliminating the kubeadmin user means
> > > you lose your failsafe login if all else fails.
> >
> > +1 here : the reason why we decided to still keep kubeadmin on the other
> > OCP clusters used for CentOS CI and Stream is exactly for that reason :
> > still be able to login, if there is a problem with the oauth setup, and
> > troubleshoot issues if (for example) ipsilon or IPA have troubles ...
> :-)
>
> We can keep it if folks like. I'd really prefer we don't use it except
> for emergency though. Having people do things as their user will make it
> way easier to see who did what. ;)
>
> > >> * Right now the api is only internal. Is it worth getting a forward
> > >> setup to allow folks to use oc locally on their machines? It would
> > >> expose that api to the world, but of course it would still need auth.
> >
> > That's what we decided to do for the CentOS CI ocp setup, and so CI
> > tenants can use oc from their laptop/infra. As long as cert exposed for
> > default ingress has it added in the SAN, it works fine :
> >
> > X509v3 Subject Alternative Name:
> > DNS:*.apps.ocp.ci.centos.org, DNS:api.ocp.ci.centos.org,
> > DNS:apps.ocp.ci.centos.org
>
Yeah, thats all fine, but to make it work for our setup, I would need
to
get RHIT to nat in port 6443 to proxy01/10 from the internet. At least I
think thats the case. Openshift 3 could just use https, but alas, I fear
OCP4 needs that 6443 port.
>
> kevin
> _______________________________________________
> infrastructure mailing list -- infrastructure(a)lists.fedoraproject.org
> To unsubscribe send an email to
> infrastructure-leave(a)lists.fedoraproject.org
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedora...
> Do not reply to spam on the list, report it:
> https://pagure.io/fedora-infrastructure
>
--
David Kirwan
Software Engineer
Community Platform Engineering @ Red Hat
T: +(353) 86-8624108 IM: @dkirwan
10:05 p.m.
On Thu, Sep 23, 2021 at 11:02 PM David Kirwan <dkirwan(a)redhat.com> wrote:
> On the storage, are we ok if a node goes down? ie, does it spread it
> over all the storage nodes/raid? Or is it just in one place and you are
> dead if that node dies?
For storage, we maintain 3 replicas for data, spread across 3 nodes. However, not all
nodes are equally resourced, we have 2 large nodes and 1 much smaller, therefore, more
replicas will be spread over these 2 larger nodes. We can likely afford to lose one of the
large physical nodes while still maintaining data integrity.
> Is there any way to backup volumes?
There are ways to clone, extend, take snapshots etc of these volumes. We've never
done it, so it'll be a learning process for us all ;). We should sync to get a better
handle on the requirements for backups. In CentOS CI we've set up backups to S3, we
can certainly use some of that, eg: backup of etcd, but may need further investigation to
backup the volumes managed by OCS. Will need to do some research here.
> should we make a playbooks/manual/ocp.yml playbook for things like
> - list of clusteradmins
> - list of clustermoniting
> - anything else we want to manage post install
Sure yep, as we're finishing up soonish, I'd imagine the next few weeks we'll
all be back focused on the Infra/Releng tasks and will be focusing on tying up any loose
ends like this, and starting migration of apps.
> Have we tried a upgrade of the clusters yet? Did everything go ok?
> Do we need any docs on upgrades?
Yes, we've already completed a number of upgrades, latest is to 4.8.11. We have SOPs
for upgrades which we can copy over from the CentOS CI infra, and will make any updates
required in the process.
> Since the control plane are vm's I assume we need to drain them one at
> a time to reboot the virthosts they are on?
If we are rebooting a single vmhost/control plane VM at a time, yes that should be good.
If we are doing more than 1 at the same time, we should do a full graceful cluster
shutdown, and then a graceful cluster startup. We have SOPs for this in CentOS CI also,
we'll get those added here and any content updates made.
> * Should we now delete the kubeadmin user? In 3.x I know they advise to
> do that after auth is setup.
We can delete it, as we have system:admin available from the os-control01 node. Best
practices might suggest we do. We can also give cluster-admin role to all users in the
sysadmin-main and sysadmin-openshift groups.
I'm in two minds about deleting it, I was hoping to wait until we get a solution that
syncs IPA groups/users to Openshift. There is an official supported solution for syncing
LDAP (think that will work?).
> * Right now the api is only internal. Is it worth getting a forward
> setup to allow folks to use oc locally on their machines? It would
> expose that api to the world, but of course it would still need auth.
We'd love to expose it, but.. all interaction with the clusters upto this point have
also only been done via Ansible, so if it turns out we can't expose the API like this
we're ok with that. With minor changes to the playbook we should be able to at least
replicate the current 3.11 experience.
>> That's what we decided to do for the CentOS CI ocp setup, and so CI
>> tenants can use oc from their laptop/infra. As long as cert exposed for
>> default ingress has it added in the SAN, it works fine :
>>
>> X509v3 Subject Alternative Name:
>> DNS:*.apps.ocp.ci.centos.org, DNS:api.ocp.ci.centos.org,
>> DNS:apps.ocp.ci.centos.org
> Yeah, thats all fine, but to make it work for our setup, I would need to
> get RHIT to nat in port 6443 to proxy01/10 from the internet. At least I
> think thats the case. Openshift 3 could just use https, but alas, I fear
> OCP4 needs that 6443 port.
Yep think you're right on that.
>Do we want to try and enable http/2 ingress?
https://docs.openshift.com/container-platform/4.5/networking/ingress-oper...
We can take a look and see if we can figure it out!
> We will want to enable kubevirt/whatever it's called...
We definitely want to make this available, but we will have to set quotas on usage. We
should enable on staging, but should we enable on production?
On CentOS CI OCP4 cluster, we have Openshift Virtualization / kubevirt installed, but I
don't think anyone is actually using it. We have several tenants which have elevated
permissions, and are then accessing KVM directly to bring up VMs on the Openshift nodes,
this is something we want to avoid, as we can't effectively set quotas on this type of
usage.
I believe the Hyperscale SIG is using it. Or if they're not, it's
because they don't know it's there.
--
真実はいつも一つ！/ Always, there's only one truth!
11:30 p.m.
I believe the Hyperscale SIG is using it. Or if they're not,
it's
because they don't know it's there.
It might also be they don't have permission to create the kubevirt api
objects on the CentOS CI cluster. If they are actively interested in this,
we can sync up and figure out exactly what permissions are required to make
use of it!
On Fri, 24 Sept 2021 at 12:08, Neal Gompa <ngompa13(a)gmail.com> wrote:
> On Thu, Sep 23, 2021 at 11:02 PM David Kirwan <dkirwan(a)redhat.com> wrote:
> >
> > > On the storage, are we ok if a node goes down? ie, does it spread it
> > > over all the storage nodes/raid? Or is it just in one place and you are
> > > dead if that node dies?
> > For storage, we maintain 3 replicas for data, spread across 3 nodes.
> However, not all nodes are equally resourced, we have 2 large nodes and 1
> much smaller, therefore, more replicas will be spread over these 2 larger
> nodes. We can likely afford to lose one of the large physical nodes while
> still maintaining data integrity.
> >
> > > Is there any way to backup volumes?
> > There are ways to clone, extend, take snapshots etc of these volumes.
> We've never done it, so it'll be a learning process for us all ;). We
> should sync to get a better handle on the requirements for backups. In
> CentOS CI we've set up backups to S3, we can certainly use some of that,
> eg: backup of etcd, but may need further investigation to backup the
> volumes managed by OCS. Will need to do some research here.
> >
> > > should we make a playbooks/manual/ocp.yml playbook for things like
> > > - list of clusteradmins
> > > - list of clustermoniting
> > > - anything else we want to manage post install
> > Sure yep, as we're finishing up soonish, I'd imagine the next few weeks
> we'll all be back focused on the Infra/Releng tasks and will be focusing on
> tying up any loose ends like this, and starting migration of apps.
> >
> > > Have we tried a upgrade of the clusters yet? Did everything go ok?
> > > Do we need any docs on upgrades?
> > Yes, we've already completed a number of upgrades, latest is to 4.8.11.
> We have SOPs for upgrades which we can copy over from the CentOS CI infra,
> and will make any updates required in the process.
> >
> >
> > > Since the control plane are vm's I assume we need to drain them one at
> > > a time to reboot the virthosts they are on?
> > If we are rebooting a single vmhost/control plane VM at a time, yes that
> should be good. If we are doing more than 1 at the same time, we should do
> a full graceful cluster shutdown, and then a graceful cluster startup. We
> have SOPs for this in CentOS CI also, we'll get those added here and any
> content updates made.
> >
> > > * Should we now delete the kubeadmin user? In 3.x I know they advise to
> > > do that after auth is setup.
> > We can delete it, as we have system:admin available from the
> os-control01 node. Best practices might suggest we do. We can also give
> cluster-admin role to all users in the sysadmin-main and sysadmin-openshift
> groups.
> > I'm in two minds about deleting it, I was hoping to wait until we get a
> solution that syncs IPA groups/users to Openshift. There is an official
> supported solution for syncing LDAP (think that will work?).
> >
> > > * Right now the api is only internal. Is it worth getting a forward
> > > setup to allow folks to use oc locally on their machines? It would
> > > expose that api to the world, but of course it would still need auth.
> > We'd love to expose it, but.. all interaction with the clusters upto
> this point have also only been done via Ansible, so if it turns out we
> can't expose the API like this we're ok with that. With minor changes to
> the playbook we should be able to at least replicate the current 3.11
> experience.
> >
> > >> That's what we decided to do for the CentOS CI ocp setup, and so
CI
> > >> tenants can use oc from their laptop/infra. As long as cert exposed
> for
> > >> default ingress has it added in the SAN, it works fine :
> > >>
> > >> X509v3 Subject Alternative Name:
> > >> DNS:*.apps.ocp.ci.centos.org, DNS:
> api.ocp.ci.centos.org,
> > >> DNS:apps.ocp.ci.centos.org
> >
> > > Yeah, thats all fine, but to make it work for our setup, I would need
> to
> > > get RHIT to nat in port 6443 to proxy01/10 from the internet. At least
> I
> > > think thats the case. Openshift 3 could just use https, but alas, I
> fear
> > > OCP4 needs that 6443 port.
> > Yep think you're right on that.
> >
> >
> > >Do we want to try and enable http/2 ingress?
> >
>
https://docs.openshift.com/container-platform/4.5/networking/ingress-oper...
> > We can take a look and see if we can figure it out!
> >
> >
> > > We will want to enable kubevirt/whatever it's called...
> > We definitely want to make this available, but we will have to set
> quotas on usage. We should enable on staging, but should we enable on
> production?
> >
> > On CentOS CI OCP4 cluster, we have Openshift Virtualization / kubevirt
> installed, but I don't think anyone is actually using it. We have several
> tenants which have elevated permissions, and are then accessing KVM
> directly to bring up VMs on the Openshift nodes, this is something we want
> to avoid, as we can't effectively set quotas on this type of usage.
> >
>
I believe the Hyperscale SIG is using it. Or if they're not,
it's
because they don't know it's there.
>
>
>
> --
> 真実はいつも一つ！/ Always, there's only one truth!
> _______________________________________________
> infrastructure mailing list -- infrastructure(a)lists.fedoraproject.org
> To unsubscribe send an email to
> infrastructure-leave(a)lists.fedoraproject.org
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedora...
> Do not reply to spam on the list, report it:
> https://pagure.io/fedora-infrastructure
>
--
David Kirwan
Software Engineer
Community Platform Engineering @ Red Hat
T: +(353) 86-8624108 IM: @dkirwan
10:47 a.m.
On Fri, Sep 24, 2021 at 12:01:27PM +0900, David Kirwan wrote:
> On the storage, are we ok if a node goes down? ie, does it
spread it
> over all the storage nodes/raid? Or is it just in one place and you are
> dead if that node dies?
For storage, we maintain 3 replicas for data, spread across 3 nodes.
However, not all nodes are equally resourced, we have 2 large nodes and 1
much smaller, therefore, more replicas will be spread over these 2 larger
nodes. We can likely afford to lose one of the large physical nodes while
still maintaining data integrity.
ok. Fair enough.
> Is there any way to backup volumes?
There are ways to clone, extend, take snapshots etc of these volumes. We've
never done it, so it'll be a learning process for us all ;). We should sync
to get a better handle on the requirements for backups. In CentOS CI we've
set up backups to S3, we can certainly use some of that, eg: backup of
etcd, but may need further investigation to backup the volumes managed by
OCS. Will need to do some research here.
Yeah, backups of etcd would be nice, but mostly I was thinking of
applications that have persistent data. Right now we have those on
netapp NFS volumes, where it keeps snapshots and mirrors to another
site. I suppose we could just keep using NFS for data that has to
persist and just use local for other things, but it's sure nice to have
it dynamically provisioned. ;)
> should we make a playbooks/manual/ocp.yml playbook for things
like
> - list of clusteradmins
> - list of clustermoniting
> - anything else we want to manage post install
Sure yep, as we're finishing up soonish, I'd imagine the next few weeks
we'll all be back focused on the Infra/Releng tasks and will be focusing on
tying up any loose ends like this, and starting migration of apps.
ok.
> Have we tried a upgrade of the clusters yet? Did everything go
ok?
> Do we need any docs on upgrades?
Yes, we've already completed a number of upgrades, latest is to 4.8.11. We
have SOPs for upgrades which we can copy over from the CentOS CI infra, and
will make any updates required in the process.
Great.
> Since the control plane are vm's I assume we need to drain
them one at
> a time to reboot the virthosts they are on?
If we are rebooting a single vmhost/control plane VM at a time, yes that
should be good. If we are doing more than 1 at the same time, we should do
a full graceful cluster shutdown, and then a graceful cluster startup. We
have SOPs for this in CentOS CI also, we'll get those added here and any
content updates made.
> * Should we now delete the kubeadmin user? In 3.x I know they advise to
> do that after auth is setup.
We can delete it, as we have system:admin available from the os-control01
node. Best practices might suggest we do. We can also give cluster-admin
role to all users in the sysadmin-main and sysadmin-openshift groups.
Yeah, we should put this in the playbook so it's very clear who has this
and when it was added, etc.
I'm in two minds about deleting it, I was hoping to wait until we
get a
solution that syncs IPA groups/users to Openshift. There is an official
supported solution for syncing LDAP (think that will work?).
Yeah, needs investigation.
> * Right now the api is only internal. Is it worth getting a
forward
> setup to allow folks to use oc locally on their machines? It would
> expose that api to the world, but of course it would still need auth.
We'd love to expose it, but.. all interaction with the clusters upto this
point have also only been done via Ansible, so if it turns out we can't
expose the API like this we're ok with that. With minor changes to the
playbook we should be able to at least replicate the current 3.11
experience.
Sure, but currently app owners can use oc on their local machines to
view logs, debug, etc. I think thats a nice thing to keep working.
>> That's what we decided to do for the CentOS CI ocp
setup, and so CI
>> tenants can use oc from their laptop/infra. As long as cert exposed for
>> default ingress has it added in the SAN, it works fine :
>>
>> X509v3 Subject Alternative Name:
>> DNS:*.apps.ocp.ci.centos.org, DNS:api.ocp.ci.centos.org,
>> DNS:apps.ocp.ci.centos.org
> Yeah, thats all fine, but to make it work for our setup, I would need to
> get RHIT to nat in port 6443 to proxy01/10 from the internet. At least I
> think thats the case. Openshift 3 could just use https, but alas, I fear
> OCP4 needs that 6443 port.
Yep think you're right on that.
I can put in a request for this.
>Do we want to try and enable http/2 ingress?
https://docs.openshift.com/container-platform/4.5/networking/ingress-oper...
We can take a look and see if we can figure it out!
ok.
> We will want to enable kubevirt/whatever it's called...
We definitely want to make this available, but we will have to set quotas
on usage. We should enable on staging, but should we enable on production?
We can start with staging and test and see what usage might be before we
go to prod.
On CentOS CI OCP4 cluster, we have Openshift Virtualization /
kubevirt
installed, but I don't think anyone is actually using *it*. We have several
tenants which have elevated permissions, and are then accessing KVM
directly to bring up VMs on the Openshift nodes, this is something we want
to avoid, as we can't effectively set quotas on this type of usage.
Yeah, I think FCOS folks are a user there? They might be migrated to our
new cluster, so they would likely need the same perms here. ;(
Thanks!
kevin
12:08 a.m.
We've updated a number of SOPs since e.g. upgrades, if you could take
another look, and +1 the PR if we're happy!
We've installed and tested Openshift Virtualization on the staging cluster,
it's working good, currently need cluster-admin to use, In the coming days
we'll continue to work on the remaining tasks such as quotas and
permissions to easily hand out to users etc.
On Sat, 25 Sept 2021 at 00:55, Kevin Fenzi <kevin(a)scrye.com> wrote:
On Fri, Sep 24, 2021 at 12:01:27PM +0900, David Kirwan wrote:
> > On the storage, are we ok if a node goes down? ie, does it spread it
> > over all the storage nodes/raid? Or is it just in one place and you are
> > dead if that node dies?
> For storage, we maintain 3 replicas for data, spread across 3 nodes.
> However, not all nodes are equally resourced, we have 2 large nodes and 1
> much smaller, therefore, more replicas will be spread over these 2 larger
> nodes. We can likely afford to lose one of the large physical nodes while
> still maintaining data integrity.
ok. Fair enough.
> > Is there any way to backup volumes?
> There are ways to clone, extend, take snapshots etc of these volumes.
We've
> never done it, so it'll be a learning process for us all ;). We should
sync
> to get a better handle on the requirements for backups. In CentOS CI
we've
> set up backups to S3, we can certainly use some of that, eg: backup of
> etcd, but may need further investigation to backup the volumes managed by
> OCS. Will need to do some research here.
Yeah, backups of etcd would be nice, but mostly I was thinking of
applications that have persistent data. Right now we have those on
netapp NFS volumes, where it keeps snapshots and mirrors to another
site. I suppose we could just keep using NFS for data that has to
persist and just use local for other things, but it's sure nice to have
it dynamically provisioned. ;)
> > should we make a playbooks/manual/ocp.yml playbook for things like
> > - list of clusteradmins
> > - list of clustermoniting
> > - anything else we want to manage post install
> Sure yep, as we're finishing up soonish, I'd imagine the next few weeks
> we'll all be back focused on the Infra/Releng tasks and will be focusing
on
> tying up any loose ends like this, and starting migration of apps.
ok.
> > Have we tried a upgrade of the clusters yet? Did everything go ok?
> > Do we need any docs on upgrades?
> Yes, we've already completed a number of upgrades, latest is to 4.8.11.
We
> have SOPs for upgrades which we can copy over from the CentOS CI infra,
and
> will make any updates required in the process.
Great.
> > Since the control plane are vm's I assume we need to drain them one at
> > a time to reboot the virthosts they are on?
> If we are rebooting a single vmhost/control plane VM at a time, yes that
> should be good. If we are doing more than 1 at the same time, we should
do
> a full graceful cluster shutdown, and then a graceful cluster startup. We
> have SOPs for this in CentOS CI also, we'll get those added here and any
> content updates made.
>
> > * Should we now delete the kubeadmin user? In 3.x I know they advise to
> > do that after auth is setup.
> We can delete it, as we have system:admin available from the os-control01
> node. Best practices might suggest we do. We can also give cluster-admin
> role to all users in the sysadmin-main and sysadmin-openshift groups.
Yeah, we should put this in the playbook so it's very clear who has this
and when it was added, etc.
> I'm in two minds about deleting it, I was hoping to wait until we get a
> solution that syncs IPA groups/users to Openshift. There is an official
> supported solution for syncing LDAP (think that will work?).
Yeah, needs investigation.
> > * Right now the api is only internal. Is it worth getting a forward
> > setup to allow folks to use oc locally on their machines? It would
> > expose that api to the world, but of course it would still need auth.
> We'd love to expose it, but.. all interaction with the clusters upto this
> point have also only been done via Ansible, so if it turns out we can't
> expose the API like this we're ok with that. With minor changes to the
> playbook we should be able to at least replicate the current 3.11
> experience.
Sure, but currently app owners can use oc on their local machines to
view logs, debug, etc. I think thats a nice thing to keep working.
> >> That's what we decided to do for the CentOS CI ocp setup, and so CI
> >> tenants can use oc from their laptop/infra. As long as cert exposed
for
> >> default ingress has it added in the SAN, it works fine :
> >>
> >> X509v3 Subject Alternative Name:
> >> DNS:*.apps.ocp.ci.centos.org, DNS:
api.ocp.ci.centos.org,
> >> DNS:apps.ocp.ci.centos.org
>
> > Yeah, thats all fine, but to make it work for our setup, I would need
to
> > get RHIT to nat in port 6443 to proxy01/10 from the internet. At least
I
> > think thats the case. Openshift 3 could just use https, but alas, I
fear
> > OCP4 needs that 6443 port.
> Yep think you're right on that.
I can put in a request for this.
> >Do we want to try and enable http/2 ingress?
>
https://docs.openshift.com/container-platform/4.5/networking/ingress-oper...
> We can take a look and see if we can figure it out!
ok.
> > We will want to enable kubevirt/whatever it's called...
> We definitely want to make this available, but we will have to set quotas
> on usage. We should enable on staging, but should we enable on
production?
We can start with staging and test and see what usage might be before we
go to prod.
> On CentOS CI OCP4 cluster, we have Openshift Virtualization / kubevirt
> installed, but I don't think anyone is actually using *it*. We have
several
> tenants which have elevated permissions, and are then accessing KVM
> directly to bring up VMs on the Openshift nodes, this is something we
want
> to avoid, as we can't effectively set quotas on this type of usage.
Yeah, I think FCOS folks are a user there? They might be migrated to our
new cluster, so they would likely need the same perms here. ;(
Thanks!
kevin
_______________________________________________
infrastructure mailing list -- infrastructure(a)lists.fedoraproject.org
To unsubscribe send an email to
infrastructure-leave(a)lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedora...
Do not reply to spam on the list, report it:
https://pagure.io/fedora-infrastructure
--
David Kirwan
Software Engineer
Community Platform Engineering @ Red Hat
T: +(353) 86-8624108 IM: @dkirwan
12:26 p.m.
On Tue, Sep 28, 2021 at 02:08:25PM +0900, David Kirwan wrote:
We've updated a number of SOPs since e.g. upgrades, if you could
take
another look, and +1 the PR if we're happy!
Looks pretty good to me. I think if there's changes to make we can make
them as we follow things. ;)
We've installed and tested Openshift Virtualization on the
staging cluster,
it's working good, currently need cluster-admin to use, In the coming days
we'll continue to work on the remaining tasks such as quotas and
permissions to easily hand out to users etc.
Sounds good.
So, as far as migrating over to the new cluster...how do we want to
handle that? Obviously we want to do staging first and sort out any
issues. Should that be a task that infra and releng takes on this next
quater? Or something you all are going to keep working on?
Also, I will have 2 more compute nodes we can add (one to stg and one to
prod) in a bit (need to get vlans sorted, etc). Perhaps it's worth
adding a sop to add new nodes I can just do it? Or hand off the machines
when they are ready for you all.
I'll put in a ticket soon also to get the api exposed to the external
net. Since we have this in our 3.x cluster I think people will be expecting
it for the new one and I don't think there should be too much danger exposing it
(it should require auth to do anything right?)
I don't think there's anything else too much pending. :)
kevin
8:35 p.m.
On Wed, 29 Sept 2021 at 02:28, Kevin Fenzi <kevin(a)scrye.com> wrote:
So, as far as migrating over to the new cluster...how do we want to
handle that? Obviously we want to do staging first and sort out any
issues. Should that be a task that infra and releng takes on this next
quater? Or something you all are going to keep working on?
I hope this is something we can continue to work on as we shift back into
the infra & releng. Might be good to catch you for a few minutes for a chat
this week to bounce a few ideas about how we could do it, we were
considering adding new roles for each individual app, it could allow us to
deploy them with any new ocp4 required changes without affecting the
existing 3.11 cluster versions.
Also, I will have 2 more compute nodes we can add (one to stg and one
to
prod) in a bit (need to get vlans sorted, etc). Perhaps it's worth
adding a sop to add new nodes I can just do it? Or hand off the machines
when they are ready for you all.
Nice! Yep we have a SOP from the CentOS CI infra we can use, it would be
great if you wanted to follow it to add these new nodes!
--
David Kirwan
Software Engineer
Community Platform Engineering @ Red Hat
T: +(353) 86-8624108 IM: @dkirwan
9:57 p.m.
Cheers folks, this is merged now.
On Wed, 29 Sept 2021 at 10:35, David Kirwan <dkirwan(a)redhat.com> wrote:
On Wed, 29 Sept 2021 at 02:28, Kevin Fenzi <kevin(a)scrye.com> wrote:
>
> So, as far as migrating over to the new cluster...how do we want to
> handle that? Obviously we want to do staging first and sort out any
> issues. Should that be a task that infra and releng takes on this next
> quater? Or something you all are going to keep working on?
>
I hope this is something we can continue to work on as we shift back into
the infra & releng. Might be good to catch you for a few minutes for a chat
this week to bounce a few ideas about how we could do it, we were
considering adding new roles for each individual app, it could allow us to
deploy them with any new ocp4 required changes without affecting the
existing 3.11 cluster versions.
> Also, I will have 2 more compute nodes we can add (one to stg and one to
> prod) in a bit (need to get vlans sorted, etc). Perhaps it's worth
> adding a sop to add new nodes I can just do it? Or hand off the machines
> when they are ready for you all.
>
Nice! Yep we have a SOP from the CentOS CI infra we can use, it would be
great if you wanted to follow it to add these new nodes!
--
David Kirwan
Software Engineer
Community Platform Engineering @ Red Hat
T: +(353) 86-8624108 IM: @dkirwan
--
David Kirwan
Software Engineer
Community Platform Engineering @ Red Hat
T: +(353) 86-8624108 IM: @dkirwan
1:40 p.m.
On Wed, Sep 29, 2021 at 10:35:07AM +0900, David Kirwan wrote:
On Wed, 29 Sept 2021 at 02:28, Kevin Fenzi <kevin(a)scrye.com>
wrote:
>
> So, as far as migrating over to the new cluster...how do we want to
> handle that? Obviously we want to do staging first and sort out any
> issues. Should that be a task that infra and releng takes on this next
> quater? Or something you all are going to keep working on?
>
I hope this is something we can continue to work on as we shift back into
the infra & releng. Might be good to catch you for a few minutes for a chat
this week to bounce a few ideas about how we could do it, we were
Sure. :)
considering adding new roles for each individual app, it could allow
us to
deploy them with any new ocp4 required changes without affecting the
existing 3.11 cluster versions.
Well, in some cases thats fine, but in other cases we really need to
actually migrate them (ie, stop old one, start new one, repoint
proxies). This would be the case for anything that had persistent data
volumes and possibly others.
I would hope for most it would just be changing the playbook to run on
os-control hosts instead of os-masters of the old cluster and possibly a
few apis changing versions. Then, changing the proxies to point to the
new cluster instead of old for that app.
> Also, I will have 2 more compute nodes we can add (one to stg
and one to
> prod) in a bit (need to get vlans sorted, etc). Perhaps it's worth
> adding a sop to add new nodes I can just do it? Or hand off the machines
> when they are ready for you all.
>
Nice! Yep we have a SOP from the CentOS CI infra we can use, it would be
great if you wanted to follow it to add these new nodes!
Sounds good. It will be a bit as I need to get them prepped and make
sure they are ready, and also we head into final freeze next tuesday, so
not much time there. ;(
kevin
933
days inactive
940
days old
infrastructure@lists.fedoraproject.org
14 comments
5 participants
participants (5)
-
darknao -
David Kirwan -
Fabian Arrotin -
Kevin Fenzi -
Neal Gompa