5:53 p.m.
Greetings everyone.
As you hopefully know, we took down our old staging env in phx2 as part
of the datacenter move. Once machines from phx2 are shipped to iad2 and
racked and installed and setup we can look at re-enabling our staging
env.
However, I'd like to ask everyone about how we want that to look.
Some questions:
* Before we had a staging openshift with staging applications in it.
This is sort of not how openshift is designed to work. In the ideal
openshift world you don't need staging, you just have enough tests and
CI and gradual rollout of new versions so everything just works.
Granted a staging openshift cluster is useful to ops folks to test
upgrades and try out things, and it's useful for developers in our case
to get all the parts setup right in ansible to deploy their application.
So, what do you think? should we setup a staging cluster as before?
Or shall we try and just use the one production cluster for staging and
prod?
* Another question is openshift 4. Openshift 3.11 is supported until
june of 2022, so we have some time, but do we want to or need to look at
moving to openshift 4 for our clusters? One thing I hate about this is
that you must have 3 master nodes, and the only machines we have are big
powerfull virthost servers, so it's very wastefull of resources to
deploy a openshift 4 cluster (with the machines we have currently
anyhow).
* In our old staging env we had a subset of things. Some of them we used
the staging instances all the time, others we almost never did. I'm not
sure we have the resources to deploy a 100% copy of our prod env, but
assuming we did, where should we shoot for on the line? ie, between 100%
duplicate of prod or nothing?
* We came up with a pretty elaborate koji sync from prod->staging. There
were lots of reasons we got to that, but I suppose if someone wants to
propose another method of doing this we could revisit that.
* Any other things we definitely want from a staging env?
* Has staging been helpful to you?
* Is there anything we could do to make it better?
Thoughts?
kevin
6:20 p.m.
On Mon, 2020-06-15 at 15:53 -0700, Kevin Fenzi wrote:
* Has staging been helpful to you?
It's been useful to me a few times. For instance it was definitely
useful when doing the initial fedmsg enablement for openQA and then for
the port from fedmsg to fedora-messaging: we did both of those in
staging first. Even though the 'staging' instance of openQA *itself*
wasn't a real staging thing, it *did* publish to the staging
fedmsg/fedora-messaging buses, so we could shake out the wrinkles in
that without affecting prod, then deploy prod once we had it working.
* Is there anything we could do to make it better?
This may be pretty niche, but I do wish the staging wiki contents were
synced more often. Of course the sync process wipes out changes made
directly on the staging wiki so there's a bit of a balance, but maybe
auto-syncing every day or every few days would be good...
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
8:04 p.m.
Finally something I feel qualified to comment on (I own Openshift at
$DAYJOB these days, and am more than happy to help out!)
On Mon, Jun 15, 2020 at 6:54 PM Kevin Fenzi <kevin(a)scrye.com> wrote:
Greetings everyone.
* Before we had a staging openshift with staging applications in it.
This is sort of not how openshift is designed to work. In the ideal
openshift world you don't need staging, you just have enough tests and
CI and gradual rollout of new versions so everything just works.
Granted a staging openshift cluster is useful to ops folks to test
upgrades and try out things, and it's useful for developers in our case
to get all the parts setup right in ansible to deploy their application.
So, what do you think? should we setup a staging cluster as before?
Or shall we try and just use the one production cluster for staging and
prod?
At $DAYJOB, we do the latter. All of our tenants (applications) only have
access to our production clusters. We do have development and staging
clusters, however they are just for us, the infrastructure team, to test
upgrades and whatnot on. All "real" workloads run on our production
clusters.
One bad thing with this setup could be that sometimes you don't see issues
until they hit production if there's weird interactions that you can't test
for. This can be *somewhat* resolved by having your "heavy hitter"
applications (which I've been out of touch with Fedora infra for so long
that I'm not sure if we have any in Openshift) test the upgrade in
conjunction with the platform team. This of course involves having willing
guinea pigs :).
* Another question is openshift 4. Openshift 3.11 is supported until
june of 2022, so we have some time, but do we want to or need to look at
moving to openshift 4 for our clusters? One thing I hate about this is
that you must have 3 master nodes, and the only machines we have are big
powerfull virthost servers, so it's very wastefull of resources to
deploy a openshift 4 cluster (with the machines we have currently
anyhow).
Yes, you want Openshift 4! (In fact, I thought that the move meant you
folks were moving to 4, guess not!). A lot of what you stated is accurate
but no different than with 3.11. You needed 3 masters in 3.11 as well for a
resilient setup. The nice thing with Openshift 4 that you couldn't do with
3 is scheduling user workloads on the masters. The masters MUST run RHCOS,
and the worker nodes I would HIGHLY recommend run RHCOS. There is work
underway (it didn't make 4.3 as mentioned, and frankly I'm not sure of its
status in 4.4) to allow 3 node clusters (
https://github.com/openshift/enhancements/blob/master/enhancements/compac...)
- currently the minimum viable cluster is 5 nodes. That said, there's
nothing that says those 5 nodes have to be bare metal - at the extreme, I
have a 5 node cluster running entirely on my desktop (a Xeon W-2155 w/192GB
RAM, but I digress....). I'd run the masters on 3 different virthosts if
possible, depending on the workload they don't actually have to be that big
(i have mine at 4x16, but it's mainly a test cluster)
* In our old staging env we had a subset of things. Some of them we used
the staging instances all the time, others we almost never did. I'm not
sure we have the resources to deploy a 100% copy of our prod env, but
assuming we did, where should we shoot for on the line? ie, between 100%
duplicate of prod or nothing?
I really think it's up to the folks that run the service if a staging
environment is useful to them or not. I'd imagine for some things it would
be extraordinarily useful, and for others a waste of resources. I think
that the individual service owners are in the best position to make that
determination.
9:01 p.m.
On Mon, Jun 15, 2020 at 09:04:27PM -0400, Jon Stanley wrote:
Finally something I feel qualified to comment on (I own Openshift at
$DAYJOB these days, and am more than happy to help out!)
Hey Jon! Nice to hear from you!
Yes, you want Openshift 4! (In fact, I thought that the move meant
you
folks were moving to 4, guess not!). A lot of what you stated is accurate
but no different than with 3.11. You needed 3 masters in 3.11 as well for a
resilient setup. The nice thing with Openshift 4 that you couldn't do with
3 is scheduling user workloads on the masters. The masters MUST run RHCOS,
and the worker nodes I would HIGHLY recommend run RHCOS. There is work
underway (it didn't make 4.3 as mentioned, and frankly I'm not sure of its
status in 4.4) to allow 3 node clusters (
https://github.com/openshift/enhancements/blob/master/enhancements/compac...)
- currently the minimum viable cluster is 5 nodes. That said, there's
nothing that says those 5 nodes have to be bare metal - at the extreme, I
have a 5 node cluster running entirely on my desktop (a Xeon W-2155 w/192GB
RAM, but I digress....). I'd run the masters on 3 different virthosts if
possible, depending on the workload they don't actually have to be that big
(i have mine at 4x16, but it's mainly a test cluster)
huh. The docs and everything I have seen seem to indicate you can't use
vm's?
I mean look at:
https://docs.openshift.com/container-platform/4.4/installing/installing_b...
no 'installing on your own vm's there' ?
I guess this is partially mitigated if you run workloads on the masters
too, but then you could run into things sucking up all the resources and
starving etcd/api/etc important things.
I just don't want to take 5 machines with 256gb ram/64 cpus and have 3
of them sit there using 16gb memory and no cpus most of the time either.
If we can do vm's that also completely works fine, thats what we are
doing with 3.11 right now.
> * In our old staging env we had a subset of things. Some of them
we used
> the staging instances all the time, others we almost never did. I'm not
> sure we have the resources to deploy a 100% copy of our prod env, but
> assuming we did, where should we shoot for on the line? ie, between 100%
> duplicate of prod or nothing?
>
I really think it's up to the folks that run the service if a staging
environment is useful to them or not. I'd imagine for some things it would
be extraordinarily useful, and for others a waste of resources. I think
that the individual service owners are in the best position to make that
determination.
Yeah, agreed.
kevin
10:34 p.m.
On Mon, Jun 15, 2020 at 10:26 PM Kevin Fenzi <kevin(a)scrye.com> wrote:
huh. The docs and everything I have seen seem to indicate you
can't use
vm's?
I mean look at:
https://docs.openshift.com/container-platform/4.4/installing/installing_b...
no 'installing on your own vm's there' ?
Look at https://access.redhat.com/articles/4207611 - which
unfortunately requires a subscription. But the relevant part reads:
Once customers have familiarized themselves with how the installation
should be performed (please refer to the bare metal installation
documentation for further details), OpenShift 4 can be successfully
deployed in most environments. You should be aware of a number of
areas where this installation method may not support your particular
provider or may even require minor modifications when deploying
OpenShift. Remember, these areas are only applicable if you are trying
to use our bare metal installation method to deploy OpenShift on
virtualization or cloud solutions that Red Hat has not yet tested or
provided a documented installation method.
Basically, it says that you should be able to get it to work. Of the 7
things that they say to look out for, I would say the number 1 is "how
am I going to get ignition configs into this thing?". For KVM, I think
that you could just pass it on the virt-install command line pretty
easily.
Also one pain point from bitter experience - once you generate the
ignition configs, you have 24 hours to have the cluster fully
functional. If you take longer than that you have to start over
because the bootstrap certs are no good anymore :). Also, once you
install the cluster, don't shut it down for about 24 hours to allow
cert rotation to happen.
I guess this is partially mitigated if you run workloads on the
masters
too, but then you could run into things sucking up all the resources and
starving etcd/api/etc important things.
Yep, totally get it - that's why we're going to try not scheduling
masters until we have to (even though in our case at $DAYJOB they're
36 core, 768GB RAM, 16TB NVMe monstrosities). However, in later
versions of Kubernetes we get nice things like priorityclasses to
somewhat mitigate that.
I just don't want to take 5 machines with 256gb ram/64 cpus and
have 3
of them sit there using 16gb memory and no cpus most of the time either.
If we can do vm's that also completely works fine, thats what we are
doing with 3.11 right now.
Yep, works great! Happy to lend the hand of experience if you want,
it's a bit of a beast to get working right.
2:03 a.m.
On 16/06/2020 00:53, Kevin Fenzi wrote:
Greetings everyone.
As you hopefully know, we took down our old staging env in phx2 as part
of the datacenter move. Once machines from phx2 are shipped to iad2 and
racked and installed and setup we can look at re-enabling our staging
env.
However, I'd like to ask everyone about how we want that to look.
Some questions:
* Before we had a staging openshift with staging applications in it.
This is sort of not how openshift is designed to work. In the ideal
openshift world you don't need staging, you just have enough tests and
CI and gradual rollout of new versions so everything just works.
Granted a staging openshift cluster is useful to ops folks to test
upgrades and try out things, and it's useful for developers in our case
to get all the parts setup right in ansible to deploy their application.
So, what do you think? should we setup a staging cluster as before?
Or shall we try and just use the one production cluster for staging and
prod?
For me the staging was helpful to synchronize changes when deploying
Anitya and the-new-hotness (it has plenty of external systems to connect
to), but I think if I would had better integration tests I don't need
staging environment anymore (I have plans to rewrite the-new-hotness
with clean architecture, which should help a lot with dependencies on
external systems).
But it could be nice to create at least some kind of playground, where
newcomers could try things without breaking anything. But we could have
one namespace dedicated to this or allow use of communishift for this.
* Another question is openshift 4. Openshift 3.11 is supported until
june of 2022, so we have some time, but do we want to or need to look at
moving to openshift 4 for our clusters? One thing I hate about this is
that you must have 3 master nodes, and the only machines we have are big
powerfull virthost servers, so it's very wastefull of resources to
deploy a openshift 4 cluster (with the machines we have currently
anyhow).
Yesterday I was working on CentOS OpenShift 4 instance and it's from
user POV much more easier to use and much more comfortable and because
CentOS CI team had it we already have knowledge in our team how to work
with it. So I would be for using the OpenShift 4 if we are doing these
breaking changes anyway.
* In our old staging env we had a subset of things. Some of them we used
the staging instances all the time, others we almost never did. I'm not
sure we have the resources to deploy a 100% copy of our prod env, but
assuming we did, where should we shoot for on the line? ie, between 100%
duplicate of prod or nothing?
* We came up with a pretty elaborate koji sync from prod->staging. There
were lots of reasons we got to that, but I suppose if someone wants to
propose another method of doing this we could revisit that.
* Any other things we definitely want from a staging env?
As I wrote above, some
kind of playground would be nice.
* Has staging been helpful to you?
It helped me a lot with Anitya and
the-new-hotness, but as I said the
better integration tests should solve most of those issues.
* Is there anything we could do to make it better?
Thoughts?
kevin
_______________________________________________
infrastructure mailing list -- infrastructure(a)lists.fedoraproject.org
To unsubscribe send an email to infrastructure-leave(a)lists.fedoraproject.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedora...
--
Role: Fedora CPE Team - Software Engineer
IRC: mkonecny
FAS: zlopez
5:07 a.m.
In our old staging env we had a subset of things. Some of them we
used
the staging instances all the time, others we almost never did. I'm not
sure we have the resources to deploy a 100% copy of our prod env, but
assuming we did, where should we shoot for on the line? ie, between 100%
duplicate of prod or nothing?
As we have everything in ansible could we leave a small amount of resources
available so that people could deploy some staging apps as needed and tear
down when done, if they are only used occasionally they would be able to
share the resources. It may take a little longer but the hardware resource
need could either be reduced overall or used elsewhere.
Mark
On Tue, Jun 16, 2020 at 8:03 AM Michal Konecny <mkonecny(a)redhat.com> wrote:
On 16/06/2020 00:53, Kevin Fenzi wrote:
Greetings everyone.
As you hopefully know, we took down our old staging env in phx2 as part
of the datacenter move. Once machines from phx2 are shipped to iad2 and
racked and installed and setup we can look at re-enabling our staging
env.
However, I'd like to ask everyone about how we want that to look.
Some questions:
* Before we had a staging openshift with staging applications in it.
This is sort of not how openshift is designed to work. In the ideal
openshift world you don't need staging, you just have enough tests and
CI and gradual rollout of new versions so everything just works.
Granted a staging openshift cluster is useful to ops folks to test
upgrades and try out things, and it's useful for developers in our case
to get all the parts setup right in ansible to deploy their application.
So, what do you think? should we setup a staging cluster as before?
Or shall we try and just use the one production cluster for staging and
prod?
For me the staging was helpful to synchronize changes when deploying
Anitya and the-new-hotness (it has plenty of external systems to connect
to), but I think if I would had better integration tests I don't need
staging environment anymore (I have plans to rewrite the-new-hotness with
clean architecture, which should help a lot with dependencies on external
systems).
But it could be nice to create at least some kind of playground, where
newcomers could try things without breaking anything. But we could have one
namespace dedicated to this or allow use of communishift for this.
* Another question is openshift 4. Openshift 3.11 is supported until
june of 2022, so we have some time, but do we want to or need to look at
moving to openshift 4 for our clusters? One thing I hate about this is
that you must have 3 master nodes, and the only machines we have are big
powerfull virthost servers, so it's very wastefull of resources to
deploy a openshift 4 cluster (with the machines we have currently
anyhow).
Yesterday I was working on CentOS OpenShift 4 instance and it's from user
POV much more easier to use and much more comfortable and because CentOS CI
team had it we already have knowledge in our team how to work with it. So I
would be for using the OpenShift 4 if we are doing these breaking changes
anyway.
* In our old staging env we had a subset of things. Some of them we used
the staging instances all the time, others we almost never did. I'm not
sure we have the resources to deploy a 100% copy of our prod env, but
assuming we did, where should we shoot for on the line? ie, between 100%
duplicate of prod or nothing?
* We came up with a pretty elaborate koji sync from prod->staging. There
were lots of reasons we got to that, but I suppose if someone wants to
propose another method of doing this we could revisit that.
* Any other things we definitely want from a staging env?
As I wrote above, some kind of playground would be nice.
* Has staging been helpful to you?
It helped me a lot with Anitya and the-new-hotness, but as I said the
better integration tests should solve most of those issues.
* Is there anything we could do to make it better?
Thoughts?
kevin
_______________________________________________
infrastructure mailing list -- infrastructure(a)lists.fedoraproject.org
To unsubscribe send an email to infrastructure-leave(a)lists.fedoraproject.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedora...
--
Role: Fedora CPE Team - Software Engineer
IRC: mkonecny
FAS: zlopez
_______________________________________________
infrastructure mailing list -- infrastructure(a)lists.fedoraproject.org
To unsubscribe send an email to
infrastructure-leave(a)lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedora...
12:51 p.m.
On Tue, Jun 16, 2020 at 11:07:59AM +0100, Mark O'Brien wrote:
> In our old staging env we had a subset of things. Some of them
we used
> the staging instances all the time, others we almost never did. I'm not
> sure we have the resources to deploy a 100% copy of our prod env, but
> assuming we did, where should we shoot for on the line? ie, between 100%
> duplicate of prod or nothing?
As we have everything in ansible could we leave a small amount of resources
available so that people could deploy some staging apps as needed and tear
down when done, if they are only used occasionally they would be able to
share the resources. It may take a little longer but the hardware resource
need could either be reduced overall or used elsewhere.
Unfortunately, a lot of things are intertwined.
In order to test say, OSBS, you would need koji, which needs ipa and
download servers and...
But yes, some of the more self contained apps could be done this way. If
they are in openshift even more so.
kevin
1:21 p.m.
On Mon, Jun 15, 2020 at 6:54 PM Kevin Fenzi <kevin(a)scrye.com> wrote:
Greetings everyone.
As you hopefully know, we took down our old staging env in phx2 as part
of the datacenter move. Once machines from phx2 are shipped to iad2 and
racked and installed and setup we can look at re-enabling our staging
env.
However, I'd like to ask everyone about how we want that to look.
Some questions:
* Before we had a staging openshift with staging applications in it.
This is sort of not how openshift is designed to work. In the ideal
openshift world you don't need staging, you just have enough tests and
CI and gradual rollout of new versions so everything just works.
Granted a staging openshift cluster is useful to ops folks to test
upgrades and try out things, and it's useful for developers in our case
to get all the parts setup right in ansible to deploy their application.
So, what do you think? should we setup a staging cluster as before?
Or shall we try and just use the one production cluster for staging and
prod?
I think having a staging OpenShift would still be useful, because we
probably still have cases where we may want to test a mixture of
cluster and app changes together, and that'd be difficult to do in the
production OpenShift system.
* Another question is openshift 4. Openshift 3.11 is supported until
june of 2022, so we have some time, but do we want to or need to look at
moving to openshift 4 for our clusters? One thing I hate about this is
that you must have 3 master nodes, and the only machines we have are big
powerfull virthost servers, so it's very wastefull of resources to
deploy a openshift 4 cluster (with the machines we have currently
anyhow).
You could make your OpenShift master nodes be able to schedule
workloads, so it's less wasteful. I think it'd make a ton of sense to
look at OpenShift/OKD 4 with this move. There *are* differences with
OpenShift 4, and we will likely have to do some significant
adaptations to our deployment code for it, but it would be worth it
for the simpler maintenance of the cluster.
* In our old staging env we had a subset of things. Some of them we
used
the staging instances all the time, others we almost never did. I'm not
sure we have the resources to deploy a 100% copy of our prod env, but
assuming we did, where should we shoot for on the line? ie, between 100%
duplicate of prod or nothing?
* We came up with a pretty elaborate koji sync from prod->staging. There
were lots of reasons we got to that, but I suppose if someone wants to
propose another method of doing this we could revisit that.
* Any other things we definitely want from a staging env?
I think that as long as we have a relatively close replica of our prod
infrastructure, it will be useful for being able to experiment with
improvements to our infra in a safe way.
* Has staging been helpful to you?
Staging has been immensely helpful for the things I work with (Package
infra, Pagure, etc.), it's helped us with testing new releases, new
changes, etc. and catching bugs before we make releases. I hope we
continue to have it.
* Is there anything we could do to make it better?
I don't particularly have a lot of access, which is kind of
frustrating. But that's a problem specific to me rather than something
wrong with staging infra env. :)
--
真実はいつも一つ！/ Always, there's only one truth!
1418
days inactive
1420
days old
infrastructure@lists.fedoraproject.org
8 comments
6 participants
participants (6)
-
Adam Williamson -
Jon Stanley -
Kevin Fenzi -
Mark O'Brien -
Michal Konecny -
Neal Gompa