So as many of you have seen in the commits lists, the staging environment is coming along and getting built. I've hit a policy issue and so I thought instead of just doing this in a black hole. I'd discuss it.
The way I see it there are two ways to do staging environments. For those of you unfamiliar with staging the general idea is to have an environment as close to production as feasible.
1) use identical configs with only minor changes and use /etc/hosts to fake things to point where you need them. Not always possible but generally good where you can do it.
2) use different configs in production and staging. The differences being able to redirect things, using different usernames, passwords, hostnames, etc.
Each has pros and cons. Right now I'd like to do 1) but I don't think its possible. 2) is going to require a lot of focus. For example... we won't be able to just git merge from staging to production as we could with 1).
Security's only an issue in that we don't want people making changes to production data from staging and vise versa. The same people will have the same access to both of these environments without exception.
I'm going to continue to think about this. I've had staging environments in the past. Both went with option 2). But still. I'd like to hold this discussion so discuss.
-Mike
On Fri, Sep 5, 2008 at 1:16 PM, Mike McGrath mmcgrath@redhat.com wrote:
So as many of you have seen in the commits lists, the staging environment is coming along and getting built. I've hit a policy issue and so I thought instead of just doing this in a black hole. I'd discuss it.
The way I see it there are two ways to do staging environments. For those of you unfamiliar with staging the general idea is to have an environment as close to production as feasible.
- use identical configs with only minor changes and use /etc/hosts to
fake things to point where you need them. Not always possible but generally good where you can do it.
- use different configs in production and staging. The differences being
able to redirect things, using different usernames, passwords, hostnames, etc.
Each has pros and cons. Right now I'd like to do 1) but I don't think its possible. 2) is going to require a lot of focus. For example... we won't be able to just git merge from staging to production as we could with 1).
there is also a combination of #1 and #2. Basically you have to create 3-4 separate network topologies (this is where you have different configs), and maybe have your bastion/proxy systems different.
Name Network Network A: Development -- 10.10.0.0/21 Servers -- 10.10.0.0/22 NFS -- 10.10.4.0/22 Network B: QA -- 10.10.8.0/21 Servers -- 10.10.8.0/22 NFS -- 10.10.12.0/22 Network C: Staging -- 10.10.16.0/21 Servers -- 10.10.16.0/22 NFS -- 10.10.20.0/22 Network D: Production -- 10.10.24.0/21 Servers -- 10.10.24.0/22 NFS -- 10.10.28.0/22 Network E: Management -- 10.10.32.0/20 Puppet -- 10.10.32.0/21 Drac/Serial -- 10.10.48.0/21 Network F: Bastion Network
[Ok I would love to have done this when I was at RH... but didn't really see it in action til later.]
Basically a box would have 3-4 network connections. The puppet and drac/serial networks are on all systems so have to be extra protected as that is where an attacker could walk from system to system. The bastion network is basically the front end that would do rewrites and other layers so that configs are the same.
And yes, this might be overkill and probably has holes in it.. I am doing it from memory on how a site seemed to be set up and had basically little downtime for critical HR services.
Security's only an issue in that we don't want people making changes to production data from staging and vise versa. The same people will have the same access to both of these environments without exception.
I'm going to continue to think about this. I've had staging environments in the past. Both went with option 2). But still. I'd like to hold this discussion so discuss.
-Mike
Fedora-infrastructure-list mailing list Fedora-infrastructure-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
On Fri, 5 Sep 2008, Stephen John Smoogen wrote:
there is also a combination of #1 and #2. Basically you have to create 3-4 separate network topologies (this is where you have different configs), and maybe have your bastion/proxy systems different.
Name NetworkNetwork A: Development -- 10.10.0.0/21 Servers -- 10.10.0.0/22 NFS -- 10.10.4.0/22 Network B: QA -- 10.10.8.0/21 Servers -- 10.10.8.0/22 NFS -- 10.10.12.0/22 Network C: Staging -- 10.10.16.0/21 Servers -- 10.10.16.0/22 NFS -- 10.10.20.0/22 Network D: Production -- 10.10.24.0/21 Servers -- 10.10.24.0/22 NFS -- 10.10.28.0/22 Network E: Management -- 10.10.32.0/20 Puppet -- 10.10.32.0/21 Drac/Serial -- 10.10.48.0/21 Network F: Bastion Network
[Ok I would love to have done this when I was at RH... but didn't really see it in action til later.]
Basically a box would have 3-4 network connections. The puppet and drac/serial networks are on all systems so have to be extra protected as that is where an attacker could walk from system to system. The bastion network is basically the front end that would do rewrites and other layers so that configs are the same.
And yes, this might be overkill and probably has holes in it.. I am doing it from memory on how a site seemed to be set up and had basically little downtime for critical HR services.
We are actually looking to get more network separation in place but right now thats slow and is going to involve the buildsystem first. But at some point in the not too distant future I would like to separate stg and production environments.
-Mike
On Fri, 05 Sep 2008, Mike McGrath wrote:
So as many of you have seen in the commits lists, the staging environment is coming along and getting built. I've hit a policy issue and so I thought instead of just doing this in a black hole. I'd discuss it.
The way I see it there are two ways to do staging environments. For those of you unfamiliar with staging the general idea is to have an environment as close to production as feasible.
- use identical configs with only minor changes and use /etc/hosts to
fake things to point where you need them. Not always possible but generally good where you can do it.
- use different configs in production and staging. The differences being
able to redirect things, using different usernames, passwords, hostnames, etc.
Each has pros and cons. Right now I'd like to do 1) but I don't think its possible. 2) is going to require a lot of focus. For example... we won't be able to just git merge from staging to production as we could with 1).
Security's only an issue in that we don't want people making changes to production data from staging and vise versa. The same people will have the same access to both of these environments without exception.
I'm going to continue to think about this. I've had staging environments in the past. Both went with option 2). But still. I'd like to hold this discussion so discuss.
What I've done is #2, but with software that accepts a second config file that overrides the first. The first config is the same between production and staging, and the staging specific config is in the second file. That way you can keep your staging settings from accidently getting migrated to production. This works best if you only have a small amount of changes between staging and production and if your software supports it.
Maybe puppet templates for all the config files that differ would be the best way to handle f-i's needs?
infrastructure@lists.fedoraproject.org
-
Daniel Drown -
Mike McGrath -
Stephen John Smoogen