Support levels and RFR adjustments

[Kevin Fenzi]

So, I sent an email a while back about this to get people thinking, but
I didn't get too much feedback from my questions, so this time I am
going to actually outline a proposal for people to look at. ;) 

Currently users expect pretty much any public service we have is fully
supported. This means things like updating status when it's down,
working anytime something is down to fix it as quickly as we can. 
New applications/services currently all pass through the (somewhat
long) RFR process which we setup to make sure we could support the
service moving forward.

This is great and all, but some services just aren't as sustainable, or
don't really fit into our RFR process very well. Also, our RFR process
makes us pretty slow to bring a new service online properly. 

In order to have support levels, we need a way to communicate that to
our users easily and the only/best way I can think of to do that easily
is via domain name. If we try and have a table or something it could
get pretty confusing for people. Tying it to domain names would make it
much easier.

So: 

fedoraproject.org - Anything with this domain is something that has
passed though our RFR process and we support fully. This means we
update status, we alert on them anytime they have issues, we work on
them anytime they are down, etc. 

fedorainfracloud.org - This comes with a lesser level of support,
simply because our cloud doesn't have any kind of HA setup, so
it will be down when doing maint or when there's problems. Services in
this domain may be down when there is scheduled cloud maint. We
monitor, but don't page off hours, we may work on issues only during
business hours, etc. Services here may not have passed through our RFR
process (perhaps we should have a parallel cloud process) 

stg.fedoraproject.org - These can be down anytime and we monitor on
them, but may not work on them off hours, etc. 

someother domain that sounds fedora related (fedorarelated.org?
fedoralinks.org? ?) - These are things that are fedora related, but not
fully controlled by fedora infrastructure. Things like the fedora
bootstrap site or the porting python3 in fedora site, or possibly cloud
instances that aren't managed by us. These we don't monitor or have
status on, and direct people to contact the managers directly. 

Any other types of sites / domains people can think of?

Any general thoughts on the idea?

kevin

[Stephen John Smoogen]

On 22 January 2016 at 14:33, Kevin Fenzi <kevin(a)scrye.com&gt; wrote:
>  So, I sent an email a while back about this to get people thinking,
> but
>  I didn't get too much feedback from my questions, so this time I am
>  going to actually outline a proposal for people to look at. ;)
>
>  Currently users expect pretty much any public service we have is fully
>  supported. This means things like updating status when it's down,
>  working anytime something is down to fix it as quickly as we can.
>  New applications/services currently all pass through the (somewhat
>  long) RFR process which we setup to make sure we could support the
>  service moving forward.
>  
This looks good but  I had to go look up what RFR meant and what its process:

Request for Resources
https://fedoraproject.org/wiki/Request_For_Resources


>  This is great and all, but some services just aren't as
> sustainable, or
>  don't really fit into our RFR process very well. Also, our RFR process
>  makes us pretty slow to bring a new service online properly.
>
>  In order to have support levels, we need a way to communicate that to
>  our users easily and the only/best way I can think of to do that easily
>  is via domain name. If we try and have a table or something it could
>  get pretty confusing for people. Tying it to domain names would make it
>  much easier.
>
>  So:
>
>  fedoraproject.org - Anything with this domain is something that has
>  passed though our RFR process and we support fully. This means we
>  update status, we alert on them anytime they have issues, we work on
>  them anytime they are down, etc.
>  
Usually when doing "support levels" there is the need to come up with
response times. I don't know if we can really do that since we don't
have business hours and such.

>  fedorainfracloud.org - This comes with a lesser level of support,
>  simply because our cloud doesn't have any kind of HA setup, so
>  it will be down when doing maint or when there's problems. Services in
>  this domain may be down when there is scheduled cloud maint. We
>  monitor, but don't page off hours, we may work on issues only during
>  business hours, etc. Services here may not have passed through our RFR
>  process (perhaps we should have a parallel cloud process)
>  
Things like copr go under fedorainfracloud.org correct?

>  stg.fedoraproject.org - These can be down anytime and we monitor on
>  them, but may not work on them off hours, etc.
>
>  someother domain that sounds fedora related (fedorarelated.org?
>  fedoralinks.org? ?) - These are things that are fedora related, but not
>  fully controlled by fedora infrastructure. Things like the fedora
>  bootstrap site or the porting python3 in fedora site, or possibly cloud
>  instances that aren't managed by us. These we don't monitor or have
>  status on, and direct people to contact the managers directly.
>
>  Any other types of sites / domains people can think of?
>  
Things we get pinged on at times which are Red Hat related but we
don't control. I know we get occasional "gnome.org?" and
"softwarecollections.org" and similar questions which we know who to
contact but have nothing but that.


>  Any general thoughts on the idea?
>
>  kevin
>
>  _______________________________________________
>  infrastructure mailing list
>  infrastructure(a)lists.fedoraproject.org
>  http://lists.fedoraproject.org/admin/lists/[e-mail]...
>  


-- 
Stephen J Smoogen.

[Kevin Fenzi]

On Fri, 22 Jan 2016 14:45:08 -0700
Stephen John Smoogen <smooge(a)gmail.com&gt; wrote:

...snip...

>  
>  Usually when doing "support levels" there is the need to come up with
>  response times. I don't know if we can really do that since we don't
>  have business hours and such. 
Well, we could try and do that, but It might be hard... we could do
some with a fair bit of wiggle room at first then adjust I suppose. 

>  Things like copr go under fedorainfracloud.org correct? 
That was the thought as currently setup. 

>  Things we get pinged on at times which are Red Hat related but we
>  don't control. I know we get occasional "gnome.org?" and
>  "softwarecollections.org" and similar questions which we know who to
>  contact but have nothing but that. 
Yeah, but not sure what we can do about those aside from saying "Sorry,
we don't handle those, try contacting XYZ"

kevin

[Patrick Uiterwijk]

>  So, I sent an email a while back about this to get people thinking,
> but
>  I didn't get too much feedback from my questions, so this time I am
>  going to actually outline a proposal for people to look at. ;)
>  
>  Currently users expect pretty much any public service we have is fully
>  supported. This means things like updating status when it's down,
>  working anytime something is down to fix it as quickly as we can.
>  New applications/services currently all pass through the (somewhat
>  long) RFR process which we setup to make sure we could support the
>  service moving forward.
>  
>  This is great and all, but some services just aren't as sustainable, or
>  don't really fit into our RFR process very well. Also, our RFR process
>  makes us pretty slow to bring a new service online properly.
>  
>  In order to have support levels, we need a way to communicate that to
>  our users easily and the only/best way I can think of to do that easily
>  is via domain name. If we try and have a table or something it could
>  get pretty confusing for people. Tying it to domain names would make it
>  much easier. 
Just domain would be a good way for us internally, but maybe we can also
get the design people to provide us with banners or different versions of
the logo to put on stg/dev/cloud/... instances, so that we also make it
clear inside the applications in a consistent manner.

>  
>  So:
>  
>  fedoraproject.org - Anything with this domain is something that has
>  passed though our RFR process and we support fully. This means we
>  update status, we alert on them anytime they have issues, we work on
>  them anytime they are down, etc. 
Maybe clearly indicate cloud.fp.o (and some others probably) as exceptions
to this rule.

getfedora.org - Same level as fedoraproject.org.

>  
>  fedorainfracloud.org - This comes with a lesser level of support,
>  simply because our cloud doesn't have any kind of HA setup, so
>  it will be down when doing maint or when there's problems. Services in
>  this domain may be down when there is scheduled cloud maint. We
>  monitor, but don't page off hours, we may work on issues only during
>  business hours, etc. Services here may not have passed through our RFR
>  process (perhaps we should have a parallel cloud process) 
cloud.fedoraproject.org - Same level as fedorainfracloud.org.

>  
>  stg.fedoraproject.org - These can be down anytime and we monitor on
>  them, but may not work on them off hours, etc.
>  
>  someother domain that sounds fedora related (fedorarelated.org?
>  fedoralinks.org? ?) - These are things that are fedora related, but not
>  fully controlled by fedora infrastructure. Things like the fedora
>  bootstrap site or the porting python3 in fedora site, or possibly cloud
>  instances that aren't managed by us. These we don't monitor or have
>  status on, and direct people to contact the managers directly.
>  
>  Any other types of sites / domains people can think of? 
Where do hosted, people and planet fall in?
I would say these are production as well, and same as fp.o.

>  
>  Any general thoughts on the idea? 
Outside of the indications to users, how about defining "SLA levels", or
however we want to call it, and display the above rules in a table, for
easy grokking by other people?

Something like:
             Status | Monitored | Paged |  Off-hours
Production |   X          X         X         -
Staging    |   -          X         -         -

Or however we want to fill this in exactly, just a quick example, we might
for example give different names to the levels or the sort.

>  
>  kevin
>   
With kind regards,
Patrick Uiterwijk
Fedora Infra

[Kevin Fenzi]

On Fri, 22 Jan 2016 16:45:49 -0500 (EST)
Patrick Uiterwijk <puiterwijk(a)redhat.com&gt; wrote:

>  Just domain would be a good way for us internally, but maybe we can
>  also get the design people to provide us with banners or different
>  versions of the logo to put on stg/dev/cloud/... instances, so that
>  we also make it clear inside the applications in a consistent manner. 
I suppose, but when something is down people would look at the banner
how? ;) Thats why I think domain names are a good way to show things. 

>  Maybe clearly indicate cloud.fp.o (and some others probably) as
>  exceptions to this rule. 
Sure, but I was hoping to phase out cloud.fp.o in favor of
fedorainfracloud. 
 
>  getfedora.org - Same level as fedoraproject.org. 
Yeah. 

In fact, further thought on this makes me think we have one level
higher too... mirrorlists, hotspot.txt, ie things end users
immediately note. However, it might just be muddying the water to try
and mention these as higher support level. 

>  cloud.fedoraproject.org - Same level as fedorainfracloud.org. 
Yeah

>  Where do hosted, people and planet fall in?
>  I would say these are production as well, and same as fp.o. 
Yeah, agreed. Although I could see a case for people and planet being
in a lower category. Probibly best to put them with fp.o

>  > Any general thoughts on the idea?  
>  
>  Outside of the indications to users, how about defining "SLA levels",
>  or however we want to call it, and display the above rules in a
>  table, for easy grokking by other people?
>  
>  Something like:
>               Status | Monitored | Paged |  Off-hours
>  Production |   X          X         X         -
>  Staging    |   -          X         -         -
>  
>  Or however we want to fill this in exactly, just a quick example, we
>  might for example give different names to the levels or the sort. 
Sounds like a good idea. Also possibly the response time there... 

kevin

[Miroslav Suchý]

Dne 25.1.2016 v 18:06 Kevin Fenzi napsal(a):
>  Sure, but I was hoping to phase out cloud.fp.o in favor of
>  fedorainfracloud.  
Hmm. There is a *lot* of people who have .repo file pointing to
copr-be.cloud.fedoraproject.org.

-- 
Miroslav Suchy, RHCA
Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys

[Miroslav Suchý]

Dne 22.1.2016 v 22:33 Kevin Fenzi napsal(a):
>  fedoraproject.org - Anything with this domain is something that has
>  passed though our RFR process and we support fully. This means we
>  update status, we alert on them anytime they have issues, we work on
>  them anytime they are down, etc. 
>  
>  fedorainfracloud.org - This comes with a lesser level of support,
>  simply because our cloud doesn't have any kind of HA setup, so
>  it will be down when doing maint or when there's problems. Services in
>  this domain may be down when there is scheduled cloud maint. We
>  monitor, but don't page off hours, we may work on issues only during
>  business hours, etc. Services here may not have passed through our RFR
>  process (perhaps we should have a parallel cloud process)  
I personally do not like fedorainfracloud.org. It is fine for hostnames of machines in
cloud. I would rather see some
subdomain under fedoraproject.org - e.g. .devel.fedoraproject.org or
.playground.fedoraproject.org or something similar.

I personally dislike the change of copr.fedoraproject.org to copr.fedorainfracloud.org. We
have been usin this name for
past two years and it is referenced a lot (35k by Google):
 
https://www.google.com/search?lr=&hl=cs&q=%22copr.fedoraproject.o...
This is not pure technical decision but marketing decision as well.
A lot of people are treating it as integral part of Fedora. So instead of changing
hostname I would rather start RFR
process.

I think that some icon at the top or link at the bottom of page, which will clearly state
the level of support will do
the same from technical POV, but will be better solution from marketing POV.

-- 
Miroslav Suchy, RHCA
Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys

[Kevin Fenzi]

On Mon, 25 Jan 2016 15:29:05 +0100
Miroslav Suchý <msuchy(a)redhat.com&gt; wrote:

>  I personally do not like fedorainfracloud.org. It is fine for
>  hostnames of machines in cloud. I would rather see some subdomain
>  under fedoraproject.org - e.g. .devel.fedoraproject.org
>  or .playground.fedoraproject.org or something similar. 
Well, it's in the cloud, so it's more descriptive, IMHO. 
And sorry for the quick change, we needed new ssl certs very soon, so
thought it would make sense to get them with these names. 
>  
>  I personally dislike the change of copr.fedoraproject.org to
>  copr.fedorainfracloud.org. We have been usin this name for past two
>  years and it is referenced a lot (35k by Google):
>
> https://www.google.com/search?lr=&hl=cs&q=%22copr.fedoraproject.o...
>  This is not pure technical decision but marketing decision as well. A
>  lot of people are treating it as integral part of Fedora. So instead
>  of changing hostname I would rather start RFR process. 
Well, for what? We have talked about moving the non builder parts of
copr into regular infrastructure in the past, but we didn't do it for
whatever reasons. Doing so would get us some advantages and some
problems: 

Advantages: could use the proxy system to cache things and also do HA
with multiple frontends or possibly even backends if we wanted. Could
mean frontend/backend/sign/git are more stable as they are not on the
cloud. 

Disadvantages: would need to figure out how to shuffle around storage,
as we have copr storage tied to the cloud right now. It would be some
work to move things around and get it all working right. 

>  I think that some icon at the top or link at the bottom of page,
>  which will clearly state the level of support will do the same from
>  technical POV, but will be better solution from marketing POV. 
The problem with that is when the thing is down, users have no way to
look at that. They just see it's down and have the expectation that
they already do in their minds. 

kevin

[Paul W. Frields]

On Mon, Jan 25, 2016 at 10:12:47AM -0700, Kevin Fenzi wrote:
>  On Mon, 25 Jan 2016 15:29:05 +0100
>  Miroslav Suchý <msuchy(a)redhat.com&gt; wrote:
>  
>  > I personally do not like fedorainfracloud.org. It is fine for
>  > hostnames of machines in cloud. I would rather see some subdomain
>  > under fedoraproject.org - e.g. .devel.fedoraproject.org
>  > or .playground.fedoraproject.org or something similar.
>  
>  Well, it's in the cloud, so it's more descriptive, IMHO. 
>  And sorry for the quick change, we needed new ssl certs very soon, so
>  thought it would make sense to get them with these names. 
>  > 
>  > I personally dislike the change of copr.fedoraproject.org to
>  > copr.fedorainfracloud.org. We have been usin this name for past two
>  > years and it is referenced a lot (35k by Google):
>  >
> https://www.google.com/search?lr=&hl=cs&q=%22copr.fedoraproject.o...
>  > This is not pure technical decision but marketing decision as well. A
>  > lot of people are treating it as integral part of Fedora. So instead
>  > of changing hostname I would rather start RFR process.
>  
>  Well, for what? We have talked about moving the non builder parts of
>  copr into regular infrastructure in the past, but we didn't do it for
>  whatever reasons. Doing so would get us some advantages and some
>  problems: 
>  
>  Advantages: could use the proxy system to cache things and also do HA
>  with multiple frontends or possibly even backends if we wanted. Could
>  mean frontend/backend/sign/git are more stable as they are not on the
>  cloud. 
>  
>  Disadvantages: would need to figure out how to shuffle around storage,
>  as we have copr storage tied to the cloud right now. It would be some
>  work to move things around and get it all working right. 
>  
>  > I think that some icon at the top or link at the bottom of page,
>  > which will clearly state the level of support will do the same from
>  > technical POV, but will be better solution from marketing POV.
>  
>  The problem with that is when the thing is down, users have no way to
>  look at that. They just see it's down and have the expectation that
>  they already do in their minds.  
I was talking to Kevin about this today and came up with what I think
is a good solution.

We have a fedoracommunity.org domain which already is provided for
communities that want to set up Fedora related sites.  Those sites
are:

* understood as owned and managed by someone other than the Fedora
  infrastructure team

* granted trademark approval to use Fedora marks to associate closely
  with the Fedora Project

Currently this domain is used exclusively for local communities.  For
example, a Czech community might run a site for Czech speaking
community members at cz.fedoracommunity.org, and we simply provide a
DNS pointer for them.

However, the use case is exactly the same for a technical project not
hosted/managed by the Infrastructure team.  The owner seeks to brand
it as Fedora and associate it with the Project, which is good for
everyone involved.  But it doesn't automatically inherit support from
the team.

Expanding fedoracommunity.org for this use makes a lot of sense to me.
*And* it brings an immediate affiliation with Fedora just from the
name, without misunderstanding.  It's very clear the site is community
run/supported.  Thoughts?


-- 
Paul W. Frields                                http://paul.frields.org/
  gpg fingerprint: 3DA6 A0AC 6D58 FEC4 0233  5906 ACDB C937 BD11 3717
  http://redhat.com/   -  -  -  -   http://pfrields.fedorapeople.org/
    The open source story continues to grow: http://opensource.com

[David Shier]

I think this is a good idea, maybe add a sub header similar to 'technical
project sites in this region' to differentiate between something that is a
more or less general community forum and the ones where you've got
development types that likely want more focused conversation related to
their projects.

[Paul W. Frields]

On Tue, Mar 08, 2016 at 07:22:44PM -0500, David Shier wrote:
>  I think this is a good idea, maybe add a sub header similar to
> 'technical
>  project sites in this region' to differentiate between something that is a
>  more or less general community forum and the ones where you've got
>  development types that likely want more focused conversation related to
>  their projects. 
Reminder since snipped: The context is that I recommended we use
*.fedoracommunity.org for projects outside regular support from the
infrastructure team, but which are good to associate with Fedora
overall, with COPR being an example.

What do we need to do to make this happen, if there are no objections?

-- 
Paul W. Frields                                http://paul.frields.org/
  gpg fingerprint: 3DA6 A0AC 6D58 FEC4 0233  5906 ACDB C937 BD11 3717
  http://redhat.com/   -  -  -  -   http://pfrields.fedorapeople.org/
    The open source story continues to grow: http://opensource.com

[Kevin Fenzi]

On Wed, 23 Mar 2016 17:38:52 -0400
"Paul W. Frields" <stickster(a)gmail.com&gt; wrote:

>  On Tue, Mar 08, 2016 at 07:22:44PM -0500, David Shier wrote:
>  > I think this is a good idea, maybe add a sub header similar to
>  > 'technical project sites in this region' to differentiate between
>  > something that is a more or less general community forum and the
>  > ones where you've got development types that likely want more
>  > focused conversation related to their projects.  
>  
>  Reminder since snipped: The context is that I recommended we use
>  *.fedoracommunity.org for projects outside regular support from the
>  infrastructure team, but which are good to associate with Fedora
>  overall, with COPR being an example. 
Hum, I am not sure copr is a good example, but perhaps I
misunderstood. :) I was thinking this was additive with other stuff: 

- fedoraproject.org -> fully supported

- fedorainfracloud.org -> lesser support 
(right now, I would say copr is here, but moving to the first one)

- stg.fedoraproject.org -> even less

- fedoracommunity.org -> we don't run it, but it's associated. 

So, here I would put things like: 

bootstrap.fedoraproject.org 
python3 porting page
etc
 
>  What do we need to do to make this happen, if there are no
> objections? 
I think: 

* we need a writeup on the wiki and/or infra-docs with this setup. 

* we need to put everything into one of the levels and get feedback

* adjust things like nagios or SOPs for the levels. 

* announce it 

* profit 

kevin

[Pierre-Yves Chibon]

On Thu, Mar 24, 2016 at 06:36:42AM -0600, Kevin Fenzi wrote:
>  On Wed, 23 Mar 2016 17:38:52 -0400
>  "Paul W. Frields" <stickster(a)gmail.com&gt; wrote:
>  
>  > On Tue, Mar 08, 2016 at 07:22:44PM -0500, David Shier wrote:
>  > > I think this is a good idea, maybe add a sub header similar to
>  > > 'technical project sites in this region' to differentiate between
>  > > something that is a more or less general community forum and the
>  > > ones where you've got development types that likely want more
>  > > focused conversation related to their projects.  
>  > 
>  > Reminder since snipped: The context is that I recommended we use
>  > *.fedoracommunity.org for projects outside regular support from the
>  > infrastructure team, but which are good to associate with Fedora
>  > overall, with COPR being an example.
>  
>  Hum, I am not sure copr is a good example, but perhaps I
>  misunderstood. :) I was thinking this was additive with other stuff: 
>  
>  - fedoraproject.org -> fully supported
>  
>  - fedorainfracloud.org -> lesser support 
>  (right now, I would say copr is here, but moving to the first one)
>  
>  - stg.fedoraproject.org -> even less
>  
>  - fedoracommunity.org -> we don't run it, but it's associated.  
I like this way of dividing our support and make clear what we run and what we
don't.

>  > What do we need to do to make this happen, if there are no
> objections?
>  
>  I think: 
>  
>  * we need a writeup on the wiki and/or infra-docs with this setup. 
>  
>  * we need to put everything into one of the levels and get feedback 
The devel/devel-announce lists? Or would we have a better place to get this
feedback?


Pierre

[Kevin Fenzi]

On Wed, 6 Apr 2016 12:41:09 +0200
Pierre-Yves Chibon <pingou(a)pingoured.fr&gt; wrote:

>  
>  The devel/devel-announce lists? Or would we have a better place to
>  get this feedback? 
I think that might be a good start yeah. 

But we first need to write up a wiki page/doc to bring to them

kevin