Flock discussions recap/request for input

[Kevin Fenzi]

Greetings. 

At the recent flock conference Infrastructure workshop, we had a nice
lively discussion on a number of items. 

However, as is normal, we don't want to make decisions about things
without being open and allowing input from everyone, including those
that couldn't be at flock. So, I thought I would write up what we
talked about and the consensus we came up with and ask for any more
input from this list before we start implementing things. It's
possible there's something we didn't think about or that needs more
discussion, so do feel free to reply to this email with any parts you
want to comment on. 

* Containers in Fedora Infrastructure: 

  * We want to look at moving things that make sense to containers. 
  * A good initial candidate is the mirrorlist servers. 
  * Would use the existing OSBS build system to build them.
  * Would run on proxies. 
  * Would have haproxy list their socket as primary and old
    mirrorlists as secondary. 
  * The container would have mirrorlist-server wsgi in it along with
    the pkl updated hourly. 
  * Could allow us to spin up more as needed, but also should allow
    faster answers from proxies as they don't have to depend on or
    query over the vpn. 

* Contributor resources in the fedorainfracloud

  * Once our cloud is upgraded, we can use ipsilon to let users login
    to the cloud and spin up instances for Fedora related needs. 
  * Outgoing restrictions would be added on port 25 and the like
  * To start with users would only get 1 external floating ip. 
  * Initial rollout would enable qa and packager groups, need to see if
    docs and i18n or other groups would have a use for it. 
  * would note that we can terminate any instance for any reason. 
  * Patrick would write some scripting to notify users after some time
    and terminate if we didn't get an answer back. 
  * Long term instances should be moved to persistent infra playbooks.

* Build setup and requirements for infrastructure applications. 
  * Will get releng to set us up some side tags that we can build from
    src.rpm in. 
  * all prod builds to be done in koji. 
  * Up to maintainers what priority they place on getting into
    EPEL/Fedora. Encouraged for many reasons. 

* FAS3 status
  * Was running in staging, but we disabled for now until we can finish
    a security audit. 
  * Need to get python-fedora changes lined up and ready/pushed out. 
  * Need to get fas3 fas_client packaged and ready to go. 
  * Need more testing in staging. 
  * Hopefully move production over after f25 is out. 

* Fedora Infrastructure support setup
  * Talked about on the list a fair bit. 
  * Support can be determined by looking at the domain: 
     fedoraproject.org - full 24x7 support, monitoring, uses RFR
     stg.fedoraproject.org - 8x5 support, monitoring
     fedoracommunity.org - some support, monitoring, uses simple RFR
     fedorainfracloud.org - unsupported, apps run by contributors

* Fedora CA and cert infrastructure.
  * Current CA expires in 2018. 
  * Plans being worked on now to back fas3 with freeipa so we could
    move to kerberos tickets for koji then
  * Need to figure out what would need to happen to sigul for that. 
  * Wait and see pending freeipa/fas3 integration. 

* koji alternative arch proposals (on devel list, fesco ticket)
  * Not too much infrastructure work here. 
  * will need to increase storage for primary koji, but can regain from
    secondaries once their last releases go end of life. 

Thats all I had notes on from the workshop, but there may well have
been other items, please do chime in with them if you think of
anything, or have any thoughts on the above. 

kevin

[Jon Stanley]

On Mon, Aug 8, 2016 at 4:40 PM, Kevin Fenzi <kevin(a)scrye.com&gt; wrote:
>  Greetings.
>
>  At the recent flock conference Infrastructure workshop, we had a nice
>  lively discussion on a number of items. 
Sorry I missed it, the first Flock/FUDCon in a long time that I
haven't at least *planned* to attend :). Good news is that I'm in good
health and one piece still :D

>  * Containers in Fedora Infrastructure:
>
>    * We want to look at moving things that make sense to containers.
>    * A good initial candidate is the mirrorlist servers.
>    * Would use the existing OSBS build system to build them.
>    * Would run on proxies.
>    * Would have haproxy list their socket as primary and old
>      mirrorlists as secondary.
>    * The container would have mirrorlist-server wsgi in it along with
>      the pkl updated hourly.
>    * Could allow us to spin up more as needed, but also should allow
>      faster answers from proxies as they don't have to depend on or
>      query over the vpn. 
Sounds like a decent proposal, but I'd like to challenge with the
question "why are we doing this?" Is it just because "containers, new
shiny things, yay!" or is there some real advantage that having this
in containers brings us? How is lifecycle management of these things
accomplished? What happens when a container host dies? Lots of
questions to be answered here. Not that I necessarily think that it's
a bad idea, but containers are disruptive to existing operational
models, and we have to be certain that we're prepared for that.

>  * Contributor resources in the fedorainfracloud
>
>    * Once our cloud is upgraded, we can use ipsilon to let users login
>      to the cloud and spin up instances for Fedora related needs.
>    * Outgoing restrictions would be added on port 25 and the like
>    * To start with users would only get 1 external floating ip.
>    * Initial rollout would enable qa and packager groups, need to see if
>      docs and i18n or other groups would have a use for it.
>    * would note that we can terminate any instance for any reason.
>    * Patrick would write some scripting to notify users after some time
>      and terminate if we didn't get an answer back. 
Even with all of these restrictions, I think that the potential for
abuse exists. I'd like to see these instances (which I think that it's
a good idea to have) be locked down tighter than Fort Knox :). I think
that the only thing that they should be able to externally communicate
with should be the rest of Infrastructure (koji and the like) just
like anybody on the Internet communicates with those services. They
should be no more trusted than the computer that I'm writing this on
right now is, and perhaps even less so (if that's possible and makes
sense).

>    * Long term instances should be moved to persistent infra
> playbooks. 
Yep.

>  * Build setup and requirements for infrastructure applications.
>    * Will get releng to set us up some side tags that we can build from
>      src.rpm in.
>    * all prod builds to be done in koji.
>    * Up to maintainers what priority they place on getting into
>      EPEL/Fedora. Encouraged for many reasons. 
Nothing earth-shattering here :). All good ideas :).

>  * FAS3 status
>    * Was running in staging, but we disabled for now until we can finish
>      a security audit.
>    * Need to get python-fedora changes lined up and ready/pushed out.
>    * Need to get fas3 fas_client packaged and ready to go.
>    * Need more testing in staging.
>    * Hopefully move production over after f25 is out.
>
>  * Fedora Infrastructure support setup
>    * Talked about on the list a fair bit.
>    * Support can be determined by looking at the domain:
>       fedoraproject.org - full 24x7 support, monitoring, uses RFR
>       stg.fedoraproject.org - 8x5 support, monitoring
>       fedoracommunity.org - some support, monitoring, uses simple RFR
>       fedorainfracloud.org - unsupported, apps run by contributors 
+1

>  * Fedora CA and cert infrastructure.
>    * Current CA expires in 2018.
>    * Plans being worked on now to back fas3 with freeipa so we could
>      move to kerberos tickets for koji then
>    * Need to figure out what would need to happen to sigul for that.
>    * Wait and see pending freeipa/fas3 integration. 
This sounds reasonable, but what happens if for some reason the
integration doesn't happen or doesn't happen in a timely manner? I
don't think that renewing the cert would be a huge deal, because the
users of the certs generated by that CA are a well-known quantity
(packagers and releng). The support burden of swapping out the CA cert
I wouldn't think would be *that* bad. I'm not sure off the top of my
head, how often do the user certs expire?

[Kevin Fenzi]

On Mon, 8 Aug 2016 21:41:16 -0400
Jon Stanley <jonstanley(a)gmail.com&gt; wrote:

>  On Mon, Aug 8, 2016 at 4:40 PM, Kevin Fenzi <kevin(a)scrye.com&gt;
> wrote:
>  > Greetings.
>  >
>  > At the recent flock conference Infrastructure workshop, we had a
>  > nice lively discussion on a number of items.  
>  
>  Sorry I missed it, the first Flock/FUDCon in a long time that I
>  haven't at least *planned* to attend :). Good news is that I'm in good
>  health and one piece still :D 
Yeah, sorry you couldn't make it either. ;( There were a number of
regulars who were quite missed. ;( 

>  > * Containers in Fedora Infrastructure:
>  >
>  >   * We want to look at moving things that make sense to containers.
>  >   * A good initial candidate is the mirrorlist servers.
>  >   * Would use the existing OSBS build system to build them.
>  >   * Would run on proxies.
>  >   * Would have haproxy list their socket as primary and old
>  >     mirrorlists as secondary.
>  >   * The container would have mirrorlist-server wsgi in it along with
>  >     the pkl updated hourly.
>  >   * Could allow us to spin up more as needed, but also should allow
>  >     faster answers from proxies as they don't have to depend on or
>  >     query over the vpn.  
>  
>  Sounds like a decent proposal, but I'd like to challenge with the
>  question "why are we doing this?" Is it just because "containers, new
>  shiny things, yay!" or is there some real advantage that having this
>  in containers brings us?  
An excellent question. :) We could of course just run a
mirrorlist-server instance on each of the proxies outside containers,
but I think containers gives us some good wins here: 

* would allow us to spin up new ons in the cloud if we know load is
  going to be high. 

* would keep logging and such in the container, so we don't need to
  untangle that from the rest of the things running there. 

>  How is lifecycle management of these things
>  accomplished? What happens when a container host dies? Lots of
>  questions to be answered here. Not that I necessarily think that it's
>  a bad idea, but containers are disruptive to existing operational
>  models, and we have to be certain that we're prepared for that. 
Sure, there will be lots of questions as we move forward on it. 
I'd guess at first lifecycle would be somewhat manual (controlled by
ansible), and if a container host dies, then we remove it from dns just
as we do now. 

>  > * Contributor resources in the fedorainfracloud
>  >
>  >   * Once our cloud is upgraded, we can use ipsilon to let users
>  > login to the cloud and spin up instances for Fedora related needs.
>  >   * Outgoing restrictions would be added on port 25 and the like
>  >   * To start with users would only get 1 external floating ip.
>  >   * Initial rollout would enable qa and packager groups, need to
>  > see if docs and i18n or other groups would have a use for it.
>  >   * would note that we can terminate any instance for any reason.
>  >   * Patrick would write some scripting to notify users after some
>  > time and terminate if we didn't get an answer back.  
>  
>  Even with all of these restrictions, I think that the potential for
>  abuse exists. I'd like to see these instances (which I think that it's
>  a good idea to have) be locked down tighter than Fort Knox :). I think
>  that the only thing that they should be able to externally communicate
>  with should be the rest of Infrastructure (koji and the like) just
>  like anybody on the Internet communicates with those services. They
>  should be no more trusted than the computer that I'm writing this on
>  right now is, and perhaps even less so (if that's possible and makes
>  sense). 
Well, we are opening these to packager and qa to start with. This is
hardly an untrusted group. Additionally it should be easy to blacklist
folks who abuse things. I guess the amount we need to lock things down
depends somewhat on the use cases people come up with. 

...snip...

>  > * Fedora CA and cert infrastructure.
>  >   * Current CA expires in 2018.
>  >   * Plans being worked on now to back fas3 with freeipa so we could
>  >     move to kerberos tickets for koji then
>  >   * Need to figure out what would need to happen to sigul for that.
>  >   * Wait and see pending freeipa/fas3 integration.  
>  
>  This sounds reasonable, but what happens if for some reason the
>  integration doesn't happen or doesn't happen in a timely manner? I
>  don't think that renewing the cert would be a huge deal, because the
>  users of the certs generated by that CA are a well-known quantity
>  (packagers and releng). The support burden of swapping out the CA cert
>  I wouldn't think would be *that* bad. I'm not sure off the top of my
>  head, how often do the user certs expire? 
It's every 6 months for user certs. Yeah, we could extend the CA expire
time if needed. 

kevin

[Patrick Uiterwijk]

... snip ...
> > > * Contributor resources in the fedorainfracloud
> > >
> > >   * Once our cloud is upgraded, we can use ipsilon to let users
> > > login to the cloud and spin up instances for Fedora related needs.
> > >   * Outgoing restrictions would be added on port 25 and the like
> > >   * To start with users would only get 1 external floating ip.
> > >   * Initial rollout would enable qa and packager groups, need to
> > > see if docs and i18n or other groups would have a use for it.
> > >   * would note that we can terminate any instance for any reason.
> > >   * Patrick would write some scripting to notify users after some
> > > time and terminate if we didn't get an answer back.
> >
> > Even with all of these restrictions, I think that the potential for
> > abuse exists. I'd like to see these instances (which I think that it's
> > a good idea to have) be locked down tighter than Fort Knox :). I think
> > that the only thing that they should be able to externally communicate
> > with should be the rest of Infrastructure (koji and the like) just
> > like anybody on the Internet communicates with those services. They
> > should be no more trusted than the computer that I'm writing this on
> > right now is, and perhaps even less so (if that's possible and makes
> > sense).
>
>  Well, we are opening these to packager and qa to start with. This is
>  hardly an untrusted group. Additionally it should be easy to blacklist
>  folks who abuse things. I guess the amount we need to lock things down
>  depends somewhat on the use cases people come up with. 
Note that the cloud instances are totally untrusted by our internal
infrastructure,
as far as the other boxes concern, they're just internet.

>  ...snip...
>
> > > * Fedora CA and cert infrastructure.
> > >   * Current CA expires in 2018.
> > >   * Plans being worked on now to back fas3 with freeipa so we could
> > >     move to kerberos tickets for koji then
> > >   * Need to figure out what would need to happen to sigul for that.
> > >   * Wait and see pending freeipa/fas3 integration.
> >
> > This sounds reasonable, but what happens if for some reason the
> > integration doesn't happen or doesn't happen in a timely manner? I
> > don't think that renewing the cert would be a huge deal, because the
> > users of the certs generated by that CA are a well-known quantity
> > (packagers and releng). The support burden of swapping out the CA cert
> > I wouldn't think would be *that* bad. I'm not sure off the top of my
> > head, how often do the user certs expire?
>
>  It's every 6 months for user certs. Yeah, we could extend the CA expire
>  time if needed. 
The integration code is now live in staging, and is fairly small and
self-contained.

>
> kevin
>
> _______________________________________________
> infrastructure mailing list
> infrastructure(a)lists.fedoraproject.org
> https://lists.fedoraproject.org/admin/lists/[e-mail]...
>