IAD2 datacenter testing/application validation

[Kevin Fenzi]

Greetings everyone.

We now have most things deployed in our new datacenter (IAD2).

Accordingly, I would like to get some testing and validation started.

Please see the following document:

https://hackmd.io/op6N_nIaR7aMzw9Ib-sDAQ

Please feel free to ask questions here and add/check off services that
appear to be running as expected. Application owners (people in
sysadmin-whatever groups) should check that they have the expected level
of access as before, that their playbooks run (idempotently!) and the
logs and any interfaces show the application appears to be running fine.

We have this week and next to get everything in good shape for migration
week (20202-06-08) when we will moving everything out of phx2.

Thanks for any help. :)

kevin

[Michal Konecny]

I don't see the-new-hotness anywhere. Will it be deployed together with 
Anitya?

Michal

On 27/05/2020 00:22, Kevin Fenzi wrote:
> Greetings everyone.
>
> We now have most things deployed in our new datacenter (IAD2).
>
> Accordingly, I would like to get some testing and validation started.
>
> Please see the following document:
>
> https://hackmd.io/op6N_nIaR7aMzw9Ib-sDAQ
>
> Please feel free to ask questions here and add/check off services that
> appear to be running as expected. Application owners (people in
> sysadmin-whatever groups) should check that they have the expected level
> of access as before, that their playbooks run (idempotently!) and the
> logs and any interfaces show the application appears to be running fine.
>
> We have this week and next to get everything in good shape for migration
> week (20202-06-08) when we will moving everything out of phx2.
>
> Thanks for any help. :)
>
> kevin
>
> infrastructure mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedoraproject.org/archives/list/[e-mail]...

[Clement Verna]

On Wed, 27 May 2020 at 09:42, Michal Konecny  wrote:
> I don't see the-new-hotness anywhere. Will it be deployed together with
> Anitya?

I think these 2 apps were in the Minimal Viable Fedora infra so they will
be deployed but in the second phase when all the hardware is available in
IAD2.
> Michal
>
> On 27/05/2020 00:22, Kevin Fenzi wrote:
>
> Greetings everyone.
>
> We now have most things deployed in our new datacenter (IAD2).
>
> Accordingly, I would like to get some testing and validation started.
>
> Please see the following document:
> https://hackmd.io/op6N_nIaR7aMzw9Ib-sDAQ
>
> Please feel free to ask questions here and add/check off services that
> appear to be running as expected. Application owners (people in
> sysadmin-whatever groups) should check that they have the expected level
> of access as before, that their playbooks run (idempotently!) and the
> logs and any interfaces show the application appears to be running fine.
>
> We have this week and next to get everything in good shape for migration
> week (20202-06-08) when we will moving everything out of phx2.
>
> Thanks for any help. :)
>
> kevin
>
> infrastructure mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedoraproject.org/archives/list/[e-mail]...
>
> --
> Role: Fedora CPE Team - Software Engineer
> IRC: mkonecny
> FAS: zlopez
>
> infrastructure mailing list -- [e-mail]
> To unsubscribe send an email to
> [e-mail]
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]...

[Pierre-Yves Chibon]

On Wed, May 27, 2020 at 10:29:20AM +0200, Clement Verna wrote:
> On Wed, 27 May 2020 at 09:42, Michal Konecny <[1][e-mail]>
>    wrote:
>  I don't see the-new-hotness anywhere. Will it be deployed together with
>  Anitya?
>
> I think these 2 apps were in the Minimal Viable Fedora infra so they will
>    be deployed but in the second phase when all the hardware is available in
>    IAD2.

Were or were not (in the Minimal Viable Fedora)? :)

Pierre

[Clement Verna]

On Wed, 27 May 2020 at 11:57, Pierre-Yves Chibon 
wrote:
> On Wed, May 27, 2020 at 10:29:20AM +0200, Clement Verna wrote:
> > On Wed, 27 May 2020 at 09:42, Michal Konecny <[1][e-mail]>
> >    wrote:
> >  I don't see the-new-hotness anywhere. Will it be deployed together
>
> with
> >  Anitya?
> >
> > I think these 2 apps were in the Minimal Viable Fedora infra so they
>
> will
> > be deployed but in the second phase when all the hardware is
>
> available in
> > IAD2.
>
> Were or were not (in the Minimal Viable Fedora)? :)

I guess "were not" but I can't be 100% sure about that :P
> Pierre
> _______________________________________________
> infrastructure mailing list -- [e-mail]
> To unsubscribe send an email to
> [e-mail]
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]...

[Kevin Fenzi]

On Wed, May 27, 2020 at 02:39:13PM +0200, Clement Verna wrote:
> On Wed, 27 May 2020 at 11:57, Pierre-Yves Chibon 
> wrote:
> > On Wed, May 27, 2020 at 10:29:20AM +0200, Clement Verna wrote:
> > > On Wed, 27 May 2020 at 09:42, Michal Konecny <[1][e-mail]>
> > >    wrote:
> > >  I don't see the-new-hotness anywhere. Will it be deployed together
> >
> > with
> > >  Anitya?
> > >
> > > I think these 2 apps were in the Minimal Viable Fedora infra so they
> >
> > will
> > > be deployed but in the second phase when all the hardware is
> >
> > available in
> > > IAD2.
> >
> > Were or were not (in the Minimal Viable Fedora)? :)
>
> I guess "were not" but I can't be 100% sure about that :P

Right, they were not in MVF.

That said, they are in openshift and we have the cluster up, so we could
go ahead and just deploy them anyhow. I'd suggest waiting until after
move week tho to avoid splitting attention... so anytime after the 12th
we could deploy them.

kevin

[Michal Konecny]

Thanks for the answer, Clement.

Michal

On 27/05/2020 10:29, Clement Verna wrote:
> On Wed, 27 May 2020 at 09:42, Michal Konecny <[e-mail] 
> > wrote:
> I don't see the-new-hotness anywhere. Will it be deployed together
> with Anitya?
>
>
> I think these 2 apps were in the Minimal Viable Fedora infra so they 
> will be deployed but in the second phase when all the hardware is 
> available in IAD2.
> Michal
>
> On 27/05/2020 00:22, Kevin Fenzi wrote:
> > Greetings everyone.
> >
> > We now have most things deployed in our new datacenter (IAD2).
> >
> > Accordingly, I would like to get some testing and validation started.
> >
> > Please see the following document:
> >
> > https://hackmd.io/op6N_nIaR7aMzw9Ib-sDAQ
> >
> > Please feel free to ask questions here and add/check off services that
> > appear to be running as expected. Application owners (people in
> > sysadmin-whatever groups) should check that they have the expected level
> > of access as before, that their playbooks run (idempotently!) and the
> > logs and any interfaces show the application appears to be running fine.
> >
> > We have this week and next to get everything in good shape for migration
> > week (20202-06-08) when we will moving everything out of phx2.
> >
> > Thanks for any help. :)
> >
> > kevin
> >
> > _______________________________________________
> > infrastructure mailing list [e-mail]  <mailto:[e-mail]>
> > To unsubscribe send an email [e-mail]  <mailto:[e-mail]>
> > Fedora Code of Conduct:https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > List Guidelines:https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives:https://lists.fedoraproject.org/archives/list/[e-mail]
> -- 
> Role: Fedora CPE Team - Software Engineer
> IRC: mkonecny
> FAS: zlopez
>
> _______________________________________________
> infrastructure mailing list --
> [e-mail]
> <mailto:[e-mail]>
> To unsubscribe send an email to
> [e-mail]
> <mailto:[e-mail]>
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines:
> https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]
>
>
> infrastructure mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedoraproject.org/archives/list/[e-mail]...

[Adrian Reber]

On Tue, May 26, 2020 at 03:22:02PM -0700, Kevin Fenzi wrote:
> Greetings everyone.
>
> We now have most things deployed in our new datacenter (IAD2).
>
> Accordingly, I would like to get some testing and validation started.
>
> Please see the following document:
>
> https://hackmd.io/op6N_nIaR7aMzw9Ib-sDAQ
>
> Please feel free to ask questions here and add/check off services that
> appear to be running as expected. Application owners (people in
> sysadmin-whatever groups) should check that they have the expected level
> of access as before, that their playbooks run (idempotently!) and the
> logs and any interfaces show the application appears to be running fine.
>
> We have this week and next to get everything in good shape for migration
> week (20202-06-08) when we will moving everything out of phx2.

I had a look at mm-backend01.iad2 and it seems work. Unfortunately it
does seem to work better than it should, because it is syncing out data
to the proxies. So now we have production data synced to the proxies
from phx2 but also data synced out from iad2, which is probably not as
up to date as phx2. I stopped cron on mm-backend01.iad2 for now to avoid
this.

Adrian

[Kevin Fenzi]

On Wed, May 27, 2020 at 11:54:36AM +0200, Adrian Reber wrote:
> On Tue, May 26, 2020 at 03:22:02PM -0700, Kevin Fenzi wrote:
> > Greetings everyone.
> >
> > We now have most things deployed in our new datacenter (IAD2).
> >
> > Accordingly, I would like to get some testing and validation started.
> >
> > Please see the following document:
> >
> > https://hackmd.io/op6N_nIaR7aMzw9Ib-sDAQ
> >
> > Please feel free to ask questions here and add/check off services that
> > appear to be running as expected. Application owners (people in
> > sysadmin-whatever groups) should check that they have the expected level
> > of access as before, that their playbooks run (idempotently!) and the
> > logs and any interfaces show the application appears to be running fine.
> >
> > We have this week and next to get everything in good shape for migration
> > week (20202-06-08) when we will moving everything out of phx2.
>
> I had a look at mm-backend01.iad2 and it seems work. Unfortunately it
> does seem to work better than it should, because it is syncing out data
> to the proxies. So now we have production data synced to the proxies
> from phx2 but also data synced out from iad2, which is probably not as
> up to date as phx2. I stopped cron on mm-backend01.iad2 for now to avoid
> this.

Oops. Sorry about that. I tried to catch these cases, but since I was
mass installing everything I am sure I missed some. ;(

Thanks for catching it.

kevin