Upcoming fedoraproject Datacenter move reminder and plans

[Kevin Fenzi]

Greetings.

As previously announced, fedoraproject is moving many of it's servers
from one datacenter (phx2 near phoenix, arizona, usa) to another (iad2:
near arlington, virginia, usa).

As we move from the old datacenter to the new, we will have a temporary
reduction in capacity. The new datacenter has a smaller, less-redundant,
lower-capacity version of our infrastructure. Over the next two weeks,
we will migrate services to it so that we can finish moving out of the
old datacenter.

After everything is moved from the old datacenter, many of the servers
there will be shipped to the new datacenter and then re-added to bring
us back to full redundency and capacity. 

Out detailed checklist for these migrations is available at
https://hackmd.io/@fedorainfra2020/rJpsA4FLL

To summarize what we are moving when: 

2020-06-03 wed: The fedoraproject master mirrors will move to IAD2. A
very small outage may be noticed as dns changes. There may be some
mirroring slowdowns as we work out bugs.

2020-06-04 thu: Our internal ansible control host and the fedoraproject
wiki will move. The wiki will be down for a few hours.

2020-06-05 fri: Our meeting minutes archive
(https://meetbot.fedoraproject.org) and our freenode irc bot (zodbot).
These two services will see a hour outage or less.

2020-06-07 sun: We will pause for the next week adding new packages and
unretiring packages to avoid problems. 

2020-06-08 mon: Our fedora-messaging bus and gateways to it
(github2fedmsg, bugzilla2fedmsg), mirrormanager, product definition
center (pdc), and our identity and authentication systems. Messages over our
message bus may be slow or missing and users may be unable to login at
various times as we migrate services over. 

Additionally, we will be stopping services that will not be back until
later in the month. 
These include: 
 * Fedocal
 * Badges
 * Nuancier
 * koschei
 * simple-koji-ci
 * All staging services (*.stg.fedoraproject.org)

2020-06-09 tue: The build and packaging ecosystem. This includes koji,
src.fedoraproject.org, osbs, odcs, container registries, bodhi (updates
system). During this day maintainers should avoid builds/updates if at
all possible as they may or may not work at various times. 

2020-06-10 wed: Various small apps (mdapi, anitya, waiverdb, greenwave,
etc), mailman/lists.fedoraproject.org, and our datagrepper/datanommer
services. Mailing lists will be down for several hours as data is
migrated. Datagrepper will be down for most of the day as it's database
is moved. Other services will be down for short amounts of time while
they are moved.

2020-06-11 thu: Various small site building apps (docs building, fedora
websites building, reviewstats, blockerbugs) and elections will be
moved. elections will be up until the currently running elections
complete. (GO VOTE! https://elections.fedoraproject.org)

2020-06-12 fri: Catch up and fix issues day, along with re-enabling
package unretirements/new packages, and other 'paused' items.

The week after this servers will be shipped and the week after that we
expect to start setting them up and getting them re-added. During this
time, we may have to make further changes to what services are available
in order to deal with load changes. 

If you have any questions or concerns, please file an infrastructure
ticket ( https://pagure.io/fedora-infrastructure) or come talk to us in
#fedora-admin on irc.freenode.net. 

Finally, I'd like to ask everyone to be patient as we do this move. I
know that it's painful when you are unable to contibute something when
you have time to do so, but rest assured that we are trying to migrate
things as quickly and smoothly as we can.

Thanks.

kevin

_______________________________________________
devel-announce mailing list -- devel-announce(a)lists.fedoraproject.org
To unsubscribe send an email to devel-announce-leave(a)lists.fedoraproject.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/[e-mail]...

[Miroslav Suchý]

Dne 02. 06. 20 v 18:40 Kevin Fenzi napsal(a):
>  2020-06-09 tue: The build and packaging ecosystem. This includes
> koji,
>  src.fedoraproject.org, osbs, odcs, container registries, bodhi (updates
>  system). During this day maintainers should avoid builds/updates if at
>  all possible as they may or may not work at various times.  
To clarify - this does NOT include Copr build system.

Copr has already been moved weeks ago, and will do one additional migration after all the
primary services will be migrated.

-- 
Miroslav Suchy, RHCA
Red Hat, Associate Manager ABRT/Copr, #brno, #fedora-buildsys


_______________________________________________
devel-announce mailing list -- devel-announce(a)lists.fedoraproject.org
To unsubscribe send an email to devel-announce-leave(a)lists.fedoraproject.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/[e-mail]...

[Miroslav Suchý]

Dne 02. 06. 20 v 18:40 Kevin Fenzi napsal(a):
>  2020-06-09 tue: The build and packaging ecosystem. This includes
> koji,
>  src.fedoraproject.org, osbs, odcs, container registries, bodhi (updates
>  system). During this day maintainers should avoid builds/updates if at
>  all possible as they may or may not work at various times.  
To clarify - this does NOT include Copr build system.

Copr has already been moved weeks ago, and will do one additional migration after all the
primary services will be migrated.

-- 
Miroslav Suchy, RHCA
Red Hat, Associate Manager ABRT/Copr, #brno, #fedora-buildsys

[Vít Ondruch]

Just wonder, will there be any convenient way to remind me that the
service is down for a reason? E.g. redirect to status page. It would be
probably harder for all the CLI utilities we are using, but maybe
something which should be addressed as well.


Vít


Dne 02. 06. 20 v 18:40 Kevin Fenzi napsal(a):
>  Greetings.
>
>  As previously announced, fedoraproject is moving many of it's servers
>  from one datacenter (phx2 near phoenix, arizona, usa) to another (iad2:
>  near arlington, virginia, usa).
>
>  As we move from the old datacenter to the new, we will have a temporary
>  reduction in capacity. The new datacenter has a smaller, less-redundant,
>  lower-capacity version of our infrastructure. Over the next two weeks,
>  we will migrate services to it so that we can finish moving out of the
>  old datacenter.
>
>  After everything is moved from the old datacenter, many of the servers
>  there will be shipped to the new datacenter and then re-added to bring
>  us back to full redundency and capacity. 
>
>  Out detailed checklist for these migrations is available at
>  https://hackmd.io/@fedorainfra2020/rJpsA4FLL
>
>  To summarize what we are moving when: 
>
>  2020-06-03 wed: The fedoraproject master mirrors will move to IAD2. A
>  very small outage may be noticed as dns changes. There may be some
>  mirroring slowdowns as we work out bugs.
>
>  2020-06-04 thu: Our internal ansible control host and the fedoraproject
>  wiki will move. The wiki will be down for a few hours.
>
>  2020-06-05 fri: Our meeting minutes archive
>  (https://meetbot.fedoraproject.org) and our freenode irc bot (zodbot).
>  These two services will see a hour outage or less.
>
>  2020-06-07 sun: We will pause for the next week adding new packages and
>  unretiring packages to avoid problems. 
>
>  2020-06-08 mon: Our fedora-messaging bus and gateways to it
>  (github2fedmsg, bugzilla2fedmsg), mirrormanager, product definition
>  center (pdc), and our identity and authentication systems. Messages over our
>  message bus may be slow or missing and users may be unable to login at
>  various times as we migrate services over. 
>
>  Additionally, we will be stopping services that will not be back until
>  later in the month. 
>  These include: 
>   * Fedocal
>   * Badges
>   * Nuancier
>   * koschei
>   * simple-koji-ci
>   * All staging services (*.stg.fedoraproject.org)
>
>  2020-06-09 tue: The build and packaging ecosystem. This includes koji,
>  src.fedoraproject.org, osbs, odcs, container registries, bodhi (updates
>  system). During this day maintainers should avoid builds/updates if at
>  all possible as they may or may not work at various times. 
>
>  2020-06-10 wed: Various small apps (mdapi, anitya, waiverdb, greenwave,
>  etc), mailman/lists.fedoraproject.org, and our datagrepper/datanommer
>  services. Mailing lists will be down for several hours as data is
>  migrated. Datagrepper will be down for most of the day as it's database
>  is moved. Other services will be down for short amounts of time while
>  they are moved.
>
>  2020-06-11 thu: Various small site building apps (docs building, fedora
>  websites building, reviewstats, blockerbugs) and elections will be
>  moved. elections will be up until the currently running elections
>  complete. (GO VOTE! https://elections.fedoraproject.org)
>
>  2020-06-12 fri: Catch up and fix issues day, along with re-enabling
>  package unretirements/new packages, and other 'paused' items.
>
>  The week after this servers will be shipped and the week after that we
>  expect to start setting them up and getting them re-added. During this
>  time, we may have to make further changes to what services are available
>  in order to deal with load changes. 
>
>  If you have any questions or concerns, please file an infrastructure
>  ticket ( https://pagure.io/fedora-infrastructure) or come talk to us in
>  #fedora-admin on irc.freenode.net. 
>
>  Finally, I'd like to ask everyone to be patient as we do this move. I
>  know that it's painful when you are unable to contibute something when
>  you have time to do so, but rest assured that we are trying to migrate
>  things as quickly and smoothly as we can.
>
>  Thanks.
>
>  kevin
>
>  _______________________________________________
>  devel-announce mailing list -- devel-announce(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-announce-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]...
>
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]

[Kevin Fenzi]

On Wed, Jun 03, 2020 at 10:46:06AM +0200, Vít Ondruch wrote:
>  Just wonder, will there be any convenient way to remind me that the
>  service is down for a reason? E.g. redirect to status page. It would be
>  probably harder for all the CLI utilities we are using, but maybe
>  something which should be addressed as well. 
Many of our services have outage pages that will tell you to look for a
outage and check status. There's nothing that covers all applications,
but any that use our proxy system would be. 

kevin

[Miro Hrončok]

On 02. 06. 20 18:40, Kevin Fenzi wrote:
>  Greetings.
>  
>  As previously announced, fedoraproject is moving many of it's servers
>  from one datacenter (phx2 near phoenix, arizona, usa) to another (iad2:
>  near arlington, virginia, usa).
>  
>  As we move from the old datacenter to the new, we will have a temporary
>  reduction in capacity. The new datacenter has a smaller, less-redundant,
>  lower-capacity version of our infrastructure. Over the next two weeks,
>  we will migrate services to it so that we can finish moving out of the
>  old datacenter.
>  
>  After everything is moved from the old datacenter, many of the servers
>  there will be shipped to the new datacenter and then re-added to bring
>  us back to full redundency and capacity.
>  
>  Out detailed checklist for these migrations is available at
>  https://hackmd.io/@fedorainfra2020/rJpsA4FLL
>  
>  ... snip ... 
Is there a publicly available more or less up to date mirror of 
src.fedoraproject.org git repos?

I know there is https://fedoraproject.org/wiki/Infrastructure/Grokmirror

But I'd rather not set it up myself, if there is some instance I can use.

We'd like to build packages in Copr during the outage, because Copr is not 
affected, but we build packages from dist git sources, so when src.fp.o goes 
down, we need a backup plan.

To have this, we need git repos + lookaside cache, but we can more or less 
workaround the lookaside cache problem (at least for packages with Sources with 
proper URLs).

-- 
Miro Hrončok
--
Phone: [phone]
IRC: mhroncok

[Kevin Fenzi]

On Fri, Jun 05, 2020 at 01:39:51PM +0200, Miro Hrončok wrote:
>  
>  Is there a publicly available more or less up to date mirror of
>  src.fedoraproject.org git repos?
>  
>  I know there is https://fedoraproject.org/wiki/Infrastructure/Grokmirror
>  
>  But I'd rather not set it up myself, if there is some instance I can use. 
Nope. Not that I know of. If there's interest we could perhaps look at
standing one up? It would give us another copy somewhere else which is
nice.

kevin

[Miro Hrončok]

On 02. 06. 20 18:40, Kevin Fenzi wrote:
>  Out detailed checklist for these migrations is available at
>  https://hackmd.io/@fedorainfra2020/rJpsA4FLL 
The https://status.fedoraproject.org/ page links to a different but similar 
hackmd.io document. Should it be replaced by this one?

-- 
Miro Hrončok
--
Phone: [phone]
IRC: mhroncok