Outage: Upgrading the Fedora Copr backend storage, 2022-10-28 -- 2022-10-30

[Pavel Raiskup]

We need to replace the volume hosting our RPM/YUM repo data with a larger one.

As a consequence, we'll have to stop the build queue processing for about 20
hours, and we'll have two "full" (but short) outages (2x5 minutes) when the
yum
repositories will be unavailable.

The anticipated start of the outage (queue stops processing) is Saturday 2022-10-29:

    $ date --date "2022-10-29 17:00 UTC"
    Sat Oct 29 07:00:00 PM CEST 2022

But a few hours before, we'll lower the build throughput a bit to prioritize the
I/O we have on the data migration task..

We'd like to kindly ask you to minimize the build throughput in the Fedora
Copr build service during the weekend in question — just to allow us to
make the transition as smooth as possible.  It would be ideal if you could
e.g. delay your mass rebuilding and other large data modifications tasks
until the storage movement is over.  That said, we don't plan to turn the
frontend machine off (API and web-UI) during the whole outage period.

For more info, see the more detailed schedule:
https://docs.google.com/spreadsheets/d/1URopuw2R533H6i4vSTH8Nt47EIhs3ubWc...

Affected services:
- Copr service (frontend):
  https://copr.fedorainfracloud.org
- Copr (backend) repositories:
  https://download.copr.fedorainfracloud.org/ (AWS CloudFront proxy)
  https://copr-be.cloud.fedoraproject.org/ (storage)

Sorry for the inconvenience,
Copr Team

[Miro Hrončok]

On 21. 10. 22 15:17, Pavel Raiskup wrote:
>  We need to replace the volume hosting our RPM/YUM repo data with a
> larger one.
>  
>  As a consequence, we'll have to stop the build queue processing for about 20
>  hours, and we'll have two "full" (but short) outages (2x5 minutes) when the
> yum
>  repositories will be unavailable.
>  
>  The anticipated start of the outage (queue stops processing) is Saturday 2022-10-29:
>  
>       $ date --date "2022-10-29 17:00 UTC"
>       Sat Oct 29 07:00:00 PM CEST 2022
>  
>  But a few hours before, we'll lower the build throughput a bit to prioritize the
>  I/O we have on the data migration task..
>  
>  We'd like to kindly ask you to minimize the build throughput in the Fedora
>  Copr build service during the weekend in question — just to allow us to
>  make the transition as smooth as possible.  It would be ideal if you could
>  e.g. delay your mass rebuilding and other large data modifications tasks
>  until the storage movement is over.  That said, we don't plan to turn the
>  frontend machine off (API and web-UI) during the whole outage period.
>  
>  For more info, see the more detailed schedule:
>
> https://docs.google.com/spreadsheets/d/1URopuw2R533H6i4vSTH8Nt47EIhs3ubWc...
>  
>  Affected services:
>  - Copr service (frontend):
>     https://copr.fedorainfracloud.org
>  - Copr (backend) repositories:
>     https://download.copr.fedorainfracloud.org/ (AWS CloudFront proxy)
>     https://copr-be.cloud.fedoraproject.org/ (storage)
>  
>  Sorry for the inconvenience,
>  Copr Team 
FWIW it seems the alert-danger banner at top of 
https://copr.fedorainfracloud.org added bootstrap css and changed (broke?) the 
design of the web interface slightly.

-- 
Miro Hrončok
-- 
Phone: [phone]
IRC: mhroncok

[Miroslav Suchý]

Dne 21. 10. 22 v 18:30 Miro Hrončok napsal(a):
>  FWIW it seems the alert-danger banner at top of
> https://copr.fedorainfracloud.org added bootstrap css and changed 
>  (broke?) the design of the web interface slightly. 
That was mistake. Link to css removed.

Miroslav

[Pavel Raiskup]

The outage just began.  Copr is still running, but with a limitted quota.
Please delay long builds and bigger rebuilds till Sunday evening.

Have a nice weekend,
Pavel

On Friday, October 21, 2022 3:17:20 PM CEST Pavel Raiskup wrote:
>  We need to replace the volume hosting our RPM/YUM repo data with a
> larger one.
>  
>  As a consequence, we'll have to stop the build queue processing for about 20
>  hours, and we'll have two "full" (but short) outages (2x5 minutes) when the
> yum
>  repositories will be unavailable.
>  
>  The anticipated start of the outage (queue stops processing) is Saturday 2022-10-29:
>  
>      $ date --date "2022-10-29 17:00 UTC"
>      Sat Oct 29 07:00:00 PM CEST 2022
>  
>  But a few hours before, we'll lower the build throughput a bit to prioritize the
>  I/O we have on the data migration task..
>  
>  We'd like to kindly ask you to minimize the build throughput in the Fedora
>  Copr build service during the weekend in question — just to allow us to
>  make the transition as smooth as possible.  It would be ideal if you could
>  e.g. delay your mass rebuilding and other large data modifications tasks
>  until the storage movement is over.  That said, we don't plan to turn the
>  frontend machine off (API and web-UI) during the whole outage period.
>  
>  For more info, see the more detailed schedule:
>
> https://docs.google.com/spreadsheets/d/1URopuw2R533H6i4vSTH8Nt47EIhs3ubWc...
>  
>  Affected services:
>  - Copr service (frontend):
>    https://copr.fedorainfracloud.org
>  - Copr (backend) repositories:
>    https://download.copr.fedorainfracloud.org/ (AWS CloudFront proxy)
>    https://copr-be.cloud.fedoraproject.org/ (storage)
>  
>  Sorry for the inconvenience,
>  Copr Team
>

[Pavel Raiskup]

And the outage is over, everything should be back online and working.

The outage eventually took little less than 7 hours (stopped queue), and
were about 5-10 minutes fully down (no yum repos available).  We'll send more
detailed email the next week (or dump a blog post or something).

Happy building!
Copr Team

On Friday, October 28, 2022 9:28:05 PM CET Pavel Raiskup wrote:
>  The outage just began.  Copr is still running, but with a limitted
> quota.
>  Please delay long builds and bigger rebuilds till Sunday evening.
>  
>  Have a nice weekend,
>  Pavel
>  
>  On Friday, October 21, 2022 3:17:20 PM CEST Pavel Raiskup wrote:
>  > We need to replace the volume hosting our RPM/YUM repo data with a larger one.
>  > 
>  > As a consequence, we'll have to stop the build queue processing for about 20
>  > hours, and we'll have two "full" (but short) outages (2x5 minutes)
> when the yum
>  > repositories will be unavailable.
>  > 
>  > The anticipated start of the outage (queue stops processing) is Saturday
> 2022-10-29:
>  > 
>  >     $ date --date "2022-10-29 17:00 UTC"
>  >     Sat Oct 29 07:00:00 PM CEST 2022
>  > 
>  > But a few hours before, we'll lower the build throughput a bit to prioritize
> the
>  > I/O we have on the data migration task..
>  > 
>  > We'd like to kindly ask you to minimize the build throughput in the Fedora
>  > Copr build service during the weekend in question — just to allow us to
>  > make the transition as smooth as possible.  It would be ideal if you could
>  > e.g. delay your mass rebuilding and other large data modifications tasks
>  > until the storage movement is over.  That said, we don't plan to turn the
>  > frontend machine off (API and web-UI) during the whole outage period.
>  > 
>  > For more info, see the more detailed schedule:
>  >
> https://docs.google.com/spreadsheets/d/1URopuw2R533H6i4vSTH8Nt47EIhs3ubWc...
>  > 
>  > Affected services:
>  > - Copr service (frontend):
>  >   https://copr.fedorainfracloud.org
>  > - Copr (backend) repositories:
>  >   https://download.copr.fedorainfracloud.org/ (AWS CloudFront proxy)
>  >   https://copr-be.cloud.fedoraproject.org/ (storage)
>  > 
>  > Sorry for the inconvenience,
>  > Copr Team
>  > 
>  
>

[Tomas Tomecek]

Congratulations, Pavel, on upgrading the storage, very well done!

I can confirm things work smoothly on the Packit side this Monday morning.


On Sun, Oct 30, 2022 at 2:29 AM Pavel Raiskup <praiskup(a)redhat.com&gt; wrote:

>  And the outage is over, everything should be back online and
> working.
>
>  The outage eventually took little less than 7 hours (stopped queue), and
>  were about 5-10 minutes fully down (no yum repos available).  We'll send
>  more
>  detailed email the next week (or dump a blog post or something).
>
>  Happy building!
>  Copr Team
>
>  On Friday, October 28, 2022 9:28:05 PM CET Pavel Raiskup wrote:
>  > The outage just began.  Copr is still running, but with a limitted quota.
>  > Please delay long builds and bigger rebuilds till Sunday evening.
>  >
>  > Have a nice weekend,
>  > Pavel
>  >
>  > On Friday, October 21, 2022 3:17:20 PM CEST Pavel Raiskup wrote:
>  > > We need to replace the volume hosting our RPM/YUM repo data with a
>  larger one.
>  > >
>  > > As a consequence, we'll have to stop the build queue processing for
>  about 20
>  > > hours, and we'll have two "full" (but short) outages (2x5
> minutes)
>  when the yum
>  > > repositories will be unavailable.
>  > >
>  > > The anticipated start of the outage (queue stops processing) is
>  Saturday 2022-10-29:
>  > >
>  > >     $ date --date "2022-10-29 17:00 UTC"
>  > >     Sat Oct 29 07:00:00 PM CEST 2022
>  > >
>  > > But a few hours before, we'll lower the build throughput a bit to
>  prioritize the
>  > > I/O we have on the data migration task..
>  > >
>  > > We'd like to kindly ask you to minimize the build throughput in the
>  Fedora
>  > > Copr build service during the weekend in question — just to allow us to
>  > > make the transition as smooth as possible.  It would be ideal if you
>  could
>  > > e.g. delay your mass rebuilding and other large data modifications
>  tasks
>  > > until the storage movement is over.  That said, we don't plan to turn
>  the
>  > > frontend machine off (API and web-UI) during the whole outage period.
>  > >
>  > > For more info, see the more detailed schedule:
>  > >
>
> https://docs.google.com/spreadsheets/d/1URopuw2R533H6i4vSTH8Nt47EIhs3ubWc...
>  > >
>  > > Affected services:
>  > > - Copr service (frontend):
>  > >   https://copr.fedorainfracloud.org
>  > > - Copr (backend) repositories:
>  > >   https://download.copr.fedorainfracloud.org/ (AWS CloudFront proxy)
>  > >   https://copr-be.cloud.fedoraproject.org/ (storage)
>  > >
>  > > Sorry for the inconvenience,
>  > > Copr Team
>  > >
>  >
>  >
>
>
>
>  _______________________________________________
>  copr-devel mailing list -- copr-devel(a)lists.fedorahosted.org
>  To unsubscribe send an email to copr-devel-leave(a)lists.fedorahosted.org
>  Fedora Code of Conduct:
>  https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
>  https://lists.fedorahosted.org/archives/list/[e-mail]...
>  Do not reply to spam, report it:
>  https://pagure.io/fedora-infrastructure/new_issue
>