Copr outage - details
by Miroslav Suchý
tl;dr: On Sunday 23rd February, there will be Copr outage. It will last the whole day.
PPC64LE builder and chroots will be deactivated. The PPC64LE builders should be back in a matter of weeks.
Hi.
As previously announced, Fedora's infrastructure is moving to a different datacenter. For some servers, the move is
trivial. Copr servers are different. Copr build system consists of four servers, plus four staging servers. Eight TB of
repos, four TB of dist-git, and several small volumes.
The original plan was to move to Washington D.C. to IAD2 datacenter by June. Copr is running in Fedora OpenStack, and
this cloud has to be evacuated by the beginning of March to free an IP range.
The plan was to move Copr to new hardware (thanks to Red Hat) and later move this HW to the new datacenter. That would
mean two outages, where the second one lasted at least 15 days (!).
We were looking for another option and we found it. We are going to move Copr to Amazon AWS, shutdown old VM on Fedora
Cloud. Move the new HW to IAD2 datacenter and then move Copr from AWS to new HW in IAD2 - FYI, the final destination is
still subject to change. This still means two outages, but they should be just a few hours. And web server with DNF
repositories should be available all the time.
The second outage, will happen in May or June.
Here is a detailed schedule. We are going to update this table during migration. You can watch the progress during
migration:
https://docs.google.com/spreadsheets/d/1jrCgdhseZwi91CTRlo9Y5DNwfl9VHoZfj...
Here is a short abstract:
* we are doing constant rsync to the new location
* we spin up staging and production instances in the new location
* on Sunday morning we stop frontend and therefore accepting new jobs. The backend with DNF repos will still be
operational.
* we do final rsync (~6 hours)
* around 13:00 UTC we switch DNS to the new location
* we then enable all services
* once we confirm that everything is operational, the outage will be over
There are several caveats:
* After we enable services on Sunday 13:00 UTC you may see some failures. Be asured that we will swiftly address them.
* Once we get out of Fedora Cloud, we lost access to PPC64LE builders. We are going to deactivate those chroots just
before the migration. After a few weeks, we should get it back. ETA is unknown. The worst-case scenario is in June 2020.
We will be aiming to bring it back as soon as possible.
* Any small issue can easily change the schedule by hours. E.g., just simple 'chown -R' on backend runs ~4 hours.
There are going to be three Copr engineers and one fedora-infrastructure member available whole Sunday. If you
experienced a problem, do not hesitate to contact us. We are on #fedora-buildsys on Freenode.
The link to the outage ticket is:
https://pagure.io/fedora-infrastructure/issue/8668
--
Miroslav Suchy, RHCARed Hat, Associate Manager ABRT/Copr, #brno, #fedora-buildsys
3 years, 7 months
mock 2.0 installed on copr builders
by Pavel Raiskup
Hey all,
just a headsup, today I pushed mock 2.0 to stable, and updated copr
builders so they use it. Our integration tests are passing, so this
shouldn't cause any issues - but please report any issues you may observe,
thank you.
Happy building!
Pavel
3 years, 7 months
Copr outage - moving to new datacenter
by Miroslav Suchý
FYI, the Copr servers have to move to the new data center by the end of this month. That will include a physical move.
In a truck.
In the worst-case scenario, it will mean at least 15 days outage.
Right now, the whole Copr team + some fedora-infrastructure guys are working on this task. We are trying several
approaches on how to handle the migration. I would love to see as small outage as possible. But it seems that at least
one-day outage is inevitable.
To get you the picture, the migration includes copying 8TB of repositories and more than one TB of dist-git data - which
is all in small files. Painfull to copy using a standard method.
As soon as we have an exact date and a precise duration of the outage, I will let you know.
--
Miroslav Suchy, RHCA
Red Hat, Associate Manager ABRT/Copr, #brno, #fedora-buildsys
3 years, 7 months
Proxies for Copr repositories
by Miroslav Suchý
Hi,
I just enabled the content delivery network (CDN) for Copr repositories.
It is provided by CloudFront from AWS. And it is provided for free by Amazon to Fedora.
Technically the original URL
copr-be.cloud.fedoraproject.org
is now accessible using
download.copr.fedorainfracloud.org
The original URL is and will be available. You can still use it. The second one is CNAME for the CDN and should be much
faster for you.
The CDN is automatically enabled for new projects. Thou, if you already have enabled some copr repository on your
workstation/server, it will still use the old URL.
It is fine; it will continue to work.
If you want to enable CDN for your repos you have to run:
dnf copr remove some/project
dnf copr enable some/project
or manually change the URL in the repo file.
If you experience any problem with CDN, please let me know.
--
Miroslav Suchy, RHCA
Red Hat, Associate Manager ABRT/Copr, #brno, #fedora-buildsys
3 years, 7 months
