copr-devel February 2020

copr-devel@lists.fedorahosted.org

6 participants
8 discussions

Outage: Migration of Copr servers - 2020-02-23 06:00 UTC
by Jakub Kadlcik 13 Apr '20

13 Apr '20

There will be an outage starting at 2020-02-23 06:00 UTC, which will last approximately 7 hours (but can take more if something goes wrong). To convert UTC to your local time, take a look at https://fedoraproject.org/wiki/UTCHowto or run: date -d '2020-02-23 06:00 UTC' Reason for outage: Moving Copr servers from OpenStack to Amazon AWS because OpenStack is going to be shut down soon. Affected Services: copr-frontend - https://copr.fedorainfracloud.org copr-backend - https://copr-be.cloud.fedoraproject.org/ copr-distgit copr-keygen Backend repositories should remain available during the outage. Ticket Link: https://pagure.io/fedora-infrastructure/issue/8668 Contact Information: Please join #fedora-admin in irc.freenode.net or add comments to the ticket for this outage above. Jakub

5 11

Chromium build timeout
by 藍挺瑋 14 Mar '20

14 Mar '20

Chromium project has decided to remove a feature called 'jumbo build' from its build system: https://groups.google.com/a/chromium.org/forum/#!topic/chromium-dev/DP9TQsz…. While this feature still exists in the build system of Chromium 79, it no longer builds and bugs related to jumbo build are considered WONTFIX: https://bugs.chromium.org/p/chromium/issues/detail?id=1017500. Therefore, I must disable jumbo build to get a working build of Chromium 79. Disabling the features significantly increases the number of build targets from about 25000 to 41000, causing the build time to reach the limit of Copr, which is currently 21 hours. https://copr.fedorainfracloud.org/coprs/lantw44/chromium/build/1129111/ https://copr.fedorainfracloud.org/coprs/lantw44/chromium/build/1129703/ https://copr.fedorainfracloud.org/coprs/lantw44/chromium/build/1131005/ Is it possible to raise the limit again to avoid the timeout? Otherwise, it is likely that building Chromium will become harder over time because Chromium build time keeps increasing.

3 4

Copr outage - details
by Miroslav Suchý 20 Feb '20

20 Feb '20

tl;dr: On Sunday 23rd February, there will be Copr outage. It will last the whole day. PPC64LE builder and chroots will be deactivated. The PPC64LE builders should be back in a matter of weeks. Hi. As previously announced, Fedora's infrastructure is moving to a different datacenter. For some servers, the move is trivial. Copr servers are different. Copr build system consists of four servers, plus four staging servers. Eight TB of repos, four TB of dist-git, and several small volumes. The original plan was to move to Washington D.C. to IAD2 datacenter by June. Copr is running in Fedora OpenStack, and this cloud has to be evacuated by the beginning of March to free an IP range. The plan was to move Copr to new hardware (thanks to Red Hat) and later move this HW to the new datacenter. That would mean two outages, where the second one lasted at least 15 days (!). We were looking for another option and we found it. We are going to move Copr to Amazon AWS, shutdown old VM on Fedora Cloud. Move the new HW to IAD2 datacenter and then move Copr from AWS to new HW in IAD2 - FYI, the final destination is still subject to change. This still means two outages, but they should be just a few hours. And web server with DNF repositories should be available all the time. The second outage, will happen in May or June. Here is a detailed schedule. We are going to update this table during migration. You can watch the progress during migration: https://docs.google.com/spreadsheets/d/1jrCgdhseZwi91CTRlo9Y5DNwfl9VHoZfjHP… Here is a short abstract: * we are doing constant rsync to the new location * we spin up staging and production instances in the new location * on Sunday morning we stop frontend and therefore accepting new jobs. The backend with DNF repos will still be operational. * we do final rsync (~6 hours) * around 13:00 UTC we switch DNS to the new location * we then enable all services * once we confirm that everything is operational, the outage will be over There are several caveats: * After we enable services on Sunday 13:00 UTC you may see some failures. Be asured that we will swiftly address them. * Once we get out of Fedora Cloud, we lost access to PPC64LE builders. We are going to deactivate those chroots just before the migration. After a few weeks, we should get it back. ETA is unknown. The worst-case scenario is in June 2020. We will be aiming to bring it back as soon as possible. * Any small issue can easily change the schedule by hours. E.g., just simple 'chown -R' on backend runs ~4 hours. There are going to be three Copr engineers and one fedora-infrastructure member available whole Sunday. If you experienced a problem, do not hesitate to contact us. We are on #fedora-buildsys on Freenode. The link to the outage ticket is: https://pagure.io/fedora-infrastructure/issue/8668 -- Miroslav Suchy, RHCARed Hat, Associate Manager ABRT/Copr, #brno, #fedora-buildsys

1 0

mock 2.0 installed on copr builders
by Pavel Raiskup 19 Feb '20

19 Feb '20

Hey all, just a headsup, today I pushed mock 2.0 to stable, and updated copr builders so they use it. Our integration tests are passing, so this shouldn't cause any issues - but please report any issues you may observe, thank you. Happy building! Pavel

1 0

Disabling Fedora 29 chroots in Copr
by Jakub Kadlcik 18 Feb '20

18 Feb '20

Hello, we have just disabled Fedora 29 chroots in Copr. According to the Fedora wiki [1], Fedora 29 reached the end of its life on 2019-11-30 and therefore we are disabling it in Copr. That effectively means that from this moment, it is no longer possible to submit builds for the following chroots: - fedora-29-x86_64 - fedora-29-i386 - fedora-29-ppc64le - fedora-29-aarch64 Additionally, according to Outdated chroots removal policy [2], Copr is going to preserve existing build results in those chroots for another 180 days and then automatically remove them unless you take an action and prolong the chroots life span in your projects. Read more about this feature in the Copr - Removing outdated chroots blog post [3]. [1] https://fedoraproject.org/wiki/End_of_life [2] https://docs.pagure.org/copr.copr/copr_outdated_chroots_removal_policy.html [3] http://frostyx.cz/posts/copr-removing-outdated-chroots Jakub

2 3

Copr outage - moving to new datacenter
by Miroslav Suchý 13 Feb '20

13 Feb '20

FYI, the Copr servers have to move to the new data center by the end of this month. That will include a physical move. In a truck. In the worst-case scenario, it will mean at least 15 days outage. Right now, the whole Copr team + some fedora-infrastructure guys are working on this task. We are trying several approaches on how to handle the migration. I would love to see as small outage as possible. But it seems that at least one-day outage is inevitable. To get you the picture, the migration includes copying 8TB of repositories and more than one TB of dist-git data - which is all in small files. Painfull to copy using a standard method. As soon as we have an exact date and a precise duration of the outage, I will let you know. -- Miroslav Suchy, RHCA Red Hat, Associate Manager ABRT/Copr, #brno, #fedora-buildsys

1 0

Proxies for Copr repositories
by Miroslav Suchý 10 Feb '20

10 Feb '20

Hi, I just enabled the content delivery network (CDN) for Copr repositories. It is provided by CloudFront from AWS. And it is provided for free by Amazon to Fedora. Technically the original URL copr-be.cloud.fedoraproject.org is now accessible using download.copr.fedorainfracloud.org The original URL is and will be available. You can still use it. The second one is CNAME for the CDN and should be much faster for you. The CDN is automatically enabled for new projects. Thou, if you already have enabled some copr repository on your workstation/server, it will still use the old URL. It is fine; it will continue to work. If you want to enable CDN for your repos you have to run: dnf copr remove some/project dnf copr enable some/project or manually change the URL in the repo file. If you experience any problem with CDN, please let me know. -- Miroslav Suchy, RHCA Red Hat, Associate Manager ABRT/Copr, #brno, #fedora-buildsys

2 2

Upgrade of Copr servers - 2020-02-06 8:30 UTC (Thursday)
by Pavel Raiskup 06 Feb '20

06 Feb '20

There will be an outage, starting at: $ date --date='2020-02-06 8:30 UTC' which will last approximately 2 hours. Reason for outage ================= Upgrade of copr packages on Copr servers, to get in bug-fixes and new features. Affected Services ================= copr-frontend - https://copr.fedorainfracloud.org copr-backend - https://copr-be.cloud.fedoraproject.org/ Though the yum/dnf repositories (the copr-backend part which is consumed by end-user by 'dnf copr enable ...') should stay available during this outage. Ticket Link =========== https://pagure.io/fedora-infrastructure/issue/8612 Contact Information =================== @praiskup Please join #fedora-admin in irc.freenode.net or add comments to the ticket for this outage above.

1 2

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

copr-devel February 2020