Outage: Migration of Copr servers - 2020-02-23 06:00 UTC

[Jakub Kadlcik]

There will be an outage starting at 2020-02-23 06:00 UTC, which will last
approximately 7 hours (but can take more if something goes wrong).

To convert UTC to your local time, take a look at
https://fedoraproject.org/wiki/UTCHowto
or run:

date -d '2020-02-23 06:00 UTC'

Reason for outage:

Moving Copr servers from OpenStack to Amazon AWS because OpenStack is
going to be shut down soon.

Affected Services:

copr-frontend - https://copr.fedorainfracloud.org
copr-backend - https://copr-be.cloud.fedoraproject.org/
copr-distgit
copr-keygen

Backend repositories should remain available during the outage.

Ticket Link:

https://pagure.io/fedora-infrastructure/issue/8668

Contact Information:

Please join #fedora-admin in irc.freenode.net or add comments to the
ticket for this outage above.

Jakub

[Pavel Raiskup]

Also note that since the next Monday we will have to shut-down all our
ppc64le builders in Fedora Copr.  Fedra infrastructure is going to move
them to (some) new datacenter - assemple, and make them available for copr
builds again.  But we don't yet have ETA.

We will disable the ppc64le chroots ourselves, so you shouldn't observe
any expected CI build failures and so on (the builds won't even start),
but if you need to have something done in ppc64le please do it now.  The
build results will stay available during the movement.

Pavel

On Wednesday, February 19, 2020 2:13:53 PM CET Jakub Kadlcik wrote:
> There will be an outage starting at 2020-02-23 06:00 UTC, which will last
> approximately 7 hours (but can take more if something goes wrong).
>
> To convert UTC to your local time, take a look at
> https://fedoraproject.org/wiki/UTCHowto
> or run:
>
> date -d '2020-02-23 06:00 UTC'
>
> Reason for outage:
>
> Moving Copr servers from OpenStack to Amazon AWS because OpenStack is
> going to be shut down soon.
>
> Affected Services:
>
> copr-frontend - https://copr.fedorainfracloud.org
> copr-backend - https://copr-be.cloud.fedoraproject.org/
> copr-distgit
> copr-keygen
>
> Backend repositories should remain available during the outage.
>
> Ticket Link:
>
> https://pagure.io/fedora-infrastructure/issue/8668
>
> Contact Information:
>
> Please join #fedora-admin in irc.freenode.net or add comments to the
> ticket for this outage above.
>
> Jakub
> _______________________________________________
> copr-devel mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedorahosted.org/archives/list/[e-mail].o...

[Pavel Raiskup]

Another thing good to warn about WRT the movement:
https://pagure.io/copr/copr/issue/1278

We can not enable Fedora 32 chroots in Fedora Copr at this moment.  It
would mean several new hundreds of gigabytes of data for us to rsync to
AWS, and we don't have time for it.

This though means that any build done in Fedora Rawhide is done against
F33 chroot, and will be later (probably Monday) copied to F32 chroot (for
those projects that follow fedora branching).  So the builds are going to
be copied even though they can be incompatible.

It might be a good idea to not build against fedora-rawhide* chroots for now,
till the next week.

Thanks for understanding,
Pavel

On Wednesday, February 19, 2020 2:13:53 PM CET Jakub Kadlcik wrote:
> There will be an outage starting at 2020-02-23 06:00 UTC, which will last
> approximately 7 hours (but can take more if something goes wrong).
>
> To convert UTC to your local time, take a look at
> https://fedoraproject.org/wiki/UTCHowto
> or run:
>
> date -d '2020-02-23 06:00 UTC'
>
> Reason for outage:
>
> Moving Copr servers from OpenStack to Amazon AWS because OpenStack is
> going to be shut down soon.
>
> Affected Services:
>
> copr-frontend - https://copr.fedorainfracloud.org
> copr-backend - https://copr-be.cloud.fedoraproject.org/
> copr-distgit
> copr-keygen
>
> Backend repositories should remain available during the outage.
>
> Ticket Link:
>
> https://pagure.io/fedora-infrastructure/issue/8668
>
> Contact Information:
>
> Please join #fedora-admin in irc.freenode.net or add comments to the
> ticket for this outage above.
>
> Jakub
> _______________________________________________
> copr-devel mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedorahosted.org/archives/list/[e-mail].o...

[Jakub Kadlcik]

I would like to happily announce, that the migration was successful
and the outage is finally over. Everything should work as expected.

The outage window in which Copr services were unavailable, took
~7 hours and most of it was spent by waiting until the final data
sync finishes, so the migration couldn't happen any faster than that.

On the positive note, we are finally going to add F32 chroots
since the migration was the only blocker for that.

Also, please know that we needed to temporarily disable ppc64le
chroots (as it was previously announced) because at this moment
we don't have access to any builders on that architecture.

Thank you for understanding,
Jakub

On Thu, Feb 20, 2020 at 10:43 AM Pavel Raiskup  wrote:
> Another thing good to warn about WRT the movement:
> https://pagure.io/copr/copr/issue/1278
>
> We can not enable Fedora 32 chroots in Fedora Copr at this moment.  It
> would mean several new hundreds of gigabytes of data for us to rsync to
> AWS, and we don't have time for it.
>
> This though means that any build done in Fedora Rawhide is done against
> F33 chroot, and will be later (probably Monday) copied to F32 chroot (for
> those projects that follow fedora branching).  So the builds are going to
> be copied even though they can be incompatible.
>
> It might be a good idea to not build against fedora-rawhide* chroots for now,
> till the next week.
>
> Thanks for understanding,
> Pavel
>
> On Wednesday, February 19, 2020 2:13:53 PM CET Jakub Kadlcik wrote:
> > There will be an outage starting at 2020-02-23 06:00 UTC, which will last
> > approximately 7 hours (but can take more if something goes wrong).
> >
> > To convert UTC to your local time, take a look at
> > https://fedoraproject.org/wiki/UTCHowto
> > or run:
> >
> > date -d '2020-02-23 06:00 UTC'
> >
> > Reason for outage:
> >
> > Moving Copr servers from OpenStack to Amazon AWS because OpenStack is
> > going to be shut down soon.
> >
> > Affected Services:
> >
> > copr-frontend - https://copr.fedorainfracloud.org
> > copr-backend - https://copr-be.cloud.fedoraproject.org/
> > copr-distgit
> > copr-keygen
> >
> > Backend repositories should remain available during the outage.
> >
> > Ticket Link:
> >
> > https://pagure.io/fedora-infrastructure/issue/8668
> >
> > Contact Information:
> >
> > Please join #fedora-admin in irc.freenode.net or add comments to the
> > ticket for this outage above.
> >
> > Jakub
> > _______________________________________________
> > copr-devel mailing list -- [e-mail]
> > To unsubscribe send an email to [e-mail]
> > Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives: https://lists.fedorahosted.org/archives/list/[e-mail].o...

[Nicolas Mailhot]

Le dimanche 23 février 2020 à 14:26 +0100, Jakub Kadlcik a écrit :

Thanks to all the people that worked on this!
> I would like to happily announce, that the migration was successful
> and the outage is finally over. Everything should work as expected.

unfortunately, things are not stable yet

Errors during downloading metadata for repository
'http_kojipkgs_fedoraproject_org_repos_rawhide_latest_basearch':
  - Curl error (23): Failed writing received data to disk/application
for 
https://kojipkgs.fedoraproject.org/repos/rawhide/latest/x86_64/repodata/4c43...
[Failed writing body (1009 != 1436)]
  - Curl error (23): Failed writing received data to disk/application
for 
https://kojipkgs.fedoraproject.org/repos/rawhide/latest/x86_64/repodata/97d0...
[Failed writing body (1785 != 3472)]
Error: Failed to download metadata for repo
'http_kojipkgs_fedoraproject_org_repos_rawhide_latest_basearch': Yum
repo downloading error: Downloading error(s):
repodata/97d08db8fa23ee7c59085ba56916b306757480bf7d221bc49f0d0d541a1129
e3-primary.xml.gz - Download failed: Curl error (23): Failed writing
received data to disk/application for 
https://kojipkgs.fedoraproject.org/repos/rawhide/latest/x86_64/repodata/97d0...
[Failed writing body (1785 != 3472)];
repodata/4c436281727a03894e2c8215ec4defa5f031a10dd55477a0c4bad037c9f6d8
48-filelists.xml.gz - Download failed: Curl error (23): Failed writing
received data to disk/application for 
https://kojipkgs.fedoraproject.org/repos/rawhide/latest/x86_64/repodata/4c43...
[Failed writing body (1009 != 1436)]

Regards,

[Pavel Raiskup]

On Sunday, February 23, 2020 5:06:07 PM CET Nicolas Mailhot wrote:
> > I would like to happily announce, that the migration was successful
> > and the outage is finally over. Everything should work as expected.
>
> unfortunately, things are not stable yet

The build boxes are deleted already, so I can not debug more ATM.  We'll need
another try.  Seems like some storage isn't recycled for heavily re-used
builders.

Pavel
> Errors during downloading metadata for repository
> 'http_kojipkgs_fedoraproject_org_repos_rawhide_latest_basearch':
> Curl error (23): Failed writing received data to disk/application
>
> for 
> https://kojipkgs.fedoraproject.org/repos/rawhide/latest/x86_64/repodata/4c43...
> [Failed writing body (1009 != 1436)]
> Curl error (23): Failed writing received data to disk/application
>
> for 
> https://kojipkgs.fedoraproject.org/repos/rawhide/latest/x86_64/repodata/97d0...
> [Failed writing body (1785 != 3472)]
> Error: Failed to download metadata for repo
> 'http_kojipkgs_fedoraproject_org_repos_rawhide_latest_basearch': Yum
> repo downloading error: Downloading error(s):
> repodata/97d08db8fa23ee7c59085ba56916b306757480bf7d221bc49f0d0d541a1129
> e3-primary.xml.gz - Download failed: Curl error (23): Failed writing
> received data to disk/application for 
> https://kojipkgs.fedoraproject.org/repos/rawhide/latest/x86_64/repodata/97d0...
> [Failed writing body (1785 != 3472)];
> repodata/4c436281727a03894e2c8215ec4defa5f031a10dd55477a0c4bad037c9f6d8
> 48-filelists.xml.gz - Download failed: Curl error (23): Failed writing
> received data to disk/application for 
> https://kojipkgs.fedoraproject.org/repos/rawhide/latest/x86_64/repodata/4c43...
> [Failed writing body (1009 != 1436)]
>
> Regards,

[Jun Aruga]

> Also, please know that we needed to temporarily disable ppc64le

chroots (as it was previously announced) because at this moment
we don't have access to any builders on that architecture.

Any update for Copr ppc64le chroots?

[Pavel Raiskup]

On Saturday, April 11, 2020 4:22:23 PM CEST Jun Aruga wrote:
> > Also, please know that we needed to temporarily disable ppc64le
>
> chroots (as it was previously announced) because at this moment
> we don't have access to any builders on that architecture.
>
> Any update for Copr ppc64le chroots?

Not yet, the HW is still not assembled in new lab.

Pavel

[Jun Aruga]

> > Any update for Copr ppc64le chroots?
>
> Not yet, the HW is still not assembled in new lab.
>
> Pavel

Do you know when it will be available?
If it takes some time, how about using qemu emulation for the ppc64le
chroot for now?

[Stephen John Smoogen]

On Sun, 12 Apr 2020 at 07:33, Jun Aruga  wrote:
> > > Any update for Copr ppc64le chroots?
> >
> > Not yet, the HW is still not assembled in new lab.
> >
> > Pavel
>
> Do you know when it will be available?
> If it takes some time, how about using qemu emulation for the ppc64le
> chroot for now?

Fedora Infrastructure are in the middle of 2 data center moves currently
with a COVID-19 lockdown affecting our ability to give any timelines of
when things will be available. I do not think the servers will be available
until May at the earliest and they will be a lower priority than keeping
koji or other core Fedora services going.

Currently copr is running out of amazon and does not have emulation
capabilities either.

[Pavel Raiskup]

On Sunday, April 12, 2020 5:52:44 PM CEST Stephen John Smoogen wrote:
> On Sun, 12 Apr 2020 at 07:33, Jun Aruga  wrote:
> > > > Any update for Copr ppc64le chroots?
> > >
> > > Not yet, the HW is still not assembled in new lab.
> > >
> > > Pavel
> >
> > Do you know when it will be available?
> > If it takes some time, how about using qemu emulation for the ppc64le
> > chroot for now?
>
> Fedora Infrastructure are in the middle of 2 data center moves currently
> with a COVID-19 lockdown affecting our ability to give any timelines of
> when things will be available. I do not think the servers will be available
> until May at the earliest and they will be a lower priority than keeping
> koji or other core Fedora services going.

Yeah, this sounds OK.  Good luck with the movement.
> Currently copr is running out of amazon and does not have emulation
> capabilities either.

We already enabled emulated armhfp (on x86_64) builders in copr (that's probably
what Jun meant)... it's the software emulation by qemu-user-static (mock
--forcearch).

@Jun wrote:
> Any update for Copr ppc64le chroots?

People were used to use native machines for ppc64le targets, so the
temporary switch to emulated variant could cause more harm then good here.
I'd rather wait till May for the native builders then mix-up.

Pavel

[Jun Aruga]

> > Fedora Infrastructure are in the middle of 2 data center moves currently
> > with a COVID-19 lockdown affecting our ability to give any timelines of
> > when things will be available. I do not think the servers will be available
> > until May at the earliest and they will be a lower priority than keeping
> > koji or other core Fedora services going.
>
> Yeah, this sounds OK.  Good luck with the movement.

Yeah, OK. Thanks for the work.
> > Currently copr is running out of amazon and does not have emulation
> > capabilities either.
>
> We already enabled emulated armhfp (on x86_64) builders in copr (that's probably
> what Jun meant)... it's the software emulation by qemu-user-static (mock
> --forcearch).

Yes, that is what I meant. The software emulation by qemu-user-static.
> @Jun wrote:
> > Any update for Copr ppc64le chroots?
>
> People were used to use native machines for ppc64le targets, so the
> temporary switch to emulated variant could cause more harm then good here.
> I'd rather wait till May for the native builders then mix-up.

I agree with you. I am okay to wait rather than starting
qemu-user-static emulation for ppc64le targets now.