Hi everyone,
I finished investigation for migration from registry.fp.o to quay.io. It is available in ARC investigation document [0]. The investigation ticket [1] is on fedora-infra tracker.
Looking forward for any feedback.
Michal
[0] - https://fedora-arc.readthedocs.io/en/latest/registry_to_quay/index.html [1] - https://pagure.io/fedora-infrastructure/issue/10386
Il 04/09/23 15:35, Michal Konecny ha scritto:
Hi everyone,
I finished investigation for migration from registry.fp.o to quay.io. It is available in ARC investigation document [0]. The investigation ticket [1] is on fedora-infra tracker.
Looking forward for any feedback.
Michal
Is this going to run on RH cloud as a service, or on premises in Fedora openshift cluster?
Just asking because I see on cloud the cost is dependent on the number of private repositories... I suppose this will be some sort of "for free", at least for the moment, but in future who knows? It also be nice to retain registry.fp.o which I think it could be used if running quay on premise.
Mattia
We already have namespace on quay.io [0] and ostree projects are already using quay.io. We don't plan to host our own instance, just use what is already available.
Michal
[0] - https://quay.io/organization/fedora
On 04. 09. 23 16:44, Mattia Verga wrote:
Il 04/09/23 15:35, Michal Konecny ha scritto:
Hi everyone,
I finished investigation for migration from registry.fp.o to quay.io. It is available in ARC investigation document [0]. The investigation ticket [1] is on fedora-infra tracker.
Looking forward for any feedback.
Michal
Is this going to run on RH cloud as a service, or on premises in Fedora openshift cluster?
Just asking because I see on cloud the cost is dependent on the number of private repositories... I suppose this will be some sort of "for free", at least for the moment, but in future who knows? It also be nice to retain registry.fp.o which I think it could be used if running quay on premise.
Mattia
infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedorapro... Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
On pondělí 4. září 2023 15:35:41 CEST Michal Konecny wrote:
Hi everyone,
I finished investigation for migration from registry.fp.o to quay.io. It is available in ARC investigation document [0]. The investigation ticket [1] is on fedora-infra tracker.
JFYI, Mock switched to "--use-bootstrap-image" (podman pulling images from various registries by default) and we had no single issue reported against the Fedora's registry, but CentOS (on quay.io) gives us random "pull" failures:
https://github.com/rpm-software-management/mock/issues/1191
So the stability might not be as ideal as with the current registry.
Pavel
Looking forward for any feedback.
Michal
[0] - https://fedora-arc.readthedocs.io/en/latest/registry_to_quay/index.html [1] - https://pagure.io/fedora-infrastructure/issue/10386 _______________________________________________ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedorapro... Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
On Mon, Sep 4, 2023 at 12:47 PM Pavel Raiskup praiskup@redhat.com wrote:
On pondělí 4. září 2023 15:35:41 CEST Michal Konecny wrote:
Hi everyone,
I finished investigation for migration from registry.fp.o to quay.io. It is available in ARC investigation document [0]. The investigation ticket [1] is on fedora-infra tracker.
JFYI, Mock switched to "--use-bootstrap-image" (podman pulling images from various registries by default) and we had no single issue reported against the Fedora's registry, but CentOS (on quay.io) gives us random "pull" failures:
https://github.com/rpm-software-management/mock/issues/1191
So the stability might not be as ideal as with the current registry.
Pavel
Looking forward for any feedback.
I'm not super-enthused about this from a few perspectives:
1. Core artifacts should be able to be produced, hosted, and consumed from Fedora infrastructure. 2. Quay ultimately does not need to care about Fedora as a stakeholder 3. Quay.io is moving into console.redhat.com[a], which makes it even less fun since RH accounts for the console require giving a lot more information.
[a]: https://issues.redhat.com/browse/PROJQUAY-5434
-- 真実はいつも一つ！/ Always, there's only one truth!
On 04. 09. 23 19:57, Neal Gompa wrote:
On Mon, Sep 4, 2023 at 12:47 PM Pavel Raiskup praiskup@redhat.com wrote:
On pondělí 4. září 2023 15:35:41 CEST Michal Konecny wrote:
Hi everyone,
I finished investigation for migration from registry.fp.o to quay.io. It is available in ARC investigation document [0]. The investigation ticket [1] is on fedora-infra tracker.
JFYI, Mock switched to "--use-bootstrap-image" (podman pulling images from various registries by default) and we had no single issue reported against the Fedora's registry, but CentOS (on quay.io) gives us random "pull" failures:
https://github.com/rpm-software-management/mock/issues/1191
So the stability might not be as ideal as with the current registry.
Pavel
Looking forward for any feedback.
I'm not super-enthused about this from a few perspectives:
- Core artifacts should be able to be produced, hosted, and consumed
from Fedora infrastructure. 2. Quay ultimately does not need to care about Fedora as a stakeholder 3. Quay.io is moving into console.redhat.com[a], which makes it even less fun since RH accounts for the console require giving a lot more information.
This investigation was about looking what needs to be done to move the current services hosted on registry.fp.o to quay.io. We also already using quay.io for ostree variants and it provides features we currently miss in registry.fp.o. So it was a logical move for us to look more closely at it.
Michal
-- 真実はいつも一つ！/ Always, there's only one truth!
I have one concern about quay.io, more of a double check that quay supports manifest lists for/in image formats that fedora is using. IIRC there have been some limitations on how manifest lists can be used, created in quay.io.
Jakub Čajka
Sep 5, 2023, 10:15 by mkonecny@redhat.com:
On 04. 09. 23 19:57, Neal Gompa wrote:
On Mon, Sep 4, 2023 at 12:47 PM Pavel Raiskup praiskup@redhat.com wrote:
On pondělí 4. září 2023 15:35:41 CEST Michal Konecny wrote:
Hi everyone,
I finished investigation for migration from registry.fp.o to quay.io. It is available in ARC investigation document [0]. The investigation ticket [1] is on fedora-infra tracker.
JFYI, Mock switched to "--use-bootstrap-image" (podman pulling images from various registries by default) and we had no single issue reported against the Fedora's registry, but CentOS (on quay.io) gives us random "pull" failures:
https://github.com/rpm-software-management/mock/issues/1191
So the stability might not be as ideal as with the current registry.
Pavel
Looking forward for any feedback.
I'm not super-enthused about this from a few perspectives:
- Core artifacts should be able to be produced, hosted, and consumed
from Fedora infrastructure. 2. Quay ultimately does not need to care about Fedora as a stakeholder 3. Quay.io is moving into console.redhat.com[a], which makes it even less fun since RH accounts for the console require giving a lot more information.
This investigation was about looking what needs to be done to move the current services hosted on registry.fp.o to quay.io. We also already using quay.io for ostree variants and it provides features we currently miss in registry.fp.o. So it was a logical move for us to look more closely at it.
Michal
-- 真実はいつも一つ！/ Always, there's only one truth!
infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedorapro... Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Hi Jakub,
On Tue, Sep 05, 2023 at 12:56:20PM +0200, Jakub Čajka wrote:
I have one concern about quay.io, more of a double check that quay supports manifest lists for/in image formats that fedora is using. IIRC there have been some limitations on how manifest lists can be used, created in quay.io.
In our experience, quay.io is pretty picky about weird manifest lists, e.g. mixing oci and docker layer types. If you create the manifest lists correctly, quay.io should be happy to accept them.
In any case, nearly all problems can be worked around by forcing skopeo and buildah to convert to a specific format via sth like `--format oci`.
Cheers
Michael
Sep 5, 2023, 10:15 by mkonecny@redhat.com:
On 04. 09. 23 19:57, Neal Gompa wrote:
On Mon, Sep 4, 2023 at 12:47 PM Pavel Raiskup praiskup@redhat.com wrote:
On pondělí 4. září 2023 15:35:41 CEST Michal Konecny wrote:
Hi everyone,
I finished investigation for migration from registry.fp.o to quay.io. It is available in ARC investigation document [0]. The investigation ticket [1] is on fedora-infra tracker.
JFYI, Mock switched to "--use-bootstrap-image" (podman pulling images from various registries by default) and we had no single issue reported against the Fedora's registry, but CentOS (on quay.io) gives us random "pull" failures:
https://github.com/rpm-software-management/mock/issues/1191
So the stability might not be as ideal as with the current registry.
Pavel
Looking forward for any feedback.
I'm not super-enthused about this from a few perspectives:
- Core artifacts should be able to be produced, hosted, and consumed
from Fedora infrastructure. 2. Quay ultimately does not need to care about Fedora as a stakeholder 3. Quay.io is moving into console.redhat.com[a], which makes it even less fun since RH accounts for the console require giving a lot more information.
This investigation was about looking what needs to be done to move the current services hosted on registry.fp.o to quay.io. We also already using quay.io for ostree variants and it provides features we currently miss in registry.fp.o. So it was a logical move for us to look more closely at it.
Michal
On 9/5/23 06:56, Jakub Čajka wrote:
I have one concern about quay.io, more of a double check that quay supports manifest lists for/in image formats that fedora is using. IIRC there have been some limitations on how manifest lists can be used, created in quay.io.
We are using manifest lists for our FCOS images/containers and it's working well:
quay.io/coreos/coreos-assembler quay.io/fedora/fedora-coreos
Dusty
On Mon, Sep 04, 2023 at 01:57:34PM -0400, Neal Gompa wrote:
On Mon, Sep 4, 2023 at 12:47 PM Pavel Raiskup praiskup@redhat.com wrote:
On pondělí 4. září 2023 15:35:41 CEST Michal Konecny wrote:
Hi everyone,
I finished investigation for migration from registry.fp.o to quay.io. It is available in ARC investigation document [0]. The investigation ticket [1] is on fedora-infra tracker.
JFYI, Mock switched to "--use-bootstrap-image" (podman pulling images from various registries by default) and we had no single issue reported against the Fedora's registry, but CentOS (on quay.io) gives us random "pull" failures:
https://github.com/rpm-software-management/mock/issues/1191
So the stability might not be as ideal as with the current registry.
Huh, good to know.
Is this something anyone has taken to upstream quay.io?
I'm not super-enthused about this from a few perspectives:
- Core artifacts should be able to be produced, hosted, and consumed
from Fedora infrastructure.
Well, they still are in koji of course...
- Quay ultimately does not need to care about Fedora as a stakeholder
Sure, but do we have complex needs that require stakeholderness (ok, thats not a word, but you know what I mean. ;)
- Quay.io is moving into console.redhat.com[a], which makes it even less
fun since RH accounts for the console require giving a lot more information.
Huh, good to know. Of course the vast majority of people will just pull from it, never look at the ui.
I think it would be good for us to try and talk to quay.io folks and see if there's any issues or reasons not to head that way.
kevin
On úterý 5. září 2023 19:02:57 CEST Kevin Fenzi wrote:
On Mon, Sep 04, 2023 at 01:57:34PM -0400, Neal Gompa wrote:
On Mon, Sep 4, 2023 at 12:47 PM Pavel Raiskup praiskup@redhat.com wrote:
On pondělí 4. září 2023 15:35:41 CEST Michal Konecny wrote:
Hi everyone,
I finished investigation for migration from registry.fp.o to quay.io. It is available in ARC investigation document [0]. The investigation ticket [1] is on fedora-infra tracker.
JFYI, Mock switched to "--use-bootstrap-image" (podman pulling images from various registries by default) and we had no single issue reported against the Fedora's registry, but CentOS (on quay.io) gives us random "pull" failures:
https://github.com/rpm-software-management/mock/issues/1191
So the stability might not be as ideal as with the current registry.
Huh, good to know.
Is this something anyone has taken to upstream quay.io?
Not yet. Unfortunately, Mock 5.0 swallows the Podman error output :( so it is hard to diagnose what is going on on Copr builders. I hope we'll have better data with Mock 5.1.
Pavel
So I contacted William Dettelback from quay.io Team about the feedback I got here.
This is the e-mail I sent: ``` 1) Mock switched to "--use-bootstrap-image" (podman pulling images from various registries by default) and we had no single issue reported against the Fedora's registry, but CentOS (on quay.io) gives us random "pull" failures:
https://github.com/rpm-software-management/mock/issues/1191
Are you aware of this issue?
2) Quay.io is moving into console.redhat.com[2], which makes it even less fun since RH accounts for the console require giving a lot more information.
Do we need to be Red Hat customers to access that? Could it be possible to allow Fedora Account System login?
3) There is a rate limiting enabled for pulling on quay.io [3]. Could it be possible to remove that if some Fedora services start hitting that? ```
And here is the response I got: ``` Thanks for reaching out- we'd certainly like to support your migration. Fedora makes perfect sense as a tenant on quay.io http://quay.io. Let me try to answer your questions:
1) Not aware of this issue- I don't believe anyone has raised a support ticket with us on it. Wasn't clear to me from the GH issue if you had a stable reproducer. If you do, please feel free to raise a bug report at https://issues.redhat.com/projects/PROJQUAY and we can take a look.
2) Our long term plan is to move all authenticated web UI access to console.redhat.com but we will keep our quay.io web UI available for unauthenticated access (e.g. google search results linking to public images). So only users who need authenticated access to your namespace(s)- for example to administer a Team, etc.. would need to sign up for a Red Hat Account. Robot account / docker CLI access will still work directly and not require RH SSO- so your automation can still push images, etc..
We have no plans to integrate the Fedora Account System login- but open to discuss what that could look like (esp. if it supports OIDC).
3) We can disable the rate limiting on your namespace(s)- it's usually not a problem, we do this for other Red Hat teams (e.g. Openshift). I would be interested to understand more of your expected traffic loads for push/pull so we can plan accordingly on our side. ```
1) Corresponds with what Pavel wrote. I sent it before I noticed the response from Pavel.
2) As FAS is supporting OIDC, we can start negotiating that. Or it would be just mandatory for maintainers of quay.io namespaces to have RedHat account (not that different from managing AWS now).
3) That is really great to hear. Do we have any traffic statistics for registry.fp.o in that regard?
Any thoughts from folks here?
Michal
On 05. 09. 23 19:02, Kevin Fenzi wrote:
On Mon, Sep 04, 2023 at 01:57:34PM -0400, Neal Gompa wrote:
On Mon, Sep 4, 2023 at 12:47 PM Pavel Raiskuppraiskup@redhat.com wrote:
On pondělí 4. září 2023 15:35:41 CEST Michal Konecny wrote:
Hi everyone,
I finished investigation for migration from registry.fp.o to quay.io. It is available in ARC investigation document [0]. The investigation ticket [1] is on fedora-infra tracker.
JFYI, Mock switched to "--use-bootstrap-image" (podman pulling images from various registries by default) and we had no single issue reported against the Fedora's registry, but CentOS (on quay.io) gives us random "pull" failures:
https://github.com/rpm-software-management/mock/issues/1191
So the stability might not be as ideal as with the current registry.
Huh, good to know.
Is this something anyone has taken to upstream quay.io?
I'm not super-enthused about this from a few perspectives:
- Core artifacts should be able to be produced, hosted, and consumed
from Fedora infrastructure.
Well, they still are in koji of course...
- Quay ultimately does not need to care about Fedora as a stakeholder
Sure, but do we have complex needs that require stakeholderness (ok, thats not a word, but you know what I mean. ;)
- Quay.io is moving into console.redhat.com[a], which makes it even less
fun since RH accounts for the console require giving a lot more information.
Huh, good to know. Of course the vast majority of people will just pull from it, never look at the ui.
I think it would be good for us to try and talk to quay.io folks and see if there's any issues or reasons not to head that way.
kevin
infrastructure mailing list --infrastructure@lists.fedoraproject.org To unsubscribe send an email toinfrastructure-leave@lists.fedoraproject.org Fedora Code of Conduct:https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines:https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives:https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedorapro... Do not reply to spam, report it:https://pagure.io/fedora-infrastructure/new_issue
On Thu, Sep 7, 2023 at 10:15 AM Michal Konecny mkonecny@redhat.com wrote:
So I contacted William Dettelback from quay.io Team about the feedback I got here.
This is the e-mail I sent:
1) Mock switched to "--use-bootstrap-image" (podman pulling images from various registries by default) and we had no single issue reported against the Fedora's registry, but CentOS (on quay.io) gives us random "pull" failures: https://github.com/rpm-software-management/mock/issues/1191 Are you aware of this issue? 2) Quay.io is moving into console.redhat.com[2], which makes it even less fun since RH accounts for the console require giving a lot more information. Do we need to be Red Hat customers to access that? Could it be possible to allow Fedora Account System login? 3) There is a rate limiting enabled for pulling on quay.io [3]. Could it be possible to remove that if some Fedora services start hitting that?And here is the response I got:
Thanks for reaching out- we'd certainly like to support your migration. Fedora makes perfect sense as a tenant on quay.io. Let me try to answer your questions: 1) Not aware of this issue- I don't believe anyone has raised a support ticket with us on it. Wasn't clear to me from the GH issue if you had a stable reproducer. If you do, please feel free to raise a bug report at https://issues.redhat.com/projects/PROJQUAY and we can take a look. 2) Our long term plan is to move all authenticated web UI access to console.redhat.com but we will keep our quay.io web UI available for unauthenticated access (e.g. google search results linking to public images). So only users who need authenticated access to your namespace(s)- for example to administer a Team, etc.. would need to sign up for a Red Hat Account. Robot account / docker CLI access will still work directly and not require RH SSO- so your automation can still push images, etc.. We have no plans to integrate the Fedora Account System login- but open to discuss what that could look like (esp. if it supports OIDC). 3) We can disable the rate limiting on your namespace(s)- it's usually not a problem, we do this for other Red Hat teams (e.g. Openshift). I would be interested to understand more of your expected traffic loads for push/pull so we can plan accordingly on our side.
Corresponds with what Pavel wrote. I sent it before I noticed the response from Pavel.
As FAS is supporting OIDC, we can start negotiating that. Or it would be just mandatory for maintainers of quay.io namespaces to have RedHat account (not that different from managing AWS now).
AWS supports being accessed via OIDC SSO, so it's possible to (for example) tie Fedora's AWS account to FAS. I would really like to see FAS supported by Red Hat SSO across the board, especially since now CentOS contributors are forced to deal with Red Hat's Jira instance with the completion of the RHEL-in-JIRA (RIJ) project.
- That is really great to hear. Do we have any traffic statistics for registry.fp.o in that regard?
Can we have an alias for registry.fp.o that goes to our quay namespace too? Breaking the world is not fun, and if Quay doesn't work out, we should be able to painlessly switch to something else.
On Thu, Sep 07, 2023 at 11:07:15AM -0400, Neal Gompa wrote:
On Thu, Sep 7, 2023 at 10:15 AM Michal Konecny mkonecny@redhat.com wrote:
So I contacted William Dettelback from quay.io Team about the feedback I got here.
This is the e-mail I sent:
1) Mock switched to "--use-bootstrap-image" (podman pulling images from various registries by default) and we had no single issue reported against the Fedora's registry, but CentOS (on quay.io) gives us random "pull" failures: https://github.com/rpm-software-management/mock/issues/1191 Are you aware of this issue? 2) Quay.io is moving into console.redhat.com[2], which makes it even less fun since RH accounts for the console require giving a lot more information. Do we need to be Red Hat customers to access that? Could it be possible to allow Fedora Account System login? 3) There is a rate limiting enabled for pulling on quay.io [3]. Could it be possible to remove that if some Fedora services start hitting that?And here is the response I got:
Thanks for reaching out- we'd certainly like to support your migration. Fedora makes perfect sense as a tenant on quay.io. Let me try to answer your questions: 1) Not aware of this issue- I don't believe anyone has raised a support ticket with us on it. Wasn't clear to me from the GH issue if you had a stable reproducer. If you do, please feel free to raise a bug report at https://issues.redhat.com/projects/PROJQUAY and we can take a look. 2) Our long term plan is to move all authenticated web UI access to console.redhat.com but we will keep our quay.io web UI available for unauthenticated access (e.g. google search results linking to public images). So only users who need authenticated access to your namespace(s)- for example to administer a Team, etc.. would need to sign up for a Red Hat Account. Robot account / docker CLI access will still work directly and not require RH SSO- so your automation can still push images, etc..
Yeah, I am not sure this is a big deal, as 99.999% of people will not have any need to login there.
We have no plans to integrate the Fedora Account System login- but open to discuss what that could look like (esp. if it supports OIDC).
- We can disable the rate limiting on your namespace(s)- it's usually not a problem, we do this
for other Red Hat teams (e.g. Openshift). I would be interested to understand more of your expected traffic loads for push/pull so we can plan accordingly on our side.
We may be able to pull that information from logs on oci-registry01/02? Or... now that we have logs going into splunk, we could ask them to just look in splunk? ;)
Corresponds with what Pavel wrote. I sent it before I noticed the response from Pavel.
As FAS is supporting OIDC, we can start negotiating that. Or it would be just mandatory for maintainers of quay.io namespaces to have RedHat account (not that different from managing AWS now).
AWS supports being accessed via OIDC SSO, so it's possible to (for
it's actually SAML2, but yeah...
example) tie Fedora's AWS account to FAS. I would really like to see FAS supported by Red Hat SSO across the board, especially since now CentOS contributors are forced to deal with Red Hat's Jira instance with the completion of the RHEL-in-JIRA (RIJ) project.
Yeah, thats a bigger conversation we should start. I'm not fully sure where...
- That is really great to hear. Do we have any traffic statistics for registry.fp.o in that regard?
Can we have an alias for registry.fp.o that goes to our quay namespace too? Breaking the world is not fun, and if Quay doesn't work out, we should be able to painlessly switch to something else.
Yep. Agree 100%. We should make it so we can switch out if needed and so things move smoothly without users having to change anything or even know too much that it happened.
kevin
infrastructure@lists.fedoraproject.org
-
Dusty Mabe -
Jakub Čajka -
Kevin Fenzi -
Mattia Verga -
Michael Hofmann -
Michal Konecny -
Neal Gompa -
Pavel Raiskup