Fedora CoreOS on multi-arch

[Jakub Cajka]

Hello,

I have pushed out first batch of Fedora CoreOS images for 
non-x86_64. You can run it on your own locally, both resulting images 
and even rebuild them on your own. Please note those images are 
unofficial, built outside of Fedora infrastructure.

Place to grab them is 
https://fedorapeople.org/groups/fcos-images/builds/. Quick start 
information are present in 
https://fedorapeople.org/groups/fcos-images/README based on the official 
FCOS 
quickstart(https://docs.fedoraproject.org/en-US/fedora-coreos/getting-started/). 
Blog post format of them is pending release on Fedora community blog. I 
would much appreciate any feedback and testing.

More onto the "boring" technical details, if you are user you can skip 
rest of the email.

I have been working on building Fedora CoreOS for non-86_64 
architectures for several past months. I'm currently in stage that most 
of the basic building is working along with kola tests for QEMU target. 
I'm also running nightly builds of the CoreOS Assembler container in 
Jenkins and push it in to quay.io and docker hub(not yet manifested, due 
to some tools issues).

quay.io/jcajka/coreos-assembler:latest[arm64.ppc64le,s390x]

docker.io/jcajka/coreso-assembler:latest[arm64.ppc64le,s390x]

Along with that I do build of FCOS. I still manually push images on to 
the fedorapeople.org, I would like to change that eventually.

There is slight deviation on ppc64le as due to some bugs in virt stack, 
I have to use more recent qemu&co for coreos-assembler image, that has 
AFAIK no impact on the resulting images and slightly relaxed filtered 
out packages on s390x(initscripts, config issue pending).

With this announcement I would like to start discussion on how to bring 
all of this on the the official Fedora infrastructure and get official 
multi-arch builds of FCOS out.

I think that coreos-assembler images is best place to start. Move build 
of it from quay to the FCOS Jenkins and source HW for the FCOS Jenkins 
instance so we can run the build on all arches and push out the results 
manifest listed to the chosen container repositories.

I'm looking forward to your feedback. Thanks to all in Fedora CoreOS for 
help.

JC

[Dusty Mabe]

On 5/8/20 4:39 PM, Jakub Cajka wrote:
> Hello,

Hey Jakub!

Thaks for sending this out and putting the effort in to make sure FCOS is at least ready for multi-arch enablement.
> I have pushed out first batch of Fedora CoreOS images for non-x86_64. You can run it on your own locally, both resulting images and even rebuild them on your own. Please note those images are unofficial, built outside of Fedora infrastructure.
>
> Place to grab them is https://fedorapeople.org/groups/fcos-images/builds/. Quick start information are present in https://fedorapeople.org/groups/fcos-images/README based on the official FCOS quickstart(https://docs.fedoraproject.org/en-US/fedora-coreos/getting-started/). Blog post format of them is pending release on Fedora community blog. I would much appreciate any feedback and testing.
>
> More onto the "boring" technical details, if you are user you can skip rest of the email.
>
> I have been working on building Fedora CoreOS for non-86_64 architectures for several past months. I'm currently in stage that most of the basic building is working along with kola tests for QEMU target. I'm also running nightly builds of the CoreOS Assembler container in Jenkins and push it in to quay.io and docker hub(not yet manifested, due to some tools issues).
>
> quay.io/jcajka/coreos-assembler:latest[arm64.ppc64le,s390x]
>
> docker.io/jcajka/coreso-assembler:latest[arm64.ppc64le,s390x]

quick side question on those tags: why `arm64` and no `aarch64` ?
> Along with that I do build of FCOS. I still manually push images on to the fedorapeople.org, I would like to change that eventually.
>
> There is slight deviation on ppc64le as due to some bugs in virt stack, I have to use more recent qemu&co for coreos-assembler image, that has AFAIK no impact on the resulting images and slightly relaxed filtered out packages on s390x(initscripts, config issue pending).
>
> With this announcement I would like to start discussion on how to bring all of this on the the official Fedora infrastructure and get official multi-arch builds of FCOS out.

I think we need to set up a meeting with:

- you and other multi-arch interested parties
- Fedora Infra
- Fedora Releng
- FCOS working group
- CentOS CI team (this is where we run our pipeline now)

WDYT?

Assuming it looks good. I'll try to find some time to set this all up for later this week or early next week.
> I think that coreos-assembler images is best place to start. Move build of it from quay to the FCOS Jenkins and source HW for the FCOS Jenkins instance so we can run the build on all arches and push out the results manifest listed to the chosen container repositories.
>
> I'm looking forward to your feedback. Thanks to all in Fedora CoreOS for help.

Thank you!

[Jonathan Lebon]

Very cool, thanks a lot for working on this!

On 5/8/20 4:39 PM, Jakub Cajka wrote:
> Place to grab them is https://fedorapeople.org/groups/fcos-images/builds/.

Interesting. Have you been "merging" each arch-specific build dirs
into this one manually for now, or do you have some wrapper layer that
does this?

On Mon, May 11, 2020 at 9:31 AM Dusty Mabe  wrote:
> I think we need to set up a meeting with:
> you and other multi-arch interested parties
> Fedora Infra
> Fedora Releng
> FCOS working group
> CentOS CI team (this is where we run our pipeline now)
>
> WDYT?

Definitely agreed with this! Let's figure out what kind of access
we'll have to multi-arch hardware, since I think that's going to be
the biggest hurdle. And we can go from there to see what it'll look
like in the pipeline, cosa, etc...

Worth linking to the upstream tickets related to this:
- https://github.com/coreos/fedora-coreos-tracker/issues/262
- https://pagure.io/fedora-infrastructure/issue/8370

[Jakub Cajka]

----- Original Message -----
> From: "Jonathan Lebon" 
> To: "Dusty Mabe" 
> Cc: [e-mail]
> Sent: Monday, May 11, 2020 4:25:06 PM
> Subject: [CoreOS] Re: Fedora CoreOS on multi-arch
>
> Very cool, thanks a lot for working on this!
>
> On 5/8/20 4:39 PM, Jakub Cajka wrote:
> > Place to grab them is https://fedorapeople.org/groups/fcos-images/builds/.
>
> Interesting. Have you been "merging" each arch-specific build dirs
> into this one manually for now, or do you have some wrapper layer that
> does this?

cosa prune ;). It is hack and bit ugly for now, but it works. I want to look in to separating the bit that regenerates the builds.json metadata(or at least allow to just regenerate with flag). Otherwise it is just copied results from different arches and "pruned"(still manually).
> On Mon, May 11, 2020 at 9:31 AM Dusty Mabe  wrote:
> > I think we need to set up a meeting with:
> > you and other multi-arch interested parties
> > Fedora Infra
> > Fedora Releng
> > FCOS working group
> > CentOS CI team (this is where we run our pipeline now)
> >
> > WDYT?
>
> Definitely agreed with this! Let's figure out what kind of access
> we'll have to multi-arch hardware, since I think that's going to be
> the biggest hurdle. And we can go from there to see what it'll look
> like in the pipeline, cosa, etc...
>
> Worth linking to the upstream tickets related to this:
> https://github.com/coreos/fedora-coreos-tracker/issues/262
> https://pagure.io/fedora-infrastructure/issue/8370

+1, from my side. What are the real hard technical requirements? Can we run of something outside of Fedora/CentOS DC? Do we really need openshift/kube cluster?....

JC
> CoreOS mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]

[Jonathan Lebon]

> > Worth linking to the upstream tickets related to this:
> > https://github.com/coreos/fedora-coreos-tracker/issues/262
> > https://pagure.io/fedora-infrastructure/issue/8370
>
> +1, from my side. What are the real hard technical requirements? Can we run of something outside of Fedora/CentOS DC? Do we really need openshift/kube cluster?....

The ideal IMO would be an OpenShift cluster with multi-arch nodes.
It's unlikely we're going to get that at the beginning though (see
https://github.com/coreos/fedora-coreos-tracker/issues/262#issuecomment-5519...).

So to start with... just really any way we can schedule work onto
multi-arch nodes. There's a gradient there of better to less ideal
options. The fedora-infrastructure ticket for example is about
leveraging Koji. But really even if it's just SSH, we could work with
that.

[Jakub Cajka]

----- Original Message -----
> From: "Dusty Mabe" 
> To: [e-mail]
> Sent: Monday, May 11, 2020 3:30:45 PM
> Subject: [CoreOS] Re: Fedora CoreOS on multi-arch
>
> On 5/8/20 4:39 PM, Jakub Cajka wrote:
> > Hello,
>
> Hey Jakub!
>
> Thaks for sending this out and putting the effort in to make sure FCOS is at
> least ready for multi-arch enablement.
> > I have pushed out first batch of Fedora CoreOS images for non-x86_64.
> >    You can run it on your own locally, both resulting images and even
> >    rebuild them on your own. Please note those images are unofficial,
> >    built outside of Fedora infrastructure.
> >
> > Place to grab them is https://fedorapeople.org/groups/fcos-images/builds/.
> > Quick start information are present in
> > https://fedorapeople.org/groups/fcos-images/README based on the official
> > FCOS
> > quickstart(https://docs.fedoraproject.org/en-US/fedora-coreos/getting-started/).
> > Blog post format of them is pending release on Fedora community blog. I
> > would much appreciate any feedback and testing.
> >
> > More onto the "boring" technical details, if you are user you can skip rest
> > of the email.
> >
> > I have been working on building Fedora CoreOS for non-86_64 architectures
> > for several past months. I'm currently in stage that most of the basic
> > building is working along with kola tests for QEMU target. I'm also
> > running nightly builds of the CoreOS Assembler container in Jenkins and
> > push it in to quay.io and docker hub(not yet manifested, due to some tools
> > issues).
> >
> > quay.io/jcajka/coreos-assembler:latest[arm64.ppc64le,s390x]
> >
> > docker.io/jcajka/coreso-assembler:latest[arm64.ppc64le,s390x]
>
> quick side question on those tags: why `arm64` and no `aarch64` ?

Go arch. All standards around containers that I have seen assume go arch string in the metadata(not really technically needed here, but did that here anyway). Unrelated side note it started to cause issues for aarch64 containers in Fedora recently as we predate the standards and used the aarch64 instead of the arm64.
> > Along with that I do build of FCOS. I still manually push images on to the
> > fedorapeople.org, I would like to change that eventually.
> >
> > There is slight deviation on ppc64le as due to some bugs in virt stack, I
> > have to use more recent qemu&co for coreos-assembler image, that has AFAIK
> > no impact on the resulting images and slightly relaxed filtered out
> > packages on s390x(initscripts, config issue pending).
> >
> > With this announcement I would like to start discussion on how to bring all
> > of this on the the official Fedora infrastructure and get official
> > multi-arch builds of FCOS out.
>
> I think we need to set up a meeting with:
> you and other multi-arch interested parties
> Fedora Infra
> Fedora Releng
> FCOS working group
> CentOS CI team (this is where we run our pipeline now)
>
> WDYT?
>
> Assuming it looks good. I'll try to find some time to set this all up for
> later this week or early next week.

Works for me. I would very much appreciate if you would be able to set it up. I particularly don't have much inside in to the CentOS side. I'm also bit worried about the Fedora side due to the DC move.
Thank you,

JC
> > I think that coreos-assembler images is best place to start. Move build of
> > it from quay to the FCOS Jenkins and source HW for the FCOS Jenkins
> > instance so we can run the build on all arches and push out the results
> > manifest listed to the chosen container repositories.
> >
> > I'm looking forward to your feedback. Thanks to all in Fedora CoreOS for
> > help.
>
> Thank you!
> _______________________________________________
> CoreOS mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]