Building VM images for Copr builders

[Pavel Raiskup]

Hello infra!

Currently, in Copr team we are periodically preparing those kinds of images for
our VM builders:

- x86_64 AMI image for AWS
- aarch64 AMI
- x86_64 image for hypervisors
- ppc64le qcow2 image (uploaded to Power9 OpenStack and to our Power8 hypervisors)
- s390x qcow2 (built in IBM Cloud)
- internally we build also x86_64 images for OpenStack

The way we do this now is pretty complicated:

- we start with the officially provided Fedora images (AMI/QCOW2)
- we modify and update them using ansible scripts and/or libguestfs
- then the images are uploaded, and tested in our development stack
- and if everything is OK, then these images are used in production

Some more info https://docs.pagure.org/copr.copr/how_to_upgrade_builders.html

This is a tedious and repetitive list of tasks, and I'm sure we
should/could automatize some parts.  At least it would be awesome if we
could try to trigger the build(s) for all the images by "a single button",
somehow, somewhere...  So I am here to ask you how you would do this.

- I thought the answer is Image Builder, but there's a missing support for
  Fedora ppc64le and s390x targets for now (rhbz#2040685).

- livemedia-creator, seems to be a bit related to Image Builder but I was told
  some time ago that AMI support is experimental, the docs say
  "At this time I have not tested the image with EC2. Feedback would be
  welcome."

- There's a Packer software (packer.io) used by testing farm folks (the tooling
  is not available in Fedora yet).

- Koji is able to build some images using Kickstarts.  This one looks like
  low-hanging fruit.  Existing and working solution, we could built on top of
  of the maintained fedora-kickstarts files that we "use" anyway (through the
  pre-built official images).  But I'm not sure if our team could be allowed to
  use Koji like this, and provide additional *ks files?

Are there any other possibilities?  Opinions?

Thanks, Pavel

[LEO PUVILLAND]

I have looked at Packer and it looks like it could be used for this. Not
sure how koji works with kickstarts but would love to check it out! On the
authorization logistics side, I'm not the right person to talk to, but if I
could help in any way that would be great.
Thanks,
Leo

On Wed, Feb 23, 2022 at 9:19 AM Pavel Raiskup <praiskup(a)redhat.com&gt; wrote:

>  Hello infra!
>
>  Currently, in Copr team we are periodically preparing those kinds of
>  images for
>  our VM builders:
>
>  - x86_64 AMI image for AWS
>  - aarch64 AMI
>  - x86_64 image for hypervisors
>  - ppc64le qcow2 image (uploaded to Power9 OpenStack and to our Power8
>  hypervisors)
>  - s390x qcow2 (built in IBM Cloud)
>  - internally we build also x86_64 images for OpenStack
>
>  The way we do this now is pretty complicated:
>
>  - we start with the officially provided Fedora images (AMI/QCOW2)
>  - we modify and update them using ansible scripts and/or libguestfs
>  - then the images are uploaded, and tested in our development stack
>  - and if everything is OK, then these images are used in production
>
>  Some more info
>  https://docs.pagure.org/copr.copr/how_to_upgrade_builders.html
>
>  This is a tedious and repetitive list of tasks, and I'm sure we
>  should/could automatize some parts.  At least it would be awesome if we
>  could try to trigger the build(s) for all the images by "a single button",
>  somehow, somewhere...  So I am here to ask you how you would do this.
>
>  - I thought the answer is Image Builder, but there's a missing support for
>    Fedora ppc64le and s390x targets for now (rhbz#2040685).
>
>  - livemedia-creator, seems to be a bit related to Image Builder but I was
>  told
>    some time ago that AMI support is experimental, the docs say
>    "At this time I have not tested the image with EC2. Feedback would be
>    welcome."
>
>  - There's a Packer software (packer.io) used by testing farm folks (the
>  tooling
>    is not available in Fedora yet).
>
>  - Koji is able to build some images using Kickstarts.  This one looks like
>    low-hanging fruit.  Existing and working solution, we could built on top
>  of
>    of the maintained fedora-kickstarts files that we "use" anyway (through
>  the
>    pre-built official images).  But I'm not sure if our team could be
>  allowed to
>    use Koji like this, and provide additional *ks files?
>
>  Are there any other possibilities?  Opinions?
>
>  Thanks, Pavel
>
>
>  _______________________________________________
>  infrastructure mailing list -- infrastructure(a)lists.fedoraproject.org
>  To unsubscribe send an email to
>  infrastructure-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct:
>  https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
>  https://lists.fedoraproject.org/archives/list/[e-mail]...
>  Do not reply to spam on the list, report it:
>  https://pagure.io/fedora-infrastructure
>

[Andrew Heath]

Pavel, I know my team uses packer.io to build our golden images for our
infrastructure. I know we build qcow2 and AMI images and have them uploaded
to our AWS, OpenStack, and RHV instances. I would be willing to sit down
and help test this with you if you would like.

On Wed, Feb 23, 2022, 12:19 PM Pavel Raiskup <praiskup(a)redhat.com&gt; wrote:

>  Hello infra!
>
>  Currently, in Copr team we are periodically preparing those kinds of
>  images for
>  our VM builders:
>
>  - x86_64 AMI image for AWS
>  - aarch64 AMI
>  - x86_64 image for hypervisors
>  - ppc64le qcow2 image (uploaded to Power9 OpenStack and to our Power8
>  hypervisors)
>  - s390x qcow2 (built in IBM Cloud)
>  - internally we build also x86_64 images for OpenStack
>
>  The way we do this now is pretty complicated:
>
>  - we start with the officially provided Fedora images (AMI/QCOW2)
>  - we modify and update them using ansible scripts and/or libguestfs
>  - then the images are uploaded, and tested in our development stack
>  - and if everything is OK, then these images are used in production
>
>  Some more info
>  https://docs.pagure.org/copr.copr/how_to_upgrade_builders.html
>
>  This is a tedious and repetitive list of tasks, and I'm sure we
>  should/could automatize some parts.  At least it would be awesome if we
>  could try to trigger the build(s) for all the images by "a single button",
>  somehow, somewhere...  So I am here to ask you how you would do this.
>
>  - I thought the answer is Image Builder, but there's a missing support for
>    Fedora ppc64le and s390x targets for now (rhbz#2040685).
>
>  - livemedia-creator, seems to be a bit related to Image Builder but I was
>  told
>    some time ago that AMI support is experimental, the docs say
>    "At this time I have not tested the image with EC2. Feedback would be
>    welcome."
>
>  - There's a Packer software (packer.io) used by testing farm folks (the
>  tooling
>    is not available in Fedora yet).
>
>  - Koji is able to build some images using Kickstarts.  This one looks like
>    low-hanging fruit.  Existing and working solution, we could built on top
>  of
>    of the maintained fedora-kickstarts files that we "use" anyway (through
>  the
>    pre-built official images).  But I'm not sure if our team could be
>  allowed to
>    use Koji like this, and provide additional *ks files?
>
>  Are there any other possibilities?  Opinions?
>
>  Thanks, Pavel
>
>
>  _______________________________________________
>  infrastructure mailing list -- infrastructure(a)lists.fedoraproject.org
>  To unsubscribe send an email to
>  infrastructure-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct:
>  https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
>  https://lists.fedoraproject.org/archives/list/[e-mail]...
>  Do not reply to spam on the list, report it:
>  https://pagure.io/fedora-infrastructure
>

[Neal Gompa]

On Wed, Feb 23, 2022 at 12:19 PM Pavel Raiskup <praiskup(a)redhat.com&gt; wrote:
>  Hello infra!
>
>  Currently, in Copr team we are periodically preparing those kinds of images for
>  our VM builders:
>
>  - x86_64 AMI image for AWS
>  - aarch64 AMI
>  - x86_64 image for hypervisors
>  - ppc64le qcow2 image (uploaded to Power9 OpenStack and to our Power8 hypervisors)
>  - s390x qcow2 (built in IBM Cloud)
>  - internally we build also x86_64 images for OpenStack
>
>  The way we do this now is pretty complicated:
>
>  - we start with the officially provided Fedora images (AMI/QCOW2)
>  - we modify and update them using ansible scripts and/or libguestfs
>  - then the images are uploaded, and tested in our development stack
>  - and if everything is OK, then these images are used in production
>
>  Some more info https://docs.pagure.org/copr.copr/how_to_upgrade_builders.html
>
>  This is a tedious and repetitive list of tasks, and I'm sure we
>  should/could automatize some parts.  At least it would be awesome if we
>  could try to trigger the build(s) for all the images by "a single button",
>  somehow, somewhere...  So I am here to ask you how you would do this.
>
>  - I thought the answer is Image Builder, but there's a missing support for
>    Fedora ppc64le and s390x targets for now (rhbz#2040685).
>
>  - livemedia-creator, seems to be a bit related to Image Builder but I was told
>    some time ago that AMI support is experimental, the docs say
>    "At this time I have not tested the image with EC2. Feedback would be
>    welcome."
>
>  - There's a Packer software (packer.io) used by testing farm folks (the tooling
>    is not available in Fedora yet).
>
>  - Koji is able to build some images using Kickstarts.  This one looks like
>    low-hanging fruit.  Existing and working solution, we could built on top of
>    of the maintained fedora-kickstarts files that we "use" anyway (through the
>    pre-built official images).  But I'm not sure if our team could be allowed to
>    use Koji like this, and provide additional *ks files?
>
>  Are there any other possibilities?  Opinions?
>  
The kiwi image builder would be another option. The Fedora Cloud and
CentOS Hyperscale teams are exploring using it for images, and we've
been collaborating with the Koji folks on kiwi support. Also, at work
I'm doing stuff with kiwi for our cloud images, and I'd be happy to
help teach y'all how to use kiwi for building images for all of your
platforms.

It's packaged in Fedora and EPEL, so it's easy to get and use.




--
真実はいつも一つ！/ Always, there's only one truth!

[Kevin Fenzi]

On Wed, Feb 23, 2022 at 06:18:24PM +0100, Pavel Raiskup wrote:
>  Hello infra!
>  
>  Currently, in Copr team we are periodically preparing those kinds of images for
>  our VM builders: 
...snip...
>  
>  - I thought the answer is Image Builder, but there's a missing support for
>    Fedora ppc64le and s390x targets for now (rhbz#2040685). 
Well, it could still be. My understanding of things is that we could run
ImageBuilder plugins in our koji and when we request builds it uses our
koji to build them instead of internal as a service builders. Of course
thats a ways out so, not sure it helps you now. 

>  - livemedia-creator, seems to be a bit related to Image Builder but I
> was told
>    some time ago that AMI support is experimental, the docs say
>    "At this time I have not tested the image with EC2. Feedback would be
>    welcome."
>  
>  - There's a Packer software (packer.io) used by testing farm folks (the tooling
>    is not available in Fedora yet).
>  
>  - Koji is able to build some images using Kickstarts.  This one looks like
>    low-hanging fruit.  Existing and working solution, we could built on top of
>    of the maintained fedora-kickstarts files that we "use" anyway (through the
>    pre-built official images).  But I'm not sure if our team could be allowed to
>    use Koji like this, and provide additional *ks files? 
We could make that work, it would need setup... (a repo for kickstarts,
permissions and tags setup, etc), but it could work. 

>  Are there any other possibilities?  Opinions? 
A related thought that occurs to me: Right now for koji builders we
install them via virt-install and then run ansible to configure them.
Since we only reinstall or upgrade them once a cycle it's not too much
hassle, but I wonder if it wouldn't make sense to collaborate with you
and come up with a base 'builder image' qcow or raw image and both copr
and fedora builders just use the same image. It might be that the use
cases diverge enough to not make that practical, but thought I would
toss it out there... ;) 

kevin