Workflow and other problems with the Fedora container infrastructure

[Lumír Balhar]

Hello.

My name is Lumír and I maintain a few layered container images in 
Fedora/RHEL/upstream related to Python and S2I (source to image) 
platform - namely python3, s2i-core, and s2i-base.

TL;DR: We (maintainers of layered container images for languages, 
runtimes and databases) are considering migrating our images from Fedora 
infrastructure and Fedora containers registry somewhere else. Below are 
the reasons for that. If you think it’s a bad idea, let us know why. If 
you are a user of those images, let us know as well so we can stay in 
touch with you.

Forgive me for my wording but I’m really sad about the current situation.

I have to honestly admit that I’m disappointed about the lack of 
attention containers in Fedora are getting. Maintaining them is painful, 
infrastructure is unstable and unreliable and there was literally no 
movement forward in recent years. Let me summarize it.

Container SIG is inactive. See important issues opened for years with no 
movement further: https://pagure.io/ContainerSIG/container-sig/issues

Even a simple task like merging rawhide branch to f35 for example is far 
from smooth because they are never the same, because manual changes are 
required for each branch: 
https://pagure.io/ContainerSIG/container-sig/issue/16 (3 years old, no 
reply)

S2I container images depend on each other (python3 -> s2i-base -> 
s2i-core) but there is no automation for chain rebuilds so if I want to 
update s2i-core image (to fix a CVE or something like that) and all 
dependant images, it takes weeks because I have to rebuild them one by 
one and let them one by one pass through Bodhi updates until I can 
rebuild next one in the queue. And it’s really frustrating given the 
insufficient stability of the infra. There seems to be a way how to make 
this faster, but nobody knows, how it works: 
https://pagure.io/ContainerSIG/container-sig/issue/52 and 
https://github.com/containerbuildsystem/atomic-reactor/issues/1730

The same applies also if the fix is done on the RPM level because there 
are no freshmaker nor automated rebuilds.

The branching structure does not really fit container needs. I maintain 
the s2i Python container and I don’t really care whether it’s based on 
Fedora 34 or 35 (both have Python 3.10 as the main one) so I’d ideally 
have one branch for Python 3.10, one for 3.9 etc. instead of two 
separated branches (f34, f35) where the Python is the same version. 
https://pagure.io/ContainerSIG/container-sig/issue/12 (3 years old)

An example of unstable infrastructure: I’m not able to build an image 
for three weeks now and a response I got is to report the problem 
upstream: https://pagure.io/fedora-infrastructure/issue/10437
I know that nothing is 100% reliable but if you take a look at these 
issues, there are a lot of them: 
https://pagure.io/fedora-infrastructure/issues?search_pattern=container&a...


And it seems that the packages in the infrastructure are outdated a lot: 
https://github.com/containerbuildsystem/atomic-reactor/issues/1742#issuec...



If we migrate our container images to some other registry (e.g. a common 
fedora space on quay.io), we’ll be able to rebuild them after every 
merged PR or every week, do chain rebuilds and push them to the registry 
directly from Github CI. This will make our lives significantly easier, 
and in turn our images better.

Are there any benefits to using Fedora infrastructure and Fedora 
containers registry which I don’t see that we should consider?

Lumír

[Kevin Fenzi]

On Tue, Jan 11, 2022 at 01:02:59PM +0100, Lumír Balhar wrote:
>  Hello.
>  
>  My name is Lumír and I maintain a few layered container images in
>  Fedora/RHEL/upstream related to Python and S2I (source to image) platform -
>  namely python3, s2i-core, and s2i-base.
>  
>  TL;DR: We (maintainers of layered container images for languages, runtimes
>  and databases) are considering migrating our images from Fedora
>  infrastructure and Fedora containers registry somewhere else. Below are the
>  reasons for that. If you think it’s a bad idea, let us know why. If you are
>  a user of those images, let us know as well so we can stay in touch with
>  you.
>  
>  Forgive me for my wording but I’m really sad about the current situation. 
yeah, it is sad. ;( 

>  I have to honestly admit that I’m disappointed about the lack of
> attention
>  containers in Fedora are getting. Maintaining them is painful,
>  infrastructure is unstable and unreliable and there was literally no
>  movement forward in recent years. Let me summarize it.
>  
>  Container SIG is inactive. See important issues opened for years with no
>  movement further: https://pagure.io/ContainerSIG/container-sig/issues
>  
>  Even a simple task like merging rawhide branch to f35 for example is far
>  from smooth because they are never the same, because manual changes are
>  required for each branch:
>  https://pagure.io/ContainerSIG/container-sig/issue/16 (3 years old, no
>  reply)
>  
>  S2I container images depend on each other (python3 -> s2i-base -> s2i-core)
>  but there is no automation for chain rebuilds so if I want to update
>  s2i-core image (to fix a CVE or something like that) and all dependant
>  images, it takes weeks because I have to rebuild them one by one and let
>  them one by one pass through Bodhi updates until I can rebuild next one in
>  the queue. And it’s really frustrating given the insufficient stability of
>  the infra. There seems to be a way how to make this faster, but nobody
>  knows, how it works: https://pagure.io/ContainerSIG/container-sig/issue/52
>  and https://github.com/containerbuildsystem/atomic-reactor/issues/1730
>  
>  The same applies also if the fix is done on the RPM level because there are
>  no freshmaker nor automated rebuilds.
>  
>  The branching structure does not really fit container needs. I maintain the
>  s2i Python container and I don’t really care whether it’s based on Fedora 34
>  or 35 (both have Python 3.10 as the main one) so I’d ideally have one branch
>  for Python 3.10, one for 3.9 etc. instead of two separated branches (f34,
>  f35) where the Python is the same version.
>  https://pagure.io/ContainerSIG/container-sig/issue/12 (3 years old) 
Yeah, I think all the above is kind of a consequence of there not being
a active containers SIG. Infrastructure and releng doesn't really have
cycles or interest in driving containers forward, so it needs some other
folks attention. ;( 

>  An example of unstable infrastructure: I’m not able to build an image
> for
>  three weeks now and a response I got is to report the problem upstream:
>  https://pagure.io/fedora-infrastructure/issue/10437
>  I know that nothing is 100% reliable but if you take a look at these issues,
>  there are a lot of them:
> https://pagure.io/fedora-infrastructure/issues?search_pattern=container&a...
Do note that that issue was filed on December 21st. 
Myself and much of the infrastructure team were off on holidays. 
I only got back yesterday and there's a ton of things to do, but I did
try and look into this some. 

The reason there's so many issues I think is because the entire pipeline
is just way overkill and complex for just building containers. Basically
we have to have koji, koji builders, and 2 openshift clusters (one for
x86 and one for aarch64) to build containers, they have all kinds of
fragile handoffs between each other as well. 

>  And it seems that the packages in the infrastructure are outdated a
> lot: 
>
> https://github.com/containerbuildsystem/atomic-reactor/issues/1742#issuec...
To clarify, thats installed in the build container... so, _fedora_
packages are outdated. ;( I have seen atomic-reactor being old, but I've
been afraid to update it since I don't know the pipeline. 
Co-maintainers of those packages in Fedora would sure help out. 

>  If we migrate our container images to some other registry (e.g. a
> common
>  fedora space on quay.io), we’ll be able to rebuild them after every merged
>  PR or every week, do chain rebuilds and push them to the registry directly
>  from Github CI. This will make our lives significantly easier, and in turn
>  our images better. 
We are looking into dropping out registry and moving to quay.io
ourselves. ;) 

>  Are there any benefits to using Fedora infrastructure and Fedora
> containers
>  registry which I don’t see that we should consider? 
I'm not sure... hopefully folks who know the pipeline will chime in
here. 

kevin

[Honza Horak]

On Wed, Jan 12, 2022 at 5:51 AM Kevin Fenzi <kevin(a)scrye.com&gt; wrote:

>  On Tue, Jan 11, 2022 at 01:02:59PM +0100, Lumír Balhar wrote:
>  > If we migrate our container images to some other registry (e.g. a common
>  > fedora space on quay.io), we’ll be able to rebuild them after every
>  merged
>  > PR or every week, do chain rebuilds and push them to the registry
>  directly
>  > from Github CI. This will make our lives significantly easier, and in
>  turn
>  > our images better.
>
>  We are looking into dropping out registry and moving to quay.io
>  ourselves. ;)
>
>  
Is this something more than a thought at this point? Something where we can
follow the design and progress?


>  > Are there any benefits to using Fedora infrastructure and
> Fedora
>  containers
>  > registry which I don’t see that we should consider?
>
>  I'm not sure... hopefully folks who know the pipeline will chime in
>  here.
>  
If you know who the folks are, we'd be glad for a gentle ping, so they
comment here and we do not stop providing something that other would start
to miss.

Thanks,
Honza


>  kevin
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct:
>  https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
>  https://lists.fedoraproject.org/archives/list/[e-mail]
>  Do not reply to spam on the list, report it:
>  https://pagure.io/fedora-infrastructure
>

[Kevin Fenzi]

On Thu, Jan 13, 2022 at 02:43:50PM +0100, Honza Horak wrote:
>  On Wed, Jan 12, 2022 at 5:51 AM Kevin Fenzi <kevin(a)scrye.com&gt;
> wrote:
>  
>  > On Tue, Jan 11, 2022 at 01:02:59PM +0100, Lumír Balhar wrote:
>  > > If we migrate our container images to some other registry (e.g. a common
>  > > fedora space on quay.io), we’ll be able to rebuild them after every
>  > merged
>  > > PR or every week, do chain rebuilds and push them to the registry
>  > directly
>  > > from Github CI. This will make our lives significantly easier, and in
>  > turn
>  > > our images better.
>  >
>  > We are looking into dropping out registry and moving to quay.io
>  > ourselves. ;)
>  >
>  >
>  Is this something more than a thought at this point? Something where we can
>  follow the design and progress? 

We have https://pagure.io/fedora-infrastructure/issue/10386
tracking it. It's definitely something we want to do, but not sure when
we will get to it. 

>  > > Are there any benefits to using Fedora infrastructure and
> Fedora
>  > containers
>  > > registry which I don’t see that we should consider?
>  >
>  > I'm not sure... hopefully folks who know the pipeline will chime in
>  > here.
>  >
>  
>  If you know who the folks are, we'd be glad for a gentle ping, so they
>  comment here and we do not stop providing something that other would start
>  to miss. 
Perhaps the Fedora CoreOS folks would have some thoughts?

kevin

[Neal Gompa]

On Tue, Jan 11, 2022 at 11:51 PM Kevin Fenzi <kevin(a)scrye.com&gt; wrote:
>  On Tue, Jan 11, 2022 at 01:02:59PM +0100, Lumír Balhar wrote:
>  > Hello.
>  >
>  > My name is Lumír and I maintain a few layered container images in
>  > Fedora/RHEL/upstream related to Python and S2I (source to image) platform -
>  > namely python3, s2i-core, and s2i-base.
>  >
>  > TL;DR: We (maintainers of layered container images for languages, runtimes
>  > and databases) are considering migrating our images from Fedora
>  > infrastructure and Fedora containers registry somewhere else. Below are the
>  > reasons for that. If you think it’s a bad idea, let us know why. If you are
>  > a user of those images, let us know as well so we can stay in touch with
>  > you.
>  >
>  > Forgive me for my wording but I’m really sad about the current situation.
>
>  yeah, it is sad. ;(
>
>  > I have to honestly admit that I’m disappointed about the lack of attention
>  > containers in Fedora are getting. Maintaining them is painful,
>  > infrastructure is unstable and unreliable and there was literally no
>  > movement forward in recent years. Let me summarize it.
>  >
>  > Container SIG is inactive. See important issues opened for years with no
>  > movement further: https://pagure.io/ContainerSIG/container-sig/issues
>  >
>  > Even a simple task like merging rawhide branch to f35 for example is far
>  > from smooth because they are never the same, because manual changes are
>  > required for each branch:
>  > https://pagure.io/ContainerSIG/container-sig/issue/16 (3 years old, no
>  > reply)
>  >
>  > S2I container images depend on each other (python3 -> s2i-base -> s2i-core)
>  > but there is no automation for chain rebuilds so if I want to update
>  > s2i-core image (to fix a CVE or something like that) and all dependant
>  > images, it takes weeks because I have to rebuild them one by one and let
>  > them one by one pass through Bodhi updates until I can rebuild next one in
>  > the queue. And it’s really frustrating given the insufficient stability of
>  > the infra. There seems to be a way how to make this faster, but nobody
>  > knows, how it works: https://pagure.io/ContainerSIG/container-sig/issue/52
>  > and https://github.com/containerbuildsystem/atomic-reactor/issues/1730
>  >
>  > The same applies also if the fix is done on the RPM level because there are
>  > no freshmaker nor automated rebuilds.
>  >
>  > The branching structure does not really fit container needs. I maintain the
>  > s2i Python container and I don’t really care whether it’s based on Fedora 34
>  > or 35 (both have Python 3.10 as the main one) so I’d ideally have one branch
>  > for Python 3.10, one for 3.9 etc. instead of two separated branches (f34,
>  > f35) where the Python is the same version.
>  > https://pagure.io/ContainerSIG/container-sig/issue/12 (3 years old)
>
>  Yeah, I think all the above is kind of a consequence of there not being
>  a active containers SIG. Infrastructure and releng doesn't really have
>  cycles or interest in driving containers forward, so it needs some other
>  folks attention. ;(
>
>  > An example of unstable infrastructure: I’m not able to build an image for
>  > three weeks now and a response I got is to report the problem upstream:
>  > https://pagure.io/fedora-infrastructure/issue/10437
>  > I know that nothing is 100% reliable but if you take a look at these issues,
>  > there are a lot of them:
> https://pagure.io/fedora-infrastructure/issues?search_pattern=container&a...
>
>  Do note that that issue was filed on December 21st.
>  Myself and much of the infrastructure team were off on holidays.
>  I only got back yesterday and there's a ton of things to do, but I did
>  try and look into this some.
>
>  The reason there's so many issues I think is because the entire pipeline
>  is just way overkill and complex for just building containers. Basically
>  we have to have koji, koji builders, and 2 openshift clusters (one for
>  x86 and one for aarch64) to build containers, they have all kinds of
>  fragile handoffs between each other as well.
>
>  > And it seems that the packages in the infrastructure are outdated a lot:
>  >
> https://github.com/containerbuildsystem/atomic-reactor/issues/1742#issuec...
>
>  To clarify, thats installed in the build container... so, _fedora_
>  packages are outdated. ;( I have seen atomic-reactor being old, but I've
>  been afraid to update it since I don't know the pipeline.
>  Co-maintainers of those packages in Fedora would sure help out.
>  
One of the things that has recently happened in the Koji space is the
addition of a kiwi-build task to build images using the KIWI tool[1].

KIWI supports building all kinds of operating system images, including
OCI containers. The Fedora Cloud WG is poking at the idea of using
KIWI for the cloud image to replace the unmaintained and brittle
ImageFactory, and we could also look at doing container builds with
KIWI to replace the OpenShift Atomic Reactor system. That would
drastically simplify the architecture and make container image builds
considerably more reasonable for the Container SIG and any other
stakeholders.

[1]: https://osinside.github.io/kiwi/




--
真実はいつも一つ！/ Always, there's only one truth!

[Kevin Fenzi]

On Thu, Jan 13, 2022 at 02:14:19PM -0500, Neal Gompa wrote:
>  
>  One of the things that has recently happened in the Koji space is the
>  addition of a kiwi-build task to build images using the KIWI tool[1].
>  
>  KIWI supports building all kinds of operating system images, including
>  OCI containers. The Fedora Cloud WG is poking at the idea of using
>  KIWI for the cloud image to replace the unmaintained and brittle
>  ImageFactory, and we could also look at doing container builds with
>  KIWI to replace the OpenShift Atomic Reactor system. That would
>  drastically simplify the architecture and make container image builds
>  considerably more reasonable for the Container SIG and any other
>  stakeholders. 
Yeah, thats quite interesting. I would be happy to move to a pipeline
thats less fragile here. :) 

There's also talk about moving things to use ImageBuilder, but I don't
think it does containers. 

kevin

[Josh Boyer]

On Sat, Jan 15, 2022, 6:57 PM Kevin Fenzi <kevin(a)scrye.com&gt; wrote:

>  On Thu, Jan 13, 2022 at 02:14:19PM -0500, Neal Gompa wrote:
>  >
>  > One of the things that has recently happened in the Koji space is the
>  > addition of a kiwi-build task to build images using the KIWI tool[1].
>  >
>  > KIWI supports building all kinds of operating system images, including
>  > OCI containers. The Fedora Cloud WG is poking at the idea of using
>  > KIWI for the cloud image to replace the unmaintained and brittle
>  > ImageFactory, and we could also look at doing container builds with
>  > KIWI to replace the OpenShift Atomic Reactor system. That would
>  > drastically simplify the architecture and make container image builds
>  > considerably more reasonable for the Container SIG and any other
>  > stakeholders.
>
>  Yeah, thats quite interesting. I would be happy to move to a pipeline
>  thats less fragile here. :)
>
>  There's also talk about moving things to use ImageBuilder, but I don't
>  think it does containers. 

We can, and should, have RFEs for Image Builder to do containers.  I know
we need this internally as well.  It may farm that out to buildah in the
background or something, but that remains to be determined.

In the interest of commonality across the Fedora/CentOS/RHEL ecosystem, I
really think using Image Builder as the tool to build images is the best
approach.

josh

[Neal Gompa]

On Sun, Jan 16, 2022 at 8:43 AM Josh Boyer <jwboyer(a)fedoraproject.org&gt; wrote:
>  On Sat, Jan 15, 2022, 6:57 PM Kevin Fenzi <kevin(a)scrye.com&gt; wrote:
> >
> > On Thu, Jan 13, 2022 at 02:14:19PM -0500, Neal Gompa wrote:
> > >
> > > One of the things that has recently happened in the Koji space is the
> > > addition of a kiwi-build task to build images using the KIWI tool[1].
> > >
> > > KIWI supports building all kinds of operating system images, including
> > > OCI containers. The Fedora Cloud WG is poking at the idea of using
> > > KIWI for the cloud image to replace the unmaintained and brittle
> > > ImageFactory, and we could also look at doing container builds with
> > > KIWI to replace the OpenShift Atomic Reactor system. That would
> > > drastically simplify the architecture and make container image builds
> > > considerably more reasonable for the Container SIG and any other
> > > stakeholders.
> >
> > Yeah, thats quite interesting. I would be happy to move to a pipeline
> > thats less fragile here. :)
> >
> > There's also talk about moving things to use ImageBuilder, but I don't
> > think it does containers.
>
>
>  We can, and should, have RFEs for Image Builder to do containers.  I know we need this
> internally as well.  It may farm that out to buildah in the background or something, but
> that remains to be determined.
>  
The "Image Builder" architecture is almost as bad as OpenShift Atomic
Reactor. It requires a whole separate service architecture and can't
directly execute inside of a Koji mock environment. That means that
it's almost as difficult and fragile to keep up with. Moreover, it
can't do nearly all of the image types that Fedora needs, notably live
media, install DVDs, or container images. The only thing it does that
almost all other tools do not is RPM-OSTree commits.

I would like OSBuild to work in Fedora the same way most of our other
image build tools do: fork+exec in dynamically spun up mock roots
rather than several services across multiple dedicated machines. The
ones that don't follow that pattern tend to be the poorly maintained
ones, such as ImageFactory and OpenShift Atomic Reactor.

>  In the interest of commonality across the Fedora/CentOS/RHEL
> ecosystem, I really think using Image Builder as the tool to build images is the best
> approach.
>  
In the interest of the Fedora ecosystem, we should use the best tools
that people can use. But also keep in mind that we don't have to use a
single tool for all the Fedora's image tooling. Heck, right now we use
over half a dozen different ones (including different tools for
producing RPM-OSTree commits depending on the Fedora variant). Going
down to 2 or 3 that are much easier for people to self-host and run is
better than the current situation.



-- 
真実はいつも一つ！/ Always, there's only one truth!

[Colin Walters]

On Thu, Jan 13, 2022, at 1:48 PM, Kevin Fenzi wrote:
>  Perhaps the Fedora CoreOS folks would have some thoughts? 
I can't speak for the whole team, but a few points.  First, the FCOS build tooling in
https://github.com/coreos/coreos-assembler is designed to run as a standard container.  In
some cases we do run it via podman over ssh as part of multi-arch, but the main approach
is to run it inside Kubernetes (OpenShift).  We designed it this way because OpenShift is
of great interest to at least my employer (in case anyone didn't know).  That's
how we run production container workloads.

Until now however, we have really had a very interesting tension because the primary
output of FCOS builds is not a container image, it's bootable disk images (as produced
by many tools, including ImageFactory, Image Builder, the kiwi thing in this thread, and
many others).

However, https://fedoraproject.org/wiki/Changes/OstreeNativeContainer is going to shift
our "center of gravity" much closer to a container build.  I'd actually like
to "decouple" the disk image builds from container builds in our pipeline more,
basically so that we generate disk images using a container image as *input* - for FCOS as
well as other ostree systems today, a bootable disk image is really just a
platform-specific wrapper shell around that.  

Related to this, I am quite strongly of the opinion that the *build* system should be
closely related to the *testing* system.  And that relates to the "running in
production" bits mentioned above.  If we're building containers, then we should
at least be testing them running inside a Kubernetes/OpenShift instance.  And if you have
that, then it just makes sense to use the same approach to run the build tooling - as a
container.  The build process is just another workload along with testing processes and
other tools inside a production Kubernetes/OpenShift cluster.

This is how it works today for the FCOS pipeline as well as downstream ones, and as
mentioned above I think the ostree native container change will be a powerful incentive to
"lift" the ostree side of things outside of Koji and into a container-native
flow.

[Pierre-Yves Chibon]

On Sun, Jan 16, 2022 at 08:42:30AM -0500, Josh Boyer wrote:
>     On Sat, Jan 15, 2022, 6:57 PM Kevin Fenzi
> <[1]kevin(a)scrye.com&gt; wrote:
>  
>       On Thu, Jan 13, 2022 at 02:14:19PM -0500, Neal Gompa wrote:
>       >
>       > One of the things that has recently happened in the Koji space is the
>       > addition of a kiwi-build task to build images using the KIWI tool[1].
>       >
>       > KIWI supports building all kinds of operating system images, including
>       > OCI containers. The Fedora Cloud WG is poking at the idea of using
>       > KIWI for the cloud image to replace the unmaintained and brittle
>       > ImageFactory, and we could also look at doing container builds with
>       > KIWI to replace the OpenShift Atomic Reactor system. That would
>       > drastically simplify the architecture and make container image builds
>       > considerably more reasonable for the Container SIG and any other
>       > stakeholders.
>  
>       Yeah, thats quite interesting. I would be happy to move to a pipeline
>       thats less fragile here. :)
>  
>       There's also talk about moving things to use ImageBuilder, but I don't
>       think it does containers.
>  
>     We can, and should, have RFEs for Image Builder to do containers.  I know
>     we need this internally as well.  It may farm that out to buildah in the
>     background or something, but that remains to be determined.
>     In the interest of commonality across the Fedora/CentOS/RHEL ecosystem, I
>     really think using Image Builder as the tool to build images is the best
>     approach. 
The underlying tool, osbuild, can already build container. They can be made
available as tarball which one can just `podman import` afterward :)


Pierre

[Josh Boyer]

On Tue, Jan 18, 2022 at 8:40 AM Pierre-Yves Chibon <pingou(a)pingoured.fr&gt; wrote:
>  On Sun, Jan 16, 2022 at 08:42:30AM -0500, Josh Boyer wrote:
>  >    On Sat, Jan 15, 2022, 6:57 PM Kevin Fenzi <[1]kevin(a)scrye.com&gt; wrote:
>  >
>  >      On Thu, Jan 13, 2022 at 02:14:19PM -0500, Neal Gompa wrote:
>  >      >
>  >      > One of the things that has recently happened in the Koji space is the
>  >      > addition of a kiwi-build task to build images using the KIWI tool[1].
>  >      >
>  >      > KIWI supports building all kinds of operating system images, including
>  >      > OCI containers. The Fedora Cloud WG is poking at the idea of using
>  >      > KIWI for the cloud image to replace the unmaintained and brittle
>  >      > ImageFactory, and we could also look at doing container builds with
>  >      > KIWI to replace the OpenShift Atomic Reactor system. That would
>  >      > drastically simplify the architecture and make container image builds
>  >      > considerably more reasonable for the Container SIG and any other
>  >      > stakeholders.
>  >
>  >      Yeah, thats quite interesting. I would be happy to move to a pipeline
>  >      thats less fragile here. :)
>  >
>  >      There's also talk about moving things to use ImageBuilder, but I don't
>  >      think it does containers.
>  >
>  >    We can, and should, have RFEs for Image Builder to do containers.  I know
>  >    we need this internally as well.  It may farm that out to buildah in the
>  >    background or something, but that remains to be determined.
>  >    In the interest of commonality across the Fedora/CentOS/RHEL ecosystem, I
>  >    really think using Image Builder as the tool to build images is the best
>  >    approach.
>
>  The underlying tool, osbuild, can already build container. They can be made
>  available as tarball which one can just `podman import` afterward :) 
Yes.  Why make people do that instead of just publishing an OCI
compliant image?  I know why, but that's the RFE :)

josh