Workflow and other problems with the Fedora container infrastructure
Hello.
My name is Lumír and I maintain a few layered container images in
Fedora/RHEL/upstream related to Python and S2I (source to image)
platform - namely python3, s2i-core, and s2i-base.
TL;DR: We (maintainers of layered container images for languages,
runtimes and databases) are considering migrating our images from Fedora
infrastructure and Fedora containers registry somewhere else. Below are
the reasons for that. If you think it’s a bad idea, let us know why. If
you are a user of those images, let us know as well so we can stay in
touch with you.
Forgive me for my wording but I’m really sad about the current situation.
I have to honestly admit that I’m disappointed about the lack of
attention containers in Fedora are getting. Maintaining them is painful,
infrastructure is unstable and unreliable and there was literally no
movement forward in recent years. Let me summarize it.
Container SIG is inactive. See important issues opened for years with no
movement further: https://pagure.io/ContainerSIG/container-sig/issues
Even a simple task like merging rawhide branch to f35 for example is far
from smooth because they are never the same, because manual changes are
required for each branch:
https://pagure.io/ContainerSIG/container-sig/issue/16 (3 years old, no
reply)
S2I container images depend on each other (python3 -> s2i-base ->
s2i-core) but there is no automation for chain rebuilds so if I want to
update s2i-core image (to fix a CVE or something like that) and all
dependant images, it takes weeks because I have to rebuild them one by
one and let them one by one pass through Bodhi updates until I can
rebuild next one in the queue. And it’s really frustrating given the
insufficient stability of the infra. There seems to be a way how to make
this faster, but nobody knows, how it works:
https://pagure.io/ContainerSIG/container-sig/issue/52 and
https://github.com/containerbuildsystem/atomic-reactor/issues/1730
The same applies also if the fix is done on the RPM level because there
are no freshmaker nor automated rebuilds.
The branching structure does not really fit container needs. I maintain
the s2i Python container and I don’t really care whether it’s based on
Fedora 34 or 35 (both have Python 3.10 as the main one) so I’d ideally
have one branch for Python 3.10, one for 3.9 etc. instead of two
separated branches (f34, f35) where the Python is the same version.
https://pagure.io/ContainerSIG/container-sig/issue/12 (3 years old)
An example of unstable infrastructure: I’m not able to build an image
for three weeks now and a response I got is to report the problem
upstream: https://pagure.io/fedora-infrastructure/issue/10437
I know that nothing is 100% reliable but if you take a look at these
issues, there are a lot of them:
https://pagure.io/fedora-infrastructure/issues?search_pattern=container&a...
And it seems that the packages in the infrastructure are outdated a lot:
https://github.com/containerbuildsystem/atomic-reactor/issues/1742#issuec...
If we migrate our container images to some other registry (e.g. a common
fedora space on quay.io), we’ll be able to rebuild them after every
merged PR or every week, do chain rebuilds and push them to the registry
directly from Github CI. This will make our lives significantly easier,
and in turn our images better.
Are there any benefits to using Fedora infrastructure and Fedora
containers registry which I don’t see that we should consider?
Lumír