On Tue, Jan 11, 2022 at 01:02:59PM +0100, Lumír Balhar wrote:
Hello.
My name is Lumír and I maintain a few layered container images in
Fedora/RHEL/upstream related to Python and S2I (source to image) platform -
namely python3, s2i-core, and s2i-base.
TL;DR: We (maintainers of layered container images for languages, runtimes
and databases) are considering migrating our images from Fedora
infrastructure and Fedora containers registry somewhere else. Below are the
reasons for that. If you think it’s a bad idea, let us know why. If you are
a user of those images, let us know as well so we can stay in touch with
you.
Forgive me for my wording but I’m really sad about the current situation.
yeah, it is sad. ;(
I have to honestly admit that I’m disappointed about the lack of
attention
containers in Fedora are getting. Maintaining them is painful,
infrastructure is unstable and unreliable and there was literally no
movement forward in recent years. Let me summarize it.
Container SIG is inactive. See important issues opened for years with no
movement further:
https://pagure.io/ContainerSIG/container-sig/issues
Even a simple task like merging rawhide branch to f35 for example is far
from smooth because they are never the same, because manual changes are
required for each branch:
https://pagure.io/ContainerSIG/container-sig/issue/16 (3 years old, no
reply)
S2I container images depend on each other (python3 -> s2i-base -> s2i-core)
but there is no automation for chain rebuilds so if I want to update
s2i-core image (to fix a CVE or something like that) and all dependant
images, it takes weeks because I have to rebuild them one by one and let
them one by one pass through Bodhi updates until I can rebuild next one in
the queue. And it’s really frustrating given the insufficient stability of
the infra. There seems to be a way how to make this faster, but nobody
knows, how it works:
https://pagure.io/ContainerSIG/container-sig/issue/52
and
https://github.com/containerbuildsystem/atomic-reactor/issues/1730
The same applies also if the fix is done on the RPM level because there are
no freshmaker nor automated rebuilds.
The branching structure does not really fit container needs. I maintain the
s2i Python container and I don’t really care whether it’s based on Fedora 34
or 35 (both have Python 3.10 as the main one) so I’d ideally have one branch
for Python 3.10, one for 3.9 etc. instead of two separated branches (f34,
f35) where the Python is the same version.
https://pagure.io/ContainerSIG/container-sig/issue/12 (3 years old)
Yeah, I think all the above is kind of a consequence of there not being
a active containers SIG. Infrastructure and releng doesn't really have
cycles or interest in driving containers forward, so it needs some other
folks attention. ;(
An example of unstable infrastructure: I’m not able to build an image
for
three weeks now and a response I got is to report the problem upstream:
https://pagure.io/fedora-infrastructure/issue/10437
I know that nothing is 100% reliable but if you take a look at these issues,
there are a lot of them:
https://pagure.io/fedora-infrastructure/issues?search_pattern=container&a...
Do note that that issue was filed on December 21st.
Myself and much of the infrastructure team were off on holidays.
I only got back yesterday and there's a ton of things to do, but I did
try and look into this some.
The reason there's so many issues I think is because the entire pipeline
is just way overkill and complex for just building containers. Basically
we have to have koji, koji builders, and 2 openshift clusters (one for
x86 and one for aarch64) to build containers, they have all kinds of
fragile handoffs between each other as well.
To clarify, thats installed in the build container... so, _fedora_
packages are outdated. ;( I have seen atomic-reactor being old, but I've
been afraid to update it since I don't know the pipeline.
Co-maintainers of those packages in Fedora would sure help out.
If we migrate our container images to some other registry (e.g. a
common
fedora space on quay.io), we’ll be able to rebuild them after every merged
PR or every week, do chain rebuilds and push them to the registry directly
from Github CI. This will make our lives significantly easier, and in turn
our images better.
We are looking into dropping out registry and moving to quay.io
ourselves. ;)
Are there any benefits to using Fedora infrastructure and Fedora
containers
registry which I don’t see that we should consider?
I'm not sure... hopefully folks who know the pipeline will chime in
here.
kevin