https://fedoraproject.org/wiki/Changes/OstreeNativeContainer
== Summary ==
Enhance the (rpm-)ostree stack to natively support OCI/Docker containers as a transport and delivery mechanism for operating system content.
This is the basis of https://github.com/coreos/enhancements/blob/main/os/coreos-layering.md
== Owner == * Name: [[User:walters| Colin Walters]] * Email: walters@verbum.org
== Detailed Description ==
Having the Fedora ecosystem (from users to release engineering) maintain tooling that operates on all three of "container images", RPMs, and OSTree updates is a maintenance burden.
This proposes that:
* The ostree stack is enhanced to support encapsulating/unencapsulating ostree commits as OCI/Docker images (DONE) * rpm-ostree is updated to consume this, while still supporting all its current features (e.g. per-machine package layering) (DONE) * We ship e.g. `quay.io/fedora/coreos:stable` and `quay.io/fedora/silverblue:36` etc. * We support '''deriving''' new user custom images from these images * We enhance this tooling to [https://github.com/ostreedev/ostree-rs-ext/issues/69 support chunking]
For more details, please see:
* [https://github.com/coreos/enhancements/blob/main/os/coreos-layering.md CoreOS layering enhancement] * [https://coreos.github.io/rpm-ostree/container/ rpm-ostree container docs] * [https://github.com/ostreedev/ostree-rs-ext/ ostree-rs-ext project]
Note that significant effort has been invested in ensuring compatibility between what exists in ostree today and OCI/Docker container image "encapsulation". For example, we will continue to reuse the GPG signature infrastructure on OSTree commits that exists today - the ostree tooling knows how to verify the signature *inside* the container image. In the future, we will also likely invest in container-native signatures.
== Benefit to Fedora ==
* Stronger focus on Docker/OCI as transport for operating system and applications * New ability to easily create derived operating system images "server side" * More benefit from e.g. work on container deltas
== Scope == * Proposal owners: Lots of detailed items listed in the rpm-ostree/CoreOS docs. * Other developers: The "other" here is vague, but certainly developing this so far has needed cooperation with e.g. the containers/ organization etc.
* Release engineering: https://pagure.io/releng/issue/10399 * Policies and guidelines: N/A (not needed for this Change) * Trademark approval: N/A (not needed for this Change) * Alignment with Objectives: No
== Upgrade/compatibility impact ==
Each individual edition/spin would need to choose when and how to make a cutover to containers as a transport. The Fedora OSTree repository would continue to be maintained until that is finished.
== How To Test ==
See the examples under https://coreos.github.io/rpm-ostree/container/
== User Experience ==
Users of rpm-ostree systems will primarily interact with container images.
== Dependencies ==
Release engineering.
== Contingency Plan ==
* Contingency mechanism: Continue to ship updates via baseline OSTree <!-- REQUIRED FOR SYSTEM WIDE CHANGES --> * Blocks release? No
== Documentation ==
Already linked above to avoid duplicating it here.
On Tue, Nov 23, 2021, at 12:18 PM, Ben Cotton wrote:
https://fedoraproject.org/wiki/Changes/OstreeNativeContainer
== Summary ==
Enhance the (rpm-)ostree stack to natively support OCI/Docker containers as a transport and delivery mechanism for operating system content.
This is the basis of https://github.com/coreos/enhancements/blob/main/os/coreos-layering.md
== Owner ==
- Name: [[User:walters| Colin Walters]]
- Email: walters@verbum.org
== Detailed Description ==
Having the Fedora ecosystem (from users to release engineering) maintain tooling that operates on all three of "container images", RPMs, and OSTree updates is a maintenance burden.
This proposes that:
- The ostree stack is enhanced to support
encapsulating/unencapsulating ostree commits as OCI/Docker images (DONE)
- rpm-ostree is updated to consume this, while still supporting all
its current features (e.g. per-machine package layering) (DONE)
- We ship e.g. `quay.io/fedora/coreos:stable` and
`quay.io/fedora/silverblue:36` etc.
- We support '''deriving''' new user custom images from these images
- We enhance this tooling to
[https://github.com/ostreedev/ostree-rs-ext/issues/69 support chunking]
For more details, please see:
CoreOS layering enhancement]
- [https://coreos.github.io/rpm-ostree/container/ rpm-ostree container docs]
- [https://github.com/ostreedev/ostree-rs-ext/ ostree-rs-ext project]
Note that significant effort has been invested in ensuring compatibility between what exists in ostree today and OCI/Docker container image "encapsulation". For example, we will continue to reuse the GPG signature infrastructure on OSTree commits that exists today - the ostree tooling knows how to verify the signature *inside* the container image. In the future, we will also likely invest in container-native signatures.
== Benefit to Fedora ==
- Stronger focus on Docker/OCI as transport for operating system and
applications
- New ability to easily create derived operating system images "server side"
- More benefit from e.g. work on container deltas
Will things be slower than native ostree?
I've got no problem with the capability being added, but I do wonder, Why not rojig, aka native RPMs plus a special metadata RPM, aka jigdo for ostree? We've already got great RPM infrastructure. Plus we could get rid of countme.timer and instead ride the rpm-md connection like the RPM-based variants do.
V/r, James Cassell
On Tue, Nov 23, 2021, at 4:28 PM, James Cassell wrote:
Will things be slower than native ostree?
The only thing that will be less efficient is wire transfer as of right now. But we're going to be working on that.
I've got no problem with the capability being added, but I do wonder, Why not rojig, aka native RPMs plus a special metadata RPM, aka jigdo for ostree?
Ah yes. I spent like 2 solid months working on all that only to have it not actually be used...
So backing up a level, really the question is: What is the "center of gravity"? Is it rpms, or ostree, or containers? This Change isn't answering that question decisively, but it is a large move closer to OCI/Docker containers.
We've already got great RPM infrastructure. Plus we could get rid of countme.timer and instead ride the rpm-md connection like the RPM-based variants do.
Mmm, I don't see a big problem with having a distinct rpm-ostree-countme.timer, it's not a lot of code.
But your reply is so far skipping over really the biggest change: making it a first class operation to build *derived* images using any Docker/OCI build system. For a lot of people that will be Dockerfile, but if you follow the CoreOS proposal we are actively discussing a more declarative/intelligent frontend.
On Wed, Nov 24, 2021 at 1:20 AM Ben Cotton bcotton@redhat.com wrote:
https://fedoraproject.org/wiki/Changes/OstreeNativeContainer
== Summary ==
Enhance the (rpm-)ostree stack to natively support OCI/Docker containers as a transport and delivery mechanism for operating system content.
This is the basis of https://github.com/coreos/enhancements/blob/main/os/coreos-layering.md
== Owner ==
- Name: [[User:walters| Colin Walters]]
- Email: walters@verbum.org
== Detailed Description ==
Having the Fedora ecosystem (from users to release engineering) maintain tooling that operates on all three of "container images", RPMs, and OSTree updates is a maintenance burden.
This proposes that:
- The ostree stack is enhanced to support
encapsulating/unencapsulating ostree commits as OCI/Docker images (DONE)
- rpm-ostree is updated to consume this, while still supporting all
its current features (e.g. per-machine package layering) (DONE)
- We ship e.g. `quay.io/fedora/coreos:stable` and
`quay.io/fedora/silverblue:36` etc.
- We support '''deriving''' new user custom images from these images
- We enhance this tooling to
[https://github.com/ostreedev/ostree-rs-ext/issues/69 support chunking]
For more details, please see:
CoreOS layering enhancement]
- [https://coreos.github.io/rpm-ostree/container/ rpm-ostree container docs]
This is a great enhancement for the OSTree world. People can use Docker hub or quay.io as an OSTree hub to maintain different ostree-based Linux distributions for users to rebase and try.
This function is unrelated to 'rpm' but unfortunately provided by 'rpm-ostree'. Maybe we should provide another standalone tool so non-rpm/dnf-based distributions can be easier to deploy.
-robin
On Wed, Nov 24, 2021, at 1:26 AM, Robin Lee wrote:
This function is unrelated to 'rpm' but unfortunately provided by 'rpm-ostree'. Maybe we should provide another standalone tool so non-rpm/dnf-based distributions can be easier to deploy.
No, all the "ostree-container" logic lives in https://github.com/ostreedev/ostree-rs-ext/ which as with the rest of ostree (and like podman/docker), has no dependency on rpm/dpkg/whatever and never will. You can put whatever you want in it and build however you want.
If you look at the PR to rpm-ostree which added this functionality (this all exists today btw!) https://github.com/coreos/rpm-ostree/pull/3139/files#diff-2e9d962a0832160594... you can see how rpm-ostree just picked up the ostree-ext changes as a library.
(While this is only somewhat related to your point, I think it is very interesting and important to note that a big part of this proposal is that we now easily expose the ability to inject non-RPM content into derived images and boot and upgrade them via rpm-ostree, so that's a "less RPM involved" angle)
ostree-ext also exposes a CLI, but see also https://github.com/ostreedev/ostree/issues/2480
That said, it is certainly true that ostree is more of a low level library and most of our effort is on rpm-ostree. Eventually perhaps it may make sense to have at least a polished shared CLI/daemon code upstream in ostree that gets shared more across distributions, but there's nontrivial logistical/political/technical aspects to that. It's perhaps somewhat analogous to librpm versus dnf/zypper.
Also omitted from this is that my job also involves making all this work really well in OpenShift/Kubernetes in addition to this change (which is basically just talking about "single node"/"baseos" functionality that applies to non-Kubernetes Fedora cases too). So juggling those two has taken most of my personal time versus polishing the "ostree but not rpm-ostree" case.
On Tue, Nov 23, 2021 at 11:23 PM Ben Cotton bcotton@redhat.com wrote:
https://fedoraproject.org/wiki/Changes/OstreeNativeContainer
== Summary ==
Enhance the (rpm-)ostree stack to natively support OCI/Docker containers as a transport and delivery mechanism for operating system content.
This is the basis of https://github.com/coreos/enhancements/blob/main/os/coreos-layering.md
== Owner ==
- Name: [[User:walters| Colin Walters]]
- Email: walters@verbum.org
== Detailed Description ==
Having the Fedora ecosystem (from users to release engineering) maintain tooling that operates on all three of "container images", RPMs, and OSTree updates is a maintenance burden.
This proposes that:
- The ostree stack is enhanced to support
encapsulating/unencapsulating ostree commits as OCI/Docker images (DONE)
- rpm-ostree is updated to consume this, while still supporting all
its current features (e.g. per-machine package layering) (DONE)
- We ship e.g. `quay.io/fedora/coreos:stable` and
`quay.io/fedora/silverblue:36` etc.
- We support '''deriving''' new user custom images from these images
- We enhance this tooling to
[https://github.com/ostreedev/ostree-rs-ext/issues/69 support chunking]
For more details, please see:
CoreOS layering enhancement]
- [https://coreos.github.io/rpm-ostree/container/ rpm-ostree container docs]
- [https://github.com/ostreedev/ostree-rs-ext/ ostree-rs-ext project]
Note that significant effort has been invested in ensuring compatibility between what exists in ostree today and OCI/Docker container image "encapsulation". For example, we will continue to reuse the GPG signature infrastructure on OSTree commits that exists today - the ostree tooling knows how to verify the signature *inside* the container image. In the future, we will also likely invest in container-native signatures.
== Benefit to Fedora ==
- Stronger focus on Docker/OCI as transport for operating system and
applications
- New ability to easily create derived operating system images "server side"
- More benefit from e.g. work on container deltas
== Scope ==
- Proposal owners: Lots of detailed items listed in the rpm-ostree/CoreOS docs.
- Other developers: The "other" here is vague, but certainly
developing this so far has needed cooperation with e.g. the containers/ organization etc.
- Release engineering: https://pagure.io/releng/issue/10399
- Policies and guidelines: N/A (not needed for this Change)
- Trademark approval: N/A (not needed for this Change)
- Alignment with Objectives: No
== Upgrade/compatibility impact ==
Each individual edition/spin would need to choose when and how to make a cutover to containers as a transport. The Fedora OSTree repository would continue to be maintained until that is finished.
== How To Test ==
See the examples under https://coreos.github.io/rpm-ostree/container/
== User Experience ==
Users of rpm-ostree systems will primarily interact with container images.
== Dependencies ==
Release engineering.
== Contingency Plan ==
- Contingency mechanism: Continue to ship updates via baseline OSTree
<!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
- Blocks release? No
== Documentation ==
Already linked above to avoid duplicating it here.
A couple of things from my perspective:
* I would like to see how this would enable CoreOS releases to go through Bodhi instead of hiding off to the side as it does now. That also means using registry.fedoraproject.org as the primary registry that replicates to quay.io. * How does this affect Kinoite, Silverblue, and CoreOS releases? Are they changing immediately to this mechanism?
On Wed, Dec 1, 2021 at 5:34 PM Neal Gompa ngompa13@gmail.com wrote:
On Tue, Nov 23, 2021 at 11:23 PM Ben Cotton bcotton@redhat.com wrote:
https://fedoraproject.org/wiki/Changes/OstreeNativeContainer
== Summary ==
Enhance the (rpm-)ostree stack to natively support OCI/Docker containers as a transport and delivery mechanism for operating system content.
This is the basis of https://github.com/coreos/enhancements/blob/main/os/coreos-layering.md
== Owner ==
- Name: [[User:walters| Colin Walters]]
- Email: walters@verbum.org
== Detailed Description ==
Having the Fedora ecosystem (from users to release engineering) maintain tooling that operates on all three of "container images", RPMs, and OSTree updates is a maintenance burden.
This proposes that:
- The ostree stack is enhanced to support
encapsulating/unencapsulating ostree commits as OCI/Docker images (DONE)
- rpm-ostree is updated to consume this, while still supporting all
its current features (e.g. per-machine package layering) (DONE)
- We ship e.g. `quay.io/fedora/coreos:stable`
http://quay.io/fedora/coreos:stable and
`quay.io/fedora/silverblue:36` http://quay.io/fedora/silverblue:36
etc.
- We support '''deriving''' new user custom images from these images
- We enhance this tooling to
[https://github.com/ostreedev/ostree-rs-ext/issues/69 support chunking]
For more details, please see:
- [
https://github.com/coreos/enhancements/blob/main/os/coreos-layering.md
CoreOS layering enhancement]
- [https://coreos.github.io/rpm-ostree/container/ rpm-ostree container
docs]
- [https://github.com/ostreedev/ostree-rs-ext/ ostree-rs-ext project]
Note that significant effort has been invested in ensuring compatibility between what exists in ostree today and OCI/Docker container image "encapsulation". For example, we will continue to reuse the GPG signature infrastructure on OSTree commits that exists today - the ostree tooling knows how to verify the signature *inside* the container image. In the future, we will also likely invest in container-native signatures.
== Benefit to Fedora ==
- Stronger focus on Docker/OCI as transport for operating system and
applications
- New ability to easily create derived operating system images "server
side"
- More benefit from e.g. work on container deltas
== Scope ==
- Proposal owners: Lots of detailed items listed in the
rpm-ostree/CoreOS docs.
- Other developers: The "other" here is vague, but certainly
developing this so far has needed cooperation with e.g. the containers/ organization etc.
- Release engineering: https://pagure.io/releng/issue/10399
- Policies and guidelines: N/A (not needed for this Change)
- Trademark approval: N/A (not needed for this Change)
- Alignment with Objectives: No
== Upgrade/compatibility impact ==
Each individual edition/spin would need to choose when and how to make a cutover to containers as a transport. The Fedora OSTree repository would continue to be maintained until that is finished.
== How To Test ==
See the examples under https://coreos.github.io/rpm-ostree/container/
== User Experience ==
Users of rpm-ostree systems will primarily interact with container
images.
== Dependencies ==
Release engineering.
== Contingency Plan ==
- Contingency mechanism: Continue to ship updates via baseline OSTree
<!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
- Blocks release? No
== Documentation ==
Already linked above to avoid duplicating it here.
A couple of things from my perspective:
- I would like to see how this would enable CoreOS releases to go
through Bodhi instead of hiding off to the side as it does now. That also means using registry.fedoraproject.org as the primary registry that replicates to quay.io.
- How does this affect Kinoite, Silverblue, and CoreOS releases? Are
they changing immediately to this mechanism?
Also, how does this intersect with Fedora IoT and their desire to move to Imagebuilder?
Overall this looks exciting to me and I love the idea that I can write an Dockerfile (or new thing) file to declare my layering and have it all built (or rebuilt) via existing container tooling.
Huzzah!
regards,
bex
-- 真実はいつも一つ!/ Always, there's only one truth! _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
On Wed, Dec 1, 2021, at 12:32 PM, Brian (bex) Exelbierd wrote:
Also, how does this intersect with Fedora IoT and their desire to move to Imagebuilder?
I'd love for one of the team members there to comment on this. From the FCOS side I can say there's interest in aligning CoreOS and Image Builder where possible, but it's not going to happen overnight. There's a fractal world of complexity there that could be its own whole thread.
I think as a short version though, all this functionality will land in (rpm-)ostree and be accessible to Image Builder should they choose to use it.
Both Image Builder today and coreos-assembler are focused on building what we'd after this change think of a *base image* (both container image and disk images).
This notion of a container style derived image is more of a new thing.
And from the CoreOS side I think I'd say it is not a goal to expose coreos-assembler to users as part of this. This proposal currently calls for supporting users to create their own derived images using any *container* build system they want. (Which, since I'm being realistic, Dockerfile is sadly the lowest common denominator here)
That said, creating derived container images this way also is going to motivate making it easy to create disk images (e.g. ISO, qcow2, AMI) that take ostree-container images as input as a user-facing productized thing. It seems to me that Image Builder is a natural place for that to happen.
I hope we can mostly avoid this need apart from special cases. For many users for example, managing (building, paying for, versioniong, garbage collecting) their own AMIs in EC2 isn't desirable. Instead being able to just build, version and test their own FCOS-derived container images is much easier (and portable across clouds).
Overall this looks exciting to me and I love the idea that I can write an Dockerfile (or new thing) file to declare my layering and have it all built (or rebuilt) via existing container tooling.
Thanks!
On Wed, Dec 1, 2021, at 12:32 PM, Brian (bex) Exelbierd wrote:
Also, how does this intersect with Fedora IoT and their desire to move to Imagebuilder?
Actually sorry you asked a specific question about IoT and I went off on a larger tangent.
The simple answer is: everything in the (rpm-)ostree stack that works today that IoT uses will continue to work and be maintained for quite some time to come.
From the build side to the client side. We're *adding* the ability client side to pull container images, not also *removing* the ability to fetch via native ostree. All that is still tested on every single pull request.
In particular, the very network-efficient ostree deltas on the wire. But, longer term as part of this I plan to pour my efforts into helping the container stack with deltas, etc.
On Wed, Dec 1, 2021, at 11:33 AM, Neal Gompa wrote:
A couple of things from my perspective:
- I would like to see how this would enable CoreOS releases to go
through Bodhi
To me, a notable chunk of the value of how we're doing FCOS is that our build, test and release processes are tightly intertwined and maintained by the same team.
It's really part of having an "image based" delivery flow - we're responsible for that "one thing", instead of a distributed amorphous notion of responsiblity that happens with packages.
We have deeply invested in a fully containerized build and test flow. Our build and test code is all in a single big coreos-assembler container.
We have a core principle that the pipeline is basically just scripting what exists in coreos-assembler (it doesn't have significant nontrivial logic itself), so it's easy to replicate locally.
That's...just not true of most other things in Fedora, and the value of wedging Bodhi (particularly the karma aspect) into the mix isn't clear to me.
I would say also that I personally am *very* strongly of the opinion that as a general rule, bespoke imperative web apps/APIs that we require humans to touch are generally a bad idea. Instead, the build and delivery pipeline should be as "git-ops" as possible.
We live in this world - see e.g. https://github.com/coreos/fedora-coreos-config/blob/testing-devel/manifest-l... which defines the content that goes into FCOS. It's JSON stored in git, and bumping it runs through CI. (I would like to consider changing things to be auto-pull-request based, just like Dependabot e.g. so that things are even more obvious; we didn't do that initially out of concerns for PR noise, but lately I'm leaning towards it)
There's no database or web API involved, it's just JSON stored in git.
This is also similar to e.g. https://github.com/NixOS/nixpkgs/pull/136462 Why should it be any harder than that?
But in the end of course, to really answer your question, this change could perhaps indeed make it easier to push ostree-containers through Bodhi.
And to be fully clear, a lot of what I wrote above applies to FCOS, but not necessarily to the other editions that use (rpm-)ostree.
That also means using registry.fedoraproject.org as the primary registry that replicates to quay.io.
Sure, I have no strong opinions there. That's already touched on in https://pagure.io/releng/issue/10399
- How does this affect Kinoite, Silverblue, and CoreOS releases? Are
they changing immediately to this mechanism?
I can't say for sure around time frames and specific editions. But *ideally* for f36:
- Client-side capability ships (done! You can try this today! https://coreos.github.io/rpm-ostree/container/ ) - Shipping e.g. quay.io/fedora/coreos:testing-devel with GPG signatures inside (we're keeping ostree GPG signatures even inside a container flow because we care about the OS! xref https://github.com/ostreedev/ostree-rs-ext/pull/76 ) - We get sufficient experience from now till f36 to mark this capability stable in rpm-ostree
After that, well it gets fuzzy. I have a lot of work to do on the client end, and I personally plan to drive this from the FCOS side.
Where things get quite interesting of course is that suddenly container-style derivation becomes not just a possibility but an obvious thing to do. IOW, it makes it more clearly possible that the Fedora Silverblue build is the equivalent of: ``` FROM quay.io/fedora/coreos:stable RUN yum -y install gnome-shell ... ```
(That said, I would like to keep all the build-side intelligence we have on rpm-ostree that is part of having a declarative input instead of Dockerfile, i.e. we'd still use treefiles etc. This is also related to https://github.com/coreos/rpm-ostree/issues/2326 )
And if we do *that*, then other editions suddenly also get to use all the investment we've made in the FCOS CI too, and it changes how we think about all of this. (Not to mention being able to use Ignition too for desktops, which is its own quite interesting aspect)