F36 Change: ostree native containers / CoreOS layering (System-Wide Change proposal)

[Ben Cotton]

https://fedoraproject.org/wiki/Changes/OstreeNativeContainer

== Summary ==

Enhance the (rpm-)ostree stack to natively support OCI/Docker
containers as a transport and delivery mechanism for operating system
content.

This is the basis of
https://github.com/coreos/enhancements/blob/main/os/coreos-layering.md

== Owner ==
* Name: [[User:walters| Colin Walters]]
* Email: [e-mail]

== Detailed Description ==

Having the Fedora ecosystem (from users to release engineering)
maintain tooling that operates on all three of "container images",
RPMs, and OSTree updates is a maintenance burden.

This proposes that:

* The ostree stack is enhanced to support
encapsulating/unencapsulating ostree commits as OCI/Docker images
(DONE)
* rpm-ostree is updated to consume this, while still supporting all
its current features (e.g. per-machine package layering) (DONE)
* We ship e.g. `quay.io/fedora/coreos:stable` and
`quay.io/fedora/silverblue:36` etc.
* We support '''deriving''' new user custom images from these images
* We enhance this tooling to
[https://github.com/ostreedev/ostree-rs-ext/issues/69 support
chunking]

For more details, please see:

* [https://github.com/coreos/enhancements/blob/main/os/coreos-layering.md
CoreOS layering enhancement]
* [https://coreos.github.io/rpm-ostree/container/ rpm-ostree container docs]
* [https://github.com/ostreedev/ostree-rs-ext/ ostree-rs-ext project]

Note that significant effort has been invested in ensuring
compatibility between what exists in ostree today and OCI/Docker
container image "encapsulation".  For example, we will continue to
reuse the GPG signature infrastructure on OSTree commits that exists
today - the ostree tooling knows how to verify the signature *inside*
the container image.  In the future, we will also likely invest in
container-native signatures.

== Benefit to Fedora ==

* Stronger focus on Docker/OCI as transport for operating system and
applications
* New ability to easily create derived operating system images "server side"
* More benefit from e.g. work on container deltas

== Scope ==
* Proposal owners: Lots of detailed items listed in the rpm-ostree/CoreOS docs.
* Other developers: The "other" here is vague, but certainly
developing this so far has needed cooperation with e.g. the
containers/ organization etc.

* Release engineering: https://pagure.io/releng/issue/10399
* Policies and guidelines: N/A (not needed for this Change)
* Trademark approval: N/A (not needed for this Change)
* Alignment with Objectives: No

== Upgrade/compatibility impact ==

Each individual edition/spin would need to choose when and how to make
a cutover to containers as a transport.  The Fedora OSTree repository
would continue to be maintained until that is finished.

== How To Test ==

See the examples under https://coreos.github.io/rpm-ostree/container/

== User Experience ==

Users of rpm-ostree systems will primarily interact with container images.

== Dependencies ==

Release engineering.

== Contingency Plan ==

* Contingency mechanism: Continue to ship updates via baseline OSTree
<!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
* Blocks release? No

== Documentation ==

Already linked above to avoid duplicating it here.

[James Cassell]

On Tue, Nov 23, 2021, at 12:18 PM, Ben Cotton wrote:
> https://fedoraproject.org/wiki/Changes/OstreeNativeContainer
>
> == Summary ==
>
> Enhance the (rpm-)ostree stack to natively support OCI/Docker
> containers as a transport and delivery mechanism for operating system
> content.
>
> This is the basis of
> https://github.com/coreos/enhancements/blob/main/os/coreos-layering.md
>
> == Owner ==
> Name: [[User:walters| Colin Walters]]
> Email: [e-mail]
>
> == Detailed Description ==
>
> Having the Fedora ecosystem (from users to release engineering)
> maintain tooling that operates on all three of "container images",
> RPMs, and OSTree updates is a maintenance burden.
>
> This proposes that:
> The ostree stack is enhanced to support
>
> encapsulating/unencapsulating ostree commits as OCI/Docker images
> (DONE)
> rpm-ostree is updated to consume this, while still supporting all
>
> its current features (e.g. per-machine package layering) (DONE)
> We ship e.g. `quay.io/fedora/coreos:stable` and
>
> `quay.io/fedora/silverblue:36` etc.
> We support '''deriving''' new user custom images from these images
> We enhance this tooling to
>
> [https://github.com/ostreedev/ostree-rs-ext/issues/69 support
> chunking]
>
> For more details, please see:
> [https://github.com/coreos/enhancements/blob/main/os/coreos-layering.md
>
> CoreOS layering enhancement]
> [https://coreos.github.io/rpm-ostree/container/ rpm-ostree container docs]
> [https://github.com/ostreedev/ostree-rs-ext/ ostree-rs-ext project]
>
> Note that significant effort has been invested in ensuring
> compatibility between what exists in ostree today and OCI/Docker
> container image "encapsulation".  For example, we will continue to
> reuse the GPG signature infrastructure on OSTree commits that exists
> today - the ostree tooling knows how to verify the signature *inside*
> the container image.  In the future, we will also likely invest in
> container-native signatures.
>
> == Benefit to Fedora ==
> Stronger focus on Docker/OCI as transport for operating system and
>
> applications
> New ability to easily create derived operating system images "server side"
> More benefit from e.g. work on container deltas

Will things be slower than native ostree?

I've got no problem with the capability being added, but I do wonder, Why not rojig, aka native RPMs plus a special metadata RPM, aka jigdo for ostree? We've already got great RPM infrastructure. Plus we could get rid of countme.timer and instead ride the rpm-md connection like the RPM-based variants do.

V/r,
James Cassell

[Colin Walters]

On Tue, Nov 23, 2021, at 4:28 PM, James Cassell wrote:
> Will things be slower than native ostree?

The only thing that will be less efficient is wire transfer as of right now.  But we're going to be working on that.
> I've got no problem with the capability being added, but I do wonder, 
> Why not rojig, aka native RPMs plus a special metadata RPM, aka jigdo 
> for ostree?

Ah yes.  I spent like 2 solid months working on all that only to have it not actually be used...

So backing up a level, really the question is:  What is the "center of gravity"?  Is it rpms, or ostree, or containers?  This Change isn't answering that question decisively, but it is a large move closer to OCI/Docker containers.
> We've already got great RPM infrastructure. Plus we could 
> get rid of countme.timer and instead ride the rpm-md connection like 
> the RPM-based variants do.

Mmm, I don't see a big problem with having a distinct rpm-ostree-countme.timer, it's not a lot of code.

But your reply is so far skipping over really the biggest change: making it a first class operation to build *derived* images using any Docker/OCI build system.  For a lot of people that will be Dockerfile, but if you follow the CoreOS proposal we are actively discussing a more declarative/intelligent frontend.

[Robin Lee]

On Wed, Nov 24, 2021 at 1:20 AM Ben Cotton  wrote:
> https://fedoraproject.org/wiki/Changes/OstreeNativeContainer
>
> == Summary ==
>
> Enhance the (rpm-)ostree stack to natively support OCI/Docker
> containers as a transport and delivery mechanism for operating system
> content.
>
> This is the basis of
> https://github.com/coreos/enhancements/blob/main/os/coreos-layering.md
>
> == Owner ==
> Name: [[User:walters| Colin Walters]]
> Email: [e-mail]
>
> == Detailed Description ==
>
> Having the Fedora ecosystem (from users to release engineering)
> maintain tooling that operates on all three of "container images",
> RPMs, and OSTree updates is a maintenance burden.
>
> This proposes that:
> The ostree stack is enhanced to support
>
> encapsulating/unencapsulating ostree commits as OCI/Docker images
> (DONE)
> rpm-ostree is updated to consume this, while still supporting all
>
> its current features (e.g. per-machine package layering) (DONE)
> We ship e.g. `quay.io/fedora/coreos:stable` and
>
> `quay.io/fedora/silverblue:36` etc.
> We support '''deriving''' new user custom images from these images
> We enhance this tooling to
>
> [https://github.com/ostreedev/ostree-rs-ext/issues/69 support
> chunking]
>
> For more details, please see:
> [https://github.com/coreos/enhancements/blob/main/os/coreos-layering.md
>
> CoreOS layering enhancement]
> [https://coreos.github.io/rpm-ostree/container/ rpm-ostree container docs]

This is a great enhancement for the OSTree world. People can use Docker hub
or quay.io as an OSTree hub to maintain different ostree-based Linux
distributions for users to rebase and try.

This function is unrelated to 'rpm' but unfortunately provided by 'rpm-ostree'.
Maybe we should provide another standalone tool so non-rpm/dnf-based
distributions can be easier to deploy.

-robin

[Colin Walters]

On Wed, Nov 24, 2021, at 1:26 AM, Robin Lee wrote:
> This function is unrelated to 'rpm' but unfortunately provided by 'rpm-ostree'.
> Maybe we should provide another standalone tool so non-rpm/dnf-based
> distributions can be easier to deploy.

No, all the "ostree-container" logic lives in
https://github.com/ostreedev/ostree-rs-ext/
which as with the rest of ostree (and like podman/docker), has no dependency on rpm/dpkg/whatever and never will.  You can put whatever you want in it and build however you want.

If you look at the PR to rpm-ostree which added this functionality (this all exists today btw!)
https://github.com/coreos/rpm-ostree/pull/3139/files#diff-2e9d962a0832160594...
you can see how rpm-ostree just picked up the ostree-ext changes as a library.

(While this is only somewhat related to your point, I think it is very interesting and important to note that a big part of this proposal is that we now easily expose the ability to inject non-RPM content into derived images and boot and upgrade them via rpm-ostree, so that's a "less RPM involved" angle)

ostree-ext also exposes a CLI, but see also https://github.com/ostreedev/ostree/issues/2480

That said, it is certainly true that ostree is more of a low level library and most of our effort is on rpm-ostree.  Eventually perhaps it may make sense to have at least a polished shared CLI/daemon code upstream in ostree that gets shared more across distributions, but there's nontrivial logistical/political/technical aspects to that.  It's perhaps somewhat analogous to librpm versus dnf/zypper.

Also omitted from this is that my job also involves making all this work really well in OpenShift/Kubernetes in addition to this change (which is basically just talking about "single node"/"baseos" functionality that applies to non-Kubernetes Fedora cases too).   So juggling those two has taken most of my personal time versus polishing the "ostree but not rpm-ostree" case.

[Neal Gompa]

On Tue, Nov 23, 2021 at 11:23 PM Ben Cotton  wrote:
> https://fedoraproject.org/wiki/Changes/OstreeNativeContainer
>
> == Summary ==
>
> Enhance the (rpm-)ostree stack to natively support OCI/Docker
> containers as a transport and delivery mechanism for operating system
> content.
>
> This is the basis of
> https://github.com/coreos/enhancements/blob/main/os/coreos-layering.md
>
> == Owner ==
> Name: [[User:walters| Colin Walters]]
> Email: [e-mail]
>
> == Detailed Description ==
>
> Having the Fedora ecosystem (from users to release engineering)
> maintain tooling that operates on all three of "container images",
> RPMs, and OSTree updates is a maintenance burden.
>
> This proposes that:
> The ostree stack is enhanced to support
>
> encapsulating/unencapsulating ostree commits as OCI/Docker images
> (DONE)
> rpm-ostree is updated to consume this, while still supporting all
>
> its current features (e.g. per-machine package layering) (DONE)
> We ship e.g. `quay.io/fedora/coreos:stable` and
>
> `quay.io/fedora/silverblue:36` etc.
> We support '''deriving''' new user custom images from these images
> We enhance this tooling to
>
> [https://github.com/ostreedev/ostree-rs-ext/issues/69 support
> chunking]
>
> For more details, please see:
> [https://github.com/coreos/enhancements/blob/main/os/coreos-layering.md
>
> CoreOS layering enhancement]
> [https://coreos.github.io/rpm-ostree/container/ rpm-ostree container docs]
> [https://github.com/ostreedev/ostree-rs-ext/ ostree-rs-ext project]
>
> Note that significant effort has been invested in ensuring
> compatibility between what exists in ostree today and OCI/Docker
> container image "encapsulation".  For example, we will continue to
> reuse the GPG signature infrastructure on OSTree commits that exists
> today - the ostree tooling knows how to verify the signature *inside*
> the container image.  In the future, we will also likely invest in
> container-native signatures.
>
> == Benefit to Fedora ==
> Stronger focus on Docker/OCI as transport for operating system and
>
> applications
> New ability to easily create derived operating system images "server side"
> More benefit from e.g. work on container deltas
>
> == Scope ==
> Proposal owners: Lots of detailed items listed in the rpm-ostree/CoreOS docs.
> Other developers: The "other" here is vague, but certainly
>
> developing this so far has needed cooperation with e.g. the
> containers/ organization etc.
> Release engineering: https://pagure.io/releng/issue/10399
> Policies and guidelines: N/A (not needed for this Change)
> Trademark approval: N/A (not needed for this Change)
> Alignment with Objectives: No
>
> == Upgrade/compatibility impact ==
>
> Each individual edition/spin would need to choose when and how to make
> a cutover to containers as a transport.  The Fedora OSTree repository
> would continue to be maintained until that is finished.
>
> == How To Test ==
>
> See the examples under https://coreos.github.io/rpm-ostree/container/
>
> == User Experience ==
>
> Users of rpm-ostree systems will primarily interact with container images.
>
> == Dependencies ==
>
> Release engineering.
>
> == Contingency Plan ==
> Contingency mechanism: Continue to ship updates via baseline OSTree
>
> <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
> Blocks release? No
>
> == Documentation ==
>
> Already linked above to avoid duplicating it here.

A couple of things from my perspective:

* I would like to see how this would enable CoreOS releases to go
through Bodhi instead of hiding off to the side as it does now. That
also means using registry.fedoraproject.org as the primary registry
that replicates to quay.io.
* How does this affect Kinoite, Silverblue, and CoreOS releases? Are
they changing immediately to this mechanism?

[Brian (bex) Exelbierd]

On Wed, Dec 1, 2021 at 5:34 PM Neal Gompa  wrote:
> On Tue, Nov 23, 2021 at 11:23 PM Ben Cotton  wrote:
> > https://fedoraproject.org/wiki/Changes/OstreeNativeContainer
> >
> > == Summary ==
> >
> > Enhance the (rpm-)ostree stack to natively support OCI/Docker
> > containers as a transport and delivery mechanism for operating system
> > content.
> >
> > This is the basis of
> > https://github.com/coreos/enhancements/blob/main/os/coreos-layering.md
> >
> > == Owner ==
> > Name: [[User:walters| Colin Walters]]
> > Email: [e-mail]
> >
> > == Detailed Description ==
> >
> > Having the Fedora ecosystem (from users to release engineering)
> > maintain tooling that operates on all three of "container images",
> > RPMs, and OSTree updates is a maintenance burden.
> >
> > This proposes that:
> > The ostree stack is enhanced to support
> >
> > encapsulating/unencapsulating ostree commits as OCI/Docker images
> > (DONE)
> > rpm-ostree is updated to consume this, while still supporting all
> >
> > its current features (e.g. per-machine package layering) (DONE)
> > We ship e.g. `quay.io/fedora/coreos:stable`
>
> http://quay.io/fedora/coreos:stable and
> > `quay.io/fedora/silverblue:36` http://quay.io/fedora/silverblue:36
>
> etc.
> > We support '''deriving''' new user custom images from these images
> > We enhance this tooling to
> >
> > [https://github.com/ostreedev/ostree-rs-ext/issues/69 support
> > chunking]
> >
> > For more details, please see:
> > [
>
> https://github.com/coreos/enhancements/blob/main/os/coreos-layering.md
> > CoreOS layering enhancement]
> > [https://coreos.github.io/rpm-ostree/container/ rpm-ostree container
>
> docs]
> > [https://github.com/ostreedev/ostree-rs-ext/ ostree-rs-ext project]
> >
> > Note that significant effort has been invested in ensuring
> > compatibility between what exists in ostree today and OCI/Docker
> > container image "encapsulation".  For example, we will continue to
> > reuse the GPG signature infrastructure on OSTree commits that exists
> > today - the ostree tooling knows how to verify the signature *inside*
> > the container image.  In the future, we will also likely invest in
> > container-native signatures.
> >
> > == Benefit to Fedora ==
> > Stronger focus on Docker/OCI as transport for operating system and
> >
> > applications
> > New ability to easily create derived operating system images "server
>
> side"
> > More benefit from e.g. work on container deltas
> >
> > == Scope ==
> > Proposal owners: Lots of detailed items listed in the
>
> rpm-ostree/CoreOS docs.
> > Other developers: The "other" here is vague, but certainly
> >
> > developing this so far has needed cooperation with e.g. the
> > containers/ organization etc.
> > Release engineering: https://pagure.io/releng/issue/10399
> > Policies and guidelines: N/A (not needed for this Change)
> > Trademark approval: N/A (not needed for this Change)
> > Alignment with Objectives: No
> >
> > == Upgrade/compatibility impact ==
> >
> > Each individual edition/spin would need to choose when and how to make
> > a cutover to containers as a transport.  The Fedora OSTree repository
> > would continue to be maintained until that is finished.
> >
> > == How To Test ==
> >
> > See the examples under https://coreos.github.io/rpm-ostree/container/
> >
> > == User Experience ==
> >
> > Users of rpm-ostree systems will primarily interact with container
>
> images.
> > == Dependencies ==
> >
> > Release engineering.
> >
> > == Contingency Plan ==
> > Contingency mechanism: Continue to ship updates via baseline OSTree
> >
> > <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
> > Blocks release? No
> >
> > == Documentation ==
> >
> > Already linked above to avoid duplicating it here.
>
> A couple of things from my perspective:
> I would like to see how this would enable CoreOS releases to go
>
> through Bodhi instead of hiding off to the side as it does now. That
> also means using registry.fedoraproject.org as the primary registry
> that replicates to quay.io.
> How does this affect Kinoite, Silverblue, and CoreOS releases? Are
>
> they changing immediately to this mechanism?

Also, how does this intersect with Fedora IoT and their desire to move to
Imagebuilder?

Overall this looks exciting to me and I love the idea that I can write an
Dockerfile (or new thing) file to declare my layering and have it all built
(or rebuilt) via existing container tooling.

Huzzah!

regards,

bex
> --
> 真実はいつも一つ！/ Always, there's only one truth!
> _______________________________________________
> devel mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]
> Do not reply to spam on the list, report it:
> https://pagure.io/fedora-infrastructure

[Colin Walters]

On Wed, Dec 1, 2021, at 12:32 PM, Brian (bex) Exelbierd wrote:
> Also, how does this intersect with Fedora IoT and their desire to move 
> to Imagebuilder?

I'd love for one of the team members there to comment on this.  From the
FCOS side I can say there's interest in aligning CoreOS and Image Builder
where possible, but it's not going to happen overnight.   There's a fractal
world of complexity there that could be its own whole thread.

I think as a short version though, all this functionality will land in
(rpm-)ostree and be accessible to Image Builder should they choose to
use it.

Both Image Builder today and coreos-assembler are focused on building
what we'd after this change think of a *base image* (both container image and
disk images).

This notion of a container style derived image is more of a new thing.

And from the CoreOS side I think I'd say it is not a goal to expose
coreos-assembler to users as part of this.  This proposal currently
calls for supporting users to create their own derived images using
any *container* build system they want.  (Which, since I'm being
realistic, Dockerfile is sadly the lowest common denominator here)

That said, creating derived container images this way also is
going to motivate making it easy to create disk images (e.g. ISO, qcow2, AMI)
that take ostree-container images as input as a user-facing productized
thing.   It seems to me that Image Builder is a natural place
for that to happen.

I hope we can mostly avoid this need apart from special cases.
For many users for example, managing (building, paying for, versioniong,
garbage collecting) their own AMIs in EC2 isn't desirable.  Instead
being able to just build, version and test their own FCOS-derived
container images is much easier (and portable across clouds).
> Overall this looks exciting to me and I love the idea that I can write 
> an Dockerfile (or new thing) file to declare my layering and have it 
> all built (or rebuilt) via existing container tooling.

Thanks!

[Colin Walters]

On Wed, Dec 1, 2021, at 12:32 PM, Brian (bex) Exelbierd wrote:
> Also, how does this intersect with Fedora IoT and their desire to move 
> to Imagebuilder?

Actually sorry you asked a specific question about IoT and I went off on a larger tangent.

The simple answer is: everything in the (rpm-)ostree stack that works today that IoT uses will continue to work and be maintained for quite some time to come.

From the build side to the client side.  We're *adding* the ability client side to pull container images, not also *removing* the ability to fetch via native ostree.  All that is still tested on every single pull request.

In particular, the very network-efficient ostree deltas on the wire.  But, longer term as part of this I plan to pour my efforts into helping the container stack with deltas, etc.

[Colin Walters]

On Wed, Dec 1, 2021, at 11:33 AM, Neal Gompa wrote:
> A couple of things from my perspective:
> I would like to see how this would enable CoreOS releases to go
>
> through Bodhi

To me, a notable chunk of the value of how we're doing FCOS is that
our build, test and release processes are tightly intertwined
and maintained by the same team.

It's really part of having an "image based" delivery flow - we're
responsible for that "one thing", instead of a distributed amorphous
notion of responsiblity that happens with packages.

We have deeply invested in a fully containerized build and test flow.
Our build and test code is all in a single big coreos-assembler
container.

We have a core principle that the pipeline is basically just scripting
what exists in coreos-assembler (it doesn't have significant nontrivial
logic itself), so it's easy to replicate locally.

That's...just not true of most other things in Fedora, and the value
of wedging Bodhi (particularly the karma aspect) into the mix isn't
clear to me.

I would say also that I personally am *very* strongly of the
opinion that as a general rule, bespoke imperative web apps/APIs
that we require humans to touch are generally a bad idea.  
Instead, the build and delivery pipeline should be as "git-ops" as possible.

We live in this world - see e.g.
https://github.com/coreos/fedora-coreos-config/blob/testing-devel/manifest-l...
which defines the content that goes into FCOS.  It's JSON stored in git,
and bumping it runs through CI.  (I would like to consider changing
things to be auto-pull-request based, just like Dependabot e.g. so
that things are even more obvious; we didn't do that initially
out of concerns for PR noise, but lately I'm leaning towards it)

There's no database or web API involved, it's just JSON stored in git.

This is also similar to e.g.
https://github.com/NixOS/nixpkgs/pull/136462
Why should it be any harder than that?

But in the end of course, to really answer your question, this change
could perhaps indeed make it easier to push ostree-containers through
Bodhi.

And to be fully clear, a lot of what I wrote above applies to FCOS,
but not necessarily to the other editions that use (rpm-)ostree.
> That
> also means using registry.fedoraproject.org as the primary registry
> that replicates to quay.io.

Sure, I have no strong opinions there.  That's already touched on in
https://pagure.io/releng/issue/10399
> How does this affect Kinoite, Silverblue, and CoreOS releases? Are
>
> they changing immediately to this mechanism?

I can't say for sure around time frames and specific editions.  But
*ideally* for f36:

- Client-side capability ships (done! You can try this today!  https://coreos.github.io/rpm-ostree/container/ )
- Shipping e.g. quay.io/fedora/coreos:testing-devel with GPG signatures inside (we're keeping ostree GPG signatures even inside a container flow because we care about the OS!  xref https://github.com/ostreedev/ostree-rs-ext/pull/76 )
- We get sufficient experience from now till f36 to mark this capability stable in rpm-ostree

After that, well it gets fuzzy.  I have a lot of work to do on the client end, and I personally plan to drive this from the FCOS side.

Where things get quite interesting of course is that suddenly container-style derivation becomes not just a possibility but an obvious thing to do.  IOW, it makes it more clearly possible that
the Fedora Silverblue build is the equivalent of:
```
FROM quay.io/fedora/coreos:stable
RUN yum -y install gnome-shell ...
```

(That said, I would like to keep all the build-side intelligence we have on rpm-ostree that is part of having a declarative input instead of Dockerfile, i.e. we'd still use treefiles etc.  This is also related to https://github.com/coreos/rpm-ostree/issues/2326 )

And if we do *that*, then other editions suddenly also get to use all the investment we've made in the FCOS CI too, and it changes how we think about all of this.  (Not to mention being able to use Ignition too for desktops, which is its own quite interesting aspect)