F38 proposal: Ostree Native Container (Phase 2, stable) (System-Wide Change proposal)

[Ben Cotton]

https://fedoraproject.org/wiki/Changes/OstreeNativeContainerStable

This document represents a proposed Change. As part of the Changes
process, proposals are publicly announced in order to receive
community feedback. This proposal will only be implemented if approved
by the Fedora Engineering Steering Committee.


== Summary ==

Continue the work done in
https://fedoraproject.org/wiki/Changes/OstreeNativeContainer but in an
officially stable format, and expanded to cover more OSTree-based
editions.  This goes "all in" on being container-native and
significantly changes the technology and user emphasis.

== Owner ==

* Name: [[User:walters| Colin Walters]], [[User:jmarrero| Joseph
Marrero]], [[User:baude| Brent Baude]]
* Email: walters(a)verbum.org, jmarrero(a)fedoraproject.org, baude(a)fedoraproject.org


== Detailed Description ==

* rpm-ostree's functionality to
[https://coreos.github.io/rpm-ostree/container/ boot and upgrade from
Docker/OCI container images] is declared stable
* Rework Fedora editions and spins (CoreOS, IoT, Silverblue, Kinoite,
etc) that use ostree to instead deliver via Docker/OCI container
images
* `rpm-ostree` is reworked to gain strong CLI compatibility with
`yum/dnf` commands (cliwrap)
* The new container native functionality is exposed via `dnf image`
* (TBD: Rebranding of rpm-ostree itself to `dnf-image` or something else)
* Support and documentation for generating derived images
* Support in Anaconda and other tools
* Support for generating bootable images


A top level goal is that users should not need to know or understand
ostree (or rpm-ostree); they only need to know containers and most key
functionality is exposed via the `yum/dnf` frontend with a few
extensions.


=== Backwards compatibility ===

If you're a user of ostree today: no need to worry.  Everything that
works today in ostree and rpm-ostree will continue to work for years
to come (it's shipped in RHEL9). However, it's expected that most
users will find the new model sufficiently compelling to switch fairly
quickly.

=== Stable Bootable OCI ===

More information about this is available in
[https://coreos.github.io/rpm-ostree/container/ the rpm-ostree
documentation].  This Change highlights that this functionality is
planned to be officially stable.

=== Changing to OCI over the wire ===

Today Fedora has an OSTree repository that contains updates for
editions such as Fedora CoreOS and Fedora Silverblue.

These editions instead will become [https://github.com/opencontainers
OCI base images], available at e.g.

* `quay.io/fedora/fedora-coreos:stable`
([https://quay.io/repository/fedora/fedora-coreos this exists today]!)
* `quay.io/fedora/fedora-silverblue:38` (does not exist, but can soon)

An update will be shipped that will cause existing systems tracking
the production ostree branches to be switched to the corresponding
container image.  Concretely for example, a system running
`fedora:fedora/36/x86_64/silverblue` will execute `rpm-ostree rebase
ostree-remote-registry:fedora:quay.io/fedora/fedora-silverblue:36`.

Note that at the current time, Fedora does not sign container images.
This should change (likely to something based on
[cosign](https://github.com/containers/skopeo/pull/1701)), but in the
mean time, the ostree-container flow supports verifying the embedded
ostree-baesd GPG signatures and this will continue to be used.

The Fedora OSTree repository will become read-only and stop receiving updates.

Note that for Fedora CoreOS specifically, there is some outstanding
design discussion around the intersection with Cincinnati:
https://github.com/coreos/fedora-coreos-tracker/issues/1263

==== Wire efficiency ====

Today the container images generated by rpm-ostree are "chunked" in a
reproducible fashion; for more information on this, see

* https://github.com/ostreedev/ostree-rs-ext/issues/69
* https://coreos.github.io/rpm-ostree/container/

However, longer term (as that first issue notes) we do want to add
[https://github.com/containers/image/pull/902 container deltas].
Doing so will benefit non-host containers too.



=== dnf/yum CLI compatibility ===

rpm-ostree has a feature called
[cliwrap](https://coreos.github.io/rpm-ostree/cliwrap/) today.  A key
motivation here is that in the switch to using containers as the
frontend, the project name "rpm-ostree" (which is very literal) no
longer makes sense.  A further emphasis on "container native" will
strongly encourage writing e.g. a `Dockerfile` like this:


<pre>
FROM quay.io/fedora/fedora-silverblue:stable
RUN dnf -y install sway && ostree container commit
</pre>

*Today*, one can spell this as `RUN rpm-ostree install -y sway &&
ostree container commit` - but a toplevel goal here is to allow host
system customization using the same patterns and tools one uses to
build application containers, and to de-emphasize the "ostree backend"
aspect.

To complete this story on the client side, what is today called
"ostree native containers" will be exposed via a new `dnf image`
subcommand on the client system.

For example:


<pre>
$ dnf image rebase quay.io/examplecorp/customos:latest
</pre>

As well as other offline/"image based" functionality such as `dnf
image rollback` and `dnf image apply-live`.

However, as many people know, rpm-ostree does today support
"per-machine" package layering/overrides - and *that will continue to
work*.  So users are not *required* to do builds on a remote server
even in "image mode".

Concretely, it will also work to `sudo dnf -y install sway` inside a
root terminal on a Fedora Workstation for example.  This will still be
a transactional operation.

More: https://github.com/coreos/rpm-ostree/issues/2883

=== Rebranding of rpm-ostree ===

This is a fundamental change in technology *and* positioning for
rpm-ostree (and for the projects that use it).  The terminology of
"immutable" gets even fuzzier now (see also [this
blog](https://blog.verbum.org/2020/08/22/immutable-%E2%86%92-reprovisiona...).

It would make sense to rebrand the project due to this.  No decision
has been made on this, but the choices are:

- Keep as is
- Rebrand to dnf-image (or, reviving a past name, yum-image)
- Rebrand to something else

More: https://github.com/coreos/rpm-ostree/issues/405

=== Support for building derivative containers ===

Until now, the model for systems like Fedora CoreOS and Silverblue
etc. has included an emphasis on containerization, but avoiding
extensive customization of the host system.  This will be weakened
somewhat - it will be fully supported to build derivative operating
system  images that in turn contain software which itself should run
directly on the host system, and not in a container.

A key aspect of this is that by generating a custom build, one binds
together the code and configuration as a transactionally updatable
unit.

The role of things like Kickstart, cloud-init, and Ignition and other
system config management shrinks significantly.

=== Example 1: Kubernetes and kubelet ===

It is not supported upstream to run the kubelet in a container image
itself.  Today, for OpenShift 4 we embed the kubelet as part of the
CoreOS builds.  The [https://www.okd.io/ OKD] project
[https://github.com/openshift/okd-machine-os/blob/fa2248d332abc9926e68428d...
already installs kubelet during a container build] deriving from the
base `fedora-coreos` image.  It is possible that RHEL CoreOS will make
[https://github.com/openshift/os/issues/799 a similar split].

=== Example 2: Agent software ===

Many organizations require agents (cloud agents, security scanners,
antivirus) which are not ready to be containerized.  Often too, these
tools come in e.g. RPM form, and also expect to be configured (remote
logging server, etc.).  This tooling allows to `dnf install` these
*and* embed the configuration, generating a standard container image
which can then be booted as an atomic unit.

(Note that in the OpenShift context, we expect this to chain with the
first example - users instead derive from a `okd-node-base` image
which itself derives from `fedora-coreos`).

=== Example 3: Podman machine ===

The `[http://podman.io Podman]` project has a function called
`[https://github.com/containers/podman/blob/main/docs/source/markdown/podman-machine.1.md
machine]` where it creates a virtual machine and boots standard CoreOS
builds. Podman developers and users has long desired a custom FCOS
image that is focused on container deployment using Podman, including
certain QEMU packages that allow for cross-architecture development.
This approach will solve a series of problems including but not
limited to getting more timely release alignment between Podman and
FCOS.

We also have a user base that wants to customize the VM image based on
restrictive environments ... bring your own virtual machine image.

=== Support in Anaconda and other tools ===

Anaconda support for custom images is tracked in this bug:
https://bugzilla.redhat.com/show_bug.cgi?id=2125655

But, as part of this Change, it should also work to generate an
Anaconda ISO which allows *user choice* for whether to install in
"image mode" or "traditional".

=== Support for generating bootable images ===

Instead of using an existing disk image/ISO (Anaconda, AWS AMI, etc.)
and then doing an in-place update to a generated container image, in
many cases will want to generate disk images using their container
image as input.

For example, generating an ISO that embeds that container image as
input and will install it.

In this proposal, we plan to ship a container image as part of Fedora
CoreOS that will do this:
https://github.com/coreos/fedora-coreos-tracker/issues/1151

But, it also makes sense to have a tool that e.g. does this and
generates an Anaconda ISO (we will inherently be doing this as part of
generating ostree-based Anaconda ISOs already!) - this would just be
supporting that in a clear external way too.  One likely target for
this is [https://github.com/osbuild osbuild] and
[https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9/...
RHEL Image Builder].



== Feedback ==


== Benefit to Fedora ==

The need to understand ostree goes away for most users; they
understand RPMs and containers.  While retaining all the current
(rpm-)ostree features.

== Scope ==
* Proposal owners:
* Other developers: May affect e.g. gnome-software and similar tools
that talk to rpm-ostree.

* Release engineering: [https://pagure.io/releng/issue/11047]
* Policies and guidelines: None.
* Trademark approval: N/A (not needed for this Change)
* Alignment with Objectives: None

== Upgrade/compatibility impact ==

We will ship a systemd unit that transitions systems currently using
OSTree on the wire over to tracking the corresponding container
images.  Note again, all features of rpm-ostree continue to work!  For
example, if you have layered packages, that will continue to be
applied.

== How To Test ==

Build container images, boot them.

== User Experience ==

Users will not need to understand ostree (at least at a superficial
level), only container images.

== Dependencies ==

Needs improvements in Anaconda in some cases, also would like to
increase alignment with dnf.  Fetching container images uses `skopeo`,
maintained by containers team.

== Contingency Plan ==
* Contingency mechanism: Continue to ship things the way we ship them today
* Contingency deadline: Dunno
* Blocks release? No

== Documentation ==

https://coreos.github.io/rpm-ostree/container/ and
https://github.com/coreos/coreos-layering-examples etc.

== Release Notes ==


-- 
Ben Cotton
He / Him / His
Fedora Program Manager
Red Hat
TZ=America/Indiana/Indianapolis

[Ben Cotton]

On Thu, Oct 13, 2022 at 3:08 PM Ben Cotton <bcotton(a)redhat.com&gt; wrote:
>  == Contingency Plan ==
>  * Contingency mechanism: Continue to ship things the way we ship them today
>  * Contingency deadline: Dunno
>  * Blocks release? No 
I suggest the branch day (2023-02-07) as the contingency deadline here.

-- 
Ben Cotton
He / Him / His
Fedora Program Manager
Red Hat
TZ=America/Indiana/Indianapolis

[Colin Walters]

On Thu, Oct 13, 2022, at 3:08 PM, Ben Cotton wrote:
>  https://fedoraproject.org/wiki/Changes/OstreeNativeContainerStable
I know there's a lot going on here, so I put together
https://github.com/cgwalters/dnfimage-config
as a demonstration system to show this all works today.  (Though there's a lot left to
do)

To say this another way...I really, really wish I had a time machine to go back and
announce *this* instead of doing Fedora Atomic Host way back in the day.  If when Docker
had come out (before Kubernetes, before podman, before CoreOS) I wish I'd said
"hey this container stuff is cool, why don't we make bootable host operating
systems configurable via containers too").  Oh well, better late than never!

(And yes, we're not the first to this nowadays, but...first, this path gives a
seamless upgrade for all the existing ostree-based systems out there, and second, you
absolutely can continue to do "dnf install cowsay" or whatever client side on
standalone systems like desktops and homelab servers, you aren't obligated to tie your
an installed OS to external build infrastructure)

[Daniel Walsh]

On 10/14/22 18:09, Colin Walters wrote:
>  On Thu, Oct 13, 2022, at 3:08 PM, Ben Cotton wrote:
> > https://fedoraproject.org/wiki/Changes/OstreeNativeContainerStable
>  I know there's a lot going on here, so I put together
>  https://github.com/cgwalters/dnfimage-config
>  as a demonstration system to show this all works today.  (Though there's a lot left
> to do)
>
>  To say this another way...I really, really wish I had a time machine to go back and
> announce *this* instead of doing Fedora Atomic Host way back in the day.  If when Docker
> had come out (before Kubernetes, before podman, before CoreOS) I wish I'd said
> "hey this container stuff is cool, why don't we make bootable host operating
> systems configurable via containers too").  Oh well, better late than never!
>
>  (And yes, we're not the first to this nowadays, but...first, this path gives a
> seamless upgrade for all the existing ostree-based systems out there, and second, you
> absolutely can continue to do "dnf install cowsay" or whatever client side on
> standalone systems like desktops and homelab servers, you aren't obligated to tie your
> an installed OS to external build infrastructure)
>
>
>
>
>
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]
>  Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
I just wanted to throw in my two cents on this, that I absolutely love 
the idea. We all wish we had Way Back machines.

[Dusty Mabe]

On 10/13/22 15:08, Ben Cotton wrote:
>  https://fedoraproject.org/wiki/Changes/OstreeNativeContainerStable
>  
>  This document represents a proposed Change. As part of the Changes
>  process, proposals are publicly announced in order to receive
>  community feedback. This proposal will only be implemented if approved
>  by the Fedora Engineering Steering Committee.
>  
>  
>  == Summary ==
>  
>  Continue the work done in
>  https://fedoraproject.org/wiki/Changes/OstreeNativeContainer but in an
>  officially stable format, and expanded to cover more OSTree-based
>  editions.  This goes "all in" on being container-native and
>  significantly changes the technology and user emphasis.
>  
>  == Owner ==
>  
>  * Name: [[User:walters| Colin Walters]], [[User:jmarrero| Joseph
>  Marrero]], [[User:baude| Brent Baude]]
>  * Email: walters(a)verbum.org, jmarrero(a)fedoraproject.org, baude(a)fedoraproject.org
>  
>  
>  == Detailed Description ==
>  
>  * rpm-ostree's functionality to
>  [https://coreos.github.io/rpm-ostree/container/ boot and upgrade from
>  Docker/OCI container images] is declared stable
>  * Rework Fedora editions and spins (CoreOS, IoT, Silverblue, Kinoite,
>  etc) that use ostree to instead deliver via Docker/OCI container
>  images
>  * `rpm-ostree` is reworked to gain strong CLI compatibility with
>  `yum/dnf` commands (cliwrap)
>  * The new container native functionality is exposed via `dnf image`
>  * (TBD: Rebranding of rpm-ostree itself to `dnf-image` or something else)
>  * Support and documentation for generating derived images
>  * Support in Anaconda and other tools
>  * Support for generating bootable images 
There are a lot of things going on in this proposal:

- shipping editions as container images in quay
- migrating existing users to the new container image based updates
- overriding yum/dnf commands via cliwrap
- rebranding `rpm-ostree`

I feel like we'd have a more cohesive discussion if we tried to break this up
into smaller pieces so we can have a more focused discussion. As written this
proposal is hard to swallow.

>  
>  
>  A top level goal is that users should not need to know or understand
>  ostree (or rpm-ostree); they only need to know containers and most key
>  functionality is exposed via the `yum/dnf` frontend with a few
>  extensions.
>  
>  
>  === Backwards compatibility ===
>  
>  If you're a user of ostree today: no need to worry.  Everything that
>  works today in ostree and rpm-ostree will continue to work for years
>  to come (it's shipped in RHEL9). However, it's expected that most
>  users will find the new model sufficiently compelling to switch fairly
>  quickly.
>  
>  === Stable Bootable OCI ===
>  
>  More information about this is available in
>  [https://coreos.github.io/rpm-ostree/container/ the rpm-ostree
>  documentation].  This Change highlights that this functionality is
>  planned to be officially stable.
>  
>  === Changing to OCI over the wire ===
>  
>  Today Fedora has an OSTree repository that contains updates for
>  editions such as Fedora CoreOS and Fedora Silverblue.
>  
>  These editions instead will become [https://github.com/opencontainers
>  OCI base images], available at e.g.
>  
>  * `quay.io/fedora/fedora-coreos:stable`
>  ([https://quay.io/repository/fedora/fedora-coreos this exists today]!)
>  * `quay.io/fedora/fedora-silverblue:38` (does not exist, but can soon)
>  
>  An update will be shipped that will cause existing systems tracking
>  the production ostree branches to be switched to the corresponding
>  container image.  Concretely for example, a system running
>  `fedora:fedora/36/x86_64/silverblue` will execute `rpm-ostree rebase
>  ostree-remote-registry:fedora:quay.io/fedora/fedora-silverblue:36`. 
I don't think I'm comfortable with making this switch by default for some
time (certainly not in the F38 time frame). We currently only have the
container image in place for one edition (Fedora CoreOS) and FCOS isn't
even ready (you mention the design issue below).

This will probably take a few cycles. What we need to do now is come up
with a plan and then implement it over time. 

>  
>  Note that at the current time, Fedora does not sign container images.
>  This should change (likely to something based on
>  [cosign](https://github.com/containers/skopeo/pull/1701)), but in the
>  mean time, the ostree-container flow supports verifying the embedded
>  ostree-baesd GPG signatures and this will continue to be used.
>  
>  The Fedora OSTree repository will become read-only and stop receiving updates. 
As mentioned above. I highly doubt it's feasible to stop shipping updates
there in a single cycle.

> 
> Note that for Fedora CoreOS specifically, there is some outstanding
> design discussion around the intersection with Cincinnati:
> https://github.com/coreos/fedora-coreos-tracker/issues/1263
> 
> ==== Wire efficiency ====
> 
> Today the container images generated by rpm-ostree are "chunked" in a
> reproducible fashion; for more information on this, see
> 
> * https://github.com/ostreedev/ostree-rs-ext/issues/69
> * https://coreos.github.io/rpm-ostree/container/
> 
> However, longer term (as that first issue notes) we do want to add
> [https://github.com/containers/image/pull/902 container deltas].
> Doing so will benefit non-host containers too.
> 
> 
> 
> === dnf/yum CLI compatibility ===
> 
> rpm-ostree has a feature called
> [cliwrap](https://coreos.github.io/rpm-ostree/cliwrap/) today.  A key
> motivation here is that in the switch to using containers as the
> frontend, the project name "rpm-ostree" (which is very literal) no
> longer makes sense.  A further emphasis on "container native" will
> strongly encourage writing e.g. a `Dockerfile` like this:
> 
> 
> <pre>
> FROM quay.io/fedora/fedora-silverblue:stable
> RUN dnf -y install sway && ostree container commit
> </pre>
> 
> *Today*, one can spell this as `RUN rpm-ostree install -y sway &&
> ostree container commit` - but a toplevel goal here is to allow host
> system customization using the same patterns and tools one uses to
> build application containers, and to de-emphasize the "ostree backend"
> aspect.
> 
> To complete this story on the client side, what is today called
> "ostree native containers" will be exposed via a new `dnf image`
> subcommand on the client system.
> 
> For example:
> 
> 
> <pre>
> $ dnf image rebase quay.io/examplecorp/customos:latest
> </pre>
> 
> As well as other offline/"image based" functionality such as `dnf
> image rollback` and `dnf image apply-live`.
> 
> However, as many people know, rpm-ostree does today support
> "per-machine" package layering/overrides - and *that will continue to
> work*.  So users are not *required* to do builds on a remote server
> even in "image mode".
> 
> Concretely, it will also work to `sudo dnf -y install sway` inside a
> root terminal on a Fedora Workstation for example.  This will still be
> a transactional operation.
> 
> More: https://github.com/coreos/rpm-ostree/issues/2883
> 
> === Rebranding of rpm-ostree ===
> 
> This is a fundamental change in technology *and* positioning for
> rpm-ostree (and for the projects that use it).  The terminology of
> "immutable" gets even fuzzier now (see also [this
>
blog](https://blog.verbum.org/2020/08/22/immutable-%E2%86%92-reprovisiona...).
> 
> It would make sense to rebrand the project due to this.  No decision
> has been made on this, but the choices are:
> 
> - Keep as is
> - Rebrand to dnf-image (or, reviving a past name, yum-image)
> - Rebrand to something else
> 
> More: https://github.com/coreos/rpm-ostree/issues/405
> 
> === Support for building derivative containers ===
> 
> Until now, the model for systems like Fedora CoreOS and Silverblue
> etc. has included an emphasis on containerization, but avoiding
> extensive customization of the host system.  This will be weakened
> somewhat - it will be fully supported to build derivative operating
> system  images that in turn contain software which itself should run
> directly on the host system, and not in a container.
> 
> A key aspect of this is that by generating a custom build, one binds
> together the code and configuration as a transactionally updatable
> unit.
> 
> The role of things like Kickstart, cloud-init, and Ignition and other
> system config management shrinks significantly.
> 
> === Example 1: Kubernetes and kubelet ===
> 
> It is not supported upstream to run the kubelet in a container image
> itself.  Today, for OpenShift 4 we embed the kubelet as part of the
> CoreOS builds.  The [https://www.okd.io/ OKD] project
>
[https://github.com/openshift/okd-machine-os/blob/fa2248d332abc9926e68428d...
> already installs kubelet during a container build] deriving from the
> base `fedora-coreos` image.  It is possible that RHEL CoreOS will make
> [https://github.com/openshift/os/issues/799 a similar split].
> 
> === Example 2: Agent software ===
> 
> Many organizations require agents (cloud agents, security scanners,
> antivirus) which are not ready to be containerized.  Often too, these
> tools come in e.g. RPM form, and also expect to be configured (remote
> logging server, etc.).  This tooling allows to `dnf install` these
> *and* embed the configuration, generating a standard container image
> which can then be booted as an atomic unit.
> 
> (Note that in the OpenShift context, we expect this to chain with the
> first example - users instead derive from a `okd-node-base` image
> which itself derives from `fedora-coreos`).
> 
> === Example 3: Podman machine ===
> 
> The `[http://podman.io Podman]` project has a function called
>
`[https://github.com/containers/podman/blob/main/docs/source/markdown/podman-machine.1.md
> machine]` where it creates a virtual machine and boots standard CoreOS
> builds. Podman developers and users has long desired a custom FCOS
> image that is focused on container deployment using Podman, including
> certain QEMU packages that allow for cross-architecture development.
> This approach will solve a series of problems including but not
> limited to getting more timely release alignment between Podman and
> FCOS.
> 
> We also have a user base that wants to customize the VM image based on
> restrictive environments ... bring your own virtual machine image.
> 
> === Support in Anaconda and other tools ===
> 
> Anaconda support for custom images is tracked in this bug:
> https://bugzilla.redhat.com/show_bug.cgi?id=2125655
> 
> But, as part of this Change, it should also work to generate an
> Anaconda ISO which allows *user choice* for whether to install in
> "image mode" or "traditional".
> 
> === Support for generating bootable images ===
> 
> Instead of using an existing disk image/ISO (Anaconda, AWS AMI, etc.)
> and then doing an in-place update to a generated container image, in
> many cases will want to generate disk images using their container
> image as input.
> 
> For example, generating an ISO that embeds that container image as
> input and will install it.
> 
> In this proposal, we plan to ship a container image as part of Fedora
> CoreOS that will do this:
> https://github.com/coreos/fedora-coreos-tracker/issues/1151
> 
> But, it also makes sense to have a tool that e.g. does this and
> generates an Anaconda ISO (we will inherently be doing this as part of
> generating ostree-based Anaconda ISOs already!) - this would just be
> supporting that in a clear external way too.  One likely target for
> this is [https://github.com/osbuild osbuild] and
>
[https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9/...
> RHEL Image Builder].
> 
> 
> 
> == Feedback ==
> 
> 
> == Benefit to Fedora ==
> 
> The need to understand ostree goes away for most users; they
> understand RPMs and containers.  While retaining all the current
> (rpm-)ostree features.
> 
> == Scope ==
> * Proposal owners:
> * Other developers: May affect e.g. gnome-software and similar tools
> that talk to rpm-ostree.
> 
> * Release engineering: [https://pagure.io/releng/issue/11047]
> * Policies and guidelines: None.
> * Trademark approval: N/A (not needed for this Change)
> * Alignment with Objectives: None
> 
> == Upgrade/compatibility impact ==
> 
> We will ship a systemd unit that transitions systems currently using
> OSTree on the wire over to tracking the corresponding container
> images.  Note again, all features of rpm-ostree continue to work!  For
> example, if you have layered packages, that will continue to be
> applied.
> 
> == How To Test ==
> 
> Build container images, boot them.
> 
> == User Experience ==
> 
> Users will not need to understand ostree (at least at a superficial
> level), only container images.
> 
> == Dependencies ==
> 
> Needs improvements in Anaconda in some cases, also would like to
> increase alignment with dnf.  Fetching container images uses `skopeo`,
> maintained by containers team.
> 
> == Contingency Plan ==
> * Contingency mechanism: Continue to ship things the way we ship them today
> * Contingency deadline: Dunno
> * Blocks release? No
> 
> == Documentation ==
> 
> https://coreos.github.io/rpm-ostree/container/ and
> https://github.com/coreos/coreos-layering-examples etc.
> 
> == Release Notes ==
> 
>

[Dusty Mabe]

On 10/24/22 23:45, Dusty Mabe wrote:
>  
>  
>  On 10/13/22 15:08, Ben Cotton wrote:
> >
> > An update will be shipped that will cause existing systems tracking
> > the production ostree branches to be switched to the corresponding
> > container image.  Concretely for example, a system running
> > `fedora:fedora/36/x86_64/silverblue` will execute `rpm-ostree rebase
> > ostree-remote-registry:fedora:quay.io/fedora/fedora-silverblue:36`.
>  
>  I don't think I'm comfortable with making this switch by default for some
>  time (certainly not in the F38 time frame). We currently only have the
>  container image in place for one edition (Fedora CoreOS) and FCOS isn't
>  even ready (you mention the design issue below).
>  
>  This will probably take a few cycles. What we need to do now is come up
>  with a plan and then implement it over time.  
And to be clear. What will take a few cycles is fully adopting this
proposal. There are certainly pieces of it that work already today.

[Colin Walters]

On Mon, Oct 24, 2022, at 11:45 PM, Dusty Mabe wrote:
>  There are a lot of things going on in this proposal:
>
>  - shipping editions as container images in quay 
https://pagure.io/releng/issue/11047

>  - migrating existing users to the new container image based updates
(No tracker yet)

>  - overriding yum/dnf commands via cliwrap 
https://github.com/coreos/rpm-ostree/issues/2883

>  - rebranding `rpm-ostree` 
https://github.com/coreos/rpm-ostree/issues/405

>  I feel like we'd have a more cohesive discussion if we tried to break this up
>  into smaller pieces so we can have a more focused discussion.  
These links were already mostly in the proposal, but I've added them again here for
convenience.

>  As written this proposal is hard to swallow. 
Two things:

- This proposal is explicitly trying to tie everything together.  I think without the
"bigger picture", it's actually *more* confusing.  For example, just pushing
the container images does little unless we invest in them as a derivation source and build
tooling and docs around them.
- At a very practical level, I find context switching in Mediawiki syntax to be rather
tedious.  I'd prefer to discuss the individual sections in the already extant tracker
issues that accept Markdown and have a well-understood and used discussion mechanism.  (Or
of course, email here)

But you're certainly not wrong, it *is* a big proposal.  Happy to make changes,
I'm mainly just saying there already are dedicated sub-proposal trackers to discuss
the details of those.

[Matthew Miller]

On Tue, Oct 25, 2022 at 09:00:40AM -0400, Colin Walters wrote:
>  Two things:
>  
>  - This proposal is explicitly trying to tie everything together.  I think without the
> "bigger picture", it's actually *more* confusing.  For example, just pushing
> the container images does little unless we invest in them as a derivation source and build
> tooling and docs around them.
>  - At a very practical level, I find context switching in Mediawiki syntax to be rather
> tedious.  I'd prefer to discuss the individual sections in the already extant tracker
> issues that accept Markdown and have a well-understood and used discussion mechanism.  (Or
> of course, email here)
>  
>  But you're certainly not wrong, it *is* a big proposal.  Happy to make changes,
> I'm mainly just saying there already are dedicated sub-proposal trackers to discuss
> the details of those. 
Would it make sense to have a Fedora Objective (at the Council level) around
this?

-- 
Matthew Miller
<mattdm(a)fedoraproject.org&gt;
Fedora Project Leader

[Colin Walters]

On Mon, Oct 31, 2022, at 5:14 PM, Matthew Miller wrote:
>  On Tue, Oct 25, 2022 at 09:00:40AM -0400, Colin Walters wrote:
> > Two things:
> > 
> > - This proposal is explicitly trying to tie everything together.  I think without the
> "bigger picture", it's actually *more* confusing.  For example, just pushing
> the container images does little unless we invest in them as a derivation source and build
> tooling and docs around them.
> > - At a very practical level, I find context switching in Mediawiki syntax to be
> rather tedious.  I'd prefer to discuss the individual sections in the already extant
> tracker issues that accept Markdown and have a well-understood and used discussion
> mechanism.  (Or of course, email here)
> > 
> > But you're certainly not wrong, it *is* a big proposal.  Happy to make changes,
> I'm mainly just saying there already are dedicated sub-proposal trackers to discuss
> the details of those.
>
>  Would it make sense to have a Fedora Objective (at the Council level) around
>  this? 
Probably?  

I would say actually that this proposal is not even the end.  I cut out a part due to
objections/concerns, but I personally (still) want to get rid of e.g. the
Silverblue/Kinoite/CoreOS type "sub-brands".   It's just e.g.
"Workstation, but in image+container-default mode" - where it even matters. 
Otherwise one can just say "I run Fedora".  This is revisiting
https://discussion.fedoraproject.org/t/its-more-fedora-than-it-is-fedora-... in
effect - I personally think "we'll actually just do what you asked for when you
type dnf -y install cowsay" is a notable leap here.

[Matthew Miller]

On Tue, Nov 01, 2022 at 09:18:19AM -0400, Colin Walters wrote:
>  > Would it make sense to have a Fedora Objective (at the Council
> level) around
>  > this?
>  
>  Probably?  
>  
>  I would say actually that this proposal is not even the end. I cut out a
>  part due to objections/concerns, but I personally (still) want to get rid
>  of e.g. the Silverblue/Kinoite/CoreOS type "sub-brands". It's just e.g.
>  "Workstation, but in image+container-default mode" - where it even
>  matters. Otherwise one can just say "I run Fedora". This is revisiting
>  https://discussion.fedoraproject.org/t/its-more-fedora-than-it-is-fedora-...
>  in effect - I personally think "we'll actually just do what you asked for
>  when you type dnf -y install cowsay" is a notable leap here. 
While I will carefully skirt the terminology issue, I love the vision. Let's
talk more about making an Objective around this.


-- 
Matthew Miller
<mattdm(a)fedoraproject.org&gt;
Fedora Project Leader

[Jonathan Lebon]

On Tue, Oct 25, 2022 at 12:43 PM Colin Walters <walters(a)verbum.org&gt; wrote:
>  - This proposal is explicitly trying to tie everything together.  I
> think without the "bigger picture", it's actually *more* confusing.  For
> example, just pushing the container images does little unless we invest in them as a
> derivation source and build tooling and docs around them. 
I agree it's helpful to discuss this at a higher level than the
individual pieces to make sense of it. But the proposal as is seems to
imply it will all be done in one cycle which I agree with Dusty is
very optimistic.

>  But you're certainly not wrong, it *is* a big proposal.  Happy to
> make changes, I'm mainly just saying there already are dedicated sub-proposal trackers
> to discuss the details of those. 
WDYT about introducing phases into the proposal? E.g. something like:
- phase 1 (f38): start pushing OSTree-based variants as container
images in Quay.io; users can opt into rebasing onto them
- phase 2 (f39): automatically transition existing users to shipping
from Quay.io; users can opt out.
- phase 3 (f40): stop pushing OSTree repo updates entirely

This is just an example, we can break it down some other way. Also,
it'd make sense to have the transition for each variant be driven by
that variant's working group (obviously with collaboration between
them).

On that topic, it doesn't seem like we got any feedback from Fedora
IoT at least. Also not sure where we stand for Fedora Silverblue and
Kinoite.

On Thu, Oct 13, 2022 at 3:09 PM Ben Cotton <bcotton(a)redhat.com&gt; wrote:
>  * `rpm-ostree` is reworked to gain strong CLI compatibility with
>  `yum/dnf` commands (cliwrap)
>  * The new container native functionality is exposed via `dnf image`
>  * (TBD: Rebranding of rpm-ostree itself to `dnf-image` or something else) 
Some concerns about this have been raised upstream already around
whether hijacking the `dnf` command in that way could cause more
confusion than it's worth. ISTM like a cleaner approach would be to
build this out into `dnf` directly instead and have it drive
rpm-ostree. Then the relationship becomes much clearer, and there's an
obvious path towards gaining parity. Conveniently with dnf5 ditching
Python, this becomes more achievable for systems that don't want
Python.

Things like `install` and `update` can more or less be passthrough as
is (some debate there about whether to provide "advice" on whether
what you're trying to install should be in a container instead; more
of that below). Things like `status` would be valid in "image-mode"
only to start, but then we could build out a traditional version of
it.

It would be great to get feedback from the DNF maintainers about this
proposal and some of the ideas here.

On Tue, Nov 1, 2022 at 9:34 AM Colin Walters <walters(a)verbum.org&gt; wrote:
>  I personally think "we'll actually just do what you asked
> for when you type dnf -y install cowsay" is a notable leap here. 
I think this probably makes sense generally, but I'll note that for
Fedora CoreOS at least, the whole point is to have users' workloads
run in containers and decoupled from the host. E.g. we've gone to
great lengths to keep Python out for that reason (also, `cowsay` pulls
in Perl BTW :) ). Having non-finger compatibility with `dnf install -y
cowsay` was kind of a feature in that sense; it made users think twice
before falling into the same patterns as other systems. Now with this
and especially container layering, it gives more power to users but
weakens that messaging. I'm not sure if it's worth adding some
artificial technological constraints to try to steer them back or just
keeping it as a documentation thing would suffice.

[Colin Walters]

On Mon, Nov 21, 2022, at 10:20 AM, Jonathan Lebon wrote:
>  On Tue, Oct 25, 2022 at 12:43 PM Colin Walters
> <walters(a)verbum.org&gt; wrote:
> > - This proposal is explicitly trying to tie everything together.  I think without the
> "bigger picture", it's actually *more* confusing.  For example, just pushing
> the container images does little unless we invest in them as a derivation source and build
> tooling and docs around them.
>
>  I agree it's helpful to discuss this at a higher level than the
>  individual pieces to make sense of it. But the proposal as is seems to
>  imply it will all be done in one cycle which I agree with Dusty is
>  very optimistic. 
I agree.

>  WDYT about introducing phases into the proposal? E.g. something
> like:
>  - phase 1 (f38): start pushing OSTree-based variants as container
>  images in Quay.io; users can opt into rebasing onto them
>
>  - phase 2 (f39): automatically transition existing users to shipping
>  from Quay.io; users can opt out.
>  - phase 3 (f40): stop pushing OSTree repo updates entirely 
I'd agree that "stop pushing ostree repo updates" probably isn't going
to happen in Fedora 38.  Dunno, I guess we could try a f40 change that calls that out
explicitly.

>  Some concerns about this have been raised upstream already around
>  whether hijacking the `dnf` command in that way could cause more
>  confusion than it's worth. ISTM like a cleaner approach would be to
>  build this out into `dnf` directly instead and have it drive
>  rpm-ostree.  
I agree with this longer term, though there's a *lot* of details here.  We have the
same problem with this that exists with dnf5 in that there are a *ton* of options that dnf
exposes that we're unlikely to make work anytime soon.  (A random example here is
"-4 resolve to IPv4 addresses only" which seems tricky to plumb through the
stack, particularly scoping in the container side).

>  It would be great to get feedback from the DNF maintainers about
> this
>  proposal and some of the ideas here. 
Agreed!  I pinged them directly again but was hoping they'd see this Change and reply
here.

>  I think this probably makes sense generally, but I'll note that
> for
>  Fedora CoreOS at least, the whole point is to have users' workloads
>  run in containers and decoupled from the host. E.g. we've gone to
>  great lengths to keep Python out for that reason (also, `cowsay` pulls
>  in Perl BTW :) ). Having non-finger compatibility with `dnf install -y
>  cowsay` was kind of a feature in that sense; it made users think twice
>  before falling into the same patterns as other systems. Now with this
>  and especially container layering, it gives more power to users but
>  weakens that messaging. 
This is true.  But weakening that messaging (and making the technology more flexible) also
*weakens the barriers* we've set up between "CoreOS" and other editions.   

(This topic gets confusing because we could be talking about the *build* side or the
client side or both; I hope most people agree the CLI compatibility on the build side just
makes sense)

BTW I wanted to give an update here specifically regarding the "dnf image" bit -
as of late, I've been working on a fresh new "bootc" CLI, see
https://github.com/ostreedev/ostree-rs-ext/pull/412 and https://github.com/cgwalters/bootc
and that may make more sense for the client side.

[Colin Walters]

On Wed, Nov 30, 2022, at 8:11 PM, Colin Walters wrote:
>  BTW I wanted to give an update here specifically regarding the "dnf 
>  image" bit - as of late, I've been working on a fresh new "bootc" CLI,
>
>  see https://github.com/ostreedev/ostree-rs-ext/pull/412 and 
>  https://github.com/cgwalters/bootc and that may make more sense for the 
>  client side. 
🆕 This is now more real - it's been moved to github.com/containers/bootc
and I've updated the Change proposal at
https://fedoraproject.org/wiki/Changes/OstreeNativeContainerStable to reflect this.  This
is all pretty new but - again I've been searching for a succinct way to describe all
this stuff, and "bootc" seems to work much better than "ostree
container" or "dnf image" etc.