11:02 a.m.
A goal of rpm-ostree is to *empower* users and developers. Being able to fearlessly have
automatic updates on and know that if your system loses power or the kernel freezes in the
middle everything will be OK - that's empowering! But some implementation choices led
to other restrictions that weren't intentional, just bugs. In this release we've
made a large advance in lifting one of the biggest restrictions.
## rpm-ostree ex apply-live
In this release the functionality formerly known as `rpm-ostree ex livefs` is now known as
`rpm-ostree ex apply-live`, and it's been placed on a much firmer technical foundation
and is considered very safe to use. It's still under `ex` because we may make some
interface changes, and we hope to gather feedback.
One can often see in discussions people say something of the form "with rpm-ostree
you need to reboot for updates" (notice this is a phrase couched in terms of
restrictions). But that's never been true in general, because the goal is to move
most software to containers. Rather than having e.g. `gcc` as part of your root
filesystem (in ostree), you have it in your development container updated independently
from the host. Server and desktop applications run as containers, etc.
For all the software that *is* part of the host though, our story until recently has been
incomplete. We've had `rpm-ostree usroverlay` for a long time, which can be very
convenient for testing things. But a core problem is that rpm-ostree has no idea
what's happening in the transient writable layer, and further we didn't offer any
tools to make changes there. The writable layer also meant that any modifications was
allowed, even if all one wanted was e.g. installing a single package.
For example, if you use `rpm-ostree usroverlay` and then `rpm -Uvh
https://example.com/some.rpm`, `rpm-ostree status` doesn't show it as layered - it
won't persist across upgrades and reboots if you did want that.
Now you can combine e.g.
```
$ rpm-ostree install fish
$ rpm-ostree ex apply-live
```
And have the newly layered `fish` package appear both for the *next* boot (persistently)
as well as your running filesystem. An obvious thing to add will be `rpm-ostree install
--apply-live fish` once we stabilize the interface (which will likely be soon).
### Implementation
The big change in implementation that makes this very safe is that changes to the running
filesystem tree go into an overlayfs with the upper directory being temporary. We no
longer create a rollback deployment; rebooting will return the previously-booted
deployment to its original state. More information in [this ostree
PR](https://github.com/ostreedev/ostree/pull/2103) and an example below.
### Still readonly
One thing that *hasn't* changed (compared to previous `ex livefs`) is that `/usr` is
still mounted read-only from the perspective of the rest of the system. It's worth
emphasizing this! In this model, rpm-ostree is still in full control, `rpm-ostree status`
is showing you truth, etc.
This is a good example how use of the term "immutable" is suboptimal when
discussing an rpm-ostree based system. See [this blog
entry](https://blog.verbum.org/2020/08/22/immutable-%E2%86%92-reprovision...
for more discussion on that.
### Testing fixes
However this isn't just for layered packages; let's take the example of something
commonly shipped in a base image like
[NetworkManager](https://gitlab.freedesktop.org/NetworkManager/NetworkMana...,
[podman](https://podman.io/), or [OpenSSH](https://www.openssh.com/).
Say you're running Fedora CoreOS stable, and you want to test a fix for podman
that's in [the testing
stream](https://docs.fedoraproject.org/en-US/fedora-coreos/update-streams/). After
you've run e.g. `rpm-ostree rebase fedora/x86_64/coreos/testing` you can now
confidently `rpm-ostree ex apply-live` to switch your running filesystem to that without
rebooting and start running `podman` commands. Then you can report success/failure to an
upstream issue or bug.
## override replace https://bodhi/...
Another notable feature in this release is support for directly pulling builds from
current Fedora testing/build tools [Bodhi](https://bodhi.fedoraproject.org/) and
[Koji](https://koji.fedoraproject.org/koji/).
A longstanding tension that rpm-ostree has created is that we want to test the *whole*
build (ostree commit) that is shipped to users, but Bodhi is oriented around per-package
changes. Switching to a `testing` stream pulls in everything. That's how it's
shipped to users so we need to test it, but it also means you're testing everything at
once.
Now with rpm-ostree v2021.1 you can run e.g.:
```
$ rpm-ostree override replace
https://bodhi.fedoraproject.org/updates/FEDORA-2020-2908628031
```
to directly pull in a single Bodhi update applied relative to your booted system
(presumably a `stable` stream) without changing anything else. You can also pull in Koji
builds (that may or may not be in a Bodhi update):
```
$ rpm-ostree override replace
https://koji.fedoraproject.org/koji/buildinfo?buildID=1625029
```
This allows you to more directly interact with Bodhi's current model of testing
individual updates.
Note that like all usage of `override replace`, these versions are "pinned"
until explicitly removed with a variant of `rpm-ostree override reset`.
## Combining features
And of course, the above two headlining features combine! Let's say that you're a
Fedora Silverblue user, and you're hitting a WiFi issue that is claimed to be fixed in
an updated NetworkManager. With a combination like:
```
$ rpm-ostree override replace
https://bodhi.fedoraproject.org/updates/FEDORA-<updateid>
$ rpm-ostree ex apply-live
$ systemctl restart NetworkManager
```
You can quickly test out that change.
What's powerful about using rpm-ostree for this is that if e.g. something goes wrong
and the updated NetworkManager is crashing or nonfunctional, you still have the base
version from the booted deployment! There's no need to carefully save previous
versions or keep a recovery USB key on hand. To undo the above:
```
$ rpm-ostree override reset -a # note actually resets all overrides
$ rpm-ostree ex apply-live
$ systemctl restart NetworkManager
```
But let's look at a more problematic scenario: starting the updated NetworkManager
triggers a latent bug in your specific laptop's WiFi card driver, causing a kernel
panic and system lockup.
In this scenario, *all you need to do is reboot* (e.g. hold the power button for 5 seconds
on most PCs) and you will be back exactly to the previous deployment with the working
kernel+NetworkManager combination. You don't even need to manually stop at the
bootloader menu to choose a previous deployment!
And a similar thing applies if the system actually crashes *while* we're mutating your
running filesystem with `apply-live` - just reboot and everything will be fine.
The reason for this is twofold. First, by default ostree "finalizes" deployments
(adds a bootloader entry) only when the system is cleanly shut down, so we're not
booted into the new root. Second, the overlayfs approach mentioned above means the
"live applied" changes are written to a transient overlay rather than changing
the persistent filesystem (underlying deployment).
## Other changes
## Countme support
rpm-ostree now ships an `rpm-ostree-countme.timer` unit which fulfills the same purpose as
the [DNF countme feature](https://fedoraproject.org/wiki/Changes/DNF_Better_Counting).
This will allow Fedora counting of installed systems to include rpm-ostree-based variants
like Fedora CoreOS, Fedora Silverblue, and Fedora IoT, while still preserving user
privacy.
Note the unit is currently *not* enabled by default. For more information, including how
to disable it, see [the docs](https://coreos.github.io/rpm-ostree/countme/).
### Default to sqlite in F34+
Fedora 34 rpm-ostree based systems will switch to the sqlite backend by default, silencing
a warning about use of the bdb backend.
### Shared library "hollowed out"
Quite a while ago rpm-ostree started shipping a shared library `librpmostree-1.so.1.0.0`;
mainly it's used by e.g. gnome-software to display package diffs between ostree
commits. However this created a problem later because we duplicated a lot of code between
our binary and the shared library, and that in turn caused unnecessary bloat. The build
system and implementation of the shared library was reworked to fork off the binary and
parse its output. This reduced the size (debug symbols stripped) from ~2.6MB to 320k
(-~88%).
### Internals: FFI and Rust
Large portions of rpm-ostree have already been ported to the [Rust programming
language](https://www.rust-lang.org/), in order to improve the overall level of safety and
resilience of the project.
To further accelerate this transition, since the last release we are using
[cxx.rs](https://cxx.rs/) as a code generation helper, which has the compelling feature of
supporting fully safe bidirectional calls between C++ and Rust.
This required switching the C code to "C that builds in C++ mode". For the sake
of clarity though, the goal here is to accelerate our transition to Rust, not to rewrite
rpm-ostree code in modern C++.
# In case you missed it: Some prior release highlights
[v2020.6](https://github.com/coreos/rpm-ostree/releases/tag/v2020.6) included an `ex
initramfs-etc` command to customize the initramfs and rewrote the documentation.
[v2020.5](https://github.com/coreos/rpm-ostree/releases/tag/v2020.5) made a notable
improvement in layered package handling, and also had some smaller milestones in porting
some C code to Rust.
[v2020.3](https://github.com/coreos/rpm-ostree/releases/tag/v2020.3) deprecated the
non-`--unified-core` code - if you're still seeing this warning, please adjust your
buildsystem!
[v2020.2](https://github.com/coreos/rpm-ostree/releases/tag/v2020.2) contained a mix of
bugfixes and features, but also continued a big push for improved continuous integration.
[v2020.1](https://github.com/coreos/rpm-ostree/releases/tag/v2020.1) added several
enhancements on the build side, and kicked off improved CI.
# Bodhi update
Here's the [rpm-ostree v2021.1 Bodhi
update](https://bodhi.fedoraproject.org/updates/FEDORA-2021-e55da2fc78). We hope you give
it a try (particularly the new `ex apply-live`)!