A recent issue was identified in the Fedora CoreOS *testing* stream where
nodes attempting to upgrade may become stuck. This is due to a missing
fedora-coreos.stream annotation in the update metadata, which causes
Zincati (versions < 0.0.31) to fail the upgrade process. This issue *did
not* reach *stable*.
Full details: https://github.com/coreos/fedora-coreos-tracker/issues/2112
While the rollout was paused, nodes on the testing stream that had already
attempted the upgrade may currently be in a "stalled" state.
*What’s Happening*
Nodes that attempted to upgrade to *43.20260217.2.1*, and failed during
staging due to this issue are now unable to upgrade to *43.20260217.2.2*.
In other words, once a node entered the failed staging loop on
*43.20260217.2.1*, it remains stuck and will not automatically progress to
newer releases.
*Why This Happened*
The root cause was dropping the fedora-coreos.stream annotation. Older
Zincati versions (< 0.0.31) still depend on that annotation during upgrade
logic. When attempting to stage *43.20260217.2.1* that no longer includes
it, Zincati fails and retries indefinitely. This only impacts older Zincati
clients upgrading across that boundary.
*How to Identify if You Are Affected*
If you are on the testing stream, check your rpm-ostree status:
rpm-ostree status
If you see multiple failed deployment attempts in Zincati, your system is
likely affected.
*Workaround*
The workaround described here resolves the issue:
https://github.com/coreos/fedora-coreos-tracker/issues/2112#issuecomment-39…
sudo systemctl stop zincati rpm-ostreed
sudo rpm-ostree cleanup -p
sudo systemctl start zincati # or run sudo /usr/bin/zincati-update-now for
immediate upgrade+reboot
You can also reboot the node, this action has been shown to clear the
stalled state and allow the upgrade to proceed.
*If you still see issues*
If you believe you’re affected and the workaround does not resolve the
issue, please comment on the tracker issue above with:
- rpm-ostree status
- rpm -q zincati
- Relevant journalctl -u zincati logs
Thanks to everyone running *testing* and helping us catch issues before
they reach *stable*!
Michael Armijo - CoreOS
Hi Everyone!
We are shipping Podman 5.8.0 to the Fedora CoreOS next stream release
*43.20260218.1.2*. Podman 5.8.0 introduces an automatic migration of the
container database from BoltDB to SQLite [1]. Since BoltDB support will be
fully removed in Podman 6.0, we are prioritizing this transition now to
ensure a smooth experience for all users.
We are temporarily "unlocking" next from the testing and stable streams.
This allows for soak-testing the migration logic without affecting testing
or stable users.
*How to help*
If you have an older FCOS install, check your current backend with:
`podman info --format {{.Host.DatabaseBackend}}`
If it says `boltdb`, update to the next stream and reboot (documentation
for switching streams
<https://docs.fedoraproject.org/en-US/fedora-coreos/update-streams/#_switchi…>).
After the update, verify that it now shows sqlite and that your containers
are running as expected.
*Reporting*
Please report successful migrations to the official Test Week ticket
<https://forge.fedoraproject.org/quality/tickets/issues/877>. If the
migration fails or you encounter issues, please file an upstream bug at
github.com/containers/podman or reach out on the FCOS tracker
<https://github.com/coreos/fedora-coreos-tracker/>.
[1]: https://github.com/coreos/fedora-coreos-tracker/issues/2110
Thanks,
Michael Armijo - CoreOS