A recent issue was identified in the Fedora CoreOS *testing* stream where
nodes attempting to upgrade may become stuck. This is due to a missing
fedora-coreos.stream annotation in the update metadata, which causes
Zincati (versions < 0.0.31) to fail the upgrade process. This issue *did
not* reach *stable*.
Full details: https://github.com/coreos/fedora-coreos-tracker/issues/2112
While the rollout was paused, nodes on the testing stream that had already
attempted the upgrade may currently be in a "stalled" state.
*What’s Happening*
Nodes that attempted to upgrade to *43.20260217.2.1*, and failed during
staging due to this issue are now unable to upgrade to *43.20260217.2.2*.
In other words, once a node entered the failed staging loop on
*43.20260217.2.1*, it remains stuck and will not automatically progress to
newer releases.
*Why This Happened*
The root cause was dropping the fedora-coreos.stream annotation. Older
Zincati versions (< 0.0.31) still depend on that annotation during upgrade
logic. When attempting to stage *43.20260217.2.1* that no longer includes
it, Zincati fails and retries indefinitely. This only impacts older Zincati
clients upgrading across that boundary.
*How to Identify if You Are Affected*
If you are on the testing stream, check your rpm-ostree status:
rpm-ostree status
If you see multiple failed deployment attempts in Zincati, your system is
likely affected.
*Workaround*
The workaround described here resolves the issue:
https://github.com/coreos/fedora-coreos-tracker/issues/2112#issuecomment-39…
sudo systemctl stop zincati rpm-ostreed
sudo rpm-ostree cleanup -p
sudo systemctl start zincati # or run sudo /usr/bin/zincati-update-now for
immediate upgrade+reboot
You can also reboot the node, this action has been shown to clear the
stalled state and allow the upgrade to proceed.
*If you still see issues*
If you believe you’re affected and the workaround does not resolve the
issue, please comment on the tracker issue above with:
- rpm-ostree status
- rpm -q zincati
- Relevant journalctl -u zincati logs
Thanks to everyone running *testing* and helping us catch issues before
they reach *stable*!
Michael Armijo - CoreOS
Minutes:
https://meetbot-raw.fedoraproject.org/meeting-1_matrix_fedoraproject-org/20…
Minutes (text):
https://meetbot-raw.fedoraproject.org/meeting-1_matrix_fedoraproject-org/20…
Log:
https://meetbot-raw.fedoraproject.org/meeting-1_matrix_fedoraproject-org/20…
=====================================
# #meeting-1:fedoraproject.org: fedora_coreos_meeting
=====================================
Meeting started by @marmijo:fedora.im at 2026-02-18 15:30:21
Meeting summary
---------------
* TOPIC: roll call (@marmijo:fedora.im, 15:30:33)
* TOPIC: Action items from last meeting (@marmijo:fedora.im, 15:35:04)
* INFO: The Fedora CoreOS Community Meeting on 2026-02-25 will be a
live video session to discuss https://github.com/coreos/ignition/pull/2185,
please join us in https://meet.google.com/iom-pedn-kdv. (@marmijo:fedora.im,
15:40:34)
* TOPIC: Review Fedora 44 Release Schedule (@marmijo:fedora.im, 15:40:58)
* LINK:
https://fedorapeople.org/groups/schedule/f-44/f-44-key-tasks.html (@marmijo:
fedora.im, 15:41:03)
* TOPIC: podman v5.8.0 boltDB database migration (@marmijo:fedora.im,
15:46:49)
* LINK: https://github.com/coreos/fedora-coreos-tracker/issues/2110
(@marmijo:fedora.im, 15:46:55)
* AGREED: Roll out podman 5.8.0 to the `next` stream first to validate
the automatic BlotDB-to-SQLite migration before shipping it in `testing`
and `stable`. We will encourage users with legacy BoltDB databases to test
the migration in the `next` stream and report any failures or issues.
(@marmijo:fedora.im, 16:06:27)
* TOPIC: New Package Request: kmscon (@marmijo:fedora.im, 16:19:46)
* LINK: https://github.com/coreos/fedora-coreos-tracker/issues/2092
(@marmijo:fedora.im, 16:19:49)
* TOPIC: Open Floor (@marmijo:fedora.im, 16:30:48)
Meeting ended at 2026-02-18 16:33:41
Action items
------------
People Present (lines said)
---------------------------
* @dustymabe:matrix.org (62)
* @marmijo:fedora.im (32)
* @hricky:fedora.im (17)
* @jbtrystram:matrix.org (10)
* @zodbot:fedora.im (6)
* @spresti:fedora.im (4)
* @jcapitao:matrix.org (3)
* @meetbot:fedora.im (2)
* @ydesouza:fedora.im (2)
Hi Everyone!
We are shipping Podman 5.8.0 to the Fedora CoreOS next stream release
*43.20260218.1.2*. Podman 5.8.0 introduces an automatic migration of the
container database from BoltDB to SQLite [1]. Since BoltDB support will be
fully removed in Podman 6.0, we are prioritizing this transition now to
ensure a smooth experience for all users.
We are temporarily "unlocking" next from the testing and stable streams.
This allows for soak-testing the migration logic without affecting testing
or stable users.
*How to help*
If you have an older FCOS install, check your current backend with:
`podman info --format {{.Host.DatabaseBackend}}`
If it says `boltdb`, update to the next stream and reboot (documentation
for switching streams
<https://docs.fedoraproject.org/en-US/fedora-coreos/update-streams/#_switchi…>).
After the update, verify that it now shows sqlite and that your containers
are running as expected.
*Reporting*
Please report successful migrations to the official Test Week ticket
<https://forge.fedoraproject.org/quality/tickets/issues/877>. If the
migration fails or you encounter issues, please file an upstream bug at
github.com/containers/podman or reach out on the FCOS tracker
<https://github.com/coreos/fedora-coreos-tracker/>.
[1]: https://github.com/coreos/fedora-coreos-tracker/issues/2110
Thanks,
Michael Armijo - CoreOS
Minutes:
https://meetbot-raw.fedoraproject.org/meeting-1_matrix_fedoraproject-org/20…
Minutes (text):
https://meetbot-raw.fedoraproject.org/meeting-1_matrix_fedoraproject-org/20…
Log:
https://meetbot-raw.fedoraproject.org/meeting-1_matrix_fedoraproject-org/20…
=====================================
# #meeting-1:fedoraproject.org: fedora_coreos_meeting
=====================================
Meeting started by @marmijo:fedora.im at 2026-02-11 15:30:24
Meeting summary
---------------
* TOPIC: roll call (@marmijo:fedora.im, 15:30:29)
* TOPIC: Action items from last meeting (@marmijo:fedora.im, 15:34:42)
* INFO: There are no action items from the last meeting. (@marmijo:
fedora.im, 15:35:11)
* TOPIC: Review Fedora 44 Release Schedule (@marmijo:fedora.im, 15:35:25)
* LINK:
https://fedorapeople.org/groups/schedule/f-44/f-44-key-tasks.html (@marmijo:
fedora.im, 15:35:31)
* TOPIC: Temporary stop building Rawhide stream with Konflux (@marmijo:
fedora.im, 15:39:07)
* LINK: https://github.com/coreos/fedora-coreos-tracker/issues/2105
(@marmijo:fedora.im, 15:39:12)
* LINK: https://issues.redhat.com/browse/KFLUXINFRA-2796 (@dustymabe:
matrix.org, 15:42:02)
* INFO: Fedora CoreOS rawhide builds have temporarily reverted from
Konflux to the standard pipeline due to infrastructure flakiness and other
issues like log accessibility, with a return planned once the ongoing
Fedora Konflux cluster refresh is complete. (@marmijo:fedora.im, 15:47:33)
* TOPIC: tracker: Fedora 44 changes considerations (@marmijo:fedora.im,
15:48:20)
* LINK: https://github.com/coreos/fedora-coreos-tracker/issues/2063
(@marmijo:fedora.im, 15:48:24)
* LINK:
https://fedoraproject.org/wiki/Changes/AtomicDesktopDropPklaCompat
(@marmijo:fedora.im, 15:49:16)
* LINK: Bump Minimum Rust Bindgen to v0.72 (@marmijo:fedora.im,
15:50:35)
* LINK:
https://fedoraproject.org/wiki/Changes/Bump_Minimum_Rust_Bindgen_To_v0.72
(@marmijo:fedora.im, 15:50:44)
* TOPIC: Open Floor (@marmijo:fedora.im, 15:54:33)
* TOPIC: New Package Request: mstflint (@marmijo:fedora.im, 15:55:42)
* LINK: https://github.com/coreos/fedora-coreos-tracker/issues/2109
(@marmijo:fedora.im, 15:55:47)
* AGREED: Proxy the `mstflint` package request downstream in RHCOS and
do not include it in Fedora CoreOS at this time, as it introduces a Python
dependency for a platform (NVIDIA Bluefield) that FCOS currently doesn't
build media for. (@marmijo:fedora.im, 16:11:12)
* TOPIC: Open Floor (@marmijo:fedora.im, 16:11:50)
* INFO: we need more volunteers to participate in our [rotation](
https://hackmd.io/@4rqq1dsYSVuBswOHTKSIBA/ByDq2EK5p) to run our weekly
community meetings (this meeting). Reach out to me (marmijo) if you're
interested or have questions. (@marmijo:fedora.im, 16:22:58)
Meeting ended at 2026-02-11 16:26:00
Action items
------------
People Present (lines said)
---------------------------
* @marmijo:fedora.im (45)
* @dustymabe:matrix.org (34)
* @jcapitao:matrix.org (7)
* @zodbot:fedora.im (7)
* @jbtrystram:matrix.org (4)
* @ravanelli:matrix.org (3)
* @ydesouza:fedora.im (3)
* @peytonrobertson:matrix.org (2)
* @meetbot:fedora.im (2)
* @hricky:fedora.im (1)