[coreos-status] Some Fedora CoreOS nodes may fail to upgrade

[Benjamin Gilbert]

Hi all,

We've recently encountered two separate problems causing nodes to get stuck
on an old release of Fedora CoreOS.  Both issues should primarily affect
machines which have been offline for an extended period.  Nevertheless, you
may wish to check whether any of your nodes are unexpectedly running a
CoreOS release older than 20200715.  If so, see below for mitigation
instructions.

Details
-------

There are two issues:

1.  Due to an interaction with ostree downgrade protection, older versions
of Fedora CoreOS could fail to apply updates [1] when there are update
barriers [2] in the update graph.  We applied a server-side workaround [3]
for several releases after the bug was fixed, but some nodes that were
offline during that period will not successfully update to current releases.

2.  In April we began migrating the Cincinnati update service from the
Fedora staging cluster at updates.coreos.stg.fedoraproject.org to the
production cluster at updates.coreos.fedoraproject.org.  The staging cluster
was removed from service as part of the Fedora datacenter move [4], and
there are no short-term plans [5] to restore it.  Any nodes that have not
updated since the migration will fail to update to current releases.

Mitigation
----------

If you have nodes affected by either problem, the following commands should
correct the issue:

sudo tee /run/zincati/config.d/99-fixes.toml << EOF
[cincinnati]
base_url = "https://updates.coreos.fedoraproject.org"
[updates]
allow_downgrade = true
EOF
sudo systemctl restart zincati.service

This procedure does not make any persistent configuration changes to the
system, so no further action is necessary after the node updates.

If you have any questions, you can connect with us on the CoreOS mailing
list [6], Fedora Discourse [7], or #fedora-coreos on Freenode.

Thanks,
--Benjamin Gilbert

[1]: https://github.com/coreos/fedora-coreos-tracker/issues/481
[2]:
https://github.com/coreos/fedora-coreos-tracker/blob/master/metadata/upda...
[3]: https://github.com/coreos/fedora-coreos-tracker/issues/481#issuecomment-6...
[4]:
https://lists.fedoraproject.org/archives/list/[e-mail]...
[5]:
https://lists.fedoraproject.org/archives/list/[e-mail]...
[6]: https://lists.fedoraproject.org/archives/list/[e-mail]....
[7]: https://discussion.fedoraproject.org/c/server/coreos