F39 Change Proposal: Automatic Cloud Reboot on Updates (Self-Contained Change)

[Aoife Moloney]

https://fedoraproject.org/wiki/Changes/Automatic_Cloud_Reboot_On_Updates

This document represents a proposed Change. As part of the Changes
process, proposals are publicly announced in order to receive
community feedback. This proposal will only be implemented if approved
by the Fedora Engineering Steering Committee.


== Summary ==
Cloud users can provide cloud-init metadata when creating a Fedora
cloud instance and that metadata can contain instructions to update
all packages on the system and reboot the system if any of those
updated packages need a reboot to go into effect. Fedora cloud
instances should write the `/var/run/reboot-required` file if a reboot
is needed after a dnf update so that cloud-init can reboot the
instance.

This issue originally surfaced in
[https://bugzilla.redhat.com/show_bug.cgi?id=1275409 RHBZ 1275409].

== Owner ==

* Name: [[User:mhayden| Major Hayden]]
* Email: major(a)redhat.com



== Detailed Description ==

Fedora cloud instances use cloud-init to do the initial configuration
of the instance. This includes setting up networking, assigning a
hostname, adding users/groups, and arbitrary scripts. There are also
two options that you can pass to cloud-init that are important for
this change:

* `package_update`: If set to `true`, all installed packages are
immediately updated on first boot
* `package_reboot_if_required`: If set to `true`, and the
`package_update` step wrote to `/var/run/reboot-required`, reboot the
system immediately after updating packages

📚 For more details, see cloud-init's module reference for
`[https://cloudinit.readthedocs.io/en/latest/reference/modules.html#package-update-upgrade-install
package_update]`.

🚨 '''WAIT A MOMENT. ARE WE TALKING ABOUT REBOOTING EVERY CLOUD
INSTANCE ON BOOT?''' 🚨 No! This change would require all three of
these things to happen before a reboot occurs:

* User provides `package_update: true` on instance creation
* '''AND''' user provides `package_reboot_if_required: true` on
instance creation
* '''AND''' `tracer` notices that at least one of the packages
need a
reboot to go into effect

🤔 '''Where does this `/var/run/reboot-required` file come from?'''
On
Debian and Ubuntu systems, `apt` automatically writes to
`/var/run/reboot-required` if a reboot is needed after a package
update. From there, `cloud-init` looks for the file
([https://github.com/canonical/cloud-init/blob/6d09df5e4786a2a6c79d6098ab41...
relevant cloud-init code]) and if present, reboots the system
immediately.

✏️ '''How do we write this file on Fedora?''' Fedora systems have
a
package called `tracer` and a corresponding dnf plugin,
`python3-dnf-plugin-tracer`, that analyzes `dnf` updates and provides
recommendations on reboots or user logouts to bring updates into
effect on the system. A recent
[https://github.com/FrostyX/tracer/pull/196 pull request] added
support for writing the `/var/run/reboot-required` file when a system
reboot is recommended. The `cloud-init` tool can read this file after
a package update and reboot if needed.

🔎 '''What does `tracer`'s output look like?'''

    [root@tracer-testing ~]# tracer
    You should restart:
    * Some applications using:
        sudo systemctl restart NetworkManager
        sudo systemctl restart auditd
        sudo systemctl restart chronyd
        sudo systemctl restart dbus-broker
        sudo systemctl restart qemu-guest-agent
        sudo systemctl restart sshd
        sudo systemctl restart systemd-journald
        sudo systemctl restart systemd-logind
        sudo systemctl restart systemd-oomd
        sudo systemctl restart systemd-resolved
        sudo systemctl restart systemd-udevd
        sudo systemctl restart systemd-userdbd

    * These applications manually:
        (sd-pam)

    Additionally, there are:
    - 3 processes requiring restart of your session (i.e. Logging out
& Logging in again)
    - 1 processes requiring reboot
    [root@tracer-testing ~]# cat /var/run/reboot-required
    Tracer says reboot is required

📋 '''What do we need to do?''' Add the
`python3-dnf-plugin-tracer`
plugin to Fedora cloud images. No additional configuration is
necessary. This action pulls in five packages that are about 2.1MB
after installation:

   
=======================================================================================
    Package                               Arch       Version
  Repository  Size
   
=======================================================================================
    Installing:
    python3-dnf-plugin-tracer             noarch     4.1.0-1.fc38
  fedora      14 k
    Installing dependencies:
    python3-dnf-plugins-extras-common     noarch     4.1.0-1.fc38
  fedora      69 k
    python3-psutil                        x86_64     5.9.2-2.fc38
  fedora     271 k
    python3-tracer                        noarch     0.7.8-5.fc38
  fedora     172 k
    tracer-common                         noarch     0.7.8-5.fc38
  fedora      22 k

    Transaction Summary
   
=======================================================================================
    Install  5 Packages

    Total download size: 547 k
    Installed size: 2.1 M

== Feedback ==

One of the other ideas was to patch `cloud-init` to run `tracer`
directly and avoid the `/var/run/reboot-required` file altogether.
That would require a lot of work upstream in `cloud-init` to enable
the functionality and we would still need the same set of packages
installed in Fedora anyway. 🥵

== Benefit to Fedora ==

This change allows Fedora cloud instances to behave in the same way
that Debian-based instances already behave. When users request package
updates with a reboot now, `cloud-init` performs the update but never
reboots the system. This is an unexpected and confusing result for
users who come to Fedora from other distributions.

Rebooting automatically could also reduce the attack surface of an
instance that just came online since it would immediately reboot to
put all package updates into effect on the system. This reduces the
time that an unpatched instance is online prior to being fully
patched.

== Scope ==
* Proposal owners: This change is fairly isolated and only affects
Fedora cloud users who request package updates followed by a reboot in
their `cloud-init` metadata.

* Other developers: N/A

* Release engineering: N/A

* Policies and guidelines: N/A

* Trademark approval: N/A

* Alignment with Community Initiatives: N/A

== Upgrade/compatibility impact ==

Since this change only applies to `cloud-init` on the very first boot
of the instance, this wouldn't affect a user upgrading from one
version of Fedora to the next.

== How To Test ==

# Ensure you have a cloud image that has an update that needs a reboot
(kernel, openssl, etc)
# Boot an instance with the following `cloud-init` user data:

    #cloud-config
    package_update: true
    package_upgrade: true
    package_reboot_if_required: true

# Wait for the package updates to finish on the instance and verify
that it rebooted after updating

== User Experience ==

First, if a user never uses the `package_upgrade` and
`package_reboot_if_required` options in their `cloud-init` user data,
they won't be affected by this change. These options are not enabled
in `cloud-init` by default.

If a user does enable both of these options, they will see their cloud
instance come online, apply updates, and reboot if required. Most
cloud providers have very fast reboots, so the delay should not be a
problem.

== Dependencies ==

Nothing depends on this change.

== Contingency Plan ==

* Contingency mechanism: Push to Fedora 40 if the work cannot be done in time
* Contingency deadline: N/A
* Blocks release? N/A

== Documentation ==

Guidance for users in a blog post (Fedora Magazine) could be helpful
for this change. Many users might not be aware that they had the
option to ask for package updates and reboots via `cloud-init` for
their Fedora cloud instances.

== Release Notes ==

Fedora cloud instances now automatically reboot when a user requests
package updates followed by a reboot on the first boot of the
instance. The reboot only occurs if an updated package requires a
reboot to go into effect (such as a kernel or critical system
library).





-- 
Aoife Moloney

Product Owner

Community Platform Engineering Team

Red Hat EMEA

Communications House

Cork Road

Waterford

[Aoife Moloney]

This proposal has now been submitted to FESCo https://pagure.io/fesco/issue/3010
_______________________________________________
devel-announce mailing list -- devel-announce(a)lists.fedoraproject.org
To unsubscribe send an email to devel-announce-leave(a)lists.fedoraproject.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/[e-mail]...
Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue

[Zbigniew Jędrzejewski-Szmek]

On Tue, May 30, 2023 at 07:57:12PM +0100, Aoife Moloney wrote:
> https://fedoraproject.org/wiki/Changes/Automatic_Cloud_Reboot_On_Updates
>  == Summary ==
>  Cloud users can provide cloud-init metadata when creating a Fedora
>  cloud instance and that metadata can contain instructions to update
>  all packages on the system and reboot the system if any of those
>  updated packages need a reboot to go into effect. Fedora cloud
>  instances should write the `/var/run/reboot-required` file if a reboot
>  is needed after a dnf update so that cloud-init can reboot the
>  instance. 
I'll add the same comment that I made in https://github.com/FrostyX/tracer/pull/196:

The correct location is /run, not /var/run. In most cases, this
doesn't matter greatly, because a compat symlink is provided, but when
/var is a separate partition, /var/run would only be accessible if
/var is mounted, i.e. rather late in boot. /run is always available.
/var/run is a compat symlink that should not be used (esp. in new code).

Stuff that is related to initial machine installation and such should
not force /var to be mounted.

Zbyszek