Fedora 32 System-Wide Change proposal: Restart services at end of rpm transaction

[Ben Cotton]

https://fedoraproject.org/wiki/Changes/Restart_services_at_end_of_rpm_tra...

== Summary ==

Scriptlets to restart each service that should be restarted in each
rpm package will be replaced by a declaration in the unit file and an
rpm transaction trigger that fires at the end and restarts all
services.

== Owner ==
* Name: [[User:zbyszek| Zbigniew Jędrzejewski-Szmek]]
* Email: zbyszek at in waw pl


== Detailed Description ==

Currently, when packages containing systemd services are upgraded,
they are restarted through %post scriptlets in each package. In other
words, the scriptlets specify which services should be restarted and
actually run the command to restart the service. An alternative
mechanism will be provided, whereby the unit file contains a setting
which specifies that the service should be restarted on upgrade, and a
%transfiletrigger in the systemd package will restart all services
marked so in upgraded packages at the end of transaction.

To mark a services as "restart on upgrade" the unit file should
contain an option to mark it so. This can be either in the main
.service file, or e.g. in a drop-in file in
<code>/usr/lib/systemd/system/<service>.service.d/needs-restart.conf</code>.
The exact name of the option should be something like
<code>X-restart-on-upgrade=true|false</code>. I'll take the proposal
for discussion in systemd upstream, since this is something that could
be used across distributions, and then the "X-" prefix could be
dropped and/or the name changed.

== Benefit to Fedora ==

Which services should be restarted is specified declaratively and the
number of scriptlets is reduced.

Admins may easily override this by providing appropriate drop-ins of
their own of higher priority.

Services are restarted after the transaction is finished, all
libraries have been upgraded, and systemd configuration reloaded. This
solves https://bugzilla.redhat.com/show_bug.cgi?id=1614751, but also a
more general problem: a service may be restarted before any of its
dependencies (files and static content and whatnot) have been
upgraded, and while some contents from the old rpm are still on disk.
Restarting after all packages have been upgraded avoids this issue.

In the future, restarting of services shall also be extended to user
managers (i.e. to restart all user services after the packages
providing those services have been upgraded). This is not part of this
proposal, but moving the information what to restart into systemd
units makes this much easier.

== Scope ==
* Proposal owners:
** implement the relevant bits in the systemd package
** submit a proposal to FPC
** convert a subset of packages

* Other developers:
** FPC: review (and accept ;)) the guidelines changes
** other maintainers: convert other packages

* Release engineering: n/a

* Policies and guidelines: a pull request will be submitted

* Trademark approval: N/A (not needed for this Change)

== Upgrade/compatibility impact ==
This change should be backwards compatible, i.e. unconverted packages
can be still installed on new systems, and services will be restarted
on upgrade like before. Converted packages may be installed on older
systems but will not get restarted on upgrades.

== How To Test ==
This change should be mostly invisible to users. Check that during
upgrades services are restarted and that no warnings are emitted.

== User Experience ==
N/A

== Dependencies ==
N/A

== Contingency Plan ==

* Contingency mechanism: Revert to previous mechanism. This will
require a revert of changes to the spec file and a rebuild of the
package.

* Contingency deadline: beta freeze
* Blocks release? No
* Blocks product? No

== Documentation ==
TBD.

== Release Notes ==
Not needed.

-- 
Ben Cotton
He / Him / His
Fedora Program Manager
Red Hat
TZ=America/Indiana/Indianapolis

[Igor Gnatenko]

I really love this idea because users (sysadmins) will get the ability
to *not* restart some production-critical services on upgrade if they
decide so without rebuilding package with removed scriptlets.

I don't have suggestions about syntax, but whatever systemd upstream
comes up with is good for me.

On Thu, Jan 2, 2020 at 4:16 PM Ben Cotton <bcotton(a)redhat.com&gt; wrote:
>
> https://fedoraproject.org/wiki/Changes/Restart_services_at_end_of_rpm_tra...
>
> == Summary ==
>
> Scriptlets to restart each service that should be restarted in each
> rpm package will be replaced by a declaration in the unit file and an
> rpm transaction trigger that fires at the end and restarts all
> services.
>
> == Owner ==
> * Name: [[User:zbyszek| Zbigniew Jędrzejewski-Szmek]]
> * Email: zbyszek at in waw pl
>
>
> == Detailed Description ==
>
> Currently, when packages containing systemd services are upgraded,
> they are restarted through %post scriptlets in each package. In other
> words, the scriptlets specify which services should be restarted and
> actually run the command to restart the service. An alternative
> mechanism will be provided, whereby the unit file contains a setting
> which specifies that the service should be restarted on upgrade, and a
> %transfiletrigger in the systemd package will restart all services
> marked so in upgraded packages at the end of transaction.
>
> To mark a services as "restart on upgrade" the unit file should
> contain an option to mark it so. This can be either in the main
> .service file, or e.g. in a drop-in file in
>
<code>/usr/lib/systemd/system/<service>.service.d/needs-restart.conf</code>.
> The exact name of the option should be something like
> <code>X-restart-on-upgrade=true|false</code>. I'll take the proposal
> for discussion in systemd upstream, since this is something that could
> be used across distributions, and then the "X-" prefix could be
> dropped and/or the name changed.
>
> == Benefit to Fedora ==
>
> Which services should be restarted is specified declaratively and the
> number of scriptlets is reduced.
>
> Admins may easily override this by providing appropriate drop-ins of
> their own of higher priority.
>
> Services are restarted after the transaction is finished, all
> libraries have been upgraded, and systemd configuration reloaded. This
> solves https://bugzilla.redhat.com/show_bug.cgi?id=1614751, but also a
> more general problem: a service may be restarted before any of its
> dependencies (files and static content and whatnot) have been
> upgraded, and while some contents from the old rpm are still on disk.
> Restarting after all packages have been upgraded avoids this issue.
>
> In the future, restarting of services shall also be extended to user
> managers (i.e. to restart all user services after the packages
> providing those services have been upgraded). This is not part of this
> proposal, but moving the information what to restart into systemd
> units makes this much easier.
>
> == Scope ==
> * Proposal owners:
> ** implement the relevant bits in the systemd package
> ** submit a proposal to FPC
> ** convert a subset of packages
>
> * Other developers:
> ** FPC: review (and accept ;)) the guidelines changes
> ** other maintainers: convert other packages
>
> * Release engineering: n/a
>
> * Policies and guidelines: a pull request will be submitted
>
> * Trademark approval: N/A (not needed for this Change)
>
> == Upgrade/compatibility impact ==
> This change should be backwards compatible, i.e. unconverted packages
> can be still installed on new systems, and services will be restarted
> on upgrade like before. Converted packages may be installed on older
> systems but will not get restarted on upgrades.
>
> == How To Test ==
> This change should be mostly invisible to users. Check that during
> upgrades services are restarted and that no warnings are emitted.
>
> == User Experience ==
> N/A
>
> == Dependencies ==
> N/A
>
> == Contingency Plan ==
>
> * Contingency mechanism: Revert to previous mechanism. This will
> require a revert of changes to the spec file and a rebuild of the
> package.
>
> * Contingency deadline: beta freeze
> * Blocks release? No
> * Blocks product? No
>
> == Documentation ==
> TBD.
>
> == Release Notes ==
> Not needed.
>
> --
> Ben Cotton
> He / Him / His
> Fedora Program Manager
> Red Hat
> TZ=America/Indiana/Indianapolis
> _______________________________________________
> devel-announce mailing list -- devel-announce(a)lists.fedoraproject.org
> To unsubscribe send an email to devel-announce-leave(a)lists.fedoraproject.org
> Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
https://lists.fedoraproject.org/archives/list/[e-mail]...

[Neal Gompa]

On Thu, Jan 2, 2020 at 10:11 AM Ben Cotton <bcotton(a)redhat.com&gt; wrote:
>  https://fedoraproject.org/wiki/Changes/Restart_services_at_end_of_rpm_tra...
>
>  == Summary ==
>
>  Scriptlets to restart each service that should be restarted in each
>  rpm package will be replaced by a declaration in the unit file and an
>  rpm transaction trigger that fires at the end and restarts all
>  services.
>
>  == Owner ==
>  * Name: [[User:zbyszek| Zbigniew Jędrzejewski-Szmek]]
>  * Email: zbyszek at in waw pl
>
>
>  == Detailed Description ==
>
>  Currently, when packages containing systemd services are upgraded,
>  they are restarted through %post scriptlets in each package. In other
>  words, the scriptlets specify which services should be restarted and
>  actually run the command to restart the service. An alternative
>  mechanism will be provided, whereby the unit file contains a setting
>  which specifies that the service should be restarted on upgrade, and a
>  %transfiletrigger in the systemd package will restart all services
>  marked so in upgraded packages at the end of transaction.
>
>  To mark a services as "restart on upgrade" the unit file should
>  contain an option to mark it so. This can be either in the main
>  .service file, or e.g. in a drop-in file in
>
> <code>/usr/lib/systemd/system/<service>.service.d/needs-restart.conf</code>.
>  The exact name of the option should be something like
>  <code>X-restart-on-upgrade=true|false</code>. I'll take the proposal
>  for discussion in systemd upstream, since this is something that could
>  be used across distributions, and then the "X-" prefix could be
>  dropped and/or the name changed.
>
>  == Benefit to Fedora ==
>
>  Which services should be restarted is specified declaratively and the
>  number of scriptlets is reduced.
>
>  Admins may easily override this by providing appropriate drop-ins of
>  their own of higher priority.
>
>  Services are restarted after the transaction is finished, all
>  libraries have been upgraded, and systemd configuration reloaded. This
>  solves https://bugzilla.redhat.com/show_bug.cgi?id=1614751, but also a
>  more general problem: a service may be restarted before any of its
>  dependencies (files and static content and whatnot) have been
>  upgraded, and while some contents from the old rpm are still on disk.
>  Restarting after all packages have been upgraded avoids this issue.
>
>  In the future, restarting of services shall also be extended to user
>  managers (i.e. to restart all user services after the packages
>  providing those services have been upgraded). This is not part of this
>  proposal, but moving the information what to restart into systemd
>  units makes this much easier.
>  
OpenMandriva actually implemented a variation of this two years ago,
as well as a way to do preset runs through file triggers (more or less
eliminating every single scriptlet for systemd units for ~90% of
packages). I considered bringing this over to Fedora, but the
mechanisms in systemd seem to still be a bit wobbly.

Even with this proposal, how do we deal with system upgrades? In the
system upgrade case, we don't really want these restarts to run,
because the system will be rebooted anyway... These kinds of questions
are why I didn't propose porting over what we did in OpenMandriva to
Fedora. It's rather wasteful in a rather common case...



--
真実はいつも一つ！/ Always, there's only one truth!

[Zbigniew Jędrzejewski-Szmek]

On Thu, Jan 02, 2020 at 10:45:07AM -0500, Neal Gompa wrote:
>  On Thu, Jan 2, 2020 at 10:11 AM Ben Cotton <bcotton(a)redhat.com&gt;
> wrote:
>  >
>  > https://fedoraproject.org/wiki/Changes/Restart_services_at_end_of_rpm_tra...
>  >
>  > == Summary ==
>  >
>  > Scriptlets to restart each service that should be restarted in each
>  > rpm package will be replaced by a declaration in the unit file and an
>  > rpm transaction trigger that fires at the end and restarts all
>  > services.
>  >
>  > == Owner ==
>  > * Name: [[User:zbyszek| Zbigniew Jędrzejewski-Szmek]]
>  > * Email: zbyszek at in waw pl
>  >
>  >
>  > == Detailed Description ==
>  >
>  > Currently, when packages containing systemd services are upgraded,
>  > they are restarted through %post scriptlets in each package. In other
>  > words, the scriptlets specify which services should be restarted and
>  > actually run the command to restart the service. An alternative
>  > mechanism will be provided, whereby the unit file contains a setting
>  > which specifies that the service should be restarted on upgrade, and a
>  > %transfiletrigger in the systemd package will restart all services
>  > marked so in upgraded packages at the end of transaction.
>  >
>  > To mark a services as "restart on upgrade" the unit file should
>  > contain an option to mark it so. This can be either in the main
>  > .service file, or e.g. in a drop-in file in
>  >
> <code>/usr/lib/systemd/system/<service>.service.d/needs-restart.conf</code>.
>  > The exact name of the option should be something like
>  > <code>X-restart-on-upgrade=true|false</code>. I'll take the
> proposal
>  > for discussion in systemd upstream, since this is something that could
>  > be used across distributions, and then the "X-" prefix could be
>  > dropped and/or the name changed.
>  >
>  > == Benefit to Fedora ==
>  >
>  > Which services should be restarted is specified declaratively and the
>  > number of scriptlets is reduced.
>  >
>  > Admins may easily override this by providing appropriate drop-ins of
>  > their own of higher priority.
>  >
>  > Services are restarted after the transaction is finished, all
>  > libraries have been upgraded, and systemd configuration reloaded. This
>  > solves https://bugzilla.redhat.com/show_bug.cgi?id=1614751, but also a
>  > more general problem: a service may be restarted before any of its
>  > dependencies (files and static content and whatnot) have been
>  > upgraded, and while some contents from the old rpm are still on disk.
>  > Restarting after all packages have been upgraded avoids this issue.
>  >
>  > In the future, restarting of services shall also be extended to user
>  > managers (i.e. to restart all user services after the packages
>  > providing those services have been upgraded). This is not part of this
>  > proposal, but moving the information what to restart into systemd
>  > units makes this much easier.
>  >
>  
>  OpenMandriva actually implemented a variation of this two years ago,
>  as well as a way to do preset runs through file triggers (more or less
>  eliminating every single scriptlet for systemd units for ~90% of
>  packages). I considered bringing this over to Fedora, but the
>  mechanisms in systemd seem to still be a bit wobbly. 
We don't do presets through file triggers because we the scriptlet is
conditionalized to only do the preset on initial package installation. But
if it is possible to do something equivalent (or better?) through a file
trigger, that'd be great. I'm all ears.

>  Even with this proposal, how do we deal with system upgrades? In the
>  system upgrade case, we don't really want these restarts to run,
>  because the system will be rebooted anyway... These kinds of questions
>  are why I didn't propose porting over what we did in OpenMandriva to
>  Fedora. It's rather wasteful in a rather common case... 
I'm not sure if skipping restarts on upgrade is worth the trouble. It
should be quite simple to implement (e.g. by having the restart scriptlet
check if the systemd target has been diverted to system-update.target),
but restarting of services would most likely take only a small fraction
of time required for the upgrade anyway.

Zbyszek

[Richard Shaw]

On Thu, Jan 2, 2020 at 10:14 AM Zbigniew Jędrzejewski-Szmek <
zbyszek(a)in.waw.pl&gt; wrote:

>  On Thu, Jan 02, 2020 at 10:45:07AM -0500, Neal Gompa wrote:
>  > On Thu, Jan 2, 2020 at 10:11 AM Ben Cotton <bcotton(a)redhat.com&gt; wrote:
>  > Even with this proposal, how do we deal with system upgrades? In the
>  > system upgrade case, we don't really want these restarts to run,
>  > because the system will be rebooted anyway... These kinds of questions
>  > are why I didn't propose porting over what we did in OpenMandriva to
>  > Fedora. It's rather wasteful in a rather common case...
>
>  I'm not sure if skipping restarts on upgrade is worth the trouble. It
>  should be quite simple to implement (e.g. by having the restart scriptlet
>  check if the systemd target has been diverted to system-update.target),
>  but restarting of services would most likely take only a small fraction
>  of time required for the upgrade anyway.
>  
Maybe it should only restart the service if it's already running. In the
case of an upgrade it's a special target, right?

Actually that's probably a good idea anyway. If I have stopped a service
(even if it's enabled) then it should not be restarted on upgrade.

Thanks,
Richard

[Adam Williamson]

On Thu, 2020-01-02 at 11:18 -0600, Richard Shaw wrote:
>  On Thu, Jan 2, 2020 at 10:14 AM Zbigniew Jędrzejewski-Szmek <
>  zbyszek(a)in.waw.pl&gt; wrote:
>  
>  > On Thu, Jan 02, 2020 at 10:45:07AM -0500, Neal Gompa wrote:
>  > > On Thu, Jan 2, 2020 at 10:11 AM Ben Cotton <bcotton(a)redhat.com&gt; wrote:
>  > > Even with this proposal, how do we deal with system upgrades? In the
>  > > system upgrade case, we don't really want these restarts to run,
>  > > because the system will be rebooted anyway... These kinds of questions
>  > > are why I didn't propose porting over what we did in OpenMandriva to
>  > > Fedora. It's rather wasteful in a rather common case...
>  > 
>  > I'm not sure if skipping restarts on upgrade is worth the trouble. It
>  > should be quite simple to implement (e.g. by having the restart scriptlet
>  > check if the systemd target has been diverted to system-update.target),
>  > but restarting of services would most likely take only a small fraction
>  > of time required for the upgrade anyway.
>  > 
>  
>  Maybe it should only restart the service if it's already running. In the
>  case of an upgrade it's a special target, right?
>  
>  Actually that's probably a good idea anyway. If I have stopped a service
>  (even if it's enabled) then it should not be restarted on upgrade. 
This is exactly what 'try-restart' does: it restarts the service only
if it's running. Per the docs:

       try-restart PATTERN...
           Stop and then start one or more units specified on the command line if the
units are running. This does
           nothing if units are not running.

The current %systemd_postun_with_restart scriptlet uses try-restart:

[adamw@adam koji ((koji-1.19.1))]$ rpm --eval "%systemd_postun_with_restart
foo"

 
if [ $1 -ge 1 ] && [ -x /usr/bin/systemctl ] ; then 
        # Package upgrade, not uninstall 
        /usr/bin/systemctl try-restart foo || : 
fi 

The Change page doesn't *specifically* state what scriptlets 
it's intended to replace, but I'd assume it would be that one. Thus I'd
expect that this mechanism would also use try-restart.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net

[Zbigniew Jędrzejewski-Szmek]

On Thu, Jan 02, 2020 at 09:39:38AM -0800, Adam Williamson wrote:
>  On Thu, 2020-01-02 at 11:18 -0600, Richard Shaw wrote:
>  > On Thu, Jan 2, 2020 at 10:14 AM Zbigniew Jędrzejewski-Szmek <
>  > zbyszek(a)in.waw.pl&gt; wrote:
>  > 
>  > > On Thu, Jan 02, 2020 at 10:45:07AM -0500, Neal Gompa wrote:
>  > > > On Thu, Jan 2, 2020 at 10:11 AM Ben Cotton <bcotton(a)redhat.com&gt;
> wrote:
>  > > > Even with this proposal, how do we deal with system upgrades? In the
>  > > > system upgrade case, we don't really want these restarts to run,
>  > > > because the system will be rebooted anyway... These kinds of questions
>  > > > are why I didn't propose porting over what we did in OpenMandriva to
>  > > > Fedora. It's rather wasteful in a rather common case...
>  > > 
>  > > I'm not sure if skipping restarts on upgrade is worth the trouble. It
>  > > should be quite simple to implement (e.g. by having the restart scriptlet
>  > > check if the systemd target has been diverted to system-update.target),
>  > > but restarting of services would most likely take only a small fraction
>  > > of time required for the upgrade anyway.
>  > > 
>  > 
>  > Maybe it should only restart the service if it's already running. In the
>  > case of an upgrade it's a special target, right?
>  > 
>  > Actually that's probably a good idea anyway. If I have stopped a service
>  > (even if it's enabled) then it should not be restarted on upgrade.
>  
>  This is exactly what 'try-restart' does: it restarts the service only
>  if it's running. Per the docs:
>  
>         try-restart PATTERN...
>             Stop and then start one or more units specified on the command line if the
> units are running. This does
>             nothing if units are not running.
>  
>  The current %systemd_postun_with_restart scriptlet uses try-restart:
>  
>  [adamw@adam koji ((koji-1.19.1))]$ rpm --eval "%systemd_postun_with_restart
> foo"
>  
>   
>  if [ $1 -ge 1 ] && [ -x /usr/bin/systemctl ] ; then 
>          # Package upgrade, not uninstall 
>          /usr/bin/systemctl try-restart foo || : 
>  fi 
>  
>  The Change page doesn't *specifically* state what scriptlets 
>  it's intended to replace, but I'd assume it would be that one. Thus I'd
>  expect that this mechanism would also use try-restart. 
Yes, exactly. Thank you for the very nice summary.
More generally, the goal is to do the exact same restarts that are done
now, but at a different point in the transaction and from a different
call site.

Zbyszek

[Björn Persson]

>  The exact name of the option should be something like
>  <code>X-restart-on-upgrade=true|false</code>. I'll take the proposal
>  for discussion in systemd upstream, since this is something that could
>  be used across distributions, and then the "X-" prefix could be
>  dropped and/or the name changed.
>  [...]
>  ** other maintainers: convert other packages 
Please don't push hard to get this widely adopted before the name is
final. Don't make packagers change their packages and then have them
change the name later. Let the feature remain experimental and optional
until the permanent name has been decided and implemented.

Other than that, the proposal sounds good to me.

Björn Persson