My experiences with KillUserProcesses=yes on F24

[Jason L Tibbitts III]

After reading (some of) the "discussion" about systemd-logind's
KillUserProcesses setting, I decided that I'd like to try enabling it
and see how it works and how I can make it useful in my environment.
Sorry, it's long, but I though folks might want to know.


Disclaimer:
I quite like systemd and I will try to avoid strong language like
"bizarre", "surprising", "WTF" and except in one case,
"bug".  This is
all based on systemd 229 in F24; I know 230 changes one thing and maybe
it or future versions will change enough that the issues in this
document go away.


Background:
I run a university department network a bunch of desktops and a pile of
other machines accessed by a few hundred users for various functions.
Around 200 machines total.  Where appropriate, I'd like to make sure
that user sessions get shut down properly even when sessions hang or get
confused at startup, but I'd also like to preserve the possibility of
users running background jobs on the desktops (which are often quite
powerful and useful for computation) even after they log out.


So:
I enabled KillUserProcesses on a desktop and rebooted.  Some testing
shows that it works as documented and kills processes at logout.  But as
with any change, I ran into some problems.


Problem 1:
Need to give users a way to run persistent background jobs without
requiring them to learn systemd-run.


Solution 1:
Provide wrappers in /usr/local/bin for nohup, screen and a couple of
local utilities which use nice and nohup internally.  Those wrappers
call systemd-run --user --scope.


Problem 2:
Even under systemd-run --user --scope, things don't persist unless
"linger" is enabled for the user.


Solution 2:
Add loginctl enable-linger to the wrappers.


Problem 3:
loginctl enable-linger requires root privs (in F24's systemd 229; seems
that 230 allows self-lingering by default).


Solution 3:
Add this in /etc/polkit-1/rules.d/50-user-linger.rules:

polkit.addRule(function(action, subject) {
    if (action.id == "org.freedesktop.login1.set-user-linger") {
        return polkit.Result.YES;
    }
});

Now users can run loginctl enable-linger.  For any user.  Oh, well, I
can live with that for now.  And things work!  But....


Problem 4:
"linger" does more than just allow sessions to persist.

A) It changes the way the "user manager" for that user runs.  Without
   linger, a user manager is started when the log in, and it goes away
   when their last process exits (which given KillUserProcesses is
   whenever they log out).  When linger is enabled for a user (in
   many curcumstances, at least), a user manager is immediately started
   if the user isn't already logged in which will never exit.  This has
   interesting consequences.

   I've found that if the user's user manager dies (for any reason you
   might choose) and linger is enable for them, a new one won't be
   started at login.  They have to disable linger, log out, and log back
   in.  Or reboot the machine.

B) It enables "user units" (probably the wrong terminology) which let
   users start up things which run periodically, or at boot, and which
   run under their UIDs.  In order to make this work, the user manager
   will run immediately at system boot time and look in
   ~/.config/systemd for units.

These seem like good ideas, except that:

i) I don't necessarily want users to be able to start units at boot
   time, but that's OK because....

ii) I have home dirs on kerberized NFS.  So this automounts the home
    directories of every lingering user at boot time (which is a problem
    for me) and that directory can't be read anyway, even by the proper
    UID, if the user doesn't have a kerberos ticket.  Plus the network
    isn't necessarily up to the point where user homedirs can be
    accessed at the time when systemd starts the user managers.
    Fortunately in the default case, it doesn't hold open a reference to
    the homedir so the automounter will remove it.  It can still cause a
    lot of mounts, which can take some time and puts more load on my
    already loaded file servers.

    The bottom line there is user sessions aren't going to work in some
    environments, period.


Solution4A:
I don't have one....  If I'm the one killing their processes (for
whatever reason I might have), I have to make sure to disable lingering
for them at the same time.  But if they kill their processes (kill -9 -1
to just terminate everything, if you want to test) then they're
screwed.  They have to disable linger, log out and log back in so that a
user manager is created for them.  Then they can enable linger again.
This just has to be a bug, so I've filed:
https://bugzilla.redhat.com/show_bug.cgi?id=1371721


Solution4B:
Add the following to /etc/systemd/system/delete-lingers.service:

[Unit]
Description=Delete lingering users
Before=network.target

[Service]
Type=oneshot
ExecStart=/usr/bin/rm -rf /var/lib/systemd/linger

[Install]
WantedBy=multi-user.target

Now at each boot, no users are set to linger and no user managers will
be started until login.  The wrapper scripts will re-enable that as
necessary so this doesn't hurt anything.


Conclusion: I can make this work, mostly, unless someones user manager
happens to die.  But really, this is all an unpleasant hack,
necessitated by a mismatch between systemd's design and what I'm trying
to accomplish.  And I don't think that what I'm trying to accomplish is
particularly unreasonable.

What I wish for is for some "property", let's call it "persist" to
be
"attached" to a scope in some way (presumably a flag to systemd-run)
which does nothing other than to indicate that the scope will continue
to run after the user's session has terminated.  This wouldn't be a
persistent user setting.  Nothing would start at boot.  The user manager
would start up if necessary at login (even if it had previously been
killed) and persist until all user processes in any scope have exited.

I am perfectly happy with wrappers around programs which indicate that
something is to persist after logins so my users don't have to learn
systemd-run.  I don't think systemd itself needs to know or care which
processes should persist.  I don't care if those wrappers are in the
base system.  If things like nohup or screen are patched to do this
automatically, I'm happy with that but it makes no difference to me.

Hope this is interesting to someone and adds useful content to the
discussion.

 - J<

[Jason L Tibbitts III]

> >>>> "JLT" == Jason L Tibbitts
> <tibbs(a)math.uh.edu&gt; writes: 
JLT>    I've found that if the user's user manager dies (for any reason
JLT>    you might choose) and linger is enable for them, a new one won't
JLT>    be started at login.  They have to disable linger, log out, and
JLT>    log back in.  Or reboot the machine.

I should add an explanation here.

This is a problem because without a user manager, systemd-run errors
out with "Failed to create bus connection: Connection refused".

Your processes are still killed when the session terminates, so if you
get into this state you basically can't run persistent jobs at all.

 - J<

[Jóhann B. Guðmundsson]

On 08/31/2016 02:25 PM, Jason L Tibbitts III wrote:

More appropriate place would be to post this upstream either on the 
mailinglist and or as an bug/rfs in the tracker so this issues can be 
addressed properly.

Lingering is a per-user thing so it probably ends up being an per user 
opt in feature/fix

JBG

[Matthew Miller]

On Wed, Aug 31, 2016 at 09:25:22AM -0500, Jason L Tibbitts III wrote:
>  Hope this is interesting to someone and adds useful content to the
>  discussion. 
I think it does — thanks for your real-world observations!

-- 
Matthew Miller
<mattdm(a)fedoraproject.org&gt;
Fedora Project Leader

[Nico Kadel-Garcia]

On Wed, Aug 31, 2016 at 10:25 AM, Jason L Tibbitts III
<tibbs(a)math.uh.edu&gt; wrote:
>  After reading (some of) the "discussion" about
> systemd-logind's
>  KillUserProcesses setting, I decided that I'd like to try enabling it
>  and see how it works and how I can make it useful in my environment.
>  Sorry, it's long, but I though folks might want to know. 
Thanks for the thoughtful work.

>  Now at each boot, no users are set to linger and no user managers
> will
>  be started until login.  The wrapper scripts will re-enable that as
>  necessary so this doesn't hurt anything. 
Wrapper scripts. *Yech*.

To me, it looks like systemd is activating a universal policy which
you, and others, need more thoughtful and tunable handling for.
Attempting to tune it by outsmarting systemd's default behavior looks
expensive and unstable.

>  Conclusion: I can make this work, mostly, unless someones user
> manager
>  happens to die.  But really, this is all an unpleasant hack,
>  necessitated by a mismatch between systemd's design and what I'm trying
>  to accomplish.  And I don't think that what I'm trying to accomplish is
>  particularly unreasonable. 
Agreed. The acceptance of the feature without any logging, whatsoever,
of what processes it killed and when shocked me. It's the sort of
breakage of working, stable tools in the name of "resource management"
that gets people fired..

>  What I wish for is for some "property", let's call it
> "persist" to be
>  "attached" to a scope in some way (presumably a flag to systemd-run)
>  which does nothing other than to indicate that the scope will continue
>  to run after the user's session has terminated.  This wouldn't be a
>  persistent user setting.  Nothing would start at boot.  The user manager
>  would start up if necessary at login (even if it had previously been
>  killed) and persist until all user processes in any scope have exited. 
Wouldn't an hourly or nightly cron job reporting such tasks, with an
option to kill them on a group or user based policy basis, be safer if
such a daemon is needed?

>  I am perfectly happy with wrappers around programs which indicate
> that
>  something is to persist after logins so my users don't have to learn
>  systemd-run.  I don't think systemd itself needs to know or care which
>  processes should persist.  I don't care if those wrappers are in the
>  base system.  If things like nohup or screen are patched to do this
>  automatically, I'm happy with that but it makes no difference to me. 
Can the tasks run with cron jobs in your environment? I realize that
Kerberized access to user home directories might prevent working
access to scripts that live in their home directory.

>  Hope this is interesting to someone and adds useful content to the
>  discussion. 
The thoughtful hands-on experience is welcome. It confirms the idea
that in multi-user environment, backgrounded sessions continuing after
logout is a common and effectively used feature worthy of support.

[Matthew Miller]

On Thu, Sep 01, 2016 at 04:13:13AM -0400, Nico Kadel-Garcia wrote:
>  Agreed. The acceptance of the feature without any logging,
> whatsoever,
>  of what processes it killed and when shocked me. It's the sort of 
But note that we _didn't accept the feature without any logging. And
see https://bugzilla.redhat.com/show_bug.cgi?id=1357426#c3:

     Zbigniew Jędrzejewski-Szmek 2016-07-26 16:38:40 EDT

     Systemd-231 was compiled in rawhide today, with the first bit of
     implementation for this change: logging of left-over processes
     killed with SIGKILL/SIGABRT at notice level.

     I'm still working on the other parts.


-- 
Matthew Miller
<mattdm(a)fedoraproject.org&gt;
Fedora Project Leader

[Jason L Tibbitts III]

> >>>> "NK" == Nico Kadel-Garcia
> <nkadel(a)gmail.com&gt; writes: 
NK> To me, it looks like systemd is activating a universal policy which
NK> you, and others, need more thoughtful and tunable handling for.
NK> Attempting to tune it by outsmarting systemd's default behavior
NK> looks expensive and unstable.

Well, please keep in mind that _I_ am the one who is activating this.  I
don't care one way or the other what the default is; a single line in a
file in /etc/systemd/logind.conf.d sets that the way I like.

NK> Agreed. The acceptance of the feature without any logging,
NK> whatsoever, of what processes it killed and when shocked me.

I'm not talking about any feature or any acceptance.  This is entirely
something I am trying, on my own, because I was intrigued by the
possibility of enforced process death at session termination.

I am making no commentary on the viability of any of this for Fedora in
general.  I'm just sharing my experiences.  Perhaps others can glean
something useful from them in some relation to what Fedora may or may
not do in the future, and that's fine, but I'm not trying to influence
that in any way.

> > What I wish for is for some "property", let's call
> it "persist" to be
> > "attached" to a scope in some way (presumably a flag to systemd-run)
> > which does nothing other than to indicate that the scope will
> > continue to run after the user's session has terminated.  This
> > wouldn't be a persistent user setting.  Nothing would start at boot.
> > The user manager would start up if necessary at login (even if it had
> > previously been killed) and persist until all user processes in any
> > scope have exited. 
NK> Wouldn't an hourly or nightly cron job reporting such tasks, with an
NK> option to kill them on a group or user based policy basis, be safer
NK> if such a daemon is needed?

I'm not entirely sure what you're suggesting.

If you're saying that I could just a cron job cleaning up jobs that
shouldn't be there, I am not certain that such a thing is possible.
Plus I've experienced a problem where users get dropped back to the
login box after a timeout, with their session destroyed but processes
still hanging around in such a way that they're prevented from logging
in a second time.  Hourly cleanup of these things is far too slow, and
sub-minute-ly would be kind of a waste.  KillUserProcesses=yes appears
to handle this case just fine as far as I've been able to test this.
(It's tough to reproduce, and does involve the confluence of other
bugs.)

NK> Can the tasks run with cron jobs in your environment? I realize that
NK> Kerberized access to user home directories might prevent working
NK> access to scripts that live in their home directory.

Well, teaching users to do that would be beyond my means.  (I'm only
human, after all.)  Something like Condor is a far better option for
using desktop cycles in the background, but it's tough to beat the
"nohup my_job" that they learned out of some book about Fortran 25 years
ago.

NK> The thoughtful hands-on experience is welcome. It confirms the idea
NK> that in multi-user environment, backgrounded sessions continuing
NK> after logout is a common and effectively used feature worthy of
NK> support.

Well, keep in mind that it will always be supported as long as you can
still set KillUserProcesses=no.  I'm trying to get the best of both
worlds, to the extent that is possible, and I'm willing to hack things
to get it.  I'm not asking anyone else to support that, but I am trying
to inform them of what they're up against should they want the same
thing.  I do have a suspicion that I'm not the only person who would
want this, though, which is why I sent the message in the first place.

 - J<