killing running vms when sanlock daemon terminated

[Jim Fehlig]

[ resend after subscribing to list ]

Hi,

I've been playing with sanlock on SUSE distros and noticed that running
vms with disk leases controlled by sanlock are killed (with unclean
state on filesystems, databases, and block devices) when the sanlock
daemon is terminated. Actually, I noticed this when simply updating the
package, which caused a restart of the daemon.

Is there a way to control this behavior? I couldn't find anything with a
quick peek at the docs and code.

Regards,
Jim

[David Teigland]

On Thu, Jan 26, 2012 at 08:44:00AM -0700, Jim Fehlig wrote:
>  [ resend after subscribing to list ]
>  
>  Hi,
>  
>  I've been playing with sanlock on SUSE distros and noticed that running
>  vms with disk leases controlled by sanlock are killed (with unclean
>  state on filesystems, databases, and block devices) when the sanlock
>  daemon is terminated. Actually, I noticed this when simply updating the
>  package, which caused a restart of the daemon.
>  
>  Is there a way to control this behavior? I couldn't find anything with a
>  quick peek at the docs and code. 
Hi Jim, the init script currently uses SIGTERM to stop the daemon, which
is equivalent to sanlock shutdown -f1 (forces shutdown even if the daemon
is holding leases.)  I think what the init script should probably do
instead is shutdown -f0, so the shutdown is ignored if the daemon is
holding leases (we could also change SIGTERM to be the equivalent of -f0
instead of -f1.)  I think think -f0 (ignore shutdown when leases are held)
is the behavior everyone would prefer from the init script stop/restart
functions.

We don't have the option of leaving pids/vms running after the sanlock
daemon exits.  Because the daemon holds their leases, the resources they
are using are no longer protected when the daemon is gone.  This is why
the sanlock daemon kills the pids/vms before it exits.  And if the daemon
is SIGKILL'ed, then the watchdog comes to the rescue and resets the entire
host so the pids/vms will be killed that way.

[Jim Fehlig]

David Teigland wrote:
>  On Thu, Jan 26, 2012 at 08:44:00AM -0700, Jim Fehlig wrote:
>    
> > [ resend after subscribing to list ]
> >
> > Hi,
> >
> > I've been playing with sanlock on SUSE distros and noticed that running
> > vms with disk leases controlled by sanlock are killed (with unclean
> > state on filesystems, databases, and block devices) when the sanlock
> > daemon is terminated. Actually, I noticed this when simply updating the
> > package, which caused a restart of the daemon.
> >
> > Is there a way to control this behavior? I couldn't find anything with a
> > quick peek at the docs and code.
> >     
>
>  Hi Jim, the init script currently uses SIGTERM to stop the daemon, which
>  is equivalent to sanlock shutdown -f1 (forces shutdown even if the daemon
>  is holding leases.)  I think what the init script should probably do
>  instead is shutdown -f0, so the shutdown is ignored if the daemon is
>  holding leases (we could also change SIGTERM to be the equivalent of -f0
>  instead of -f1.)  I think think -f0 (ignore shutdown when leases are held)
>  is the behavior everyone would prefer from the init script stop/restart
>  functions.
>     
Perhaps the init script can check if the daemon is holding leases and
emit a message without stopping it, otherwise shutdown as it currently
does?  Looking at the docs, I'm not quite sure how to do this though.  I
don't think it is as simple as

  if $prog client status | egrep ^r > /dev/null; then
    echo -n "Resources controlled by sanlock, refusing to shutdown "
  else
    echo -n $"Stopping $prog: "
    killproc -p $lockfile $prog -TERM
  fi
  ...
  
>  We don't have the option of leaving pids/vms running after the
> sanlock
>  daemon exits.  Because the daemon holds their leases, the resources they
>  are using are no longer protected when the daemon is gone.  This is why
>  the sanlock daemon kills the pids/vms before it exits.  And if the daemon
>  is SIGKILL'ed, then the watchdog comes to the rescue and resets the entire
>  host so the pids/vms will be killed that way.
>     
Ah, understood.  Thanks for the explanation David!

Regards,
Jim