service restart question

Hi,

I recently encountered a problem with the Openstack Quantum service. The
service was installed by doing the following steps:
     - sudo yum install openstack-quantum
     - sudo systemctl enable quantum-server.service
     - sudo systemctl start quantum-server.service
Due to a bug the service terminated.

The contents of the file
/etc/systemd/system/multi-user.target.wants/quantum-server.service is below:
[Unit]
Description=OpenStack Quantum Server
After=syslog.target network.target

[Service]
Type=simple
User=quantum
ExecStart=/usr/bin/quantum-server --config-file
/etc/quantum/quantum.conf --log-file /var/log/quantum/server.log
PrivateTmp=true

[Install]
WantedBy=multi-user.target

My understanding is that if there is a entry in the "Service" section
"Restart=always", then we can rely on systemd to restart the service if
it dies.

Can someone please explain or clarify why this is not the default value?
I can understand that this should not be set if there is another
"watcher" process that can restart a failed service.

Thanks in advance
Gary

[Reindl Harald]

Am 25.06.2012 15:57, schrieb Gary Kotton:
>  My understanding is that if there is a entry in the
> "Service" section
>  "Restart=always", then we can rely on systemd to restart the service if
>  it dies. 
true and that is why i am change each systemd-unit from early
F15 days on

>  Can someone please explain or clarify why this is not the default
> value?
>  I can understand that this should not be set if there is another
>  "watcher" process that can restart a failed service 
interesting question why the most useful feature of
systemd is wasted which has much more benefit than 2
seconds faster boot

there are only very rare cases where a enabled and started servcie
is expected to die and not start it again is fine for the user

[Lennart Poettering]

On Mon, 25.06.12 16:57, Gary Kotton (gkotton(a)redhat.com) wrote:

>  Hi,
>  
>  I recently encountered a problem with the Openstack Quantum service. The
>  service was installed by doing the following steps:
>      - sudo yum install openstack-quantum
>      - sudo systemctl enable quantum-server.service
>      - sudo systemctl start quantum-server.service
>  Due to a bug the service terminated.
>  
>  The contents of the file
>  /etc/systemd/system/multi-user.target.wants/quantum-server.service is below:
>  [Unit]
>  Description=OpenStack Quantum Server
>  After=syslog.target network.target
>  
>  [Service]
>  Type=simple
>  User=quantum
>  ExecStart=/usr/bin/quantum-server --config-file
>  /etc/quantum/quantum.conf --log-file /var/log/quantum/server.log
>  PrivateTmp=true
>  
>  [Install]
>  WantedBy=multi-user.target
>  
>  My understanding is that if there is a entry in the "Service" section
>  "Restart=always", then we can rely on systemd to restart the service if
>  it dies.
>  
>  Can someone please explain or clarify why this is not the default value?
>  I can understand that this should not be set if there is another
>  "watcher" process that can restart a failed service. 
Well, simply because we have no policy about it. I think it would make a
lot of sense however, to amend the fedora policy to suggest usage of
this by default for normal long running services.

Of course, there are some services where this functionality makes no
sense, for example all those which are of type "oneshot", i.e. are just
quickly started at boot to initialize something but then don't run any
longer.

Or Historically on SysV this functionality was not available, and so far
we only did the minimum packaging policy updates for translation of
SysV. We probably should fix that, but be careful to make this a
recommendation but not a requirement, and leave a lot of room for the
individual cases.

Also, we need to think about whether Restart=always or
Restart=on-failure or even Restart=on-abort is the best option to
suggest.

I think it would be great if somebody would file an FPC ticket about
this, so that the policy gets amended. But for that we'd first have to
make our mind up what the best option to recommend is.

Lennart

-- 
Lennart Poettering - Red Hat, Inc.

[Lennart Poettering]

On Mon, 25.06.12 16:13, Lennart Poettering (mzerqung(a)0pointer.de) wrote:

>  I think it would be great if somebody would file an FPC ticket about
>  this, so that the policy gets amended. But for that we'd first have to
>  make our mind up what the best option to recommend is. 
Hmm, ok, I decided to be that somebody and have now filed an FPC ticket
with my recommendations:

https://fedorahosted.org/fpc/ticket/191

Lennart

-- 
Lennart Poettering - Red Hat, Inc.

[Tom Lane]

Lennart Poettering <mzerqung(a)0pointer.de&gt; writes:
>  On Mon, 25.06.12 16:57, Gary Kotton (gkotton(a)redhat.com) wrote:
> > My understanding is that if there is a entry in the "Service" section
> > "Restart=always", then we can rely on systemd to restart the service if
> > it dies.
> > 
> > Can someone please explain or clarify why this is not the default value?
> > I can understand that this should not be set if there is another
> > "watcher" process that can restart a failed service. 
>  Well, simply because we have no policy about it. 
See also bug #832029 before being in too much of a hurry to decide that
this Must Be A Good Thing.  At minimum, it currently seems that we might
need per-service tuning of the restart timing parameters before being
sure that enabling restart is safe.  So while recommending that services
enable this after suitable testing *might* be a good idea, turning it on
by default seems like a horribly bad one.

			regards, tom lane

[Bruno Wolff III]

On Mon, Jun 25, 2012 at 10:30:00 -0400,
   Tom Lane <tgl(a)redhat.com&gt; wrote:
> See also bug #832029 before being in too much of a hurry to decide that
> this Must Be A Good Thing.  At minimum, it currently seems that we might
> need per-service tuning of the restart timing parameters before being
> sure that enabling restart is safe.  So while recommending that services
> enable this after suitable testing *might* be a good idea, turning it on
> by default seems like a horribly bad one. 
Since 832029 is not a public bug, can you give the gist of the issue?

[Tom Lane]

Bruno Wolff III <bruno(a)wolff.to&gt; writes:
>     Tom Lane <tgl(a)redhat.com&gt; wrote:
> > See also bug #832029 before being in too much of a hurry to decide that
> > this Must Be A Good Thing.  At minimum, it currently seems that we might
> > need per-service tuning of the restart timing parameters before being
> > sure that enabling restart is safe.  So while recommending that services
> > enable this after suitable testing *might* be a good idea, turning it on
> > by default seems like a horribly bad one. 
>  Since 832029 is not a public bug, can you give the gist of the issue?
Ah, sorry about that.  The deal is that mysqld has historically been
automatically restarted after crashes by a supervisor script
mysqld_safe.  When we went over to systemd-land we said "hey, systemd
can do that, and we'll have one less process required".  However,
it's not working so well:

(1) systemd is not able to distinguish a crash that should be restarted
from, say, failure due to misconfiguration in /etc/my.cnf.  (It's not
clear whether restart settings other than "always" would help here,
but in general it seems obvious that there are likely to be service-
specific reasons for restarting after some failures and not others.)

(2) Right now it appears that there is a bug in systemd that causes
it to ignore its respawn limits, such that if mysqld is indeed exiting
immediately due to misconfiguration, it gets restarted immediately.
Lather, rinse, repeat.  Indefinitely.

(3) Even if StartLimitInterval/StartLimitBurst were operating properly,
there are scenarios where mysqld will fail to start up, but be slow
enough about it (like a couple of seconds) that systemd's respawn
suppression logic would not get triggered, so it'd keep on restarting
it.  So we'd probably need custom-tuned values of those settings if we
decide to stay with using systemd for restart logic.

I assume that (2) is just a bug that's going to get fixed pretty quick.
But (1) and (3) seem like likely risk spots for other services.

In the meantime I'm seriously considering reverting to mysqld_safe.

			regards, tom lane

[Paul Wouters]

On Mon, 25 Jun 2012, Tom Lane wrote:

>  Subject: Re: service restart question 
>  (1) systemd is not able to distinguish a crash that should be
> restarted 
>  (2) Right now it appears that there is a bug in systemd that causes
>  it to ignore its respawn limits 
>  (3) Even if StartLimitInterval/StartLimitBurst were operating
> properly,
>  there are scenarios where mysqld will fail to start up, but be slow
>  enough about it (like a couple of seconds) that systemd's respawn
>  suppression logic would not get triggered, so it'd keep on restarting 
>  In the meantime I'm seriously considering reverting to
> mysqld_safe. 
Good to know, because I was thinking about the same thing for openswan's
wrapper script. It's a little more complicated there because it supports
an option plutorestartoncrash=yes|no. It's really nice for dev's to not
get a chain reaction of cores, but in production we really want it to
always restart. I was hoping systemd would support that properly for all
daemons now via generic options.

I'll keep an eye on this issue,

Paul

[Bill Nottingham]

Paul Wouters (pwouters(a)redhat.com) said: 
>  On Mon, 25 Jun 2012, Tom Lane wrote:
>  
>  >Subject: Re: service restart question
>  
>  >(1) systemd is not able to distinguish a crash that should be restarted
>  
>  >(2) Right now it appears that there is a bug in systemd that causes
>  >it to ignore its respawn limits
>  
>  >(3) Even if StartLimitInterval/StartLimitBurst were operating properly,
>  >there are scenarios where mysqld will fail to start up, but be slow
>  >enough about it (like a couple of seconds) that systemd's respawn
>  >suppression logic would not get triggered, so it'd keep on restarting
>  
>  >In the meantime I'm seriously considering reverting to mysqld_safe.
>  
>  Good to know, because I was thinking about the same thing for openswan's
>  wrapper script. It's a little more complicated there because it supports
>  an option plutorestartoncrash=yes|no. It's really nice for dev's to not
>  get a chain reaction of cores, but in production we really want it to
>  always restart. I was hoping systemd would support that properly for all
>  daemons now via generic options. 
I think it's a safe option that systemd would *like* to support this without
external helpers where possible. For your case, Restart=on-abort seems like
what you want?

Bill

[Lennart Poettering]

On Mon, 25.06.12 15:40, Tom Lane (tgl(a)redhat.com) wrote:

>  (1) systemd is not able to distinguish a crash that should be
> restarted
>  from, say, failure due to misconfiguration in /etc/my.cnf.  (It's not
>  clear whether restart settings other than "always" would help here,
>  but in general it seems obvious that there are likely to be service-
>  specific reasons for restarting after some failures and not others.) 
Hmm, it can actually do that. For example if you set Restart=on-abort
we'll restart mysql only if it crashes abnormally, via a signal -- but
it wouldn't be if mysql exits with exit(x) regardless what that x is.

Also, if you use Restart=always and a service terminates during its
initialization phase then we don't try restarts either (well, at least
in theory, there might be a bug in this, too). We'd only restart it if
it terminates during the normal runtime.

Anyway, I will spend some time today to make sure this all works
properly, as intended.

Lennart

-- 
Lennart Poettering - Red Hat, Inc.

[Honza Horak]

On 06/26/2012 11:01 AM, Lennart Poettering wrote:
>  Also, if you use Restart=always and a service terminates during its
>  initialization phase then we don't try restarts either (well, at least
>  in theory, there might be a bug in this, too). We'd only restart it if
>  it terminates during the normal runtime. 
That sounds interesting, but I wonder how systemd can distinguish 
between initialization phase and normal runtime?

Cheers,
Honza

[Lennart Poettering]

On Wed, 27.06.12 13:34, Honza Horak (hhorak(a)redhat.com) wrote:

>  
>  On 06/26/2012 11:01 AM, Lennart Poettering wrote:
>  >Also, if you use Restart=always and a service terminates during its
>  >initialization phase then we don't try restarts either (well, at least
>  >in theory, there might be a bug in this, too). We'd only restart it if
>  >it terminates during the normal runtime.
>  
>  That sounds interesting, but I wonder how systemd can distinguish
>  between initialization phase and normal runtime? 
This really depends on the service type. For classic SysV services the
init phase is the time spent between where we fork the daemon off and
it forked off and returned in the parent. This is in-line with how SysV
defined init phase for daemons. For bus services this is the time
between where the service started and finished taking the name on the
bus. And then there is also a way how services can notify the init
system about completed initialization by issuing sd_notify("READY=1").

For more information see the systemd.service(5), the section about
Type=.

Lennart

-- 
Lennart Poettering - Red Hat, Inc.

[Bill Nottingham]

Tom Lane (tgl(a)redhat.com) said: 
>  > Well, simply because we have no policy about it.
>  
>  See also bug #832029 before being in too much of a hurry to decide that
>  this Must Be A Good Thing.  At minimum, it currently seems that we might
>  need per-service tuning of the restart timing parameters before being
>  sure that enabling restart is safe.  So while recommending that services
>  enable this after suitable testing *might* be a good idea, turning it on
>  by default seems like a horribly bad one. 
Also, having different services have different restart settings can
certainly lead to confusion; as it stands now, the policy where no services
restart automatically, while possibly suboptimal, is at least predictable
for admins.

Bill