cobblerd dead but subsys locked

[Rodrique Heron]

My installation of  cobbler 1.6.2-1 fails to start when you issue "service
cobblerd start" command, when you check the status "service cobblerd
status", I get this output: cobblerd dead but snippets.

Any ideas? the logs for cobbler does  has no errors.

[Rodrique Heron]

On Thu, Apr 16, 2009 at 2:55 PM, Rodrique Heron  wrote:
> My installation of  cobbler 1.6.2-1 fails to start when you issue "service
> cobblerd start" command, when you check the status "service cobblerd
> status", I get this output: cobblerd dead but snippets.
>
> Any ideas? the logs for cobbler does  has no errors.
>
> Sorry, the error message I included was wrong, its :

[root@w-inf-cob-01 snippets]# service cobblerd status
cobblerd dead but subsys locked

I am on centos 5.2

Thanks

[Rodrique Heron]

On Thu, Apr 16, 2009 at 3:04 PM, Rodrique Heron  wrote:
> On Thu, Apr 16, 2009 at 2:55 PM, Rodrique Heron wrote:
> > My installation of  cobbler 1.6.2-1 fails to start when you issue "service
> > cobblerd start" command, when you check the status "service cobblerd
> > status", I get this output: cobblerd dead but snippets.
> >
> > Any ideas? the logs for cobbler does  has no errors.
> >
> > Sorry, the error message I included was wrong, its :
>
> [root@w-inf-cob-01 snippets]# service cobblerd status
> cobblerd dead but subsys locked
>
> I am on centos 5.2
>
> Thanks

I removed the lock and tried starting cobbler again, but got the same error.
So it appears cobbler stating then immediately crashing. There's nothing in
the logs.

[root@w-inf-cob-01 subsys]# rm /var/lock/subsys/cobblerd
rm: remove regular empty file `/var/lock/subsys/cobblerd'? y
[root@w-inf-cob-01 subsys]# service cobblerd status
cobblerd is stopped
[root@w-inf-cob-01 subsys]# service cobblerd start
Starting cobbler daemon:                                   [  OK  ]
[root@w-inf-cob-01 subsys]# service cobblerd status
cobblerd dead but subsys locked

[Michael DeHaan]

Rodrique Heron wrote:
> On Thu, Apr 16, 2009 at 3:04 PM, Rodrique Heron <[e-mail] 
> > wrote:
> On Thu, Apr 16, 2009 at 2:55 PM, Rodrique Heron
> <[e-mail] <mailto:[e-mail]>> wrote:
>
>     My installation of  cobbler 1.6.2-1 fails to start when you
>     issue "service cobblerd start" command, when you check the
>     status "service cobblerd status", I get this output: cobblerd
>     dead but snippets.
>
>     Any ideas? the logs for cobbler does  has no errors.
>
>
> Sorry, the error message I included was wrong, its :
>
> [root@w-inf-cob-01 snippets]# service cobblerd status
> cobblerd dead but subsys locked
>
> I am on centos 5.2
>
> Thanks
>
>
> I removed the lock and tried starting cobbler again, but got the same 
> error. So it appears cobbler stating then immediately crashing. 
> There's nothing in the logs.
>
> [root@w-inf-cob-01 subsys]# rm /var/lock/subsys/cobblerd
> rm: remove regular empty file `/var/lock/subsys/cobblerd'? y
> [root@w-inf-cob-01 subsys]# service cobblerd status
> cobblerd is stopped
> [root@w-inf-cob-01 subsys]# service cobblerd start
> Starting cobbler daemon:                                   [  OK  ]
> [root@w-inf-cob-01 subsys]# service cobblerd status
> cobblerd dead but subsys locked

This came up on IRC yesterday.  In this case, starting cobblerd with 
"--no-daemonize" showed that the address was already in use.

In which case you are probably using cobbler with manage_dhcp on, and 
dhcpd has restarted (from cobbler) and claimed cobbler's port (and lsof 
claims that dhcpd also owns cobblerd.log, which is nonsensical)

As a fit for this, cobbler 1.6.3 has replaced all "os.system" calls with 
usage of the subprocess module with close_fds=True, which, should, in 
theory, keep things a bit better behaved.

I'm not positive that resolves the problem, but we're testing that 
now.    Previously cobblerd daemonized itself with os.system.

[Rodrique Heron]

On Thu, Apr 16, 2009 at 3:22 PM, Michael DeHaan  wrote:
> Rodrique Heron wrote:
> > On Thu, Apr 16, 2009 at 3:04 PM, Rodrique Heron <[e-mail]
> > > wrote:
> > On Thu, Apr 16, 2009 at 2:55 PM, Rodrique Heron
> > <[e-mail] <mailto:[e-mail]>> wrote:
> >
> >     My installation of  cobbler 1.6.2-1 fails to start when you
> >     issue "service cobblerd start" command, when you check the
> >     status "service cobblerd status", I get this output: cobblerd
> >     dead but snippets.
> >
> >     Any ideas? the logs for cobbler does  has no errors.
> >
> >
> > Sorry, the error message I included was wrong, its :
> >
> > [root@w-inf-cob-01 snippets]# service cobblerd status
> > cobblerd dead but subsys locked
> >
> > I am on centos 5.2
> >
> > Thanks
> >
> >
> > I removed the lock and tried starting cobbler again, but got the same
> > error. So it appears cobbler stating then immediately crashing.
> > There's nothing in the logs.
> >
> > [root@w-inf-cob-01 subsys]# rm /var/lock/subsys/cobblerd
> > rm: remove regular empty file `/var/lock/subsys/cobblerd'? y
> > [root@w-inf-cob-01 subsys]# service cobblerd status
> > cobblerd is stopped
> > [root@w-inf-cob-01 subsys]# service cobblerd start
> > Starting cobbler daemon:                                   [  OK  ]
> > [root@w-inf-cob-01 subsys]# service cobblerd status
> > cobblerd dead but subsys locked
>
> This came up on IRC yesterday.  In this case, starting cobblerd with
> "--no-daemonize" showed that the address was already in use.
>
> In which case you are probably using cobbler with manage_dhcp on, and
> dhcpd has restarted (from cobbler) and claimed cobbler's port (and lsof
> claims that dhcpd also owns cobblerd.log, which is nonsensical)
>
> As a fit for this, cobbler 1.6.3 has replaced all "os.system" calls with
> usage of the subprocess module with close_fds=True, which, should, in
> theory, keep things a bit better behaved.
>
> I'm not positive that resolves the problem, but we're testing that
> now.    Previously cobblerd daemonized itself with os.system.

Thanks for the explanation, I am using cobbler with manage_dhcp on. This
isn't the first I time I ran experienced this issue, the last time, I backed
up my cobbler install and reinstalled. That's what I am doing now,
especially since I just noticed 1.6.2 is available in EPEL and I was using
your SRPM.

[Ian Meyer]

The workaround for this issue is after doing a cobbler sync from
either command line or the web interface was to sync then restart
dhcpd.

if you do a lsof | grep dhcpd, you will see
/var/log/cobbler/cobblerd.log is owned by dhcp, that's bad, as is
dhcpd owning localhost.localdomain:25151.. so check for those things
in the lsof | grep dhcpd output, and if you see neither of those
things, you should be good to go.

Hope that helps somewhat.
- Ian

On Thu, Apr 16, 2009 at 3:35 PM, Rodrique Heron  wrote:
> On Thu, Apr 16, 2009 at 3:22 PM, Michael DeHaan  wrote:
> > Rodrique Heron wrote:
> > > On Thu, Apr 16, 2009 at 3:04 PM, Rodrique Heron <[e-mail]
> > > > wrote:
> > >
> > > On Thu, Apr 16, 2009 at 2:55 PM, Rodrique Heron
> > >     <[e-mail] > wrote:
> > >
> > > My installation of  cobbler 1.6.2-1 fails to start when you
> > >         issue "service cobblerd start" command, when you check the
> > >         status "service cobblerd status", I get this output: cobblerd
> > >         dead but snippets.
> > >
> > > Any ideas? the logs for cobbler does  has no errors.
> > >
> > > Sorry, the error message I included was wrong, its :
> > >
> > > [root@w-inf-cob-01 snippets]# service cobblerd status
> > >     cobblerd dead but subsys locked
> > >
> > > I am on centos 5.2
> > >
> > > Thanks
> > >
> > > I removed the lock and tried starting cobbler again, but got the same
> > > error. So it appears cobbler stating then immediately crashing.
> > > There's nothing in the logs.
> > >
> > > [root@w-inf-cob-01 subsys]# rm /var/lock/subsys/cobblerd
> > > rm: remove regular empty file `/var/lock/subsys/cobblerd'? y
> > > [root@w-inf-cob-01 subsys]# service cobblerd status
> > > cobblerd is stopped
> > > [root@w-inf-cob-01 subsys]# service cobblerd start
> > > Starting cobbler daemon:                                   [  OK  ]
> > > [root@w-inf-cob-01 subsys]# service cobblerd status
> > > cobblerd dead but subsys locked
> >
> > This came up on IRC yesterday.  In this case, starting cobblerd with
> > "--no-daemonize" showed that the address was already in use.
> >
> > In which case you are probably using cobbler with manage_dhcp on, and
> > dhcpd has restarted (from cobbler) and claimed cobbler's port (and lsof
> > claims that dhcpd also owns cobblerd.log, which is nonsensical)
> >
> > As a fit for this, cobbler 1.6.3 has replaced all "os.system" calls with
> > usage of the subprocess module with close_fds=True, which, should, in
> > theory, keep things a bit better behaved.
> >
> > I'm not positive that resolves the problem, but we're testing that
> > now.    Previously cobblerd daemonized itself with os.system.
>
> Thanks for the explanation, I am using cobbler with manage_dhcp on. This
> isn't the first I time I ran experienced this issue, the last time, I backed
> up my cobbler install and reinstalled. That's what I am doing now,
> especially since I just noticed 1.6.2 is available in EPEL and I was using
> your SRPM.
>
> cobbler mailing list
> [e-mail]
> https://fedorahosted.org/mailman/listinfo/cobbler

[Rodrique Heron]

On Thu, Apr 16, 2009 at 3:38 PM, Ian Meyer  wrote:
> The workaround for this issue is after doing a cobbler sync from
> either command line or the web interface was to sync then restart
> dhcpd.
>
> if you do a lsof | grep dhcpd, you will see
> /var/log/cobbler/cobblerd.log is owned by dhcp, that's bad, as is
> dhcpd owning localhost.localdomain:25151.. so check for those things
> in the lsof | grep dhcpd output, and if you see neither of those
> things, you should be good to go.

Ian-

This worked for me, as reinstalling did not fix it. I stop cobbler, then
stop dhcpd, then started cobbler, and it worked.

Thanks

[Michael DeHaan]

Rodrique Heron wrote:
> On Thu, Apr 16, 2009 at 3:38 PM, Ian Meyer <[e-mail] 
> > wrote:
> The workaround for this issue is after doing a cobbler sync from
> either command line or the web interface was to sync then restart
> dhcpd.
>
> if you do a lsof | grep dhcpd, you will see
> /var/log/cobbler/cobblerd.log is owned by dhcp, that's bad, as is
> dhcpd owning localhost.localdomain:25151.. so check for those things
> in the lsof | grep dhcpd output, and if you see neither of those
> things, you should be good to go.
>
>
> Ian-
>
> This worked for me, as reinstalling did not fix it. I stop cobbler, 
> then stop dhcpd, then started cobbler, and it worked.
>
> Thanks
>
> cobbler mailing list
> [e-mail]
> https://fedorahosted.org/mailman/listinfo/cobbler

I'm curious if the aforementioned close_fds=True fix has any effect on 
this.   Seeing sockets /are/ file descriptors, I think it would prevent 
the assignment of the fd to dhcpd.

Latest git, master branch, for those that are interested.

--Michael

[Christian Horn]

On Thu, Apr 16, 2009 at 03:04:27PM -0400, Rodrique Heron wrote:
> On Thu, Apr 16, 2009 at 2:55 PM, Rodrique Heron  wrote:
> > My installation of  cobbler 1.6.2-1 fails to start when you issue "service
> > cobblerd start" command, when you check the status "service cobblerd
> > status", I get this output: cobblerd dead but snippets.
>
> [root@w-inf-cob-01 snippets]# service cobblerd status
> cobblerd dead but subsys locked
>
> I am on centos 5.2

Call the cobblerd-script in verbos-mode and interpret
the output, i.e.:

bash -x /etc/init.d/cobblerd start

Christian