Error occurs when I try to change the log level in sanlock v2.7 and fedora 18. Error Info: lockfile setlk error /var/run/sanlock/sanlock.pid: Resource temporarily unavailable

[Qixiaozhen]

Hi,all

There were no packages of rpm of sanlock v2.7 for fedora 18 in the internet. I built these packages manually on fedora 18.

The source code and spec file were downloaded in the web. http://arm.koji.fedoraproject.org//packages/sanlock/2.7/1.fc19/src/sanlock-2...http://arm.koji.fedoraproject.org/packages/sanlock/2.7/1.fc19/src/sanlock-2.7-1.fc19.src.rpm

These packages were installed in fedora 18 successfully, however, if I tried to change the log level to DEBUG with the command 'sanlock daemon -L 7', one lockfile error was found in the logfile /var/log/sanlock.log.

The logfile said:
2013-07-29 16:00:16-0400 338 [1317]: sanlock daemon started 2.7 host 48f0a719-03ff-4fc8-b069-46532dcddb94.sanlock-de
2013-07-29 16:00:50-0400 372 [1369]: lockfile setlk error /var/run/sanlock/sanlock.pid: Resource temporarily unavailable

Its code fragment in src/lockfile.c of sanlock v2.7:
         lock.l_type = F_WRLCK;
         lock.l_start = 0;
         lock.l_whence = SEEK_SET;
         lock.l_len = 0;

rv = fcntl(fd, F_SETLK, &lock);
         if (rv < 0) {
                   log_error("lockfile setlk error %s: %s",
                              path, strerror(errno));
                   goto fail;
         }

Is there anyone meet this problem? Can the package named sanlock-2.7.tar.gz in sanlock-2.7-1.fc19.src.rpm be used directly in Fedora18?

Thanks for your help.

Qi

[David Teigland]

On Mon, Jul 29, 2013 at 08:23:26AM +0000, Qixiaozhen wrote:
> 2013-07-29 16:00:50-0400 372 [1369]: lockfile setlk error /var/run/sanlock/sanlock.pid: Resource temporarily unavailable
>      rv = fcntl(fd, F_SETLK, &lock);
>      if (rv < 0) {
>                log_error("lockfile setlk error %s: %s",
>                           path, strerror(errno));
>                goto fail;
>      }

EAGAIN means that another process is holding that lock.
Did you verify that no other sanlock daemon is running?
You could also try "lsof /var/run/sanlock/sanlock.pid".
Dave

Two daemons existing and more details. RE: Error occurs when I try to change the log level in sanlock v2.7 and fedora 18. Error Info: lockfile setlk error /var/run/sanlock/sanlock.pid: Resource temporarily unavailable

[Qixiaozhen]

Hi, David,

Thank you for your reply.

There are two daemons named " sanlock daemon -U sanlock -G sanlock" running in the host, however, those daemons of sanlock have the relationship between parent and child.

Like this:
[root@localhost sanlock2.7_rpm_f18]# ps -ef | grep sanlock
sanlock    959     1  0 21:16 ?        00:00:00 sanlock daemon -U sanlock -G sanlock
root       960     1  0 21:16 ?        00:00:00 wdmd -G sanlock
root       962   959  0 21:16 ?        00:00:00 sanlock daemon -U sanlock -G sanlock

More details list here:
[root@localhost sanlock2.7_rpm_f18]#
[root@localhost sanlock2.7_rpm_f18]# ps -ef | grep sanlock
root       937   783  0 21:15 pts/0    00:00:00 grep --color=auto sanlock
[root@localhost sanlock2.7_rpm_f18]#
[root@localhost sanlock2.7_rpm_f18]# service sanlock start
Redirecting to /bin/systemctl start  sanlock.service
[root@localhost sanlock2.7_rpm_f18]#
[root@localhost sanlock2.7_rpm_f18]# ps -ef | grep sanlock
sanlock    959     1  0 21:16 ?        00:00:00 sanlock daemon -U sanlock -G sanlock
root       960     1  0 21:16 ?        00:00:00 wdmd -G sanlock
root       962   959  0 21:16 ?        00:00:00 sanlock daemon -U sanlock -G sanlock
root       969   783  0 21:16 pts/0    00:00:00 grep --color=auto sanlock
[root@localhost sanlock2.7_rpm_f18]#
[root@localhost sanlock2.7_rpm_f18]# lsof /var/run/sanlock/sanlock.pid
COMMAND PID    USER   FD   TYPE DEVICE SIZE/OFF  NODE NAME
sanlock 959 sanlock    6wW  REG   0,17        4 16287 /var/../run/sanlock/sanlock.pid
[root@localhost sanlock2.7_rpm_f18]#
[root@localhost sanlock2.7_rpm_f18]#
[root@localhost sanlock2.7_rpm_f18]# ps -ef | grep 959
sanlock    959     1  0 21:16 ?        00:00:00 sanlock daemon -U sanlock -G sanlock
root       962   959  0 21:16 ?        00:00:00 sanlock daemon -U sanlock -G sanlock
root      1041   783  0 21:26 pts/0    00:00:00 grep --color=auto 959
[root@localhost sanlock2.7_rpm_f18]#
[root@localhost sanlock2.7_rpm_f18]#
[root@localhost sanlock2.7_rpm_f18]#
[root@localhost sanlock2.7_rpm_f18]# fuser -m /var/run/sanlock/sanlock.pid
/run/sanlock/sanlock.pid:     1   206   209m   330   331   332   334   344   676   711   778   959   960
[root@localhost sanlock2.7_rpm_f18]# ps -ef | grep 206 | grep -v grep
root       206     1  0 21:08 ?        00:00:00 /usr/lib/systemd/systemd-udevd
root       282   206  0 21:08 ?        00:00:00 /usr/lib/systemd/systemd-udevd
root       283   206  0 21:08 ?        00:00:00 /usr/lib/systemd/systemd-udevd
[root@localhost sanlock2.7_rpm_f18]# ps -ef | grep 209 | grep -v grep
root       209     1  0 21:08 ?        00:00:00 /usr/lib/systemd/systemd-journald
[root@localhost sanlock2.7_rpm_f18]# ps -ef | grep 330 | grep -v grep
root       330     1  0 21:08 ?        00:00:00 /sbin/rsyslogd -n -c 5
[root@localhost sanlock2.7_rpm_f18]# ps -ef | grep 331 | grep -v grep
root       331     1  0 21:08 ?        00:00:00 /usr/lib/systemd/systemd-logind
[root@localhost sanlock2.7_rpm_f18]# ps -ef | grep 332 | grep -v grep
dbus       332     1  0 21:08 ?        00:00:00 /bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
[root@localhost sanlock2.7_rpm_f18]# ps -ef | grep 334 | grep -v grep
root       334     1  0 21:08 ?        00:00:00 /usr/sbin/crond -n
[root@localhost sanlock2.7_rpm_f18]# ps -ef | grep 344 | grep -v grep
root       344     1  0 21:08 ?        00:00:00 login -- root
root       721   344  0 21:08 tty1     00:00:00 -bash
[root@localhost sanlock2.7_rpm_f18]# ps -ef | grep 676 | grep -v grep
root       676     1  0 21:08 ?        00:00:00 sendmail: accepting connections
[root@localhost sanlock2.7_rpm_f18]# ps -ef | grep 711 | grep -v grep
smmsp      711     1  0 21:08 ?        00:00:00 sendmail: Queue runner@01:00:00 for /var/spool/clientmqueue
[root@localhost sanlock2.7_rpm_f18]# ps -ef | grep 778 | grep -v grep
root       778   653  0 21:11 ?        00:00:00 sshd: root@pts/0
root       783   778  0 21:11 pts/0    00:00:00 -bash
[root@localhost sanlock2.7_rpm_f18]# ps -ef | grep 959 | grep -v grep
sanlock    959     1  0 21:16 ?        00:00:00 sanlock daemon -U sanlock -G sanlock
root       962   959  0 21:16 ?        00:00:00 sanlock daemon -U sanlock -G sanlock
[root@localhost sanlock2.7_rpm_f18]# ps -ef | grep 960 | grep -v grep
root       960     1  0 21:16 ?        00:00:00 wdmd -G sanlock
[root@localhost sanlock2.7_rpm_f18]#
[root@localhost sanlock2.7_rpm_f18]# sanlock daemon -L 7
[root@localhost sanlock2.7_rpm_f18]#
[root@localhost sanlock2.7_rpm_f18]# tail /var/log/sanlock.log
2013-07-30 21:16:07+0800 473 [959]: sanlock daemon started 2.7 host 408f9139-5a7b-423c-81ba-48310373f8be.localhost.
2013-07-30 21:36:10+0800 1676 [1059]: lockfile setlk error /var/run/sanlock/sanlock.pid: Resource temporarily unavailable
[root@localhost sanlock2.7_rpm_f18]#

The source was rebuild and packaged with the src and spec in http://arm.koji.fedoraproject.org//packages/sanlock/2.7/1.fc19/src/sanlock-2... .

Thank you.

Qi

-----Original Message-----
From: David Teigland [mailto:[e-mail]] 
Sent: Tuesday, July 30, 2013 12:47 AM
To: Qixiaozhen
Cc: [e-mail]
Subject: Re: Error occurs when I try to change the log level in sanlock v2.7 and fedora 18. Error Info: lockfile setlk error /var/run/sanlock/sanlock.pid: Resource temporarily unavailable

On Mon, Jul 29, 2013 at 08:23:26AM +0000, Qixiaozhen wrote:
> 2013-07-29 16:00:50-0400 372 [1369]: lockfile setlk error /var/run/sanlock/sanlock.pid: Resource temporarily unavailable
>      rv = fcntl(fd, F_SETLK, &lock);
>      if (rv < 0) {
>                log_error("lockfile setlk error %s: %s",
>                           path, strerror(errno));
>                goto fail;
>      }

EAGAIN means that another process is holding that lock.
Did you verify that no other sanlock daemon is running?
You could also try "lsof /var/run/sanlock/sanlock.pid".
Dave

Two daemons existing and more details. RE: Error occurs when I try to change the log level in sanlock v2.7 and fedora 18. Error Info: lockfile setlk error /var/run/sanlock/sanlock.pid: Resource temporarily unavailable

[David Teigland]

On Tue, Jul 30, 2013 at 02:00:32AM +0000, Qixiaozhen wrote:
> [root@localhost sanlock2.7_rpm_f18]# service sanlock start
> Redirecting to /bin/systemctl start  sanlock.service
> [root@localhost sanlock2.7_rpm_f18]# ps -ef | grep sanlock
> sanlock    959     1  0 21:16 ?        00:00:00 sanlock daemon -U sanlock -G sanlock
> root       962   959  0 21:16 ?        00:00:00 sanlock daemon -U sanlock -G sanlock
> [root@localhost sanlock2.7_rpm_f18]# lsof /var/run/sanlock/sanlock.pid
> COMMAND PID    USER   FD   TYPE DEVICE SIZE/OFF  NODE NAME
> sanlock 959 sanlock    6wW  REG   0,17        4 16287 /var/../run/sanlock/sanlock.pid
> [root@localhost sanlock2.7_rpm_f18]# sanlock daemon -L 7
> [root@localhost sanlock2.7_rpm_f18]# tail /var/log/sanlock.log
> 2013-07-30 21:16:07+0800 473 [959]: sanlock daemon started 2.7 host 408f9139-5a7b-423c-81ba-48310373f8be.localhost.
> 2013-07-30 21:36:10+0800 1676 [1059]: lockfile setlk error /var/run/sanlock/sanlock.pid: Resource temporarily unavailable

Perhaps you misunderstand what "sanlock daemon -L 7" does.
Anything that begins "sanlock daemon" is a command to start a
new daemon, which fails here because you already started it.

I suspect you want this command to change the logging level of
the existing daemon, but that is not currently possible.

Dave

Two daemons existing and more details. RE: Error occurs when I try to change the log level in sanlock v2.7 and fedora 18. Error Info: lockfile setlk error /var/run/sanlock/sanlock.pid: Resource temporarily unavailable

[Qixiaozhen]

Sorry, I hava misunderstood the usage of the 'sanlock daemon'.  In fact, I want to change the logging level dynamically with the 'sanlock' command.

The function of different commands:
sanlock daemon --- start daemon
sanlock client --- send request to daemon (default command if none given)
sanlock direct --- access storage directly (no coordination with daemon)

Thank you, Dave

-----Original Message-----
From: David Teigland [mailto:[e-mail]] 
Sent: Tuesday, July 30, 2013 10:14 PM
To: Qixiaozhen
Cc: [e-mail]
Subject: Re: Two daemons existing and more details. RE: Error occurs when I try to change the log level in sanlock v2.7 and fedora 18. Error Info: lockfile setlk error /var/run/sanlock/sanlock.pid: Resource temporarily unavailable

On Tue, Jul 30, 2013 at 02:00:32AM +0000, Qixiaozhen wrote:
> [root@localhost sanlock2.7_rpm_f18]# service sanlock start
> Redirecting to /bin/systemctl start  sanlock.service
> [root@localhost sanlock2.7_rpm_f18]# ps -ef | grep sanlock
> sanlock    959     1  0 21:16 ?        00:00:00 sanlock daemon -U sanlock -G sanlock
> root       962   959  0 21:16 ?        00:00:00 sanlock daemon -U sanlock -G sanlock
> [root@localhost sanlock2.7_rpm_f18]# lsof /var/run/sanlock/sanlock.pid
> COMMAND PID    USER   FD   TYPE DEVICE SIZE/OFF  NODE NAME
> sanlock 959 sanlock    6wW  REG   0,17        4 16287 /var/../run/sanlock/sanlock.pid
> [root@localhost sanlock2.7_rpm_f18]# sanlock daemon -L 7
> [root@localhost sanlock2.7_rpm_f18]# tail /var/log/sanlock.log
> 2013-07-30 21:16:07+0800 473 [959]: sanlock daemon started 2.7 host 408f9139-5a7b-423c-81ba-48310373f8be.localhost.
> 2013-07-30 21:36:10+0800 1676 [1059]: lockfile setlk error /var/run/sanlock/sanlock.pid: Resource temporarily unavailable

Perhaps you misunderstand what "sanlock daemon -L 7" does.
Anything that begins "sanlock daemon" is a command to start a
new daemon, which fails here because you already started it.

I suspect you want this command to change the logging level of
the existing daemon, but that is not currently possible.

Dave

The same to sanlock v3.0. RE: Two daemons existing and more details. RE: Error occurs when I try to change the log level in sanlock v2.7 and fedora 18. Error Info: lockfile setlk error /var/run/sanlock/sanlock.pid: Resource temporarily unavailable

[Qixiaozhen]

Hi, David

I have traced this problem today.

Both sanlock v2.7 and v3.0 are the same in fedora 18.

[root@localhost x86_64]# rpm -qa | grep sanlock
sanlock-lib-3.0.0-1.fc18.x86_64
fence-sanlock-3.0.0-1.fc18.x86_64
sanlock-python-3.0.0-1.fc18.x86_64
sanlock-3.0.0-1.fc18.x86_64
sanlock-devel-3.0.0-1.fc18.x86_64
[root@localhost x86_64]#
[root@localhost x86_64]# sanlock daemon -w 0
[root@localhost x86_64]# tail /var/log/sanlock.log
2013-07-31 09:10:42+0800 35338 [5380]: lockfile setlk error /var/run/sanlock/sanlock.pid: Resource temporarily unavailable
[root@localhost x86_64]# ps -ef | grep sanlock
root      5328     1  0 09:08 ?        00:00:00 wdmd -G sanlock
sanlock   5330     1  0 09:08 ?        00:00:00 sanlock daemon -U sanlock -G sanlock
root      5333  5330  0 09:08 ?        00:00:00 sanlock daemon -U sanlock -G sanlock
root      5385  5267  0 09:11 pts/1    00:00:00 grep --color=auto sanlock
[root@localhost x86_64]#
[root@localhost x86_64]# fuser -m /var/run/sanlock/sanlock.pid
/run/sanlock/sanlock.pid:     1   207   210m   330   331   332   334   688   704   878  5263  5328  5330
[root@localhost x86_64]#
[root@localhost x86_64]# lsof /var/run/sanlock/sanlock.pid
COMMAND  PID    USER   FD   TYPE DEVICE SIZE/OFF  NODE NAME
sanlock 5330 sanlock    6wW  REG   0,17        5 16588 /var/../run/sanlock/sanlock.pid
[root@localhost x86_64]#
[root@localhost x86_64]#

How can I solve this problem?

Appreciate for your reply. Thank you.

Qi

-----Original Message-----
From: Qixiaozhen 
Sent: Tuesday, July 30, 2013 10:01 AM
To: 'David Teigland'
Cc: [e-mail]
Subject: Two daemons existing and more details. RE: Error occurs when I try to change the log level in sanlock v2.7 and fedora 18. Error Info: lockfile setlk error /var/run/sanlock/sanlock.pid: Resource temporarily unavailable

Hi, David,

Thank you for your reply.

There are two daemons named " sanlock daemon -U sanlock -G sanlock" running in the host, however, those daemons of sanlock have the relationship between parent and child.

Like this:
[root@localhost sanlock2.7_rpm_f18]# ps -ef | grep sanlock
sanlock    959     1  0 21:16 ?        00:00:00 sanlock daemon -U sanlock -G sanlock
root       960     1  0 21:16 ?        00:00:00 wdmd -G sanlock
root       962   959  0 21:16 ?        00:00:00 sanlock daemon -U sanlock -G sanlock

More details list here:
[root@localhost sanlock2.7_rpm_f18]#
[root@localhost sanlock2.7_rpm_f18]# ps -ef | grep sanlock
root       937   783  0 21:15 pts/0    00:00:00 grep --color=auto sanlock
[root@localhost sanlock2.7_rpm_f18]#
[root@localhost sanlock2.7_rpm_f18]# service sanlock start
Redirecting to /bin/systemctl start  sanlock.service
[root@localhost sanlock2.7_rpm_f18]#
[root@localhost sanlock2.7_rpm_f18]# ps -ef | grep sanlock
sanlock    959     1  0 21:16 ?        00:00:00 sanlock daemon -U sanlock -G sanlock
root       960     1  0 21:16 ?        00:00:00 wdmd -G sanlock
root       962   959  0 21:16 ?        00:00:00 sanlock daemon -U sanlock -G sanlock
root       969   783  0 21:16 pts/0    00:00:00 grep --color=auto sanlock
[root@localhost sanlock2.7_rpm_f18]#
[root@localhost sanlock2.7_rpm_f18]# lsof /var/run/sanlock/sanlock.pid
COMMAND PID    USER   FD   TYPE DEVICE SIZE/OFF  NODE NAME
sanlock 959 sanlock    6wW  REG   0,17        4 16287 /var/../run/sanlock/sanlock.pid
[root@localhost sanlock2.7_rpm_f18]#
[root@localhost sanlock2.7_rpm_f18]#
[root@localhost sanlock2.7_rpm_f18]# ps -ef | grep 959
sanlock    959     1  0 21:16 ?        00:00:00 sanlock daemon -U sanlock -G sanlock
root       962   959  0 21:16 ?        00:00:00 sanlock daemon -U sanlock -G sanlock
root      1041   783  0 21:26 pts/0    00:00:00 grep --color=auto 959
[root@localhost sanlock2.7_rpm_f18]#
[root@localhost sanlock2.7_rpm_f18]#
[root@localhost sanlock2.7_rpm_f18]#
[root@localhost sanlock2.7_rpm_f18]# fuser -m /var/run/sanlock/sanlock.pid
/run/sanlock/sanlock.pid:     1   206   209m   330   331   332   334   344   676   711   778   959   960
[root@localhost sanlock2.7_rpm_f18]# ps -ef | grep 206 | grep -v grep
root       206     1  0 21:08 ?        00:00:00 /usr/lib/systemd/systemd-udevd
root       282   206  0 21:08 ?        00:00:00 /usr/lib/systemd/systemd-udevd
root       283   206  0 21:08 ?        00:00:00 /usr/lib/systemd/systemd-udevd
[root@localhost sanlock2.7_rpm_f18]# ps -ef | grep 209 | grep -v grep
root       209     1  0 21:08 ?        00:00:00 /usr/lib/systemd/systemd-journald
[root@localhost sanlock2.7_rpm_f18]# ps -ef | grep 330 | grep -v grep
root       330     1  0 21:08 ?        00:00:00 /sbin/rsyslogd -n -c 5
[root@localhost sanlock2.7_rpm_f18]# ps -ef | grep 331 | grep -v grep
root       331     1  0 21:08 ?        00:00:00 /usr/lib/systemd/systemd-logind
[root@localhost sanlock2.7_rpm_f18]# ps -ef | grep 332 | grep -v grep
dbus       332     1  0 21:08 ?        00:00:00 /bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
[root@localhost sanlock2.7_rpm_f18]# ps -ef | grep 334 | grep -v grep
root       334     1  0 21:08 ?        00:00:00 /usr/sbin/crond -n
[root@localhost sanlock2.7_rpm_f18]# ps -ef | grep 344 | grep -v grep
root       344     1  0 21:08 ?        00:00:00 login -- root
root       721   344  0 21:08 tty1     00:00:00 -bash
[root@localhost sanlock2.7_rpm_f18]# ps -ef | grep 676 | grep -v grep
root       676     1  0 21:08 ?        00:00:00 sendmail: accepting connections
[root@localhost sanlock2.7_rpm_f18]# ps -ef | grep 711 | grep -v grep
smmsp      711     1  0 21:08 ?        00:00:00 sendmail: Queue runner@01:00:00 for /var/spool/clientmqueue
[root@localhost sanlock2.7_rpm_f18]# ps -ef | grep 778 | grep -v grep
root       778   653  0 21:11 ?        00:00:00 sshd: root@pts/0
root       783   778  0 21:11 pts/0    00:00:00 -bash
[root@localhost sanlock2.7_rpm_f18]# ps -ef | grep 959 | grep -v grep
sanlock    959     1  0 21:16 ?        00:00:00 sanlock daemon -U sanlock -G sanlock
root       962   959  0 21:16 ?        00:00:00 sanlock daemon -U sanlock -G sanlock
[root@localhost sanlock2.7_rpm_f18]# ps -ef | grep 960 | grep -v grep
root       960     1  0 21:16 ?        00:00:00 wdmd -G sanlock
[root@localhost sanlock2.7_rpm_f18]#
[root@localhost sanlock2.7_rpm_f18]# sanlock daemon -L 7
[root@localhost sanlock2.7_rpm_f18]#
[root@localhost sanlock2.7_rpm_f18]# tail /var/log/sanlock.log
2013-07-30 21:16:07+0800 473 [959]: sanlock daemon started 2.7 host 408f9139-5a7b-423c-81ba-48310373f8be.localhost.
2013-07-30 21:36:10+0800 1676 [1059]: lockfile setlk error /var/run/sanlock/sanlock.pid: Resource temporarily unavailable
[root@localhost sanlock2.7_rpm_f18]#

The source was rebuild and packaged with the src and spec in http://arm.koji.fedoraproject.org//packages/sanlock/2.7/1.fc19/src/sanlock-2... .

Thank you.

Qi

-----Original Message-----
From: David Teigland [mailto:[e-mail]] 
Sent: Tuesday, July 30, 2013 12:47 AM
To: Qixiaozhen
Cc: [e-mail]
Subject: Re: Error occurs when I try to change the log level in sanlock v2.7 and fedora 18. Error Info: lockfile setlk error /var/run/sanlock/sanlock.pid: Resource temporarily unavailable

On Mon, Jul 29, 2013 at 08:23:26AM +0000, Qixiaozhen wrote:
> 2013-07-29 16:00:50-0400 372 [1369]: lockfile setlk error /var/run/sanlock/sanlock.pid: Resource temporarily unavailable
>      rv = fcntl(fd, F_SETLK, &lock);
>      if (rv < 0) {
>                log_error("lockfile setlk error %s: %s",
>                           path, strerror(errno));
>                goto fail;
>      }

EAGAIN means that another process is holding that lock.
Did you verify that no other sanlock daemon is running?
You could also try "lsof /var/run/sanlock/sanlock.pid".
Dave