September 2011 - sanlock-devel - Fedora Mailing-Lists

by David Teigland

Changes since the dawn of time: Daniel P. Berrange (15): Fix const-ness of many APIs. Fix warnings in watchdog module Fix function prototypes for no-arg methods Remove use of 'index' as a variable name Make many functions static Fix missing include in logging file Annotate many unused parameters to avoid warnings Remove redundant redeclaration of 'to' variable Fix args to execv() Remove redundant arg to 'run_command' Rename optarg to optionarg to avoid clashing with getopt.h Disable the read_request method since it is unused Add many more compiler warning flags & safety checks Hard code a sector size of 512 if the lease volume is a regular file Ensure libsanlock.so is built with debug/warning flags David Teigland (292): sync_manager: initial commit sync_manager: misc updates sync_manager: misc updates sync_manager: misc updates sync_manager: misc updates sync_manager: add more logging sync_manager: misc updates sync_manager: misc updates sync_manager: num_hosts/MAX_HOSTS stuff daemon: reworking notions of resource_id/token_name sync_manager: resource lockfiles sync_manager: lease arg processing sync_manager: Began multiple lease support sync_manager: use first command line arg as action sync_manager: leader record changes and verify sync_manager: clean up released leases sync_manager: move functions around sync_manager: add more tool actions sync_manager: naming changes sync_manager: separate token index and id sync_manager: fix index usage and other misc sync_manager: use pthread cond in acquire sync_manager: write all log entries sync_manager: simplify polling sync_manager: fix waitpid use sync_manager: acquire can fail early sync_manager: write log entries at exit sync_manager: add test program sync_manager: move secondary pid check sync_manager: fix disk paxos contention devcount: fix verify checks sync_manager: add GPL license file sync_manager: fix leader block offsets sync_manager: increase COMMAND_MAX sync_manager: renewal should verify sync_manager: use sector size from libblkid sync_manager: use a real checksum function sync_manager: add libblkid to spec file sync_manager: print status info sync_manager: one watchdog file per lease sync_manager: lease_threads handle watchdog files sync_manager: fix/add some text/comments sync_manager: refactor read/write sync_manager: move disk io functions sync_manager: remove token arg sync_manager: rename paxos_disk sync_disk sync_manager: add aio read/write sync_manager: make io_timeout_seconds a diskio arg sync_manager: forgot to add new files sync_manager: use log thread sync_manager: client connections sync_manager: connection processing sync_manager: send/recv pid sync_manager: add write_sectors sync_manager: restructuring sync_manager: write_sectors code factoring sync_manager: daemonize sync_manager: releasing leases sync_manager: async releasing sync_manager: release fixes sync_manager: add direct and indirect acquire/release sync_manager: reacquire resources sync_manager: move code sync_manager: same pid may reacquire resource sync_manager: lease migration sync_manager: handle client errors sync_manager: improve error handling sync_manager: host_id leases sync_manager: remove empty files sync_manager: print initialization info sync_manager: rename files sync_manager: clean up header org sync_manager: delta_lease implementation sync_manager: accept offset units sync_manager: fix up init output sync_manager: put back watchdog calls sync_manager: fix start_host_id error paths sync_manager: add log_error's for watchdog file errors sync_manager: actual timeouts sync_manager: change timeouts on cmd line sanlock: create new external api sanlock: build libsanlock sanlock: use MAX_LEASES everywhere sanlock: add libvirt plugin sanlock plugin: couple minor fixes sanlock: clean up /var file names sanlock plugin: fix symbol needed by libvirt sanlock: add some debug output sanlock plugin: fix uuid copy sanlock plugin: fix names sanlock: add "owner_name" sanlock: fix renewal checks sanlock: clean up host_id types sanlock: set_host_id command sanlock: fix killing pids sanlock: add status command sanlock: set version to 1.0 sanlock: delta_lease cleanup sanlock: changing num_hosts sanlock: add dump command sanlock: renewal timings sanlock: add direct option sanlock: check for watchdog file sanlock: recovery fixes lock_driver_sanlock: fix compile problems sanlock: improve command options sanlock: tidying help text sanlock: move binary to /usr/sbin sanlock: add init script sanlock: fix sigterm shutdown sanlock: init stop sanlock: add wdtest command sanlock.spec: new url lock_driver_sanlock: remove close sanlock: introduce lockspaces lock_driver_sanlock: remove files sanlock: better logging functions sanlock: misc log message sanlock.spec: sbin not libexec sanlock init: remove watchdog reference wdmd: watchdog multiplexing daemon sanlock: add code to use wdmd sanlock/wdmd: use wdmd in sanlock sanlock/wdmd: add copyright header to source files sanlock: rename sanlock source dir sanlock: move tests dir move COPYING file wdmd: use signalfd for signal handling Fix Makefile comments wdmd: fix daemon debug option wdmd: add init script sanlock.spec: updates sanlock.spec: src dir sanlock: build with uninstalled libwdmd sanlock: version 1.1 sanlock: high priority options wdmd: high priority options sanlock: return migration state sanlock: migration.txt describes libvirt/sanlock steps libsanlock: include admin functions sanlock: fix host_id expiration check sanlock: migration working devcount: migrate test sanlock: setowner improvements sanlock: migrate to target fix sanlock: fix wdmd stop order sanlock: various fixes sanlock: remove wdtest sanlock: remove migration sanlock: clean up command return data sanlock: add resource string conversion functions sanlock: rework internal structs devcount: add relock test sanlock: fix release and inquire sanlock: add_lockspace EEXIST sanlock: rework client handling sanlock: clean up warnings sanlock: debug message changes sanlock: add lockspace checks wdmd: enable test scripts sanlock: add str_to_lockspace to lib WIP devcount migrate devcount: new migrate test sanlock: read_id and live_id commands sanlock: check lockspace name and host_id sanlock: remove remaining cluster_mode sanlock: add libsanlock_direct devcountn: start multiple devcount tests devcount: small changes sanlock: new return values sanlock: misc changes and fix sanlock: log error of full bad block sanlock: interval between renewal checks sanlock: renewal changes sanlock: fix log_dump sanlock: fix find_client_pid sanlock: fix host_id reads from paxos_acquire sanlock: init with one write devcount: improve output devcount: new pause/resume devcount: add expire test sanlock: correct paxos usage sanlock: direct read_leader sanlock: paxos delays sanlock: use thread pool sanlock: client status output format changes sanlock: fix inquire of dead pid sanlock: use native linux aio sanlock: i/o changes sanlock: aio changes sanlock: reduce paxos acquire read ops sanlock: quiet error case sanlock: don't free aio buf until event completes sanlock: io timeout related changes sanlock: read dblocks in single aligned io sanlock: add sanlock_restrict api sanlock: add sanlock_direct_sector_size api sanlock: add checksum to dblocks sanlock: fix init restart sanlock: don't release tokens in dead lockspace sanlock: fix adding lockspace sanlock: official 1MB/8MB alignment devcount: use aio in init libsanlock: link with LDFLAGS sanlock: increase version to 1.3 sanlock/wdmd: shut up warnings sanlock: fix libwdmd linking remove spec file sanlock: use a completed read after renewal timeout sanlock: use unique host name in delta leases sanlock: remove sector_size api sanlock: abort delta wait on shutdown sanlock: fix add_lockspace failure sanlk_load: add new test sanlock: fix recv and inquire sanlock: initial pid_dead check in acquire sanlock: release 1.4 sanlock: generate a uuid for host id sanlock: return -EINPROGRESS from add_lockspace sanlk_load: periodically kill and replace a pid sanlock: zero num_hosts uses DEFAULT_MAX_HOSTS tests: misc changes sanlock: break paxos_acquire wait loop sanlock: increase log line to 512 bytes sanlock: change a log_error to log_debug sanlock: fail host_id when corrupted sanlock: release 1.5 sanlock: release 1.6 sanlock: handle colon escaping in path strings wdmd: add option for high priority wdmd: use accept4 with SOCK_NONBLOCK wdmd: tidy sun_addr snprintf wdmd: pid and sock file changes wdmd: add man page wdmd: disable test scripts sanlock: use accept4 with SOCK_NONBLOCK sanlock: tidy sun_addr snprintf sanlock: add explicit -luuid sanlock: pid and sock file changes sanlock: add man page sanlock/wdmd: improve mkdir of run dir wdmd: new build flags sanlock: new build flags sanlock/wdmd: use monotonic time sanlock: build with pie sanlock/wdmd: nonblocking listening/accept sanlock: add missing monotime files sanlock: update man page sanlock: man page update sanlock: update man page and help text sanlock: print connections limit release: sanlock 1.7 makefile: install mode for man pages sanlock: read align_size in renewal sanlock: check other host_id leases sanlock: minor fixes and cleanups sanlock: add request api/cmd sanlock: crc code tidying sanlock/wdmd: add license header to files sanlock: create libsanlock_client sanlock: move client code remove COPYING file sanlock: remove internal header from sanlock_sock libsanlock_client: use LGPLv2+ libwdmd: use LGPLv2+ sanlock_rv.h: switch to LGPLv2+ README.license: document licenses sanlock: client align and init sanlock: write request record sanlock: request struct magic and version numbers sanlock: set bitmap for requests sanlock: examine resource requests sanlock: update man page sanlock: remove old comment sanlock: renaming functions sanlock: optimize paxos wait sanlock: use flags in struct resource sanlock: restructure lockspace checks sanlock: remove BLOCK_WD force mode libsanlock: fix function stubs sanlock: new status and host_status sanlock: improve status output sanlock: status output sorting sanlock: SIGTERM and SIGKILL for REQ_KILL_PID sanlock: add flag RESTRICT_SIGKILL sanlock: setup fewer aio events sanlock: move cmd processing sanlock: suppress log messages sanlock: rename source files sanlock: improve killing pids sanlock: fix log_dump version 1.8 Fabio M. Di Nitto (5): build: sanlock should link with libsanlock build: install shared lib header files and fix DESTDIR usage build: drop rpm target spec file: do first cut for total spec file build: fix linking with libsanlock and install target Federico Simoncelli (33): rpm: sync specfile with fedora rpm: add sanlock_admin.h header rpm: add the lib package, install the initscripts python: remove unused python code python: add python binding python: release the gil during sanlock operations python: wrap sanlock extension with a module rpm: add python binding package python: pass a lockspace copy to str_to_lockspace makefile: fix install typo rpm: add sanlock_direct header python: add sanlock init functions and exception direct: close disks after initialization python: register process only once daemon: configurable socket permissions rpm: add sanlock user and group python: exceptions must contain the errno rpm: add missing libaio-devel dependency rpm: add daemon options in the init file python: add missing aio library python: add get_alignment function libs: include libsanlock_direct into libsanlock python: align num_hosts and max_hosts defaults python: expose sanlock file descriptor python: improve error reporting python: parse lockspaces and resources natively python: add usage example python: initial support for sanlock errors python: document the sanlock module python: module cleanup build: fix documentation install path client: return appropriate errno on failure python: expose errno in the exception Saggi Mizrahi (21): Added the begining of the testing and debugging tools Better handling of max hosts sync_manager: Updated tests to work with new lease struct sync_manager: fixed skipping first arg in command sync_manager: acquire and release actions sync_manager: minor fixes sync_manager: renamed stuff sync_manager: made acquire synchronous again sync_manager: added set_host_id action sync_manager: use kill(0) for secondary pid check sync_manager: make rpm and install sync_manager: spec file update sync_manager: Allow longer resource names sync_manager: allow repeated set_host_id sync_manager: Added escaping for the leases arg sync_manager: Created the python bindings for sync_manager sync_manager: listener socket permissions sync_manager: Updated python binding and tests sync_manager: Made 'token' a const in log_level sync_manager: refactor messaging system sync_manager: use getsockopt PEERCRED

12 years, 2 months

1
0
0 / 0

src/main.c wdmd/main.c

by David Teigland

src/main.c | 2 +- wdmd/main.c | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) New commits: commit 68aa2ca480d93737b46b0464490f85316835fc39 Author: David Teigland <teigland(a)redhat.com> Date: Fri Sep 16 13:12:37 2011 -0500 version 1.8 diff --git a/src/main.c b/src/main.c index e5aa9c6..e2d0c0b 100644 --- a/src/main.c +++ b/src/main.c @@ -1317,7 +1317,7 @@ static int parse_arg_resource(char *arg) return 0; } -#define RELEASE_VERSION "1.7" +#define RELEASE_VERSION "1.8" /* * daemon: acquires leases for the local host_id, associates them with a local diff --git a/wdmd/main.c b/wdmd/main.c index b31d9f6..b3e9d67 100644 --- a/wdmd/main.c +++ b/wdmd/main.c @@ -888,8 +888,8 @@ static void print_usage(void) would be for wdmd to fail starting if it found a pid file left over from its previous run. */ -#define RELEASE_VERSION "1.1" - +#define RELEASE_VERSION "1.8" + int main(int argc, char *argv[]) { char optchar;

12 years, 2 months

1
0
0 / 0

src/client_cmd.c src/client_cmd.h src/cmd.c src/log.c src/log.h src/main.c src/sanlock_internal.h

by David Teigland

src/client_cmd.c | 38 ++++++++++++++++++++++++++------------ src/client_cmd.h | 2 +- src/cmd.c | 17 +++++++++++++---- src/log.c | 42 ++++++++++++++++++++++++++++++------------ src/log.h | 2 +- src/main.c | 2 +- src/sanlock_internal.h | 2 +- 7 files changed, 73 insertions(+), 32 deletions(-) New commits: commit 0b2e562ff5f5a719cbb320868d8a7629dac4cc0e Author: David Teigland <teigland(a)redhat.com> Date: Fri Sep 16 13:08:10 2011 -0500 sanlock: fix log_dump and make it more efficient diff --git a/src/client_cmd.c b/src/client_cmd.c index 119149f..2662122 100644 --- a/src/client_cmd.c +++ b/src/client_cmd.c @@ -470,15 +470,22 @@ int sanlock_host_status(int debug, char *lockspace_name) return rv; } -int sanlock_log_dump(void) +int sanlock_log_dump(int max_size) { struct sm_header h; - char buf[4096]; + char *buf; int fd, rv; + buf = malloc(max_size); + if (!buf) + return -ENOMEM; + memset(buf, 0, max_size); + fd = send_command(SM_CMD_LOG_DUMP, 0); - if (fd < 0) + if (fd < 0) { + free(buf); return fd; + } memset(&h, 0, sizeof(h)); @@ -492,20 +499,27 @@ int sanlock_log_dump(void) goto out; } + if (h.data <= 0 || h.data > max_size) + goto out; - while (1) { - memset(buf, 0, sizeof(buf)); - - rv = recv(fd, buf, sizeof(buf) - 1, MSG_WAITALL); - - if (rv > 0) - printf("%s", buf); - else - break; + rv = recv(fd, buf, h.data, MSG_WAITALL); + if (rv < 0) { + rv = -errno; + goto out; } + if (!rv) { + rv = -1; + goto out; + } + + printf("%s", buf); printf("\n"); + + if (rv != h.data) + printf("partial dump %d of %d\n", rv, h.data); out: close(fd); + free(buf); return rv; } diff --git a/src/client_cmd.h b/src/client_cmd.h index dceb7ca..6759c93 100644 --- a/src/client_cmd.h +++ b/src/client_cmd.h @@ -11,7 +11,7 @@ int sanlock_status(int debug, char sort_arg); int sanlock_host_status(int debug, char *lockspace_name); -int sanlock_log_dump(void); +int sanlock_log_dump(int max_size); int sanlock_shutdown(void); #endif diff --git a/src/cmd.c b/src/cmd.c index bbab319..97c4eba 100644 --- a/src/cmd.c +++ b/src/cmd.c @@ -1196,13 +1196,15 @@ static int print_state_daemon(char *str) "io_timeout=%d " "id_renewal=%d " "id_renewal_fail=%d " - "id_renewal_warn=%d", + "id_renewal_warn=%d " + "monotime=%llu", our_host_name_global, main_task.use_aio, main_task.io_timeout_seconds, main_task.id_renewal_seconds, main_task.id_renewal_fail_seconds, - main_task.id_renewal_warn_seconds); + main_task.id_renewal_warn_seconds, + (unsigned long long)monotime()); return strlen(str) + 1; } @@ -1537,11 +1539,18 @@ static void cmd_host_status(int fd, struct sm_header *h_recv) free(status); } +static char send_log_dump[LOG_DUMP_SIZE]; + static void cmd_log_dump(int fd, struct sm_header *h_recv) { - send(fd, h_recv, sizeof(struct sm_header), MSG_DONTWAIT); + int len; + + copy_log_dump(send_log_dump, &len); + + h_recv->data = len; - write_log_dump(fd); + send(fd, h_recv, sizeof(struct sm_header), MSG_NOSIGNAL); + send(fd, send_log_dump, len, MSG_NOSIGNAL); } static void cmd_restrict(int ci, int fd, struct sm_header *h_recv) diff --git a/src/log.c b/src/log.c index 72b424f..7513152 100644 --- a/src/log.c +++ b/src/log.c @@ -33,7 +33,7 @@ static pthread_t thread_handle; static pthread_mutex_t log_mutex = PTHREAD_MUTEX_INITIALIZER; static pthread_cond_t log_cond = PTHREAD_COND_INITIALIZER; -static char log_dump[SM_LOG_DUMP_SIZE]; +static char log_dump[LOG_DUMP_SIZE]; static unsigned int log_point; static unsigned int log_wrap; @@ -42,9 +42,9 @@ struct entry { char str[LOG_STR_LEN]; }; -#define SM_LOG_DEFAULT_ENTRIES 4096 +#define LOG_DEFAULT_ENTRIES 4096 static struct entry *log_ents; -static unsigned int log_num_ents = SM_LOG_DEFAULT_ENTRIES; +static unsigned int log_num_ents = LOG_DEFAULT_ENTRIES; static unsigned int log_head_ent; /* add at head */ static unsigned int log_tail_ent; /* remove from tail */ static unsigned int log_dropped; @@ -62,10 +62,21 @@ static void _log_save_dump(int level GNUC_UNUSED, int len) { int i; + if (len < LOG_DUMP_SIZE - log_point) { + memcpy(log_dump+log_point, log_str, len); + log_point += len; + + if (log_point == LOG_DUMP_SIZE) { + log_point = 0; + log_wrap = 1; + } + return; + } + for (i = 0; i < len; i++) { log_dump[log_point++] = log_str[i]; - if (log_point == SM_LOG_DUMP_SIZE) { + if (log_point == LOG_DUMP_SIZE) { log_point = 0; log_wrap = 1; } @@ -175,17 +186,24 @@ static void write_dropped(int level, int num) write_entry(level, str); } -void write_log_dump(int fd) +void copy_log_dump(char *buf, int *len) { - pthread_mutex_lock(&log_mutex); - - if (log_wrap) - send(fd, log_dump + log_point, SM_LOG_DUMP_SIZE - log_point, MSG_DONTWAIT); + int tail_len; - log_dump[log_point] = '\0'; - - send(fd, log_dump, log_point, MSG_DONTWAIT); + pthread_mutex_lock(&log_mutex); + if (!log_wrap && !log_point) { + *len = 0; + } else if (log_wrap) { + tail_len = LOG_DUMP_SIZE - log_point; + memcpy(buf, log_dump+log_point, tail_len); + if (log_point) + memcpy(buf+tail_len, log_dump, log_point); + *len = LOG_DUMP_SIZE; + } else { + memcpy(buf, log_dump, log_point-1); + *len = log_point-1; + } pthread_mutex_unlock(&log_mutex); } diff --git a/src/log.h b/src/log.h index 8661eb0..63168eb 100644 --- a/src/log.h +++ b/src/log.h @@ -14,7 +14,7 @@ void log_level(uint32_t space_id, uint32_t token_id, char *name_in, int level, c int setup_logging(void); void close_logging(void); -void write_log_dump(int fd); +void copy_log_dump(char *buf, int *len); #define log_debug(fmt, args...) log_level(0, 0, NULL, LOG_DEBUG, fmt, ##args) #define log_space(space, fmt, args...) log_level(space->space_id, 0, NULL, LOG_DEBUG, fmt, ##args) diff --git a/src/main.c b/src/main.c index 3034f47..e5aa9c6 100644 --- a/src/main.c +++ b/src/main.c @@ -1708,7 +1708,7 @@ static int do_client(void) break; case ACT_LOG_DUMP: - rv = sanlock_log_dump(); + rv = sanlock_log_dump(LOG_DUMP_SIZE); break; case ACT_SHUTDOWN: diff --git a/src/sanlock_internal.h b/src/sanlock_internal.h index 24ceb2f..46e2b36 100644 --- a/src/sanlock_internal.h +++ b/src/sanlock_internal.h @@ -35,7 +35,7 @@ #define DEFAULT_MAX_HOSTS 2000 -#define SM_LOG_DUMP_SIZE (1024*1024) +#define LOG_DUMP_SIZE (1024*1024) /* this is just the path to the executable, not full command line */

12 years, 2 months

1
0
0 / 0

src/cmd.c src/main.c src/sanlock_internal.h src/sanlock_resource.h src/task.c

by David Teigland

src/cmd.c | 6 +- src/main.c | 129 +++++++++++++++++++++++++------------------------ src/sanlock_internal.h | 7 +- src/sanlock_resource.h | 1 src/task.c | 14 ++++- 5 files changed, 88 insertions(+), 69 deletions(-) New commits: commit ba4b71a6cd7b27ea3adcd011916b4b93b6da253e Author: David Teigland <teigland(a)redhat.com> Date: Thu Sep 15 16:23:18 2011 -0500 sanlock: improve killing pids try to kill a pid once a second for a configured max number of attempts (60), after which we give up. The first 10 attempts use SIGTERM (if not restricted), after which we use SIGKILL (if not restricted). diff --git a/src/cmd.c b/src/cmd.c index 4810876..bbab319 100644 --- a/src/cmd.c +++ b/src/cmd.c @@ -1219,7 +1219,8 @@ static int print_state_client(struct client *cl, int ci, char *str) "cmd_active=%d " "cmd_last=%d " "pid_dead=%d " - "killing=%d " + "kill_count=%d " + "kill_last=%llu " "suspend=%d " "need_free=%d", ci, @@ -1229,7 +1230,8 @@ static int print_state_client(struct client *cl, int ci, char *str) cl->cmd_active, cl->cmd_last, cl->pid_dead, - cl->killing, + cl->kill_count, + (unsigned long long)cl->kill_last, cl->suspend, cl->need_free); diff --git a/src/main.c b/src/main.c index 25f2b09..3034f47 100644 --- a/src/main.c +++ b/src/main.c @@ -150,7 +150,8 @@ static void _client_free(int ci) cl->pid_dead = 0; cl->suspend = 0; cl->need_free = 0; - cl->killing = 0; + cl->kill_count = 0; + cl->kill_last = 0; cl->restrict = 0; memset(cl->owner_name, 0, sizeof(cl->owner_name)); cl->workfn = NULL; @@ -417,7 +418,7 @@ static int client_using_space(struct client *cl, struct space *sp) if (strncmp(token->r.lockspace_name, sp->space_name, NAME_ID_SIZE)) continue; - if (cl->killing < 10) + if (!cl->kill_count) log_spoke(sp, token, "client_using_space pid %d", cl->pid); token->space_dead = sp->space_dead; rv = 1; @@ -425,92 +426,89 @@ static int client_using_space(struct client *cl, struct space *sp) return rv; } -/* - * TODO: move to time-based kill tracking instead of repetition based, e.g. - * kill pid once a second until a max pid_exit_wait after which we quit - * trying to kill it. half way through pid_exit_wait shift from SIGTERM - * to SIGKILL (if no restriction). - * - * sp->killing pids begins at 1 - * don't increment sp->killing pids each time - * when all clients have gone past pid_exit_wait, set sp->killing_pids to 2 - * when sp->killing_pids is 2, then return immediately from kill_pids() - * - * Remove the usleep delay after a kill. - */ +/* TODO: try killscript first if one is provided */ static void kill_pids(struct space *sp) { struct client *cl; - int ci, pid; - int sig; - int found = 0; - - /* TODO: try killscript first if one is provided */ + uint64_t now; + int ci, fd, pid, sig; + int do_kill; - if (sp->killing_pids > 11) + /* + * all remaining pids using sp are stuck, we've made max attempts to + * kill all, don't bother cycling through them + */ + if (sp->killing_pids > 1) return; - log_space(sp, "kill_pids %d", sp->killing_pids); - - if (sp->killing_pids > 10) { - sp->killing_pids++; - goto do_dump; - } - - sp->killing_pids++; + now = monotime(); for (ci = 0; ci <= client_maxi; ci++) { - pid = -1; + do_kill = 0; cl = &client[ci]; pthread_mutex_lock(&cl->mutex); if (!cl->used) goto unlock; + if (cl->pid <= 0) goto unlock; + + /* NB this cl may not be using sp, but trying to + avoid the expensive client_using_space check */ + + if (cl->kill_count >= main_task.kill_count_max) + goto unlock; + + if (cl->kill_count && (now - cl->kill_last < 1)) + goto unlock; + if (!client_using_space(cl, sp)) goto unlock; - if (!(cl->restrict & SANLK_RESTRICT_SIGKILL) && cl->killing > 1) + cl->kill_last = now; + cl->kill_count++; + + fd = cl->fd; + pid = cl->pid; + + if (cl->restrict & SANLK_RESTRICT_SIGKILL) + sig = SIGTERM; + else if (cl->restrict & SANLK_RESTRICT_SIGTERM) sig = SIGKILL; - else + else if (cl->kill_count <= main_task.kill_count_term) sig = SIGTERM; + else + sig = SIGKILL; - pid = cl->pid; - cl->killing++; - found++; + do_kill = 1; unlock: pthread_mutex_unlock(&cl->mutex); - if (pid > 0) - kill(pid, sig); - } - - if (found) { - log_space(sp, "kill_pids %d found %d pids", sig, found); - usleep(500000); - } - - return; + if (!do_kill) + continue; - do_dump: - for (ci = 0; ci <= client_maxi; ci++) { - if (client[ci].pid && client[ci].killing) { - log_error("kill_pids %d stuck", client[ci].pid); + if (cl->kill_count == main_task.kill_count_max) { + log_erros(sp, "kill %d,%d,%d sig %d count %d final attempt", + ci, fd, pid, sig, cl->kill_count); + } else { + log_space(sp, "kill %d,%d,%d sig %d count %d", + ci, fd, pid, sig, cl->kill_count); } + + kill(pid, sig); } } static int all_pids_dead(struct space *sp) { struct client *cl; - int ci, pid; + int stuck = 0, check = 0; + int ci; for (ci = 0; ci <= client_maxi; ci++) { - pid = -1; - cl = &client[ci]; pthread_mutex_lock(&cl->mutex); @@ -521,18 +519,23 @@ static int all_pids_dead(struct space *sp) if (!client_using_space(cl, sp)) goto unlock; - pid = cl->pid; + if (cl->kill_count >= main_task.kill_count_max) + stuck++; + else + check++; unlock: pthread_mutex_unlock(&cl->mutex); + } - if (pid > 0) { - if (cl->killing < 10) { - log_space(sp, "used by pid %d killing %d", - pid, cl->killing); - } - return 0; - } + if (stuck && !check && sp->killing_pids < 2) { + log_erros(sp, "killing pids stuck %d", stuck); + /* cause kill_pids to give up */ + sp->killing_pids = 2; } + + if (stuck || check) + return 0; + log_space(sp, "used by no pids"); return 1; } @@ -887,9 +890,9 @@ static void process_cmd_thread_registered(int ci_in, struct sm_header *h_recv) goto out; } - if (cl->killing) { - log_error("cmd %d %d,%d,%d killing", - h_recv->cmd, ci_target, cl->fd, cl->pid); + if (cl->kill_count) { + log_error("cmd %d %d,%d,%d kill_count %d", + h_recv->cmd, ci_target, cl->fd, cl->pid, cl->kill_count); result = -EBUSY; goto out; } diff --git a/src/sanlock_internal.h b/src/sanlock_internal.h index fb13afb..24ceb2f 100644 --- a/src/sanlock_internal.h +++ b/src/sanlock_internal.h @@ -450,10 +450,10 @@ struct task { int id_renewal_seconds; /* configured */ int id_renewal_fail_seconds; /* configured */ int id_renewal_warn_seconds; /* configured */ - int host_dead_seconds; /* calculated */ - int request_finish_seconds; /* calculated */ + int kill_count_term; /* constant */ + int kill_count_max; /* constant */ unsigned int io_count; /* stats */ unsigned int to_count; /* stats */ @@ -477,8 +477,9 @@ struct client { int pid_dead; int suspend; int need_free; - int killing; + int kill_count; uint32_t restrict; + uint64_t kill_last; char owner_name[SANLK_NAME_LEN+1]; pthread_mutex_t mutex; void *workfn; diff --git a/src/sanlock_resource.h b/src/sanlock_resource.h index ae8dfa4..b86bde0 100644 --- a/src/sanlock_resource.h +++ b/src/sanlock_resource.h @@ -23,6 +23,7 @@ /* restrict flags */ #define SANLK_RESTRICT_ALL 0x00000001 #define SANLK_RESTRICT_SIGKILL 0x00000002 +#define SANLK_RESTRICT_SIGTERM 0x00000004 /* release flags */ #define SANLK_REL_ALL 0x00000001 diff --git a/src/task.c b/src/task.c index 9635f92..e06a6fa 100644 --- a/src/task.c +++ b/src/task.c @@ -53,13 +53,24 @@ void setup_task_timeouts(struct task *task, int io_timeout_arg) int paxos_acquire_held_min = host_dead_seconds; int paxos_acquire_free_max = 6 * io_timeout_seconds; int paxos_acquire_free_min = 0; + int request_finish_seconds = 3 * id_renewal_seconds; /* random */ task->io_timeout_seconds = io_timeout_seconds; task->id_renewal_seconds = id_renewal_seconds; task->id_renewal_fail_seconds = id_renewal_fail_seconds; task->id_renewal_warn_seconds = id_renewal_warn_seconds; task->host_dead_seconds = host_dead_seconds; - task->request_finish_seconds = 3 * id_renewal_seconds; /* random */ + task->request_finish_seconds = request_finish_seconds; + + /* interval between each kill count is approx 1 sec, so we + spend about 10 seconds sending 10 SIGTERMs to a pid, + then send SIGKILLs to it. after 60 attempts the watchdog + should have fired if the kills are due to failed renewal; + otherwise we just give up at that point */ + + task->kill_count_term = 10; + task->kill_count_max = 60; + /* the rest are calculated as needed in place */ /* hack to make just main thread log this info */ @@ -84,6 +95,7 @@ void setup_task_timeouts(struct task *task, int io_timeout_arg) log_debug("paxos_acquire_held_min %d", paxos_acquire_held_min); log_debug("paxos_acquire_free_max %d", paxos_acquire_free_max); log_debug("paxos_acquire_free_min %d", paxos_acquire_free_min); + log_debug("request_finish_seconds %d", request_finish_seconds); } void setup_task_aio(struct task *task, int use_aio, int cb_size)

12 years, 2 months

1
0
0 / 0

2 commits - src/main.c src/sanlock_internal.h src/sanlock_resource.h src/token_manager.c src/token_manager.h tests/devcount.c

by David Teigland

src/main.c | 23 +++++++++----- src/sanlock_internal.h | 11 +++--- src/sanlock_resource.h | 1 src/token_manager.c | 80 ++++++++++++++++++++++++++++++++----------------- src/token_manager.h | 2 - tests/devcount.c | 25 +++++++++++++++ 6 files changed, 101 insertions(+), 41 deletions(-) New commits: commit e3077bf925432aa852f6eb4f3eb552c21a09d353 Author: David Teigland <teigland(a)redhat.com> Date: Wed Sep 14 13:50:33 2011 -0500 sanlock: setup fewer aio events since there is a system limit on these, we'd unnecessarily cause io_setup to fail and not use aio at all. We really shouldn't need many anyway. diff --git a/src/sanlock_internal.h b/src/sanlock_internal.h index 0bd3d4c..6e7aae6 100644 --- a/src/sanlock_internal.h +++ b/src/sanlock_internal.h @@ -431,11 +431,11 @@ struct space { * The delta lease algorithm expects real io timeouts. */ -#define HOSTID_AIO_CB_SIZE 64 -#define WORKER_AIO_CB_SIZE 8 -#define DIRECT_AIO_CB_SIZE 8 -#define RESOURCE_AIO_CB_SIZE 64 -#define LIB_AIO_CB_SIZE 8 +#define HOSTID_AIO_CB_SIZE 4 +#define WORKER_AIO_CB_SIZE 2 +#define DIRECT_AIO_CB_SIZE 1 +#define RESOURCE_AIO_CB_SIZE 2 +#define LIB_AIO_CB_SIZE 1 struct aicb { int used; commit dd6791b3844e16144fbe81330371f5d4f4847757 Author: David Teigland <teigland(a)redhat.com> Date: Wed Sep 14 13:24:51 2011 -0500 sanlock: add flag RESTRICT_SIGKILL which causes sanlock to only use SIGTERM to kill pid, not to escalate to SIGKILL. A process may be holding a lease based on state outside itself; the fact that the pid exits does not imply it's safe to release the lease. The pid would choose to ignore the SIGTERM if it's not safe to release the lease, and using SIGKILL makes it impossible for the pid to make that choice. diff --git a/src/main.c b/src/main.c index d3d0c4f..2d6eeee 100644 --- a/src/main.c +++ b/src/main.c @@ -66,7 +66,7 @@ struct client { int suspend; int need_free; int killing; - uint32_t cmd_flags; + uint32_t restrict; char owner_name[SANLK_NAME_LEN+1]; pthread_mutex_t mutex; void *workfn; @@ -183,7 +183,7 @@ static void _client_free(int ci) cl->suspend = 0; cl->need_free = 0; cl->killing = 0; - cl->cmd_flags = 0; + cl->restrict = 0; memset(cl->owner_name, 0, sizeof(cl->owner_name)); cl->workfn = NULL; cl->deadfn = NULL; @@ -432,7 +432,7 @@ static void kill_pids(struct space *sp) { struct client *cl; int ci, pid; - int sig = SIGTERM; + int sig; int found = 0; log_space(sp, "kill_pids %d", sp->killing_pids); @@ -447,8 +447,6 @@ static void kill_pids(struct space *sp) goto do_dump; } - if (sp->killing_pids > 1) - sig = SIGKILL; sp->killing_pids++; for (ci = 0; ci <= client_maxi; ci++) { @@ -464,6 +462,11 @@ static void kill_pids(struct space *sp) if (!client_using_space(cl, sp)) goto unlock; + if (!(cl->restrict & SANLK_RESTRICT_SIGKILL) && cl->killing > 1) + sig = SIGKILL; + else + sig = SIGTERM; + pid = cl->pid; cl->killing++; found++; @@ -777,6 +780,7 @@ static void cmd_acquire(struct task *task, struct cmd_args *ca) char *opt_str; uint64_t acquire_lver = 0; uint32_t new_num_hosts = 0; + uint32_t cl_restrict; int token_len, disks_len; int fd, rv, i, j, empty_slots; int alloc_count = 0, add_count = 0, acquire_count = 0; @@ -809,6 +813,7 @@ static void cmd_acquire(struct task *task, struct cmd_args *ca) pthread_mutex_unlock(&cl->mutex); goto done; } + cl_restrict = cl->restrict; empty_slots = 0; for (i = 0; i < SANLK_MAX_RESOURCES; i++) { @@ -962,7 +967,7 @@ static void cmd_acquire(struct task *task, struct cmd_args *ca) for (i = 0; i < new_tokens_count; i++) { token = new_tokens[i]; - rv = add_resource(token, cl_pid); + rv = add_resource(token, cl_pid, cl_restrict); if (rv < 0) { if (!com.quiet_fail) log_errot(token, "cmd_acquire %d,%d,%d add_resource %d", @@ -1967,6 +1972,7 @@ static int print_state_client(struct client *cl, int ci, char *str) "ci=%d " "fd=%d " "pid=%d " + "restrict=%x " "cmd_active=%d " "cmd_last=%d " "pid_dead=%d " @@ -1976,6 +1982,7 @@ static int print_state_client(struct client *cl, int ci, char *str) ci, cl->fd, cl->pid, + cl->restrict, cl->cmd_active, cl->cmd_last, cl->pid_dead, @@ -2297,7 +2304,7 @@ static void cmd_restrict(int ci, int fd, struct sm_header *h_recv) log_debug("cmd_restrict ci %d fd %d pid %d flags %x", ci, fd, client[ci].pid, h_recv->cmd_flags); - client[ci].cmd_flags = h_recv->cmd_flags; + client[ci].restrict = h_recv->cmd_flags; send_result(fd, h_recv, 0); } @@ -2532,7 +2539,7 @@ static void process_connection(int ci) ci, rv, h.magic, SM_MAGIC); goto dead; } - if (client[ci].cmd_flags & SANLK_RESTRICT_ALL) { + if (client[ci].restrict & SANLK_RESTRICT_ALL) { log_error("ci %d fd %d pid %d cmd %d restrict all", ci, client[ci].fd, client[ci].pid, h.cmd); goto dead; diff --git a/src/sanlock_internal.h b/src/sanlock_internal.h index 0b39934..0bd3d4c 100644 --- a/src/sanlock_internal.h +++ b/src/sanlock_internal.h @@ -90,6 +90,7 @@ struct token { }; #define R_EXAMINE 0x00000001 +#define R_NO_SIGKILL 0x00000002 struct resource { struct list_head list; diff --git a/src/sanlock_resource.h b/src/sanlock_resource.h index 770ebfc..ae8dfa4 100644 --- a/src/sanlock_resource.h +++ b/src/sanlock_resource.h @@ -22,6 +22,7 @@ /* restrict flags */ #define SANLK_RESTRICT_ALL 0x00000001 +#define SANLK_RESTRICT_SIGKILL 0x00000002 /* release flags */ #define SANLK_REL_ALL 0x00000001 diff --git a/src/token_manager.c b/src/token_manager.c index 33de66a..824a202 100644 --- a/src/token_manager.c +++ b/src/token_manager.c @@ -98,7 +98,7 @@ static void save_resource_lver(struct token *token, uint64_t lver) } -int add_resource(struct token *token, int pid) +int add_resource(struct token *token, int pid, uint32_t cl_restrict) { struct resource *r; int rv, disks_len, r_len; @@ -135,6 +135,8 @@ int add_resource(struct token *token, int pid) r->token_id = token->token_id; r->token = token; r->pid = pid; + if (cl_restrict & SANLK_RESTRICT_SIGKILL) + r->flags |= R_NO_SIGKILL; list_add_tail(&r->list, &resources); rv = 0; out: @@ -343,6 +345,55 @@ static int examine_token(struct task *task, struct token *token, return rv; } +static void do_req_kill_pid(struct token *tt, int pid) +{ + struct resource *r; + uint32_t flags; + int found = 0; + + pthread_mutex_lock(&resource_mutex); + r = find_resource(tt, &resources); + if (r && r->pid == pid) { + found = 1; + flags = r->flags; + } + pthread_mutex_unlock(&resource_mutex); + + if (!found) { + log_error("req pid %d %.48s:%.48s not found", + pid, tt->r.lockspace_name, tt->r.name); + return; + } + + log_debug("do_req_kill_pid %d flags %x %.48s:%.48s", + pid, flags, tt->r.lockspace_name, tt->r.name); + + /* TODO: share code with kill_pids() to gradually + * escalate from killscript, SIGTERM, SIGKILL */ + + kill(pid, SIGTERM); + + if (flags & R_NO_SIGKILL) + return; + + sleep(1); + kill(pid, SIGKILL); +} + +/* + * TODO? add force_mode SANLK_REQ_KILL_PID_OR_RESET + * which would attempt to kill the pid like KILL_PID, + * but if the pid doesn't exit will block watchdog + * updates to reset the host. + * Here set r->block_wd_time = now + pid_exit_time, + * In renewal check for any r in resources with + * block_wd_time <= now, and if found will not + * update the watchdog. If the pid continues to + * not exit, the wd will fire and reset the machine. + * If the pid exits before pid_exit_time, no wd + * updates will be skipped. + */ + /* * - releases tokens of pid's that die * - examines request blocks of resources @@ -440,33 +491,8 @@ static void *resource_thread(void *arg GNUC_UNUSED) continue; } - /* - * TODO: add force_mode SANLK_REQ_KILL_PID_OR_RESET - * which would attempt to kill the pid like KILL_PID, - * but if the pid doesn't exit will block watchdog - * updates to reset the host. - * Here set r->block_wd_time = now + pid_exit_time, - * In renewal check for any r in resources with - * block_wd_time <= now, and if found will not - * update the watchdog. If the pid continues to - * not exit, the wd will fire and reset the machine. - * If the pid exits before pid_exit_time, no wd - * updates will be skipped. - */ - if (req.force_mode == SANLK_REQ_KILL_PID) { - /* look up r again? verify it exists and pid same */ - - log_debug("req force_mode %u pid %d %.48s:%.48s", - req.force_mode, pid, - tt->r.lockspace_name, tt->r.name); - - /* TODO: share code with kill_pids() to gradually - * escalate from killscript, SIGTERM, SIGKILL */ - - kill(pid, SIGTERM); - sleep(1); - kill(pid, SIGKILL); + do_req_kill_pid(tt, pid); } else { log_error("req force_mode %u unknown", req.force_mode); } diff --git a/src/token_manager.h b/src/token_manager.h index da34baa..751b636 100644 --- a/src/token_manager.h +++ b/src/token_manager.h @@ -19,7 +19,7 @@ void release_token_async(struct token *token); int request_token(struct task *task, struct token *token, uint32_t force_mode, uint64_t *owner_id); -int add_resource(struct token *token, int pid); +int add_resource(struct token *token, int pid, uint32_t cl_restrict); void del_resource(struct token *token); int set_resource_examine(char *space_name, char *res_name); diff --git a/tests/devcount.c b/tests/devcount.c index 9db5a43..6c51c6c 100644 --- a/tests/devcount.c +++ b/tests/devcount.c @@ -58,6 +58,18 @@ do { \ printf("ERROR %llu " fmt "\n", (unsigned long long)time(NULL), ##args); \ } while (0) +static void sigterm_handler(int sig) +{ + log_debug("sigterm_handler %d", sig); +} + +static void setup_sigterm(void) +{ + struct sigaction act; + memset(&act, 0, sizeof(act)); + act.sa_handler = sigterm_handler; + sigaction(SIGTERM, &act, NULL); +} static int kill_pid(int pid) { @@ -836,6 +848,13 @@ static int do_wrap(int argc, char *argv[]) exit(-1); } + rv = sanlock_restrict(sock, SANLK_RESTRICT_SIGKILL); + if (rv < 0) { + log_error("%s c %d sanlock_restrict error %d", + count_path, pid, sock); + exit(-1); + } + rv = sanlock_acquire(sock, -1, 0, 1, &res, NULL); if (rv < 0) { log_error("%s c %d sanlock_acquire error %d", @@ -1489,6 +1508,12 @@ int main(int argc, char *argv[]) else if (!strcmp(argv[1], "rw") || !strcmp(argv[1], "wr")) rv = do_count(argc, argv); + else if (!strcmp(argv[1], "rwsig")) { + setup_sigterm(); + argv[1] = "rw"; + rv = do_count(argc, argv); + } + else if (!strcmp(argv[1], "lock")) rv = do_lock(argc, argv);

12 years, 2 months

1
0
0 / 0

src/host_id.c src/token_manager.c

by David Teigland

src/host_id.c | 5 ++++- src/token_manager.c | 28 ++++++++++++++++++++++++---- 2 files changed, 28 insertions(+), 5 deletions(-) New commits: commit 3d14379e4aea8eef07cfd0ff7dee586a89ef4b9d Author: David Teigland <teigland(a)redhat.com> Date: Tue Sep 13 14:40:34 2011 -0500 sanlock: SIGTERM and SIGKILL for REQ_KILL_PID and add some comments about another req mode diff --git a/src/host_id.c b/src/host_id.c index 4d39d82..18ef28d 100644 --- a/src/host_id.c +++ b/src/host_id.c @@ -519,6 +519,7 @@ static void *lockspace_thread(void *arg_in) /* * pet the watchdog + * (don't update on thread_stop because it's probably unlinked) */ if (delta_result == SANLK_OK && !sp->thread_stop) @@ -543,7 +544,9 @@ static void *lockspace_thread(void *arg_in) } } - /* unlink called below to get it done ASAP */ + /* watchdog unlink was done in main_loop when thread_stop was set, to + get it done as quickly as possible in case the wd is about to fire. */ + close_watchdog_file(sp); out: if (delta_result == SANLK_OK) diff --git a/src/token_manager.c b/src/token_manager.c index 6568af3..33de66a 100644 --- a/src/token_manager.c +++ b/src/token_manager.c @@ -440,15 +440,35 @@ static void *resource_thread(void *arg GNUC_UNUSED) continue; } + /* + * TODO: add force_mode SANLK_REQ_KILL_PID_OR_RESET + * which would attempt to kill the pid like KILL_PID, + * but if the pid doesn't exit will block watchdog + * updates to reset the host. + * Here set r->block_wd_time = now + pid_exit_time, + * In renewal check for any r in resources with + * block_wd_time <= now, and if found will not + * update the watchdog. If the pid continues to + * not exit, the wd will fire and reset the machine. + * If the pid exits before pid_exit_time, no wd + * updates will be skipped. + */ + if (req.force_mode == SANLK_REQ_KILL_PID) { - /* look up r again to check it still exists and - pid is same? */ + /* look up r again? verify it exists and pid same */ - log_error("req_kill_pid %d %.48s:%.48s", pid, + log_debug("req force_mode %u pid %d %.48s:%.48s", + req.force_mode, pid, tt->r.lockspace_name, tt->r.name); + + /* TODO: share code with kill_pids() to gradually + * escalate from killscript, SIGTERM, SIGKILL */ + + kill(pid, SIGTERM); + sleep(1); kill(pid, SIGKILL); } else { - log_error("req force_mode unknown %u", req.force_mode); + log_error("req force_mode %u unknown", req.force_mode); } } }

12 years, 2 months

1
0
0 / 0

2 commits - src/client_cmd.c src/client_cmd.h src/main.c src/sanlock.8 src/sanlock_internal.h

by David Teigland

src/client_cmd.c | 336 +++++++++++++++++++++++++++++++++++++++++-------- src/client_cmd.h | 2 src/main.c | 32 +++- src/sanlock.8 | 1 src/sanlock_internal.h | 1 5 files changed, 311 insertions(+), 61 deletions(-) New commits: commit 758e7c83b6bdd3e8a8203de6b541c38385ad9196 Author: David Teigland <teigland(a)redhat.com> Date: Tue Sep 13 11:39:23 2011 -0500 sanlock: status output sorting -o p|s to sort resources by pid/lockspace. Also fix host_status bug when -s not found. diff --git a/src/client_cmd.c b/src/client_cmd.c index 73404bf..119149f 100644 --- a/src/client_cmd.c +++ b/src/client_cmd.c @@ -56,16 +56,15 @@ static void print_debug(char *str, int len) printf(" %s\n", p); } -static void status_daemon(int fd GNUC_UNUSED, struct sanlk_state *st, char *str, int debug) +static void status_daemon(struct sanlk_state *st, char *str, int debug) { printf("daemon %.48s\n", st->name); - if (st->str_len && debug) { + if (st->str_len && debug) print_debug(str, st->str_len); - } } -static void status_client(int fd GNUC_UNUSED, struct sanlk_state *st, char *str, int debug) +static void status_client(struct sanlk_state *st, char *str, int debug) { printf("p %d ", st->data32); printf("%.48s\n", st->name); @@ -74,38 +73,33 @@ static void status_client(int fd GNUC_UNUSED, struct sanlk_state *st, char *str, print_debug(str, st->str_len); } -static void status_lockspace(int fd, struct sanlk_state *st, char *str, int debug) +static void status_lockspace(struct sanlk_state *st, char *str, char *bin, int debug) { - struct sanlk_lockspace lockspace; - int rv; - - rv = recv(fd, &lockspace, sizeof(lockspace), MSG_WAITALL); + struct sanlk_lockspace *ls = (struct sanlk_lockspace *)bin; printf("s %.48s:%llu:%s:%llu\n", - lockspace.name, - (unsigned long long)lockspace.host_id, - lockspace.host_id_disk.path, - (unsigned long long)lockspace.host_id_disk.offset); + ls->name, + (unsigned long long)ls->host_id, + ls->host_id_disk.path, + (unsigned long long)ls->host_id_disk.offset); if (st->str_len && debug) print_debug(str, st->str_len); } -static void status_resource(int fd, struct sanlk_state *st, char *str, int debug) +static void status_resource(struct sanlk_state *st, char *str, char *bin, int debug) { - struct sanlk_resource resource; - struct sanlk_disk disk; - int i, rv; - - rv = recv(fd, &resource, sizeof(resource), MSG_WAITALL); + struct sanlk_resource *res = (struct sanlk_resource *)bin; + struct sanlk_disk *disk; + int i; - printf("r %.48s:%.48s", resource.lockspace_name, resource.name); + printf("r %.48s:%.48s", res->lockspace_name, res->name); - for (i = 0; i < resource.num_disks; i++) { - rv = recv(fd, &disk, sizeof(disk), MSG_WAITALL); + for (i = 0; i < res->num_disks; i++) { + disk = (struct sanlk_disk *)(bin + sizeof(struct sanlk_resource) + i * sizeof(struct sanlk_disk)); printf(":%s:%llu", - disk.path, (unsigned long long)disk.offset); + disk->path, (unsigned long long)disk->offset); } printf(":%llu p %u\n", (unsigned long long)st->data64, st->data32); @@ -113,7 +107,7 @@ static void status_resource(int fd, struct sanlk_state *st, char *str, int debug print_debug(str, st->str_len); } -static void status_host(int fd GNUC_UNUSED, struct sanlk_state *st, char *str, int debug) +static void status_host(struct sanlk_state *st, char *str, int debug) { printf("%u timestamp %llu\n", st->data32, (unsigned long long)st->data64); @@ -122,12 +116,213 @@ static void status_host(int fd GNUC_UNUSED, struct sanlk_state *st, char *str, i print_debug(str, st->str_len); } -int sanlock_status(int debug) +static void print_st(struct sanlk_state *st, char *str, char *bin, int debug) +{ + switch (st->type) { + case SANLK_STATE_DAEMON: + status_daemon(st, str, debug); + break; + case SANLK_STATE_CLIENT: + status_client(st, str, debug); + break; + case SANLK_STATE_LOCKSPACE: + status_lockspace(st, str, bin, debug); + break; + case SANLK_STATE_RESOURCE: + status_resource(st, str, bin, debug); + break; + } +} + +#define MAX_SORT_ENTRIES 1024 +static char *sort_bufs[MAX_SORT_ENTRIES]; +static int sort_count; +static int sort_done; + +static void print_type(int type, int debug) +{ + struct sanlk_state *st; + char *buf, *str, *bin; + int i; + + for (i = 0; i < sort_count; i++) { + buf = sort_bufs[i]; + if (!buf) + continue; + st = (struct sanlk_state *)buf; + str = buf + sizeof(struct sanlk_state); + bin = buf + sizeof(struct sanlk_state) + SANLK_STATE_MAXSTR; + + if (!type || st->type == type) { + print_st(st, str, bin, debug); + free(buf); + sort_bufs[i] = NULL; + sort_done++; + } + } +} + +static void print_p(int p, int debug) +{ + struct sanlk_state *st; + char *buf, *str, *bin; + int i; + + for (i = 0; i < sort_count; i++) { + buf = sort_bufs[i]; + if (!buf) + continue; + st = (struct sanlk_state *)buf; + str = buf + sizeof(struct sanlk_state); + bin = buf + sizeof(struct sanlk_state) + SANLK_STATE_MAXSTR; + + if (st->type != SANLK_STATE_CLIENT) + continue; + + if (st->data32 == p) { + print_st(st, str, bin, debug); + free(buf); + sort_bufs[i] = NULL; + sort_done++; + } + } +} + +static int find_type(int type, int *sort_index) +{ + struct sanlk_state *st; + char *buf; + int i; + + for (i = 0; i < sort_count; i++) { + buf = sort_bufs[i]; + if (!buf) + continue; + st = (struct sanlk_state *)buf; + + if (st->type == type) { + *sort_index = i; + return 0; + } + } + return -1; +} + +static void print_r(int p, char *s, int debug) +{ + struct sanlk_resource *res; + struct sanlk_state *st; + char *buf, *str, *bin; + int i; + + for (i = 0; i < sort_count; i++) { + buf = sort_bufs[i]; + if (!buf) + continue; + st = (struct sanlk_state *)buf; + str = buf + sizeof(struct sanlk_state); + bin = buf + sizeof(struct sanlk_state) + SANLK_STATE_MAXSTR; + + if (st->type != SANLK_STATE_RESOURCE) + continue; + + res = (struct sanlk_resource *)bin; + + if ((p && st->data32 == p) || + (s && !strncmp(s, res->lockspace_name, SANLK_NAME_LEN))) { + print_st(st, str, bin, debug); + free(buf); + sort_bufs[i] = NULL; + sort_done++; + } + } +} + +static void print_r_by_p(int debug) +{ + struct sanlk_state *st; + char *buf, *str, *bin; + int rv, i; + + while (1) { + rv = find_type(SANLK_STATE_CLIENT, &i); + if (rv < 0) + return; + + buf = sort_bufs[i]; + st = (struct sanlk_state *)buf; + str = buf + sizeof(struct sanlk_state); + bin = buf + sizeof(struct sanlk_state) + SANLK_STATE_MAXSTR; + + print_st(st, str, bin, debug); + + print_r(st->data32, NULL, debug); + + free(buf); + sort_bufs[i] = NULL; + sort_done++; + } +} + +static void print_r_by_s(int debug) +{ + struct sanlk_state *st; + char *buf, *str, *bin; + int rv, i; + + while (1) { + rv = find_type(SANLK_STATE_LOCKSPACE, &i); + if (rv < 0) + return; + + buf = sort_bufs[i]; + st = (struct sanlk_state *)buf; + str = buf + sizeof(struct sanlk_state); + bin = buf + sizeof(struct sanlk_state) + SANLK_STATE_MAXSTR; + + print_st(st, str, bin, debug); + + print_r(0, st->name, debug); + + free(buf); + sort_bufs[i] = NULL; + sort_done++; + } +} + +static void recv_bin(int fd, struct sanlk_state *st, char *bin) +{ + struct sanlk_resource *res; + + if (st->type == SANLK_STATE_LOCKSPACE) { + recv(fd, bin, sizeof(struct sanlk_lockspace), MSG_WAITALL); + + } else if (st->type == SANLK_STATE_RESOURCE) { + recv(fd, bin, sizeof(struct sanlk_resource), MSG_WAITALL); + + res = (struct sanlk_resource *)bin; + + recv(fd, bin+sizeof(struct sanlk_resource), + res->num_disks * sizeof(struct sanlk_disk), + MSG_WAITALL); + } +} + +int sanlock_status(int debug, char sort_arg) { struct sm_header h; - struct sanlk_state st; - char str[SANLK_STATE_MAXSTR]; - int fd, rv; + struct sanlk_state state; + char maxstr[SANLK_STATE_MAXSTR]; + char maxbin[SANLK_STATE_MAXSTR]; + struct sanlk_state *st; + char *buf, *str, *bin; + int fd, rv, len; + int sort_p = 0, sort_s = 0; + + if (sort_arg == 'p') + sort_p = 1; + else if (sort_arg == 's') + sort_s = 1; fd = send_command(SM_CMD_STATUS, 0); if (fd < 0) @@ -143,34 +338,74 @@ int sanlock_status(int debug) goto out; } + st = &state; + str = maxstr; + bin = maxbin; + while (1) { - rv = recv(fd, &st, sizeof(st), MSG_WAITALL); + if (sort_p || sort_s) { + len = sizeof(struct sanlk_state) + SANLK_STATE_MAXSTR*4; + buf = malloc(len); + if (!buf) + return -ENOMEM; + memset(buf, 0, len); + st = (struct sanlk_state *)buf; + str = buf + sizeof(struct sanlk_state); + bin = buf + sizeof(struct sanlk_state) + SANLK_STATE_MAXSTR; + } else { + memset(&state, 0, sizeof(state)); + memset(maxstr, 0, sizeof(maxstr)); + memset(maxbin, 0, sizeof(maxbin)); + } + + rv = recv(fd, st, sizeof(struct sanlk_state), MSG_WAITALL); if (!rv) break; - if (rv != sizeof(st)) + if (rv != sizeof(struct sanlk_state)) break; - if (st.str_len) { - rv = recv(fd, str, st.str_len, MSG_WAITALL); - if (rv != st.str_len) + if (st->str_len) { + rv = recv(fd, str, st->str_len, MSG_WAITALL); + if (rv != st->str_len) break; } - switch (st.type) { - case SANLK_STATE_DAEMON: - status_daemon(fd, &st, str, debug); - break; - case SANLK_STATE_CLIENT: - status_client(fd, &st, str, debug); - break; - case SANLK_STATE_LOCKSPACE: - status_lockspace(fd, &st, str, debug); - break; - case SANLK_STATE_RESOURCE: - status_resource(fd, &st, str, debug); - break; + recv_bin(fd, st, bin); + + if (sort_p || sort_s) { + if (sort_count == MAX_SORT_ENTRIES) { + printf("cannot sort over %d\n", MAX_SORT_ENTRIES); + goto out; + } + sort_bufs[sort_count++] = buf; + continue; } + + /* no sorting, print as received */ + + print_st(st, str, bin, debug); } + + if (sort_p) { + print_type(SANLK_STATE_DAEMON, debug); + print_p(-1, debug); + print_type(SANLK_STATE_LOCKSPACE, debug); + print_r_by_p(debug); + if (sort_done < sort_count) { + printf("-\n"); + print_type(0, debug); + } + } else if (sort_s) { + print_type(SANLK_STATE_DAEMON, debug); + print_p(-1, debug); + print_type(SANLK_STATE_CLIENT, debug); + print_r_by_s(debug); + if (sort_done < sort_count) { + printf("-\n"); + print_type(0, debug); + } + } + rv = 0; out: close(fd); @@ -224,12 +459,12 @@ int sanlock_host_status(int debug, char *lockspace_name) switch (st.type) { case SANLK_STATE_HOST: - status_host(fd, &st, str, debug); + status_host(&st, str, debug); break; } } - rv = 0; + rv = h.data; out: close(fd); return rv; diff --git a/src/client_cmd.h b/src/client_cmd.h index 3599659..dceb7ca 100644 --- a/src/client_cmd.h +++ b/src/client_cmd.h @@ -9,7 +9,7 @@ #ifndef __CLIENT_CMD_H__ #define __CLIENT_CMD_H__ -int sanlock_status(int debug); +int sanlock_status(int debug, char sort_arg); int sanlock_host_status(int debug, char *lockspace_name); int sanlock_log_dump(void); int sanlock_shutdown(void); diff --git a/src/main.c b/src/main.c index 5b7477d..d3d0c4f 100644 --- a/src/main.c +++ b/src/main.c @@ -2027,7 +2027,7 @@ static int print_state_resource(struct resource *r, char *str, const char *list_ "list=%s " "flags=%x " "lver=%llu " - "token_id=%u ", + "token_id=%u", list_name, r->flags, (unsigned long long)r->lver, @@ -2046,7 +2046,7 @@ static int print_state_host(struct host_status *hs, char *str) "last_req=%llu " "owner_id=%llu " "owner_generation=%llu " - "timestamp=%llu ", + "timestamp=%llu", (unsigned long long)hs->last_check, (unsigned long long)hs->last_live, (unsigned long long)hs->last_req, @@ -2255,13 +2255,19 @@ static void cmd_host_status(int fd, struct sm_header *h_recv) goto fail; } - send(fd, &h, sizeof(h), MSG_NOSIGNAL); - pthread_mutex_lock(&spaces_mutex); sp = find_lockspace(lockspace.name); - memcpy(status, &sp->host_status, status_len); + if (sp) + memcpy(status, &sp->host_status, status_len); pthread_mutex_unlock(&spaces_mutex); + if (!sp) { + h.data = -ENOSPC; + goto fail; + } + + send(fd, &h, sizeof(h), MSG_NOSIGNAL); + for (i = 0; i < DEFAULT_MAX_HOSTS; i++) { hs = &status[i]; if (!hs->last_live && !hs->owner_id) @@ -3007,7 +3013,7 @@ static void print_usage(void) printf(" -o 0|1 io timeout in seconds (%d)\n", DEFAULT_IO_TIMEOUT); printf("\n"); printf("sanlock client <action> [options]\n"); - printf("sanlock client status [-D]\n"); + printf("sanlock client status [-D] [-o p|s]\n"); printf("sanlock client host_status -s LOCKSPACE [-D]\n"); printf("sanlock client log_dump\n"); printf("sanlock client shutdown\n"); @@ -3231,9 +3237,13 @@ static int read_command_line(int argc, char *argv[]) com.high_priority = atoi(optionarg); break; case 'o': - com.io_timeout_arg = atoi(optionarg); - if (!com.io_timeout_arg) - com.io_timeout_arg = DEFAULT_IO_TIMEOUT; + if (com.action == ACT_STATUS) { + com.sort_arg = *optionarg; + } else { + com.io_timeout_arg = atoi(optionarg); + if (!com.io_timeout_arg) + com.io_timeout_arg = DEFAULT_IO_TIMEOUT; + } break; case 'n': com.num_hosts = atoi(optionarg); @@ -3344,7 +3354,7 @@ static int do_client(void) switch (com.action) { case ACT_STATUS: - rv = sanlock_status(com.debug); + rv = sanlock_status(com.debug, com.sort_arg); break; case ACT_HOST_STATUS: diff --git a/src/sanlock.8 b/src/sanlock.8 index 4462280..891ab86 100644 --- a/src/sanlock.8 +++ b/src/sanlock.8 @@ -247,6 +247,7 @@ io timeout in seconds Print processes, lockspaces, and resources being manged by the sanlock daemon. Add -D to show extra internal daemon status for debugging. +Add -o p to show resources by pid, or -o s to show resources by lockspace. .BR "sanlock client host_status -s" " LOCKSPACE" diff --git a/src/sanlock_internal.h b/src/sanlock_internal.h index 2621317..0b39934 100644 --- a/src/sanlock_internal.h +++ b/src/sanlock_internal.h @@ -492,6 +492,7 @@ struct command_line { int uid; /* -U */ int gid; /* -G */ int pid; /* -p */ + char sort_arg; uint64_t local_host_id; /* -i */ uint64_t local_host_generation; /* -g */ int num_hosts; /* -n */ commit ad80edea0471fc32221da1d2610d3197f9e31275 Author: David Teigland <teigland(a)redhat.com> Date: Mon Sep 12 14:26:58 2011 -0500 sanlock: improve status output diff --git a/src/client_cmd.c b/src/client_cmd.c index a3949d1..73404bf 100644 --- a/src/client_cmd.c +++ b/src/client_cmd.c @@ -58,15 +58,16 @@ static void print_debug(char *str, int len) static void status_daemon(int fd GNUC_UNUSED, struct sanlk_state *st, char *str, int debug) { - printf("daemon\n"); + printf("daemon %.48s\n", st->name); - if (st->str_len && debug) + if (st->str_len && debug) { print_debug(str, st->str_len); + } } static void status_client(int fd GNUC_UNUSED, struct sanlk_state *st, char *str, int debug) { - printf("pid %d ", st->data32); + printf("p %d ", st->data32); printf("%.48s\n", st->name); if (st->str_len && debug) @@ -80,7 +81,7 @@ static void status_lockspace(int fd, struct sanlk_state *st, char *str, int debu rv = recv(fd, &lockspace, sizeof(lockspace), MSG_WAITALL); - printf("lockspace %.48s:%llu:%s:%llu\n", + printf("s %.48s:%llu:%s:%llu\n", lockspace.name, (unsigned long long)lockspace.host_id, lockspace.host_id_disk.path, @@ -98,7 +99,7 @@ static void status_resource(int fd, struct sanlk_state *st, char *str, int debug rv = recv(fd, &resource, sizeof(resource), MSG_WAITALL); - printf("pid %d resource %.48s:%.48s", st->data32, resource.lockspace_name, resource.name); + printf("r %.48s:%.48s", resource.lockspace_name, resource.name); for (i = 0; i < resource.num_disks; i++) { rv = recv(fd, &disk, sizeof(disk), MSG_WAITALL); @@ -106,7 +107,7 @@ static void status_resource(int fd, struct sanlk_state *st, char *str, int debug printf(":%s:%llu", disk.path, (unsigned long long)disk.offset); } - printf("\n"); + printf(":%llu p %u\n", (unsigned long long)st->data64, st->data32); if (st->str_len && debug) print_debug(str, st->str_len); diff --git a/src/main.c b/src/main.c index b7cfe91..5b7477d 100644 --- a/src/main.c +++ b/src/main.c @@ -2064,6 +2064,7 @@ static void send_state_daemon(int fd) int str_len; memset(&st, 0, sizeof(st)); + strncpy(st.name, our_host_name_global, NAME_ID_SIZE); st.type = SANLK_STATE_DAEMON; @@ -2137,6 +2138,7 @@ static void send_state_resource(int fd, struct resource *r, const char *list_nam st.type = SANLK_STATE_RESOURCE; st.data32 = r->pid; + st.data64 = r->lver; strncpy(st.name, r->r.name, NAME_ID_SIZE); str_len = print_state_resource(r, str, list_name);

12 years, 2 months

1
0
0 / 0

src/Makefile src/client.c src/client_cmd.c src/client_cmd.h src/direct_lib.c src/host_id.c src/host_id.h src/log.c src/log.h src/main.c src/sanlock.8 src/sanlock_admin.h src/sanlock_internal.h src/sanlock_sock.h src/token_manager.c

by David Teigland

src/Makefile | 1 src/client.c | 172 ------------------- src/client_cmd.c | 287 ++++++++++++++++++++++++++++++++ src/client_cmd.h | 17 + src/direct_lib.c | 4 src/host_id.c | 56 +----- src/host_id.h | 5 src/log.c | 2 src/log.h | 2 src/main.c | 434 +++++++++++++++++++++++++++++++++---------------- src/sanlock.8 | 6 src/sanlock_admin.h | 8 src/sanlock_internal.h | 27 ++- src/sanlock_sock.h | 6 src/token_manager.c | 22 -- 15 files changed, 652 insertions(+), 397 deletions(-) New commits: commit 241606a1ae1d4db45cff28902e052e00d541b9e6 Author: David Teigland <teigland(a)redhat.com> Date: Thu Sep 8 17:27:13 2011 -0500 sanlock: new status and host_status Change the client status command to operate on global resource lists instead of per client tokens. Add new client host_status command to print a lockspace's saved info about hosts. diff --git a/src/Makefile b/src/Makefile index 8afbd8a..5d710e4 100644 --- a/src/Makefile +++ b/src/Makefile @@ -30,6 +30,7 @@ CMD_SOURCE = \ token_manager.c \ watchdog.c \ monotime.c \ + client_cmd.c \ sanlock_sock.c LIB_ENTIRE_SOURCE = \ diff --git a/src/client.c b/src/client.c index 4987867..ea25832 100644 --- a/src/client.c +++ b/src/client.c @@ -81,7 +81,9 @@ static int send_header(int sock, int cmd, uint32_t cmd_flags, int datalen, return 0; } -static int send_command(int cmd, uint32_t data) +int send_command(int cmd, uint32_t data); + +int send_command(int cmd, uint32_t data) { int rv, sock; @@ -114,174 +116,6 @@ static int recv_result(int fd) return (int)h.data; } -int sanlock_shutdown(void) -{ - int fd; - - fd = send_command(SM_CMD_SHUTDOWN, 0); - if (fd < 0) - return fd; - - close(fd); - return 0; -} - -int sanlock_log_dump(void) -{ - struct sm_header h; - char buf[4096]; - int fd, rv; - - fd = send_command(SM_CMD_LOG_DUMP, 0); - if (fd < 0) - return fd; - - memset(&h, 0, sizeof(h)); - - rv = recv(fd, &h, sizeof(h), MSG_WAITALL); - if (rv != sizeof(h)) { - rv = -1; - goto out; - } - - while (1) { - memset(buf, 0, sizeof(buf)); - - rv = recv(fd, buf, sizeof(buf) - 1, MSG_WAITALL); - - if (rv > 0) - printf("%s", buf); - else - break; - } - printf("\n"); - out: - close(fd); - return rv; -} - -static void print_debug(char *str, int len) -{ - char *p; - int i; - - p = &str[0]; - for (i = 0; i < len-1; i++) { - if (str[i] == ' ') { - str[i] = '\0'; - printf(" %s\n", p); - p = &str[i+1]; - } - } - - if (p) - printf(" %s\n", p); -} - -static void status_daemon(int fd GNUC_UNUSED, struct sanlk_state *st, char *str, int debug) -{ - printf("daemon\n"); - - if (st->str_len && debug) - print_debug(str, st->str_len); -} - -static void status_lockspace(int fd, struct sanlk_state *st, char *str, int debug) -{ - struct sanlk_lockspace lockspace; - int rv; - - rv = recv(fd, &lockspace, sizeof(lockspace), MSG_WAITALL); - - printf(" lockspace %.48s:%llu:%s:%llu\n", - lockspace.name, - (unsigned long long)lockspace.host_id, - lockspace.host_id_disk.path, - (unsigned long long)lockspace.host_id_disk.offset); - - if (st->str_len && debug) - print_debug(str, st->str_len); -} - -static void status_client(int fd GNUC_UNUSED, struct sanlk_state *st, char *str, int debug) -{ - printf("pid %d ", st->data32); - printf("%.48s\n", st->name); - - if (st->str_len && debug) - print_debug(str, st->str_len); -} - -static void status_resource(int fd, struct sanlk_state *st, char *str, int debug) -{ - struct sanlk_resource resource; - struct sanlk_disk disk; - int i, rv; - - rv = recv(fd, &resource, sizeof(resource), MSG_WAITALL); - - printf(" resource %.48s:%.48s", resource.lockspace_name, resource.name); - - for (i = 0; i < resource.num_disks; i++) { - rv = recv(fd, &disk, sizeof(disk), MSG_WAITALL); - - printf(":%s:%llu", - disk.path, (unsigned long long)disk.offset); - } - printf("\n"); - - if (st->str_len && debug) - print_debug(str, st->str_len); -} - -int sanlock_status(int debug) -{ - struct sm_header h; - struct sanlk_state st; - char str[SANLK_STATE_MAXSTR]; - int fd, rv; - - fd = send_command(SM_CMD_STATUS, 0); - if (fd < 0) - return fd; - - rv = recv(fd, &h, sizeof(h), MSG_WAITALL); - if (rv != sizeof(h)) - return -1; - - - while (1) { - rv = recv(fd, &st, sizeof(st), MSG_WAITALL); - if (!rv) - break; - if (rv != sizeof(st)) - return -1; - - if (st.str_len) { - rv = recv(fd, str, st.str_len, MSG_WAITALL); - if (rv != st.str_len) - return -1; - } - - switch (st.type) { - case SANLK_STATE_DAEMON: - status_daemon(fd, &st, str, debug); - break; - case SANLK_STATE_LOCKSPACE: - status_lockspace(fd, &st, str, debug); - break; - case SANLK_STATE_CLIENT: - status_client(fd, &st, str, debug); - break; - case SANLK_STATE_RESOURCE: - status_resource(fd, &st, str, debug); - break; - } - } - - return 0; -} - static int cmd_lockspace(int cmd, struct sanlk_lockspace *ls, uint32_t flags) { int rv, fd; diff --git a/src/client_cmd.c b/src/client_cmd.c new file mode 100644 index 0000000..a3949d1 --- /dev/null +++ b/src/client_cmd.c @@ -0,0 +1,287 @@ +/* + * Copyright (C) 2010-2011 Red Hat, Inc. All rights reserved. + * + * This copyrighted material is made available to anyone wishing to use, + * modify, copy, or redistribute it subject to the terms and conditions + * of the GNU General Public License v.2. + */ + +#include <inttypes.h> +#include <unistd.h> +#include <stdio.h> +#include <stdlib.h> +#include <stdint.h> +#include <stddef.h> +#include <fcntl.h> +#include <string.h> +#include <errno.h> +#include <limits.h> +#include <time.h> +#include <syslog.h> +#include <pthread.h> +#include <poll.h> +#include <sys/types.h> +#include <sys/wait.h> +#include <sys/stat.h> +#include <sys/socket.h> +#include <sys/time.h> +#include <sys/un.h> + +#include "sanlock.h" +#include "sanlock_sock.h" + +#include "client_cmd.h" + +#ifndef GNUC_UNUSED +#define GNUC_UNUSED __attribute__((__unused__)) +#endif + +extern int send_command(int cmd, uint32_t data); + +static void print_debug(char *str, int len) +{ + char *p; + int i; + + p = &str[0]; + for (i = 0; i < len-1; i++) { + if (str[i] == ' ') { + str[i] = '\0'; + printf(" %s\n", p); + p = &str[i+1]; + } + } + + if (p) + printf(" %s\n", p); +} + +static void status_daemon(int fd GNUC_UNUSED, struct sanlk_state *st, char *str, int debug) +{ + printf("daemon\n"); + + if (st->str_len && debug) + print_debug(str, st->str_len); +} + +static void status_client(int fd GNUC_UNUSED, struct sanlk_state *st, char *str, int debug) +{ + printf("pid %d ", st->data32); + printf("%.48s\n", st->name); + + if (st->str_len && debug) + print_debug(str, st->str_len); +} + +static void status_lockspace(int fd, struct sanlk_state *st, char *str, int debug) +{ + struct sanlk_lockspace lockspace; + int rv; + + rv = recv(fd, &lockspace, sizeof(lockspace), MSG_WAITALL); + + printf("lockspace %.48s:%llu:%s:%llu\n", + lockspace.name, + (unsigned long long)lockspace.host_id, + lockspace.host_id_disk.path, + (unsigned long long)lockspace.host_id_disk.offset); + + if (st->str_len && debug) + print_debug(str, st->str_len); +} + +static void status_resource(int fd, struct sanlk_state *st, char *str, int debug) +{ + struct sanlk_resource resource; + struct sanlk_disk disk; + int i, rv; + + rv = recv(fd, &resource, sizeof(resource), MSG_WAITALL); + + printf("pid %d resource %.48s:%.48s", st->data32, resource.lockspace_name, resource.name); + + for (i = 0; i < resource.num_disks; i++) { + rv = recv(fd, &disk, sizeof(disk), MSG_WAITALL); + + printf(":%s:%llu", + disk.path, (unsigned long long)disk.offset); + } + printf("\n"); + + if (st->str_len && debug) + print_debug(str, st->str_len); +} + +static void status_host(int fd GNUC_UNUSED, struct sanlk_state *st, char *str, int debug) +{ + printf("%u timestamp %llu\n", st->data32, + (unsigned long long)st->data64); + + if (st->str_len && debug) + print_debug(str, st->str_len); +} + +int sanlock_status(int debug) +{ + struct sm_header h; + struct sanlk_state st; + char str[SANLK_STATE_MAXSTR]; + int fd, rv; + + fd = send_command(SM_CMD_STATUS, 0); + if (fd < 0) + return fd; + + rv = recv(fd, &h, sizeof(h), MSG_WAITALL); + if (rv < 0) { + rv = -errno; + goto out; + } + if (rv != sizeof(h)) { + rv = -1; + goto out; + } + + while (1) { + rv = recv(fd, &st, sizeof(st), MSG_WAITALL); + if (!rv) + break; + if (rv != sizeof(st)) + break; + + if (st.str_len) { + rv = recv(fd, str, st.str_len, MSG_WAITALL); + if (rv != st.str_len) + break; + } + + switch (st.type) { + case SANLK_STATE_DAEMON: + status_daemon(fd, &st, str, debug); + break; + case SANLK_STATE_CLIENT: + status_client(fd, &st, str, debug); + break; + case SANLK_STATE_LOCKSPACE: + status_lockspace(fd, &st, str, debug); + break; + case SANLK_STATE_RESOURCE: + status_resource(fd, &st, str, debug); + break; + } + } + rv = 0; + out: + close(fd); + return rv; +} + +int sanlock_host_status(int debug, char *lockspace_name) +{ + struct sm_header h; + struct sanlk_state st; + struct sanlk_lockspace lockspace; + char str[SANLK_STATE_MAXSTR]; + int fd, rv; + + if (!lockspace_name || !lockspace_name[0]) + return -1; + + fd = send_command(SM_CMD_HOST_STATUS, 0); + if (fd < 0) + return fd; + + memset(&lockspace, 0, sizeof(lockspace)); + snprintf(lockspace.name, SANLK_NAME_LEN, "%s", lockspace_name); + + rv = send(fd, &lockspace, sizeof(lockspace), 0); + if (rv < 0) + goto out; + + rv = recv(fd, &h, sizeof(h), MSG_WAITALL); + if (rv < 0) { + rv = -errno; + goto out; + } + if (rv != sizeof(h)) { + rv = -1; + goto out; + } + + while (1) { + rv = recv(fd, &st, sizeof(st), MSG_WAITALL); + if (!rv) + break; + if (rv != sizeof(st)) + break; + + if (st.str_len) { + rv = recv(fd, str, st.str_len, MSG_WAITALL); + if (rv != st.str_len) + break; + } + + switch (st.type) { + case SANLK_STATE_HOST: + status_host(fd, &st, str, debug); + break; + } + } + + rv = 0; + out: + close(fd); + return rv; +} + +int sanlock_log_dump(void) +{ + struct sm_header h; + char buf[4096]; + int fd, rv; + + fd = send_command(SM_CMD_LOG_DUMP, 0); + if (fd < 0) + return fd; + + memset(&h, 0, sizeof(h)); + + rv = recv(fd, &h, sizeof(h), MSG_WAITALL); + if (rv < 0) { + rv = -errno; + goto out; + } + if (rv != sizeof(h)) { + rv = -1; + goto out; + } + + + while (1) { + memset(buf, 0, sizeof(buf)); + + rv = recv(fd, buf, sizeof(buf) - 1, MSG_WAITALL); + + if (rv > 0) + printf("%s", buf); + else + break; + } + printf("\n"); + out: + close(fd); + return rv; +} + +int sanlock_shutdown(void) +{ + int fd; + + fd = send_command(SM_CMD_SHUTDOWN, 0); + if (fd < 0) + return fd; + + close(fd); + return 0; +} + diff --git a/src/client_cmd.h b/src/client_cmd.h new file mode 100644 index 0000000..3599659 --- /dev/null +++ b/src/client_cmd.h @@ -0,0 +1,17 @@ +/* + * Copyright (C) 2010-2011 Red Hat, Inc. All rights reserved. + * + * This copyrighted material is made available to anyone wishing to use, + * modify, copy, or redistribute it subject to the terms and conditions + * of the GNU General Public License v.2. + */ + +#ifndef __CLIENT_CMD_H__ +#define __CLIENT_CMD_H__ + +int sanlock_status(int debug); +int sanlock_host_status(int debug, char *lockspace_name); +int sanlock_log_dump(void); +int sanlock_shutdown(void); + +#endif diff --git a/src/direct_lib.c b/src/direct_lib.c index 7d7f9a9..59513ab 100644 --- a/src/direct_lib.c +++ b/src/direct_lib.c @@ -21,11 +21,11 @@ #include "direct.h" #include "task.h" -void log_level(int space_id GNUC_UNUSED, int token_id GNUC_UNUSED, +void log_level(uint32_t space_id GNUC_UNUSED, uint32_t token_id GNUC_UNUSED, char *name GNUC_UNUSED, int level GNUC_UNUSED, const char *fmt GNUC_UNUSED, ...); -void log_level(int space_id GNUC_UNUSED, int token_id GNUC_UNUSED, +void log_level(uint32_t space_id GNUC_UNUSED, uint32_t token_id GNUC_UNUSED, char *name GNUC_UNUSED, int level GNUC_UNUSED, const char *fmt GNUC_UNUSED, ...) { diff --git a/src/host_id.c b/src/host_id.c index 86df60b..4d39d82 100644 --- a/src/host_id.c +++ b/src/host_id.c @@ -34,43 +34,7 @@ #include "direct.h" #include "token_manager.h" -static unsigned int space_id_counter = 1; - -struct list_head spaces; -struct list_head spaces_add; -struct list_head spaces_rem; -pthread_mutex_t spaces_mutex = PTHREAD_MUTEX_INITIALIZER; - -int print_space_state(struct space *sp, char *str) -{ - memset(str, 0, SANLK_STATE_MAXSTR); - - snprintf(str, SANLK_STATE_MAXSTR-1, - "space_id=%u " - "host_generation=%llu " - "space_dead=%d " - "killing_pids=%d " - "corrupt_result=%d " - "acquire_last_result=%d " - "renewal_last_result=%d " - "acquire_last_attempt=%llu " - "acquire_last_success=%llu " - "renewal_last_attempt=%llu " - "renewal_last_success=%llu", - sp->space_id, - (unsigned long long)sp->host_generation, - sp->space_dead, - sp->killing_pids, - sp->lease_status.corrupt_result, - sp->lease_status.acquire_last_result, - sp->lease_status.renewal_last_result, - (unsigned long long)sp->lease_status.acquire_last_attempt, - (unsigned long long)sp->lease_status.acquire_last_success, - (unsigned long long)sp->lease_status.renewal_last_attempt, - (unsigned long long)sp->lease_status.renewal_last_success); - - return strlen(str) + 1; -} +static uint32_t space_id_counter = 1; static struct space *_search_space(char *name, struct sync_disk *disk, @@ -123,6 +87,11 @@ static struct space *_search_space(char *name, return NULL; } +struct space *find_lockspace(char *name) +{ + return _search_space(name, NULL, 0, &spaces, &spaces_rem, &spaces_add); +} + int _lockspace_info(char *space_name, struct space *sp_out) { struct space *sp; @@ -290,9 +259,6 @@ void check_other_leases(struct task *task, struct space *sp, char *buf) new = 0; for (i = 0; i < DEFAULT_MAX_HOSTS; i++) { - if (i+1 == sp->host_id) - continue; - hs = &sp->host_status[i]; hs->last_check = now; @@ -309,6 +275,9 @@ void check_other_leases(struct task *task, struct space *sp, char *buf) hs->timestamp = leader->timestamp; hs->last_live = now; + if (i+1 == sp->host_id) + continue; + bitmap = (char *)leader + HOSTID_BITMAP_OFFSET; if (!test_id_bit(sp->host_id, bitmap)) @@ -834,10 +803,3 @@ void free_lockspaces(int wait) pthread_mutex_unlock(&spaces_mutex); } -void setup_spaces(void) -{ - INIT_LIST_HEAD(&spaces); - INIT_LIST_HEAD(&spaces_add); - INIT_LIST_HEAD(&spaces_rem); -} - diff --git a/src/host_id.h b/src/host_id.h index fba2d03..00dbcf7 100644 --- a/src/host_id.h +++ b/src/host_id.h @@ -9,20 +9,17 @@ #ifndef __HOST_ID_H__ #define __HOST_ID__H__ -int print_space_state(struct space *sp, char *str); +struct space *find_lockspace(char *name); int _lockspace_info(char *space_name, struct space *sp_out); int lockspace_info(char *space_name, struct space *sp_out); int lockspace_disk(char *space_name, struct sync_disk *disk); -void block_watchdog_updates(char *space_name); int host_info(char *space_name, uint64_t host_id, struct host_status *hs_out); int host_status_set_bit(char *space_name, uint64_t host_id); -int host_status_clear_bit(char *space_name, uint64_t host_id); int test_id_bit(int host_id, char *bitmap); int check_our_lease(struct task *task, struct space *sp, int *check_all, char *check_buf); void check_other_leases(struct task *task, struct space *sp, char *buf); int add_lockspace(struct sanlk_lockspace *ls); int rem_lockspace(struct sanlk_lockspace *ls); void free_lockspaces(int wait); -void setup_spaces(void); #endif diff --git a/src/log.c b/src/log.c index 3cb4064..72b424f 100644 --- a/src/log.c +++ b/src/log.c @@ -100,7 +100,7 @@ static void _log_save_ent(int level, int len) * logfile and/or syslog (so callers don't block writing messages to files) */ -void log_level(int space_id, int token_id, char *name_in, int level, const char *fmt, ...) +void log_level(uint32_t space_id, uint32_t token_id, char *name_in, int level, const char *fmt, ...) { va_list ap; char name[NAME_ID_SIZE + 1]; diff --git a/src/log.h b/src/log.h index eacb91b..8661eb0 100644 --- a/src/log.h +++ b/src/log.h @@ -9,7 +9,7 @@ #ifndef __LOG_H__ #define __LOG_H__ -void log_level(int space_id, int token_id, char *name_in, int level, const char *fmt, ...) +void log_level(uint32_t space_id, uint32_t token_id, char *name_in, int level, const char *fmt, ...) __attribute__((format(printf, 5, 6))); int setup_logging(void); diff --git a/src/main.c b/src/main.c index c249ace..b7cfe91 100644 --- a/src/main.c +++ b/src/main.c @@ -49,6 +49,7 @@ #include "lockfile.h" #include "watchdog.h" #include "task.h" +#include "client_cmd.h" /* priorities are LOG_* from syslog.h */ int log_logfile_priority = LOG_WARNING; @@ -82,7 +83,7 @@ static struct pollfd *pollfd = NULL; static char command[COMMAND_MAX]; static int cmd_argc; static char **cmd_argv; -static unsigned int token_id_counter = 1; +static uint32_t token_id_counter = 1; struct cmd_args { struct list_head list; /* thread_pool data */ @@ -106,10 +107,6 @@ struct thread_pool { static struct thread_pool pool; -extern struct list_head spaces; -extern struct list_head spaces_rem; -extern pthread_mutex_t spaces_mutex; - static struct random_data rand_data; static char rand_state[32]; static pthread_mutex_t rand_mutex = PTHREAD_MUTEX_INITIALIZER; @@ -1921,7 +1918,27 @@ static int thread_pool_create(int min_workers, int max_workers) return rv; } -static int print_daemon_state(char *str) +/* + * sanlock client status + * + * 1. send_state_daemon + * + * 2. for each cl in clients + * send_state_client() [sanlk_state + str_len] + * + * 3. for each sp in spaces, spaces_add, spaces_rem + * send_state_lockspace() [sanlk_state + str_len + sanlk_lockspace] + * + * 4. for each r in resources, dispose_resources + * send_state_resource() [sanlk_state + str_len + sanlk_resource + sanlk_disk * num_disks] + * + * sanlock client host_status <lockspace_name> + * + * 1. for each hs in sp->host_status + * send_state_host() + */ + +static int print_state_daemon(char *str) { memset(str, 0, SANLK_STATE_MAXSTR); @@ -1942,7 +1959,7 @@ static int print_daemon_state(char *str) return strlen(str) + 1; } -static int print_client_state(struct client *cl, int ci, char *str) +static int print_state_client(struct client *cl, int ci, char *str) { memset(str, 0, SANLK_STATE_MAXSTR); @@ -1969,180 +1986,295 @@ static int print_client_state(struct client *cl, int ci, char *str) return strlen(str) + 1; } -static int print_token_state(struct token *t, char *str) +static int print_state_lockspace(struct space *sp, char *str, const char *list_name) { memset(str, 0, SANLK_STATE_MAXSTR); snprintf(str, SANLK_STATE_MAXSTR-1, - "token_id=%u " - "acquire_result=%d " - "release_result=%d " - "leader.lver=%llu " - "leader.timestamp=%llu " - "leader.owner_id=%llu " - "leader.owner_generation=%llu", - t->token_id, - t->acquire_result, - t->release_result, - (unsigned long long)t->leader.lver, - (unsigned long long)t->leader.timestamp, - (unsigned long long)t->leader.owner_id, - (unsigned long long)t->leader.owner_generation); + "list=%s " + "space_id=%u " + "host_generation=%llu " + "space_dead=%d " + "killing_pids=%d " + "corrupt_result=%d " + "acquire_last_result=%d " + "renewal_last_result=%d " + "acquire_last_attempt=%llu " + "acquire_last_success=%llu " + "renewal_last_attempt=%llu " + "renewal_last_success=%llu", + list_name, + sp->space_id, + (unsigned long long)sp->host_generation, + sp->space_dead, + sp->killing_pids, + sp->lease_status.corrupt_result, + sp->lease_status.acquire_last_result, + sp->lease_status.renewal_last_result, + (unsigned long long)sp->lease_status.acquire_last_attempt, + (unsigned long long)sp->lease_status.acquire_last_success, + (unsigned long long)sp->lease_status.renewal_last_attempt, + (unsigned long long)sp->lease_status.renewal_last_success); return strlen(str) + 1; } -/* - * 0. header - * 1. dst (sanlk_state DAEMON) - * 2. dst.str (dst.len) - * 3. lst (sanlk_state LOCKSPACE) - * 4. lst.str (lst.len) print_space_state() - * 5. lockspace (sanlk_lockspace) - * 6. [repeat 3-5 for each space] - * 7. cst (sanlk_state CLIENT) - * 8. cst.str (cst.len) print_client_state() - * 9. rst (sanlk_state RESOURCE) - * 10. rst.str (rst.len) print_token_state() - * 11. resource (sanlk_resource) - * 12. disks (sanlk_disk * resource.num_disks) - * 13. [repeat 9-12 for each token] - * 14. [repeat 7-13 for each client] - */ +static int print_state_resource(struct resource *r, char *str, const char *list_name) +{ + memset(str, 0, SANLK_STATE_MAXSTR); -/* - * TODO: - * increase max transfer size - * send spaces/spaces_add/spaces_rem instead of just spaces - * use resources/dispose_resources to send resource state instead of cl->tokens - * - * . daemon - * . clients - * . lockspaces from spaces/spaces_add/spaces_rem - * . resources from resources/dispose_resources - * - * in print function, - * for each client pid, go through resources and print any for that pid, clear the res - * print any remaining detached resources (orphan, dispose) - * - * sanlock client host_status <lockspace_name> - * send sp->host_status[] for the named lockspace - */ + snprintf(str, SANLK_STATE_MAXSTR-1, + "list=%s " + "flags=%x " + "lver=%llu " + "token_id=%u ", + list_name, + r->flags, + (unsigned long long)r->lver, + r->token_id); -static void cmd_status(int fd, struct sm_header *h_recv) + return strlen(str) + 1; +} + +static int print_state_host(struct host_status *hs, char *str) { - struct sm_header h; - struct sanlk_state dst; - struct sanlk_state lst; - struct sanlk_state cst; - struct sanlk_state rst; - struct sanlk_lockspace lockspace; + memset(str, 0, SANLK_STATE_MAXSTR); + + snprintf(str, SANLK_STATE_MAXSTR-1, + "last_check=%llu " + "last_live=%llu " + "last_req=%llu " + "owner_id=%llu " + "owner_generation=%llu " + "timestamp=%llu ", + (unsigned long long)hs->last_check, + (unsigned long long)hs->last_live, + (unsigned long long)hs->last_req, + (unsigned long long)hs->owner_id, + (unsigned long long)hs->owner_generation, + (unsigned long long)hs->timestamp); + + return strlen(str) + 1; +} + +static void send_state_daemon(int fd) +{ + struct sanlk_state st; char str[SANLK_STATE_MAXSTR]; - struct token *token; - struct space *sp; - struct client *cl; - int ci, i, j, str_len; + int str_len; - /* - * send header: h - */ + memset(&st, 0, sizeof(st)); - memset(&h, 0, sizeof(h)); - memcpy(&h, h_recv, sizeof(struct sm_header)); - h.length = sizeof(h); - h.data = 0; + st.type = SANLK_STATE_DAEMON; - send(fd, &h, sizeof(h), MSG_NOSIGNAL); + str_len = print_state_daemon(str); - /* - * send daemon state: dst, dst.str - */ + st.str_len = str_len; - str_len = print_daemon_state(str); - memset(&dst, 0, sizeof(dst)); - dst.type = SANLK_STATE_DAEMON; - dst.str_len = str_len; + send(fd, &st, sizeof(st), MSG_NOSIGNAL); + if (str_len) + send(fd, str, str_len, MSG_NOSIGNAL); +} - send(fd, &dst, sizeof(dst), MSG_NOSIGNAL); +static void send_state_client(int fd, struct client *cl, int ci) +{ + struct sanlk_state st; + char str[SANLK_STATE_MAXSTR]; + int str_len; + + memset(&st, 0, sizeof(st)); + + st.type = SANLK_STATE_CLIENT; + st.data32 = cl->pid; + strncpy(st.name, cl->owner_name, NAME_ID_SIZE); + + str_len = print_state_client(cl, ci, str); + + st.str_len = str_len; + + send(fd, &st, sizeof(st), MSG_NOSIGNAL); if (str_len) send(fd, str, str_len, MSG_NOSIGNAL); +} - if (h_recv->data == SANLK_STATE_DAEMON) - return; +static void send_state_lockspace(int fd, struct space *sp, const char *list_name) +{ + struct sanlk_state st; + struct sanlk_lockspace lockspace; + char str[SANLK_STATE_MAXSTR]; + int str_len; - /* - * send lockspace state: lst, lst.str, sanlk_lockspace - */ + memset(&st, 0, sizeof(st)); - pthread_mutex_lock(&spaces_mutex); - list_for_each_entry(sp, &spaces, list) { - str_len = print_space_state(sp, str); - memset(&lst, 0, sizeof(lst)); - lst.type = SANLK_STATE_LOCKSPACE; - lst.data64 = sp->host_id; - strncpy(lst.name, sp->space_name, NAME_ID_SIZE); - lst.str_len = str_len; + st.type = SANLK_STATE_LOCKSPACE; + st.data64 = sp->host_id; + strncpy(st.name, sp->space_name, NAME_ID_SIZE); + + str_len = print_state_lockspace(sp, str, list_name); + + st.str_len = str_len; + + send(fd, &st, sizeof(st), MSG_NOSIGNAL); + if (str_len) + send(fd, str, str_len, MSG_NOSIGNAL); - send(fd, &lst, sizeof(lst), MSG_NOSIGNAL); - if (str_len) - send(fd, str, str_len, MSG_NOSIGNAL); + memset(&lockspace, 0, sizeof(struct sanlk_lockspace)); + strncpy(lockspace.name, sp->space_name, NAME_ID_SIZE); + lockspace.host_id = sp->host_id; + memcpy(&lockspace.host_id_disk, &sp->host_id_disk, sizeof(struct sanlk_disk)); - memset(&lockspace, 0, sizeof(struct sanlk_lockspace)); - strncpy(lockspace.name, sp->space_name, NAME_ID_SIZE); - lockspace.host_id = sp->host_id; - memcpy(&lockspace.host_id_disk, &sp->host_id_disk, sizeof(struct sanlk_disk)); + send(fd, &lockspace, sizeof(lockspace), MSG_NOSIGNAL); +} + +static void send_state_resource(int fd, struct resource *r, const char *list_name) +{ + struct sanlk_state st; + char str[SANLK_STATE_MAXSTR]; + int str_len; + int i; - send(fd, &lockspace, sizeof(lockspace), MSG_NOSIGNAL); + memset(&st, 0, sizeof(st)); + + st.type = SANLK_STATE_RESOURCE; + st.data32 = r->pid; + strncpy(st.name, r->r.name, NAME_ID_SIZE); + + str_len = print_state_resource(r, str, list_name); + + st.str_len = str_len; + + send(fd, &st, sizeof(st), MSG_NOSIGNAL); + if (str_len) + send(fd, str, str_len, MSG_NOSIGNAL); + + send(fd, &r->r, sizeof(struct sanlk_resource), MSG_NOSIGNAL); + + for (i = 0; i < r->r.num_disks; i++) { + send(fd, &r->r.disks[i], sizeof(struct sanlk_disk), MSG_NOSIGNAL); } - pthread_mutex_unlock(&spaces_mutex); +} - if (h_recv->data == SANLK_STATE_LOCKSPACE) - return; +static void send_state_host(int fd, struct host_status *hs, int host_id) +{ + struct sanlk_state st; + char str[SANLK_STATE_MAXSTR]; + int str_len; - /* - * send client and resource state: - * cst, cst.str, (rst, rst.str, resource, disk*N)*M - */ + memset(&st, 0, sizeof(st)); + + st.type = SANLK_STATE_HOST; + st.data32 = host_id; + st.data64 = hs->timestamp; + + str_len = print_state_host(hs, str); + + st.str_len = str_len; + + send(fd, &st, sizeof(st), MSG_NOSIGNAL); + if (str_len) + send(fd, str, str_len, MSG_NOSIGNAL); +} + +static void cmd_status(int fd, struct sm_header *h_recv) +{ + struct sm_header h; + struct client *cl; + struct space *sp; + struct resource *r; + int ci; + + memset(&h, 0, sizeof(h)); + memcpy(&h, h_recv, sizeof(struct sm_header)); + h.length = sizeof(h); + h.data = 0; + + send(fd, &h, sizeof(h), MSG_NOSIGNAL); + + send_state_daemon(fd); + + if (h_recv->data == SANLK_STATE_DAEMON) + return; for (ci = 0; ci <= client_maxi; ci++) { cl = &client[ci]; - if (!cl->used) continue; + send_state_client(fd, cl, ci); + } - str_len = print_client_state(cl, ci, str); - memset(&cst, 0, sizeof(cst)); - cst.type = SANLK_STATE_CLIENT; - cst.data32 = cl->pid; - strncpy(cst.name, cl->owner_name, NAME_ID_SIZE); - cst.str_len = str_len; + if (h_recv->data == SANLK_STATE_CLIENT) + return; - send(fd, &cst, sizeof(cst), MSG_NOSIGNAL); - if (str_len) - send(fd, str, str_len, MSG_NOSIGNAL); + pthread_mutex_lock(&spaces_mutex); + list_for_each_entry(sp, &spaces, list) + send_state_lockspace(fd, sp, "spaces"); + list_for_each_entry(sp, &spaces_rem, list) + send_state_lockspace(fd, sp, "spaces_rem"); + list_for_each_entry(sp, &spaces_rem, list) + send_state_lockspace(fd, sp, "spaces_add"); + pthread_mutex_unlock(&spaces_mutex); - for (i = 0; i < SANLK_MAX_RESOURCES; i++) { - token = cl->tokens[i]; - if (!token) - continue; + if (h_recv->data == SANLK_STATE_LOCKSPACE) + return; + + pthread_mutex_lock(&resource_mutex); + list_for_each_entry(r, &resources, list) + send_state_resource(fd, r, "resources"); + list_for_each_entry(r, &dispose_resources, list) + send_state_resource(fd, r, "dispose_resources"); + pthread_mutex_unlock(&resource_mutex); +} + +static void cmd_host_status(int fd, struct sm_header *h_recv) +{ + struct sm_header h; + struct sanlk_lockspace lockspace; + struct space *sp; + struct host_status *hs, *status = NULL; + int status_len; + int i, rv; - str_len = print_token_state(token, str); - memset(&rst, 0, sizeof(rst)); - rst.type = SANLK_STATE_RESOURCE; - strncpy(rst.name, token->r.name, NAME_ID_SIZE); - rst.str_len = str_len; + memset(&h, 0, sizeof(h)); + memcpy(&h, h_recv, sizeof(struct sm_header)); + h.length = sizeof(h); + h.data = 0; - send(fd, &rst, sizeof(rst), MSG_NOSIGNAL); - if (str_len) - send(fd, str, str_len, MSG_NOSIGNAL); + status_len = sizeof(struct host_status) * DEFAULT_MAX_HOSTS; - send(fd, &token->r, sizeof(struct sanlk_resource), MSG_NOSIGNAL); + status = malloc(status_len); + if (!status) { + h.data = -ENOMEM; + goto fail; + } - for (j = 0; j < token->r.num_disks; j++) { - send(fd, &token->disks[j], sizeof(struct sanlk_disk), MSG_NOSIGNAL); - } - } + rv = recv(fd, &lockspace, sizeof(struct sanlk_lockspace), MSG_WAITALL); + if (rv != sizeof(struct sanlk_lockspace)) { + h.data = -ENOTCONN; + goto fail; + } + + send(fd, &h, sizeof(h), MSG_NOSIGNAL); + + pthread_mutex_lock(&spaces_mutex); + sp = find_lockspace(lockspace.name); + memcpy(status, &sp->host_status, status_len); + pthread_mutex_unlock(&spaces_mutex); + + for (i = 0; i < DEFAULT_MAX_HOSTS; i++) { + hs = &status[i]; + if (!hs->last_live && !hs->owner_id) + continue; + send_state_host(fd, hs, i+1); } + + if (status) + free(status); + return; + fail: + send(fd, &h, sizeof(h), MSG_NOSIGNAL); + + if (status) + free(status); } static void cmd_log_dump(int fd, struct sm_header *h_recv) @@ -2354,6 +2486,10 @@ static void process_cmd_daemon(int ci, struct sm_header *h_recv) strcpy(client[ci].owner_name, "status"); cmd_status(fd, h_recv); break; + case SM_CMD_HOST_STATUS: + strcpy(client[ci].owner_name, "host_status"); + cmd_host_status(fd, h_recv); + break; case SM_CMD_LOG_DUMP: strcpy(client[ci].owner_name, "log_dump"); cmd_log_dump(fd, h_recv); @@ -2401,6 +2537,7 @@ static void process_connection(int ci) case SM_CMD_RESTRICT: case SM_CMD_SHUTDOWN: case SM_CMD_STATUS: + case SM_CMD_HOST_STATUS: case SM_CMD_LOG_DUMP: process_cmd_daemon(ci, &h); break; @@ -2644,8 +2781,6 @@ static int do_daemon(void) if (rv < 0) goto out_threads; - setup_spaces(); - main_loop(); close_token_manager(); @@ -2871,6 +3006,7 @@ static void print_usage(void) printf("\n"); printf("sanlock client <action> [options]\n"); printf("sanlock client status [-D]\n"); + printf("sanlock client host_status -s LOCKSPACE [-D]\n"); printf("sanlock client log_dump\n"); printf("sanlock client shutdown\n"); printf("sanlock client init -s LOCKSPACE | -r RESOURCE\n"); @@ -2968,6 +3104,8 @@ static int read_command_line(int argc, char *argv[]) case COM_CLIENT: if (!strcmp(act, "status")) com.action = ACT_STATUS; + else if (!strcmp(act, "host_status")) + com.action = ACT_HOST_STATUS; else if (!strcmp(act, "log_dump")) com.action = ACT_LOG_DUMP; else if (!strcmp(act, "shutdown")) @@ -3207,6 +3345,10 @@ static int do_client(void) rv = sanlock_status(com.debug); break; + case ACT_HOST_STATUS: + rv = sanlock_host_status(com.debug, com.lockspace.name); + break; + case ACT_LOG_DUMP: rv = sanlock_log_dump(); break; @@ -3468,6 +3610,16 @@ int main(int argc, char *argv[]) BUILD_BUG_ON(sizeof(struct sanlk_disk) != sizeof(struct sync_disk)); BUILD_BUG_ON(sizeof(struct leader_record) > LEADER_RECORD_MAX); + + /* initialize global variables */ + pthread_mutex_init(&spaces_mutex, NULL); + pthread_mutex_init(&resource_mutex, NULL); + pthread_cond_init(&resource_cond, NULL); + INIT_LIST_HEAD(&spaces); + INIT_LIST_HEAD(&spaces_rem); + INIT_LIST_HEAD(&spaces_add); + INIT_LIST_HEAD(&resources); + INIT_LIST_HEAD(&dispose_resources); memset(&com, 0, sizeof(com)); com.use_watchdog = DEFAULT_USE_WATCHDOG; diff --git a/src/sanlock.8 b/src/sanlock.8 index e3196e9..4462280 100644 --- a/src/sanlock.8 +++ b/src/sanlock.8 @@ -248,6 +248,12 @@ io timeout in seconds Print processes, lockspaces, and resources being manged by the sanlock daemon. Add -D to show extra internal daemon status for debugging. +.BR "sanlock client host_status -s" " LOCKSPACE" + +Print state of host_id delta leases read during the last renewal. +Only lockspace_name is used from the LOCKSPACE argument. +Add -D to show extra internal daemon status for debugging. + .B sanlock client log_dump Print the sanlock daemon internal debug log. diff --git a/src/sanlock_admin.h b/src/sanlock_admin.h index 3a64a2d..802beee 100644 --- a/src/sanlock_admin.h +++ b/src/sanlock_admin.h @@ -11,14 +11,6 @@ #define __SANLOCK_ADMIN_H__ /* - * daemon admin/managment - */ - -int sanlock_status(int debug); -int sanlock_log_dump(void); -int sanlock_shutdown(void); - -/* * add_lockspace returns: * 0: the lockspace has been added successfully * -EEXIST: the lockspace already exists diff --git a/src/sanlock_internal.h b/src/sanlock_internal.h index 7a26749..2621317 100644 --- a/src/sanlock_internal.h +++ b/src/sanlock_internal.h @@ -79,7 +79,7 @@ struct token { uint64_t host_generation; /* internal */ - int token_id; /* used to refer to this token instance in log messages */ + uint32_t token_id; /* used to refer to this token instance in log messages */ int space_dead; /* don't bother trying to release if ls is dead */ int acquire_result; int release_result; @@ -89,6 +89,18 @@ struct token { struct sanlk_resource r; }; +#define R_EXAMINE 0x00000001 + +struct resource { + struct list_head list; + struct token *token; + int pid; + uint32_t token_id; + uint32_t flags; + uint64_t lver; + struct sanlk_resource r; +}; + struct lease_status { int corrupt_result; int acquire_last_result; @@ -116,11 +128,11 @@ struct host_status { struct space { struct list_head list; char space_name[NAME_ID_SIZE]; + uint32_t space_id; /* used to refer to this space instance in log messages */ uint64_t host_id; uint64_t host_generation; struct sync_disk host_id_disk; int align_size; - int space_id; /* used to refer to this space instance in log messages */ int space_dead; int killing_pids; int external_remove; @@ -502,6 +514,7 @@ EXTERN struct command_line com; enum { ACT_STATUS = 1, + ACT_HOST_STATUS, ACT_LOG_DUMP, ACT_SHUTDOWN, ACT_ADD_LOCKSPACE, @@ -527,5 +540,15 @@ enum { EXTERN int external_shutdown; EXTERN char our_host_name_global[SANLK_NAME_LEN+1]; +EXTERN struct list_head spaces; +EXTERN struct list_head spaces_rem; +EXTERN struct list_head spaces_add; +EXTERN pthread_mutex_t spaces_mutex; + +EXTERN struct list_head resources; +EXTERN struct list_head dispose_resources; +EXTERN pthread_mutex_t resource_mutex; +EXTERN pthread_cond_t resource_cond; + #endif diff --git a/src/sanlock_sock.h b/src/sanlock_sock.h index 5c6a004..8279bd4 100644 --- a/src/sanlock_sock.h +++ b/src/sanlock_sock.h @@ -34,6 +34,7 @@ enum { SM_CMD_INIT_RESOURCE = 14, SM_CMD_EXAMINE_LOCKSPACE = 15, SM_CMD_EXAMINE_RESOURCE = 16, + SM_CMD_HOST_STATUS = 17, }; struct sm_header { @@ -50,9 +51,10 @@ struct sm_header { #define SANLK_STATE_MAXSTR 4096 #define SANLK_STATE_DAEMON 1 -#define SANLK_STATE_LOCKSPACE 2 -#define SANLK_STATE_CLIENT 3 +#define SANLK_STATE_CLIENT 2 +#define SANLK_STATE_LOCKSPACE 3 #define SANLK_STATE_RESOURCE 4 +#define SANLK_STATE_HOST 5 struct sanlk_state { uint32_t type; /* SANLK_STATE_ */ diff --git a/src/token_manager.c b/src/token_manager.c index a701cc3..6568af3 100644 --- a/src/token_manager.c +++ b/src/token_manager.c @@ -31,25 +31,10 @@ #include "task.h" #include "host_id.h" -static struct list_head resources; -static struct list_head dispose_resources; -static pthread_mutex_t resource_mutex = PTHREAD_MUTEX_INITIALIZER; -static pthread_cond_t resource_cond = PTHREAD_COND_INITIALIZER; static pthread_t resource_pt; static int resource_thread_stop; static int resource_examine; -#define R_EXAMINE 0x00000001 - -struct resource { - struct list_head list; - struct token *token; - int pid; - uint32_t flags; - uint64_t lver; - struct sanlk_resource r; -}; - int set_resource_examine(char *space_name, char *res_name) { struct resource *r; @@ -147,6 +132,7 @@ int add_resource(struct token *token, int pid) memset(r, 0, r_len); memcpy(&r->r, &token->r, sizeof(struct sanlk_resource)); memcpy(&r->r.disks, &token->r.disks, disks_len); + r->token_id = token->token_id; r->token = token; r->pid = pid; list_add_tail(&r->list, &resources); @@ -208,10 +194,9 @@ int acquire_token(struct task *task, struct token *token, if (rv < 0) return rv; - save_resource_lver(token, token->leader.lver); - memcpy(&token->leader, &leader_ret, sizeof(struct leader_record)); token->r.lver = token->leader.lver; + save_resource_lver(token, token->leader.lver); return rv; /* SANLK_OK */ } @@ -498,9 +483,6 @@ int setup_token_manager(void) { int rv; - INIT_LIST_HEAD(&resources); - INIT_LIST_HEAD(&dispose_resources); - rv = pthread_create(&resource_pt, NULL, resource_thread, NULL); if (rv) return -1;

12 years, 2 months

1
0
0 / 0

4 commits - src/direct_lib.c src/host_id.c src/main.c src/sanlock.8 src/sanlock_internal.h src/sanlock_resource.h src/token_manager.c

by David Teigland

src/direct_lib.c | 11 +++ src/host_id.c | 15 ----- src/main.c | 143 +++++++++++++++++++++++++++++++++++++------------ src/sanlock.8 | 7 -- src/sanlock_internal.h | 1 src/sanlock_resource.h | 1 src/token_manager.c | 17 +++-- 7 files changed, 130 insertions(+), 65 deletions(-) New commits: commit dd9dfa9c9d544ff12684ab94b5e30fc165206052 Author: David Teigland <teigland(a)redhat.com> Date: Wed Sep 7 16:49:13 2011 -0500 libsanlock: fix function stubs would like to get rid of this ugliness diff --git a/src/direct_lib.c b/src/direct_lib.c index e2c7d7a..7d7f9a9 100644 --- a/src/direct_lib.c +++ b/src/direct_lib.c @@ -31,9 +31,16 @@ void log_level(int space_id GNUC_UNUSED, int token_id GNUC_UNUSED, { } -int host_id_disk_info(char *name GNUC_UNUSED, struct sync_disk *disk GNUC_UNUSED); +int lockspace_disk(char *space_name GNUC_UNUSED, struct sync_disk *disk GNUC_UNUSED); -int host_id_disk_info(char *name GNUC_UNUSED, struct sync_disk *disk GNUC_UNUSED) +int lockspace_disk(char *space_name GNUC_UNUSED, struct sync_disk *disk GNUC_UNUSED) +{ + return -1; +} + +int host_info(char *space_name, uint64_t host_id, struct host_status *hs_out); + +int host_info(char *space_name GNUC_UNUSED, uint64_t host_id GNUC_UNUSED, struct host_status *hs_out GNUC_UNUSED) { return -1; } commit 4d1ed9632189bc2752ad6a684145ca38971ab959 Author: David Teigland <teigland(a)redhat.com> Date: Wed Sep 7 15:54:53 2011 -0500 sanlock: remove BLOCK_WD force mode don't include this option until we're ready to use it, in case the usage specifics end up being different diff --git a/src/host_id.c b/src/host_id.c index 1b82eeb..86df60b 100644 --- a/src/host_id.c +++ b/src/host_id.c @@ -163,17 +163,6 @@ int lockspace_disk(char *space_name, struct sync_disk *disk) return rv; } -void block_watchdog_updates(char *space_name) -{ - struct space *sp; - - pthread_mutex_lock(&spaces_mutex); - sp = _search_space(space_name, NULL, 0, &spaces, NULL, NULL); - if (sp) - sp->block_watchdog_updates = 1; - pthread_mutex_unlock(&spaces_mutex); -} - #if 0 static void clear_bit(int host_id, char *bitmap) { @@ -563,9 +552,7 @@ static void *lockspace_thread(void *arg_in) * pet the watchdog */ - if (delta_result == SANLK_OK && - !sp->thread_stop && - !sp->block_watchdog_updates) + if (delta_result == SANLK_OK && !sp->thread_stop) update_watchdog_file(sp, last_success); pthread_mutex_unlock(&sp->mutex); diff --git a/src/sanlock.8 b/src/sanlock.8 index 9a33990..e3196e9 100644 --- a/src/sanlock.8 +++ b/src/sanlock.8 @@ -477,11 +477,6 @@ take: process has exited, the resource lease will be released, and can then be acquired by anyone. -\fB2\fP (BLOCK_WD): stop updating the watchdog (/dev/watchdog keepalive -via wdmd_test_live) for the lockspace, which will lead to /dev/watchdog -firing and reseting the host. The resource lease can be acquired after -the timeout for a failed host. - .SH SEE ALSO .BR wdmd (8) diff --git a/src/sanlock_internal.h b/src/sanlock_internal.h index 70c696d..7a26749 100644 --- a/src/sanlock_internal.h +++ b/src/sanlock_internal.h @@ -124,7 +124,6 @@ struct space { int space_dead; int killing_pids; int external_remove; - int block_watchdog_updates; int thread_stop; int wd_fd; pthread_t thread; diff --git a/src/sanlock_resource.h b/src/sanlock_resource.h index 4998c16..770ebfc 100644 --- a/src/sanlock_resource.h +++ b/src/sanlock_resource.h @@ -28,7 +28,6 @@ /* request flags */ #define SANLK_REQ_KILL_PID 0x00000001 -#define SANLK_REQ_BLOCK_WD 0x00000002 int sanlock_register(void); diff --git a/src/token_manager.c b/src/token_manager.c index a15e47a..a701cc3 100644 --- a/src/token_manager.c +++ b/src/token_manager.c @@ -462,10 +462,8 @@ static void *resource_thread(void *arg GNUC_UNUSED) log_error("req_kill_pid %d %.48s:%.48s", pid, tt->r.lockspace_name, tt->r.name); kill(pid, SIGKILL); - - } else if (req.force_mode == SANLK_REQ_BLOCK_WD) { - log_error("req_block_wd %.48s", tt->r.lockspace_name); - block_watchdog_updates(tt->r.lockspace_name); + } else { + log_error("req force_mode unknown %u", req.force_mode); } } } commit 0a356d94812872ebc7d8c39f79cac7cc88997ab1 Author: David Teigland <teigland(a)redhat.com> Date: Wed Sep 7 12:12:34 2011 -0500 sanlock: restructure lockspace checks shifts some code around, should be no functional change. not checking sp after the list_move is better. diff --git a/src/main.c b/src/main.c index 51c1fe7..c249ace 100644 --- a/src/main.c +++ b/src/main.c @@ -424,10 +424,6 @@ static int client_using_space(struct client *cl, struct space *sp) log_spoke(sp, token, "client_using_space pid %d", cl->pid); token->space_dead = sp->space_dead; rv = 1; - - /* we could break here after finding one if we didn't care - * about setting token->space_dead which isn't really - * necessary; it just avoids trying to release the token */ } return rv; } @@ -586,9 +582,42 @@ static int main_loop(void) last_check = now; check_interval = STANDARD_CHECK_INTERVAL; + /* + * check the condition of each lockspace, + * if pids are being killed, have pids all exited? + * is its host_id being renewed?, if not kill pids + */ + pthread_mutex_lock(&spaces_mutex); list_for_each_entry_safe(sp, safe, &spaces, list) { - check_all = 0; + + if (sp->killing_pids && all_pids_dead(sp)) { + /* + * move sp to spaces_rem so main_loop + * will no longer see it. + */ + log_space(sp, "set thread_stop"); + pthread_mutex_lock(&sp->mutex); + sp->thread_stop = 1; + unlink_watchdog_file(sp); + pthread_mutex_unlock(&sp->mutex); + list_move(&sp->list, &spaces_rem); + continue; + } + + if (sp->killing_pids) { + /* + * continue to kill the pids with increasing + * levels of severity until they all exit + */ + kill_pids(sp); + check_interval = RECOVERY_CHECK_INTERVAL; + continue; + } + + /* + * check host_id lease renewal + */ if (sp->align_size > check_buf_len) { if (check_buf) @@ -599,37 +628,21 @@ static int main_loop(void) if (check_buf) memset(check_buf, 0, check_buf_len); - if (sp->killing_pids) { - if (all_pids_dead(sp)) { - log_space(sp, "set thread_stop"); - pthread_mutex_lock(&sp->mutex); - sp->thread_stop = 1; - unlink_watchdog_file(sp); - pthread_mutex_unlock(&sp->mutex); - list_move(&sp->list, &spaces_rem); - } else { - kill_pids(sp); - } - } else { - rv = check_our_lease(&main_task, sp, - &check_all, check_buf); - - if (rv || external_shutdown || sp->external_remove) { - log_space(sp, "set killing_pids check %d " - "shutdown %d remove %d", - rv, external_shutdown, - sp->external_remove); - sp->space_dead = 1; - sp->killing_pids = 1; - kill_pids(sp); - } - } + check_all = 0; - if (!sp->killing_pids && check_all) - check_other_leases(&main_task, sp, check_buf); + rv = check_our_lease(&main_task, sp, &check_all, check_buf); - if (sp->killing_pids) + if (rv || external_shutdown || sp->external_remove) { + log_space(sp, "set killing_pids check %d shutdown %d remove %d", + rv, external_shutdown, sp->external_remove); + sp->space_dead = 1; + sp->killing_pids = 1; + kill_pids(sp); check_interval = RECOVERY_CHECK_INTERVAL; + + } else if (check_all) { + check_other_leases(&main_task, sp, check_buf); + } } empty = list_empty(&spaces); pthread_mutex_unlock(&spaces_mutex); @@ -1516,6 +1529,51 @@ static void cmd_add_lockspace(struct cmd_args *ca) client_resume(ca->ci_in); } +/* + * TODO: rem_lockspace works like a renewal failure would, and abandons + * resource leases (tokens) without releasing them. Unlike the renewal + * failure case, rem_lockspace most likely releases the host_id. + * + * What might be nice is an option where rem_lockspace would try to + * release resource leases before releasing the lockspace host_id. + * (We don't really want to be releasing tokens after we've released + * our host_id for the token's lockspace.) + * + * - kill all pids (by looking at struct resource pid?) + * - wait for all pids to exit + * o have us or other thread release their tokens/resources + * o wait for tokens/resources to be released, although the release + * may fail or time out, we don't want to wait too long + * - set sp->external_remove + * - main_loop sets sp->thread_stop (should find no pids) + * - main_loop unlinks watchdog + * - lockspace_thread releases host_id + * + * The aim is that we kill pids and wait for resources to be released + * before main_loop gets involved and before the lockspace_thread is + * told to stop. + * + * An alternative messy is to add another condition to the current + * main_loop checks: + * + * if (sp->killing_pids && all_pids_dead(sp) && all_tokens_released(sp)) { + * sp->thread_stop = 1; + * unlink_watchdog_file(sp); + * list_move(spaces_rem); + * } + * + * all_tokens_released would just return 1 in case we're not doing + * the releases + * + * release_token_async would need to learn to put the resources onto + * dispose list in this case + * + * consider using the resources/dispose_resources list for all_pids_dead + * and kill_pids? instead of the clients[].tokens[] loops? actually, + * could we remove tokens and cl->tokens altogether and just use the + * resources list? + */ + static void cmd_rem_lockspace(struct cmd_args *ca) { struct sanlk_lockspace lockspace; diff --git a/src/sanlock.8 b/src/sanlock.8 index f004159..9a33990 100644 --- a/src/sanlock.8 +++ b/src/sanlock.8 @@ -278,7 +278,7 @@ This will allow resources to be acquired in the lockspace. Tell the sanlock daemon to release the specified host_id in the lockspace. Any processes holding resource leases in this lockspace will be killed, -and the leases released. +and the resource leases not released. .BR "sanlock client command -r" " RESOURCE " \ \fB-c\fP " " \fIpath\fP " " \fIargs\fP commit 65de7ce8f6f67bcc54803d56e499dc611668aad0 Author: David Teigland <teigland(a)redhat.com> Date: Wed Sep 7 11:16:29 2011 -0500 sanlock: use flags in struct resource diff --git a/src/main.c b/src/main.c index f30960b..51c1fe7 100644 --- a/src/main.c +++ b/src/main.c @@ -1952,6 +1952,25 @@ static int print_token_state(struct token *t, char *str) * 14. [repeat 7-13 for each client] */ +/* + * TODO: + * increase max transfer size + * send spaces/spaces_add/spaces_rem instead of just spaces + * use resources/dispose_resources to send resource state instead of cl->tokens + * + * . daemon + * . clients + * . lockspaces from spaces/spaces_add/spaces_rem + * . resources from resources/dispose_resources + * + * in print function, + * for each client pid, go through resources and print any for that pid, clear the res + * print any remaining detached resources (orphan, dispose) + * + * sanlock client host_status <lockspace_name> + * send sp->host_status[] for the named lockspace + */ + static void cmd_status(int fd, struct sm_header *h_recv) { struct sm_header h; diff --git a/src/token_manager.c b/src/token_manager.c index 4d447a6..a15e47a 100644 --- a/src/token_manager.c +++ b/src/token_manager.c @@ -39,11 +39,13 @@ static pthread_t resource_pt; static int resource_thread_stop; static int resource_examine; +#define R_EXAMINE 0x00000001 + struct resource { struct list_head list; struct token *token; int pid; - int examine; + uint32_t flags; uint64_t lver; struct sanlk_resource r; }; @@ -59,7 +61,7 @@ int set_resource_examine(char *space_name, char *res_name) continue; if (res_name && strncmp(r->r.name, res_name, NAME_ID_SIZE)) continue; - r->examine = 1; + r->flags |= R_EXAMINE; resource_examine = 1; count++; } @@ -75,7 +77,7 @@ static struct resource *find_resource_examine(void) struct resource *r; list_for_each_entry(r, &resources, list) { - if (r->examine) + if (r->flags & R_EXAMINE) return r; } return NULL; @@ -421,7 +423,7 @@ static void *resource_thread(void *arg GNUC_UNUSED) pthread_mutex_unlock(&resource_mutex); continue; } - r->examine = 0; + r->flags &= ~R_EXAMINE; /* we can't safely access r->token here, and r may be freed after we release mutex, so copy @@ -482,6 +484,7 @@ void release_token_async(struct token *token) r = find_resource(token, &resources); if (r) { /* assert r->token == token ? */ + if (token->space_dead || (token->acquire_result != SANLK_OK)) { _del_resource(r); free(token);

12 years, 2 months

1
0
0 / 0

4 commits - src/host_id.c src/host_id.h src/main.c src/paxos_lease.c src/sanlock.8 src/sanlock_internal.h

by David Teigland

src/host_id.c | 100 +++++++++++++++++++--------------- src/host_id.h | 11 ++- src/main.c | 6 +- src/paxos_lease.c | 144 +++++++++++++++++++++++++++---------------------- src/sanlock.8 | 96 +++++++++++++++++++++++++++++++- src/sanlock_internal.h | 6 +- 6 files changed, 241 insertions(+), 122 deletions(-) New commits: commit 63faee433069cbb9e3c16a85c3a66c2be0d33e75 Author: David Teigland <teigland(a)redhat.com> Date: Fri Sep 2 16:17:21 2011 -0500 sanlock: optimize paxos wait When paxos_lease_acquire waits for host_dead_seconds for a delta lease to expire, the start of our waiting time can be the last timestamp update we saved from the host during our host_id renewal. This avoids always waiting for the full host_dead_seconds and could often mean waiting no additional time. diff --git a/src/host_id.c b/src/host_id.c index ecab8fd..1b82eeb 100644 --- a/src/host_id.c +++ b/src/host_id.c @@ -236,6 +236,29 @@ int host_status_set_bit(char *space_name, uint64_t host_id) return 0; } +int host_info(char *space_name, uint64_t host_id, struct host_status *hs_out) +{ + struct space *sp; + int found = 0; + + if (!host_id || host_id > DEFAULT_MAX_HOSTS) + return -EINVAL; + + pthread_mutex_lock(&spaces_mutex); + list_for_each_entry(sp, &spaces, list) { + if (strncmp(sp->space_name, space_name, NAME_ID_SIZE)) + continue; + memcpy(hs_out, &sp->host_status[host_id-1], sizeof(struct host_status)); + found = 1; + break; + } + pthread_mutex_unlock(&spaces_mutex); + + if (!found) + return -ENOSPC; + return 0; +} + static void create_bitmap(struct task *task, struct space *sp, char *bitmap) { uint64_t now; @@ -263,17 +286,6 @@ static void create_bitmap(struct task *task, struct space *sp, char *bitmap) pthread_mutex_unlock(&sp->mutex); } -/* - * when entering the monitor loop in paxos_lease, once - * last_check - last_live > host_dead_seconds, it's expired - * - * at local time t=last_live, we read timstamp=X - * at local time t=last_check, we read timestamp=X - * so once the difference between last_live and last_check - * is > host_dead_seconds, the host has not renewed it's - * timestamp in host_dead_seconds. - */ - void check_other_leases(struct task *task, struct space *sp, char *buf) { struct leader_record *leader; diff --git a/src/host_id.h b/src/host_id.h index 5fab69d..fba2d03 100644 --- a/src/host_id.h +++ b/src/host_id.h @@ -14,6 +14,7 @@ int _lockspace_info(char *space_name, struct space *sp_out); int lockspace_info(char *space_name, struct space *sp_out); int lockspace_disk(char *space_name, struct sync_disk *disk); void block_watchdog_updates(char *space_name); +int host_info(char *space_name, uint64_t host_id, struct host_status *hs_out); int host_status_set_bit(char *space_name, uint64_t host_id); int host_status_clear_bit(char *space_name, uint64_t host_id); int test_id_bit(int host_id, char *bitmap); diff --git a/src/paxos_lease.c b/src/paxos_lease.c index 7323196..e7494e8 100644 --- a/src/paxos_lease.c +++ b/src/paxos_lease.c @@ -1011,15 +1011,16 @@ int paxos_lease_acquire(struct task *task, struct leader_record new_leader; struct paxos_dblock our_dblock; struct paxos_dblock dblock; - time_t start; + struct host_status hs; + uint64_t wait_start, now; + uint64_t last_timestamp; uint64_t next_lver; uint64_t our_mbal = 0; - uint64_t last_timestamp = 0; int copy_cur_leader = 0; int disk_open = 0; int error, rv, us; - log_token(token, "paxos_acquire begin acquire_lver %llu flags %x", + log_token(token, "paxos_acquire begin lver %llu flags %x", (unsigned long long)acquire_lver, flags); restart: @@ -1063,10 +1064,6 @@ int paxos_lease_acquire(struct task *task, * its watchdog has triggered and we can go for the paxos lease. */ - log_token(token, "paxos_acquire check owner_id %llu gen %llu", - (unsigned long long)cur_leader.owner_id, - (unsigned long long)cur_leader.owner_generation); - if (!disk_open) { memset(&host_id_disk, 0, sizeof(host_id_disk)); @@ -1085,17 +1082,28 @@ int paxos_lease_acquire(struct task *task, error = SANLK_ACQUIRE_IDDISK; goto out; } + } - log_token(token, "paxos_acquire lockspace %.48s " - "path %s offset %llu sector_size %u fd %d", - cur_leader.space_name, - host_id_disk.path, - (unsigned long long)host_id_disk.offset, - host_id_disk.sector_size, - host_id_disk.fd); + rv = host_info(cur_leader.space_name, cur_leader.owner_id, &hs); + if (!rv && hs.last_check && hs.last_live && + hs.owner_id == cur_leader.owner_id && + hs.owner_generation == cur_leader.owner_generation) { + wait_start = hs.last_live; + last_timestamp = hs.timestamp; + } else { + wait_start = monotime(); + last_timestamp = 0; } - start = monotime(); + log_token(token, "paxos_acquire owner %llu %llu %llu " + "host_status %llu %llu %llu wait_start %llu", + (unsigned long long)cur_leader.owner_id, + (unsigned long long)cur_leader.owner_generation, + (unsigned long long)cur_leader.timestamp, + (unsigned long long)hs.owner_id, + (unsigned long long)hs.owner_generation, + (unsigned long long)hs.timestamp, + (unsigned long long)wait_start); while (1) { error = delta_lease_leader_read(task, &host_id_disk, @@ -1104,9 +1112,14 @@ int paxos_lease_acquire(struct task *task, &host_id_leader, "paxos_acquire"); if (error < 0) { - log_errot(token, "paxos_acquire host_id %llu read %d", + log_errot(token, "paxos_acquire owner %llu %llu %llu " + "delta read %d fd %d path %s off %llu ss %u", (unsigned long long)cur_leader.owner_id, - error); + (unsigned long long)cur_leader.owner_generation, + (unsigned long long)cur_leader.timestamp, + error, host_id_disk.fd, host_id_disk.path, + (unsigned long long)host_id_disk.offset, + host_id_disk.sector_size); goto out; } @@ -1116,62 +1129,82 @@ int paxos_lease_acquire(struct task *task, and acquiring cannot take less than host_dead_seconds */ if (host_id_leader.timestamp == LEASE_FREE) { - log_token(token, "paxos_acquire host_id %llu free", + log_token(token, "paxos_acquire owner %llu delta free", (unsigned long long)cur_leader.owner_id); goto run; } /* another host has acquired the host_id of the host that owned this paxos lease; acquiring a host_id also cannot be - done in less than host_dead_seconds */ - - if (host_id_leader.owner_id != cur_leader.owner_id) { - log_token(token, "paxos_acquire host_id %llu owner %llu", - (unsigned long long)cur_leader.owner_id, - (unsigned long long)host_id_leader.owner_id); - goto run; - } + done in less than host_dead_seconds, or - /* the host_id that owns this lease may be alive, but it + the host_id that owns this lease may be alive, but it owned the lease in a previous generation without freeing it, and no longer owns it */ - if (host_id_leader.owner_generation > cur_leader.owner_generation) { - log_token(token, "paxos_acquire host_id %llu " - "generation now %llu old %llu", + if (host_id_leader.owner_id != cur_leader.owner_id || + host_id_leader.owner_generation > cur_leader.owner_generation) { + log_errot(token, "paxos_acquire owner %llu %llu %llu " + "delta %llu %llu %llu mismatch", (unsigned long long)cur_leader.owner_id, + (unsigned long long)cur_leader.owner_generation, + (unsigned long long)cur_leader.timestamp, + (unsigned long long)host_id_leader.owner_id, (unsigned long long)host_id_leader.owner_generation, - (unsigned long long)cur_leader.owner_generation); + (unsigned long long)host_id_leader.timestamp); goto run; } - /* if the owner hasn't renewed its host_id lease for - host_dead_seconds then its watchdog should have fired - by now */ - - if (monotime() - start > task->host_dead_seconds) { - log_token(token, "paxos_acquire host_id %llu expired %llu", - (unsigned long long)cur_leader.owner_id, - (unsigned long long)host_id_leader.timestamp); - goto run; + if (!last_timestamp) { + last_timestamp = host_id_leader.timestamp; + goto skip_live_check; } /* the owner is renewing its host_id so it's alive */ - if (last_timestamp && (host_id_leader.timestamp != last_timestamp)) { + if (host_id_leader.timestamp != last_timestamp) { if (flags & PAXOS_ACQUIRE_QUIET_FAIL) { - log_token(token, "paxos_acquire host_id %llu alive", - (unsigned long long)cur_leader.owner_id); + log_token(token, "paxos_acquire owner %llu " + "delta %llu %llu %llu alive", + (unsigned long long)cur_leader.owner_id, + (unsigned long long)host_id_leader.owner_id, + (unsigned long long)host_id_leader.owner_generation, + (unsigned long long)host_id_leader.timestamp); } else { - log_errot(token, "paxos_acquire host_id %llu alive", - (unsigned long long)cur_leader.owner_id); + log_errot(token, "paxos_acquire owner %llu " + "delta %llu %llu %llu alive", + (unsigned long long)cur_leader.owner_id, + (unsigned long long)host_id_leader.owner_id, + (unsigned long long)host_id_leader.owner_generation, + (unsigned long long)host_id_leader.timestamp); } error = SANLK_ACQUIRE_IDLIVE; goto out; } - last_timestamp = host_id_leader.timestamp; + /* if the owner hasn't renewed its host_id lease for + host_dead_seconds then its watchdog should have fired + by now */ + + now = monotime(); + + if (now - wait_start > task->host_dead_seconds) { + log_token(token, "paxos_acquire owner %llu %llu %llu " + "delta %llu %llu %llu dead %llu-%llu>%d", + (unsigned long long)cur_leader.owner_id, + (unsigned long long)cur_leader.owner_generation, + (unsigned long long)cur_leader.timestamp, + (unsigned long long)host_id_leader.owner_id, + (unsigned long long)host_id_leader.owner_generation, + (unsigned long long)host_id_leader.timestamp, + (unsigned long long)now, + (unsigned long long)wait_start, + task->host_dead_seconds); + goto run; + } + + skip_live_check: /* TODO: test with sleep(2) here */ sleep(1); commit af80a0f1ef076010ad98c9fb57e9188918269d61 Author: David Teigland <teigland(a)redhat.com> Date: Thu Sep 1 17:28:32 2011 -0500 sanlock: renaming functions get_space_info() -> lockspace_info() struct host_info -> struct host_status diff --git a/src/host_id.c b/src/host_id.c index badccd9..ecab8fd 100644 --- a/src/host_id.c +++ b/src/host_id.c @@ -123,7 +123,7 @@ static struct space *_search_space(char *name, return NULL; } -int _get_space_info(char *space_name, struct space *sp_out) +int _lockspace_info(char *space_name, struct space *sp_out) { struct space *sp; @@ -136,35 +136,24 @@ int _get_space_info(char *space_name, struct space *sp_out) return -1; } -int get_space_info(char *space_name, struct space *sp_out) +int lockspace_info(char *space_name, struct space *sp_out) { int rv; pthread_mutex_lock(&spaces_mutex); - rv = _get_space_info(space_name, sp_out); + rv = _lockspace_info(space_name, sp_out); pthread_mutex_unlock(&spaces_mutex); return rv; } -void block_watchdog_updates(char *space_name) -{ - struct space *sp; - - pthread_mutex_lock(&spaces_mutex); - sp = _search_space(space_name, NULL, 0, &spaces, NULL, NULL); - if (sp) - sp->block_watchdog_updates = 1; - pthread_mutex_unlock(&spaces_mutex); -} - -int host_id_disk_info(char *name, struct sync_disk *disk) +int lockspace_disk(char *space_name, struct sync_disk *disk) { struct space space; int rv; pthread_mutex_lock(&spaces_mutex); - rv = _get_space_info(name, &space); + rv = _lockspace_info(space_name, &space); if (!rv) { memcpy(disk, &space.host_id_disk, sizeof(struct sync_disk)); disk->fd = -1; @@ -174,6 +163,17 @@ int host_id_disk_info(char *name, struct sync_disk *disk) return rv; } +void block_watchdog_updates(char *space_name) +{ + struct space *sp; + + pthread_mutex_lock(&spaces_mutex); + sp = _search_space(space_name, NULL, 0, &spaces, NULL, NULL); + if (sp) + sp->block_watchdog_updates = 1; + pthread_mutex_unlock(&spaces_mutex); +} + #if 0 static void clear_bit(int host_id, char *bitmap) { @@ -210,7 +210,7 @@ int test_id_bit(int host_id, char *bitmap) return (*byte & mask); } -int host_info_set_bit(char *space_name, uint64_t host_id) +int host_status_set_bit(char *space_name, uint64_t host_id) { struct space *sp; int found = 0; @@ -231,7 +231,7 @@ int host_info_set_bit(char *space_name, uint64_t host_id) return -ENOSPC; pthread_mutex_lock(&sp->mutex); - sp->host_info[host_id-1].set_bit_time = monotime(); + sp->host_status[host_id-1].set_bit_time = monotime(); pthread_mutex_unlock(&sp->mutex); return 0; } @@ -249,12 +249,12 @@ static void create_bitmap(struct task *task, struct space *sp, char *bitmap) if (i+1 == sp->host_id) continue; - if (!sp->host_info[i].set_bit_time) + if (!sp->host_status[i].set_bit_time) continue; - if (now - sp->host_info[i].set_bit_time > task->request_finish_seconds) { + if (now - sp->host_status[i].set_bit_time > task->request_finish_seconds) { log_space(sp, "bitmap clear host_id %d", i+1); - sp->host_info[i].set_bit_time = 0; + sp->host_status[i].set_bit_time = 0; } else { set_id_bit(i+1, bitmap, &c); log_space(sp, "bitmap set host_id %d byte %x", i+1, c); @@ -278,7 +278,7 @@ void check_other_leases(struct task *task, struct space *sp, char *buf) { struct leader_record *leader; struct sync_disk *disk; - struct host_info *info; + struct host_status *hs; char *bitmap; uint64_t now; int i, new; @@ -292,21 +292,21 @@ void check_other_leases(struct task *task, struct space *sp, char *buf) if (i+1 == sp->host_id) continue; - info = &sp->host_info[i]; - info->last_check = now; + hs = &sp->host_status[i]; + hs->last_check = now; leader = (struct leader_record *)(buf + (i * disk->sector_size)); - if (info->owner_id == leader->owner_id && - info->owner_generation == leader->owner_generation && - info->timestamp == leader->timestamp) { + if (hs->owner_id == leader->owner_id && + hs->owner_generation == leader->owner_generation && + hs->timestamp == leader->timestamp) { continue; } - info->owner_id = leader->owner_id; - info->owner_generation = leader->owner_generation; - info->timestamp = leader->timestamp; - info->last_live = now; + hs->owner_id = leader->owner_id; + hs->owner_generation = leader->owner_generation; + hs->timestamp = leader->timestamp; + hs->last_live = now; bitmap = (char *)leader + HOSTID_BITMAP_OFFSET; @@ -316,11 +316,11 @@ void check_other_leases(struct task *task, struct space *sp, char *buf) /* this host has made a request for us, we won't take a new request from this host for another request_finish_seconds */ - if (now - info->last_req < task->request_finish_seconds) + if (now - hs->last_req < task->request_finish_seconds) continue; log_space(sp, "request from host_id %d", i+1); - info->last_req = now; + hs->last_req = now; new = 1; } diff --git a/src/host_id.h b/src/host_id.h index ea5c225..5fab69d 100644 --- a/src/host_id.h +++ b/src/host_id.h @@ -10,12 +10,12 @@ #define __HOST_ID__H__ int print_space_state(struct space *sp, char *str); -int _get_space_info(char *space_name, struct space *sp_out); -int get_space_info(char *space_name, struct space *sp_out); +int _lockspace_info(char *space_name, struct space *sp_out); +int lockspace_info(char *space_name, struct space *sp_out); +int lockspace_disk(char *space_name, struct sync_disk *disk); void block_watchdog_updates(char *space_name); -int host_id_disk_info(char *name, struct sync_disk *disk); -int host_info_set_bit(char *space_name, uint64_t host_id); -int host_info_clear_bit(char *space_name, uint64_t host_id); +int host_status_set_bit(char *space_name, uint64_t host_id); +int host_status_clear_bit(char *space_name, uint64_t host_id); int test_id_bit(int host_id, char *bitmap); int check_our_lease(struct task *task, struct space *sp, int *check_all, char *check_buf); void check_other_leases(struct task *task, struct space *sp, char *buf); diff --git a/src/main.c b/src/main.c index ef98a45..f30960b 100644 --- a/src/main.c +++ b/src/main.c @@ -734,7 +734,7 @@ static int check_new_tokens_space(struct client *cl, for (i = 0; i < new_tokens_count; i++) { token = new_tokens[i]; - rv = _get_space_info(token->r.lockspace_name, &space); + rv = _lockspace_info(token->r.lockspace_name, &space); if (!rv && !space.killing_pids && space.host_id == token->host_id) continue; @@ -928,7 +928,7 @@ static void cmd_acquire(struct task *task, struct cmd_args *ca) for (i = 0; i < new_tokens_count; i++) { token = new_tokens[i]; - rv = get_space_info(token->r.lockspace_name, &space); + rv = lockspace_info(token->r.lockspace_name, &space); if (rv < 0 || space.killing_pids) { log_errot(token, "cmd_acquire %d,%d,%d invalid lockspace " "found %d failed %d name %.48s", @@ -1429,7 +1429,7 @@ static void cmd_request(struct task *task, struct cmd_args *ca) goto reply; if (owner_id) - host_info_set_bit(token->r.lockspace_name, owner_id); + host_status_set_bit(token->r.lockspace_name, owner_id); reply: free(token); log_debug("cmd_request %d,%d done %d", ca->ci_in, fd, result); diff --git a/src/paxos_lease.c b/src/paxos_lease.c index 1d2edf0..7323196 100644 --- a/src/paxos_lease.c +++ b/src/paxos_lease.c @@ -1070,7 +1070,7 @@ int paxos_lease_acquire(struct task *task, if (!disk_open) { memset(&host_id_disk, 0, sizeof(host_id_disk)); - rv = host_id_disk_info(cur_leader.space_name, &host_id_disk); + rv = lockspace_disk(cur_leader.space_name, &host_id_disk); if (rv < 0) { log_errot(token, "paxos_acquire no lockspace info %.48s", cur_leader.space_name); diff --git a/src/sanlock_internal.h b/src/sanlock_internal.h index 3d78a0c..70c696d 100644 --- a/src/sanlock_internal.h +++ b/src/sanlock_internal.h @@ -103,7 +103,7 @@ struct lease_status { char *renewal_read_buf; }; -struct host_info { +struct host_status { uint64_t last_check; /* local monotime */ uint64_t last_live; /* local monotime */ uint64_t last_req; /* local monotime */ @@ -126,11 +126,11 @@ struct space { int external_remove; int block_watchdog_updates; int thread_stop; + int wd_fd; pthread_t thread; pthread_mutex_t mutex; /* protects lease_status, thread_stop */ struct lease_status lease_status; - int wd_fd; - struct host_info host_info[DEFAULT_MAX_HOSTS]; + struct host_status host_status[DEFAULT_MAX_HOSTS]; }; /* commit b6e7633e8c5a276be7acd8da5e9188c93bd18ce2 Author: David Teigland <teigland(a)redhat.com> Date: Thu Sep 1 17:10:00 2011 -0500 sanlock: remove old comment We don't use real time timestamps any longer, only monotonic timestamps. The comment about comparing real timestamps among nodes being a problem due to out of sync date/time settings is not relevant. monotonic timestamp comparisons among nodes is even more impossible. diff --git a/src/paxos_lease.c b/src/paxos_lease.c index b5a7282..1d2edf0 100644 --- a/src/paxos_lease.c +++ b/src/paxos_lease.c @@ -1147,16 +1147,7 @@ int paxos_lease_acquire(struct task *task, /* if the owner hasn't renewed its host_id lease for host_dead_seconds then its watchdog should have fired - by now - - if we trust that the clocks are in sync among hosts, then this - check could be: if (time() - host_id_leader.timestamp > - task->host_dead_seconds), but if the clocks are out of sync, - this check would easily give two hosts the lease. - - N.B. we need to be careful about ever comparing local time - to a time value we read off disk from another node that may - have different time. */ + by now */ if (monotime() - start > task->host_dead_seconds) { log_token(token, "paxos_acquire host_id %llu expired %llu", @@ -1164,14 +1155,6 @@ int paxos_lease_acquire(struct task *task, (unsigned long long)host_id_leader.timestamp); goto run; } -#if 0 - if (time(NULL) - host_id_leader.timestamp > task->host_dead_seconds) { - log_token(token, "paxos_acquire host_id %llu expired %llu", - (unsigned long long)cur_leader.owner_id, - (unsigned long long)host_id_leader.timestamp); - goto run; - } -#endif /* the owner is renewing its host_id so it's alive */ commit 5859f32c8503e21db9bece4024341b4111dbc1c9 Author: David Teigland <teigland(a)redhat.com> Date: Thu Sep 1 16:30:40 2011 -0500 sanlock: update man page diff --git a/src/sanlock.8 b/src/sanlock.8 index 6a9787a..f004159 100644 --- a/src/sanlock.8 +++ b/src/sanlock.8 @@ -257,6 +257,18 @@ Print the sanlock daemon internal debug log. Tell the sanlock daemon to exit. Any registered processes will be killed, their resource leases released, and lockspaces removed. +.BR "sanlock client init -s" " LOCKSPACE" +.br +.BR "sanlock client init -r" " RESOURCE" + +Tell the sanlock daemon to initialize storage for lease areas. +(See sanlock direct init.) + +.BR "sanlock client align -s" " LOCKSPACE" + +Tell the sanlock daemon to report the required lease alignment for a storage +path. Only path is used from the LOCKSPACE argument. + .BR "sanlock client add_lockspace -s" " LOCKSPACE" Tell the sanlock daemon to acquire the specified host_id in the lockspace. @@ -292,6 +304,23 @@ Print the resource leases held the given pid. The format is a versioned RESOURCE string "RESOURCE:lver" where lver is the version of the lease held. +.BR "sanlock client request -r" " RESOURCE " \ +\fB-f\fP " " \fIforce_mode\fP + +Request the owner of a resource do something specified by force_mode. A +versioned RESOURCE:lver string must be used with a greater version than is +presently held. Zero lver and force_mode clears the request. + +.BR "sanlock client examine -r" " RESOURCE" + +Examine the request record for the currently held resource lease and carry +out the action specified by the requested force_mode. + +.BR "sanlock client examine -s" " LOCKSPACE" + +Examine requests for all resource leases currently held in the named +lockspace. Only lockspace_name is used from the LOCKSPACE argument. + .B "sanlock direct" .I action [options] @@ -353,7 +382,9 @@ sanlock daemon's conclusion unless the configured timeouts match.) .BI "sanlock direct dump" " path" \ \fR[\fP\fB:\fP\fIoffset\fP\fR]\fP -Read disk sectors and print leader records for delta or paxos leases. +Read disk sectors and print leader records for delta or paxos leases. Add +-f 1 to print the request record values for paxos leases, and host_ids set +in delta lease bitmaps. .SS LOCKSPACE option string @@ -381,8 +412,13 @@ RESOURCE option string .IR path " path to storage reserved for leases" .br .IR offset " offset on path (bytes)" -.br -.IR lver " optional leader version" + +.SS +RESOURCE option string with version + +.BR \-r " " \fIlockspace_name\fP:\fIresource_name\fP:\fIpath\fP:\fIoffset\fP:\fIlver\fP +.P +.IR lver " leader version" .SS Defaults @@ -392,6 +428,60 @@ shows the default values for the options above. .B sanlock version shows the build version. +.SH USAGE + +.SS Request/Examine + +The first part of making a request for a resource is writing the request +record of the resource (the sector following the leader record). To make +a successful request: +.IP \(bu 3 +RESOURCE:lver must be greater than the lver presently held by the other +host. This implies the leader record must be read to discover the lver, +prior to making a request. +.IP \(bu 3 +RESOURCE:lver must be greater than or equal to the lver presently +written to the request record. Two hosts may write a new request at the +same time for the same lver, in which case both would succeed, but the +force_mode from the last would win. +.IP \(bu 3 +The force_mode must be greater than zero. +.IP \(bu 3 +To unconditionally clear the request record (set both lver and +force_mode to 0), make request with RESOURCE:0 and force_mode 0. +.PP + +The owner of the requested resource will not know of the request unless it +is explicitly told to examine its resources via the "examine" api/command, +or otherwise notfied. + +The second part of making a request is notifying the resource lease owner +that it should examine the request records of its resource leases. The +notification will cause the lease owner to automatically run the +equivalent of "sanlock client examine -s LOCKSPACE" for the lockspace of +the requested resource. + +The notification is made using a bitmap in each host_id delta lease. Each +bit represents each of the possible host_ids (1-2000). If host A wants to +notify host B to examine its resources, A sets the bit in its own bitmap +that corresponds to the host_id of B. When B next renews its delta lease, +it reads the delta leases for all hosts and checks each bitmap to see if +its own host_id has been set. It finds the bit for its own host_id set in +A's bitmap, and examines its resource request records. (The bit remains +set in A's bitmap for request_finish_seconds.) + +\fIforce_mode\fP determines the action the resource lease owner should +take: + +\fB1\fP (KILL_PID): kill the process holding the resource lease. When the +process has exited, the resource lease will be released, and can then be +acquired by anyone. + +\fB2\fP (BLOCK_WD): stop updating the watchdog (/dev/watchdog keepalive +via wdmd_test_live) for the lockspace, which will lead to /dev/watchdog +firing and reseting the host. The resource lease can be acquired after +the timeout for a failed host. + .SH SEE ALSO .BR wdmd (8)

12 years, 3 months

1
0
0 / 0

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

sanlock-devel September 2011