[PATCH 0/6] Add github CI
by Nir Soffer
Replace the broken travis CI configuration with github actions.
The new CI revealed some issues with chown(), and some flaky or broken tests
with some python versions.
The chown() issues are fixed in this change, the tests issues need
investigation.
Nir Soffer (6):
Remove travis CI configuration
Log uid and gid when running in unprivileged mode
Change rundir ownership only if running as root
Don't change socket ownership
Add python 3.11 to tox
Add github actions based CI
.github/workflows/ci.yml | 48 ++++++++++++++++++++++++++++++++++++++++
.travis.yml | 26 ----------------------
src/lockfile.c | 12 +++++-----
src/main.c | 10 ++-------
tox.ini | 4 ++--
5 files changed, 59 insertions(+), 41 deletions(-)
create mode 100644 .github/workflows/ci.yml
delete mode 100644 .travis.yml
--
2.38.1
1 year, 3 months
[PATCH 0/2] Improve README.dev
by Nir Soffer
- Improve instructions for creating python virtual environment
- Remove stale info about python 2.7
Nir Soffer (2):
Add section for creating and using venv
Update README.dev for python 3
README.dev | 68 +++++++++++++++++++++++++-----------------------------
1 file changed, 32 insertions(+), 36 deletions(-)
--
2.38.1
1 year, 3 months
[PATCH 0/3] Improve disk validation
by Nir Soffer
Fix failing disk validation test since python 3.9, and improve error messages
when disk validation fails.
The tests succeeds now on python3.6-python3.11:
https://github.com/nirs/sanlock/actions/runs/3559586867
Nir Soffer (3):
Improve error message if PyObject_Repr failed
Improve error messages when validating disks
Fix disk validation on python >= 3.9
python/sanlock.c | 11 ++++++++---
1 file changed, 8 insertions(+), 3 deletions(-)
--
2.38.1
1 year, 3 months
[PATCH v3 0/4] New patch for updating userstorage package
by Albert Esteve
Sanlock is using an old version of userstorage in its tests. The package it's been
installable through package manager for a while now, and it has been recently updated to
its version 0.5.1.
So I have created a patch to use the latest version that can be installed from PyPI, and
remove the old embedded package. The structure of the test storage folder remains the
same.
Also added a couple commits to add latest python versions, and a prerrequisite to the
documentation that I needed to install to be able to compile. But I can remove these
unrelated commits.
Albert Esteve (4):
README.dev: update prerequisites
tox.ini: add py310 to environments
userstorage: replace by PyPI userstorage
README.dev: update userstorage doc
README.dev | 22 +++++--
requirements.txt | 1 +
tests/conftest.py | 22 ++++---
tests/storage.py | 161 +++++++---------------------------------------
tox.ini | 5 +-
5 files changed, 55 insertions(+), 156 deletions(-)
create mode 100644 requirements.txt
--
2.37.3
1 year, 3 months
[sanlock] 04/04: README.dev: update userstorage doc
by pagure@pagure.io
This is an automated email from the git hooks/post-receive script.
nsoffer pushed a commit to branch master
in repository sanlock.
commit 1b306f0abb173393cb478f09d58a9b0777bc1387
Author: Albert Esteve <aesteve(a)redhat.com>
AuthorDate: Wed Nov 9 14:50:57 2022 +0100
README.dev: update userstorage doc
Update documentation regarding userstorage
setup and teardown.
Signed-off-by: Albert Esteve <aesteve(a)redhat.com>
Signed-off-by: Nir Soffer <nsoffer(a)redhat.com>
---
README.dev | 30 +++++++++++++++++++++++++++---
1 file changed, 27 insertions(+), 3 deletions(-)
diff --git a/README.dev b/README.dev
index 1bf1d17..46b0088 100644
--- a/README.dev
+++ b/README.dev
@@ -54,9 +54,29 @@ Or using environment variable:
Testing 4K support
==================
-To enable the 4k tests, you need to setup 4k stroage for the tests:
+To enable the 4k tests, you need to have userstorage installed.
+First, you need to create a virtual environment
+(https://docs.python.org/3/library/venv.html), and then install
+the userstorage library from the requirements.txt.
+This only needs to be done once:
- $ python tests/storage.py setup
+ $ python3 -m venv ~/.venv/sanlock
+ $ source ~/.venv/sanlock/bin/activate
+ $ pip install --upgrade pip
+ $ pip install -r requirements.txt
+ $ deactivate
+
+Before running the tests, you need to enter the virtual environment:
+
+ $ source ~/.venv/sanlock/bin/activate
+
+The shell prompt will change to show the virtual environment name:
+
+ (sanlock) [user@hostname sanlock]$
+
+Then, you just need to setup 4k storage for the tests:
+
+ $ userstorage create tests/storage.py
This creates two loop devices with 4k sector size; one for testing sanlock with
4k block device, and the other for testing with a filesystem backed by a 4k
@@ -64,10 +84,14 @@ block device.
To teardown the storage:
- $ python tests/storage.py teardown
+ $ userstorage delete tests/storage.py
The script unmounts the filesystem and detaches the loop devices.
The storage helper script uses sudo to perform privileged operations. The best
way to use it is to setup the environment once at the start of the session, and
teardown when you finish.
+
+To deactivate the virtual environment:
+
+ $ deactivate
--
To stop receiving notification emails like this one, please contact
the administrator of this repository.
1 year, 3 months
[sanlock] 03/04: userstorage: replace by PyPI userstorage
by pagure@pagure.io
This is an automated email from the git hooks/post-receive script.
nsoffer pushed a commit to branch master
in repository sanlock.
commit 38ff68588078af1664787e20256ffba3ae166489
Author: Albert Esteve <aesteve(a)redhat.com>
AuthorDate: Wed Nov 9 14:50:56 2022 +0100
userstorage: replace by PyPI userstorage
Remove the outdated version of userstorage
that is stored locally at tests/storage.py
The userstorage library is required
to create the storage before running the
tests with tox, and also in the test
virtual environment. In order to satisfy
this, add a requirements.txt to the root
folder to obtain the latests userstorage
version from PyPI, and add it as a tox
dependency for tests.
Behaviour for missing storage and folder
structure remains as originally intended.
Fixture is adapted to new style so that
is uses setup and teardown before and
after every test, respectively.
Signed-off-by: Albert Esteve <aesteve(a)redhat.com>
---
requirements.txt | 2 +
tests/conftest.py | 22 ++++----
tests/storage.py | 161 ++++++++----------------------------------------------
tox.ini | 1 +
4 files changed, 37 insertions(+), 149 deletions(-)
diff --git a/requirements.txt b/requirements.txt
new file mode 100644
index 0000000..2dfdbcd
--- /dev/null
+++ b/requirements.txt
@@ -0,0 +1,2 @@
+tox
+userstorage>=0.5.1
diff --git a/tests/conftest.py b/tests/conftest.py
index b40852b..79c514b 100644
--- a/tests/conftest.py
+++ b/tests/conftest.py
@@ -9,13 +9,17 @@ Fixtures for sanlock testing.
"""
from __future__ import absolute_import
-import os
-
import pytest
+import userstorage
-from . import storage
from . import util
+# Mark tests with skip if userstorage is missing
+userstorage.missing_handler = pytest.skip
+
+# Requires relative path from tox basedir
+BACKENDS = userstorage.load_config("./tests/storage.py").BACKENDS
+
class SanlockIsRunning(Exception):
""" Raised if sanlock running when it should not """
@@ -38,8 +42,8 @@ def sanlock_daemon():
@pytest.fixture(params=[
- pytest.param(storage.BLOCK, id="block"),
- pytest.param(storage.FILE, id="file"),
+ BACKENDS["block"],
+ BACKENDS["file"],
])
def user_4k_path(request):
"""
@@ -48,11 +52,9 @@ def user_4k_path(request):
If storage is not available, skip the tests.
"""
- if not os.path.exists(request.param):
- pytest.skip(
- "user storage available - run 'python tests/strorage.py setup' "
- "to enable 4k storage tests")
- return request.param
+ backend = request.param
+ with backend:
+ yield backend.path
@pytest.fixture
diff --git a/tests/storage.py b/tests/storage.py
index 0fe6f56..0b12e47 100644
--- a/tests/storage.py
+++ b/tests/storage.py
@@ -8,145 +8,28 @@
storage - provide storage for sanlock tests.
"""
-import argparse
-import errno
-import logging
-import os
-import subprocess
+from userstorage import File, Mount, LoopDevice
-BASE_DIR = "/var/tmp/sanlock-storage"
-MOUNTPOINT = os.path.join(BASE_DIR, "mnt")
-
-# For testing sanlock with 4k block device.
-LOOP1 = os.path.join(BASE_DIR, "loop1")
-BACKING1 = os.path.join(BASE_DIR, "backing1")
-
-# For testing sanlock with a filesystem backed by 4k block device.
-LOOP2 = os.path.join(BASE_DIR, "loop2")
-BACKING2 = os.path.join(BASE_DIR, "backing2")
-
-# Test paths.
-BLOCK = LOOP1
-FILE = os.path.join(MOUNTPOINT, "file")
-
-log = logging.getLogger("storage")
-
-
-def main():
- parser = argparse.ArgumentParser(
- description='Storage helper for sanlock tests')
- parser.add_argument("command", choices=["setup", "teardown"])
- args = parser.parse_args()
-
- logging.basicConfig(level=logging.INFO, format="storage: %(message)s")
-
- if args.command == "setup":
- setup()
- elif args.command == "teardown":
- teardown()
-
-
-def setup():
- create_dir(BASE_DIR)
-
- if not os.path.exists(LOOP1):
- create_loop_device(LOOP1, BACKING1)
-
- if not os.path.exists(LOOP2):
- create_loop_device(LOOP2, BACKING2)
- create_dir(MOUNTPOINT)
- create_filesystem(LOOP2, MOUNTPOINT)
-
- # Sanlock allocates spaces as needed.
- with open(FILE, "wb") as f:
- f.truncate(0)
-
-
-def teardown():
- if is_mounted(MOUNTPOINT):
- remove_filesystem(MOUNTPOINT)
-
- if os.path.exists(LOOP2):
- remove_loop_device(LOOP2, BACKING2)
-
- if os.path.exists(LOOP1):
- remove_loop_device(LOOP1, BACKING1)
-
-
-def create_loop_device(link_path, backing_file, size=1024**3,
- sector_size=4096):
- log.info("Creating loop device %s", link_path)
-
- with open(backing_file, "wb") as f:
- f.truncate(size)
-
- out = subprocess.check_output([
- "sudo",
- "losetup",
- "-f", backing_file,
- "--sector-size", str(sector_size),
- "--show",
- ])
-
- device = out.decode("utf-8").strip()
-
- # Remove stale symlink.
- if os.path.islink(link_path):
- os.unlink(link_path)
-
- os.symlink(device, link_path)
- chown(link_path)
-
-
-def remove_loop_device(link_path, backing_file):
- log.info("Removing loop device %s", link_path)
-
- subprocess.check_call(["sudo", "losetup", "-d", link_path])
- remove_file(link_path)
- remove_file(backing_file)
-
-
-def create_filesystem(device, mountpoint):
- log.info("Creating filesystem %s", mountpoint)
-
- subprocess.check_call(["sudo", "mkfs.xfs", "-q", device])
- subprocess.check_call(["sudo", "mount", device, mountpoint])
- chown(mountpoint)
-
-
-def remove_filesystem(mountpoint):
- log.info("Removing filesystem %s", mountpoint)
- subprocess.check_call(["sudo", "umount", mountpoint])
-
-
-def is_mounted(mountpoint):
- with open("/proc/self/mounts") as f:
- for line in f:
- if mountpoint in line:
- return True
- return False
-
-
-def chown(path):
- user_group = "%(USER)s:%(USER)s" % os.environ
- subprocess.check_call(["sudo", "chown", user_group, path])
-
-
-def create_dir(path):
- try:
- os.makedirs(path)
- except EnvironmentError as e:
- if e.errno != errno.EEXIST:
- raise
-
-
-def remove_file(path):
- try:
- os.remove(path)
- except EnvironmentError as e:
- if e.errno != errno.ENOENT:
- raise
+GiB = 1024**3
+BASE_DIR = "/var/tmp/sanlock-storage"
-if __name__ == "__main__":
- main()
+BACKENDS = {
+
+ "file":
+ File(
+ Mount(
+ LoopDevice(
+ base_dir=BASE_DIR,
+ name="file",
+ size=GiB,
+ sector_size=4096))),
+
+ "block":
+ LoopDevice(
+ base_dir=BASE_DIR,
+ name="loop",
+ size=GiB,
+ sector_size=4096),
+
+}
diff --git a/tox.ini b/tox.ini
index 8847b34..c0cf5a1 100644
--- a/tox.ini
+++ b/tox.ini
@@ -17,6 +17,7 @@ setenv =
whitelist_externals = make
deps =
pytest
+ userstorage>=0.5.1
commands =
py{36,37,38,39,310}: make BUILDARGS="--build-lib={envsitepackagesdir}"
pytest {posargs}
--
To stop receiving notification emails like this one, please contact
the administrator of this repository.
1 year, 3 months
[sanlock] 02/04: tox.ini: add py310 to environments
by pagure@pagure.io
This is an automated email from the git hooks/post-receive script.
nsoffer pushed a commit to branch master
in repository sanlock.
commit 7de76581b25d36375638478fb487ac45ef9925df
Author: Albert Esteve <aesteve(a)redhat.com>
AuthorDate: Wed Nov 9 14:50:55 2022 +0100
tox.ini: add py310 to environments
Add py310 to the list of tox environments,
so that we can tests newer versions too.
Signed-off-by: Albert Esteve <aesteve(a)redhat.com>
---
tox.ini | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/tox.ini b/tox.ini
index ee871ef..8847b34 100644
--- a/tox.ini
+++ b/tox.ini
@@ -4,7 +4,7 @@
# and then run "tox" from this directory.
[tox]
-envlist = py{36,37,38,39},flake8
+envlist = py{36,37,38,39,310},flake8
skipsdist = True
skip_missing_interpreters = True
@@ -18,7 +18,7 @@ whitelist_externals = make
deps =
pytest
commands =
- py{36,37,38}: make BUILDARGS="--build-lib={envsitepackagesdir}"
+ py{36,37,38,39,310}: make BUILDARGS="--build-lib={envsitepackagesdir}"
pytest {posargs}
[testenv:flake8]
--
To stop receiving notification emails like this one, please contact
the administrator of this repository.
1 year, 3 months
[sanlock] 01/04: README.dev: update prerequisites
by pagure@pagure.io
This is an automated email from the git hooks/post-receive script.
nsoffer pushed a commit to branch master
in repository sanlock.
commit 0cb0978b7125609443bd69bba2059e68b4b18e14
Author: Albert Esteve <aesteve(a)redhat.com>
AuthorDate: Wed Nov 9 14:50:54 2022 +0100
README.dev: update prerequisites
Add uuid to the listed prerequisites to
avoid the build error due to a missing include:
fatal error: uuid/uuid.h: No such file or directory
#include <uuid/uuid.h>
Signed-off-by: Albert Esteve <aesteve(a)redhat.com>
---
README.dev | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/README.dev b/README.dev
index 7f3df2c..1bf1d17 100644
--- a/README.dev
+++ b/README.dev
@@ -3,11 +3,11 @@ Prerequisites
For Fedora/CentOS install the following packages:
- $ sudo yum install -y gcc make libaio-devel libblkid-devel
+ $ sudo yum install -y gcc make libaio-devel libblkid-devel libuuid-devel
For Ubuntu/Debian install the following packages:
- $ sudo apt install -y gcc make libaio-dev libblkid-dev
+ $ sudo apt install -y gcc make libaio-dev libblkid-dev uuid-dev
How to test sanlock
--
To stop receiving notification emails like this one, please contact
the administrator of this repository.
1 year, 3 months
[sanlock] 02/02: wdmd: close watchdog when not used
by pagure@pagure.io
This is an automated email from the git hooks/post-receive script.
teigland pushed a commit to branch master
in repository sanlock.
commit 45563a6e0c9d388f58178e03c9dd0b2708b2969c
Author: David Teigland <teigland(a)redhat.com>
AuthorDate: Mon Nov 14 16:04:39 2022 -0600
wdmd: close watchdog when not used
Previously, the watchdog would remain open with keepalives
even after all clients (sanlock lockspaces) were removed.
This was an unnecessary liability in case it couldn't be
kept alive while not needed. It also prevented sanlock
from being restarted using a new watchdog timeout.
---
wdmd/main.c | 59 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++---
1 file changed, 56 insertions(+), 3 deletions(-)
diff --git a/wdmd/main.c b/wdmd/main.c
index e89ab5e..3c60b4e 100644
--- a/wdmd/main.c
+++ b/wdmd/main.c
@@ -101,6 +101,7 @@ struct client {
int pid;
int pid_dead;
int refcount;
+ int internal;
uint64_t renewal;
uint64_t expire;
void *workfn;
@@ -281,12 +282,12 @@ static void dump_debug(int fd)
now = monotime();
memset(line, 0, sizeof(line));
- snprintf(line, 255, "wdmd %d socket_gid %d high_priority %d now %llu last_keepalive %llu last_closeunclean %llu allow_scripts %d kill_script_sec %d\n",
+ snprintf(line, 255, "wdmd %d socket_gid %d high_priority %d now %llu last_keepalive %llu last_closeunclean %llu allow_scripts %d kill_script_sec %d fire_timeout %d\n",
getpid(), socket_gid, high_priority,
(unsigned long long)now,
(unsigned long long)last_keepalive,
(unsigned long long)last_closeunclean,
- allow_scripts, kill_script_sec);
+ allow_scripts, kill_script_sec, fire_timeout);
line_len = strlen(line);
strncat(debug_buf, line, LINE_SIZE);
@@ -648,6 +649,7 @@ static int setup_clients(void)
ci = client_add(fd, process_listener, client_pid_dead);
strncpy(client[ci].name, "listen", WDMD_NAME_SIZE);
+ client[ci].internal = 1;
return 0;
}
@@ -731,6 +733,25 @@ static int active_clients(void)
return 0;
}
+static void count_clients(int *active, int *external)
+{
+ int act = 0, ext = 0;
+ int i;
+
+ for (i = 0; i < client_size; i++) {
+ if (!client[i].used)
+ continue;
+ if (client[i].refcount)
+ act++;
+ if (!client[i].internal)
+ ext++;
+ }
+ if (active)
+ *active = act;
+ if (external)
+ *external = ext;
+}
+
#ifdef TEST_FILES
#define FILES_DIR "/run/wdmd/test_files"
@@ -1447,6 +1468,7 @@ static int setup_signals(void)
ci = client_add(fd, process_signals, client_pid_dead);
strncpy(client[ci].name, "signal", WDMD_NAME_SIZE);
+ client[ci].internal = 1;
return 0;
}
@@ -1490,6 +1512,8 @@ static int test_loop(void)
void (*workfn) (int ci);
void (*deadfn) (int ci);
uint64_t test_time;
+ int resetting = 0;
+ int active_usage, external_usage;
int poll_timeout;
int sleep_seconds;
int fail_count;
@@ -1520,7 +1544,9 @@ static int test_loop(void)
}
}
- if (daemon_quit && !active_clients())
+ count_clients(&active_usage, &external_usage);
+
+ if (daemon_quit && !active_usage)
break;
/*
@@ -1529,6 +1555,31 @@ static int test_loop(void)
if (!test_loop_enable)
continue;
+ /*
+ * active_usage are client connections with a refcount.
+ * external_usage are any clients other than internal.
+ * (open_watchdog happens with external but not active
+ * connections.)
+ *
+ * checking resetting here is critical to avoiding
+ * unnecessary resets: while in recovery mode we
+ * have done close_watchdog_unclean, then all clients
+ * are cleared, and we need the loop below to see
+ * no further failures and reopen and pet the watchdog
+ * again to avoid a reset. After it's been reopened,
+ * and no longer used due to all clients being cleared,
+ * then it's ok to get here and close cleanly.
+ */
+ if (!active_usage && !external_usage && !resetting) {
+ log_debug("close watchdog unused");
+ close_watchdog();
+ test_loop_enable = 0;
+ test_interval = standard_test_interval;
+ poll_timeout = test_interval * 1000;
+ test_time = 0;
+ continue;
+ }
+
if (monotime() - test_time >= test_interval) {
test_time = monotime();
log_debug("test_time %llu",
@@ -1549,6 +1600,7 @@ static int test_loop(void)
}
test_interval = standard_test_interval;
+ resetting = 0;
} else {
/* If we can patch the kernel so that close
does not generate a ping, then we can skip
@@ -1557,6 +1609,7 @@ static int test_loop(void)
close_watchdog_unclean();
test_interval = RECOVER_TEST_INTERVAL;
+ resetting = 1;
}
}
--
To stop receiving notification emails like this one, please contact
the administrator of this repository.
1 year, 4 months