9:12 a.m.
Sanlock is using an old version of userstorage in its tests. The package it's been
installable through package manager for a while now, and it has been recently updated to
its version 0.5.1.
So I have created a patch to use the latest version that can be installed from PyPI, and
remove the old embedded package. The structure of the test storage folder remains the
same.
Also added a couple commits to add latest python versions, and a prerrequisite to the
documentation that I needed to install to be able to compile. But I can remove these
unrelated commits.
Albert Esteve (4):
README.dev: update prerequisites
tox.ini: add py310 to environments
userstorage: replace by PyPI userstorage
README.dev: update userstorage doc
README.dev | 14 ++--
tests/conftest.py | 21 +++---
tests/storage.py | 161 +++++++---------------------------------------
tox.ini | 5 +-
4 files changed, 45 insertions(+), 156 deletions(-)
--
2.37.3
9:12 a.m.
New subject: [PATCH 1/4] README.dev: update prerequisites
Add uuid to the listed prerequisites to
avoid the build error due to a missing include:
fatal error: uuid/uuid.h: No such file or directory
#include <uuid/uuid.h>
Signed-off-by: Albert Esteve <aesteve(a)redhat.com>
---
README.dev | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/README.dev b/README.dev
index 7f3df2c..aafbf34 100644
--- a/README.dev
+++ b/README.dev
@@ -3,11 +3,11 @@ Prerequisites
For Fedora/CentOS install the following packages:
- $ sudo yum install -y gcc make libaio-devel libblkid-devel
+ $ sudo yum install -y gcc make libaio-devel libblkid-devel libuuid libuuid-devel
For Ubuntu/Debian install the following packages:
- $ sudo apt install -y gcc make libaio-dev libblkid-dev
+ $ sudo apt install -y gcc make libaio-dev libblkid-dev uuid-dev
How to test sanlock
--
2.37.3
9:49 a.m.
New subject: [PATCH 1/4] README.dev: update prerequisites
On Thu, Oct 6, 2022 at 5:12 PM Albert Esteve <aesteve(a)redhat.com> wrote:
Add uuid to the listed prerequisites to
avoid the build error due to a missing include:
fatal error: uuid/uuid.h: No such file or directory
#include <uuid/uuid.h>
Signed-off-by: Albert Esteve <aesteve(a)redhat.com>
---
README.dev | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/README.dev b/README.dev
index 7f3df2c..aafbf34 100644
--- a/README.dev
+++ b/README.dev
@@ -3,11 +3,11 @@ Prerequisites
For Fedora/CentOS install the following packages:
- $ sudo yum install -y gcc make libaio-devel libblkid-devel
+ $ sudo yum install -y gcc make libaio-devel libblkid-devel libuuid
libuuid-devel
Only libuuid-devel is needed:
$ dnf repoquery --requires libuuid-devel
...
/usr/bin/pkg-config
libuuid(x86-32) = 2.38-0.2.fc36
libuuid(x86-32) = 2.38-1.fc36
libuuid(x86-64) = 2.38-0.2.fc36
libuuid(x86-64) = 2.38-1.fc36
libuuid.so.1
libuuid.so.1()(64bit)
pkgconfig
For Ubuntu/Debian install the following packages:
- $ sudo apt install -y gcc make libaio-dev libblkid-dev
+ $ sudo apt install -y gcc make libaio-dev libblkid-dev uuid-dev
As you did here
9:12 a.m.
New subject: [PATCH 2/4] tox.ini: add py310 to environments
Add py310 to the list of tox environments,
so that we can tests newer versions too.
Signed-off-by: Albert Esteve <aesteve(a)redhat.com>
---
tox.ini | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/tox.ini b/tox.ini
index ee871ef..8847b34 100644
--- a/tox.ini
+++ b/tox.ini
@@ -4,7 +4,7 @@
# and then run "tox" from this directory.
[tox]
-envlist = py{36,37,38,39},flake8
+envlist = py{36,37,38,39,310},flake8
skipsdist = True
skip_missing_interpreters = True
@@ -18,7 +18,7 @@ whitelist_externals = make
deps =
pytest
commands =
- py{36,37,38}: make BUILDARGS="--build-lib={envsitepackagesdir}"
+ py{36,37,38,39,310}: make BUILDARGS="--build-lib={envsitepackagesdir}"
pytest {posargs}
[testenv:flake8]
--
2.37.3
9:12 a.m.
New subject: [PATCH 3/4] userstorage: replace by PyPI userstorage
Remove the outdated version of userstorage
that is stored locally at tests/storage.py
Change it so that it uses the version obtained
from PyPI, and add it as a tox dependency for
tests.
Behaviour for missing storage and folder
structure remains as originally intended.
Fixture is adapted to new style so that
is uses setup and teardown before and
after every test, respectively.
Signed-off-by: Albert Esteve <aesteve(a)redhat.com>
---
tests/conftest.py | 21 +++---
tests/storage.py | 161 +++++++---------------------------------------
tox.ini | 1 +
3 files changed, 34 insertions(+), 149 deletions(-)
diff --git a/tests/conftest.py b/tests/conftest.py
index b40852b..5434772 100644
--- a/tests/conftest.py
+++ b/tests/conftest.py
@@ -9,13 +9,16 @@ Fixtures for sanlock testing.
"""
from __future__ import absolute_import
-import os
-
import pytest
+import userstorage
-from . import storage
from . import util
+# Mark tests with xfail if userstorage is missing
+userstorage.missing_handler = pytest.xfail
+
+# Requires relative path from tox basedir
+BACKENDS = userstorage.load_config("./tests/storage.py").BACKENDS
class SanlockIsRunning(Exception):
""" Raised if sanlock running when it should not """
@@ -38,8 +41,8 @@ def sanlock_daemon():
@pytest.fixture(params=[
- pytest.param(storage.BLOCK, id="block"),
- pytest.param(storage.FILE, id="file"),
+ BACKENDS["block"],
+ BACKENDS["file"],
])
def user_4k_path(request):
"""
@@ -48,11 +51,9 @@ def user_4k_path(request):
If storage is not available, skip the tests.
"""
- if not os.path.exists(request.param):
- pytest.skip(
- "user storage available - run 'python tests/strorage.py setup'
"
- "to enable 4k storage tests")
- return request.param
+ backend = request.param
+ with backend:
+ yield backend.path
@pytest.fixture
diff --git a/tests/storage.py b/tests/storage.py
index 0fe6f56..0b12e47 100644
--- a/tests/storage.py
+++ b/tests/storage.py
@@ -8,145 +8,28 @@
storage - provide storage for sanlock tests.
"""
-import argparse
-import errno
-import logging
-import os
-import subprocess
+from userstorage import File, Mount, LoopDevice
-BASE_DIR = "/var/tmp/sanlock-storage"
-MOUNTPOINT = os.path.join(BASE_DIR, "mnt")
-
-# For testing sanlock with 4k block device.
-LOOP1 = os.path.join(BASE_DIR, "loop1")
-BACKING1 = os.path.join(BASE_DIR, "backing1")
-
-# For testing sanlock with a filesystem backed by 4k block device.
-LOOP2 = os.path.join(BASE_DIR, "loop2")
-BACKING2 = os.path.join(BASE_DIR, "backing2")
-
-# Test paths.
-BLOCK = LOOP1
-FILE = os.path.join(MOUNTPOINT, "file")
-
-log = logging.getLogger("storage")
-
-
-def main():
- parser = argparse.ArgumentParser(
- description='Storage helper for sanlock tests')
- parser.add_argument("command", choices=["setup",
"teardown"])
- args = parser.parse_args()
-
- logging.basicConfig(level=logging.INFO, format="storage: %(message)s")
-
- if args.command == "setup":
- setup()
- elif args.command == "teardown":
- teardown()
-
-
-def setup():
- create_dir(BASE_DIR)
-
- if not os.path.exists(LOOP1):
- create_loop_device(LOOP1, BACKING1)
-
- if not os.path.exists(LOOP2):
- create_loop_device(LOOP2, BACKING2)
- create_dir(MOUNTPOINT)
- create_filesystem(LOOP2, MOUNTPOINT)
-
- # Sanlock allocates spaces as needed.
- with open(FILE, "wb") as f:
- f.truncate(0)
-
-
-def teardown():
- if is_mounted(MOUNTPOINT):
- remove_filesystem(MOUNTPOINT)
-
- if os.path.exists(LOOP2):
- remove_loop_device(LOOP2, BACKING2)
-
- if os.path.exists(LOOP1):
- remove_loop_device(LOOP1, BACKING1)
-
-
-def create_loop_device(link_path, backing_file, size=1024**3,
- sector_size=4096):
- log.info("Creating loop device %s", link_path)
-
- with open(backing_file, "wb") as f:
- f.truncate(size)
-
- out = subprocess.check_output([
- "sudo",
- "losetup",
- "-f", backing_file,
- "--sector-size", str(sector_size),
- "--show",
- ])
-
- device = out.decode("utf-8").strip()
-
- # Remove stale symlink.
- if os.path.islink(link_path):
- os.unlink(link_path)
-
- os.symlink(device, link_path)
- chown(link_path)
-
-
-def remove_loop_device(link_path, backing_file):
- log.info("Removing loop device %s", link_path)
-
- subprocess.check_call(["sudo", "losetup", "-d",
link_path])
- remove_file(link_path)
- remove_file(backing_file)
-
-
-def create_filesystem(device, mountpoint):
- log.info("Creating filesystem %s", mountpoint)
-
- subprocess.check_call(["sudo", "mkfs.xfs", "-q",
device])
- subprocess.check_call(["sudo", "mount", device, mountpoint])
- chown(mountpoint)
-
-
-def remove_filesystem(mountpoint):
- log.info("Removing filesystem %s", mountpoint)
- subprocess.check_call(["sudo", "umount", mountpoint])
-
-
-def is_mounted(mountpoint):
- with open("/proc/self/mounts") as f:
- for line in f:
- if mountpoint in line:
- return True
- return False
-
-
-def chown(path):
- user_group = "%(USER)s:%(USER)s" % os.environ
- subprocess.check_call(["sudo", "chown", user_group, path])
-
-
-def create_dir(path):
- try:
- os.makedirs(path)
- except EnvironmentError as e:
- if e.errno != errno.EEXIST:
- raise
-
-
-def remove_file(path):
- try:
- os.remove(path)
- except EnvironmentError as e:
- if e.errno != errno.ENOENT:
- raise
+GiB = 1024**3
+BASE_DIR = "/var/tmp/sanlock-storage"
-if __name__ == "__main__":
- main()
+BACKENDS = {
+
+ "file":
+ File(
+ Mount(
+ LoopDevice(
+ base_dir=BASE_DIR,
+ name="file",
+ size=GiB,
+ sector_size=4096))),
+
+ "block":
+ LoopDevice(
+ base_dir=BASE_DIR,
+ name="loop",
+ size=GiB,
+ sector_size=4096),
+
+}
diff --git a/tox.ini b/tox.ini
index 8847b34..c0cf5a1 100644
--- a/tox.ini
+++ b/tox.ini
@@ -17,6 +17,7 @@ setenv =
whitelist_externals = make
deps =
pytest
+ userstorage>=0.5.1
commands =
py{36,37,38,39,310}: make BUILDARGS="--build-lib={envsitepackagesdir}"
pytest {posargs}
--
2.37.3
9:58 a.m.
New subject: [PATCH 3/4] userstorage: replace by PyPI userstorage
On Thu, Oct 6, 2022 at 5:12 PM Albert Esteve <aesteve(a)redhat.com> wrote:
Remove the outdated version of userstorage
that is stored locally at tests/storage.py
Change it so that it uses the version obtained
from PyPI, and add it as a tox dependency for
tests.
Behaviour for missing storage and folder
structure remains as originally intended.
Fixture is adapted to new style so that
is uses setup and teardown before and
after every test, respectively.
Signed-off-by: Albert Esteve <aesteve(a)redhat.com>
---
tests/conftest.py | 21 +++---
tests/storage.py | 161 +++++++---------------------------------------
tox.ini | 1 +
3 files changed, 34 insertions(+), 149 deletions(-)
diff --git a/tests/conftest.py b/tests/conftest.py
index b40852b..5434772 100644
--- a/tests/conftest.py
+++ b/tests/conftest.py
@@ -9,13 +9,16 @@ Fixtures for sanlock testing.
"""
from __future__ import absolute_import
-import os
-
import pytest
+import userstorage
-from . import storage
from . import util
+# Mark tests with xfail if userstorage is missing
+userstorage.missing_handler = pytest.xfail
Previously we skipped tests if user storage was missing. It would be nice
to keep the previous behaviour, although xfail makes sense as well.
Changing skip to xfail in a separate commit is the right way to change the
policy.
+
+# Requires relative path from tox basedir
+BACKENDS = userstorage.load_config("./tests/storage.py").BACKENDS
Missing whitespace here - reported by flake8:
tests/conftest.py:23:1: E302 expected 2 blank lines, found 1
class SanlockIsRunning(Exception):
^
1 E302 expected 2 blank lines, found 1
ERROR: InvocationError for command
/home/nsoffer/src/sanlock/.tox/flake8/bin/flake8 --statistics tests python
(exited with code 1)
Did you run "tox" before posting the patches? It runs all the tests.
class SanlockIsRunning(Exception):
""" Raised if sanlock running when it should not """
@@ -38,8 +41,8 @@ def sanlock_daemon():
@pytest.fixture(params=[
- pytest.param(storage.BLOCK, id="block"),
- pytest.param(storage.FILE, id="file"),
+ BACKENDS["block"],
+ BACKENDS["file"],
])
def user_4k_path(request):
"""
@@ -48,11 +51,9 @@ def user_4k_path(request):
If storage is not available, skip the tests.
"""
- if not os.path.exists(request.param):
- pytest.skip(
- "user storage available - run 'python tests/strorage.py
setup' "
- "to enable 4k storage tests")
- return request.param
+ backend = request.param
+ with backend:
+ yield backend.path
@pytest.fixture
diff --git a/tests/storage.py b/tests/storage.py
index 0fe6f56..0b12e47 100644
--- a/tests/storage.py
+++ b/tests/storage.py
@@ -8,145 +8,28 @@
storage - provide storage for sanlock tests.
"""
-import argparse
-import errno
-import logging
-import os
-import subprocess
+from userstorage import File, Mount, LoopDevice
-BASE_DIR = "/var/tmp/sanlock-storage"
-MOUNTPOINT = os.path.join(BASE_DIR, "mnt")
-
-# For testing sanlock with 4k block device.
-LOOP1 = os.path.join(BASE_DIR, "loop1")
-BACKING1 = os.path.join(BASE_DIR, "backing1")
-
-# For testing sanlock with a filesystem backed by 4k block device.
-LOOP2 = os.path.join(BASE_DIR, "loop2")
-BACKING2 = os.path.join(BASE_DIR, "backing2")
-
-# Test paths.
-BLOCK = LOOP1
-FILE = os.path.join(MOUNTPOINT, "file")
-
-log = logging.getLogger("storage")
-
-
-def main():
- parser = argparse.ArgumentParser(
- description='Storage helper for sanlock tests')
- parser.add_argument("command", choices=["setup",
"teardown"])
- args = parser.parse_args()
-
- logging.basicConfig(level=logging.INFO, format="storage: %(message)s")
-
- if args.command == "setup":
- setup()
- elif args.command == "teardown":
- teardown()
-
-
-def setup():
- create_dir(BASE_DIR)
-
- if not os.path.exists(LOOP1):
- create_loop_device(LOOP1, BACKING1)
-
- if not os.path.exists(LOOP2):
- create_loop_device(LOOP2, BACKING2)
- create_dir(MOUNTPOINT)
- create_filesystem(LOOP2, MOUNTPOINT)
-
- # Sanlock allocates spaces as needed.
- with open(FILE, "wb") as f:
- f.truncate(0)
-
-
-def teardown():
- if is_mounted(MOUNTPOINT):
- remove_filesystem(MOUNTPOINT)
-
- if os.path.exists(LOOP2):
- remove_loop_device(LOOP2, BACKING2)
-
- if os.path.exists(LOOP1):
- remove_loop_device(LOOP1, BACKING1)
-
-
-def create_loop_device(link_path, backing_file, size=1024**3,
- sector_size=4096):
- log.info("Creating loop device %s", link_path)
-
- with open(backing_file, "wb") as f:
- f.truncate(size)
-
- out = subprocess.check_output([
- "sudo",
- "losetup",
- "-f", backing_file,
- "--sector-size", str(sector_size),
- "--show",
- ])
-
- device = out.decode("utf-8").strip()
-
- # Remove stale symlink.
- if os.path.islink(link_path):
- os.unlink(link_path)
-
- os.symlink(device, link_path)
- chown(link_path)
-
-
-def remove_loop_device(link_path, backing_file):
- log.info("Removing loop device %s", link_path)
-
- subprocess.check_call(["sudo", "losetup", "-d",
link_path])
- remove_file(link_path)
- remove_file(backing_file)
-
-
-def create_filesystem(device, mountpoint):
- log.info("Creating filesystem %s", mountpoint)
-
- subprocess.check_call(["sudo", "mkfs.xfs", "-q",
device])
- subprocess.check_call(["sudo", "mount", device, mountpoint])
- chown(mountpoint)
-
-
-def remove_filesystem(mountpoint):
- log.info("Removing filesystem %s", mountpoint)
- subprocess.check_call(["sudo", "umount", mountpoint])
-
-
-def is_mounted(mountpoint):
- with open("/proc/self/mounts") as f:
- for line in f:
- if mountpoint in line:
- return True
- return False
-
-
-def chown(path):
- user_group = "%(USER)s:%(USER)s" % os.environ
- subprocess.check_call(["sudo", "chown", user_group, path])
-
-
-def create_dir(path):
- try:
- os.makedirs(path)
- except EnvironmentError as e:
- if e.errno != errno.EEXIST:
- raise
-
-
-def remove_file(path):
- try:
- os.remove(path)
- except EnvironmentError as e:
- if e.errno != errno.ENOENT:
- raise
+GiB = 1024**3
+BASE_DIR = "/var/tmp/sanlock-storage"
-if __name__ == "__main__":
- main()
+BACKENDS = {
+
+ "file":
+ File(
+ Mount(
+ LoopDevice(
+ base_dir=BASE_DIR,
+ name="file",
+ size=GiB,
+ sector_size=4096))),
+
+ "block":
+ LoopDevice(
+ base_dir=BASE_DIR,
+ name="loop",
+ size=GiB,
+ sector_size=4096),
+
+}
diff --git a/tox.ini b/tox.ini
index 8847b34..c0cf5a1 100644
--- a/tox.ini
+++ b/tox.ini
@@ -17,6 +17,7 @@ setenv =
whitelist_externals = make
deps =
pytest
+ userstorage>=0.5.1
commands =
py{36,37,38,39,310}: make BUILDARGS="--build-lib={envsitepackagesdir}"
pytest {posargs}
--
2.37.3
10:05 a.m.
New subject: [PATCH 3/4] userstorage: replace by PyPI userstorage
On Thu, Oct 6, 2022 at 5:12 PM Albert Esteve <aesteve(a)redhat.com> wrote:
--- a/tox.ini
+++ b/tox.ini
@@ -17,6 +17,7 @@ setenv =
whitelist_externals = make
deps =
pytest
+ userstorage>=0.5.1
This is better required using a requirements.txt file with tox. Since we
must create
the storage before running tox, the userstorage package should be installed
with
tox in a virtual environment.
commands =
py{36,37,38,39,310}: make BUILDARGS="--build-lib={envsitepackagesdir}"
pytest {posargs}
--
2.37.3
9:12 a.m.
New subject: [PATCH 4/4] README.dev: update userstorage doc
Update documentation regarding userstorage
setup and teardown.
Signed-off-by: Albert Esteve <aesteve(a)redhat.com>
---
README.dev | 10 +++++++---
1 file changed, 7 insertions(+), 3 deletions(-)
diff --git a/README.dev b/README.dev
index aafbf34..6d9ed14 100644
--- a/README.dev
+++ b/README.dev
@@ -54,9 +54,13 @@ Or using environment variable:
Testing 4K support
==================
-To enable the 4k tests, you need to setup 4k stroage for the tests:
+To enable the 4k tests, you need to have userstorage installed:
- $ python tests/storage.py setup
+ $ python -m pip install userstorage
+
+Then, you just need to setup 4k storage for the tests:
+
+ $ userstorage create tests/storage.py
This creates two loop devices with 4k sector size; one for testing sanlock with
4k block device, and the other for testing with a filesystem backed by a 4k
@@ -64,7 +68,7 @@ block device.
To teardown the storage:
- $ python tests/storage.py teardown
+ $ userstorage delete tests/storage.py
The script unmounts the filesystem and detaches the loop devices.
--
2.37.3
10:01 a.m.
New subject: [PATCH 4/4] README.dev: update userstorage doc
On Thu, Oct 6, 2022 at 5:12 PM Albert Esteve <aesteve(a)redhat.com> wrote:
Update documentation regarding userstorage
setup and teardown.
Signed-off-by: Albert Esteve <aesteve(a)redhat.com>
---
README.dev | 10 +++++++---
1 file changed, 7 insertions(+), 3 deletions(-)
diff --git a/README.dev b/README.dev
index aafbf34..6d9ed14 100644
--- a/README.dev
+++ b/README.dev
@@ -54,9 +54,13 @@ Or using environment variable:
Testing 4K support
==================
-To enable the 4k tests, you need to setup 4k stroage for the tests:
+To enable the 4k tests, you need to have userstorage installed:
- $ python tests/storage.py setup
+ $ python -m pip install userstorage
Installing user storage like this is not a good idea - this will try to
install it
as root.
We show here how to create a virtual environment and install the packages
required for sanlock testing (tox, userstorage), and how to activate and
deactivate
the environment.
+
+Then, you just need to setup 4k storage for the tests:
+
+ $ userstorage create tests/storage.py
This creates two loop devices with 4k sector size; one for testing
sanlock with
4k block device, and the other for testing with a filesystem backed by a
4k
@@ -64,7 +68,7 @@ block device.
To teardown the storage:
- $ python tests/storage.py teardown
+ $ userstorage delete tests/storage.py
The script unmounts the filesystem and detaches the loop devices.
--
2.37.3
579
days inactive
579
days old
sanlock-devel@lists.fedorahosted.org
8 comments
2 participants
participants (2)
-
Albert Esteve -
Nir Soffer