Here's a rough proof of concept showing how we could run tests for passt with Avocado and the exeter library I recently created. It includes Cleber's patch adding some basic Avocado tests and builds on that. The current draft is pretty janky: * The build rules to download and install the necessary pieces are messy * We create the Avocado job files from the exeter sources in the Makefile. Ideally Avocado would eventually be extended to handle this itself * The names that Avocado sees for each test are overlong * There's some hacks to make sure things are executed from the right working directory But, it's a starting point. Stefano, If you could look particularly at 6/22 and 22/22 which add the real tests for passt/pasta, that would be great. The more specific you can be about what you find ugly about how the tests are written, then better I can try to address that. I suspect it will be easier to actually apply the series, then look at the new test files (test/build/build.py, and test/pasta/pasta.py particularly). From there you can look at as much of the support library as you need to, rather than digging through the actual patches to look for that. Cleber, If you could look at 4..6/22 particularly to review how I'm connecting the actual tests to the Avocado runner, that would be helpful. Cleber Rosa (1): test: run static checkers with Avocado and JSON definitions David Gibson (21): nstool: Fix some trivial typos nstool: Propagate SIGTERM to processes executed in the namespace test: Extend make targets to run Avocado tests test: Exeter based static tests test: Add exeter+Avocado based build tests test: Add linters for Python code tasst: Introduce library of common test helpers tasst: "snh" module for simulated network hosts tasst: Add helper to get network interface names for a site tasst: Add helpers to run commands with nstool tasst: Add ifup and network address helpers to SimNetHost tasst: Helper for creating veth devices between namespaces tasst: Add helper for getting MTU of a network interface tasst: Add helper to wait for IP address to appear tasst: Add helpers for getting a SimNetHost's routes tasst: Helpers to test transferring data between sites tasst: IP address allocation helpers tasst: Helpers for testing NDP behaviour tasst: Helpers for testing DHCP & DHCPv6 behaviour tasst: Helpers to construct a simple network environment for tests avocado: Convert basic pasta tests test/.gitignore | 2 + test/Makefile | 63 ++++++- test/avocado/static_checkers.json | 16 ++ test/build/.gitignore | 2 + test/build/build.py | 105 +++++++++++ test/build/static_checkers.sh | 28 +++ test/meta/.gitignore | 1 + test/meta/lint.sh | 28 +++ test/nstool.c | 30 ++- test/pasta/.gitignore | 1 + test/pasta/pasta.py | 138 ++++++++++++++ test/run_avocado | 51 ++++++ test/tasst/.gitignore | 1 + test/tasst/__init__.py | 11 ++ test/tasst/__main__.py | 22 +++ test/tasst/address.py | 79 ++++++++ test/tasst/dhcp.py | 132 ++++++++++++++ test/tasst/dhcpv6.py | 89 +++++++++ test/tasst/ndp.py | 116 ++++++++++++ test/tasst/nstool.py | 186 +++++++++++++++++++ test/tasst/pasta.py | 52 ++++++ test/tasst/scenario/__init__.py | 12 ++ test/tasst/scenario/simple.py | 109 +++++++++++ test/tasst/selftest/__init__.py | 16 ++ test/tasst/selftest/static_ifup.py | 60 ++++++ test/tasst/selftest/veth.py | 106 +++++++++++ test/tasst/snh.py | 283 +++++++++++++++++++++++++++++ test/tasst/transfer.py | 194 ++++++++++++++++++++ 28 files changed, 1928 insertions(+), 5 deletions(-) create mode 100644 test/avocado/static_checkers.json create mode 100644 test/build/.gitignore create mode 100644 test/build/build.py create mode 100644 test/build/static_checkers.sh create mode 100644 test/meta/.gitignore create mode 100644 test/meta/lint.sh create mode 100644 test/pasta/.gitignore create mode 100644 test/pasta/pasta.py create mode 100755 test/run_avocado create mode 100644 test/tasst/.gitignore create mode 100644 test/tasst/__init__.py create mode 100644 test/tasst/__main__.py create mode 100644 test/tasst/address.py create mode 100644 test/tasst/dhcp.py create mode 100644 test/tasst/dhcpv6.py create mode 100644 test/tasst/ndp.py create mode 100644 test/tasst/nstool.py create mode 100644 test/tasst/pasta.py create mode 100644 test/tasst/scenario/__init__.py create mode 100644 test/tasst/scenario/simple.py create mode 100644 test/tasst/selftest/__init__.py create mode 100644 test/tasst/selftest/static_ifup.py create mode 100644 test/tasst/selftest/veth.py create mode 100644 test/tasst/snh.py create mode 100644 test/tasst/transfer.py -- 2.45.2
Signed-off-by: David Gibson <david(a)gibson.dropbear.id.au>
---
test/nstool.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/test/nstool.c b/test/nstool.c
index 1bdf44e8..a6aca981 100644
--- a/test/nstool.c
+++ b/test/nstool.c
@@ -359,7 +359,7 @@ static void wait_for_child(pid_t pid)
if (rc != pid)
die("waitpid() on %d returned %d", pid, rc);
if (WIFSTOPPED(status)) {
- /* Stop the parent to patch */
+ /* Stop the parent to match */
kill(getpid(), SIGSTOP);
/* We must have resumed, resume the child */
kill(pid, SIGCONT);
@@ -508,7 +508,7 @@ static void cmd_exec(int argc, char *argv[])
/* CHILD */
if (argc > optind + 1) {
exe = argv[optind + 1];
- xargs = (const char * const*)(argv + optind + 1);
+ xargs = (const char *const *)(argv + optind + 1);
} else {
exe = getenv("SHELL");
if (!exe)
--
2.45.2
Particularly in shell it's sometimes natural to save the pid from a process
run and later kill it. If doing this with nstool exec, however, it will
kill nstool itself, not the program it is running, which isn't usually what
you want or expect.
Address this by having nstool propagate SIGTERM to its child process. It
may make sense to propagate some other signals, but some introduce extra
complications, so we'll worry about them when and if it seems useful.
Signed-off-by: David Gibson <david(a)gibson.dropbear.id.au>
---
test/nstool.c | 26 ++++++++++++++++++++++++--
1 file changed, 24 insertions(+), 2 deletions(-)
diff --git a/test/nstool.c b/test/nstool.c
index a6aca981..fc357d8a 100644
--- a/test/nstool.c
+++ b/test/nstool.c
@@ -345,17 +345,39 @@ static int openns(const char *fmt, ...)
return fd;
}
+static pid_t sig_pid;
+static void sig_handler(int signum)
+{
+ int err;
+
+ err = kill(sig_pid, signum);
+ if (err)
+ die("Propagating %s: %s\n", strsignal(signum), strerror(errno));
+}
+
static void wait_for_child(pid_t pid)
{
- int status;
+ struct sigaction sa = {
+ .sa_handler = sig_handler,
+ .sa_flags = SA_RESETHAND,
+ };
+ int status, err;
+
+ sig_pid = pid;
+ err = sigaction(SIGTERM, &sa, NULL);
+ if (err)
+ die("sigaction(SIGTERM): %s\n", strerror(errno));
/* Match the child's exit status, if possible */
for (;;) {
pid_t rc;
rc = waitpid(pid, &status, WUNTRACED);
- if (rc < 0)
+ if (rc < 0) {
+ if (errno == EINTR)
+ continue;
die("waitpid() on %d: %s\n", pid, strerror(errno));
+ }
if (rc != pid)
die("waitpid() on %d returned %d", pid, rc);
if (WIFSTOPPED(status)) {
--
2.45.2
On Mon, 5 Aug 2024 22:36:41 +1000 David Gibson <david(a)gibson.dropbear.id.au> wrote:Particularly in shell it's sometimes natural to save the pid from a process run and later kill it. If doing this with nstool exec, however, it will kill nstool itself, not the program it is running, which isn't usually what you want or expect. Address this by having nstool propagate SIGTERM to its child process. It may make sense to propagate some other signals, but some introduce extra complications, so we'll worry about them when and if it seems useful. Signed-off-by: David Gibson <david(a)gibson.dropbear.id.au> --- test/nstool.c | 26 ++++++++++++++++++++++++-- 1 file changed, 24 insertions(+), 2 deletions(-) diff --git a/test/nstool.c b/test/nstool.c index a6aca981..fc357d8a 100644 --- a/test/nstool.c +++ b/test/nstool.c @@ -345,17 +345,39 @@ static int openns(const char *fmt, ...) return fd; } +static pid_t sig_pid; +static void sig_handler(int signum) +{ + int err; + + err = kill(sig_pid, signum); + if (err) + die("Propagating %s: %s\n", strsignal(signum), strerror(errno));As I've just been bitten by this, f30ed68c5273 ("pasta: Save errno on signal handler entry, restore on return when needed"), I was kind of wondering if we should save and restore errno, regardless of the fact it's not needed here (if kill() affects ernno, we won't return). On the other hand this handler at the moment is simple enough that we would notice if it's needed because of some further changes. -- Stefano
From: Cleber Rosa <crosa(a)redhat.com>
This adds a script and configuration to use the Avocado Testing
Framework to run, at this time, the static checkers.
The actual tests are defined using (JSON based) files, that are known
to Avocado as "recipes". The JSON files are parsed and "resolved"
into tests by Avocado's "runnables-recipe" resolver. The syntax
allows for any kind of test supported by Avocado to be defined there,
including a mix of different test types.
By the nature of Avocado's default configuration, those will run in
parallel in the host system. For more complex tests or different use
cases, Avocado could help in future versions by running those in
different environments such as containers.
The entry point ("test/run_avocado") is intended to be an optional
tool at this point, coexisting with the current implementation to run
tests. It uses Avocado's Job API to create a job with, at this point,
the static checkers suite.
The installation of Avocado itself is left to users, given that the
details on how to install it (virtual environments and specific
tooling) can be a very different and long discussion.
Signed-off-by: Cleber Rosa <crosa(a)redhat.com>
Message-ID: <20240629121342.3284907-1-crosa(a)redhat.com>
---
test/avocado/static_checkers.json | 16 ++++++++++
test/run_avocado | 49 +++++++++++++++++++++++++++++++
2 files changed, 65 insertions(+)
create mode 100644 test/avocado/static_checkers.json
create mode 100755 test/run_avocado
diff --git a/test/avocado/static_checkers.json b/test/avocado/static_checkers.json
new file mode 100644
index 00000000..5fae43ed
--- /dev/null
+++ b/test/avocado/static_checkers.json
@@ -0,0 +1,16 @@
+[
+ {
+ "kind": "exec-test",
+ "uri": "make",
+ "args": [
+ "clang-tidy"
+ ]
+ },
+ {
+ "kind": "exec-test",
+ "uri": "make",
+ "args": [
+ "cppcheck"
+ ]
+ }
+]
diff --git a/test/run_avocado b/test/run_avocado
new file mode 100755
index 00000000..37db17c3
--- /dev/null
+++ b/test/run_avocado
@@ -0,0 +1,49 @@
+#!/usr/bin/env python3
+
+import os
+import sys
+
+
+def check_avocado_version():
+ minimum_version = 106.0
+
+ def error_out():
+ print(
+ f"Avocado version {minimum_version} or later is required.\n"
+ f"You may install it with: \n"
+ f" python3 -m pip install avocado-framework",
+ file=sys.stderr,
+ )
+ sys.exit(1)
+
+ try:
+ from avocado import VERSION
+
+ if (float(VERSION)) < minimum_version:
+ error_out()
+ except ImportError:
+ error_out()
+
+
+check_avocado_version()
+from avocado.core.job import Job
+from avocado.core.suite import TestSuite
+
+
+def main():
+ repo_root_path = os.path.abspath(
+ os.path.dirname(os.path.dirname(os.path.dirname(__file__)))
+ )
+ config = {
+ "resolver.references": [
+ os.path.join(repo_root_path, "test", "avocado",
"static_checkers.json")
+ ],
+ "runner.identifier_format": "{args[0]}",
+ }
+ suite = TestSuite.from_config(config, name="static_checkers")
+ with Job(config, [suite]) as j:
+ return j.run()
+
+
+if __name__ == "__main__":
+ sys.exit(main())
--
2.45.2
Add a new 'avocado' target to the test/ Makefile, which will install
avocado into a Python venv, and run the Avocado based tests with it.
Signed-off-by: David Gibson <david(a)gibson.dropbear.id.au>
---
test/.gitignore | 1 +
test/Makefile | 16 ++++++++++++++++
test/run_avocado | 9 +++++----
3 files changed, 22 insertions(+), 4 deletions(-)
diff --git a/test/.gitignore b/test/.gitignore
index 6dd4790b..a79d5b6f 100644
--- a/test/.gitignore
+++ b/test/.gitignore
@@ -10,3 +10,4 @@ QEMU_EFI.fd
nstool
guest-key
guest-key.pub
+/venv/
diff --git a/test/Makefile b/test/Makefile
index 35a3b559..fda62984 100644
--- a/test/Makefile
+++ b/test/Makefile
@@ -63,6 +63,12 @@ LOCAL_ASSETS = mbuto.img mbuto.mem.img podman/bin/podman QEMU_EFI.fd \
ASSETS = $(DOWNLOAD_ASSETS) $(LOCAL_ASSETS)
+SYSTEM_PYTHON = python3
+VENV = venv
+PYTHON = $(VENV)/bin/python3
+PIP = $(VENV)/bin/pip3
+RUN_AVOCADO = cd .. && test/$(PYTHON) test/run_avocado
+
CFLAGS = -Wall -Werror -Wextra -pedantic -std=c99
assets: $(ASSETS)
@@ -116,6 +122,15 @@ medium.bin:
big.bin:
dd if=/dev/urandom bs=1M count=10 of=$@
+.PHONY: venv
+venv:
+ $(SYSTEM_PYTHON) -m venv $(VENV)
+ $(PIP) install avocado-framework
+
+.PHONY: avocado
+avocado: venv
+ $(RUN_AVOCADO) avocado
+
check: assets
./run
@@ -127,6 +142,7 @@ clean:
rm -f $(LOCAL_ASSETS)
rm -rf test_logs
rm -f prepared-*.qcow2 prepared-*.img
+ rm -rf $(VENV)
realclean: clean
rm -rf $(DOWNLOAD_ASSETS)
diff --git a/test/run_avocado b/test/run_avocado
index 37db17c3..19a94a8f 100755
--- a/test/run_avocado
+++ b/test/run_avocado
@@ -32,12 +32,13 @@ from avocado.core.suite import TestSuite
def main():
repo_root_path = os.path.abspath(
- os.path.dirname(os.path.dirname(os.path.dirname(__file__)))
+ os.path.dirname(os.path.dirname(__file__))
)
+
+ references = [os.path.join(repo_root_path, 'test', x) for x in
sys.argv[1:]]
+
config = {
- "resolver.references": [
- os.path.join(repo_root_path, "test", "avocado",
"static_checkers.json")
- ],
+ "resolver.references": references,
"runner.identifier_format": "{args[0]}",
}
suite = TestSuite.from_config(config, name="static_checkers")
--
2.45.2
Introduce some trivial testcases based on the exeter library. These run the C static checkers, which is redundant with the included Avocado json file, but are useful as an example. We extend the make avocado target to generate Avocado job files from the exeter tests and include them in the test run. Signed-off-by: David Gibson <david(a)gibson.dropbear.id.au> --- test/.gitignore | 1 + test/Makefile | 18 +++++++++++++++--- test/build/.gitignore | 1 + test/build/static_checkers.sh | 30 ++++++++++++++++++++++++++++++ test/run_avocado | 2 +- 5 files changed, 48 insertions(+), 4 deletions(-) create mode 100644 test/build/.gitignore create mode 100644 test/build/static_checkers.sh diff --git a/test/.gitignore b/test/.gitignore index a79d5b6f..bded349b 100644 --- a/test/.gitignore +++ b/test/.gitignore @@ -11,3 +11,4 @@ nstool guest-key guest-key.pub /venv/ +/exeter/ diff --git a/test/Makefile b/test/Makefile index fda62984..dae25312 100644 --- a/test/Makefile +++ b/test/Makefile @@ -52,7 +52,7 @@ UBUNTU_NEW_IMGS = xenial-server-cloudimg-powerpc-disk1.img \ jammy-server-cloudimg-s390x.img UBUNTU_IMGS = $(UBUNTU_OLD_IMGS) $(UBUNTU_NEW_IMGS) -DOWNLOAD_ASSETS = mbuto podman \ +DOWNLOAD_ASSETS = exeter mbuto podman \ $(DEBIAN_IMGS) $(FEDORA_IMGS) $(OPENSUSE_IMGS) $(UBUNTU_IMGS) TESTDATA_ASSETS = small.bin big.bin medium.bin LOCAL_ASSETS = mbuto.img mbuto.mem.img podman/bin/podman QEMU_EFI.fd \ @@ -63,6 +63,11 @@ LOCAL_ASSETS = mbuto.img mbuto.mem.img podman/bin/podman QEMU_EFI.fd \ ASSETS = $(DOWNLOAD_ASSETS) $(LOCAL_ASSETS) +EXETER_SH = build/static_checkers.sh +EXETER_JOBS = $(EXETER_SH:%.sh=%.json) + +AVOCADO_JOBS = $(EXETER_JOBS) avocado/static_checkers.json + SYSTEM_PYTHON = python3 VENV = venv PYTHON = $(VENV)/bin/python3 @@ -77,6 +82,9 @@ assets: $(ASSETS) pull-%: % git -C $* pull +exeter: + git clone https://gitlab.com/dgibson/exeter.git + mbuto: git clone git://mbuto.sh/mbuto @@ -127,9 +135,12 @@ venv: $(SYSTEM_PYTHON) -m venv $(VENV) $(PIP) install avocado-framework +%.json: %.sh pull-exeter + cd ..; sh test/$< --avocado > test/$@ + .PHONY: avocado -avocado: venv - $(RUN_AVOCADO) avocado +avocado: venv $(AVOCADO_JOBS) + $(RUN_AVOCADO) $(AVOCADO_JOBS) check: assets ./run @@ -143,6 +154,7 @@ clean: rm -rf test_logs rm -f prepared-*.qcow2 prepared-*.img rm -rf $(VENV) + rm -f $(EXETER_JOBS) realclean: clean rm -rf $(DOWNLOAD_ASSETS) diff --git a/test/build/.gitignore b/test/build/.gitignore new file mode 100644 index 00000000..a6c57f5f --- /dev/null +++ b/test/build/.gitignore @@ -0,0 +1 @@ +*.json diff --git a/test/build/static_checkers.sh b/test/build/static_checkers.sh new file mode 100644 index 00000000..ec159ea2 --- /dev/null +++ b/test/build/static_checkers.sh @@ -0,0 +1,30 @@ +#! /bin/sh +# +# SPDX-License-Identifier: GPL-2.0-or-later +# +# PASST - Plug A Simple Socket Transport +# for qemu/UNIX domain socket mode +# +# PASTA - Pack A Subtle Tap Abstraction +# for network namespace/tap device mode +# +# test/build/static_checkers.sh - Run static checkers +# +# Copyright Red Hat +# Author: David Gibson <david(a)gibson.dropbear.id.au> + +source $(dirname $0)/../exeter/sh/exeter.sh + +cppcheck () { + make cppcheck +} +exeter_register cppcheck + +clang_tidy () { + make clang-tidy +} +exeter_register clang_tidy + +exeter_main "$@" + + diff --git a/test/run_avocado b/test/run_avocado index 19a94a8f..d518b9ec 100755 --- a/test/run_avocado +++ b/test/run_avocado @@ -39,7 +39,7 @@ def main(): config = { "resolver.references": references, - "runner.identifier_format": "{args[0]}", + "runner.identifier_format": "{args}", } suite = TestSuite.from_config(config, name="static_checkers") with Job(config, [suite]) as j: -- 2.45.2
Add a new test script to run the equivalent of the tests in build/all
using exeter and Avocado. This new version of the tests is more robust
than the original, since it makes a temporary copy of the source tree so
will not be affected by concurrent manual builds.
Signed-off-by: David Gibson <david(a)gibson.dropbear.id.au>
---
test/Makefile | 19 +++++---
test/build/.gitignore | 1 +
test/build/build.py | 105 ++++++++++++++++++++++++++++++++++++++++++
test/run_avocado | 2 +-
4 files changed, 120 insertions(+), 7 deletions(-)
create mode 100644 test/build/build.py
diff --git a/test/Makefile b/test/Makefile
index dae25312..d24fce14 100644
--- a/test/Makefile
+++ b/test/Makefile
@@ -64,15 +64,19 @@ LOCAL_ASSETS = mbuto.img mbuto.mem.img podman/bin/podman QEMU_EFI.fd
\
ASSETS = $(DOWNLOAD_ASSETS) $(LOCAL_ASSETS)
EXETER_SH = build/static_checkers.sh
-EXETER_JOBS = $(EXETER_SH:%.sh=%.json)
+EXETER_PY = build/build.py
+EXETER_JOBS = $(EXETER_SH:%.sh=%.json) $(EXETER_PY:%.py=%.json)
AVOCADO_JOBS = $(EXETER_JOBS) avocado/static_checkers.json
-SYSTEM_PYTHON = python3
+PYTHON = python3
VENV = venv
-PYTHON = $(VENV)/bin/python3
PIP = $(VENV)/bin/pip3
-RUN_AVOCADO = cd .. && test/$(PYTHON) test/run_avocado
+PYPATH = exeter/py3
+SPACE = $(subst ,, )
+PYPATH_TEST = $(subst $(SPACE),:,$(PYPATH))
+PYPATH_BASE = $(subst $(SPACE),:,$(PYPATH:%=test/%))
+RUN_AVOCADO = cd .. && PYTHONPATH=$(PYPATH_BASE) test/$(VENV)/bin/python3
test/run_avocado
CFLAGS = -Wall -Werror -Wextra -pedantic -std=c99
@@ -131,13 +135,16 @@ big.bin:
dd if=/dev/urandom bs=1M count=10 of=$@
.PHONY: venv
-venv:
- $(SYSTEM_PYTHON) -m venv $(VENV)
+venv: pull-exeter
+ $(PYTHON) -m venv $(VENV)
$(PIP) install avocado-framework
%.json: %.sh pull-exeter
cd ..; sh test/$< --avocado > test/$@
+%.json: %.py pull-exeter
+ cd ..; PYTHONPATH=$(PYPATH_BASE) $(PYTHON) test/$< --avocado > test/$@
+
.PHONY: avocado
avocado: venv $(AVOCADO_JOBS)
$(RUN_AVOCADO) $(AVOCADO_JOBS)
diff --git a/test/build/.gitignore b/test/build/.gitignore
index a6c57f5f..4ef40dd0 100644
--- a/test/build/.gitignore
+++ b/test/build/.gitignore
@@ -1 +1,2 @@
*.json
+build.exeter
diff --git a/test/build/build.py b/test/build/build.py
new file mode 100644
index 00000000..79668672
--- /dev/null
+++ b/test/build/build.py
@@ -0,0 +1,105 @@
+#! /usr/bin/env python3
+#
+# SPDX-License-Identifier: GPL-2.0-or-later
+#
+# PASST - Plug A Simple Socket Transport
+# for qemu/UNIX domain socket mode
+#
+# PASTA - Pack A Subtle Tap Abstraction
+# for network namespace/tap device mode
+#
+# test/build/build.sh - Test build and install targets
+#
+# Copyright Red Hat
+# Author: David Gibson <david(a)gibson.dropbear.id.au>
+
+import contextlib
+import os.path
+import shutil
+import subprocess
+import tempfile
+
+import exeter
+
+
+def host_run(*cmd, **kwargs):
+ return subprocess.run(cmd, check=True, encoding='UTF-8', **kwargs)
+
+
+def host_out(*cmd, **kwargs):
+ return host_run(*cmd, capture_output=True, **kwargs).stdout
+
+
+(a)contextlib.contextmanager
+def clone_source_tree():
+ with tempfile.TemporaryDirectory(ignore_cleanup_errors=False) as tmpdir:
+ # Make a temporary copy of the sources
+ srcfiles = host_out('git', 'ls-files').splitlines()
+ for src in srcfiles:
+ dst = os.path.join(tmpdir, src)
+ os.makedirs(os.path.dirname(dst), exist_ok=True)
+ shutil.copy(src, dst)
+ os.chdir(tmpdir)
+ yield tmpdir
+
+
+def build_target(target, outputs):
+ with clone_source_tree():
+ for o in outputs:
+ assert not os.path.exists(o)
+ host_run('make', f'{target}',
'CFLAGS="-Werror"')
+ for o in outputs:
+ assert os.path.exists(o)
+ host_run('make', 'clean')
+ for o in outputs:
+ assert not os.path.exists(o)
+
+
+(a)exeter.test
+def test_make_passt():
+ build_target('passt', ['passt'])
+
+
+(a)exeter.test
+def test_make_pasta():
+ build_target('pasta', ['pasta'])
+
+
+(a)exeter.test
+def test_make_qrap():
+ build_target('qrap', ['qrap'])
+
+
+(a)exeter.test
+def test_make_all():
+ build_target('all', ['passt', 'pasta',
'qrap'])
+
+
+(a)exeter.test
+def test_make_install_uninstall():
+ with clone_source_tree():
+ with tempfile.TemporaryDirectory(ignore_cleanup_errors=False) \
+ as prefix:
+ bindir = os.path.join(prefix, 'bin')
+ mandir = os.path.join(prefix, 'share', 'man')
+ exes = ['passt', 'pasta', 'qrap']
+
+ # Install
+ host_run('make', 'install',
'CFLAGS="-Werror"', f'prefix={prefix}')
+
+ for t in exes:
+ assert os.path.isfile(os.path.join(bindir, t))
+ host_run('man', '-M', f'{mandir}',
'-W', 'passt')
+
+ # Uninstall
+ host_run('make', 'uninstall',
f'prefix={prefix}')
+
+ for t in exes:
+ assert not os.path.exists(os.path.join(bindir, t))
+ cmd = ['man', '-M', f'{mandir}',
'-W', 'passt']
+ exeter.assert_raises(subprocess.CalledProcessError,
+ host_run, *cmd)
+
+
+if __name__ == '__main__':
+ exeter.main()
diff --git a/test/run_avocado b/test/run_avocado
index d518b9ec..26a226ce 100755
--- a/test/run_avocado
+++ b/test/run_avocado
@@ -41,7 +41,7 @@ def main():
"resolver.references": references,
"runner.identifier_format": "{args}",
}
- suite = TestSuite.from_config(config, name="static_checkers")
+ suite = TestSuite.from_config(config, name="all")
with Job(config, [suite]) as j:
return j.run()
--
2.45.2
On Mon, 5 Aug 2024 22:36:45 +1000 David Gibson <david(a)gibson.dropbear.id.au> wrote:Add a new test script to run the equivalent of the tests in build/all using exeter and Avocado. This new version of the tests is more robust than the original, since it makes a temporary copy of the source tree so will not be affected by concurrent manual builds.I think this is much more readable than the previous Python attempt. On the other hand, I guess it's not an ideal candidate for a fair comparison because this is exactly the kind of stuff where shell scripting shines: it's a simple test that needs a few basic shell commands. On that subject, the shell test is about half the lines of code (just skipping headers, it's 48 lines instead of 90... and yes, this version now uses a copy of the source code, but that would be two lines). In terms of time overhead, dropping delays to make the display capture nice (a feature that we would anyway lose with exeter plus Avocado, if I understood correctly): $ time (make clean; make passt; make clean; make pasta; make clean; make qrap; make clean; make; d=$(mktemp -d); prefix=$d make install; prefix=$d make uninstall; ) [...] real 0m17.449s user 0m15.616s sys 0m2.136s compared to: $ time ./run [...] real 0m18.217s user 0m0.010s sys 0m0.001s ...which I would call essentially no overhead. I didn't try out this version yet, I suspect it would be somewhere in between.Signed-off-by: David Gibson <david(a)gibson.dropbear.id.au> --- test/Makefile | 19 +++++--- test/build/.gitignore | 1 + test/build/build.py | 105 ++++++++++++++++++++++++++++++++++++++++++ test/run_avocado | 2 +- 4 files changed, 120 insertions(+), 7 deletions(-) create mode 100644 test/build/build.py diff --git a/test/Makefile b/test/Makefile index dae25312..d24fce14 100644 --- a/test/Makefile +++ b/test/Makefile @@ -64,15 +64,19 @@ LOCAL_ASSETS = mbuto.img mbuto.mem.img podman/bin/podman QEMU_EFI.fd \ ASSETS = $(DOWNLOAD_ASSETS) $(LOCAL_ASSETS) EXETER_SH = build/static_checkers.sh -EXETER_JOBS = $(EXETER_SH:%.sh=%.json) +EXETER_PY = build/build.py +EXETER_JOBS = $(EXETER_SH:%.sh=%.json) $(EXETER_PY:%.py=%.json) AVOCADO_JOBS = $(EXETER_JOBS) avocado/static_checkers.json -SYSTEM_PYTHON = python3 +PYTHON = python3 VENV = venv -PYTHON = $(VENV)/bin/python3 PIP = $(VENV)/bin/pip3 -RUN_AVOCADO = cd .. && test/$(PYTHON) test/run_avocado +PYPATH = exeter/py3 +SPACE = $(subst ,, ) +PYPATH_TEST = $(subst $(SPACE),:,$(PYPATH)) +PYPATH_BASE = $(subst $(SPACE),:,$(PYPATH:%=test/%)) +RUN_AVOCADO = cd .. && PYTHONPATH=$(PYPATH_BASE) test/$(VENV)/bin/python3 test/run_avocadoAt least intuitively, I would have no clue what this all does. But it doesn't matter so much, I could try to find out the day that something doesn't work.CFLAGS = -Wall -Werror -Wextra -pedantic -std=c99 @@ -131,13 +135,16 @@ big.bin: dd if=/dev/urandom bs=1M count=10 of=$@ .PHONY: venv -venv: - $(SYSTEM_PYTHON) -m venv $(VENV) +venv: pull-exeter + $(PYTHON) -m venv $(VENV) $(PIP) install avocado-framework %.json: %.sh pull-exeter cd ..; sh test/$< --avocado > test/$@ +%.json: %.py pull-exeter + cd ..; PYTHONPATH=$(PYPATH_BASE) $(PYTHON) test/$< --avocado > test/$@ +Same here..PHONY: avocado avocado: venv $(AVOCADO_JOBS) $(RUN_AVOCADO) $(AVOCADO_JOBS) diff --git a/test/build/.gitignore b/test/build/.gitignore index a6c57f5f..4ef40dd0 100644 --- a/test/build/.gitignore +++ b/test/build/.gitignore @@ -1 +1,2 @@ *.json +build.exeter diff --git a/test/build/build.py b/test/build/build.py new file mode 100644 index 00000000..79668672 --- /dev/null +++ b/test/build/build.py @@ -0,0 +1,105 @@ +#! /usr/bin/env python3 +# +# SPDX-License-Identifier: GPL-2.0-or-later +# +# PASST - Plug A Simple Socket Transport +# for qemu/UNIX domain socket mode +# +# PASTA - Pack A Subtle Tap Abstraction +# for network namespace/tap device mode +# +# test/build/build.sh - Test build and install targets +# +# Copyright Red Hat +# Author: David Gibson <david(a)gibson.dropbear.id.au> + +import contextlib +import os.path +import shutil +import subprocess +import tempfile + +import exeter + + +def host_run(*cmd, **kwargs): + return subprocess.run(cmd, check=True, encoding='UTF-8', **kwargs) + + +def host_out(*cmd, **kwargs): + return host_run(*cmd, capture_output=True, **kwargs).stdoutA vague idea only, so far, but I guess it's fine to have some amount of boilerplate.+ + +(a)contextlib.contextmanager +def clone_source_tree(): + with tempfile.TemporaryDirectory(ignore_cleanup_errors=False) as tmpdir: + # Make a temporary copy of the sources + srcfiles = host_out('git', 'ls-files').splitlines() + for src in srcfiles: + dst = os.path.join(tmpdir, src) + os.makedirs(os.path.dirname(dst), exist_ok=True) + shutil.copy(src, dst) + os.chdir(tmpdir) + yield tmpdirThis all makes sense. Of course it would be more readable in shell script (including the trap to remove the temporary directory on failure/interrupt), but I think it's as clear as it can get in any other language.+ + +def build_target(target, outputs): + with clone_source_tree(): + for o in outputs: + assert not os.path.exists(o) + host_run('make', f'{target}', 'CFLAGS="-Werror"')Compared to: host CFLAGS="-Werror" make I would say it's not great, but again, it makes sense, and it's as good as it gets, I suppose.+ for o in outputs: + assert os.path.exists(o) + host_run('make', 'clean') + for o in outputs: + assert not os.path.exists(o)Same here, check [ -f passt ] check [ -h pasta ] check [ -f qrap ]+ + +(a)exeter.test +def test_make_passt(): + build_target('passt', ['passt']) + + +(a)exeter.test +def test_make_pasta(): + build_target('pasta', ['pasta']) + + +(a)exeter.test +def test_make_qrap(): + build_target('qrap', ['qrap']) + + +(a)exeter.test +def test_make_all(): + build_target('all', ['passt', 'pasta', 'qrap'])These all make sense and look relatively readable (while not as... writable as shell commands "everybody" is familiar with).+ +(a)exeter.test +def test_make_install_uninstall(): + with clone_source_tree(): + with tempfile.TemporaryDirectory(ignore_cleanup_errors=False) \ + as prefix: + bindir = os.path.join(prefix, 'bin') + mandir = os.path.join(prefix, 'share', 'man') + exes = ['passt', 'pasta', 'qrap'] + + # Install + host_run('make', 'install', 'CFLAGS="-Werror"', f'prefix={prefix}') + + for t in exes: + assert os.path.isfile(os.path.join(bindir, t)) + host_run('man', '-M', f'{mandir}', '-W', 'passt') + + # Uninstall + host_run('make', 'uninstall', f'prefix={prefix}') + + for t in exes: + assert not os.path.exists(os.path.join(bindir, t)) + cmd = ['man', '-M', f'{mandir}', '-W', 'passt']Same, up to here: it's much more readable and obvious to write in shell script, but I don't find it impossible to grasp in Python, either.+ exeter.assert_raises(subprocess.CalledProcessError, + host_run, *cmd)This, I have no idea why. Why is it only in this loop? How does it affect the control flow?+ + +if __name__ == '__main__': + exeter.main() diff --git a/test/run_avocado b/test/run_avocado index d518b9ec..26a226ce 100755 --- a/test/run_avocado +++ b/test/run_avocado @@ -41,7 +41,7 @@ def main(): "resolver.references": references, "runner.identifier_format": "{args}", } - suite = TestSuite.from_config(config, name="static_checkers") + suite = TestSuite.from_config(config, name="all") with Job(config, [suite]) as j: return j.run()Patch 22/22 will take me a bit longer (I'm just looking at these two for the moment, as you suggested). -- Stefano
On Wed, Aug 07, 2024 at 12:11:26AM +0200, Stefano Brivio wrote:On Mon, 5 Aug 2024 22:36:45 +1000 David Gibson <david(a)gibson.dropbear.id.au> wrote:That's encouraging.Add a new test script to run the equivalent of the tests in build/all using exeter and Avocado. This new version of the tests is more robust than the original, since it makes a temporary copy of the source tree so will not be affected by concurrent manual builds.I think this is much more readable than the previous Python attempt.On the other hand, I guess it's not an ideal candidate for a fair comparison because this is exactly the kind of stuff where shell scripting shines: it's a simple test that needs a few basic shell commands.Right.On that subject, the shell test is about half the lines of code (just skipping headers, it's 48 lines instead of 90... and yes, this versionEven ignoring the fact that this case is particularly suited to shell, I don't think that's really an accurate comparison, but getting to one is pretty hard. The existing test isn't 48 lines of shell, but of "passt test DSL". There are several hundred additional lines of shell to interpret that. Now obviously we don't need all of that for just this test. Likewise the new Python test needs at least exeter - that's only a couple of hundred lines - but also Avocado (huge, but only a small amount is really relevant here).now uses a copy of the source code, but that would be two lines).I feel like it would be a bit more than two lines, to copy exactly what youwant, and to clean up after yourself.In terms of time overhead, dropping delays to make the display capture nice (a feature that we would anyway lose with exeter plus Avocado, if I understood correctly):Yes. Unlike you, I'm really not convinced of the value of the display capture versus log files, at least in the majority of cases. I certainly don't think it's worth slowing down the test running in the normal case.$ time (make clean; make passt; make clean; make pasta; make clean; make qrap; make clean; make; d=$(mktemp -d); prefix=$d make install; prefix=$d make uninstall; ) [...] real 0m17.449s user 0m15.616s sys 0m2.136sOn my system: [...] real 0m20.325s user 0m15.595s sys 0m5.287scompared to: $ time ./run [...] real 0m18.217s user 0m0.010s sys 0m0.001s ...which I would call essentially no overhead. I didn't try out this version yet, I suspect it would be somewhere in between.Well.. $ time PYTHONPATH=test/exeter/py3 test/venv/bin/avocado run test/build/build.json [...] RESULTS : PASS 5 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0 JOB TIME : 10.85 s real 0m11.000s user 0m23.439s sys 0m7.315s Because parallel. It looks like the avocado start up time is reasonably substantial too, so that should look better with a larger set of tests.Yeah, this makefile stuff is very mucky, I'm certainly hoping this can be improved.Signed-off-by: David Gibson <david(a)gibson.dropbear.id.au> --- test/Makefile | 19 +++++--- test/build/.gitignore | 1 + test/build/build.py | 105 ++++++++++++++++++++++++++++++++++++++++++ test/run_avocado | 2 +- 4 files changed, 120 insertions(+), 7 deletions(-) create mode 100644 test/build/build.py diff --git a/test/Makefile b/test/Makefile index dae25312..d24fce14 100644 --- a/test/Makefile +++ b/test/Makefile @@ -64,15 +64,19 @@ LOCAL_ASSETS = mbuto.img mbuto.mem.img podman/bin/podman QEMU_EFI.fd \ ASSETS = $(DOWNLOAD_ASSETS) $(LOCAL_ASSETS) EXETER_SH = build/static_checkers.sh -EXETER_JOBS = $(EXETER_SH:%.sh=%.json) +EXETER_PY = build/build.py +EXETER_JOBS = $(EXETER_SH:%.sh=%.json) $(EXETER_PY:%.py=%.json) AVOCADO_JOBS = $(EXETER_JOBS) avocado/static_checkers.json -SYSTEM_PYTHON = python3 +PYTHON = python3 VENV = venv -PYTHON = $(VENV)/bin/python3 PIP = $(VENV)/bin/pip3 -RUN_AVOCADO = cd .. && test/$(PYTHON) test/run_avocado +PYPATH = exeter/py3 +SPACE = $(subst ,, ) +PYPATH_TEST = $(subst $(SPACE),:,$(PYPATH)) +PYPATH_BASE = $(subst $(SPACE),:,$(PYPATH:%=test/%)) +RUN_AVOCADO = cd .. && PYTHONPATH=$(PYPATH_BASE) test/$(VENV)/bin/python3 test/run_avocadoAt least intuitively, I would have no clue what this all does. But it doesn't matter so much, I could try to find out the day that something doesn't work.It looks messy because of the (interim, I hope) path & cwd wrangling stuff. But the basis is very simple. We run the exeter program: $(PYTHON) test/$< with the '--avocado' flag --avocado and send the output to a json fileCFLAGS = -Wall -Werror -Wextra -pedantic -std=c99 @@ -131,13 +135,16 @@ big.bin: dd if=/dev/urandom bs=1M count=10 of=$@ .PHONY: venv -venv: - $(SYSTEM_PYTHON) -m venv $(VENV) +venv: pull-exeter + $(PYTHON) -m venv $(VENV) $(PIP) install avocado-framework %.json: %.sh pull-exeter cd ..; sh test/$< --avocado > test/$@ +%.json: %.py pull-exeter + cd ..; PYTHONPATH=$(PYPATH_BASE) $(PYTHON) test/$< --avocado > test/$@ +Same here.$@Later..> .PHONY: avocado > avocado: venv $(AVOCADO_JOBS) > $(RUN_AVOCADO) $(AVOCADO_JOBS)..we feed that json file to avocado to actually run the tests.Right. These are loosely equivalent to the implementation of the "host" and "hout" directives in the existing DSL.diff --git a/test/build/.gitignore b/test/build/.gitignore index a6c57f5f..4ef40dd0 100644 --- a/test/build/.gitignore +++ b/test/build/.gitignore @@ -1 +1,2 @@ *.json +build.exeter diff --git a/test/build/build.py b/test/build/build.py new file mode 100644 index 00000000..79668672 --- /dev/null +++ b/test/build/build.py @@ -0,0 +1,105 @@ +#! /usr/bin/env python3 +# +# SPDX-License-Identifier: GPL-2.0-or-later +# +# PASST - Plug A Simple Socket Transport +# for qemu/UNIX domain socket mode +# +# PASTA - Pack A Subtle Tap Abstraction +# for network namespace/tap device mode +# +# test/build/build.sh - Test build and install targets +# +# Copyright Red Hat +# Author: David Gibson <david(a)gibson.dropbear.id.au> + +import contextlib +import os.path +import shutil +import subprocess +import tempfile + +import exeter + + +def host_run(*cmd, **kwargs): + return subprocess.run(cmd, check=True, encoding='UTF-8', **kwargs) + + +def host_out(*cmd, **kwargs): + return host_run(*cmd, capture_output=True, **kwargs).stdoutA vague idea only, so far, but I guess it's fine to have some amount of boilerplate.I don't think that's a fair comparison. The Python equivalent to the DSL line is just: host_run('make', f'{target}', 'CFLAGS="-Werror"') The loop before it is verifying that the targets didn't exist before the make - i.e. we won't spuriously pass because of a stile build. The shell version didn't do that. The with clone_source_tree(): is essentially equivalent saying which (elswhere defined) setup we want. Invoking that explicitly in each test is more verbose, but makes it much easier to see what setup each test needs, and much easier to have lots of different tests with lots of different setups.+(a)contextlib.contextmanager +def clone_source_tree(): + with tempfile.TemporaryDirectory(ignore_cleanup_errors=False) as tmpdir: + # Make a temporary copy of the sources + srcfiles = host_out('git', 'ls-files').splitlines() + for src in srcfiles: + dst = os.path.join(tmpdir, src) + os.makedirs(os.path.dirname(dst), exist_ok=True) + shutil.copy(src, dst) + os.chdir(tmpdir) + yield tmpdirThis all makes sense. Of course it would be more readable in shell script (including the trap to remove the temporary directory on failure/interrupt), but I think it's as clear as it can get in any other language.+ + +def build_target(target, outputs): + with clone_source_tree(): + for o in outputs: + assert not os.path.exists(o) + host_run('make', f'{target}', 'CFLAGS="-Werror"')Compared to: host CFLAGS="-Werror" make I would say it's not great, but again, it makes sense, and it's as good as it gets, I suppose.So, unlike the shell version I'm using a parameterized helper rather than copy-pasting each case. So, that's a readability / brevity trade-off independent of the shell vs. python difference.+ for o in outputs: + assert os.path.exists(o) + host_run('make', 'clean') + for o in outputs: + assert not os.path.exists(o)Same here, check [ -f passt ] check [ -h pasta ] check [ -f qrap ]+ + +(a)exeter.test +def test_make_passt(): + build_target('passt', ['passt']) + + +(a)exeter.test +def test_make_pasta(): + build_target('pasta', ['pasta']) + + +(a)exeter.test +def test_make_qrap(): + build_target('qrap', ['qrap']) + + +(a)exeter.test +def test_make_all(): + build_target('all', ['passt', 'pasta', 'qrap'])These all make sense and look relatively readable (while not as... writable as shell commands "everybody" is familiar with).So, this is essentially check ! man -M ... Now that we've uninstalled, we're re-running (host_run), the same man command (*cmd) as we used before, and checking that it fails (raises the CalledProcessError exception). Come to think of it, I can definitely write this more simply. I'll improve it in the next spin. [snip]+ +(a)exeter.test +def test_make_install_uninstall(): + with clone_source_tree(): + with tempfile.TemporaryDirectory(ignore_cleanup_errors=False) \ + as prefix: + bindir = os.path.join(prefix, 'bin') + mandir = os.path.join(prefix, 'share', 'man') + exes = ['passt', 'pasta', 'qrap'] + + # Install + host_run('make', 'install', 'CFLAGS="-Werror"', f'prefix={prefix}') + + for t in exes: + assert os.path.isfile(os.path.join(bindir, t)) + host_run('man', '-M', f'{mandir}', '-W', 'passt') + + # Uninstall + host_run('make', 'uninstall', f'prefix={prefix}') + + for t in exes: + assert not os.path.exists(os.path.join(bindir, t)) + cmd = ['man', '-M', f'{mandir}', '-W', 'passt']Same, up to here: it's much more readable and obvious to write in shell script, but I don't find it impossible to grasp in Python, either.+ exeter.assert_raises(subprocess.CalledProcessError, + host_run, *cmd)This, I have no idea why. Why is it only in this loop? How does it affect the control flow?Patch 22/22 will take me a bit longer (I'm just looking at these two for the moment, as you suggested).Sure. -- David Gibson (he or they) | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you, not the other way | around. http://www.ozlabs.org/~dgibson
On Wed, 7 Aug 2024 20:51:08 +1000 David Gibson <david(a)gibson.dropbear.id.au> wrote:On Wed, Aug 07, 2024 at 12:11:26AM +0200, Stefano Brivio wrote:Yeah, but the 48 lines is all I have to look at, which is what matters I would argue. That's exactly why I wrote that interpreter. Here, it's 90 lines of *test file*.On Mon, 5 Aug 2024 22:36:45 +1000 David Gibson <david(a)gibson.dropbear.id.au> wrote:That's encouraging.Add a new test script to run the equivalent of the tests in build/all using exeter and Avocado. This new version of the tests is more robust than the original, since it makes a temporary copy of the source tree so will not be affected by concurrent manual builds.I think this is much more readable than the previous Python attempt.On the other hand, I guess it's not an ideal candidate for a fair comparison because this is exactly the kind of stuff where shell scripting shines: it's a simple test that needs a few basic shell commands.Right.On that subject, the shell test is about half the lines of code (just skipping headers, it's 48 lines instead of 90... and yes, this versionEven ignoring the fact that this case is particularly suited to shell, I don't think that's really an accurate comparison, but getting to one is pretty hard. The existing test isn't 48 lines of shell, but of "passt test DSL". There are several hundred additional lines of shell to interpret that.Now obviously we don't need all of that for just this test. Likewise the new Python test needs at least exeter - that's only a couple of hundred lines - but also Avocado (huge, but only a small amount is really relevant here).host mkdir __STATEDIR__/sources host cp --parents $(git ls-files) __STATEDIR__/sources ...which is actually an improvement on the original as __STATEDIR__ can be handled in a centralised way, if one wants to keep that after the single test case, after the whole test run, or not at all.now uses a copy of the source code, but that would be two lines).I feel like it would be a bit more than two lines, to copy exactly what youwant, and to clean up after yourself.Well, but I use that... By the way, openQA nowadays takes periodic screenshots. That's certainly not as useful, but I'm indeed not the only one who benefits from _seeing_ tests as they run instead of correlating log files from different contexts, especially when you have a client, a server, and what you're testing in between.In terms of time overhead, dropping delays to make the display capture nice (a feature that we would anyway lose with exeter plus Avocado, if I understood correctly):Yes. Unlike you, I'm really not convinced of the value of the display capture versus log files, at least in the majority of cases.I certainly don't think it's worth slowing down the test running in the normal case.It doesn't significantly slow things down, but it certainly makes it more complicated to run test cases in parallel... which you can't do anyway for throughput and latency tests (which take 22 out of the 37 minutes of a current CI run), unless you set up VMs with CPU pinning and cgroups, or a server farm. I mean, I see the value of running things in parallel in a general case, but I don't think you should just ignore everything else.With the current set of tests, I doubt it's ever going to pay off. Even if you run the non-perf tests in 10% of the time, it's going to be 24 minutes instead of 37. I guess it will start making sense with larger matrices of network environments, or with more test cases (but really a lot of them).$ time (make clean; make passt; make clean; make pasta; make clean; make qrap; make clean; make; d=$(mktemp -d); prefix=$d make install; prefix=$d make uninstall; ) [...] real 0m17.449s user 0m15.616s sys 0m2.136sOn my system: [...] real 0m20.325s user 0m15.595s sys 0m5.287scompared to: $ time ./run [...] real 0m18.217s user 0m0.010s sys 0m0.001s ...which I would call essentially no overhead. I didn't try out this version yet, I suspect it would be somewhere in between.Well.. $ time PYTHONPATH=test/exeter/py3 test/venv/bin/avocado run test/build/build.json [...] RESULTS : PASS 5 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0 JOB TIME : 10.85 s real 0m11.000s user 0m23.439s sys 0m7.315s Because parallel. It looks like the avocado start up time is reasonably substantial too, so that should look better with a larger set of tests.Yes, that's exactly what I meant...Yeah, this makefile stuff is very mucky, I'm certainly hoping this can be improved.Signed-off-by: David Gibson <david(a)gibson.dropbear.id.au> --- test/Makefile | 19 +++++--- test/build/.gitignore | 1 + test/build/build.py | 105 ++++++++++++++++++++++++++++++++++++++++++ test/run_avocado | 2 +- 4 files changed, 120 insertions(+), 7 deletions(-) create mode 100644 test/build/build.py diff --git a/test/Makefile b/test/Makefile index dae25312..d24fce14 100644 --- a/test/Makefile +++ b/test/Makefile @@ -64,15 +64,19 @@ LOCAL_ASSETS = mbuto.img mbuto.mem.img podman/bin/podman QEMU_EFI.fd \ ASSETS = $(DOWNLOAD_ASSETS) $(LOCAL_ASSETS) EXETER_SH = build/static_checkers.sh -EXETER_JOBS = $(EXETER_SH:%.sh=%.json) +EXETER_PY = build/build.py +EXETER_JOBS = $(EXETER_SH:%.sh=%.json) $(EXETER_PY:%.py=%.json) AVOCADO_JOBS = $(EXETER_JOBS) avocado/static_checkers.json -SYSTEM_PYTHON = python3 +PYTHON = python3 VENV = venv -PYTHON = $(VENV)/bin/python3 PIP = $(VENV)/bin/pip3 -RUN_AVOCADO = cd .. && test/$(PYTHON) test/run_avocado +PYPATH = exeter/py3 +SPACE = $(subst ,, ) +PYPATH_TEST = $(subst $(SPACE),:,$(PYPATH)) +PYPATH_BASE = $(subst $(SPACE),:,$(PYPATH:%=test/%)) +RUN_AVOCADO = cd .. && PYTHONPATH=$(PYPATH_BASE) test/$(VENV)/bin/python3 test/run_avocadoAt least intuitively, I would have no clue what this all does. But it doesn't matter so much, I could try to find out the day that something doesn't work.It looks messy because of the (interim, I hope) path & cwd wrangling stuff. But the basis is very simple. We run the exeter program: $(PYTHON) test/$< with the '--avocado' flag --avocado and send the output to a json fileCFLAGS = -Wall -Werror -Wextra -pedantic -std=c99 @@ -131,13 +135,16 @@ big.bin: dd if=/dev/urandom bs=1M count=10 of=$@ .PHONY: venv -venv: - $(SYSTEM_PYTHON) -m venv $(VENV) +venv: pull-exeter + $(PYTHON) -m venv $(VENV) $(PIP) install avocado-framework %.json: %.sh pull-exeter cd ..; sh test/$< --avocado > test/$@ +%.json: %.py pull-exeter + cd ..; PYTHONPATH=$(PYPATH_BASE) $(PYTHON) test/$< --avocado > test/$@ +Same here.$@Later..> .PHONY: avocado > avocado: venv $(AVOCADO_JOBS) > $(RUN_AVOCADO) $(AVOCADO_JOBS)..we feed that json file to avocado to actually run the tests.Right. These are loosely equivalent to the implementation of the "host" and "hout" directives in the existing DSL.diff --git a/test/build/.gitignore b/test/build/.gitignore index a6c57f5f..4ef40dd0 100644 --- a/test/build/.gitignore +++ b/test/build/.gitignore @@ -1 +1,2 @@ *.json +build.exeter diff --git a/test/build/build.py b/test/build/build.py new file mode 100644 index 00000000..79668672 --- /dev/null +++ b/test/build/build.py @@ -0,0 +1,105 @@ +#! /usr/bin/env python3 +# +# SPDX-License-Identifier: GPL-2.0-or-later +# +# PASST - Plug A Simple Socket Transport +# for qemu/UNIX domain socket mode +# +# PASTA - Pack A Subtle Tap Abstraction +# for network namespace/tap device mode +# +# test/build/build.sh - Test build and install targets +# +# Copyright Red Hat +# Author: David Gibson <david(a)gibson.dropbear.id.au> + +import contextlib +import os.path +import shutil +import subprocess +import tempfile + +import exeter + + +def host_run(*cmd, **kwargs): + return subprocess.run(cmd, check=True, encoding='UTF-8', **kwargs) + + +def host_out(*cmd, **kwargs): + return host_run(*cmd, capture_output=True, **kwargs).stdoutA vague idea only, so far, but I guess it's fine to have some amount of boilerplate.I don't think that's a fair comparison. The Python equivalent to the DSL line is just: host_run('make', f'{target}', 'CFLAGS="-Werror"')+(a)contextlib.contextmanager +def clone_source_tree(): + with tempfile.TemporaryDirectory(ignore_cleanup_errors=False) as tmpdir: + # Make a temporary copy of the sources + srcfiles = host_out('git', 'ls-files').splitlines() + for src in srcfiles: + dst = os.path.join(tmpdir, src) + os.makedirs(os.path.dirname(dst), exist_ok=True) + shutil.copy(src, dst) + os.chdir(tmpdir) + yield tmpdirThis all makes sense. Of course it would be more readable in shell script (including the trap to remove the temporary directory on failure/interrupt), but I think it's as clear as it can get in any other language.+ + +def build_target(target, outputs): + with clone_source_tree(): + for o in outputs: + assert not os.path.exists(o) + host_run('make', f'{target}', 'CFLAGS="-Werror"')Compared to: host CFLAGS="-Werror" make I would say it's not great, but again, it makes sense, and it's as good as it gets, I suppose.The loop before it is verifying that the targets didn't exist before the make - i.e. we won't spuriously pass because of a stile build. The shell version didn't do that. The with clone_source_tree(): is essentially equivalent saying which (elswhere defined) setup we want. Invoking that explicitly in each test is more verbose, but makes it much easier to see what setup each test needs, and much easier to have lots of different tests with lots of different setups.No, absolutely, the rest is actually clear enough, I guess.-- StefanoSo, unlike the shell version I'm using a parameterized helper rather than copy-pasting each case. So, that's a readability / brevity trade-off independent of the shell vs. python difference.+ for o in outputs: + assert os.path.exists(o) + host_run('make', 'clean') + for o in outputs: + assert not os.path.exists(o)Same here, check [ -f passt ] check [ -h pasta ] check [ -f qrap ]+ + +(a)exeter.test +def test_make_passt(): + build_target('passt', ['passt']) + + +(a)exeter.test +def test_make_pasta(): + build_target('pasta', ['pasta']) + + +(a)exeter.test +def test_make_qrap(): + build_target('qrap', ['qrap']) + + +(a)exeter.test +def test_make_all(): + build_target('all', ['passt', 'pasta', 'qrap'])These all make sense and look relatively readable (while not as... writable as shell commands "everybody" is familiar with).So, this is essentially check ! man -M ... Now that we've uninstalled, we're re-running (host_run), the same man command (*cmd) as we used before, and checking that it fails (raises the CalledProcessError exception). Come to think of it, I can definitely write this more simply. I'll improve it in the next spin. [snip]+ +(a)exeter.test +def test_make_install_uninstall(): + with clone_source_tree(): + with tempfile.TemporaryDirectory(ignore_cleanup_errors=False) \ + as prefix: + bindir = os.path.join(prefix, 'bin') + mandir = os.path.join(prefix, 'share', 'man') + exes = ['passt', 'pasta', 'qrap'] + + # Install + host_run('make', 'install', 'CFLAGS="-Werror"', f'prefix={prefix}') + + for t in exes: + assert os.path.isfile(os.path.join(bindir, t)) + host_run('man', '-M', f'{mandir}', '-W', 'passt') + + # Uninstall + host_run('make', 'uninstall', f'prefix={prefix}') + + for t in exes: + assert not os.path.exists(os.path.join(bindir, t)) + cmd = ['man', '-M', f'{mandir}', '-W', 'passt']Same, up to here: it's much more readable and obvious to write in shell script, but I don't find it impossible to grasp in Python, either.+ exeter.assert_raises(subprocess.CalledProcessError, + host_run, *cmd)This, I have no idea why. Why is it only in this loop? How does it affect the control flow?Patch 22/22 will take me a bit longer (I'm just looking at these two for the moment, as you suggested).Sure.
On Wed, Aug 07, 2024 at 03:06:44PM +0200, Stefano Brivio wrote:On Wed, 7 Aug 2024 20:51:08 +1000 David Gibson <david(a)gibson.dropbear.id.au> wrote:Fair point. Fwiw, it's down to 77 so far for my next draft.On Wed, Aug 07, 2024 at 12:11:26AM +0200, Stefano Brivio wrote:Yeah, but the 48 lines is all I have to look at, which is what matters I would argue. That's exactly why I wrote that interpreter. Here, it's 90 lines of *test file*.On Mon, 5 Aug 2024 22:36:45 +1000 David Gibson <david(a)gibson.dropbear.id.au> wrote:That's encouraging.Add a new test script to run the equivalent of the tests in build/all using exeter and Avocado. This new version of the tests is more robust than the original, since it makes a temporary copy of the source tree so will not be affected by concurrent manual builds.I think this is much more readable than the previous Python attempt.On the other hand, I guess it's not an ideal candidate for a fair comparison because this is exactly the kind of stuff where shell scripting shines: it's a simple test that needs a few basic shell commands.Right.On that subject, the shell test is about half the lines of code (just skipping headers, it's 48 lines instead of 90... and yes, this versionEven ignoring the fact that this case is particularly suited to shell, I don't think that's really an accurate comparison, but getting to one is pretty hard. The existing test isn't 48 lines of shell, but of "passt test DSL". There are several hundred additional lines of shell to interpret that.Huh, I didn't know about cp --parents, which does exactly what's needed. In the Python library there are, alas, several things that do almost but not quite what's needed. I guess I could just invoke 'cp --parents' myself.Now obviously we don't need all of that for just this test. Likewise the new Python test needs at least exeter - that's only a couple of hundred lines - but also Avocado (huge, but only a small amount is really relevant here).host mkdir __STATEDIR__/sources host cp --parents $(git ls-files) __STATEDIR__/sources ...which is actually an improvement on the original as __STATEDIR__ can be handled in a centralised way, if one wants to keep that after the single test case, after the whole test run, or not at all.now uses a copy of the source code, but that would be two lines).I feel like it would be a bit more than two lines, to copy exactly what youwant, and to clean up after yourself.If you have to correlate multiple logs that's a pain, yes. My approach here is, as much as possible, to have a single "log" (actually stdout & stderr) from the top level test logic, so the logical ordering is kind of built in.Well, but I use that... By the way, openQA nowadays takes periodic screenshots. That's certainly not as useful, but I'm indeed not the only one who benefits from _seeing_ tests as they run instead of correlating log files from different contexts, especially when you have a client, a server, and what you're testing in between.In terms of time overhead, dropping delays to make the display capture nice (a feature that we would anyway lose with exeter plus Avocado, if I understood correctly):Yes. Unlike you, I'm really not convinced of the value of the display capture versus log files, at least in the majority of cases.It does if you explicitly add delays to make the display capture nice as mentioned above.I certainly don't think it's worth slowing down the test running in the normal case.It doesn't significantly slow things down,but it certainly makes it more complicated to run test cases in parallel... which you can't do anyway for throughput and latency tests (which take 22 out of the 37 minutes of a current CI run), unless you set up VMs with CPU pinning and cgroups, or a server farm.So, yes, the perf tests take the majority of the runtime for CI, but I'm less concerned about runtime for CI tests. I'm more interested in runtime for a subset of functional tests you can run repeatedly while developing. I routinely disable the perf and other slow tests, to get a subset taking 5-7 minutes. That's ok, but I'm pretty confident I can get better coverage in significantly less time using parallel tests.I mean, I see the value of running things in parallel in a general case, but I don't think you should just ignore everything else.Including the perf tests, probably not. Excluding them (which is extremely useful when actively coding) I think it will.With the current set of tests, I doubt it's ever going to pay off. Even if you run the non-perf tests in 10% of the time, it's going to be 24 minutes instead of 37.$ time (make clean; make passt; make clean; make pasta; make clean; make qrap; make clean; make; d=$(mktemp -d); prefix=$d make install; prefix=$d make uninstall; ) [...] real 0m17.449s user 0m15.616s sys 0m2.136sOn my system: [...] real 0m20.325s user 0m15.595s sys 0m5.287scompared to: $ time ./run [...] real 0m18.217s user 0m0.010s sys 0m0.001s ...which I would call essentially no overhead. I didn't try out this version yet, I suspect it would be somewhere in between.Well.. $ time PYTHONPATH=test/exeter/py3 test/venv/bin/avocado run test/build/build.json [...] RESULTS : PASS 5 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0 JOB TIME : 10.85 s real 0m11.000s user 0m23.439s sys 0m7.315s Because parallel. It looks like the avocado start up time is reasonably substantial too, so that should look better with a larger set of tests.I guess it will start making sense with larger matrices of network environments, or with more test cases (but really a lot of them).We could certainly do with a lot more tests, though I expect it will take a while to get them. -- David Gibson (he or they) | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you, not the other way | around. http://www.ozlabs.org/~dgibson
On Thu, 8 Aug 2024 11:28:50 +1000 David Gibson <david(a)gibson.dropbear.id.au> wrote:On Wed, Aug 07, 2024 at 03:06:44PM +0200, Stefano Brivio wrote:That's not necessarily helpful: if I have a client and a server, things are much clearer to me if I have two different logs, side-by-side. Even more so if you have a guest, a host, and a namespace "in between". I see the difference as I'm often digging through Podman CI's logs, where there's a single log (including stdout and stderr), because bats doesn't offer a context functionality like we have right now. It's sometimes really not easy to understand what's going on in Podman's tests without copying and pasting into an editor and manually marking things.On Wed, 7 Aug 2024 20:51:08 +1000 David Gibson <david(a)gibson.dropbear.id.au> wrote:Fair point. Fwiw, it's down to 77 so far for my next draft.On Wed, Aug 07, 2024 at 12:11:26AM +0200, Stefano Brivio wrote:Yeah, but the 48 lines is all I have to look at, which is what matters I would argue. That's exactly why I wrote that interpreter. Here, it's 90 lines of *test file*.On Mon, 5 Aug 2024 22:36:45 +1000 David Gibson <david(a)gibson.dropbear.id.au> wrote: > Add a new test script to run the equivalent of the tests in build/all > using exeter and Avocado. This new version of the tests is more robust > than the original, since it makes a temporary copy of the source tree so > will not be affected by concurrent manual builds. I think this is much more readable than the previous Python attempt.That's encouraging.On the other hand, I guess it's not an ideal candidate for a fair comparison because this is exactly the kind of stuff where shell scripting shines: it's a simple test that needs a few basic shell commands.Right.On that subject, the shell test is about half the lines of code (just skipping headers, it's 48 lines instead of 90... and yes, this versionEven ignoring the fact that this case is particularly suited to shell, I don't think that's really an accurate comparison, but getting to one is pretty hard. The existing test isn't 48 lines of shell, but of "passt test DSL". There are several hundred additional lines of shell to interpret that.Huh, I didn't know about cp --parents, which does exactly what's needed. In the Python library there are, alas, several things that do almost but not quite what's needed. I guess I could just invoke 'cp --parents' myself.Now obviously we don't need all of that for just this test. Likewise the new Python test needs at least exeter - that's only a couple of hundred lines - but also Avocado (huge, but only a small amount is really relevant here).host mkdir __STATEDIR__/sources host cp --parents $(git ls-files) __STATEDIR__/sources ...which is actually an improvement on the original as __STATEDIR__ can be handled in a centralised way, if one wants to keep that after the single test case, after the whole test run, or not at all.now uses a copy of the source code, but that would be two lines).I feel like it would be a bit more than two lines, to copy exactly what youwant, and to clean up after yourself.If you have to correlate multiple logs that's a pain, yes. My approach here is, as much as possible, to have a single "log" (actually stdout & stderr) from the top level test logic, so the logical ordering is kind of built in.Well, but I use that... By the way, openQA nowadays takes periodic screenshots. That's certainly not as useful, but I'm indeed not the only one who benefits from _seeing_ tests as they run instead of correlating log files from different contexts, especially when you have a client, a server, and what you're testing in between.In terms of time overhead, dropping delays to make the display capture nice (a feature that we would anyway lose with exeter plus Avocado, if I understood correctly):Yes. Unlike you, I'm really not convinced of the value of the display capture versus log files, at least in the majority of cases.Okay, I didn't realise the amount of eye-candy I left in even when ${FAST} is set (which probably only makes sense when run as './ci'). With the patch attached I get: $ time ./run [...] real 17m17.686s user 0m0.010s sys 0m0.014s I also cut the duration of throughput and latency tests down to one second. After we fixed lot of issues in passt, and some in QEMU and kernel, results are now surprisingly consistent. Still, a significant part of it is Podman's tests (which I'm working on speeding up, for the sake of Podman's own CI), and performance tests anyway. Without those: $ time ./run [...] real 5m57.612s user 0m0.011s sys 0m0.009sIt does if you explicitly add delays to make the display capture nice as mentioned above.I certainly don't think it's worth slowing down the test running in the normal case.It doesn't significantly slow things down,Probably, yes, but still I would like to point out that the difference between five and ten minutes is not as relevant in terms of workflow as the difference between one and five minutes.but it certainly makes it more complicated to run test cases in parallel... which you can't do anyway for throughput and latency tests (which take 22 out of the 37 minutes of a current CI run), unless you set up VMs with CPU pinning and cgroups, or a server farm.So, yes, the perf tests take the majority of the runtime for CI, but I'm less concerned about runtime for CI tests. I'm more interested in runtime for a subset of functional tests you can run repeatedly while developing. I routinely disable the perf and other slow tests, to get a subset taking 5-7 minutes. That's ok, but I'm pretty confident I can get better coverage in significantly less time using parallel tests.-- StefanoI mean, I see the value of running things in parallel in a general case, but I don't think you should just ignore everything else.Including the perf tests, probably not. Excluding them (which is extremely useful when actively coding) I think it will.With the current set of tests, I doubt it's ever going to pay off. Even if you run the non-perf tests in 10% of the time, it's going to be 24 minutes instead of 37.$ time (make clean; make passt; make clean; make pasta; make clean; make qrap; make clean; make; d=$(mktemp -d); prefix=$d make install; prefix=$d make uninstall; ) [...] real 0m17.449s user 0m15.616s sys 0m2.136sOn my system: [...] real 0m20.325s user 0m15.595s sys 0m5.287scompared to: $ time ./run [...] real 0m18.217s user 0m0.010s sys 0m0.001s ...which I would call essentially no overhead. I didn't try out this version yet, I suspect it would be somewhere in between.Well.. $ time PYTHONPATH=test/exeter/py3 test/venv/bin/avocado run test/build/build.json [...] RESULTS : PASS 5 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0 JOB TIME : 10.85 s real 0m11.000s user 0m23.439s sys 0m7.315s Because parallel. It looks like the avocado start up time is reasonably substantial too, so that should look better with a larger set of tests.I guess it will start making sense with larger matrices of network environments, or with more test cases (but really a lot of them).We could certainly do with a lot more tests, though I expect it will take a while to get them.
We use both cppcheck and clang-tidy to lint our C code. Now that we're
introducing Python code in the tests, use linters pycodestyle and flake8.
Add a "make meta" target to run tests of the test infrastructure. For now
it just has the linters, but we'll add more in future.
Signed-off-by: David Gibson <david(a)gibson.dropbear.id.au>
---
test/Makefile | 22 +++++++++++++++++++---
test/build/static_checkers.sh | 2 --
test/meta/.gitignore | 1 +
test/meta/lint.sh | 28 ++++++++++++++++++++++++++++
test/run_avocado | 5 +++--
5 files changed, 51 insertions(+), 7 deletions(-)
create mode 100644 test/meta/.gitignore
create mode 100644 test/meta/lint.sh
diff --git a/test/Makefile b/test/Makefile
index d24fce14..0b3ed3d0 100644
--- a/test/Makefile
+++ b/test/Makefile
@@ -6,6 +6,8 @@
# Author: David Gibson <david(a)gibson.dropbear.id.au>
WGET = wget -c
+FLAKE8 = flake8-3
+PYCODESTYLE = pycodestyle-3
DEBIAN_IMGS = debian-8.11.0-openstack-amd64.qcow2 \
debian-9-nocloud-amd64-daily-20200210-166.qcow2 \
@@ -66,9 +68,13 @@ ASSETS = $(DOWNLOAD_ASSETS) $(LOCAL_ASSETS)
EXETER_SH = build/static_checkers.sh
EXETER_PY = build/build.py
EXETER_JOBS = $(EXETER_SH:%.sh=%.json) $(EXETER_PY:%.py=%.json)
-
AVOCADO_JOBS = $(EXETER_JOBS) avocado/static_checkers.json
+EXETER_META = meta/lint.json
+META_JOBS = $(EXETER_META)
+
+PYPKGS = $(EXETER_PY)
+
PYTHON = python3
VENV = venv
PIP = $(VENV)/bin/pip3
@@ -147,7 +153,17 @@ venv: pull-exeter
.PHONY: avocado
avocado: venv $(AVOCADO_JOBS)
- $(RUN_AVOCADO) $(AVOCADO_JOBS)
+ $(RUN_AVOCADO) all $(AVOCADO_JOBS)
+
+.PHONY: meta
+meta: venv $(META_JOBS)
+ $(RUN_AVOCADO) meta $(META_JOBS)
+
+flake8:
+ $(FLAKE8) $(PYPKGS)
+
+pycodestyle:
+ $(PYCODESTYLE) $(PYPKGS)
check: assets
./run
@@ -161,7 +177,7 @@ clean:
rm -rf test_logs
rm -f prepared-*.qcow2 prepared-*.img
rm -rf $(VENV)
- rm -f $(EXETER_JOBS)
+ rm -f $(EXETER_JOBS) $(EXETER_META)
realclean: clean
rm -rf $(DOWNLOAD_ASSETS)
diff --git a/test/build/static_checkers.sh b/test/build/static_checkers.sh
index ec159ea2..fa07f8fd 100644
--- a/test/build/static_checkers.sh
+++ b/test/build/static_checkers.sh
@@ -26,5 +26,3 @@ clang_tidy () {
exeter_register clang_tidy
exeter_main "$@"
-
-
diff --git a/test/meta/.gitignore b/test/meta/.gitignore
new file mode 100644
index 00000000..a6c57f5f
--- /dev/null
+++ b/test/meta/.gitignore
@@ -0,0 +1 @@
+*.json
diff --git a/test/meta/lint.sh b/test/meta/lint.sh
new file mode 100644
index 00000000..6cbaa5d4
--- /dev/null
+++ b/test/meta/lint.sh
@@ -0,0 +1,28 @@
+#! /bin/sh
+#
+# SPDX-License-Identifier: GPL-2.0-or-later
+#
+# PASST - Plug A Simple Socket Transport
+# for qemu/UNIX domain socket mode
+#
+# PASTA - Pack A Subtle Tap Abstraction
+# for network namespace/tap device mode
+#
+# test/meta/lint.sh - Linters for the test code
+#
+# Copyright Red Hat
+# Author: David Gibson <david(a)gibson.dropbear.id.au>
+
+source $(dirname $0)/../exeter/sh/exeter.sh
+
+flake8 () {
+ make -C test flake8
+}
+exeter_register flake8
+
+pycodestyle () {
+ make -C test pycodestyle
+}
+exeter_register pycodestyle
+
+exeter_main "$@"
diff --git a/test/run_avocado b/test/run_avocado
index 26a226ce..b62864f6 100755
--- a/test/run_avocado
+++ b/test/run_avocado
@@ -35,13 +35,14 @@ def main():
os.path.dirname(os.path.dirname(__file__))
)
- references = [os.path.join(repo_root_path, 'test', x) for x in
sys.argv[1:]]
+ suitename = sys.argv[1]
+ references = [os.path.join(repo_root_path, 'test', x) for x in
sys.argv[2:]]
config = {
"resolver.references": references,
"runner.identifier_format": "{args}",
}
- suite = TestSuite.from_config(config, name="all")
+ suite = TestSuite.from_config(config, name=suitename)
with Job(config, [suite]) as j:
return j.run()
--
2.45.2
Create a Python package "tasst" with common helper code for use in passt and pasta. Initially it just has a placeholder selftest. Extend the meta tests to include selftests within the tasst library. This lets us test the functionality of the library itself without involving actual passt or pasta. Signed-off-by: David Gibson <david(a)gibson.dropbear.id.au> --- test/Makefile | 13 +++++++++---- test/tasst/.gitignore | 1 + test/tasst/__init__.py | 11 +++++++++++ test/tasst/__main__.py | 22 ++++++++++++++++++++++ 4 files changed, 43 insertions(+), 4 deletions(-) create mode 100644 test/tasst/.gitignore create mode 100644 test/tasst/__init__.py create mode 100644 test/tasst/__main__.py diff --git a/test/Makefile b/test/Makefile index 0b3ed3d0..81f94f70 100644 --- a/test/Makefile +++ b/test/Makefile @@ -70,15 +70,17 @@ EXETER_PY = build/build.py EXETER_JOBS = $(EXETER_SH:%.sh=%.json) $(EXETER_PY:%.py=%.json) AVOCADO_JOBS = $(EXETER_JOBS) avocado/static_checkers.json -EXETER_META = meta/lint.json +TASST_SRCS = __init__.py __main__.py + +EXETER_META = meta/lint.json meta/tasst.json META_JOBS = $(EXETER_META) -PYPKGS = $(EXETER_PY) +PYPKGS = tasst $(EXETER_PY) PYTHON = python3 VENV = venv PIP = $(VENV)/bin/pip3 -PYPATH = exeter/py3 +PYPATH = . exeter/py3 SPACE = $(subst ,, ) PYPATH_TEST = $(subst $(SPACE),:,$(PYPATH)) PYPATH_BASE = $(subst $(SPACE),:,$(PYPATH:%=test/%)) @@ -151,6 +153,9 @@ venv: pull-exeter %.json: %.py pull-exeter cd ..; PYTHONPATH=$(PYPATH_BASE) $(PYTHON) test/$< --avocado > test/$@ +meta/tasst.json: $(TASST_SRCS:%=tasst/%) $(VENV) pull-exeter + cd ..; PYTHONPATH=$(PYPATH_BASE) $(PYTHON) -m tasst --avocado > test/$@ + .PHONY: avocado avocado: venv $(AVOCADO_JOBS) $(RUN_AVOCADO) all $(AVOCADO_JOBS) @@ -176,7 +181,7 @@ clean: rm -f $(LOCAL_ASSETS) rm -rf test_logs rm -f prepared-*.qcow2 prepared-*.img - rm -rf $(VENV) + rm -rf $(VENV) tasst/__pycache__ rm -f $(EXETER_JOBS) $(EXETER_META) realclean: clean diff --git a/test/tasst/.gitignore b/test/tasst/.gitignore new file mode 100644 index 00000000..c18dd8d8 --- /dev/null +++ b/test/tasst/.gitignore @@ -0,0 +1 @@ +__pycache__/ diff --git a/test/tasst/__init__.py b/test/tasst/__init__.py new file mode 100644 index 00000000..c1d5d9dd --- /dev/null +++ b/test/tasst/__init__.py @@ -0,0 +1,11 @@ +#! /usr/bin/env python3 + +# SPDX-License-Identifier: GPL-2.0-or-later +# +# Copyright Red Hat +# Author: David Gibson <david(a)gibson.dropbear.id.au> + +""" +Test A Simple Socket Transport +library of test helpers for passt & pasta +""" diff --git a/test/tasst/__main__.py b/test/tasst/__main__.py new file mode 100644 index 00000000..c365b986 --- /dev/null +++ b/test/tasst/__main__.py @@ -0,0 +1,22 @@ +#! /usr/bin/env python3 + +# SPDX-License-Identifier: GPL-2.0-or-later +# +# Copyright Red Hat +# Author: David Gibson <david(a)gibson.dropbear.id.au> + +""" +Test A Simple Socket Transport +library of test helpers for passt & pasta +""" + +import exeter + + +(a)exeter.test +def placeholder(): + pass + + +if __name__ == '__main__': + exeter.main() -- 2.45.2
Add to the tasst library a SimNetHost class used to represent a
simulated network host of some type (e.g. namespaces, VMs). For now
all it does is lets you execute commands, either foreground or
background in the context of the simulated host. We add some "meta"
exeter tests for it.
Signed-off-by: David Gibson <david(a)gibson.dropbear.id.au>
---
test/Makefile | 2 +-
test/tasst/__main__.py | 6 +-
test/tasst/snh.py | 187 +++++++++++++++++++++++++++++++++++++++++
3 files changed, 190 insertions(+), 5 deletions(-)
create mode 100644 test/tasst/snh.py
diff --git a/test/Makefile b/test/Makefile
index 81f94f70..8373ae77 100644
--- a/test/Makefile
+++ b/test/Makefile
@@ -70,7 +70,7 @@ EXETER_PY = build/build.py
EXETER_JOBS = $(EXETER_SH:%.sh=%.json) $(EXETER_PY:%.py=%.json)
AVOCADO_JOBS = $(EXETER_JOBS) avocado/static_checkers.json
-TASST_SRCS = __init__.py __main__.py
+TASST_SRCS = __init__.py __main__.py snh.py
EXETER_META = meta/lint.json meta/tasst.json
META_JOBS = $(EXETER_META)
diff --git a/test/tasst/__main__.py b/test/tasst/__main__.py
index c365b986..91499128 100644
--- a/test/tasst/__main__.py
+++ b/test/tasst/__main__.py
@@ -12,10 +12,8 @@ library of test helpers for passt & pasta
import exeter
-
-(a)exeter.test
-def placeholder():
- pass
+# We import just to get the exeter tests, which flake8 can't see
+from . import snh # noqa: F401
if __name__ == '__main__':
diff --git a/test/tasst/snh.py b/test/tasst/snh.py
new file mode 100644
index 00000000..dfbe2c84
--- /dev/null
+++ b/test/tasst/snh.py
@@ -0,0 +1,187 @@
+#! /usr/bin/env python3
+
+# SPDX-License-Identifier: GPL-2.0-or-later
+#
+# Copyright Red Hat
+# Author: David Gibson <david(a)gibson.dropbear.id.au>
+
+"""
+Test A Simple Socket Transport
+
+tasst/snh.py - Simulated network hosts for testing
+"""
+
+
+import contextlib
+import subprocess
+import sys
+
+import exeter
+
+
+STDOUT = 1
+
+
+class SnhProcess(contextlib.AbstractContextManager):
+ """
+ A background process running on a SimNetHost
+ """
+
+ def __init__(self, snh, *cmd, check=True, context_timeout=1.0, **kwargs):
+ self.snh = snh
+ self.cmd = cmd
+ self.check = check
+ self.context_timeout = float(context_timeout)
+
+ self.kwargs = kwargs
+
+ def __enter__(self):
+ self.popen = subprocess.Popen(self.cmd, **self.kwargs)
+ return self
+
+ def run(self, **kwargs):
+ stdout, stderr = self.popen.communicate(**kwargs)
+ cp = subprocess.CompletedProcess(self.popen.args,
+ self.popen.returncode,
+ stdout, stderr)
+ if self.check:
+ cp.check_returncode()
+ return cp
+
+ def terminate(self):
+ self.popen.terminate()
+
+ def kill(self):
+ self.popen.kill()
+
+ def __exit__(self, *exc_details):
+ try:
+ self.popen.wait(timeout=self.context_timeout)
+ except subprocess.TimeoutExpired as e:
+ self.terminate()
+ try:
+ self.popen.wait(timeout=self.context_timeout)
+ except subprocess.TimeoutExpired:
+ self.kill()
+ raise e
+
+
+class SimNetHost(contextlib.AbstractContextManager):
+ """
+ A (usually virtual or simulated) location where we can execute
+ commands and configure networks.
+
+ """
+
+ def __init__(self, name):
+ self.name = name # For debugging
+
+ def hostify(self, *cmd, **kwargs):
+ raise NotImplementedError
+
+ def __enter__(self):
+ return self
+
+ def __exit__(self, *exc_details):
+ pass
+
+ def output(self, *cmd, **kwargs):
+ proc = self.fg(*cmd, capture=STDOUT, **kwargs)
+ return proc.stdout
+
+ def fg(self, *cmd, timeout=None, **kwargs):
+ # We don't use subprocess.run() because it kills without
+ # attempting to terminate on timeout
+ with self.bg(*cmd, **kwargs) as proc:
+ res = proc.run(timeout=timeout)
+ return res
+
+ def bg(self, *cmd, capture=None, **kwargs):
+ if capture == STDOUT:
+ kwargs['stdout'] = subprocess.PIPE
+ hostcmd, kwargs = self.hostify(*cmd, **kwargs)
+ proc = SnhProcess(self, *hostcmd, **kwargs)
+ print(f"SimNetHost {self.name}: Started {cmd} => {proc}",
+ file=sys.stderr)
+ return proc
+
+ # Internal tests
+ def test_true(self):
+ with self as snh:
+ snh.fg('true')
+
+ def test_false(self):
+ with self as snh:
+ exeter.assert_raises(subprocess.CalledProcessError,
+ snh.fg, 'false')
+
+ def test_echo(self):
+ msg = 'Hello tasst'
+ with self as snh:
+ out = snh.output('echo', f'{msg}')
+ exeter.assert_eq(out, msg.encode('utf-8') + b'\n')
+
+ def test_timeout(self):
+ with self as snh:
+ exeter.assert_raises(subprocess.TimeoutExpired, snh.fg,
+ 'sleep', 'infinity', timeout=0.1,
check=False)
+
+ def test_bg_true(self):
+ with self as snh:
+ with snh.bg('true'):
+ pass
+
+ def test_bg_false(self):
+ with self as snh:
+ with snh.bg('false') as proc:
+ exeter.assert_raises(subprocess.CalledProcessError, proc.run)
+
+ def test_bg_echo(self):
+ msg = 'Hello tasst'
+ with self as snh:
+ with snh.bg('echo', f'{msg}', capture=STDOUT) as proc:
+ res = proc.run()
+ exeter.assert_eq(res.stdout, msg.encode('utf-8') +
b'\n')
+
+ def test_bg_timeout(self):
+ with self as snh:
+ with snh.bg('sleep', 'infinity') as proc:
+ exeter.assert_raises(subprocess.TimeoutExpired,
+ proc.run, timeout=0.1)
+ proc.terminate()
+
+ def test_bg_context_timeout(self):
+ with self as snh:
+ def run_timeout():
+ with snh.bg('sleep', 'infinity',
context_timeout=0.1):
+ pass
+ exeter.assert_raises(subprocess.TimeoutExpired, run_timeout)
+
+ SELFTESTS = [test_true, test_false, test_echo, test_timeout,
+ test_bg_true, test_bg_false, test_bg_echo, test_bg_timeout,
+ test_bg_context_timeout]
+
+ @classmethod
+ def selftest(cls, setup):
+ "Register standard snh tests for instance returned by setup"
+ for t in cls.SELFTESTS:
+ testid = f'{setup.__qualname__}|{t.__qualname__}'
+ exeter.register_pipe(testid, setup, t)
+
+
+class RealHost(SimNetHost):
+ """Represents the host on which the tests are running (as opposed
+ to some simulated host created by the tests)
+
+ """
+
+ def __init__(self):
+ super().__init__('REAL_HOST')
+
+ def hostify(self, *cmd, capable=False, **kwargs):
+ assert not capable, \
+ "BUG: Shouldn't run commands with capabilities on host"
+ return cmd, kwargs
+
+
+SimNetHost.selftest(RealHost)
--
2.45.2
Start adding convenience functions for handling sites as places with
network setup with a simple helper which lists the network interface names
for a site.
Signed-off-by: David Gibson <david(a)gibson.dropbear.id.au>
---
test/tasst/snh.py | 12 +++++++++++-
1 file changed, 11 insertions(+), 1 deletion(-)
diff --git a/test/tasst/snh.py b/test/tasst/snh.py
index dfbe2c84..8ee9802a 100644
--- a/test/tasst/snh.py
+++ b/test/tasst/snh.py
@@ -13,6 +13,7 @@ tasst/snh.py - Simulated network hosts for testing
import contextlib
+import json
import subprocess
import sys
@@ -105,6 +106,10 @@ class SimNetHost(contextlib.AbstractContextManager):
file=sys.stderr)
return proc
+ def ifs(self):
+ info = json.loads(self.output('ip', '-j', 'link',
'show'))
+ return [i['ifname'] for i in info]
+
# Internal tests
def test_true(self):
with self as snh:
@@ -157,9 +162,14 @@ class SimNetHost(contextlib.AbstractContextManager):
pass
exeter.assert_raises(subprocess.TimeoutExpired, run_timeout)
+ def test_has_lo(self):
+ with self as snh:
+ assert 'lo' in snh.ifs()
+
SELFTESTS = [test_true, test_false, test_echo, test_timeout,
test_bg_true, test_bg_false, test_bg_echo, test_bg_timeout,
- test_bg_context_timeout]
+ test_bg_context_timeout,
+ test_has_lo]
@classmethod
def selftest(cls, setup):
--
2.45.2
Use our existing nstool C helper, add python wrappers to easily run
commands in various namespaces.
Signed-off-by: David Gibson <david(a)gibson.dropbear.id.au>
---
test/Makefile | 8 +-
test/tasst/__main__.py | 2 +-
test/tasst/nstool.py | 170 +++++++++++++++++++++++++++++++++++++++++
test/tasst/snh.py | 16 ++++
4 files changed, 192 insertions(+), 4 deletions(-)
create mode 100644 test/tasst/nstool.py
diff --git a/test/Makefile b/test/Makefile
index 8373ae77..83725f59 100644
--- a/test/Makefile
+++ b/test/Makefile
@@ -64,13 +64,15 @@ LOCAL_ASSETS = mbuto.img mbuto.mem.img podman/bin/podman QEMU_EFI.fd
\
$(TESTDATA_ASSETS)
ASSETS = $(DOWNLOAD_ASSETS) $(LOCAL_ASSETS)
+AVOCADO_ASSETS =
+META_ASSETS = nstool
EXETER_SH = build/static_checkers.sh
EXETER_PY = build/build.py
EXETER_JOBS = $(EXETER_SH:%.sh=%.json) $(EXETER_PY:%.py=%.json)
AVOCADO_JOBS = $(EXETER_JOBS) avocado/static_checkers.json
-TASST_SRCS = __init__.py __main__.py snh.py
+TASST_SRCS = __init__.py __main__.py nstool.py snh.py
EXETER_META = meta/lint.json meta/tasst.json
META_JOBS = $(EXETER_META)
@@ -157,11 +159,11 @@ meta/tasst.json: $(TASST_SRCS:%=tasst/%) $(VENV) pull-exeter
cd ..; PYTHONPATH=$(PYPATH_BASE) $(PYTHON) -m tasst --avocado > test/$@
.PHONY: avocado
-avocado: venv $(AVOCADO_JOBS)
+avocado: venv $(AVOCADO_ASSETS) $(AVOCADO_JOBS)
$(RUN_AVOCADO) all $(AVOCADO_JOBS)
.PHONY: meta
-meta: venv $(META_JOBS)
+meta: venv $(META_ASSETS) $(META_JOBS)
$(RUN_AVOCADO) meta $(META_JOBS)
flake8:
diff --git a/test/tasst/__main__.py b/test/tasst/__main__.py
index 91499128..9fd6174e 100644
--- a/test/tasst/__main__.py
+++ b/test/tasst/__main__.py
@@ -13,7 +13,7 @@ library of test helpers for passt & pasta
import exeter
# We import just to get the exeter tests, which flake8 can't see
-from . import snh # noqa: F401
+from . import nstool, snh # noqa: F401
if __name__ == '__main__':
diff --git a/test/tasst/nstool.py b/test/tasst/nstool.py
new file mode 100644
index 00000000..0b23fbfb
--- /dev/null
+++ b/test/tasst/nstool.py
@@ -0,0 +1,170 @@
+#! /usr/bin/env python3
+
+# SPDX-License-Identifier: GPL-2.0-or-later
+#
+# Copyright Red Hat
+# Author: David Gibson <david(a)gibson.dropbear.id.au>
+
+"""
+Test A Simple Socket Transport
+
+nstool.py - Run commands in namespaces via 'nstool'
+"""
+
+import contextlib
+import os
+import subprocess
+import tempfile
+
+import exeter
+
+from .snh import RealHost, SimNetHost
+
+# FIXME: Can this be made more portable?
+UNIX_PATH_MAX = 108
+
+NSTOOL_BIN = 'test/nstool'
+
+
+class NsTool(SimNetHost):
+ """A bundle of Linux namespaces managed by nstool"""
+
+ def __init__(self, name, sockpath, parent=RealHost()):
+ if len(sockpath) > UNIX_PATH_MAX:
+ raise ValueError(
+ f'Unix domain socket path "{sockpath}" is too long'
+ )
+
+ super().__init__(name)
+ self.sockpath = sockpath
+ self.parent = parent
+ self._pid = None
+
+ def __enter__(self):
+ cmd = [f'{NSTOOL_BIN}', 'info', '-wp',
f'{self.sockpath}']
+ pid = self.parent.output(*cmd, timeout=1)
+ self._pid = int(pid)
+ return self
+
+ def __exit__(self, *exc_details):
+ pass
+
+ # PID of the nstool hold process as seen by the parent snh
+ def pid(self):
+ return self._pid
+
+ # PID of the nstool hold process as seen by another snh which can
+ # see the nstool socket (important when using PID namespaces)
+ def relative_pid(self, relative_to):
+ cmd = [f'{NSTOOL_BIN}', 'info', '-p',
f'{self.sockpath}']
+ relpid = relative_to.output(*cmd)
+ return int(relpid)
+
+ def hostify(self, *cmd, capable=False, **kwargs):
+ hostcmd = [f'{NSTOOL_BIN}', 'exec']
+ if capable:
+ hostcmd.append('--keep-caps')
+ hostcmd += [self.sockpath, '--']
+ hostcmd += list(cmd)
+ return hostcmd, kwargs
+
+
+(a)contextlib.contextmanager
+def unshare_snh(name, *opts, parent=RealHost(), capable=False):
+ # Create path for temporary nstool Unix socket
+ with tempfile.TemporaryDirectory() as tmpd:
+ sockpath = os.path.join(tmpd, name)
+ cmd = ['unshare'] + list(opts)
+ cmd += ['--', f'{NSTOOL_BIN}', 'hold',
f'{sockpath}']
+ with parent.bg(*cmd, capable=capable) as holder:
+ try:
+ with NsTool(name, sockpath, parent=parent) as snh:
+ yield snh
+ finally:
+ try:
+ parent.fg(f'{NSTOOL_BIN}', 'stop',
f'{sockpath}')
+ finally:
+ try:
+ holder.run(timeout=0.1)
+ holder.kill()
+ finally:
+ try:
+ os.remove(sockpath)
+ except FileNotFoundError:
+ pass
+
+
+TEST_EXC = ValueError
+
+
+def test_sockdir_cleanup(s):
+ def mess(sockpaths):
+ with s as snh:
+ ns = snh
+ while isinstance(ns, NsTool):
+ sockpaths.append(ns.sockpath)
+ ns = ns.parent
+ raise TEST_EXC
+
+ sockpaths = []
+ exeter.assert_raises(TEST_EXC, mess, sockpaths)
+ assert sockpaths
+ for path in sockpaths:
+ assert not os.path.exists(os.path.dirname(path))
+
+
+def userns_snh():
+ return unshare_snh('usernetns', '-Ucn')
+
+
+(a)exeter.test
+def test_userns():
+ cmd = ['capsh', '--has-p=CAP_SETUID']
+ with RealHost() as realhost:
+ status = realhost.fg(*cmd, check=False)
+ assert status.returncode != 0
+ with userns_snh() as ns:
+ ns.fg(*cmd, capable=True)
+
+
+(a)contextlib.contextmanager
+def nested_snh():
+ with unshare_snh('userns', '-Uc') as userns:
+ with unshare_snh('netns', '-n', parent=userns, capable=True)
as netns:
+ yield netns
+
+
+def pidns_snh():
+ return unshare_snh('pidns', '-Upfn')
+
+
+(a)exeter.test
+def test_relative_pid():
+ with pidns_snh() as snh:
+ # The holder is init (pid 1) within its own pidns
+ exeter.assert_eq(snh.relative_pid(snh), 1)
+
+
+# General tests for all the nstool examples
+for setup in [userns_snh, nested_snh, pidns_snh]:
+ # Common snh tests
+ SimNetHost.selftest_isolated(setup)
+ exeter.register_pipe(f'{setup.__qualname__}|test_sockdir_cleanup',
+ setup, test_sockdir_cleanup)
+
+
+(a)contextlib.contextmanager
+def connect_snh():
+ with tempfile.TemporaryDirectory() as tmpd:
+ sockpath = os.path.join(tmpd, 'nons')
+ holdcmd = [f'{NSTOOL_BIN}', 'hold', f'{sockpath}']
+ with subprocess.Popen(holdcmd) as holder:
+ try:
+ with NsTool("fakens", sockpath) as snh:
+ yield snh
+ finally:
+ holder.kill()
+ os.remove(sockpath)
+
+
+SimNetHost.selftest(connect_snh)
diff --git a/test/tasst/snh.py b/test/tasst/snh.py
index 8ee9802a..598ea979 100644
--- a/test/tasst/snh.py
+++ b/test/tasst/snh.py
@@ -178,6 +178,22 @@ class SimNetHost(contextlib.AbstractContextManager):
testid = f'{setup.__qualname__}|{t.__qualname__}'
exeter.register_pipe(testid, setup, t)
+ # Additional tests only valid if the snh is isolated (no outside
+ # network connections)
+ def test_is_isolated(self):
+ with self as snh:
+ exeter.assert_eq(snh.ifs(), ['lo'])
+
+ ISOLATED_SELFTESTS = [test_is_isolated]
+
+ @classmethod
+ def selftest_isolated(cls, setup):
+ "Register self tests for an isolated snh example"
+ cls.selftest(setup)
+ for t in cls.ISOLATED_SELFTESTS:
+ testid = f'{setup.__qualname__}|{t.__qualname__}'
+ exeter.register_pipe(testid, setup, t)
+
class RealHost(SimNetHost):
"""Represents the host on which the tests are running (as opposed
--
2.45.2
Add a helper to bring network interfaces up on an snh, and to retrieve
configured IP addresses.
Signed-off-by: David Gibson <david(a)gibson.dropbear.id.au>
---
test/tasst/nstool.py | 11 +++++++++--
test/tasst/snh.py | 28 +++++++++++++++++++++++++++-
2 files changed, 36 insertions(+), 3 deletions(-)
diff --git a/test/tasst/nstool.py b/test/tasst/nstool.py
index 0b23fbfb..d852d81e 100644
--- a/test/tasst/nstool.py
+++ b/test/tasst/nstool.py
@@ -113,8 +113,11 @@ def test_sockdir_cleanup(s):
assert not os.path.exists(os.path.dirname(path))
+(a)contextlib.contextmanager
def userns_snh():
- return unshare_snh('usernetns', '-Ucn')
+ with unshare_snh('usernetns', '-Ucn') as ns:
+ ns.ifup('lo')
+ yield ns
@exeter.test
@@ -131,11 +134,15 @@ def test_userns():
def nested_snh():
with unshare_snh('userns', '-Uc') as userns:
with unshare_snh('netns', '-n', parent=userns, capable=True)
as netns:
+ netns.ifup('lo')
yield netns
+(a)contextlib.contextmanager
def pidns_snh():
- return unshare_snh('pidns', '-Upfn')
+ with unshare_snh('pidns', '-Upfn') as ns:
+ ns.ifup('lo')
+ yield ns
@exeter.test
diff --git a/test/tasst/snh.py b/test/tasst/snh.py
index 598ea979..fd8f6f13 100644
--- a/test/tasst/snh.py
+++ b/test/tasst/snh.py
@@ -13,6 +13,7 @@ tasst/snh.py - Simulated network hosts for testing
import contextlib
+import ipaddress
import json
import subprocess
import sys
@@ -110,6 +111,25 @@ class SimNetHost(contextlib.AbstractContextManager):
info = json.loads(self.output('ip', '-j', 'link',
'show'))
return [i['ifname'] for i in info]
+ def ifup(self, ifname):
+ self.fg('ip', 'link', 'set', f'{ifname}',
'up', capable=True)
+
+ def addrinfos(self, ifname, **criteria):
+ info = json.loads(self.output('ip', '-j', 'addr',
'show', f'{ifname}'))
+ assert len(info) == 1 # We specified a specific interface
+
+ ais = list(ai for ai in info[0]['addr_info'])
+ for key, value in criteria.items():
+ ais = [ai for ai in ais if key in ai and ai[key] == value]
+
+ return ais
+
+ def addrs(self, ifname, **criteria):
+ # Return just the parsed, non-tentative addresses
+ return
[ipaddress.ip_interface(f'{ai["local"]}/{ai["prefixlen"]}')
+ for ai in self.addrinfos(ifname, **criteria)
+ if 'tentative' not in ai]
+
# Internal tests
def test_true(self):
with self as snh:
@@ -166,10 +186,16 @@ class SimNetHost(contextlib.AbstractContextManager):
with self as snh:
assert 'lo' in snh.ifs()
+ def test_lo_addrs(self):
+ expected = set(ipaddress.ip_interface(a)
+ for a in ['127.0.0.1/8', '::1/128'])
+ with self as snh:
+ assert set(snh.addrs('lo')) == expected
+
SELFTESTS = [test_true, test_false, test_echo, test_timeout,
test_bg_true, test_bg_false, test_bg_echo, test_bg_timeout,
test_bg_context_timeout,
- test_has_lo]
+ test_has_lo, test_lo_addrs]
@classmethod
def selftest(cls, setup):
--
2.45.2
Signed-off-by: David Gibson <david(a)gibson.dropbear.id.au>
---
test/Makefile | 3 ++-
test/tasst/__main__.py | 1 +
test/tasst/nstool.py | 9 +++++++++
test/tasst/selftest/__init__.py | 16 ++++++++++++++++
test/tasst/selftest/veth.py | 33 +++++++++++++++++++++++++++++++++
5 files changed, 61 insertions(+), 1 deletion(-)
create mode 100644 test/tasst/selftest/__init__.py
create mode 100644 test/tasst/selftest/veth.py
diff --git a/test/Makefile b/test/Makefile
index 83725f59..e13c49c8 100644
--- a/test/Makefile
+++ b/test/Makefile
@@ -72,7 +72,8 @@ EXETER_PY = build/build.py
EXETER_JOBS = $(EXETER_SH:%.sh=%.json) $(EXETER_PY:%.py=%.json)
AVOCADO_JOBS = $(EXETER_JOBS) avocado/static_checkers.json
-TASST_SRCS = __init__.py __main__.py nstool.py snh.py
+TASST_SRCS = __init__.py __main__.py nstool.py snh.py \
+ selftest/__init__.py selftest/veth.py
EXETER_META = meta/lint.json meta/tasst.json
META_JOBS = $(EXETER_META)
diff --git a/test/tasst/__main__.py b/test/tasst/__main__.py
index 9fd6174e..d52f9c55 100644
--- a/test/tasst/__main__.py
+++ b/test/tasst/__main__.py
@@ -14,6 +14,7 @@ import exeter
# We import just to get the exeter tests, which flake8 can't see
from . import nstool, snh # noqa: F401
+from .selftest import veth # noqa: F401
if __name__ == '__main__':
diff --git a/test/tasst/nstool.py b/test/tasst/nstool.py
index d852d81e..bf0174eb 100644
--- a/test/tasst/nstool.py
+++ b/test/tasst/nstool.py
@@ -68,6 +68,15 @@ class NsTool(SimNetHost):
hostcmd += list(cmd)
return hostcmd, kwargs
+ def veth(self, ifname, peername, peer=None):
+ self.fg('ip', 'link', 'add', f'{ifname}',
'type', 'veth',
+ 'peer', 'name', f'{peername}',
capable=True)
+ if peer is not None:
+ if not isinstance(peer, NsTool):
+ raise TypeError
+ self.fg('ip', 'link', 'set',
f'{peername}',
+ 'netns', f'{peer.relative_pid(self)}',
capable=True)
+
@contextlib.contextmanager
def unshare_snh(name, *opts, parent=RealHost(), capable=False):
diff --git a/test/tasst/selftest/__init__.py b/test/tasst/selftest/__init__.py
new file mode 100644
index 00000000..d7742930
--- /dev/null
+++ b/test/tasst/selftest/__init__.py
@@ -0,0 +1,16 @@
+#! /usr/bin/python3
+
+# SPDX-License-Identifier: GPL-2.0-or-later
+#
+# Copyright Red Hat
+# Author: David Gibson <david(a)gibson.dropbear.id.au>
+
+"""Test A Simple Socket Transport
+
+selftest/ - Selftests for the tasst library
+
+Usually, tests for the tasst helper library itself should go next to
+the implementation of the thing being tested. Sometimes that's
+inconvenient or impossible (usually because it would cause a circular
+module dependency). In that case those tests can go here.
+"""
diff --git a/test/tasst/selftest/veth.py b/test/tasst/selftest/veth.py
new file mode 100644
index 00000000..3c0b3f5b
--- /dev/null
+++ b/test/tasst/selftest/veth.py
@@ -0,0 +1,33 @@
+#! /usr/bin/env python3
+
+# SPDX-License-Identifier: GPL-2.0-or-later
+#
+# Copyright Red Hat
+# Author: David Gibson <david(a)gibson.dropbear.id.au>
+
+"""
+Test A Simple Socket Transport
+
+selftest/veth.py - Test various veth configurations
+"""
+
+import contextlib
+
+import exeter
+
+from tasst import nstool
+
+
+(a)contextlib.contextmanager
+def unconfigured_veth():
+ with nstool.unshare_snh('ns1', '-Un') as ns1:
+ with nstool.unshare_snh('ns2', '-n', parent=ns1,
capable=True) as ns2:
+ ns1.veth('veth1', 'veth2', ns2)
+ yield (ns1, ns2)
+
+
+(a)exeter.test
+def test_ifs():
+ with unconfigured_veth() as (ns1, ns2):
+ exeter.assert_eq(set(ns1.ifs()), set(['lo', 'veth1']))
+ exeter.assert_eq(set(ns2.ifs()), set(['lo', 'veth2']))
--
2.45.2
Signed-off-by: David Gibson <david(a)gibson.dropbear.id.au>
---
test/tasst/selftest/veth.py | 7 +++++++
test/tasst/snh.py | 11 ++++++++++-
2 files changed, 17 insertions(+), 1 deletion(-)
diff --git a/test/tasst/selftest/veth.py b/test/tasst/selftest/veth.py
index 3c0b3f5b..5c8f0c0b 100644
--- a/test/tasst/selftest/veth.py
+++ b/test/tasst/selftest/veth.py
@@ -31,3 +31,10 @@ def test_ifs():
with unconfigured_veth() as (ns1, ns2):
exeter.assert_eq(set(ns1.ifs()), set(['lo', 'veth1']))
exeter.assert_eq(set(ns2.ifs()), set(['lo', 'veth2']))
+
+
+(a)exeter.test
+def test_mtu():
+ with unconfigured_veth() as (ns1, ns2):
+ exeter.assert_eq(ns1.mtu('veth1'), 1500)
+ exeter.assert_eq(ns2.mtu('veth2'), 1500)
diff --git a/test/tasst/snh.py b/test/tasst/snh.py
index fd8f6f13..0554fbd0 100644
--- a/test/tasst/snh.py
+++ b/test/tasst/snh.py
@@ -130,6 +130,11 @@ class SimNetHost(contextlib.AbstractContextManager):
for ai in self.addrinfos(ifname, **criteria)
if 'tentative' not in ai]
+ def mtu(self, ifname):
+ cmd = ['ip', '-j', 'link', 'show',
f'{ifname}']
+ (info,) = json.loads(self.output(*cmd))
+ return info['mtu']
+
# Internal tests
def test_true(self):
with self as snh:
@@ -192,10 +197,14 @@ class SimNetHost(contextlib.AbstractContextManager):
with self as snh:
assert set(snh.addrs('lo')) == expected
+ def test_lo_mtu(self):
+ with self as snh:
+ exeter.assert_eq(snh.mtu('lo'), 65536)
+
SELFTESTS = [test_true, test_false, test_echo, test_timeout,
test_bg_true, test_bg_false, test_bg_echo, test_bg_timeout,
test_bg_context_timeout,
- test_has_lo, test_lo_addrs]
+ test_has_lo, test_lo_addrs, test_lo_mtu]
@classmethod
def selftest(cls, setup):
--
2.45.2
Add a helper to the Site() class to wait for an address with specified
characteristics to be ready on an interface. In particular this is useful
for waiting for IPv6 SLAAC & DAD (Duplicate Address Detection) to complete.
Because DAD is not going to be useful in many of our scenarios, also extend
Site.ifup() to allow DAD to be switched to optimistic mode or disabled.
Signed-off-by: David Gibson <david(a)gibson.dropbear.id.au>
---
test/Makefile | 2 +-
test/tasst/__main__.py | 2 +-
test/tasst/selftest/static_ifup.py | 40 ++++++++++++++++++++++++++++++
test/tasst/selftest/veth.py | 27 ++++++++++++++++++++
test/tasst/snh.py | 24 +++++++++++++++++-
5 files changed, 92 insertions(+), 3 deletions(-)
create mode 100644 test/tasst/selftest/static_ifup.py
diff --git a/test/Makefile b/test/Makefile
index e13c49c8..139a0b14 100644
--- a/test/Makefile
+++ b/test/Makefile
@@ -73,7 +73,7 @@ EXETER_JOBS = $(EXETER_SH:%.sh=%.json) $(EXETER_PY:%.py=%.json)
AVOCADO_JOBS = $(EXETER_JOBS) avocado/static_checkers.json
TASST_SRCS = __init__.py __main__.py nstool.py snh.py \
- selftest/__init__.py selftest/veth.py
+ selftest/__init__.py selftest/static_ifup.py selftest/veth.py
EXETER_META = meta/lint.json meta/tasst.json
META_JOBS = $(EXETER_META)
diff --git a/test/tasst/__main__.py b/test/tasst/__main__.py
index d52f9c55..f3f88424 100644
--- a/test/tasst/__main__.py
+++ b/test/tasst/__main__.py
@@ -14,7 +14,7 @@ import exeter
# We import just to get the exeter tests, which flake8 can't see
from . import nstool, snh # noqa: F401
-from .selftest import veth # noqa: F401
+from .selftest import static_ifup, veth # noqa: F401
if __name__ == '__main__':
diff --git a/test/tasst/selftest/static_ifup.py b/test/tasst/selftest/static_ifup.py
new file mode 100644
index 00000000..0c6375d4
--- /dev/null
+++ b/test/tasst/selftest/static_ifup.py
@@ -0,0 +1,40 @@
+#! /usr/bin/env python3
+
+# SPDX-License-Identifier: GPL-2.0-or-later
+#
+# Copyright Red Hat
+# Author: David Gibson <david(a)gibson.dropbear.id.au>
+
+"""
+Test A Simple Socket Transport
+
+meta/static_ifup - Static address configuration
+"""
+
+import contextlib
+import ipaddress
+
+import exeter
+
+from tasst import nstool
+
+
+IFNAME = 'testveth'
+IFNAME_PEER = 'vethpeer'
+TEST_IPS = set([ipaddress.ip_interface('192.0.2.1/24'),
+ ipaddress.ip_interface('2001:db8:9a55::1/112'),
+ ipaddress.ip_interface('10.1.2.3/8')])
+
+
+(a)contextlib.contextmanager
+def setup_ns():
+ with nstool.unshare_snh('ns', '-Un') as ns:
+ ns.veth(IFNAME, IFNAME_PEER)
+ ns.ifup(IFNAME, *TEST_IPS, dad='disable')
+ yield ns
+
+
+(a)exeter.test
+def test_addr():
+ with setup_ns() as ns:
+ exeter.assert_eq(set(ns.addrs(IFNAME, scope='global')), TEST_IPS)
diff --git a/test/tasst/selftest/veth.py b/test/tasst/selftest/veth.py
index 5c8f0c0b..24bbdc27 100644
--- a/test/tasst/selftest/veth.py
+++ b/test/tasst/selftest/veth.py
@@ -12,6 +12,7 @@ selftest/veth.py - Test various veth configurations
"""
import contextlib
+import ipaddress
import exeter
@@ -38,3 +39,29 @@ def test_mtu():
with unconfigured_veth() as (ns1, ns2):
exeter.assert_eq(ns1.mtu('veth1'), 1500)
exeter.assert_eq(ns2.mtu('veth2'), 1500)
+
+
+(a)exeter.test
+def test_slaac(dad=None):
+ TESTMAC = '02:aa:bb:cc:dd:ee'
+ TESTIP = ipaddress.ip_interface('fe80::aa:bbff:fecc:ddee/64')
+
+ with unconfigured_veth() as (ns1, ns2):
+ ns1.fg('ip', 'link', 'set', 'dev',
'veth1', 'address', f'{TESTMAC}',
+ capable=True)
+
+ ns1.ifup('veth1', dad=dad)
+ ns2.ifup('veth2')
+
+ addrs = ns1.addr_wait('veth1', family='inet6',
scope='link')
+ exeter.assert_eq(addrs, [TESTIP])
+
+
+(a)exeter.test
+def test_optimistic_dad():
+ test_slaac(dad='optimistic')
+
+
+(a)exeter.test
+def test_no_dad():
+ test_slaac(dad='disable')
diff --git a/test/tasst/snh.py b/test/tasst/snh.py
index 0554fbd0..a1225ff0 100644
--- a/test/tasst/snh.py
+++ b/test/tasst/snh.py
@@ -111,7 +111,23 @@ class SimNetHost(contextlib.AbstractContextManager):
info = json.loads(self.output('ip', '-j', 'link',
'show'))
return [i['ifname'] for i in info]
- def ifup(self, ifname):
+ def ifup(self, ifname, *addrs, dad=None):
+ if dad == 'disable':
+ self.fg('sysctl',
f'net.ipv6.conf.{ifname}.accept_dad=0',
+ capable=True)
+ elif dad == 'optimistic':
+ self.fg('sysctl',
f'net.ipv6.conf.{ifname}.optimistic_dad=1',
+ capable=True)
+ elif dad is not None:
+ raise ValueError
+
+ for a in addrs:
+ if not isinstance(a, ipaddress.IPv4Interface) \
+ and not isinstance(a, ipaddress.IPv6Interface):
+ raise TypeError
+ self.fg('ip', 'addr', 'add',
f'{a.with_prefixlen}',
+ 'dev', f'{ifname}', capable=True)
+
self.fg('ip', 'link', 'set', f'{ifname}',
'up', capable=True)
def addrinfos(self, ifname, **criteria):
@@ -135,6 +151,12 @@ class SimNetHost(contextlib.AbstractContextManager):
(info,) = json.loads(self.output(*cmd))
return info['mtu']
+ def addr_wait(self, ifname, **criteria):
+ while True:
+ addrs = self.addrs(ifname, **criteria)
+ if addrs:
+ return addrs
+
# Internal tests
def test_true(self):
with self as snh:
--
2.45.2
Signed-off-by: David Gibson <david(a)gibson.dropbear.id.au>
---
test/tasst/selftest/static_ifup.py | 20 ++++++++++++++++++++
test/tasst/snh.py | 13 +++++++++++++
2 files changed, 33 insertions(+)
diff --git a/test/tasst/selftest/static_ifup.py b/test/tasst/selftest/static_ifup.py
index 0c6375d4..2627b579 100644
--- a/test/tasst/selftest/static_ifup.py
+++ b/test/tasst/selftest/static_ifup.py
@@ -38,3 +38,23 @@ def setup_ns():
def test_addr():
with setup_ns() as ns:
exeter.assert_eq(set(ns.addrs(IFNAME, scope='global')), TEST_IPS)
+
+
+(a)exeter.test
+def test_routes4():
+ with setup_ns() as ns:
+ expected_routes = set(i.network for i in TEST_IPS
+ if isinstance(i, ipaddress.IPv4Interface))
+ actual_routes = set(ipaddress.ip_interface(r['dst']).network
+ for r in ns.routes4(dev=IFNAME))
+ exeter.assert_eq(expected_routes, actual_routes)
+
+
+(a)exeter.test
+def test_routes6():
+ with setup_ns() as ns:
+ expected_routes = set(i.network for i in TEST_IPS
+ if isinstance(i, ipaddress.IPv6Interface))
+ actual_routes = set(ipaddress.ip_interface(r['dst']).network
+ for r in ns.routes6(dev=IFNAME))
+ exeter.assert_eq(expected_routes, actual_routes)
diff --git a/test/tasst/snh.py b/test/tasst/snh.py
index a1225ff0..4ddcbb16 100644
--- a/test/tasst/snh.py
+++ b/test/tasst/snh.py
@@ -157,6 +157,19 @@ class SimNetHost(contextlib.AbstractContextManager):
if addrs:
return addrs
+ def _routes(self, ipv, **criteria):
+ routes = json.loads(self.output('ip', '-j',
f'-{ipv}', 'route'))
+ for key, value in criteria.items():
+ routes = [r for r in routes if key in r and r[key] == value]
+
+ return routes
+
+ def routes4(self, **criteria):
+ return self._routes('4', **criteria)
+
+ def routes6(self, **criteria):
+ return self._routes('6', **criteria)
+
# Internal tests
def test_true(self):
with self as snh:
--
2.45.2
Many of our existing tests are based on using socat to transfer between
various locations connected via pasta or passt. Add helpers to make
avocado tests performing similar transfers. Add selftests to verify those
work as expected when we don't have pasta or passt involved yet.
Signed-off-by: David Gibson <david(a)gibson.dropbear.id.au>
---
test/Makefile | 4 +-
test/tasst/__main__.py | 2 +-
test/tasst/transfer.py | 194 +++++++++++++++++++++++++++++++++++++++++
3 files changed, 197 insertions(+), 3 deletions(-)
create mode 100644 test/tasst/transfer.py
diff --git a/test/Makefile b/test/Makefile
index 139a0b14..584f56e9 100644
--- a/test/Makefile
+++ b/test/Makefile
@@ -65,14 +65,14 @@ LOCAL_ASSETS = mbuto.img mbuto.mem.img podman/bin/podman QEMU_EFI.fd
\
ASSETS = $(DOWNLOAD_ASSETS) $(LOCAL_ASSETS)
AVOCADO_ASSETS =
-META_ASSETS = nstool
+META_ASSETS = nstool small.bin medium.bin big.bin
EXETER_SH = build/static_checkers.sh
EXETER_PY = build/build.py
EXETER_JOBS = $(EXETER_SH:%.sh=%.json) $(EXETER_PY:%.py=%.json)
AVOCADO_JOBS = $(EXETER_JOBS) avocado/static_checkers.json
-TASST_SRCS = __init__.py __main__.py nstool.py snh.py \
+TASST_SRCS = __init__.py __main__.py nstool.py snh.py transfer.py \
selftest/__init__.py selftest/static_ifup.py selftest/veth.py
EXETER_META = meta/lint.json meta/tasst.json
diff --git a/test/tasst/__main__.py b/test/tasst/__main__.py
index f3f88424..98a94011 100644
--- a/test/tasst/__main__.py
+++ b/test/tasst/__main__.py
@@ -13,7 +13,7 @@ library of test helpers for passt & pasta
import exeter
# We import just to get the exeter tests, which flake8 can't see
-from . import nstool, snh # noqa: F401
+from . import nstool, snh, transfer # noqa: F401
from .selftest import static_ifup, veth # noqa: F401
diff --git a/test/tasst/transfer.py b/test/tasst/transfer.py
new file mode 100644
index 00000000..be3eebc2
--- /dev/null
+++ b/test/tasst/transfer.py
@@ -0,0 +1,194 @@
+#! /usr/bin/env python3
+
+# SPDX-License-Identifier: GPL-2.0-or-later
+#
+# Copyright Red Hat
+# Author: David Gibson <david(a)gibson.dropbear.id.au>
+
+"""
+Test A Simple Socket Transport
+
+transfer.py - Helpers for testing data transfers
+"""
+
+import contextlib
+from ipaddress import IPv4Address, IPv6Address
+import time
+
+import exeter
+
+from . import nstool, snh
+
+
+# HACK: how long to wait for the server to be ready and listening (s)
+SERVER_READY_DELAY = 0.1 # 1/10th of a second
+
+
+# socat needs IPv6 addresses in square brackets
+def socat_ip(ip):
+ if isinstance(ip, IPv6Address):
+ return f'[{ip}]'
+ if isinstance(ip, IPv4Address):
+ return f'{ip}'
+ raise TypeError
+
+
+def socat_upload(datafile, csnh, ssnh, connect, listen):
+ srcdata = csnh.output('cat', f'{datafile}')
+ with ssnh.bg('socat', '-u', f'{listen}',
'STDOUT',
+ capture=snh.STDOUT) as server:
+ time.sleep(SERVER_READY_DELAY)
+
+ # Can't use csnh.fg() here, because while we wait for the
+ # client to complete we won't be reading from the output pipe
+ # of the server, meaning it will freeze once the buffers fill
+ with csnh.bg('socat', '-u', f'OPEN:{datafile}',
f'{connect}') \
+ as client:
+ res = server.run()
+ client.run()
+ exeter.assert_eq(srcdata, res.stdout)
+
+
+def socat_download(datafile, csnh, ssnh, connect, listen):
+ srcdata = ssnh.output('cat', f'{datafile}')
+ with ssnh.bg('socat', '-u', f'OPEN:{datafile}',
f'{listen}'):
+ time.sleep(SERVER_READY_DELAY)
+ dstdata = csnh.output('socat', '-u', f'{connect}',
'STDOUT')
+ exeter.assert_eq(srcdata, dstdata)
+
+
+def _tcp_socat(connectip, connectport, listenip, listenport, fromip):
+ v6 = isinstance(connectip, IPv6Address)
+ if listenport is None:
+ listenport = connectport
+ if v6:
+ connect = f'TCP6:[{connectip}]:{connectport},ipv6only'
+ listen = f'TCP6-LISTEN:{listenport},ipv6only'
+ else:
+ connect = f'TCP4:{connectip}:{connectport}'
+ listen = f'TCP4-LISTEN:{listenport}'
+ if listenip is not None:
+ listen += f',bind={socat_ip(listenip)}'
+ if fromip is not None:
+ connect += f',bind={socat_ip(fromip)}'
+ return (connect, listen)
+
+
+def tcp_upload(datafile, cs, ss, connectip, connectport,
+ listenip=None, listenport=None, fromip=None):
+ connect, listen = _tcp_socat(connectip, connectport, listenip, listenport,
+ fromip)
+ socat_upload(datafile, cs, ss, connect, listen)
+
+
+def tcp_download(datafile, cs, ss, connectip, connectport,
+ listenip=None, listenport=None, fromip=None):
+ connect, listen = _tcp_socat(connectip, connectport, listenip, listenport,
+ fromip)
+ socat_download(datafile, cs, ss, connect, listen)
+
+
+def udp_transfer(datafile, cs, ss, connectip, connectport,
+ listenip=None, listenport=None, fromip=None):
+ v6 = isinstance(connectip, IPv6Address)
+ if listenport is None:
+ listenport = connectport
+ if v6:
+ connect = f'UDP6:[{connectip}]:{connectport},ipv6only,shut-null'
+ listen = f'UDP6-LISTEN:{listenport},ipv6only,null-eof'
+ else:
+ connect = f'UDP4:{connectip}:{connectport},shut-null'
+ listen = f'UDP4-LISTEN:{listenport},null-eof'
+ if listenip is not None:
+ listen += f',bind={socat_ip(listenip)}'
+ if fromip is not None:
+ connect += f',bind={socat_ip(fromip)}'
+
+ socat_upload(datafile, cs, ss, connect, listen)
+
+
+SMALL_DATA = 'test/small.bin'
+BIG_DATA = 'test/big.bin'
+UDP_DATA = 'test/medium.bin'
+
+
+class TransferTestScenario:
+ def __init__(self, *, client, server, connect_ip, connect_port,
+ listen_ip=None, listen_port=None, from_ip=None):
+ self.client = client
+ self.server = server
+ if isinstance(connect_ip, IPv4Address):
+ self.ip = connect_ip
+ self.listen_ip = listen_ip
+ self.from_ip = from_ip
+ elif isinstance(connect_ip, IPv6Address):
+ self.ip = connect_ip
+ self.listen_ip = listen_ip
+ self.from_ip = from_ip
+ self.port = connect_port
+ self.listen_port = listen_port
+
+
+def test_tcp_upload(setup, datafile=SMALL_DATA):
+ with setup as scn:
+ tcp_upload(datafile, scn.client, scn.server, scn.ip, scn.port,
+ listenip=scn.listen_ip, listenport=scn.listen_port,
+ fromip=scn.from_ip)
+
+
+def test_tcp_big_upload(setup):
+ return test_tcp_upload(setup, datafile=BIG_DATA)
+
+
+def test_tcp_download(setup, datafile=SMALL_DATA):
+ with setup as scn:
+ tcp_download(datafile, scn.client, scn.server, scn.ip, scn.port,
+ listenip=scn.listen_ip, listenport=scn.listen_port,
+ fromip=scn.from_ip)
+
+
+def test_tcp_big_download(setup):
+ return test_tcp_download(setup, datafile=BIG_DATA)
+
+
+def test_udp_transfer(setup, datafile=UDP_DATA):
+ with setup as scn:
+ udp_transfer(datafile, scn.client, scn.server,
+ scn.ip, scn.port,
+ listenip=scn.listen_ip, listenport=scn.listen_port,
+ fromip=scn.from_ip)
+
+
+TRANSFER_TESTS = [test_tcp_upload, test_tcp_big_upload,
+ test_tcp_download, test_tcp_big_download,
+ test_udp_transfer]
+
+
+def transfer_tests(setup):
+ for t in TRANSFER_TESTS:
+ testid = f'{setup.__qualname__}|{t.__qualname__}'
+ exeter.register_pipe(testid, setup, t)
+
+
+(a)contextlib.contextmanager
+def local_transfer4():
+ with nstool.unshare_snh('ns', '-Un') as ns:
+ ns.ifup('lo')
+ yield TransferTestScenario(client=ns, server=ns,
+ connect_ip=IPv4Address('127.0.0.1'),
+ connect_port=10000)
+
+
+transfer_tests(local_transfer4)
+
+
+(a)contextlib.contextmanager
+def local_transfer6():
+ with nstool.unshare_snh('ns', '-Un') as ns:
+ ns.ifup('lo')
+ yield TransferTestScenario(client=ns, server=ns,
+ connect_ip=IPv6Address('::1'),
+ connect_port=10000)
+
+
+transfer_tests(local_transfer6)
--
2.45.2
A bunch of our test scenarious will require us to allocate IPv4 and IPv6
addresses in example networks. Make helpers to do this easily.
Signed-off-by: David Gibson <david(a)gibson.dropbear.id.au>
---
test/Makefile | 2 +-
test/tasst/address.py | 79 +++++++++++++++++++++++++++++++++++++
test/tasst/selftest/veth.py | 41 ++++++++++++++++++-
test/tasst/transfer.py | 6 +--
4 files changed, 123 insertions(+), 5 deletions(-)
create mode 100644 test/tasst/address.py
diff --git a/test/Makefile b/test/Makefile
index 584f56e9..f3a3cc58 100644
--- a/test/Makefile
+++ b/test/Makefile
@@ -72,7 +72,7 @@ EXETER_PY = build/build.py
EXETER_JOBS = $(EXETER_SH:%.sh=%.json) $(EXETER_PY:%.py=%.json)
AVOCADO_JOBS = $(EXETER_JOBS) avocado/static_checkers.json
-TASST_SRCS = __init__.py __main__.py nstool.py snh.py transfer.py \
+TASST_SRCS = __init__.py __main__.py address.py nstool.py snh.py transfer.py \
selftest/__init__.py selftest/static_ifup.py selftest/veth.py
EXETER_META = meta/lint.json meta/tasst.json
diff --git a/test/tasst/address.py b/test/tasst/address.py
new file mode 100644
index 00000000..70899789
--- /dev/null
+++ b/test/tasst/address.py
@@ -0,0 +1,79 @@
+#! /usr/bin/env python3
+
+# SPDX-License-Identifier: GPL-2.0-or-later
+#
+# Copyright Red Hat
+# Author: David Gibson <david(a)gibson.dropbear.id.au>
+
+"""
+Test A Simple Socket Transport
+
+address.py - Address allocation helpers
+"""
+
+import ipaddress
+
+import exeter
+
+# Loopback addresses, for convenience
+LOOPBACK4 = ipaddress.ip_address('127.0.0.1')
+LOOPBACK6 = ipaddress.ip_address('::1')
+
+# Documentation test networks defined by RFC 5737
+TEST_NET_1 = ipaddress.ip_network('192.0.2.0/24')
+TEST_NET_2 = ipaddress.ip_network('198.51.100.0/24')
+TEST_NET_3 = ipaddress.ip_network('203.0.113.0/24')
+
+# Documentation test network defined by RFC 3849
+TEST_NET6 = ipaddress.ip_network('2001:db8::/32')
+# Some subnets of that for our usage
+TEST_NET6_TASST_A = ipaddress.ip_network('2001:db8:9a55:aaaa::/64')
+TEST_NET6_TASST_B = ipaddress.ip_network('2001:db8:9a55:bbbb::/64')
+TEST_NET6_TASST_C = ipaddress.ip_network('2001:db8:9a55:cccc::/64')
+
+
+class IpiAllocator:
+ """IP address allocator"""
+
+ DEFAULT_NETS = [TEST_NET_1, TEST_NET6_TASST_A]
+
+ def __init__(self, *nets):
+ if not nets:
+ nets = self.DEFAULT_NETS
+
+ self.nets = [ipaddress.ip_network(n) for n in nets]
+ self.hostses = [n.hosts() for n in self.nets]
+
+ def next_ipis(self):
+ addrs = [next(h) for h in self.hostses]
+ return [ipaddress.ip_interface(f'{a}/{n.prefixlen}')
+ for a, n in zip(addrs, self.nets)]
+
+
+(a)exeter.test
+def ipa_test(nets=None, count=12):
+ if nets is None:
+ ipa = IpiAllocator()
+ nets = IpiAllocator.DEFAULT_NETS
+ else:
+ ipa = IpiAllocator(*nets)
+
+ addrsets = [set() for i in range(len(nets))]
+ for i in range(count):
+ addrs = ipa.next_ipis()
+ # Check we got as many addresses as expected
+ exeter.assert_eq(len(addrs), len(nets))
+ for s, a, n in zip(addrsets, addrs, nets):
+ # Check the addresses belong to the right network
+ exeter.assert_eq(a.network, ipaddress.ip_network(n))
+ s.add(a)
+
+ print(addrsets)
+ # Check the addresses are unique
+ for s in addrsets:
+ exeter.assert_eq(len(s), count)
+
+
+(a)exeter.test
+def ipa_test_custom():
+ ipa_test(nets=['10.55.0.0/16', '192.168.55.0/24',
'fd00:9a57:a000::/48'])
diff --git a/test/tasst/selftest/veth.py b/test/tasst/selftest/veth.py
index 24bbdc27..39ac947d 100644
--- a/test/tasst/selftest/veth.py
+++ b/test/tasst/selftest/veth.py
@@ -16,7 +16,7 @@ import ipaddress
import exeter
-from tasst import nstool
+from tasst import address, nstool, transfer
@contextlib.contextmanager
@@ -65,3 +65,42 @@ def test_optimistic_dad():
@exeter.test
def test_no_dad():
test_slaac(dad='disable')
+
+
+(a)contextlib.contextmanager
+def configured_veth(ip1, ip2):
+ with unconfigured_veth() as (ns1, ns2):
+ ns1.ifup('lo')
+ ns1.ifup('veth1', ip1, dad='disable')
+
+ ns2.ifup('lo')
+ ns2.ifup('veth2', ip2, dad='disable')
+
+ yield (ns1, ns2)
+
+
+(a)contextlib.contextmanager
+def veth_transfer(ip1, ip2):
+ with configured_veth(ip1, ip2) as (ns1, ns2):
+ yield transfer.TransferTestScenario(client=ns1, server=ns2,
+ connect_ip=ip2.ip,
+ connect_port=10000)
+
+
+ipa = address.IpiAllocator()
+NS1_IP4, NS1_IP6 = ipa.next_ipis()
+NS2_IP4, NS2_IP6 = ipa.next_ipis()
+
+
+def veth_transfer4():
+ return veth_transfer(NS1_IP4, NS2_IP4)
+
+
+transfer.transfer_tests(veth_transfer4)
+
+
+def veth_transfer6():
+ return veth_transfer(NS1_IP6, NS2_IP6)
+
+
+transfer.transfer_tests(veth_transfer6)
diff --git a/test/tasst/transfer.py b/test/tasst/transfer.py
index be3eebc2..a5aa0614 100644
--- a/test/tasst/transfer.py
+++ b/test/tasst/transfer.py
@@ -17,7 +17,7 @@ import time
import exeter
-from . import nstool, snh
+from . import address, nstool, snh
# HACK: how long to wait for the server to be ready and listening (s)
@@ -175,7 +175,7 @@ def local_transfer4():
with nstool.unshare_snh('ns', '-Un') as ns:
ns.ifup('lo')
yield TransferTestScenario(client=ns, server=ns,
- connect_ip=IPv4Address('127.0.0.1'),
+ connect_ip=address.LOOPBACK4,
connect_port=10000)
@@ -187,7 +187,7 @@ def local_transfer6():
with nstool.unshare_snh('ns', '-Un') as ns:
ns.ifup('lo')
yield TransferTestScenario(client=ns, server=ns,
- connect_ip=IPv6Address('::1'),
+ connect_ip=address.LOOPBACK6,
connect_port=10000)
--
2.45.2
Signed-iff-by: David Gibson <david(a)gibson.dropbear.id.au>
---
test/Makefile | 3 +-
test/tasst/__main__.py | 2 +-
test/tasst/ndp.py | 116 +++++++++++++++++++++++++++++++++++++++++
3 files changed, 119 insertions(+), 2 deletions(-)
create mode 100644 test/tasst/ndp.py
diff --git a/test/Makefile b/test/Makefile
index f3a3cc58..248329e5 100644
--- a/test/Makefile
+++ b/test/Makefile
@@ -72,7 +72,8 @@ EXETER_PY = build/build.py
EXETER_JOBS = $(EXETER_SH:%.sh=%.json) $(EXETER_PY:%.py=%.json)
AVOCADO_JOBS = $(EXETER_JOBS) avocado/static_checkers.json
-TASST_SRCS = __init__.py __main__.py address.py nstool.py snh.py transfer.py \
+TASST_SRCS = __init__.py __main__.py address.py ndp.py nstool.py snh.py \
+ transfer.py \
selftest/__init__.py selftest/static_ifup.py selftest/veth.py
EXETER_META = meta/lint.json meta/tasst.json
diff --git a/test/tasst/__main__.py b/test/tasst/__main__.py
index 98a94011..6a95eec1 100644
--- a/test/tasst/__main__.py
+++ b/test/tasst/__main__.py
@@ -13,7 +13,7 @@ library of test helpers for passt & pasta
import exeter
# We import just to get the exeter tests, which flake8 can't see
-from . import nstool, snh, transfer # noqa: F401
+from . import ndp, nstool, snh, transfer # noqa: F401
from .selftest import static_ifup, veth # noqa: F401
diff --git a/test/tasst/ndp.py b/test/tasst/ndp.py
new file mode 100644
index 00000000..1c18385c
--- /dev/null
+++ b/test/tasst/ndp.py
@@ -0,0 +1,116 @@
+#! /usr/bin/env avocado-runner-avocado-classless
+
+# SPDX-License-Identifier: GPL-2.0-or-later
+#
+# Copyright Red Hat
+# Author: David Gibson <david(a)gibson.dropbear.id.au>
+
+"""
+Test A Simple Socket Transport
+
+ndp.py - Helpers for testing NDP
+"""
+
+import contextlib
+import ipaddress
+import os
+import tempfile
+
+import exeter
+
+from . import address, nstool
+
+
+class NdpTestScenario:
+ def __init__(self, *, client, ifname, network, gateway):
+ self.client = client
+ self.ifname = ifname
+ self.network = network
+ self.gateway = gateway
+
+
+def test_ndp_addr(setup):
+ with setup as scn:
+ # Wait for NDP to do its thing
+ (addr,) = scn.client.addr_wait(scn.ifname, family='inet6',
+ scope='global')
+
+ # The SLAAC address is derived from the guest ns MAC, so
+ # probably won't exactly match the host address (we need
+ # DHCPv6 for that). It should be in the right network though.
+ exeter.assert_eq(addr.network, scn.network)
+
+
+def test_ndp_route(setup):
+ with setup as scn:
+ defroutes = scn.client.routes6(dst='default')
+ while not defroutes:
+ defroutes = scn.client.routes6(dst='default')
+
+ exeter.assert_eq(len(defroutes), 1)
+ gw = ipaddress.ip_address(defroutes[0]['gateway'])
+ exeter.assert_eq(gw, scn.gateway)
+
+
+NDP_TESTS = [test_ndp_addr, test_ndp_route]
+
+
+def ndp_tests(setup):
+ for t in NDP_TESTS:
+ testid = f'{setup.__qualname__}|{t.__qualname__}'
+ exeter.register_pipe(testid, setup, t)
+
+
+IFNAME = 'clientif'
+NETWORK = address.TEST_NET6_TASST_A
+ipa = address.IpiAllocator(NETWORK)
+(ROUTER_IP6,) = ipa.next_ipis()
+
+
+(a)contextlib.contextmanager
+def setup_radvd():
+ router_ifname = 'routerif'
+
+ with nstool.unshare_snh('client', '-Un') as client, \
+ nstool.unshare_snh('router', '-n',
+ parent=client, capable=True) as router:
+ with tempfile.TemporaryDirectory() as tmpdir:
+ client.veth(IFNAME, router_ifname, router)
+
+ # Configure the simulated router
+ confpath = os.path.join(tmpdir, 'radvd.conf')
+ pidfile = os.path.join(tmpdir, 'radvd.pid')
+ open(confpath, 'w', encoding='UTF-8').write(
+ f'''
+ interface {router_ifname} {{
+ AdvSendAdvert on;
+ prefix {NETWORK} {{
+ }};
+ }};
+ '''
+ )
+
+ router.ifup('lo')
+ router.ifup('routerif', ROUTER_IP6)
+
+ # Configure the client
+ client.ifup('lo')
+ client.ifup(IFNAME)
+
+ # Get the router's link-local-address
+ (router_ll,) = router.addr_wait(router_ifname,
+ family='inet6',
scope='link')
+
+ # Run radvd
+ router.fg('radvd', '-c', '-C',
f'{confpath}')
+ radvd_cmd = ['radvd', '-C', f'{confpath}',
'-n',
+ '-p', f'{pidfile}', '-d',
'5']
+ with router.bg(*radvd_cmd, capable=True) as radvd:
+ yield NdpTestScenario(client=client,
+ ifname=IFNAME,
+ network=NETWORK,
+ gateway=router_ll.ip)
+ radvd.terminate()
+
+
+ndp_tests(setup_radvd)
--
2.45.2
Signed-off-by: David Gibson <david(a)gibson.dropbear.id.au>
---
test/Makefile | 4 +-
test/tasst/__main__.py | 2 +-
test/tasst/dhcp.py | 132 +++++++++++++++++++++++++++++++++++++++++
test/tasst/dhcpv6.py | 89 +++++++++++++++++++++++++++
4 files changed, 224 insertions(+), 3 deletions(-)
create mode 100644 test/tasst/dhcp.py
create mode 100644 test/tasst/dhcpv6.py
diff --git a/test/Makefile b/test/Makefile
index 248329e5..0eeaf82e 100644
--- a/test/Makefile
+++ b/test/Makefile
@@ -72,8 +72,8 @@ EXETER_PY = build/build.py
EXETER_JOBS = $(EXETER_SH:%.sh=%.json) $(EXETER_PY:%.py=%.json)
AVOCADO_JOBS = $(EXETER_JOBS) avocado/static_checkers.json
-TASST_SRCS = __init__.py __main__.py address.py ndp.py nstool.py snh.py \
- transfer.py \
+TASST_SRCS = __init__.py __main__.py address.py dhcp.py dhcpv6.py ndp.py \
+ nstool.py snh.py transfer.py \
selftest/__init__.py selftest/static_ifup.py selftest/veth.py
EXETER_META = meta/lint.json meta/tasst.json
diff --git a/test/tasst/__main__.py b/test/tasst/__main__.py
index 6a95eec1..8c4efd74 100644
--- a/test/tasst/__main__.py
+++ b/test/tasst/__main__.py
@@ -13,7 +13,7 @@ library of test helpers for passt & pasta
import exeter
# We import just to get the exeter tests, which flake8 can't see
-from . import ndp, nstool, snh, transfer # noqa: F401
+from . import dhcp, dhcpv6, ndp, nstool, snh, transfer # noqa: F401
from .selftest import static_ifup, veth # noqa: F401
diff --git a/test/tasst/dhcp.py b/test/tasst/dhcp.py
new file mode 100644
index 00000000..d86df2de
--- /dev/null
+++ b/test/tasst/dhcp.py
@@ -0,0 +1,132 @@
+#! /usr/bin/env avocado-runner-avocado-classless
+
+# SPDX-License-Identifier: GPL-2.0-or-later
+#
+# Copyright Red Hat
+# Author: David Gibson <david(a)gibson.dropbear.id.au>
+
+"""
+Test A Simple Socket Transport
+
+dhcp.py - Helpers for testing DHCP
+"""
+
+import contextlib
+import ipaddress
+import os
+import tempfile
+
+import exeter
+
+from . import address, nstool
+
+
+DHCLIENT = '/sbin/dhclient'
+
+
+(a)contextlib.contextmanager
+def dhclient(snh, ifname, ipv='4'):
+ with tempfile.TemporaryDirectory() as tmpdir:
+ pidfile = os.path.join(tmpdir, 'dhclient.pid')
+ leasefile = os.path.join(tmpdir, 'dhclient.leases')
+
+ # We need '-nc' because we may be running with
+ # capabilities but not UID 0. Without -nc dhclient drops
+ # capabilities before invoking dhclient-script, so it's
+ # unable to actually configure the interface
+ opts = [f'-{ipv}', '-v', '-nc', '-pf',
f'{pidfile}',
+ '-lf', f'{leasefile}', f'{ifname}']
+ snh.fg(f'{DHCLIENT}', *opts, capable=True)
+ yield
+ snh.fg(f'{DHCLIENT}', '-x', '-pf',
f'{pidfile}', capable=True)
+
+
+class DhcpTestScenario:
+ def __init__(self, *, client, ifname, addr, gateway, mtu):
+ self.client = client
+ self.ifname = ifname
+ self.addr = addr
+ self.gateway = gateway
+ self.mtu = mtu
+
+
+def test_dhcp_addr(setup):
+ with setup as scn, dhclient(scn.client, scn.ifname):
+ (actual_addr,) = scn.client.addrs(scn.ifname,
+ family='inet',
scope='global')
+ exeter.assert_eq(actual_addr.ip, scn.addr)
+
+
+def test_dhcp_route(setup):
+ with setup as scn, dhclient(scn.client, scn.ifname):
+ (defroute,) = scn.client.routes4(dst='default')
+ exeter.assert_eq(ipaddress.ip_address(defroute['gateway']),
+ scn.gateway)
+
+
+def test_dhcp_mtu(setup):
+ with setup as scn, dhclient(scn.client, scn.ifname):
+ exeter.assert_eq(scn.client.mtu(scn.ifname), scn.mtu)
+
+
+DHCP_TESTS = [test_dhcp_addr, test_dhcp_route, test_dhcp_mtu]
+
+
+def dhcp_tests(setup):
+ for t in DHCP_TESTS:
+ testid = f'{setup.__qualname__}|{t.__qualname__}'
+ exeter.register_pipe(testid, setup, t)
+
+
+DHCPD = 'dhcpd'
+SUBNET = address.TEST_NET_1
+ipa = address.IpiAllocator(SUBNET)
+(SERVER_IP4,) = ipa.next_ipis()
+(CLIENT_IP4,) = ipa.next_ipis()
+IFNAME = 'clientif'
+
+
+(a)contextlib.contextmanager
+def setup_dhcpd_common(ifname, server_ifname):
+ with nstool.unshare_snh('client', '-Un') as client, \
+ nstool.unshare_snh('server', '-n',
+ parent=client, capable=True) as server:
+ client.veth(ifname, server_ifname, server)
+
+ with tempfile.TemporaryDirectory() as tmpdir:
+ yield (client, server, tmpdir)
+
+
+(a)contextlib.contextmanager
+def setup_dhcpd():
+ server_ifname = 'serverif'
+
+ with setup_dhcpd_common(IFNAME, server_ifname) as (client, server, tmpdir):
+ # Configure dhcpd
+ confpath = os.path.join(tmpdir, 'dhcpd.conf')
+ open(confpath, 'w', encoding='UTF-8').write(
+ f'''subnet {SUBNET.network_address} netmask {SUBNET.netmask}
{{
+ option routers {SERVER_IP4.ip};
+ range {CLIENT_IP4.ip} {CLIENT_IP4.ip};
+ }}'''
+ )
+ pidfile = os.path.join(tmpdir, 'dhcpd.pid')
+ leasepath = os.path.join(tmpdir, 'dhcpd.leases')
+ open(leasepath, 'wb').write(b'')
+
+ server.ifup('lo')
+ server.ifup(server_ifname, SERVER_IP4)
+
+ opts = ['-f', '-d', '-4', '-cf',
f'{confpath}',
+ '-lf', f'{leasepath}', '-pf',
f'{pidfile}']
+ server.fg(f'{DHCPD}', '-t', *opts) # test config
+ with server.bg(f'{DHCPD}', *opts, capable=True, check=False) as dhcpd:
+ # Configure the client
+ client.ifup('lo')
+ yield DhcpTestScenario(client=client, ifname=IFNAME,
+ addr=CLIENT_IP4.ip,
+ gateway=SERVER_IP4.ip, mtu=1500)
+ dhcpd.terminate()
+
+
+dhcp_tests(setup_dhcpd)
diff --git a/test/tasst/dhcpv6.py b/test/tasst/dhcpv6.py
new file mode 100644
index 00000000..ab119ae7
--- /dev/null
+++ b/test/tasst/dhcpv6.py
@@ -0,0 +1,89 @@
+#! /usr/bin/env avocado-runner-avocado-classless
+
+# SPDX-License-Identifier: GPL-2.0-or-later
+#
+# Copyright Red Hat
+# Author: David Gibson <david(a)gibson.dropbear.id.au>
+
+"""
+Test A Simple Socket Transport
+
+dhcpv6.py - Helpers for testing DHCPv6
+"""
+
+import contextlib
+import os
+
+import exeter
+
+from . import address, dhcp
+
+
+def dhclientv6(snh, ifname):
+ return dhcp.dhclient(snh, ifname, '6')
+
+
+class Dhcpv6TestScenario:
+ def __init__(self, *, client, ifname, addr):
+ self.client = client
+ self.ifname = ifname
+ self.addr = addr
+
+
+def test_dhcp6_addr(setup):
+ with setup as scn, dhclientv6(scn.client, scn.ifname):
+ addrs = [a.ip for a in scn.client.addrs(scn.ifname, family='inet6',
+ scope='global')]
+ assert scn.addr in addrs # Might also have a SLAAC address
+
+
+DHCP6_TESTS = [test_dhcp6_addr]
+
+
+def dhcp6_tests(setup):
+ for t in DHCP6_TESTS:
+ testid = f'{setup.__qualname__}|{t.__qualname__}'
+ exeter.register_pipe(testid, setup, t)
+
+
+DHCPD = 'dhcpd'
+SUBNET = address.TEST_NET6_TASST_A
+ipa = address.IpiAllocator(SUBNET)
+(SERVER_IP6,) = ipa.next_ipis()
+(CLIENT_IP6,) = ipa.next_ipis()
+IFNAME = 'clientif'
+
+
+(a)contextlib.contextmanager
+def setup_dhcpdv6():
+ server_ifname = 'serverif'
+
+ with dhcp.setup_dhcpd_common(IFNAME, server_ifname) \
+ as (client, server, tmpdir):
+ # Sort out link local addressing
+ server.ifup('lo')
+ server.ifup(server_ifname, SERVER_IP6)
+ client.ifup('lo')
+ client.ifup(IFNAME)
+ server.addr_wait(server_ifname, family='inet6',
scope='link')
+
+ # Configure the DHCP server
+ confpath = os.path.join(tmpdir, 'dhcpd.conf')
+ open(confpath, 'w', encoding='UTF-8').write(
+ f'''subnet6 {SUBNET} {{
+ range6 {CLIENT_IP6.ip} {CLIENT_IP6.ip};
+ }}''')
+ pidfile = os.path.join(tmpdir, 'dhcpd.pid')
+ leasepath = os.path.join(tmpdir, 'dhcpd.leases')
+ open(leasepath, 'wb').write(b'')
+
+ opts = ['-f', '-d', '-6', '-cf',
f'{confpath}',
+ '-lf', f'{leasepath}', '-pf',
f'{pidfile}']
+ server.fg(f'{DHCPD}', '-t', *opts) # test config
+ with server.bg(f'{DHCPD}', *opts, capable=True, check=False) as dhcpd:
+ yield Dhcpv6TestScenario(client=client, ifname=IFNAME,
+ addr=CLIENT_IP6.ip)
+ dhcpd.terminate()
+
+
+dhcp6_tests(setup_dhcpdv6)
--
2.45.2
This constructs essentially the simplest sensible network for passt/pasta
to operate in. We have one netns "simhost" to represent the host where we
will run passt or pasta, and a second "gw" to represent its default
gateway.
Signed-off-by: David Gibson <david(a)gibson.dropbear.id.au>
---
test/Makefile | 3 +-
test/tasst/__main__.py | 1 +
test/tasst/scenario/__init__.py | 12 ++++
test/tasst/scenario/simple.py | 109 ++++++++++++++++++++++++++++++++
4 files changed, 124 insertions(+), 1 deletion(-)
create mode 100644 test/tasst/scenario/__init__.py
create mode 100644 test/tasst/scenario/simple.py
diff --git a/test/Makefile b/test/Makefile
index 0eeaf82e..6748d38a 100644
--- a/test/Makefile
+++ b/test/Makefile
@@ -74,7 +74,8 @@ AVOCADO_JOBS = $(EXETER_JOBS) avocado/static_checkers.json
TASST_SRCS = __init__.py __main__.py address.py dhcp.py dhcpv6.py ndp.py \
nstool.py snh.py transfer.py \
- selftest/__init__.py selftest/static_ifup.py selftest/veth.py
+ selftest/__init__.py selftest/static_ifup.py selftest/veth.py \
+ scenario/__init__.py scenario/simple.py
EXETER_META = meta/lint.json meta/tasst.json
META_JOBS = $(EXETER_META)
diff --git a/test/tasst/__main__.py b/test/tasst/__main__.py
index 8c4efd74..491c68c9 100644
--- a/test/tasst/__main__.py
+++ b/test/tasst/__main__.py
@@ -14,6 +14,7 @@ import exeter
# We import just to get the exeter tests, which flake8 can't see
from . import dhcp, dhcpv6, ndp, nstool, snh, transfer # noqa: F401
+from .scenario import simple # noqa: F401
from .selftest import static_ifup, veth # noqa: F401
diff --git a/test/tasst/scenario/__init__.py b/test/tasst/scenario/__init__.py
new file mode 100644
index 00000000..4ea4584d
--- /dev/null
+++ b/test/tasst/scenario/__init__.py
@@ -0,0 +1,12 @@
+#! /usr/bin/python3
+
+# SPDX-License-Identifier: GPL-2.0-or-later
+#
+# Copyright Red Hat
+# Author: David Gibson <david(a)gibson.dropbear.id.au>
+
+"""
+Test A Simple Socket Transport
+
+scenario/ - Helpers to set up various sample network topologies
+"""
diff --git a/test/tasst/scenario/simple.py b/test/tasst/scenario/simple.py
new file mode 100644
index 00000000..d8b78568
--- /dev/null
+++ b/test/tasst/scenario/simple.py
@@ -0,0 +1,109 @@
+#! /usr/bin/env python3
+
+# SPDX-License-Identifier: GPL-2.0-or-later
+#
+# Copyright Red Hat
+# Author: David Gibson <david(a)gibson.dropbear.id.au>
+
+"""
+Test A Simple Socket Transport
+
+scenario/simple.py - Smallest sensible network to use passt/pasta
+"""
+
+import contextlib
+
+from .. import address, nstool, transfer
+
+
+class __SimpleNet: # pylint: disable=R0903
+ """A simple network setup scenario
+
+ The sample network has 2 snhs (network namespaces) connected with
+ a veth link:
+ [simhost] <-veth-> [gw]
+
+ gw is set up as the default router for simhost.
+
+ simhost has addresses:
+ self.IP4 (IPv4), self.IP6 (IPv6), self.ip6_ll (IPv6 link local)
+
+ gw has addresses:
+ self.GW_IP4 (IPv4), self.GW_IP6 (IPv6),
+ self.gw_ip6_ll (IPv6 link local)
+ self.REMOTE_IP4 (IPv4), self.REMOTE_IP6 (IPv6)
+
+ The "remote" addresses are on a different subnet from the others,
+ so the only way for simhost to reach them is via its default
+ route. This helps to exercise that we're actually using that,
+ rather than just local net routes.
+
+ """
+
+ IFNAME = 'veth'
+ ipa_local = address.IpiAllocator()
+ (IP4, IP6) = ipa_local.next_ipis()
+ (GW_IP4, GW_IP6) = ipa_local.next_ipis()
+
+ ipa_remote = address.IpiAllocator(address.TEST_NET_2,
+ address.TEST_NET6_TASST_B)
+ (REMOTE_IP4, REMOTE_IP6) = ipa_remote.next_ipis()
+
+ def __init__(self, simhost, gw):
+ self.simhost = simhost
+ self.gw = gw
+
+ ifname = self.IFNAME
+ self.gw_ifname = 'gw' + ifname
+ self.simhost.veth(self.IFNAME, self.gw_ifname, self.gw)
+
+ self.gw.ifup('lo')
+ self.gw.ifup(self.gw_ifname, self.GW_IP4, self.GW_IP6,
+ self.REMOTE_IP4, self.REMOTE_IP6)
+
+ self.simhost.ifup('lo')
+ self.simhost.ifup(ifname, self.IP4, self.IP6)
+
+ # Once link is up on both sides, SLAAC will run
+ self.gw_ip6_ll = self.gw.addr_wait(self.gw_ifname,
+ family='inet6',
scope='link')[0]
+ self.ip6_ll = self.simhost.addr_wait(ifname,
+ family='inet6',
scope='link')[0]
+
+ # Set up the default route
+ self.simhost.fg('ip', '-4', 'route',
'add', 'default',
+ 'via', f'{self.GW_IP4.ip}', capable=True)
+ self.simhost.fg('ip', '-6', 'route',
'add', 'default',
+ 'via', f'{self.gw_ip6_ll.ip}',
'dev', f'{ifname}',
+ capable=True)
+
+
+(a)contextlib.contextmanager
+def simple_net():
+ with nstool.unshare_snh('simhost', '-Ucnpf',
'--mount-proc') as simhost, \
+ nstool.unshare_snh('gw', '-n', parent=simhost, capable=True)
as gw:
+ yield __SimpleNet(simhost, gw)
+
+
+(a)contextlib.contextmanager
+def simple_transfer4_setup():
+ with simple_net() as snet:
+ yield transfer.TransferTestScenario(client=snet.simhost,
+ server=snet.gw,
+ connect_ip=snet.REMOTE_IP4.ip,
+ connect_port=10000)
+
+
+transfer.transfer_tests(simple_transfer4_setup)
+
+
+(a)contextlib.contextmanager
+def simple_transfer6_setup():
+ with simple_net() as snet:
+ yield transfer.TransferTestScenario(client=snet.simhost,
+ server=snet.gw,
+ connect_ip=snet.REMOTE_IP6.ip,
+ connect_port=10000)
+
+
+transfer.transfer_tests(simple_transfer6_setup)
--
2.45.2
Convert the old-style tests for pasta (DHCP, NDP, TCP and UDP transfers)
to using avocado. There are a few differences in what we test, but this
should generally improve coverage:
* We run in a constructed network environment, so we no longer depend on
the real host's networking configuration
* We do independent setup for each individual test
* We add explicit tests for --config-net, which we use to accelerate that
setup for the TCP and UDP tests
* The TCP and UDP tests now test transfers between the guest and a
(simulated) remote site that's on a different network from the simulated
pasta host. Thus testing the no NAT case that passt/pasta emphasizes.
(We need to add tests for the NAT cases back in).
Signed-off-by: David Gibson <david(a)gibson.dropbear.id.au>
---
test/Makefile | 4 +-
test/pasta/.gitignore | 1 +
test/pasta/pasta.py | 138 +++++++++++++++++++++++++++++++++++++++++
test/tasst/__main__.py | 2 +-
test/tasst/pasta.py | 52 ++++++++++++++++
5 files changed, 194 insertions(+), 3 deletions(-)
create mode 100644 test/pasta/.gitignore
create mode 100644 test/pasta/pasta.py
create mode 100644 test/tasst/pasta.py
diff --git a/test/Makefile b/test/Makefile
index 6748d38a..ba249a5d 100644
--- a/test/Makefile
+++ b/test/Makefile
@@ -64,11 +64,11 @@ LOCAL_ASSETS = mbuto.img mbuto.mem.img podman/bin/podman QEMU_EFI.fd
\
$(TESTDATA_ASSETS)
ASSETS = $(DOWNLOAD_ASSETS) $(LOCAL_ASSETS)
-AVOCADO_ASSETS =
+AVOCADO_ASSETS = nstool small.bin medium.bin big.bin
META_ASSETS = nstool small.bin medium.bin big.bin
EXETER_SH = build/static_checkers.sh
-EXETER_PY = build/build.py
+EXETER_PY = build/build.py pasta/pasta.py
EXETER_JOBS = $(EXETER_SH:%.sh=%.json) $(EXETER_PY:%.py=%.json)
AVOCADO_JOBS = $(EXETER_JOBS) avocado/static_checkers.json
diff --git a/test/pasta/.gitignore b/test/pasta/.gitignore
new file mode 100644
index 00000000..a6c57f5f
--- /dev/null
+++ b/test/pasta/.gitignore
@@ -0,0 +1 @@
+*.json
diff --git a/test/pasta/pasta.py b/test/pasta/pasta.py
new file mode 100644
index 00000000..491927a6
--- /dev/null
+++ b/test/pasta/pasta.py
@@ -0,0 +1,138 @@
+#! /usr/bin/env avocado-runner-avocado-classless
+
+# SPDX-License-Identifier: GPL-2.0-or-later
+#
+# Copyright Red Hat
+# Author: David Gibson <david(a)gibson.dropbear.id.au>
+
+"""
+avocado/pasta.py - Basic tests for pasta mode
+"""
+
+import contextlib
+import ipaddress
+
+import exeter
+
+from tasst import dhcp, dhcpv6, ndp, nstool
+from tasst.pasta import Pasta
+from tasst.scenario.simple import simple_net
+
+IN_FWD_PORT = 10002
+SPLICE_FWD_PORT = 10003
+FWD_OPTS = ['-t', f'{IN_FWD_PORT}', '-u',
f'{IN_FWD_PORT}',
+ '-T', f'{SPLICE_FWD_PORT}', '-U',
f'{SPLICE_FWD_PORT}']
+
+
+(a)contextlib.contextmanager
+def pasta_unconfigured(*opts):
+ with simple_net() as simnet:
+ with nstool.unshare_snh('pastans', '-Ucnpf',
'--mount-proc',
+ parent=simnet.simhost, capable=True) \
+ as guestns:
+ with Pasta(host=simnet.simhost, opts=opts, ns=guestns) as pasta:
+ yield simnet, pasta.ns
+
+
+(a)exeter.test
+def test_ifname():
+ with pasta_unconfigured() as (simnet, ns):
+ expected = set(['lo', simnet.IFNAME])
+ exeter.assert_eq(set(ns.ifs()), expected)
+
+
+(a)contextlib.contextmanager
+def pasta_ndp_setup():
+ with pasta_unconfigured() as (simnet, guestns):
+ guestns.ifup(simnet.IFNAME)
+ yield ndp.NdpTestScenario(client=guestns,
+ ifname=simnet.IFNAME,
+ network=simnet.IP6.network,
+ gateway=simnet.gw_ip6_ll.ip)
+
+
+ndp.ndp_tests(pasta_ndp_setup)
+
+
+(a)contextlib.contextmanager
+def pasta_dhcp():
+ with pasta_unconfigured() as (simnet, guestns):
+ yield dhcp.DhcpTestScenario(client=guestns,
+ ifname=simnet.IFNAME,
+ addr=simnet.IP4.ip,
+ gateway=simnet.GW_IP4.ip,
+ mtu=65520)
+
+
+dhcp.dhcp_tests(pasta_dhcp)
+
+
+(a)contextlib.contextmanager
+def pasta_dhcpv6():
+ with pasta_unconfigured() as (simnet, guestns):
+ yield dhcpv6.Dhcpv6TestScenario(client=guestns,
+ ifname=simnet.IFNAME,
+ addr=simnet.IP6.ip)
+
+
+dhcpv6.dhcp6_tests(pasta_dhcpv6)
+
+
+(a)contextlib.contextmanager
+def pasta_configured():
+ with pasta_unconfigured('--config-net', *FWD_OPTS) as (simnet, ns):
+ # Wait for DAD to complete on the --config-net address
+ ns.addr_wait(simnet.IFNAME, family='inet6', scope='global')
+ yield simnet, ns
+
+
+(a)exeter.test
+def test_config_net_addr():
+ with pasta_configured() as (simnet, ns):
+ addrs = ns.addrs(simnet.IFNAME, scope='global')
+ exeter.assert_eq(set(addrs), set([simnet.IP4, simnet.IP6]))
+
+
+(a)exeter.test
+def test_config_net_route4():
+ with pasta_configured() as (simnet, ns):
+ (defroute,) = ns.routes4(dst='default')
+ gateway = ipaddress.ip_address(defroute['gateway'])
+ exeter.assert_eq(gateway, simnet.GW_IP4.ip)
+
+
+(a)exeter.test
+def test_config_net_route6():
+ with pasta_configured() as (simnet, ns):
+ (defroute,) = ns.routes6(dst='default')
+ gateway = ipaddress.ip_address(defroute['gateway'])
+ exeter.assert_eq(gateway, simnet.gw_ip6_ll.ip)
+
+
+(a)exeter.test
+def test_config_net_mtu():
+ with pasta_configured() as (simnet, ns):
+ mtu = ns.mtu(simnet.IFNAME)
+ exeter.assert_eq(mtu, 65520)
+
+
+(a)contextlib.contextmanager
+def outward_transfer():
+ with pasta_configured() as (simnet, ns):
+ yield ns, simnet.gw
+
+
+(a)contextlib.contextmanager
+def inward_transfer():
+ with pasta_configured() as (simnet, ns):
+ yield simnet.gw, ns
+
+
+(a)contextlib.contextmanager
+def spliced_transfer():
+ with pasta_configured() as (simnet, ns):
+ yield ns, simnet.simhost
+
+
+if __name__ == '__main__':
+ exeter.main()
diff --git a/test/tasst/__main__.py b/test/tasst/__main__.py
index 491c68c9..058b3746 100644
--- a/test/tasst/__main__.py
+++ b/test/tasst/__main__.py
@@ -13,7 +13,7 @@ library of test helpers for passt & pasta
import exeter
# We import just to get the exeter tests, which flake8 can't see
-from . import dhcp, dhcpv6, ndp, nstool, snh, transfer # noqa: F401
+from . import dhcp, dhcpv6, ndp, nstool, pasta, snh, transfer # noqa: F401
from .scenario import simple # noqa: F401
from .selftest import static_ifup, veth # noqa: F401
diff --git a/test/tasst/pasta.py b/test/tasst/pasta.py
new file mode 100644
index 00000000..030affce
--- /dev/null
+++ b/test/tasst/pasta.py
@@ -0,0 +1,52 @@
+#! /usr/bin/python3
+
+# SPDX-License-Identifier: GPL-2.0-or-later
+#
+# Copyright Red Hat
+# Author: David Gibson <david(a)gibson.dropbear.id.au>
+
+"""
+Test A Simple Socket Transport
+
+pasta.py - Helpers for starting pasta
+"""
+
+import contextlib
+import os.path
+import tempfile
+
+
+PASTA_BIN = './pasta'
+
+
+class Pasta(contextlib.AbstractContextManager):
+ """A managed pasta instance"""
+
+ def __init__(self, *, host, ns, opts):
+ self.host = host
+ self.ns = ns
+ self.opts = opts
+ self.proc = None
+
+ def __enter__(self):
+ self.tmpdir = tempfile.TemporaryDirectory()
+ piddir = self.tmpdir.__enter__()
+ pidfile = os.path.join(piddir, 'pasta.pid')
+ relpid = self.ns.relative_pid(self.host)
+ cmd = [f'{PASTA_BIN}', '-f', '-P',
f'{pidfile}'] + list(self.opts) + \
+ [f'{relpid}']
+ self.proc = self.host.bg(*cmd)
+ self.proc.__enter__()
+ # Wait for the PID file to be written
+ pidstr = None
+ while not pidstr:
+ pidstr = self.host.output('cat', f'{pidfile}',
check=False)
+ self.pid = int(pidstr)
+ return self
+
+ def __exit__(self, *exc_details):
+ try:
+ self.host.fg('kill', '-TERM', f'{self.pid}')
+ self.proc.__exit__(*exc_details)
+ finally:
+ self.tmpdir.__exit__(*exc_details)
--
2.45.2
On Mon, Aug 05, 2024 at 10:36:39PM +1000, David Gibson wrote:Here's a rough proof of concept showing how we could run tests for passt with Avocado and the exeter library I recently created. It includes Cleber's patch adding some basic Avocado tests and builds on that. The current draft is pretty janky: * The build rules to download and install the necessary pieces are messy * We create the Avocado job files from the exeter sources in the Makefile. Ideally Avocado would eventually be extended to handle this itself * The names that Avocado sees for each test are overlong * There's some hacks to make sure things are executed from the right working directory But, it's a starting point. Stefano, If you could look particularly at 6/22 and 22/22 which add the real tests for passt/pasta, that would be great. The more specific you can be about what you find ugly about how the tests are written, then better I can try to address that. I suspect it will be easier to actually apply the series, then look at the new test files (test/build/build.py, and test/pasta/pasta.py particularly). From there you can look at as much of the support library as you need to, rather than digging through the actual patches to look for that.Forgot to mention. Patches 1 & 2 should be good to go regardless of what we do with the rest of the testing stuff. -- David Gibson (he or they) | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you, not the other way | around. http://www.ozlabs.org/~dgibson
On Tue, 6 Aug 2024 22:28:19 +1000 David Gibson <david(a)gibson.dropbear.id.au> wrote:On Mon, Aug 05, 2024 at 10:36:39PM +1000, David Gibson wrote:Applied up to 2/22. -- StefanoHere's a rough proof of concept showing how we could run tests for passt with Avocado and the exeter library I recently created. It includes Cleber's patch adding some basic Avocado tests and builds on that. The current draft is pretty janky: * The build rules to download and install the necessary pieces are messy * We create the Avocado job files from the exeter sources in the Makefile. Ideally Avocado would eventually be extended to handle this itself * The names that Avocado sees for each test are overlong * There's some hacks to make sure things are executed from the right working directory But, it's a starting point. Stefano, If you could look particularly at 6/22 and 22/22 which add the real tests for passt/pasta, that would be great. The more specific you can be about what you find ugly about how the tests are written, then better I can try to address that. I suspect it will be easier to actually apply the series, then look at the new test files (test/build/build.py, and test/pasta/pasta.py particularly). From there you can look at as much of the support library as you need to, rather than digging through the actual patches to look for that.Forgot to mention. Patches 1 & 2 should be good to go regardless of what we do with the rest of the testing stuff.