September 2022 - dev

[PATCH v2 00/32] Improve handling of test temporary files
by David Gibson 08 Sep '22

08 Sep '22

The tests create temporary files and fifos in a number of places. Some of them can interfere with later test runs, and are awkward to clean up. Consolidate them in a single per-run directory in /tmp which gets cleaned up automatically. This is based on the earlier userns cleanup series. Chances since v1: * Fixed a number of straightforward bugs where things were missed * Also removed files we're no longer putting into the source tree from .gitignore * Added an extra patch moving the video processing files David Gibson (32): test: Correctly match "background" with "wait" commands test: Context execution helpers test: Allow a tmux pane to watch commands executed in contexts test: Integration of old-style pane execution and new context execution test: Issue host commands via context for most tests test: Use new-style contexts for passt pane in the pasta and passt tests test: Add nsholder utility test: Extend context system to run commands in namespace for pasta tests test: Use context system for guest commands test: Use context system for two_guests tests test: Use new-style command issue for passt_in_ns tests Don't store UID & GID persistently in the context structure Split checking for root from dropping root privilege Consolidate determination of UID/GID to run as Safer handling if we can't open /proc/self/uid_map Move self-isolation code into a separate file Consolidate validation of pasta namespace options Clean up and rename conf_ns_open() Correctly handle --netns-only in pasta_start_ns() Handle userns isolation and dropping root at the same time Allow --userns when pasta spawns a command test: Group tests by context then protocol, rather than the reverse test: Remove unused variable FFMPEG_PID_FILE test: Actually run cleanup function test: Create common state directories for temporary files test: Move context temporary files to state dir test: Dont regnerate small test file in pasta/tcp test: Use paths in __STATEDIR__ instead of 'temp' and 'tempdir' directives test: Move pause temporary file to state directory test: Store pcap files in $LOGDIR instead of /tmp test: Move pidfiles and nsholder sockets into state directory test: Move video processing files to $STATEBASE .gitignore | 1 - Makefile | 8 +- conf.c | 236 +++++++------- isolation.c | 210 +++++++++++++ isolation.h | 15 + passt.1 | 5 +- passt.c | 116 +------ passt.h | 9 - pasta.c | 91 ++++-- pasta.h | 1 + test/.gitignore | 6 +- test/Makefile | 13 +- test/build/all | 31 +- test/demo/passt | 9 +- test/demo/pasta | 12 +- test/demo/podman | 11 +- test/distro/debian | 2 +- test/distro/fedora | 2 +- test/distro/opensuse | 16 +- test/distro/ubuntu | 8 +- test/lib/context | 128 ++++++++ test/lib/layout | 61 ++-- test/lib/setup | 296 +++++++++--------- test/lib/term | 98 +++++- test/lib/test | 152 ++++----- test/lib/video | 20 +- test/nsholder.c | 117 +++++++ test/passt.mbuto | 32 +- test/{dhcp/passt => passt/dhcp} | 2 +- test/{ndp/passt => passt/ndp} | 2 +- test/{shutdown/passt => passt/shutdown} | 8 +- test/{tcp/passt => passt/tcp} | 8 +- test/{udp/passt => passt/udp} | 9 +- test/{icmp/passt_in_ns => passt_in_ns/icmp} | 2 +- .../passt_in_ns => passt_in_ns/shutdown} | 8 +- test/{tcp/passt_in_ns => passt_in_ns/tcp} | 27 +- test/{udp/passt_in_ns => passt_in_ns/udp} | 13 +- test/{dhcp/pasta => pasta/dhcp} | 2 +- test/{ndp/pasta => pasta/ndp} | 2 +- test/{tcp/pasta => pasta/tcp} | 14 +- test/{udp/pasta => pasta/udp} | 21 +- test/run | 42 +-- util.c | 83 ----- util.h | 2 - 44 files changed, 1159 insertions(+), 792 deletions(-) create mode 100644 isolation.c create mode 100644 isolation.h create mode 100644 test/lib/context create mode 100644 test/nsholder.c rename test/{dhcp/passt => passt/dhcp} (98%) rename test/{ndp/passt => passt/ndp} (95%) rename test/{shutdown/passt => passt/shutdown} (75%) rename test/{tcp/passt => passt/tcp} (95%) rename test/{udp/passt => passt/udp} (88%) rename test/{icmp/passt_in_ns => passt_in_ns/icmp} (94%) rename test/{shutdown/passt_in_ns => passt_in_ns/shutdown} (75%) rename test/{tcp/passt_in_ns => passt_in_ns/tcp} (97%) rename test/{udp/passt_in_ns => passt_in_ns/udp} (97%) rename test/{dhcp/pasta => pasta/dhcp} (96%) rename test/{ndp/pasta => pasta/ndp} (95%) rename test/{tcp/pasta => pasta/tcp} (95%) rename test/{udp/pasta => pasta/udp} (74%) -- 2.37.3

1 32

[PATCH 00/12] Improve handling of test temporary files
by David Gibson 08 Sep '22

08 Sep '22

The tests create temporary files and fifos in a number of places. Some of them can interfere with later test runs, and are awkward to clean up. Consolidate them in a single per-run directory in /tmp which gets cleaned up automatically. This is based on the earlier userns cleanup series. David Gibson (12): test: Group tests by context then protocol, rather than the reverse test: Remove stale comments from context_setup_guest() test: Remove unused variable FFMPEG_PID_FILE test: Actually run cleanup function test: Create common state directories for temporary files test: Move passt_test_log_pipe to state directory test: Move context temporary files to state dir test: Dont regnerate small test file in pasta/tcp test: Use paths in __STATEDIR__ instead of 'temp' and 'tempdir' directives test: Move pause temporary file to state directory test: Store pcap files in $LOGDIR instead of /tmp test: Move pidfiles and nsholder sockets into state directory test/build/all | 31 ++++--- test/demo/passt | 9 +- test/demo/pasta | 12 ++- test/demo/podman | 11 ++- test/distro/debian | 2 +- test/distro/fedora | 2 +- test/distro/opensuse | 16 ++-- test/distro/ubuntu | 8 +- test/lib/context | 61 +++++++------- test/lib/layout | 16 ++-- test/lib/setup | 84 ++++++++++--------- test/lib/term | 18 ++-- test/lib/test | 14 +--- test/lib/video | 1 - test/{dhcp/passt => passt/dhcp} | 2 +- test/{ndp/passt => passt/ndp} | 2 +- test/{shutdown/passt => passt/shutdown} | 4 +- test/{tcp/passt => passt/tcp} | 6 +- test/{udp/passt => passt/udp} | 8 +- test/{icmp/passt_in_ns => passt_in_ns/icmp} | 2 +- .../passt_in_ns => passt_in_ns/shutdown} | 4 +- test/{tcp/passt_in_ns => passt_in_ns/tcp} | 10 +-- test/{udp/passt_in_ns => passt_in_ns/udp} | 8 +- test/{dhcp/pasta => pasta/dhcp} | 2 +- test/{ndp/pasta => pasta/ndp} | 2 +- test/{tcp/pasta => pasta/tcp} | 13 ++- test/{udp/pasta => pasta/udp} | 20 ++--- test/run | 48 +++++------ 28 files changed, 202 insertions(+), 214 deletions(-) rename test/{dhcp/passt => passt/dhcp} (98%) rename test/{ndp/passt => passt/ndp} (95%) rename test/{shutdown/passt => passt/shutdown} (80%) rename test/{tcp/passt => passt/tcp} (96%) rename test/{udp/passt => passt/udp} (88%) rename test/{icmp/passt_in_ns => passt_in_ns/icmp} (94%) rename test/{shutdown/passt_in_ns => passt_in_ns/shutdown} (79%) rename test/{tcp/passt_in_ns => passt_in_ns/tcp} (97%) rename test/{udp/passt_in_ns => passt_in_ns/udp} (97%) rename test/{dhcp/pasta => pasta/dhcp} (96%) rename test/{ndp/pasta => pasta/ndp} (95%) rename test/{tcp/pasta => pasta/tcp} (95%) rename test/{udp/pasta => pasta/udp} (74%) -- 2.37.3

1 13

[PATCH v2 0/4] Cleanups to performance tests
by David Gibson 07 Sep '22

07 Sep '22

A handful of changes to the performance tests to make them more robust and faster. Changes since v1: * Addressed missing change in test time parameterization * Updated commit messages for accuracy in final version David Gibson (4): gitignore pidfiles other than passt.pid test: Combine iperf3c and iperf3s into a single DSL command test: Parameterize run time for throughput performance tests test: Rewrite test_iperf3 .gitignore | 3 +- test/lib/test | 113 +++++++++++++++++++------------------------- test/perf/passt_tcp | 68 ++++++++++---------------- test/perf/passt_udp | 63 +++++++++--------------- test/perf/pasta_tcp | 61 +++++++++--------------- test/perf/pasta_udp | 61 ++++++++++-------------- 6 files changed, 144 insertions(+), 225 deletions(-) -- 2.37.2

2 9

[PATCH 00/10] Clean up handling of userns
by David Gibson 07 Sep '22

07 Sep '22

Our handling of user namespaces is more complex than it needs to be. This simplifies the handling by identifying and entering (or creating) the correct userns earlier, so that later code doesn't need to deal with it any more. Along the way we make a number of other cleanups to handling of userns and setting our user and group. This is based on my earlier test command dispatch and performance test cleanup series. David Gibson (10): Don't store UID & GID persistently in the context structure Split checking for root from dropping root privilege Consolidate determination of UID/GID to run as Safer handling if we can't open /proc/self/uid_map Move self-isolation code into a separate file Consolidate validation of pasta namespace options Clean up and rename conf_ns_open() Correctly handle --netns-only in pasta_start_ns() Handle userns isolation and dropping root at the same time Allow --userns when pasta spawns a command Makefile | 8 +- conf.c | 236 ++++++++++++++++++++++++++-------------------------- isolation.c | 209 ++++++++++++++++++++++++++++++++++++++++++++++ isolation.h | 15 ++++ passt.1 | 5 +- passt.c | 117 +------------------------- passt.h | 9 -- pasta.c | 91 ++++++++++++-------- pasta.h | 1 + util.c | 84 ------------------- util.h | 2 - 11 files changed, 411 insertions(+), 366 deletions(-) create mode 100644 isolation.c create mode 100644 isolation.h -- 2.37.3

1 10

[PATCH] Makefile: Honour LDFLAGS for binary targets
by Stefano Brivio 07 Sep '22

07 Sep '22

We don't set any, but we should use them if they are passed in the environment. On a Fedora Rawhide package build, annocheck (https://sourceware.org/annobin/) reports: Hardened: /usr/bin/passt: FAIL: bind-now test because not linked with -Wl,-z,now ...despite the build system exporting -Wl,-z,now in LDFLAGS. Signed-off-by: Stefano Brivio <sbrivio(a)redhat.com> --- Makefile | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/Makefile b/Makefile index 26e64d5..644a541 100644 --- a/Makefile +++ b/Makefile @@ -105,11 +105,11 @@ seccomp.h: $(PASST_SRCS) $(PASST_HEADERS) @ EXTRA_SYSCALLS=$(EXTRA_SYSCALLS) ./seccomp.sh $^ passt: $(PASST_SRCS) $(PASST_HEADERS) seccomp.h - $(CC) $(CFLAGS) $(PASST_SRCS) -o passt + $(CC) $(CFLAGS) $(PASST_SRCS) -o passt $(LDFLAGS) passt.avx2: CFLAGS += -Ofast -mavx2 -ftree-vectorize -funroll-loops passt.avx2: $(PASST_SRCS) $(PASST_HEADERS) seccomp.h - $(CC) $(filter-out -O2,$(CFLAGS)) $(PASST_SRCS) -o passt.avx2 + $(CC) $(filter-out -O2,$(CFLAGS)) $(PASST_SRCS) -o passt.avx2 $(LDFLAGS) passt.avx2: passt @@ -117,7 +117,7 @@ pasta.avx2 pasta.1 pasta: pasta%: passt% ln -s $< $@ qrap: $(QRAP_SRCS) passt.h - $(CC) $(CFLAGS) $(QRAP_SRCS) -o qrap + $(CC) $(CFLAGS) $(QRAP_SRCS) -o qrap $(LDFLAGS) valgrind: EXTRA_SYSCALLS="rt_sigprocmask rt_sigtimedwait rt_sigaction \ getpid gettid kill clock_gettime mmap munmap open \ -- 2.35.1

2 1

[PATCH] fedora: Escape % characters in spec file's changelog
by Stefano Brivio 06 Sep '22

06 Sep '22

...rpmbuild otherwise expands valid macro names in changelog entries. Signed-off-by: Stefano Brivio <sbrivio(a)redhat.com> --- contrib/fedora/rpkg.macros | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/contrib/fedora/rpkg.macros b/contrib/fedora/rpkg.macros index df9dfc5..06359ab 100644 --- a/contrib/fedora/rpkg.macros +++ b/contrib/fedora/rpkg.macros @@ -36,7 +36,7 @@ function passt_git_changelog_entry { IFS=' ' for l in $(git log ${__from}..${__to} --pretty=format:"- %s" -- contrib/fedora); do - printf "%s%s\n" '-' "${l#*:}" + printf "%s%s\n" '-' "${l#*:}" | sed s/%/%%/g done unset IFS -- 2.35.1

1 0

[PATCH] test: Wait for systemd-resolved to be ready on Ubuntu 22.04 for s390x
by Stefano Brivio 05 Sep '22

05 Sep '22

On new Ubuntu 22.04 images, stopping systemd-resolved to get the dhclient script override resolv.conf doesn't work anymore. I originally used that hack to avoid introducing a delay which is needed when running it on TCG. Keep systemd-resolved running instead, and wait for it to be ready by retrying to resolve a domain a few times before installing packages, so that we don't add another ugly delay that might unnecessarily slow down things even further. Signed-off-by: Stefano Brivio <sbrivio(a)redhat.com> --- test/distro/ubuntu | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/test/distro/ubuntu b/test/distro/ubuntu index 6666f38..5e67a46 100644 --- a/test/distro/ubuntu +++ b/test/distro/ubuntu @@ -54,6 +54,11 @@ hout INIT_MSG cat /tmp/init_msg check [ __INIT_MSG__ = "from_ns" ] endef +# With systemd-resolved and TCG, DNS might take a while to work +def dns_ready_wait +host r=10; while [ \${r} -gt 0 ]; do host ubuntu.com && break; sleep 5; r=\$((r - 1)); done +endef + # Start passt, set common variables hostb ./passt -P __PIDFILE__ & sleep 1 @@ -185,11 +190,10 @@ test Ubuntu 22.04 (Jammy Jellyfish), s390x host ./qrap 5 qemu-system-s390x -m 2048 -smp 2 -serial stdio -nodefaults -nographic __BASEPATH__/prepared-jammy-server-cloudimg-s390x.img -net socket,fd=5 -net nic,model=virtio -device virtio-rng-ccw -snapshot -host service systemd-resolved stop host export DEBIAN_FRONTEND=noninteractive host apt-get -y remove needrestart snapd host dhclient -4 -sleep 2 +dns_ready_wait host apt-get update host apt-get -y install make gcc socat -- 2.35.1

1 0

[PATCH] conf: Fix getopt_long() optstring for current semantics of -D, -S, -p
by Stefano Brivio 03 Sep '22

03 Sep '22

Declaring them as required_argument in the longopts array specifies validation, but doesn't affect how optind is increased after parsing their values. Currently, passing one of these options as last option causes pasta to handle their own values as path to a binary to execute. Fixes: aae2a9bbf7d1 ("conf: Use "-D none" and "-S none" instead of missing empty option arguments") Fixes: bf95322fc1ef ("conf: Make the argument to --pcap option mandatory") Signed-off-by: Stefano Brivio <sbrivio(a)redhat.com> --- conf.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/conf.c b/conf.c index e6d1c62..2edb4ae 100644 --- a/conf.c +++ b/conf.c @@ -1094,9 +1094,9 @@ void conf(struct ctx *c, int argc, char **argv) const char *optstring; if (c->mode == MODE_PASST) - optstring = "dqfehs:p::P:m:a:n:M:g:i:D::S::46t:u:"; + optstring = "dqfehs:p:P:m:a:n:M:g:i:D:S:46t:u:"; else - optstring = "dqfehI:p::P:m:a:n:M:g:i:D::S::46t:u:T:U:"; + optstring = "dqfehI:p:P:m:a:n:M:g:i:D:S:46t:u:T:U:"; name = getopt_long(argc, argv, optstring, options, NULL); -- 2.35.1

2 1

[PATCH v2 0/2] Two further fixes for Fedora's spec file issues
by Stefano Brivio 02 Sep '22

02 Sep '22

...both reported by Daniel. v2: Details in single patch messages Stefano Brivio (2): fedora: Add %dir entries for own SELinux policy directory and documentation fedora: Add selinux-policy Requires: tag contrib/fedora/passt.spec | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-) -- 2.35.1

1 2

[PATCH 0/2] Two further fixes for Fedora's spec file issues
by Stefano Brivio 02 Sep '22

02 Sep '22

...both reported by Daniel. Stefano Brivio (2): fedora: Add %dir entries for own SELinux policy directory and documentation fedora: Add selinux-policy Requires: tag contrib/fedora/passt.spec | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) -- 2.35.1

2 7