September 2022 - dev

[PATCH 0/2] Don't drop outbound zero-length UDP packets over tap
by David Gibson 13 Sep '22

13 Sep '22

passt/pasta was incorrectly dropping UDP packets with a zero-length payload when travelling out via the tap interface. This is incorrect, since for a datagram protocol, zero-length packets are still meaningful. Based on my earlier series for test command dispatch, user namespace cleanup and test temporary file handling. Bugzilla: https://bugs.passt.top/show_bug.cgi?id=19 David Gibson (2): udp: Don't drop zero-length outbound UDP packets test: Simpler termination handling for UDP tests test/passt/udp | 23 +++++++------- test/passt_in_ns/udp | 73 ++++++++++++++++++++++---------------------- test/pasta/udp | 31 +++++++++---------- udp.c | 10 +++--- 4 files changed, 67 insertions(+), 70 deletions(-) -- 2.37.3

2 8

[PATCH v2 0/3] Don't drop outbound zero-length UDP packets over tap
by David Gibson 13 Sep '22

13 Sep '22

passt/pasta was incorrectly dropping UDP packets with a zero-length payload when travelling out via the tap interface. This is incorrect, since for a datagram protocol, zero-length packets are still meaningful. Based on my earlier series for test command dispatch, user namespace cleanup and test temporary file handling. Bugzilla: https://bugs.passt.top/show_bug.cgi?id=19 Changes since v1: * Also remove unnecessary initialization of mm[] David Gibson (3): udp: Don't pre-initialize msghdr array udp: Don't drop zero-length outbound UDP packets test: Simpler termination handling for UDP tests test/passt/udp | 23 +++++++------- test/passt_in_ns/udp | 73 ++++++++++++++++++++++---------------------- test/pasta/udp | 31 +++++++++---------- udp.c | 23 +++++++++----- 4 files changed, 77 insertions(+), 73 deletions(-) -- 2.37.3

1 3

[PATCH v2 00/10] Clean up handling of userns
by David Gibson 13 Sep '22

13 Sep '22

Sorry for the resend, but I found a bug that means this will fail to build on some distros / versions. Our handling of user namespaces is more complex than it needs to be. This simplifies the handling by identifying and entering (or creating) the correct userns earlier, so that later code doesn't need to deal with it any more. Along the way we make a number of other cleanups to handling of userns and setting our user and group. This is based on my earlier test command dispatch and performance test cleanup series. Changes since v1: * Fixed overenthusiastic pruning of #includes when moving the self-isolation code which broke compile on some distro versions David Gibson (10): Don't store UID & GID persistently in the context structure Split checking for root from dropping root privilege Consolidate determination of UID/GID to run as Safer handling if we can't open /proc/self/uid_map Move self-isolation code into a separate file Consolidate validation of pasta namespace options Clean up and rename conf_ns_open() Correctly handle --netns-only in pasta_start_ns() Handle userns isolation and dropping root at the same time Allow --userns when pasta spawns a command Makefile | 8 +- conf.c | 236 ++++++++++++++++++++++++++-------------------------- isolation.c | 210 ++++++++++++++++++++++++++++++++++++++++++++++ isolation.h | 15 ++++ passt.1 | 5 +- passt.c | 116 +------------------------- passt.h | 9 -- pasta.c | 91 ++++++++++++-------- pasta.h | 1 + util.c | 83 ------------------ util.h | 2 - 11 files changed, 412 insertions(+), 364 deletions(-) create mode 100644 isolation.c create mode 100644 isolation.h -- 2.37.3

2 32

[PATCH v4 00/12] Improve handling of test temporary files
by David Gibson 13 Sep '22

13 Sep '22

Sigh. Even more silly errors The tests create temporary files and fifos in a number of places. Some of them can interfere with later test runs, and are awkward to clean up. Consolidate them in a single per-run directory in /tmp which gets cleaned up automatically. This is based on the earlier userns cleanup series. Changes since v3: * Fixed problem with ssh hosts file naming which caused a bunch of non-fatal warnings Changes since v2: * Reinstated patch to move the log pipe accidentally dropped during a rebase Chances since v1: * Fixed a number of straightforward bugs where things were missed * Also removed files we're no longer putting into the source tree from .gitignore * Added an extra patch moving the video processing files David Gibson (12): test: Group tests by context then protocol, rather than the reverse test: Remove unused variable FFMPEG_PID_FILE test: Actually run cleanup function test: Create common state directories for temporary files test: Move passt_test_log_pipe to state directory test: Move context temporary files to state dir test: Dont regnerate small test file in pasta/tcp test: Use paths in __STATEDIR__ instead of 'temp' and 'tempdir' directives test: Move pause temporary file to state directory test: Store pcap files in $LOGDIR instead of /tmp test: Move pidfiles and nsholder sockets into state directory test: Move video processing files to $STATEBASE .gitignore | 1 - test/.gitignore | 3 - test/build/all | 31 ++++--- test/demo/passt | 9 +- test/demo/pasta | 12 ++- test/demo/podman | 11 ++- test/distro/debian | 2 +- test/distro/fedora | 2 +- test/distro/opensuse | 16 ++-- test/distro/ubuntu | 8 +- test/lib/context | 61 +++++++------- test/lib/layout | 16 ++-- test/lib/setup | 84 ++++++++++--------- test/lib/term | 26 +++--- test/lib/test | 14 +--- test/lib/video | 20 ++--- test/{dhcp/passt => passt/dhcp} | 2 +- test/{ndp/passt => passt/ndp} | 2 +- test/{shutdown/passt => passt/shutdown} | 4 +- test/{tcp/passt => passt/tcp} | 6 +- test/{udp/passt => passt/udp} | 8 +- test/{icmp/passt_in_ns => passt_in_ns/icmp} | 2 +- .../passt_in_ns => passt_in_ns/shutdown} | 4 +- test/{tcp/passt_in_ns => passt_in_ns/tcp} | 10 +-- test/{udp/passt_in_ns => passt_in_ns/udp} | 8 +- test/{dhcp/pasta => pasta/dhcp} | 2 +- test/{ndp/pasta => pasta/ndp} | 2 +- test/{tcp/pasta => pasta/tcp} | 13 ++- test/{udp/pasta => pasta/udp} | 20 ++--- test/run | 48 +++++------ 30 files changed, 217 insertions(+), 230 deletions(-) rename test/{dhcp/passt => passt/dhcp} (98%) rename test/{ndp/passt => passt/ndp} (95%) rename test/{shutdown/passt => passt/shutdown} (80%) rename test/{tcp/passt => passt/tcp} (96%) rename test/{udp/passt => passt/udp} (88%) rename test/{icmp/passt_in_ns => passt_in_ns/icmp} (94%) rename test/{shutdown/passt_in_ns => passt_in_ns/shutdown} (79%) rename test/{tcp/passt_in_ns => passt_in_ns/tcp} (97%) rename test/{udp/passt_in_ns => passt_in_ns/udp} (97%) rename test/{dhcp/pasta => pasta/dhcp} (96%) rename test/{ndp/pasta => pasta/ndp} (95%) rename test/{tcp/pasta => pasta/tcp} (95%) rename test/{udp/pasta => pasta/udp} (74%) -- 2.37.3

2 19

[PATCH v5 00/14] Improve handling of test temporary files
by David Gibson 13 Sep '22

13 Sep '22

The tests create temporary files and fifos in a number of places. Some of them can interfere with later test runs, and are awkward to clean up. Consolidate them in a single per-run directory in /tmp which gets cleaned up automatically. This is based on the earlier userns cleanup series. Changes since v4: * Also moved .pid files from demo into state directory * Moved the .cast video files into $LOGDIR/web instead of plain $LOGDIR * Added a new patch moving the perf.js report into $LOGDIR/web too * Updated hooks/pre-push to take things from their new directories Changes since v3: * Fixed problem with ssh hosts file naming which caused a bunch of non-fatal warnings Changes since v2: * Reinstated patch to move the log pipe accidentally dropped during a rebase Chances since v1: * Fixed a number of straightforward bugs where things were missed * Also removed files we're no longer putting into the source tree from .gitignore * Added an extra patch moving the video processing files David Gibson (14): test: Group tests by mode then protocol, rather than the reverse test: Remove unused variable FFMPEG_PID_FILE test: Actually run cleanup function test: Create common state directories for temporary files test: Move passt_test_log_pipe to state directory test: Move context temporary files to state dir test: Don't redundantly regenerate small test file in pasta/tcp test: Use paths in __STATEDIR__ instead of 'temp' and 'tempdir' directives test: Move pause temporary file to state directory test: Store pcap files in $LOGDIR instead of /tmp test: Move pidfiles and nsholder sockets into state directory demo: Move pidfiles to state directory test: Move video processing files to $STATEBASE test: Move perf.js report file to $LOGDIR/web .gitignore | 1 - hooks/pre-push | 12 +-- test/.gitignore | 5 -- test/build/all | 31 ++++--- test/demo/passt | 14 ++-- test/demo/pasta | 20 ++--- test/demo/podman | 11 ++- test/distro/debian | 2 +- test/distro/fedora | 2 +- test/distro/opensuse | 16 ++-- test/distro/ubuntu | 8 +- test/lib/context | 61 +++++++------- test/lib/layout | 16 ++-- test/lib/perf_report | 3 +- test/lib/setup | 84 ++++++++++--------- test/lib/term | 26 +++--- test/lib/test | 14 +--- test/lib/video | 38 +++++---- test/{dhcp/passt => passt/dhcp} | 2 +- test/{ndp/passt => passt/ndp} | 2 +- test/{shutdown/passt => passt/shutdown} | 4 +- test/{tcp/passt => passt/tcp} | 6 +- test/{udp/passt => passt/udp} | 8 +- test/{icmp/passt_in_ns => passt_in_ns/icmp} | 2 +- .../passt_in_ns => passt_in_ns/shutdown} | 4 +- test/{tcp/passt_in_ns => passt_in_ns/tcp} | 10 +-- test/{udp/passt_in_ns => passt_in_ns/udp} | 8 +- test/{dhcp/pasta => pasta/dhcp} | 2 +- test/{ndp/pasta => pasta/ndp} | 2 +- test/{tcp/pasta => pasta/tcp} | 13 ++- test/{udp/pasta => pasta/udp} | 20 ++--- test/run | 48 +++++------ 32 files changed, 243 insertions(+), 252 deletions(-) rename test/{dhcp/passt => passt/dhcp} (98%) rename test/{ndp/passt => passt/ndp} (95%) rename test/{shutdown/passt => passt/shutdown} (80%) rename test/{tcp/passt => passt/tcp} (96%) rename test/{udp/passt => passt/udp} (88%) rename test/{icmp/passt_in_ns => passt_in_ns/icmp} (94%) rename test/{shutdown/passt_in_ns => passt_in_ns/shutdown} (79%) rename test/{tcp/passt_in_ns => passt_in_ns/tcp} (97%) rename test/{udp/passt_in_ns => passt_in_ns/udp} (97%) rename test/{dhcp/pasta => pasta/dhcp} (96%) rename test/{ndp/pasta => pasta/ndp} (95%) rename test/{tcp/pasta => pasta/tcp} (95%) rename test/{udp/pasta => pasta/udp} (74%) -- 2.37.3

1 14

[PATCH v3 00/10] Clean up handling of userns
by David Gibson 12 Sep '22

12 Sep '22

Our handling of user namespaces is more complex than it needs to be. This simplifies the handling by identifying and entering (or creating) the correct userns earlier, so that later code doesn't need to deal with it any more. Along the way we make a number of other cleanups to handling of userns and setting our user and group. This is based on my earlier test command dispatch and performance test cleanup series. Changes since v2: * Correction to man page to match behaviour change in 10/10 * Minor changes to spacing and wording of comments Changes since v1: * Fixed overenthusiastic pruning of #includes when moving the self-isolation code which broke compile on some distro versions David Gibson (10): Don't store UID & GID persistently in the context structure Split checking for root from dropping root privilege Consolidate determination of UID/GID to run as Safer handling if we can't open /proc/self/uid_map Move self-isolation code into a separate file Consolidate validation of pasta namespace options Clean up and rename conf_ns_open() Correctly handle --netns-only in pasta_start_ns() Handle userns isolation and dropping root at the same time Allow --userns when pasta spawns a command Makefile | 8 +- conf.c | 236 ++++++++++++++++++++++++++-------------------------- isolation.c | 210 ++++++++++++++++++++++++++++++++++++++++++++++ isolation.h | 15 ++++ passt.1 | 7 +- passt.c | 116 +------------------------- passt.h | 9 -- pasta.c | 91 ++++++++++++-------- pasta.h | 1 + util.c | 83 ------------------ util.h | 2 - 11 files changed, 412 insertions(+), 366 deletions(-) create mode 100644 isolation.c create mode 100644 isolation.h -- 2.37.3

1 10

[PATCH v3 00/11] Improve command dispatch in test scripts
by David Gibson 12 Sep '22

12 Sep '22

Sorry for the resend, but I found and fixed a couple of warts in this series. The test scripts need to dispatch commands in various different contexts: on the host, in namespaces, and in guests. Currently this is done by running each context in a tmux pane and simulating typing into that pane using tmux commands. This has a number of problems: * It's slow * Getting the results from programs is tricky and error prone. We can misparse output if: * The window isn't large enough * Commands produce output which looks like a shell prompt * The shell doesn't have the prompt we expect * A accidental keypress while the tests are running can mess up the test run * We have to issue explicit "echo $?" commands to check if things succeeded This series adds a new subsystem to the test scripts for dispatching commands which accurately preserves output (both stdout and stderr), and exit codes. Most of the testsuite is converted to this new system. For now the distro tests still rely on the old approach, since they have some complications that require additional handling. That does, alas, mean we have some ugly transitional code remaining. Oh well, we should be able to clean that up at some point. This series is pased on the previous series cleaning up the performance tests. Changes since v2: * Whitespace fixes based on review comments * Added proper copyright, license and banner information to nsholder Changes since v1: * Removed a couple of commented debugging lines in patch 9/11 * Removed a no longer necessary pipe-pane in patch 9/11 David Gibson (11): test: Correctly match "background" with "wait" commands test: Context execution helpers test: Allow a tmux pane to watch commands executed in contexts test: Integration of old-style pane execution and new context execution test: Issue host commands via context for most tests test: Use new-style contexts for passt pane in the pasta and passt tests test: Add nsholder utility test: Extend context system to run commands in namespace for pasta tests test: Use context system for guest commands test: Use context system for two_guests tests test: Use new-style command issue for passt_in_ns tests test/.gitignore | 3 + test/Makefile | 13 +- test/lib/context | 125 +++++++++++++++++ test/lib/layout | 61 +++------ test/lib/setup | 274 +++++++++++++++++++------------------- test/lib/term | 84 ++++++++++++ test/lib/test | 138 ++++++++----------- test/nsholder.c | 139 +++++++++++++++++++ test/passt.mbuto | 32 ++++- test/run | 1 + test/shutdown/passt | 4 +- test/shutdown/passt_in_ns | 4 +- test/tcp/passt | 2 + test/tcp/passt_in_ns | 17 ++- test/tcp/pasta | 1 - test/udp/passt | 1 + test/udp/passt_in_ns | 5 +- test/udp/pasta | 1 - 18 files changed, 628 insertions(+), 277 deletions(-) create mode 100644 test/lib/context create mode 100644 test/nsholder.c -- 2.37.3

1 11

[PATCH v2 00/11] Improve command dispatch in test scripts
by David Gibson 10 Sep '22

10 Sep '22

Sorry for the resend, but I found and fixed a couple of warts in this series. The test scripts need to dispatch commands in various different contexts: on the host, in namespaces, and in guests. Currently this is done by running each context in a tmux pane and simulating typing into that pane using tmux commands. This has a number of problems: * It's slow * Getting the results from programs is tricky and error prone. We can misparse output if: * The window isn't large enough * Commands produce output which looks like a shell prompt * The shell doesn't have the prompt we expect * A accidental keypress while the tests are running can mess up the test run * We have to issue explicit "echo $?" commands to check if things succeeded This series adds a new subsystem to the test scripts for dispatching commands which accurately preserves output (both stdout and stderr), and exit codes. Most of the testsuite is converted to this new system. For now the distro tests still rely on the old approach, since they have some complications that require additional handling. That does, alas, mean we have some ugly transitional code remaining. Oh well, we should be able to clean that up at some point. This series is pased on the previous series cleaning up the performance tests. Changes since v1: * Removed a couple of commented debugging lines in patch 9/11 * Removed a no longer necessary pipe-pane in patch 9/11 David Gibson (11): test: Correctly match "background" with "wait" commands test: Context execution helpers test: Allow a tmux pane to watch commands executed in contexts test: Integration of old-style pane execution and new context execution test: Issue host commands via context for most tests test: Use new-style contexts for passt pane in the pasta and passt tests test: Add nsholder utility test: Extend context system to run commands in namespace for pasta tests test: Use context system for guest commands test: Use context system for two_guests tests test: Use new-style command issue for passt_in_ns tests test/.gitignore | 3 + test/Makefile | 13 +- test/lib/context | 125 +++++++++++++++++ test/lib/layout | 61 +++------ test/lib/setup | 274 +++++++++++++++++++------------------- test/lib/term | 84 ++++++++++++ test/lib/test | 138 ++++++++----------- test/nsholder.c | 117 ++++++++++++++++ test/passt.mbuto | 32 ++++- test/run | 1 + test/shutdown/passt | 4 +- test/shutdown/passt_in_ns | 4 +- test/tcp/passt | 2 + test/tcp/passt_in_ns | 17 ++- test/tcp/pasta | 1 - test/udp/passt | 1 + test/udp/passt_in_ns | 5 +- test/udp/pasta | 1 - 18 files changed, 606 insertions(+), 277 deletions(-) create mode 100644 test/lib/context create mode 100644 test/nsholder.c -- 2.37.3

2 22

passt (and KubeVirt) at KVM Forum 2022
by Stefano Brivio 09 Sep '22

09 Sep '22

Hi, passt will be presented, along with its proposed KubeVirt usage, at the KVM Forum (https://events.linuxfoundation.org/kvm-forum/) next week! Presentation deck: https://static.sched.com/hosted_files/kvmforum2022/d7/Slirp%20is%20dead%2C%… Event details: https://kvmforum2022.sched.com/event/15jJY/slirp-is-dead-long-live-slirp-a-… -- Stefano

1 1

[PATCH v3 00/12] Improve handling of test temporary files
by David Gibson 08 Sep '22

08 Sep '22

Urgh.. so easy to miss dumb errors working in shell The tests create temporary files and fifos in a number of places. Some of them can interfere with later test runs, and are awkward to clean up. Consolidate them in a single per-run directory in /tmp which gets cleaned up automatically. This is based on the earlier userns cleanup series. Changes since v2: * Reinstated patch to move the log pipe accidentally dropped during a rebase Chances since v1: * Fixed a number of straightforward bugs where things were missed * Also removed files we're no longer putting into the source tree from .gitignore * Added an extra patch moving the video processing files David Gibson (12): test: Group tests by context then protocol, rather than the reverse test: Remove unused variable FFMPEG_PID_FILE test: Actually run cleanup function test: Create common state directories for temporary files test: Move passt_test_log_pipe to state directory test: Move context temporary files to state dir test: Dont regnerate small test file in pasta/tcp test: Use paths in __STATEDIR__ instead of 'temp' and 'tempdir' directives test: Move pause temporary file to state directory test: Store pcap files in $LOGDIR instead of /tmp test: Move pidfiles and nsholder sockets into state directory test: Move video processing files to $STATEBASE .gitignore | 1 - test/.gitignore | 3 - test/build/all | 31 ++++--- test/demo/passt | 9 +- test/demo/pasta | 12 ++- test/demo/podman | 11 ++- test/distro/debian | 2 +- test/distro/fedora | 2 +- test/distro/opensuse | 16 ++-- test/distro/ubuntu | 8 +- test/lib/context | 61 +++++++------- test/lib/layout | 16 ++-- test/lib/setup | 84 ++++++++++--------- test/lib/term | 26 +++--- test/lib/test | 14 +--- test/lib/video | 20 ++--- test/{dhcp/passt => passt/dhcp} | 2 +- test/{ndp/passt => passt/ndp} | 2 +- test/{shutdown/passt => passt/shutdown} | 4 +- test/{tcp/passt => passt/tcp} | 6 +- test/{udp/passt => passt/udp} | 8 +- test/{icmp/passt_in_ns => passt_in_ns/icmp} | 2 +- .../passt_in_ns => passt_in_ns/shutdown} | 4 +- test/{tcp/passt_in_ns => passt_in_ns/tcp} | 10 +-- test/{udp/passt_in_ns => passt_in_ns/udp} | 8 +- test/{dhcp/pasta => pasta/dhcp} | 2 +- test/{ndp/pasta => pasta/ndp} | 2 +- test/{tcp/pasta => pasta/tcp} | 13 ++- test/{udp/pasta => pasta/udp} | 20 ++--- test/run | 48 +++++------ 30 files changed, 217 insertions(+), 230 deletions(-) rename test/{dhcp/passt => passt/dhcp} (98%) rename test/{ndp/passt => passt/ndp} (95%) rename test/{shutdown/passt => passt/shutdown} (80%) rename test/{tcp/passt => passt/tcp} (96%) rename test/{udp/passt => passt/udp} (88%) rename test/{icmp/passt_in_ns => passt_in_ns/icmp} (94%) rename test/{shutdown/passt_in_ns => passt_in_ns/shutdown} (79%) rename test/{tcp/passt_in_ns => passt_in_ns/tcp} (97%) rename test/{udp/passt_in_ns => passt_in_ns/udp} (97%) rename test/{dhcp/pasta => pasta/dhcp} (96%) rename test/{ndp/pasta => pasta/ndp} (95%) rename test/{tcp/pasta => pasta/tcp} (95%) rename test/{udp/pasta => pasta/udp} (74%) -- 2.37.3

1 12