- dev - passt

[PATCH v2 0/4] Clean up reset of tap connections
by David Gibson 07 Aug '23

07 Aug '23

I ran into some mildly confusing stuff in the tap reset path while working on the epoll rework. Here are some fixes for it. Changes since v1: * Two extra patches that further clean up the reset path David Gibson (4): tap: Clean up tap reset path tap: More sensible behaviour for error on listening qemu socket tap: Fold reset handling into tap_handler_pasta() tap: Fold reset handling into tap_handler_passt() tap.c | 94 ++++++++++++++++++++++++++++++++--------------------------- 1 file changed, 51 insertions(+), 43 deletions(-) -- 2.41.0

1 4

[PATCH 0/3] Better report errors failing to open namespace tap device
by David Gibson 04 Aug '23

04 Aug '23

In https://github.com/containers/podman/issues/19428, pasta is failing to open the namespace tap device. Paul Holzinger correctly noted that pasta isn't very helpful in this case, with no information beyond "it failed". He suggested a patch for that, however it wasn't quite sufficient: errno may not be propagated back from the ephemeral thread which enters the namespace, and even if it does the errno alone won't tell us which of the possible failure points actually failed. 2/3 here is a more robust change to address the problem. The other patches are minor cleanups I noticed along the way. Link: https://bugs.passt.top/show_bug.cgi?id=69 David Gibson (3): util: Make ns_enter() a void function and report setns() errors tap: More detailed error reporting in tap_ns_tun() tap: Remove unnecessary global tun_ns_fd conf.c | 3 ++- tap.c | 33 ++++++++++++++++++--------------- udp.c | 6 ++---- util.c | 8 +++----- util.h | 2 +- 5 files changed, 26 insertions(+), 26 deletions(-) -- 2.41.0

2 5

[PATCH 0/2] Clean up reset of tap connections
by David Gibson 04 Aug '23

04 Aug '23

I ran into some mildly confusing stuff in the tap reset path while working on the epoll rework. Here's a couple of fixes for it. David Gibson (2): tap: Clean up tap reset path tap: More sensible behaviour for error on listening qemu socket tap.c | 57 ++++++++++++++++++++++++++++++++------------------------- 1 file changed, 32 insertions(+), 25 deletions(-) -- 2.41.0

1 2

[PATCH v2 00/17] netlink fixes and cleanups
by David Gibson 04 Aug '23

04 Aug '23

We've had several bugs in the past that were quite tricky to debug, but would have been much easier if we'd known that a netlink operation had failed. So, it would be desirable to actually detect and report failures of netlink operations. While working on that, I discovered that there are a number of other issues ranging from very small to medium sized with the way we use netlink. This series addresses many of them. Link: https://bugs.passt.top/show_bug.cgi?id=60 Link: https://bugs.passt.top/show_bug.cgi?id=67 This series was tested as based on the pending patches adding C11 support, though I believe it trivially rebases onto current main. Changes since v1: * Assorted minor fixes based on Stefano's review * Rebased on C11 branch David Gibson (17): netlink: Split up functionality of nl_link() netlink: Split nl_addr() into separate operation functions netlink: Split nl_route() into separate operation functions netlink: Use struct in_addr for IPv4 addresses, not bare uint32_t netlink: Explicitly pass netlink sockets to operations netlink: Make nl_*_dup() use a separate datagram for each request netlink: Start sequence number from 1 instead of 0 netlink: Treat send() or recv() errors as fatal netlink: Fill in netlink header fields from nl_req() netlink: Add nl_do() helper for simple operations with error checking netlink: Clearer reasoning about the netlink response buffer size netlink: Split nl_req() to allow processing multiple response datagrams netlink: Add nl_foreach_oftype to filter response message types netlink: Propagate errors for "set" operations netlink: Always process all responses to a netlink request netlink: Propagate errors for "dump" operations netlink: Propagate errors for "dup" operations conf.c | 66 ++++- netlink.c | 843 ++++++++++++++++++++++++++++++++++-------------------- netlink.h | 27 +- pasta.c | 75 +++-- 4 files changed, 658 insertions(+), 353 deletions(-) -- 2.41.0

2 18

[PATCH 0/3] RFC: Allow C11 extensions in the passt/pasta code
by David Gibson 04 Aug '23

04 Aug '23

As discussed on our recent calls, the C11 standard introduces anonymous structure and union members and static assertions, amongst other things. Both of these could be useful in a few places in passt/pasta to make the code more readable and safer. However, at the moment, the compiler flags we use only allow C99 code. This series allows C11 code, and makes some fairly obvious cleanups by using it. It would be nice to get an opinion on this reasonably quickly, because I have other patches in the works that will look different depending on whether or not they can use C11 features. David Gibson (3): Allow C11 code, not just C99 code Use C11 anonymous members to make poll refs less verbose to use Use static assertion to verify that union epoll_ref is the right size Makefile | 4 ++-- icmp.c | 22 +++++++++++----------- icmp.h | 2 +- passt.c | 8 ++++---- passt.h | 8 ++++++-- tcp.c | 46 +++++++++++++++++++++++----------------------- tcp.h | 2 +- tcp_splice.c | 11 +++++------ udp.c | 50 +++++++++++++++++++++++--------------------------- udp.h | 2 +- util.c | 4 ++-- 11 files changed, 79 insertions(+), 80 deletions(-) -- 2.41.0

2 7

[PATCH 00/17] netlink fixes and cleanups
by David Gibson 03 Aug '23

03 Aug '23

We've had several bugs in the past that were quite tricky to debug, but would have been much easier if we'd known that a netlink operation had failed. So, it would be desirable to actually detect and report failures of netlink operations. While working on that, I discovered that there are a number of other issues ranging from very small to medium sized with the way we use netlink. This series addresses many of them. Link: https://bugs.passt.top/show_bug.cgi?id=60 Link: https://bugs.passt.top/show_bug.cgi?id=67 David Gibson (17): netlink: Split up functionality if nl_link() netlink: Split nl_addr() into separate operation functions netlink: Split nl_route() into separate operation functions netlink: Use struct in_addr for IPv4 addresses, not bare uint32_t netlink: Explicitly pass netlink sockets to operations netlink: Make nl_*_dup() use a separate datagram for each request netlink: Start sequence number from 1 instead of 0 netlink: Treat send() or recv() errors as fatal netlink: Fill in netlink header fields from nl_req() netlink: Add nl_do() helper for simple operations with error checking netlink: Clearer reasoning about the netlink response buffer size netlink: Split nl_req() to allow processing multiple response datagrams netlink: Add nl_foreach_oftype to filter response message types netlink: Propagate errors for "set" operations netlink: Always process all responses to a netlink request netlink: Propagate errors for "dump" operations netlink: Propagate errors for "dup" operations conf.c | 66 ++++- netlink.c | 844 ++++++++++++++++++++++++++++++++++-------------------- netlink.h | 27 +- pasta.c | 75 +++-- 4 files changed, 659 insertions(+), 353 deletions(-) -- 2.41.0

2 34

[PATCH] tap: include errno in error when tap_ns_tun() fails
by Paul Holzinger 03 Aug '23

03 Aug '23

It is important to know why a syscall failed so pasta should include the errno in the error message. This is still not perfect as we do not know which of functions (open, ioctl, if_nametoindex) failed but it should at least include more important context. This change was inspiered by a podman issue[1]. [1] https://github.com/containers/podman/issues/19428 Signed-off-by: Paul Holzinger <pholzing(a)redhat.com> --- tap.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/tap.c b/tap.c index a6a73d3..c212616 100644 --- a/tap.c +++ b/tap.c @@ -1205,7 +1205,8 @@ static void tap_sock_tun_init(struct ctx *c) NS_CALL(tap_ns_tun, c); if (tun_ns_fd == -1) - die("Failed to open tun socket in namespace"); + die("Failed to open tun socket in namespace: %s", + strerror(errno)); pasta_ns_conf(c); -- 2.41.0

2 3

[PATCH 0/8] RFC: Generalize flow tracking, part 1
by David Gibson 28 Jul '23

28 Jul '23

This is a first draft of the first part of implementing a more general flow table (connection tracking) as described at: https://pad.passt.top/p/NewForwardingModel This is by no means complete. So far it doesn't really do anything new, it just reorganizes the TCP connection table to be closer to the more general flow table. Still it's ready for preliminary review. David Gibson (8): tap: Don't clobber source address in tap6_handler() tap: Pass source address to protocol handler functions tcp: More precise terms for addresses and ports tcp, udp: Don't include destination address in partially precomputed csums tcp, udp: Don't pre-fill IPv4 destination address in headers tcp: Track guest-side correspondent address tcp, flow: Introduce struct demiflow tcp, flow: Perform TCP hash calculations based on demiflow structure flow.h | 66 +++++++++++++++ icmp.c | 12 ++- icmp.h | 3 +- passt.c | 10 +-- passt.h | 4 +- pasta.c | 2 +- siphash.c | 1 + tap.c | 29 ++++--- tcp.c | 227 ++++++++++++++++++++------------------------------- tcp.h | 5 +- tcp_conn.h | 9 +- tcp_splice.c | 2 + udp.c | 37 +++------ udp.h | 5 +- util.h | 4 +- 15 files changed, 209 insertions(+), 207 deletions(-) create mode 100644 flow.h -- 2.41.0

1 8

ip_nonlocal_bind causes havoc with local connection detection
by Valtteri Vuorikoski 20 Jul '23

20 Jul '23

If net.ipv4.ip_nonlocal_bind is enabled, the following code in tcp_conn_from_tap gets very confused: if (!bind(s, sa, sl)) { tcp_rst(c, conn); /* Nobody is listening then */ return; } if (errno != EADDRNOTAVAIL && errno != EACCES) conn_flag(c, conn, LOCAL); This is especially visible if net.ipv4.ip_unprivileged_port_start is set to a value lower than the default. For example, if net.ipv4.ip_unprivileged_port_start=443 and net.ipv4.ip_nonlocal_bind=1, the bind()==0 branch will be hit for all outgoing connections going to port 443 because bind() succeeds even when "sa" contains the remote address, and pretty much nothing will work. It might the best to skip the check and marking connections as LOCAL if net.ipv4.ip_nonlocal_bind is enabled? If that doesn't seem reasonable, then maybe show a warning at start and/or just document that the ip_nonlocal_bind setting shouldn't be used with passt? -Valtteri

2 2

[PATCH 00/27] RFC: Start converting passt & pasta tests to Avocado using special plugin
by David Gibson 10 Jul '23

10 Jul '23

I've sent previous series converting some of the passt & pasta testsuite to use Avocado. However the jUnit-like structure of avocado tests wasn't a great fit for our needs (plus Stefano and I just dislike the style). Here's another attempt, where instead we use a new "avocado-classless" plugin to avoid jUnit-ism. This is definitely a work-in-progress. Some early observations: * It is noticeably less verbose than jUnit style * It does avoid some of the confusing stuff that arises from jUnit style * The present draft has some different pretty confusing stuff, involving nested and stacked function decorators, and some other non-entirely-obvious use of higher order functions * I hope I can improve this at least a bit, but it will need some thought * I was pleasantly surprised at how simple it was to write the plugin itself David Gibson (27): avocado: Make a duplicate copy of testsuite for comparison purposes avocado: Don't double download assets for test/ and oldtest/ avocado: Move static checkers to avocado avocado: Introduce "avocado-classless" plugin, runner and outline avocado, test: Add static checkers for Python code avocado: Resolver implementation for avocado-classless plugin avocado: Add basic assertion helpers to avocado-classless plugin tasst, avocado: Introduce library of common test helpers avocado-classless: Test matrices by composition tasst: Helper functions for executing commands in different places avocado-classless: Allow overriding default timeout avocado: Convert build tests to avocado tasst: Add helpers for running background commands on sites tasst: Add helper to get network interface names for a site tasst: Add helpers to run commands with nstool tasst: Add ifup and network address helpers to Site tasst: Helper for creating veth devices between namespaces tasst: Add helper for getting MTU of a network interface tasst: Add helper to wait for IP address to appear tasst: Add helpers for getting a site's routes tasst: Helpers to test transferring data between sites tasst: IP address allocation helpers tasst: Helpers for running daemons with a pidfile tasst: Helpers for testing NDP behaviour tasst: Helpers for testing DHCP & DHCPv6 behaviour tasst: Helpers to construct a simple network environment for tests avocado: Convert basic pasta tests Makefile | 9 + oldtest/.gitignore | 11 + oldtest/Makefile | 119 +++ oldtest/README.md | 137 +++ {test => oldtest}/build/all | 0 {test => oldtest}/build/clang_tidy | 0 {test => oldtest}/build/cppcheck | 0 oldtest/ci | 1 + oldtest/demo/passt | 245 ++++++ oldtest/demo/pasta | 274 ++++++ oldtest/demo/podman | 819 ++++++++++++++++++ oldtest/distro/debian | 252 ++++++ oldtest/distro/fedora | 396 +++++++++ oldtest/distro/opensuse | 208 +++++ oldtest/distro/ubuntu | 216 +++++ oldtest/env/mate-terminal.profile | 42 + oldtest/find-arm64-firmware.sh | 13 + oldtest/lib/context | 130 +++ oldtest/lib/layout | 259 ++++++ oldtest/lib/layout_ugly | 113 +++ oldtest/lib/perf_report | 272 ++++++ oldtest/lib/setup | 385 ++++++++ oldtest/lib/setup_ugly | 58 ++ oldtest/lib/term | 750 ++++++++++++++++ oldtest/lib/test | 398 +++++++++ oldtest/lib/util | 133 +++ oldtest/lib/video | 152 ++++ oldtest/memory/passt | 187 ++++ oldtest/nstool.c | 565 ++++++++++++ oldtest/passt.mbuto | 83 ++ oldtest/passt.mem.mbuto | 44 + oldtest/passt/dhcp | 70 ++ oldtest/passt/ndp | 33 + oldtest/passt/shutdown | 19 + oldtest/passt/tcp | 76 ++ oldtest/passt/udp | 46 + oldtest/passt_in_ns/icmp | 32 + oldtest/passt_in_ns/shutdown | 19 + oldtest/passt_in_ns/tcp | 256 ++++++ oldtest/passt_in_ns/udp | 138 +++ {test => oldtest}/pasta/dhcp | 0 {test => oldtest}/pasta/ndp | 0 {test => oldtest}/pasta/tcp | 0 {test => oldtest}/pasta/udp | 0 oldtest/pasta_options/log_to_file | 93 ++ oldtest/perf/passt_tcp | 215 +++++ oldtest/perf/passt_udp | 165 ++++ oldtest/perf/pasta_tcp | 300 +++++++ oldtest/perf/pasta_udp | 219 +++++ oldtest/prepare-distro-img.sh | 18 + oldtest/run | 238 +++++ oldtest/run_demo | 1 + oldtest/two_guests/basic | 80 ++ oldtest/valgrind.supp | 9 + test/.gitignore | 2 + test/Makefile | 69 +- test/avocado/build.py | 94 ++ test/avocado/pasta.py | 129 +++ test/avocado/static_checkers/clang-tidy.sh | 3 + test/avocado/static_checkers/cppcheck.sh | 3 + test/avocado/static_checkers/flake8.sh | 3 + test/avocado/static_checkers/pylint.sh | 3 + test/avocado_classless/.gitignore | 1 + .../avocado_classless/__init__.py | 11 + .../avocado_classless/manifest.py | 48 + .../avocado_classless/plugin.py | 231 +++++ .../avocado_classless/test.py | 80 ++ test/avocado_classless/examples.py | 57 ++ test/avocado_classless/selftests.py | 76 ++ test/avocado_classless/setup.py | 32 + test/lib/layout | 31 - test/lib/setup | 47 - test/run | 14 - test/tasst/__init__.py | 11 + test/tasst/address.py | 80 ++ test/tasst/dhcp.py | 114 +++ test/tasst/dhcpv6.py | 74 ++ test/tasst/exesite.py | 285 ++++++ test/tasst/meta/__init__.py | 16 + test/tasst/meta/static_ifup.py | 60 ++ test/tasst/meta/veth.py | 86 ++ test/tasst/ndp.py | 99 +++ test/tasst/nstool.py | 184 ++++ test/tasst/pasta.py | 42 + test/tasst/scenario/__init__.py | 12 + test/tasst/scenario/simple.py | 98 +++ test/tasst/transfer.py | 174 ++++ test/tasst/typecheck.py | 47 + 88 files changed, 10518 insertions(+), 96 deletions(-) create mode 100644 oldtest/.gitignore create mode 100644 oldtest/Makefile create mode 100644 oldtest/README.md rename {test => oldtest}/build/all (100%) rename {test => oldtest}/build/clang_tidy (100%) rename {test => oldtest}/build/cppcheck (100%) create mode 120000 oldtest/ci create mode 100644 oldtest/demo/passt create mode 100644 oldtest/demo/pasta create mode 100644 oldtest/demo/podman create mode 100644 oldtest/distro/debian create mode 100644 oldtest/distro/fedora create mode 100644 oldtest/distro/opensuse create mode 100644 oldtest/distro/ubuntu create mode 100644 oldtest/env/mate-terminal.profile create mode 100755 oldtest/find-arm64-firmware.sh create mode 100644 oldtest/lib/context create mode 100644 oldtest/lib/layout create mode 100644 oldtest/lib/layout_ugly create mode 100755 oldtest/lib/perf_report create mode 100755 oldtest/lib/setup create mode 100755 oldtest/lib/setup_ugly create mode 100755 oldtest/lib/term create mode 100755 oldtest/lib/test create mode 100755 oldtest/lib/util create mode 100755 oldtest/lib/video create mode 100644 oldtest/memory/passt create mode 100644 oldtest/nstool.c create mode 100755 oldtest/passt.mbuto create mode 100755 oldtest/passt.mem.mbuto create mode 100644 oldtest/passt/dhcp create mode 100644 oldtest/passt/ndp create mode 100644 oldtest/passt/shutdown create mode 100644 oldtest/passt/tcp create mode 100644 oldtest/passt/udp create mode 100644 oldtest/passt_in_ns/icmp create mode 100644 oldtest/passt_in_ns/shutdown create mode 100644 oldtest/passt_in_ns/tcp create mode 100644 oldtest/passt_in_ns/udp rename {test => oldtest}/pasta/dhcp (100%) rename {test => oldtest}/pasta/ndp (100%) rename {test => oldtest}/pasta/tcp (100%) rename {test => oldtest}/pasta/udp (100%) create mode 100644 oldtest/pasta_options/log_to_file create mode 100644 oldtest/perf/passt_tcp create mode 100644 oldtest/perf/passt_udp create mode 100644 oldtest/perf/pasta_tcp create mode 100644 oldtest/perf/pasta_udp create mode 100755 oldtest/prepare-distro-img.sh create mode 100755 oldtest/run create mode 120000 oldtest/run_demo create mode 100644 oldtest/two_guests/basic create mode 100644 oldtest/valgrind.supp create mode 100644 test/avocado/build.py create mode 100644 test/avocado/pasta.py create mode 100755 test/avocado/static_checkers/clang-tidy.sh create mode 100755 test/avocado/static_checkers/cppcheck.sh create mode 100755 test/avocado/static_checkers/flake8.sh create mode 100755 test/avocado/static_checkers/pylint.sh create mode 100644 test/avocado_classless/.gitignore create mode 100644 test/avocado_classless/avocado_classless/__init__.py create mode 100644 test/avocado_classless/avocado_classless/manifest.py create mode 100644 test/avocado_classless/avocado_classless/plugin.py create mode 100644 test/avocado_classless/avocado_classless/test.py create mode 100644 test/avocado_classless/examples.py create mode 100644 test/avocado_classless/selftests.py create mode 100644 test/avocado_classless/setup.py create mode 100644 test/tasst/__init__.py create mode 100644 test/tasst/address.py create mode 100644 test/tasst/dhcp.py create mode 100644 test/tasst/dhcpv6.py create mode 100644 test/tasst/exesite.py create mode 100644 test/tasst/meta/__init__.py create mode 100644 test/tasst/meta/static_ifup.py create mode 100644 test/tasst/meta/veth.py create mode 100644 test/tasst/ndp.py create mode 100644 test/tasst/nstool.py create mode 100644 test/tasst/pasta.py create mode 100644 test/tasst/scenario/__init__.py create mode 100644 test/tasst/scenario/simple.py create mode 100644 test/tasst/transfer.py create mode 100644 test/tasst/typecheck.py -- 2.41.0

2 31