- dev - passt

passt: new version 2025_12_10.d04c480 available
by Stefano Brivio 10 Dec '25

10 Dec '25

Subject: passt: new version 2025_12_10.d04c480 available The new version with tag 2025_12_10.d04c480 includes the following changes: d04c480 pif: Correctly set scope_id for guest-side link local addresses 696709d tcp: Correct timer expiry value in trace message https://passt.top/passt/log/?qt=range&q=2025_12_09.c3f1ba7..2025_12_10.d04c… Packages: - Alpine Linux: https://pkgs.alpinelinux.org/packages?name=passt - Arch Linux: https://www.archlinux.org/packages/extra/x86_64/passt/ https://archlinuxarm.org/packages/aarch64/passt https://archlinuxarm.org/packages/armv7h/passt - Chimera: https://pkgs.chimera-linux.org/packages?name=passt - Clear Linux: https://github.com/clearlinux-pkgs/passt/ - Copr (CentOS Stream, EPEL, Fedora, Mageia): https://copr.fedorainfracloud.org/coprs/sbrivio/passt/build/9896122/ permanent mirror: https://passt.top/builds/copr/0^20251210.gd04c480/ - Debian tracker: https://tracker.debian.org/pkg/passt - Fedora updates: https://bodhi.fedoraproject.org/updates/?packages=passt - Gentoo versions: https://packages.gentoo.org/packages/net-misc/passt - GNU Guix: https://packages.guix.gnu.org/packages/passt/ - Homebrew: https://formulae.brew.sh/formula/passt - NixOS: https://github.com/NixOS/nixpkgs/tree/nixos-unstable/pkgs/by-name/pa/passt - openSUSE: https://software.opensuse.org/package/passt - OpenMandriva: https://github.com/OpenMandrivaAssociation/passt/tree/master - PLD Linux: https://git.pld-linux.org/cgi-bin/gitweb.cgi?p=packages/passt.git - Solus: https://github.com/getsolus/packages/tree/main/packages/p/passt - Ubuntu tracker: https://launchpad.net/ubuntu/+source/passt - Void Linux: https://voidlinux.org/packages/?q=passt - Static builds: - Package for other RPM-based distributions, x86_64 only: https://passt.top/builds/latest/x86_64/passt-gd04c480-1.x86_64.rpm - x86_64 static binaries: https://passt.top/builds/latest/x86_64/ - Debian package, from x86_64 static build: https://passt.top/builds/latest/x86_64/passt_0d04c480-1_all.deb -- Stefano

1 0

[PATCH 1/2] treewide: Introduce passt_exit() helper
by David Gibson 10 Dec '25

10 Dec '25

In d0006fa78 ("treewide: use _exit() over exit()"), we replaced use of the normal exit(3) with direct calls to _exit(2). That was because glibc exit(3) made some unexpected futex() calls, which hit our seccomp profile. We've since had some bugs due to missing the extra cleanup that exit(3) implemented, for which we've added explicit cleanup calls. Specifically, we have fflush() calls in some places to avoid leaving incomplete messages on stdout/stderr, and in other places fsync_pcap_and_log() to avoid leaving incomplete log or pcap files. It's easy to forget these when adding new error paths, so instead, implement our own passt_exit() wrapper to perform vital cleanup then call _exit(2). This also provides an obvious place to add any additional cleanups we discover we need in future. Signed-off-by: David Gibson <david(a)gibson.dropbear.id.au> --- conf.c | 11 ++++------- log.h | 9 +++++++-- passt.c | 5 ++--- pasta.c | 10 ++++------ tap.c | 6 ++---- util.c | 24 ++++++++++++++++++------ util.h | 1 - vhost_user.c | 4 ++-- 8 files changed, 39 insertions(+), 31 deletions(-) diff --git a/conf.c b/conf.c index fdc19e81..24b44419 100644 --- a/conf.c +++ b/conf.c @@ -826,7 +826,7 @@ static void conf_ip6_local(struct ip6_ctx *ip6) * usage() - Print usage, exit with given status code * @name: Executable name * @f: Stream to print usage info to - * @status: Status code for _exit() + * @status: Status code for exit(2) */ static void usage(const char *name, FILE *f, int status) { @@ -997,8 +997,7 @@ static void usage(const char *name, FILE *f, int status) " SPEC is as described for TCP above\n" " default: none\n"); - (void)fflush(f); - _exit(status); + passt_exit(status); pasta_opts: @@ -1052,8 +1051,7 @@ pasta_opts: " --ns-mac-addr ADDR Set MAC address on tap interface\n" " --no-splice Disable inbound socket splicing\n"); - (void)fflush(f); - _exit(status); + passt_exit(status); } /** @@ -1621,8 +1619,7 @@ void conf(struct ctx *c, int argc, char **argv) FPRINTF(stdout, c->mode == MODE_PASTA ? "pasta " : "passt "); FPRINTF(stdout, VERSION_BLOB); - (void)fflush(stdout); - _exit(EXIT_SUCCESS); + passt_exit(EXIT_SUCCESS); case 15: ret = snprintf(c->ip4.ifname_out, sizeof(c->ip4.ifname_out), "%s", optarg); diff --git a/log.h b/log.h index c8473c0d..b7b20672 100644 --- a/log.h +++ b/log.h @@ -9,6 +9,11 @@ #include <stdbool.h> #include <syslog.h> +/* This would make more sense in util.h, but because we use it in die(), that + * would cause awkward circular reference problems. + */ +void passt_exit(int status) __attribute__((noreturn)); + #define LOGFILE_SIZE_DEFAULT (1024 * 1024UL) #define LOGFILE_CUT_RATIO 30 /* When full, cut ~30% size */ #define LOGFILE_SIZE_MIN (5UL * MAX(BUFSIZ, PAGE_SIZE)) @@ -32,13 +37,13 @@ void logmsg_perror(int pri, const char *format, ...) #define die(...) \ do { \ err(__VA_ARGS__); \ - _exit(EXIT_FAILURE); \ + passt_exit(EXIT_FAILURE); \ } while (0) #define die_perror(...) \ do { \ err_perror(__VA_ARGS__); \ - _exit(EXIT_FAILURE); \ + passt_exit(EXIT_FAILURE); \ } while (0) extern int log_file; diff --git a/passt.c b/passt.c index 5ed88d07..cf38822f 100644 --- a/passt.c +++ b/passt.c @@ -177,8 +177,7 @@ static void exit_handler(int signal) { (void)signal; - fsync_pcap_and_log(); - _exit(EXIT_SUCCESS); + passt_exit(EXIT_SUCCESS); } /** @@ -399,7 +398,7 @@ int main(int argc, char **argv) flow_init(); if ((!c.no_udp && udp_init(&c)) || (!c.no_tcp && tcp_init(&c))) - _exit(EXIT_FAILURE); + passt_exit(EXIT_FAILURE); proto_update_l2_buf(c.guest_mac); diff --git a/pasta.c b/pasta.c index 674b5542..5c693de1 100644 --- a/pasta.c +++ b/pasta.c @@ -70,15 +70,13 @@ void pasta_child_handler(int signal) if (pasta_child_pid && !waitid(P_PID, pasta_child_pid, &infop, WEXITED | WNOHANG)) { if (infop.si_pid == pasta_child_pid) { - fsync_pcap_and_log(); - if (infop.si_code == CLD_EXITED) - _exit(infop.si_status); + passt_exit(infop.si_status); /* If killed by a signal, si_status is the number. * Follow common shell convention of returning it + 128. */ - _exit(infop.si_status + 128); + passt_exit(infop.si_status + 128); /* Nothing to do, detached PID namespace going away */ } @@ -511,7 +509,7 @@ void pasta_netns_quit_inotify_handler(struct ctx *c, int inotify_fd) if (!strncmp(ev->name, c->netns_base, sizeof(c->netns_base))) { info("Namespace %s is gone, exiting", c->netns_base); - _exit(EXIT_SUCCESS); + passt_exit(EXIT_SUCCESS); } } } @@ -539,7 +537,7 @@ void pasta_netns_quit_timer_handler(struct ctx *c, union epoll_ref ref) return; info("Namespace %s is gone, exiting", c->netns_base); - _exit(EXIT_SUCCESS); + passt_exit(EXIT_SUCCESS); } close(fd); diff --git a/tap.c b/tap.c index e3ea61cb..9d1344b2 100644 --- a/tap.c +++ b/tap.c @@ -1149,10 +1149,8 @@ void tap_sock_reset(struct ctx *c) { info("Client connection closed%s", c->one_off ? ", exiting" : ""); - if (c->one_off) { - fsync_pcap_and_log(); - _exit(EXIT_SUCCESS); - } + if (c->one_off) + passt_exit(EXIT_SUCCESS); /* Close the connected socket, wait for a new connection */ epoll_del(c->epollfd, c->fd_tap); diff --git a/util.c b/util.c index bfeb6191..e266c396 100644 --- a/util.c +++ b/util.c @@ -555,7 +555,7 @@ void pidfile_write(int fd, pid_t pid) if (write(fd, pid_buf, n) < 0) { perror("PID file write"); - _exit(EXIT_FAILURE); + passt_exit(EXIT_FAILURE); } close(fd); @@ -592,12 +592,12 @@ int __daemon(int pidfile_fd, int devnull_fd) if (pid == -1) { perror("fork"); - _exit(EXIT_FAILURE); + passt_exit(EXIT_FAILURE); } if (pid) { pidfile_write(pidfile_fd, pid); - _exit(EXIT_SUCCESS); + passt_exit(EXIT_SUCCESS); } if (setsid() < 0 || @@ -605,7 +605,7 @@ int __daemon(int pidfile_fd, int devnull_fd) dup2(devnull_fd, STDOUT_FILENO) < 0 || dup2(devnull_fd, STDERR_FILENO) < 0 || close(devnull_fd)) - _exit(EXIT_FAILURE); + passt_exit(EXIT_FAILURE); return 0; } @@ -1225,17 +1225,29 @@ void abort_with_msg(const char *fmt, ...) } /** - * fsync_pcap_and_log() - Flush pcap and log files as needed + * passt_exit() - Perform vital cleanup and exit + * + * We don't use exit(3) because on some C library versions it can do unexpected + * things that hit our seccomp profile (e.g. futex() calls). This is a bespoke + * wrapper around _exit(2) performing just the cleanup that we need. * * #syscalls fsync */ -void fsync_pcap_and_log(void) +void passt_exit(int status) { + /* Make sure we don't leave the pcap file truncated */ if (pcap_fd != -1 && fsync(pcap_fd)) warn_perror("Failed to flush pcap file, it might be truncated"); + /* Make sure we don't leave an incomplete log */ if (log_file != -1) (void)fsync(log_file); + + /* Make sure we dont leave any messages incomplete */ + (void)fflush(stderr); + (void)fflush(stdout); + + _exit(status); } /** diff --git a/util.h b/util.h index f7a941fd..4cbb5da3 100644 --- a/util.h +++ b/util.h @@ -242,7 +242,6 @@ int read_all_buf(int fd, void *buf, size_t len); int read_remainder(int fd, const struct iovec *iov, size_t cnt, size_t skip); void close_open_files(int argc, char **argv); bool snprintf_check(char *str, size_t size, const char *format, ...); -void fsync_pcap_and_log(void); long clamped_scale(long x, long y, long lo, long hi, long f); /** diff --git a/vhost_user.c b/vhost_user.c index aa7c869d..f9a5646c 100644 --- a/vhost_user.c +++ b/vhost_user.c @@ -60,7 +60,7 @@ void vu_print_capabilities(void) info("{"); info(" \"type\": \"net\""); info("}"); - _exit(EXIT_SUCCESS); + passt_exit(EXIT_SUCCESS); } /** @@ -1202,7 +1202,7 @@ void vu_control_handler(struct vu_dev *vdev, int fd, uint32_t events) !vdev->context->migrate_target) { if (vdev->context->migrate_exit) { info("Migration complete, exiting"); - _exit(EXIT_SUCCESS); + passt_exit(EXIT_SUCCESS); } info("Migration complete"); -- 2.52.0

2 1

[PATCH 0/2] Bug 181 & bug 182
by David Gibson 10 Dec '25

10 Dec '25

These are fixes for two regressions in the latest release. David Gibson (2): tcp: Correct timer expiry value in trace message pif: Correctly set scope_id for guest-side link local addresses pif.c | 10 +++++++--- tcp.c | 10 +++++----- 2 files changed, 12 insertions(+), 8 deletions(-) -- 2.52.0

2 1

[PATCH 2/2] pasta: Clean up waiting pasta child on failures
by David Gibson 10 Dec '25

10 Dec '25

When pasta is invoked with a command rather than an existing namespace to attach to, it spawns a child process to run a shell or other command. We create that process during conf(), since we need the namespace to exist for much of our setup. However, we don't want the specified command to run until the pasta network interface is ready for use. Therefore, pasta_spawn_cmd() executing in the child waits before exec()ing. main() signals the child to continue with SIGUSR1 shortly before entering the main forwarding loop. This has the downside that if we exit due to any kind of failure between conf() and the SIGUSR1, the child process will be around waiting indefinitely. The user must manually clean this up. Make this cleaner, by having passt_exit() terminate the child, when called during this window. Signed-off-by: David Gibson <david(a)gibson.dropbear.id.au> --- passt.c | 1 + pasta.c | 2 ++ pasta.h | 1 + util.c | 5 +++++ 4 files changed, 9 insertions(+) diff --git a/passt.c b/passt.c index cf38822f..955c7091 100644 --- a/passt.c +++ b/passt.c @@ -431,6 +431,7 @@ int main(int argc, char **argv) if (pasta_child_pid) { kill(pasta_child_pid, SIGUSR1); log_stderr = false; + pasta_child_signalled = true; } isolate_postfork(&c); diff --git a/pasta.c b/pasta.c index 5c693de1..8ac4511f 100644 --- a/pasta.c +++ b/pasta.c @@ -54,6 +54,8 @@ /* PID of child, in case we created a namespace */ int pasta_child_pid; +/* Has the child been signalled to start a shell or command */ +bool pasta_child_signalled; /** * pasta_child_handler() - Exit once shell exits (if we started it), reap clones diff --git a/pasta.h b/pasta.h index 4b063d13..55028c74 100644 --- a/pasta.h +++ b/pasta.h @@ -7,6 +7,7 @@ #define PASTA_H extern int pasta_child_pid; +extern bool pasta_child_signalled; void pasta_open_ns(struct ctx *c, const char *netns); void pasta_start_ns(struct ctx *c, uid_t uid, gid_t gid, diff --git a/util.c b/util.c index e266c396..4744f09e 100644 --- a/util.c +++ b/util.c @@ -35,6 +35,7 @@ #include "log.h" #include "pcap.h" #include "epoll_ctl.h" +#include "pasta.h" #ifdef HAS_GETRANDOM #include <sys/random.h> #endif @@ -1235,6 +1236,10 @@ void abort_with_msg(const char *fmt, ...) */ void passt_exit(int status) { + /* If we're starting our own namespace, don't leave it in limbo */ + if (pasta_child_pid && !pasta_child_signalled) + kill(pasta_child_pid, SIGTERM); + /* Make sure we don't leave the pcap file truncated */ if (pcap_fd != -1 && fsync(pcap_fd)) warn_perror("Failed to flush pcap file, it might be truncated"); -- 2.52.0

2 1

[PATCH 2/2] pif: Correctly set scope_id for guest-side link local addresses
by David Gibson 10 Dec '25

10 Dec '25

pif_sockaddr() is supposed to generate a suitable socket address, either for the host, or for the guest, depending on the 'pif' parameter. When given a link-local address, this means it needs to generate a suitable scope_id to specify which link. It does this for the host, but not for the guest. I think this was done on the assumption that we won't ever generate guest side link local addresses when forwarding connections. That, however, is not the case, at least with the recent extensions to "local mode". Fix the problem by properly populating the scope_id field for guest addresses. Link: https://bugs.passt.top/show_bug.cgi?id=181 Signed-off-by: David Gibson <david(a)gibson.dropbear.id.au> --- pif.c | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/pif.c b/pif.c index 3d7a90e5..6ae970a0 100644 --- a/pif.c +++ b/pif.c @@ -50,10 +50,14 @@ void pif_sockaddr(const struct ctx *c, union sockaddr_inany *sa, sa->sa_family = AF_INET6; sa->sa6.sin6_addr = addr->a6; sa->sa6.sin6_port = htons(port); - if (pif == PIF_HOST && IN6_IS_ADDR_LINKLOCAL(&addr->a6)) - sa->sa6.sin6_scope_id = c->ifi6; - else + if (IN6_IS_ADDR_LINKLOCAL(&addr->a6)) { + if (pif == PIF_HOST) + sa->sa6.sin6_scope_id = c->ifi6; + else if (pif == PIF_SPLICE) + sa->sa6.sin6_scope_id = c->pasta_ifi; + } else { sa->sa6.sin6_scope_id = 0; + } sa->sa6.sin6_flowinfo = 0; } } -- 2.52.0

1 0

[PATCH 1/2] tcp: Correct timer expiry value in trace message
by David Gibson 10 Dec '25

10 Dec '25

000601ba8 ("tcp: Adaptive interval based on RTT for socket-side acknowledgement checks") added (amongst other things) a new trace message showing the expiry time for the TCP timer in ms rather than s. Unfortunately there were some arithmetic errors in the message, meaning it will print incorrect numbers. Correct them Fixes: 000601ba86da ("tcp: Adaptive interval based on RTT for socket-side acknowledgement checks") Link: https://bugs.passt.top/show_bug.cgi?id=182 Signed-off-by: David Gibson <david(a)gibson.dropbear.id.au> --- tcp.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/tcp.c b/tcp.c index 8dabfd30..ce36bbd3 100644 --- a/tcp.c +++ b/tcp.c @@ -626,11 +626,11 @@ static void tcp_timer_ctl(const struct ctx *c, struct tcp_tap_conn *conn) } if (conn->flags & ACK_TO_TAP_DUE) { - flow_trace(conn, "timer expires in %llu.%03llums", - (unsigned long)it.it_value.tv_sec * 1000 + - (unsigned long long)it.it_value.tv_nsec % - ((long)1000 * 1000), - (unsigned long long)it.it_value.tv_nsec / 1000); + flow_trace(conn, "timer expires in %llu.%02llums", + (unsigned long long)it.it_value.tv_sec * 1000 + + it.it_value.tv_nsec / 1000 / 1000, + (unsigned long long)it.it_value.tv_nsec + / 1000 / 10 % 100); } else { flow_dbg(conn, "timer expires in %llu.%03llus", (unsigned long long)it.it_value.tv_sec, -- 2.52.0

1 0

[PATCH 0/2] Improved exit()/_exit() handling
by David Gibson 10 Dec '25

10 Dec '25

As discussed on IRC, while working on the forwarding extensions, I discovered that failures at certain points of initialisation (in particular tcp_init() or udp_init() can result in a zombie pasta child process. Here's a fix. David Gibson (2): treewide: Introduce passt_exit() helper pasta: Clean up waiting pasta child on failures conf.c | 11 ++++------- log.h | 9 +++++++-- passt.c | 6 +++--- pasta.c | 12 ++++++------ pasta.h | 1 + tap.c | 6 ++---- util.c | 29 +++++++++++++++++++++++------ util.h | 1 - vhost_user.c | 4 ++-- 9 files changed, 48 insertions(+), 31 deletions(-) -- 2.52.0

1 0

[PATCH v3 0/6] vhost-user: Add multiqueue support
by Laurent Vivier 10 Dec '25

10 Dec '25

This series implements multiqueue support for vhost-user mode, allowing passt to utilize multiple queue pairs for improved network throughput when used with multi-CPU guest VMs. While this version uses a single thread for packet processing, it enables the guest kernel to distribute network traffic across multiple queues and vCPUs. The implementation advertises support for up to 16 queue pairs (32 virtqueues) by setting VIRTIO_NET_F_MQ and VHOST_USER_PROTOCOL_F_MQ feature flags. Packets are routed to the appropriate RX queue based on which TX queue they originated from, following the virtio specification's automatic receive steering requirements. This series adds: - Multiqueue capability advertisement (VIRTIO_NET_F_MQ and VHOST_USER_PROTOCOL_F_MQ features) - Per-queue-pair packet pools to support concurrent queue operations - Queue pair parameter throughout the network stack, propagated through all protocol handlers (TCP, UDP, ICMP, ARP, DHCP, DHCPv6, NDP) - Flow-aware queue routing that tracks the originating TX queue for each flow and routes return packets to the corresponding RX queue - Test coverage with VHOST_USER_MQ environment variable to validate multiqueue functionality across all protocols (TCP, UDP, ICMP) and services (DHCP, NDP) Current behavior: - TX queue selection is controlled by the guest kernel's networking stack - RX packets are routed to queues based on their associated flows, with the queue assignment updated on each packet from TX to maintain affinity - Host-initiated flows (e.g., from socket-side connections) currently default to queue pair 0 The changes are transparent to single-queue operation - passt/pasta modes and single-queue vhost-user configurations continue to work unchanged, always using queue pair 0. v3: - Removed --max-qpairs configuration option - multiqueue support is now always enabled up to 16 queue pairs without requiring explicit configuration - Replaced "tap: Add queue pair parameter throughout the packet processing path" with "tap: Convert packet pools to per-queue-pair arrays for multiqueue" - simplified implementation by converting global pools to arrays rather than passing pool parameters throughout - Changed qpair parameter type from int to unsigned int throughout the codebase - Simplified test infrastructure - queues parameter is always set on netdev, mq=true added to virtio-net only when VHOST_USER_MQ > 1 - Updated QEMU usage hints to always show multiqueue-capable command line v2: - New patch: "tap: Remove pool parameter from tap4_handler() and tap6_handler()" to clean up unused parameters before adding queue pair parameter - Changed to one packet pool per queue pair instead of shared pools across all queue pairs - Split "multiqueue: Add queue-aware flow management..." into two patches: - "tap: Add queue pair parameter throughout the packet processing path" - "flow: Add queue pair tracking to flow management" - Updated test infrastructure patch with refined implementation Laurent Vivier (6): tap: Remove pool parameter from tap4_handler() and tap6_handler() vhost-user: Enable multiqueue test: Add multiqueue support to vhost-user test infrastructure vhost-user: Add queue pair parameter throughout the network stack tap: Convert packet pools to per-queue-pair arrays for multiqueue flow: Add queue pair tracking to flow management arp.c | 15 +++-- arp.h | 6 +- dhcp.c | 5 +- dhcp.h | 2 +- dhcpv6.c | 12 ++-- dhcpv6.h | 2 +- flow.c | 33 +++++++++ flow.h | 17 +++++ fwd.c | 18 ++--- fwd.h | 5 +- icmp.c | 25 ++++--- icmp.h | 4 +- ndp.c | 35 ++++++---- ndp.h | 7 +- netlink.c | 2 +- tap.c | 177 ++++++++++++++++++++++++++++--------------------- tap.h | 20 +++--- tcp.c | 47 +++++++------ tcp.h | 7 +- tcp_vu.c | 8 ++- test/lib/setup | 21 +++--- test/run | 23 +++++++ udp.c | 47 +++++++------ udp.h | 6 +- udp_flow.c | 8 ++- udp_flow.h | 2 +- udp_internal.h | 4 +- udp_vu.c | 4 +- vhost_user.c | 10 +-- virtio.h | 2 +- vu_common.c | 15 +++-- vu_common.h | 3 +- 32 files changed, 374 insertions(+), 218 deletions(-) -- 2.51.1

2 9

[PATCH] tcp: Suppress false positive cppcheck warning in tcp_update_seqack_wnd()
by Laurent Vivier 09 Dec '25

09 Dec '25

Cppcheck doesn't track that tinfo is reassigned to &tinfo_new when NULL, leading to a false uninitvar warning at line 1099. Add inline suppression. Signed-off-by: Laurent Vivier <lvivier(a)redhat.com> --- tcp.c | 1 + 1 file changed, 1 insertion(+) diff --git a/tcp.c b/tcp.c index 17fda5dcd7db..8dabfd3042d2 100644 --- a/tcp.c +++ b/tcp.c @@ -1132,6 +1132,7 @@ int tcp_update_seqack_wnd(const struct ctx *c, struct tcp_tap_conn *conn, return 0; } + /* cppcheck-suppress uninitvar */ if ((unsigned)SNDBUF_GET(conn) > (long long)tinfo->tcpi_rtt * tinfo->tcpi_delivery_rate / 1000 / 1000 * -- 2.51.1

2 1

[PATCH v3 00/10] tcp: Fix throughput issues with non-local peers
by Stefano Brivio 09 Dec '25

09 Dec '25

Patch 3/10 is the most relevant fix here, as we currently advertise a window that might be too big for what we can write to the socket, causing retransmissions right away and occasional high latency on short transfers to non-local peers. Mostly as a consequence of fixing that, we now need several improvements and small fixes, including, most notably, an adaptive approach to pick the interval between checks for socket-side ACKs (patch 4/10), and several tricks to reliably trigger TCP buffer size auto-tuning as implemented by the Linux kernel (patches 6/10 and 8/10). These changes make some existing issues more relevant, fixed by the other patches. With this series, I'm getting the expected (wirespeed) throughput for transfers between peers with varying non-local RTTs: I checked different guests bridged on the same machine (~500 us) and hosts with increasing distance using iperf3, as well as HTTP transfers only for some hosts I have control over (500 us and 5 ms case). With increasing RTTs, I can finally see the throughput converging to the available bandwidth reasonably fast: * 500 us RTT, ~10 Gbps available: [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 785 MBytes 6.57 Gbits/sec 13 1.84 MBytes [ 5] 1.00-2.00 sec 790 MBytes 6.64 Gbits/sec 0 1.92 MBytes * 5 ms, 1 Gbps: [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 4.88 MBytes 40.9 Mbits/sec 22 240 KBytes [ 5] 1.00-2.00 sec 46.2 MBytes 388 Mbits/sec 34 900 KBytes [ 5] 2.00-3.00 sec 110 MBytes 923 Mbits/sec 0 1.11 MBytes * 10 ms, 1 Gbps: [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 67.9 MBytes 569 Mbits/sec 2 960 KBytes [ 5] 1.00-2.00 sec 110 MBytes 926 Mbits/sec 0 1.05 MBytes [ 5] 2.00-3.00 sec 111 MBytes 934 Mbits/sec 0 1.17 MBytes * 24 ms, 1 Gbps: [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 2.50 MBytes 20.9 Mbits/sec 16 240 KBytes [ 5] 1.00-2.00 sec 1.50 MBytes 12.6 Mbits/sec 9 120 KBytes [ 5] 2.00-3.00 sec 99.2 MBytes 832 Mbits/sec 4 2.40 MBytes [ 5] 3.00-4.00 sec 122 MBytes 1.03 Gbits/sec 1 3.16 MBytes [ 5] 4.00-5.00 sec 119 MBytes 1.00 Gbits/sec 0 4.16 MBytes * 40 ms, ~600 Mbps: [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 2.12 MBytes 17.8 Mbits/sec 0 600 KBytes [ 5] 1.00-2.00 sec 3.25 MBytes 27.3 Mbits/sec 14 420 KBytes [ 5] 2.00-3.00 sec 31.5 MBytes 264 Mbits/sec 11 1.29 MBytes [ 5] 3.00-4.00 sec 72.5 MBytes 608 Mbits/sec 0 1.46 MBytes * 100 ms, 1 Gbps: [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 4.88 MBytes 40.9 Mbits/sec 1 840 KBytes [ 5] 1.00-2.00 sec 1.62 MBytes 13.6 Mbits/sec 9 240 KBytes [ 5] 2.00-3.00 sec 5.25 MBytes 44.0 Mbits/sec 5 780 KBytes [ 5] 3.00-4.00 sec 9.75 MBytes 81.8 Mbits/sec 0 1.29 MBytes [ 5] 4.00-5.00 sec 15.8 MBytes 132 Mbits/sec 0 1.99 MBytes [ 5] 5.00-6.00 sec 22.9 MBytes 192 Mbits/sec 0 3.05 MBytes [ 5] 6.00-7.00 sec 132 MBytes 1.11 Gbits/sec 0 7.62 MBytes * 114 ms, 1 Gbps: [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 1.62 MBytes 13.6 Mbits/sec 8 420 KBytes [ 5] 1.00-2.00 sec 2.12 MBytes 17.8 Mbits/sec 15 120 KBytes [ 5] 2.00-3.00 sec 26.0 MBytes 218 Mbits/sec 50 1.82 MBytes [ 5] 3.00-4.00 sec 103 MBytes 865 Mbits/sec 31 5.10 MBytes [ 5] 4.00-5.00 sec 111 MBytes 930 Mbits/sec 0 5.92 MBytes * 153 ms, 1 Gbps: [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 1.12 MBytes 9.43 Mbits/sec 2 180 KBytes [ 5] 1.00-2.00 sec 2.12 MBytes 17.8 Mbits/sec 11 180 KBytes [ 5] 2.00-3.00 sec 12.6 MBytes 106 Mbits/sec 40 1.29 MBytes [ 5] 3.00-4.00 sec 44.5 MBytes 373 Mbits/sec 22 2.75 MBytes [ 5] 4.00-5.00 sec 86.0 MBytes 721 Mbits/sec 0 6.68 MBytes [ 5] 5.00-6.00 sec 120 MBytes 1.01 Gbits/sec 119 6.97 MBytes [ 5] 6.00-7.00 sec 110 MBytes 927 Mbits/sec 0 6.97 MBytes * 186 ms, 1 Gbps: [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 1.12 MBytes 9.43 Mbits/sec 3 180 KBytes [ 5] 1.00-2.00 sec 512 KBytes 4.19 Mbits/sec 4 120 KBytes [ 5] 2.00-3.00 sec 2.12 MBytes 17.8 Mbits/sec 6 360 KBytes [ 5] 3.00-4.00 sec 27.1 MBytes 228 Mbits/sec 6 1.11 MBytes [ 5] 4.00-5.00 sec 38.2 MBytes 321 Mbits/sec 0 1.99 MBytes [ 5] 5.00-6.00 sec 38.2 MBytes 321 Mbits/sec 0 2.46 MBytes [ 5] 6.00-7.00 sec 69.2 MBytes 581 Mbits/sec 71 3.63 MBytes [ 5] 7.00-8.00 sec 110 MBytes 919 Mbits/sec 0 5.92 MBytes * 271 ms, 1 Gbps: [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 1.12 MBytes 9.43 Mbits/sec 0 320 KBytes [ 5] 1.00-2.00 sec 0.00 Bytes 0.00 bits/sec 0 600 KBytes [ 5] 2.00-3.00 sec 512 KBytes 4.19 Mbits/sec 7 420 KBytes [ 5] 3.00-4.00 sec 896 KBytes 7.34 Mbits/sec 8 60.0 KBytes [ 5] 4.00-5.00 sec 2.62 MBytes 22.0 Mbits/sec 13 420 KBytes [ 5] 5.00-6.00 sec 12.1 MBytes 102 Mbits/sec 7 1020 KBytes [ 5] 6.00-7.00 sec 19.9 MBytes 167 Mbits/sec 0 1.82 MBytes [ 5] 7.00-8.00 sec 17.9 MBytes 150 Mbits/sec 44 1.76 MBytes [ 5] 8.00-9.00 sec 57.4 MBytes 481 Mbits/sec 30 2.70 MBytes [ 5] 9.00-10.00 sec 88.0 MBytes 738 Mbits/sec 0 6.45 MBytes * 292 ms, ~ 600 Mbps: [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 1.12 MBytes 9.43 Mbits/sec 0 450 KBytes [ 5] 1.00-2.00 sec 0.00 Bytes 0.00 bits/sec 3 180 KBytes [ 5] 2.00-3.00 sec 640 KBytes 5.24 Mbits/sec 4 120 KBytes [ 5] 3.00-4.00 sec 384 KBytes 3.15 Mbits/sec 2 120 KBytes [ 5] 4.00-5.00 sec 4.50 MBytes 37.7 Mbits/sec 1 660 KBytes [ 5] 5.00-6.00 sec 13.0 MBytes 109 Mbits/sec 3 960 KBytes [ 5] 6.00-7.00 sec 64.5 MBytes 541 Mbits/sec 0 2.17 MBytes * 327 ms, 1 Gbps: [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 1.12 MBytes 9.43 Mbits/sec 0 600 KBytes [ 5] 1.00-2.00 sec 0.00 Bytes 0.00 bits/sec 4 240 KBytes [ 5] 2.00-3.00 sec 768 KBytes 6.29 Mbits/sec 4 120 KBytes [ 5] 3.00-4.00 sec 1.62 MBytes 13.6 Mbits/sec 5 120 KBytes [ 5] 4.00-5.00 sec 1.88 MBytes 15.7 Mbits/sec 0 480 KBytes [ 5] 5.00-6.00 sec 17.6 MBytes 148 Mbits/sec 14 1.05 MBytes [ 5] 6.00-7.00 sec 35.1 MBytes 295 Mbits/sec 0 2.58 MBytes [ 5] 7.00-8.00 sec 45.2 MBytes 380 Mbits/sec 0 4.63 MBytes [ 5] 8.00-9.00 sec 27.0 MBytes 226 Mbits/sec 96 3.93 MBytes [ 5] 9.00-10.00 sec 85.9 MBytes 720 Mbits/sec 67 4.22 MBytes [ 5] 10.00-11.00 sec 118 MBytes 986 Mbits/sec 0 9.67 MBytes [ 5] 11.00-12.00 sec 124 MBytes 1.04 Gbits/sec 0 15.9 MBytes For short transfers, we strictly stick to the available sending buffer size to (almost) make sure we avoid local retransmissions, and significantly decrease transfer time as a result: from 1.2 s to 60 ms for a 5 MB HTTP transfer from a container hosted in a virtual machine to another guest. v3: - split 1/9 into 1/10 and 2/10 (one adding the function, one changing the usage factor) - in 1/10, rename function and clarify that the factor can be less than 100% - fix timer expiration formatting in 4/10 - in 6/10, use tcpi_rtt directly instead of its stored approximation v2: - Add 1/9, factoring out a generic version of the scaling function we already use for tcp_get_sndbuf(), as I'm now using it in 7/9 as well - in 3/9, use 4 bits instead of 3 to represent the RTT, which is important as we now use RTT values for more than just the ACK checks - in 5/9, instead of just comparing the sending buffer to SNDBUF_BIG to decide when to acknowledge data, use an adaptive approach based on the bandwidth-delay product - in 6/9, clarify the relationship between SWS avoidance and Nagle's algorithm, and introduce a reference to RFC 813, Section 4 - in 7/9, use an adaptive approach based on the product of bytes sent (and acknowledged) so far and RTT, instead of the previous approach based on bytes sent only, as it allows us to converge to the expected throughput much quicker with high RTT destinations *** BLURB HERE *** Stefano Brivio (10): tcp, util: Add function for scaling to linearly interpolated factor, use it tcp: Change usage factor of sending buffer in tcp_get_sndbuf() to 75% tcp: Limit advertised window to available, not total sending buffer size tcp: Adaptive interval based on RTT for socket-side acknowledgement checks tcp: Don't clear ACK_TO_TAP_DUE if we're advertising a zero-sized window tcp: Acknowledge everything if it looks like bulk traffic, not interactive tcp: Don't limit window to less-than-MSS values, use zero instead tcp: Allow exceeding the available sending buffer size in window advertisements tcp: Send a duplicate ACK also on complete sendmsg() failure tcp: Skip redundant ACK on partial sendmsg() failure README.md | 2 +- tcp.c | 170 ++++++++++++++++++++++++++++++++++++++++++----------- tcp_conn.h | 9 +++ util.c | 52 ++++++++++++++++ util.h | 2 + 5 files changed, 199 insertions(+), 36 deletions(-) -- 2.43.0

3 22