- dev - passt

[PATCH] tcp: Set ACK flag on *all* RST segments, even for client in SYN-SENT state
by Stefano Brivio 22 Jan '25

22 Jan '25

Somewhat curiously, RFC 9293, section 3.10.7.3, states: If the state is SYN-SENT, then [...] Second, check the RST bit: - If the RST bit is set, [...] o If the ACK was acceptable, then signal to the user "error: connection reset", drop the segment, enter CLOSED state, delete TCB, and return. Otherwise (no ACK), drop the segment and return. which matches verbatim RFC 793, pages 66-67, and is implemented as-is by tcp_rcv_synsent_state_process() in the Linux kernel, that is: /* No ACK in the segment */ if (th->rst) { /* rfc793: * "If the RST bit is set * * Otherwise (no ACK) drop the segment and return." */ goto discard_and_undo; } meaning that if a client is in SYN-SENT state, and we send a RST segment once we realise that we can't establish the outbound connection, the client will ignore our segment and will need to pointlessly wait until the connection times out instead of aborting it right away. The ACK flag on a RST, in this case, doesn't really seem to have any function, but we must set it nevertheless. The ACK sequence number is already correct because we always set it before calling tcp_prepare_flags(), whenever relevant. This leaves us with no cases where we should *not* set the ACK flag on non-SYN segments, so always set the ACK flag for RST segments. Note that non-SYN, non-RST segments were already covered by commit 4988e2b40631 ("tcp: Unconditionally force ACK for all !SYN, !RST packets"). Reported-by: Dirk Janssen <Dirk.Janssen(a)schiphol.nl> Reported-by: Roeland van de Pol <Roeland.van.de.Pol(a)schiphol.nl> Reported-by: Robert Floor <Robert.Floor(a)schiphol.nl> Signed-off-by: Stefano Brivio <sbrivio(a)redhat.com> --- tcp.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tcp.c b/tcp.c index 4d6a6b3..c89f323 100644 --- a/tcp.c +++ b/tcp.c @@ -1147,7 +1147,7 @@ int tcp_prepare_flags(const struct ctx *c, struct tcp_tap_conn *conn, *opts = TCP_SYN_OPTS(mss, conn->ws_to_tap); *optlen = sizeof(*opts); - } else if (!(flags & RST)) { + } else { flags |= ACK; } -- 2.43.0

2 1

[PATCH] tcp: Set PSH flag for last incoming packets in a batch
by Stefano Brivio 22 Jan '25

22 Jan '25

So far we omitted setting PSH flags for inbound traffic altogether: as we ignore the nature of the data we're sending, we can't conclude that some data is more or less urgent. This works fine with Linux guests, as the Linux kernel doesn't do much with it, on input: it will generally deliver data to the application layer without delay. However, with Windows, things change: if we don't set the PSH flag on interactive inbound traffic, we can expect long delays before the data is delivered to the application. This is very visible with RDP, where packets we send on behalf of the RDP client are delivered with delays exceeding one second: $ tshark -r rdp.pcap -td -Y 'frame.number in { 33170 .. 33173 }' --disable-protocol tls 33170 0.030296 93.235.154.248 → 88.198.0.164 54 TCP 49012 → 3389 [ACK] Seq=13820 Ack=285229 Win=387968 Len=0 33171 0.985412 88.198.0.164 → 93.235.154.248 105 TCP 3389 → 49012 [PSH, ACK] Seq=285229 Ack=13820 Win=63198 Len=51 33172 0.030373 93.235.154.248 → 88.198.0.164 54 TCP 49012 → 3389 [ACK] Seq=13820 Ack=285280 Win=387968 Len=0 33173 1.383776 88.198.0.164 → 93.235.154.248 424 TCP 3389 → 49012 [PSH, ACK] Seq=285280 Ack=13820 Win=63198 Len=370 in this example (packet capture taken by passt), frame #33172 is a mouse event sent by the RDP client, and frame #33173 is the first event (display reacting to click) sent back by the server. This appears as a 1.4 s delay before we get frame #33173. If we set PSH, instead: $ tshark -r rdp_psh.pcap -td -Y 'frame.number in { 314 .. 317 }' --disable-protocol tls 314 0.002503 93.235.154.248 → 88.198.0.164 170 TCP 51066 → 3389 [PSH, ACK] Seq=7779 Ack=74047 Win=31872 Len=116 315 0.000557 88.198.0.164 → 93.235.154.248 54 TCP 3389 → 51066 [ACK] Seq=79162 Ack=7895 Win=62872 Len=0 316 0.012752 93.235.154.248 → 88.198.0.164 170 TCP 51066 → 3389 [PSH, ACK] Seq=7895 Ack=79162 Win=31872 Len=116 317 0.011927 88.198.0.164 → 93.235.154.248 107 TCP 3389 → 51066 [PSH, ACK] Seq=79162 Ack=8011 Win=62756 Len=53 here, in frame #116, our mouse event is delivered without a delay and receives a response in approximately 12 ms. Set PSH on the last segment for any batch we dequeue from the socket, that is, set it whenever we know that we might not be sending data to the same port for a while. Reported-by: NN708 Link: https://bugs.passt.top/show_bug.cgi?id=107 Signed-off-by: Stefano Brivio <sbrivio(a)redhat.com> --- tcp_buf.c | 11 ++++++++--- tcp_vu.c | 7 +++++-- 2 files changed, 13 insertions(+), 5 deletions(-) diff --git a/tcp_buf.c b/tcp_buf.c index cbefa42..72d99c5 100644 --- a/tcp_buf.c +++ b/tcp_buf.c @@ -239,9 +239,10 @@ int tcp_buf_send_flag(const struct ctx *c, struct tcp_tap_conn *conn, int flags) * @dlen: TCP payload length * @no_csum: Don't compute IPv4 checksum, use the one from previous buffer * @seq: Sequence number to be sent + * @push: Set PSH flag, last segment in a batch */ static void tcp_data_to_tap(const struct ctx *c, struct tcp_tap_conn *conn, - ssize_t dlen, int no_csum, uint32_t seq) + ssize_t dlen, int no_csum, uint32_t seq, bool push) { struct tcp_payload_t *payload; const uint16_t *check = NULL; @@ -268,6 +269,7 @@ static void tcp_data_to_tap(const struct ctx *c, struct tcp_tap_conn *conn, payload->th.th_x2 = 0; payload->th.th_flags = 0; payload->th.ack = 1; + payload->th.psh = push; iov[TCP_IOV_PAYLOAD].iov_len = dlen + sizeof(struct tcphdr); tcp_l2_buf_fill_headers(conn, iov, check, seq, false); if (++tcp_payload_used > TCP_FRAMES_MEM - 1) @@ -402,11 +404,14 @@ int tcp_buf_data_from_sock(const struct ctx *c, struct tcp_tap_conn *conn) seq = conn->seq_to_tap; for (i = 0; i < send_bufs; i++) { int no_csum = i && i != send_bufs - 1 && tcp_payload_used; + bool push = false; - if (i == send_bufs - 1) + if (i == send_bufs - 1) { dlen = last_len; + push = true; + } - tcp_data_to_tap(c, conn, dlen, no_csum, seq); + tcp_data_to_tap(c, conn, dlen, no_csum, seq, push); seq += dlen; } diff --git a/tcp_vu.c b/tcp_vu.c index a216bb1..fad7065 100644 --- a/tcp_vu.c +++ b/tcp_vu.c @@ -289,10 +289,11 @@ static ssize_t tcp_vu_sock_recv(const struct ctx *c, * @iov_cnt: Number of entries in @iov * @check: Checksum, if already known * @no_tcp_csum: Do not set TCP checksum + * @push: Set PSH flag, last segment in a batch */ static void tcp_vu_prepare(const struct ctx *c, struct tcp_tap_conn *conn, struct iovec *iov, size_t iov_cnt, - const uint16_t **check, bool no_tcp_csum) + const uint16_t **check, bool no_tcp_csum, bool push) { const struct flowside *toside = TAPFLOW(conn); bool v6 = !(inany_v4(&toside->eaddr) && inany_v4(&toside->oaddr)); @@ -334,6 +335,7 @@ static void tcp_vu_prepare(const struct ctx *c, struct tcp_tap_conn *conn, memset(th, 0, sizeof(*th)); th->doff = sizeof(*th) / 4; th->ack = 1; + th->psh = push; tcp_fill_headers(conn, NULL, ip4h, ip6h, th, &payload, *check, conn->seq_to_tap, no_tcp_csum); @@ -443,6 +445,7 @@ int tcp_vu_data_from_sock(const struct ctx *c, struct tcp_tap_conn *conn) struct iovec *iov = &elem[head[i]].in_sg[0]; int buf_cnt = head[i + 1] - head[i]; ssize_t dlen = iov_size(iov, buf_cnt) - hdrlen; + bool push = i == head_cnt - 1; vu_set_vnethdr(vdev, iov->iov_base, buf_cnt); @@ -451,7 +454,7 @@ int tcp_vu_data_from_sock(const struct ctx *c, struct tcp_tap_conn *conn) check = NULL; previous_dlen = dlen; - tcp_vu_prepare(c, conn, iov, buf_cnt, &check, !*c->pcap); + tcp_vu_prepare(c, conn, iov, buf_cnt, &check, !*c->pcap, push); if (*c->pcap) { pcap_iov(iov, buf_cnt, -- 2.43.0

2 2

passt: new version 2025_01_21.4f2c8e7 available
by Stefano Brivio 21 Jan '25

21 Jan '25

The new version with tag 2025_01_21.4f2c8e7 includes the following changes: 4f2c8e7 vhost_user: Drop packet with unsupported iovec array ec5c4d9 tcp: Set PSH flag for last incoming packets in a batch db2c91a tcp: Set ACK flag on *all* RST segments, even for client in SYN-SENT state 54bb972 tcp: Disable Nagle's algorithm (set TCP_NODELAY) on all sockets 8757834 tcp: Buffer sizes are *not* inherited on accept()/accept4() c96a88d vhost_user: remove ASSERT() on iovec number 412ed4f vhost-user: Report to front-end we support VHOST_USER_PROTOCOL_F_DEVICE_STATE 31d7002 vhost-user: add VHOST_USER_SET_DEVICE_STATE_FD command 878e163 vhost-user: add VHOST_USER_CHECK_DEVICE_STATE command 78c73e9 vhost-user: Report to front-end we support VHOST_USER_PROTOCOL_F_LOG_SHMFD 3c1d91b vhost-user: add VHOST_USER_SET_LOG_BASE command 538312a vhost-user: Pass vu_dev to more virtio functions b04195c vhost-user: add VHOST_USER_SET_LOG_FD command 6016e04 vhost-user: update protocol features and commands list a8f4fc4 tcp: Mask EPOLLIN altogether if we're blocked waiting on an ACK from the guest b8f573c tcp: Set EPOLLET when when reading from a socket fails with EAGAIN 22cf08b tcp: Don't subscribe to EPOLLOUT events on STALLED 707f77b tcp: Fix ACK sequence getting out of sync on EPOLLOUT wake-up 1b95bd6 vhost_user: fix multibuffer from linux f04b483 test/pasta_podman: Run Podman tests on a single CPU thread 2c174f1 checksum: fix checksum with odd base address 725acd1 tcp_splice: Set (again) TCP_NODELAY on both sides 3876fc7 seccomp: Unconditionally allow accept(2) even if accept4(2) is present 898e853 virtio: Use const pointer for vu_dev 324233b udp_flow: Don't block multicast and broadcast messages 2385b69 Makefile: Report error and stop if we can't set TARGET e5ba8ad README: Mark vhost-user as supported https://passt.top/passt/log/?qt=range&q=2024_12_11.09478d5..2025_01_21.4f2c… Packages: - Alpine Linux: https://pkgs.alpinelinux.org/packages?name=passt - Arch Linux: https://www.archlinux.org/packages/extra/x86_64/passt/ https://archlinuxarm.org/packages/aarch64/passt https://archlinuxarm.org/packages/armv7h/passt - Chimera: https://pkgs.chimera-linux.org/packages?name=passt - Clear Linux: https://github.com/clearlinux-pkgs/passt/ - Copr (CentOS Stream, EPEL, Fedora, Mageia): https://copr.fedorainfracloud.org/coprs/sbrivio/passt/build/8556325/ permanent mirror: https://passt.top/builds/copr/0^20250121.g4f2c8e7/ - Debian tracker: https://tracker.debian.org/pkg/passt - Fedora updates: https://bodhi.fedoraproject.org/updates/?packages=passt - Gentoo versions: https://packages.gentoo.org/packages/net-misc/passt - GNU Guix: https://packages.guix.gnu.org/packages/passt/ - Homebrew: https://formulae.brew.sh/formula/passt - NixOS: https://github.com/NixOS/nixpkgs/tree/nixos-unstable/pkgs/by-name/pa/passt - openSUSE: https://software.opensuse.org/package/passt - OpenMandriva: https://github.com/OpenMandrivaAssociation/passt/tree/master - PLD Linux: https://git.pld-linux.org/cgi-bin/gitweb.cgi?p=packages/passt.git - Solus: https://github.com/getsolus/packages/tree/main/packages/p/passt - Ubuntu tracker: https://launchpad.net/ubuntu/+source/passt - Void Linux: https://voidlinux.org/packages/?q=passt - Static builds: - Package for other RPM-based distributions, x86_64 only: https://passt.top/builds/latest/x86_64/passt-g4f2c8e7-1.x86_64.rpm - x86_64 static binaries: https://passt.top/builds/latest/x86_64/ - Debian package, from x86_64 static build: https://passt.top/builds/latest/x86_64/passt_4f2c8e7-1_all.deb -- Stefano

1 0

[PATCH] vhost_user: Drop packet with unsupported iovec array
by Laurent Vivier 21 Jan '25

21 Jan '25

If the iovec array cannot be managed, drop it rather than passing the second entry to tap_add_packet(). Signed-off-by: Laurent Vivier <lvivier(a)redhat.com> --- vu_common.c | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/vu_common.c b/vu_common.c index aa5ca7b0e750..f43d8ac45748 100644 --- a/vu_common.c +++ b/vu_common.c @@ -195,15 +195,17 @@ static void vu_handle_tx(struct vu_dev *vdev, int index, hdrlen); } else { /* vnet header can be in a separate iovec */ - if (elem[count].out_num != 2) + if (elem[count].out_num != 2) { debug("virtio-net transmit queue contains more than one buffer ([%d]: %u)", count, elem[count].out_num); - if (elem[count].out_sg[0].iov_len != (size_t)hdrlen) + } else if (elem[count].out_sg[0].iov_len != (size_t)hdrlen) { debug("virtio-net transmit queue entry not aligned on hdrlen ([%d]: %d != %zu)", - count, hdrlen, elem[count].out_sg[0].iov_len); - tap_add_packet(vdev->context, - elem[count].out_sg[1].iov_len, - (char *)elem[count].out_sg[1].iov_base); + count, hdrlen, elem[count].out_sg[0].iov_len); + } else { + tap_add_packet(vdev->context, + elem[count].out_sg[1].iov_len, + (char *)elem[count].out_sg[1].iov_base); + } } count++; -- 2.47.1

2 1

[PATCH v2] tcp: Disable Nagle's algorithm (set TCP_NODELAY) on all sockets
by Stefano Brivio 21 Jan '25

21 Jan '25

Following up on 725acd111ba3 ("tcp_splice: Set (again) TCP_NODELAY on both sides"), David argues that, in general, we don't know what kind of TCP traffic we're dealing with, on any side or path. TCP segments might have been delivered to our socket with a PSH flag, but we don't have a way to know about it. Similarly, the guest might send us segments with PSH or URG set, but we don't know if we should generally TCP_CORK sockets and uncork on those flags, because that would assume they're running a Linux kernel (and a particular version of it) matching the kernel that delivers outbound packets for us. Given that we can't make any assumption and everything might very well be interactive traffic, disable Nagle's algorithm on all non-spliced sockets as well. After all, John Nagle himself is nowadays recommending that delayed ACKs should never be enabled together with his algorithm, but we don't have a practical way to ensure that our environment is free from delayed ACKs (TCP_QUICKACK is not really usable for this purpose): https://news.ycombinator.com/item?id=34180239 Suggested-by: David Gibson <david(a)gibson.dropbear.id.au> Signed-off-by: Stefano Brivio <sbrivio(a)redhat.com> --- v2: Set TCP_NODELAY on inbound socket after accept4(), not on the listening sockets, and change failure message to debug() instead of trace() tcp.c | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/tcp.c b/tcp.c index a012b81..4d6a6b3 100644 --- a/tcp.c +++ b/tcp.c @@ -756,6 +756,19 @@ static void tcp_sock_set_bufsize(const struct ctx *c, int s) trace("TCP: failed to set SO_SNDBUF to %i", v); } +/** + * tcp_sock_set_nodelay() - Set TCP_NODELAY option (disable Nagle's algorithm) + * @s: Socket, can be -1 to avoid check in the caller + */ +static void tcp_sock_set_nodelay(int s) +{ + if (s == -1) + return; + + if (setsockopt(s, SOL_TCP, TCP_NODELAY, &((int){ 1 }), sizeof(int))) + debug("TCP: failed to set TCP_NODELAY on socket %i", s); +} + /** * tcp_update_csum() - Calculate TCP checksum * @psum: Unfolded partial checksum of the IPv4 or IPv6 pseudo-header @@ -1285,6 +1298,7 @@ static int tcp_conn_new_sock(const struct ctx *c, sa_family_t af) return -errno; tcp_sock_set_bufsize(c, s); + tcp_sock_set_nodelay(s); return s; } @@ -2058,6 +2072,7 @@ void tcp_listen_handler(const struct ctx *c, union epoll_ref ref, goto cancel; tcp_sock_set_bufsize(c, s); + tcp_sock_set_nodelay(s); /* FIXME: When listening port has a specific bound address, record that * as our address -- 2.43.0

2 4

[PATCH] vhost_user: remove ASSERT() on iovec number
by Laurent Vivier 21 Jan '25

21 Jan '25

Replace ASSERT() on the number of iovec in the element and on the first entry length by a debug() message. Signed-off-by: Laurent Vivier <lvivier(a)redhat.com> --- vu_common.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/vu_common.c b/vu_common.c index 431fba6be0c0..d34ae6dc1df3 100644 --- a/vu_common.c +++ b/vu_common.c @@ -195,8 +195,12 @@ static void vu_handle_tx(struct vu_dev *vdev, int index, hdrlen); } else { /* vnet header can be in a separate iovec */ - ASSERT(elem[count].out_num == 2); - ASSERT(elem[count].out_sg[0].iov_len == (size_t)hdrlen); + if (elem[count].out_num != 2) + debug("virtio-net tranmit queue contains more than one buffer ([%d]: %u)", + count, elem[count].out_num); + if (elem[count].out_sg[0].iov_len != (size_t)hdrlen) + debug("virtio-net transmit queue entry not aligned on hdrlen ([%d]: %d != %zu)", + count, hdrlen, elem[count].out_sg[0].iov_len); tap_add_packet(vdev->context, elem[count].out_sg[1].iov_len, (char *)elem[count].out_sg[1].iov_base); -- 2.47.1

3 3

[PATCH] tcp: Buffer sizes are *not* inherited on accept()/accept4()
by Stefano Brivio 21 Jan '25

21 Jan '25

...so it's pointless to set SO_RCVBUF and SO_SNDBUF on listening sockets. Call tcp_sock_set_bufsize() after accept4(), for inbound sockets. As we didn't have large buffer sizes set for inbound sockets for a long time (they are set explicitly only if the maximum size is big enough, more than than the ~200 KiB default), I ran some more throughput tests for this one, and I see slightly better numbers (say, 17 gbps instead of 15 gbps guest to host without vhost-user). Fixes: 904b86ade7db ("tcp: Rework window handling, timers, add SO_RCVLOWAT and pools for sockets/pipes") Signed-off-by: Stefano Brivio <sbrivio(a)redhat.com> --- tcp.c | 11 ++++------- 1 file changed, 4 insertions(+), 7 deletions(-) diff --git a/tcp.c b/tcp.c index 3b3193a..a012b81 100644 --- a/tcp.c +++ b/tcp.c @@ -2057,6 +2057,8 @@ void tcp_listen_handler(const struct ctx *c, union epoll_ref ref, if (s < 0) goto cancel; + tcp_sock_set_bufsize(c, s); + /* FIXME: When listening port has a specific bound address, record that * as our address */ @@ -2260,7 +2262,6 @@ static int tcp_sock_init_one(const struct ctx *c, const union inany_addr *addr, if (s < 0) return s; - tcp_sock_set_bufsize(c, s); return s; } @@ -2317,9 +2318,7 @@ static void tcp_ns_sock_init4(const struct ctx *c, in_port_t port) s = pif_sock_l4(c, EPOLL_TYPE_TCP_LISTEN, PIF_SPLICE, &inany_loopback4, NULL, port, tref.u32); - if (s >= 0) - tcp_sock_set_bufsize(c, s); - else + if (s < 0) s = -1; if (c->tcp.fwd_out.mode == FWD_AUTO) @@ -2343,9 +2342,7 @@ static void tcp_ns_sock_init6(const struct ctx *c, in_port_t port) s = pif_sock_l4(c, EPOLL_TYPE_TCP_LISTEN, PIF_SPLICE, &inany_loopback6, NULL, port, tref.u32); - if (s >= 0) - tcp_sock_set_bufsize(c, s); - else + if (s < 0) s = -1; if (c->tcp.fwd_out.mode == FWD_AUTO) -- 2.43.0

2 1

[PATCH 0/9] vhost-user: Migration support
by Laurent Vivier 20 Jan '25

20 Jan '25

This series allows a QEMU guest to be migrated while it is connected to Passt using vhost-user interface. There are two parts: - first part enables the migration of QEMU without transferring the internal state of Passt. All connections are lost. This is done by implementing VHOST_USER_SET_LOG_FD and VHOST_USER_SET_LOG_BASE commands (and enabling VHOST_USER_PROTOCOL_F_LOG_SHMFD feature) "vhost-user: add VHOST_USER_SET_LOG_FD command" "vhost-user: add VHOST_USER_SET_LOG_BASE command" "vhost-user: Report to front-end we support VHOST_USER_PROTOCOL_F_LOG_SHMFD" - second part allows source Passt instance to send its internal state using QEMU migration channel to destination Passt instance. This is done implementing VHOST_USER_SET_DEVICE_STATE_FD and VHOST_USER_CHECK_DEVICE_STATE (and enabling VHOST_USER_PROTOCOL_F_DEVICE_STATE feature). "vhost-user: add VHOST_USER_CHECK_DEVICE_STATE command" "vhost-user: add VHOST_USER_SET_DEVICE_STATE_FD command" "vhost-user: Report to front-end we support VHOST_USER_PROTOCOL_F_DEVICE_STATE" For now, it only implements the function needed to transfer the state but no state is transferred. VHOST_USER_PROTOCOL_F_DEVICE_STATE is implemented in QEMU only for vhost-user-fs, to be able to use it with virtio-net I have proposed a patch upstream: https://patchew.org/QEMU/20241218143453.1573185-1-lvivier@redhat.com/ Laurent Vivier (9): virtio: Use const pointer for vu_dev vhost-user: update protocol features and commands list vhost-user: add VHOST_USER_SET_LOG_FD command vhost-user: Pass vu_dev to more virtio functions vhost-user: add VHOST_USER_SET_LOG_BASE command vhost-user: Report to front-end we support VHOST_USER_PROTOCOL_F_LOG_SHMFD vhost-user: add VHOST_USER_CHECK_DEVICE_STATE command vhost-user: add VHOST_USER_SET_DEVICE_STATE_FD command vhost-user: Report to front-end we support VHOST_USER_PROTOCOL_F_DEVICE_STATE epoll_type.h | 2 + passt.c | 4 + util.h | 3 + vhost_user.c | 251 ++++++++++++++++++++++++++++++++++++++++++++++++++- vhost_user.h | 50 +++++++++- virtio.c | 116 +++++++++++++++++++++--- virtio.h | 32 +++++-- vu_common.c | 59 +++++++++++- vu_common.h | 3 +- 9 files changed, 484 insertions(+), 36 deletions(-) -- 2.47.1

3 30

[PATCH] tcp: Disable Nagle's algorithm (set TCP_NODELAY) on all sockets
by Stefano Brivio 20 Jan '25

20 Jan '25

Following up on 725acd111ba3 ("tcp_splice: Set (again) TCP_NODELAY on both sides"), David argues that, in general, we don't know what kind of TCP traffic we're dealing with, on any side or path. TCP segments might have been delivered to our socket with a PSH flag, but we don't have a way to know about it. Similarly, the guest might send us segments with PSH or URG set, but we don't know if we should generally TCP_CORK sockets and uncork on those flags, because that would assume they're running a Linux kernel (and a particular version of it) matching the kernel that delivers outbound packets for us. Given that we can't make any assumption and everything might very well be interactive traffic, disable Nagle's algorithm on all non-spliced sockets as well. After all, John Nagle himself is nowadays recommending that delayed ACKs should never be enabled together with his algorithm, but we don't have a practical way to ensure that our environment is free from delayed ACKs (TCP_QUICKACK is not really usable for this purpose): https://news.ycombinator.com/item?id=34180239 Suggested-by: David Gibson <david(a)gibson.dropbear.id.au> Signed-off-by: Stefano Brivio <sbrivio(a)redhat.com> --- tcp.c | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/tcp.c b/tcp.c index 3b3193a..c570f42 100644 --- a/tcp.c +++ b/tcp.c @@ -756,6 +756,19 @@ static void tcp_sock_set_bufsize(const struct ctx *c, int s) trace("TCP: failed to set SO_SNDBUF to %i", v); } +/** + * tcp_sock_set_nodelay() - Set TCP_NODELAY option (disable Nagle's algorithm) + * @s: Socket, can be -1 to avoid check in the caller + */ +static void tcp_sock_set_nodelay(int s) +{ + if (s == -1) + return; + + if (setsockopt(s, SOL_TCP, TCP_NODELAY, &((int){ 1 }), sizeof(int))) + trace("TCP: failed to set TCP_NODELAY on socket %i", s); +} + /** * tcp_update_csum() - Calculate TCP checksum * @psum: Unfolded partial checksum of the IPv4 or IPv6 pseudo-header @@ -1285,6 +1298,7 @@ static int tcp_conn_new_sock(const struct ctx *c, sa_family_t af) return -errno; tcp_sock_set_bufsize(c, s); + tcp_sock_set_nodelay(s); return s; } @@ -2261,6 +2275,8 @@ static int tcp_sock_init_one(const struct ctx *c, const union inany_addr *addr, return s; tcp_sock_set_bufsize(c, s); + tcp_sock_set_nodelay(s); + return s; } @@ -2322,6 +2338,8 @@ static void tcp_ns_sock_init4(const struct ctx *c, in_port_t port) else s = -1; + tcp_sock_set_nodelay(s); + if (c->tcp.fwd_out.mode == FWD_AUTO) tcp_sock_ns[port][V4] = s; } @@ -2348,6 +2366,8 @@ static void tcp_ns_sock_init6(const struct ctx *c, in_port_t port) else s = -1; + tcp_sock_set_nodelay(s); + if (c->tcp.fwd_out.mode == FWD_AUTO) tcp_sock_ns[port][V6] = s; } -- 2.43.0

2 2

[net, v2.1] tcp: correct handling of extreme memory squeeze
by Jon Maloy 17 Jan '25

17 Jan '25

Testing with iperf3 using the "pasta" protocol splicer has revealed a bug in the way tcp handles window advertising in extreme memory squeeze situations. The problem occurs on the server side, and the socket in question is a completely regular socket with the default settings for the fedora40 kernel. We do not use SO_PEEK or SO_RCVBUF on this socket. A brief summary: Under memory pressure, a socket endpoint may temporarily advertise a zero-sized window, but this is not stored as part of the socket data. The reasoning behind this is that it is considered a temporary setting which shouldn't influence any further calculations. However, if we happen to stall at an unfortunate value of the current window size, the algorithm selecting a new value will consistently fail to advertise a non-zero window once we have freed up enough memory. This means that this side's notion of the current window size is different from the one last advertised to the peer, causing the latter to not send any data to resolve the sitution. The following excerpt of a logging session, with own comments added, shows more in detail what is happening: // tcp_v4_rcv(->) // tcp_rcv_established(->) [5201<->39222]: ==== Activating log @ net/ipv4/tcp_input.c/tcp_data_queue()/5257 ==== [5201<->39222]: tcp_data_queue(->) [5201<->39222]: DROPPING skb [265600160..265665640], reason: SKB_DROP_REASON_PROTO_MEM [rcv_nxt 265600160, rcv_wnd 262144, snt_ack 265469200, win_now 131184] [copied_seq 259909392->260034360 (124968), unread 5565800, qlen 85, ofoq 0] [5201<->39222]: tcp_data_queue(<-) OFO queue: gap: 65480, len: 0 [5201<->39222]: __tcp_transmit_skb(->) [5201<->39222]: tcp_select_window(->) tp->rcv_wup: 265469200, tp->rcv_wnd: 262144, tp->rcv_nxt 265600160 [5201<->39222]: (inet_csk(sk)->icsk_ack.pending & ICSK_ACK_NOMEM) --> TRUE [5201<->39222]: tcp_select_window(<-) tp->rcv_wup: 265469200, tp->rcv_wnd: 262144, tp->rcv_nxt 265600160, returning 0 [5201<->39222]: ADVERTISING WIN 0, ACK_SEQ: 265600160 [5201<->39222]: __tcp_transmit_skb(<-) tp->rcv_wup: 265469200, tp->rcv_wnd: 262144, tp->rcv_nxt 265600160 [5201<->39222]: tcp_rcv_established(<-) [5201<->39222]: tcp_v4_rcv(<-) // Receive queue is at 85 buffers and we are out of memory. // We drop the incoming buffer, although it is in sequence, and decide // to send an advertisement with a window of zero. // We don't update tp->rcv_wnd and tp->rcv_wup accordingly, which means // we unconditionally shrink the window. [5201<->39222]: tcp_recvmsg_locked(->) [5201<->39222]: __tcp_cleanup_rbuf(->) tp->rcv_wup: 265469200, tp->rcv_wnd: 262144, tp->rcv_nxt 265600160 [5201<->39222]: [new_win = 0, win_now = 131184, 2 * win_now = 262368] [5201<->39222]: [new_win >= (2 * win_now) ? --> time_to_ack = 0] [5201<->39222]: NOT calling tcp_send_ack() [5201<->39222]: __tcp_cleanup_rbuf(<-) tp->rcv_wup: 265469200, tp->rcv_wnd: 262144, tp->rcv_nxt 265600160 [5201<->39222]: tcp_recvmsg_locked(<-) returning 6104 bytes. [rcv_nxt 265600160, rcv_wnd 262144, snt_ack 265469200, win_now 131184] [copied_seq 260040464->260040464 (0), unread 5559696, qlen 85, ofoq 0] // After each read, the algorithm for calculating the new receive // window in __tcp_cleanup_rbuf() finds it is too small to advertise // or to update tp->rcv_wnd. // Meanwhile, the peer thinks the window is zero, and will not send // any more data to trigger an update from the interrupt mode side. [5201<->39222]: tcp_recvmsg_locked(->) [5201<->39222]: __tcp_cleanup_rbuf(->) tp->rcv_wup: 265469200, tp->rcv_wnd: 262144, tp->rcv_nxt 265600160 [5201<->39222]: [new_win = 262144, win_now = 131184, 2 * win_now = 262368] [5201<->39222]: [new_win >= (2 * win_now) ? --> time_to_ack = 0] [5201<->39222]: NOT calling tcp_send_ack() [5201<->39222]: __tcp_cleanup_rbuf(<-) tp->rcv_wup: 265469200, tp->rcv_wnd: 262144, tp->rcv_nxt 265600160 [5201<->39222]: tcp_recvmsg_locked(<-) returning 131072 bytes. [rcv_nxt 265600160, rcv_wnd 262144, snt_ack 265469200, win_now 131184] [copied_seq 260099840->260171536 (71696), unread 5428624, qlen 83, ofoq 0] // The above pattern repeats again and again, since nothing changes // between the reads. [...] [5201<->39222]: tcp_recvmsg_locked(->) [5201<->39222]: __tcp_cleanup_rbuf(->) tp->rcv_wup: 265469200, tp->rcv_wnd: 262144, tp->rcv_nxt 265600160 [5201<->39222]: [new_win = 262144, win_now = 131184, 2 * win_now = 262368] [5201<->39222]: [new_win >= (2 * win_now) ? --> time_to_ack = 0] [5201<->39222]: NOT calling tcp_send_ack() [5201<->39222]: __tcp_cleanup_rbuf(<-) tp->rcv_wup: 265469200, tp->rcv_wnd: 262144, tp->rcv_nxt 265600160 [5201<->39222]: tcp_recvmsg_locked(<-) returning 131072 bytes. [rcv_nxt 265600160, rcv_wnd 262144, snt_ack 265469200, win_now 131184] [copied_seq 265469200->265545488 (76288), unread 54672, qlen 1, ofoq 0] [5201<->39222]: tcp_recvmsg_locked(->) [5201<->39222]: __tcp_cleanup_rbuf(->) tp->rcv_wup: 265469200, tp->rcv_wnd: 262144, tp->rcv_nxt 265600160 [5201<->39222]: [new_win = 262144, win_now = 131184, 2 * win_now = 262368] [5201<->39222]: [new_win >= (2 * win_now) ? --> time_to_ack = 0] [5201<->39222]: NOT calling tcp_send_ack() [5201<->39222]: __tcp_cleanup_rbuf(<-) tp->rcv_wup: 265469200, tp->rcv_wnd: 262144, tp->rcv_nxt 265600160 [5201<->39222]: tcp_recvmsg_locked(<-) returning 54672 bytes. [rcv_nxt 265600160, rcv_wnd 262144, snt_ack 265469200, win_now 131184] [copied_seq 265600160->265600160 (0), unread 0, qlen 0, ofoq 0] // The receive queue is empty, but no new advertisement is sent. // The peer still thinks the receive window is zero, and sends nothing. // We have ended up in a deadlock situation. Furthermore, we have observed that in these situations this side may send out an updated 'th->ack_seq´ which is not stored in tp->rcv_wup as it should be. Backing ack_seq seems to be harmless, but is of course still wrong from a protocol viewpoint. We fix this by setting tp->rcv_wnd and tp->rcv_wup even when a packet has been dropped because of memory exhaustion and we have to advertize a zero window. Further testing shows that the connection recovers neatly from the squeeze situation, and traffic can continue indefinitely. Fixes: e2142825c120 ("net: tcp: send zero-window ACK when no memory") Signed-off-by: Jon Maloy <jmaloy(a)redhat.com> --- v1: -Posted on Apr 6, 2024 v2: -Improved commit log to clarify how we end up in this situation. -After feedback from Eric Dumazet, removed references to use of SO_PEEK and SO_PEEK_OFF which may lead to a misunderstanding about how this situation occurs. Those flags are used at the peer side's incoming connection, and not on this one. --- net/ipv4/tcp_output.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/net/ipv4/tcp_output.c b/net/ipv4/tcp_output.c index 0e5b9a654254..ba295f798e5e 100644 --- a/net/ipv4/tcp_output.c +++ b/net/ipv4/tcp_output.c @@ -265,11 +265,13 @@ static u16 tcp_select_window(struct sock *sk) u32 cur_win, new_win; /* Make the window 0 if we failed to queue the data because we - * are out of memory. The window is temporary, so we don't store - * it on the socket. + * are out of memory. */ - if (unlikely(inet_csk(sk)->icsk_ack.pending & ICSK_ACK_NOMEM)) + if (unlikely(inet_csk(sk)->icsk_ack.pending & ICSK_ACK_NOMEM)) { + tp->rcv_wnd = 0; + tp->rcv_wup = tp->rcv_nxt; return 0; + } cur_win = tcp_receive_window(tp); new_win = __tcp_select_window(sk); -- 2.48.0

2 1