- dev - passt

[PATCH] tcp: Fix subtle bug in fast re-transmit path
by David Gibson 12 Feb '24

12 Feb '24

When a duplicate ack from the tap side triggers a fast re-transmit, we set both conn->seq_ack_from_tap and conn->seq_to_tap to the sequence number of the duplicate ack. Setting seq_to_tap is correct: this is what triggers the retransmit from this point onwards. Setting seq_ack_from_tap is not correct, though. In most cases setting seq_ack_from_tap will be redundant but harmless: it will have already been updated to the same value by tcp_update_seqack_from_tap() a few lines above. However that call can be skipped if tcp_sock_consume() fails, which is rare but possible. In that case this update will cause problems. We use seq_ack_from_tap to track two logically distinct things: how much of the stream has been acked by the guest, and how much of the stream from the socket has been read and discarded (as opposed to MSG_PEEKed). We attempt to keep those values the same, because we discard data exactly when it is acked by the guest. However tcp_sock_consume() failing means we weren't able to disard the acked data. To handle that case, we skip the usual update of seq_ack_from_tap, effectively ignoring the ack assuming we'll get one which supersedes it soon enough. Setting seq_ack_from_tap in the fast retransmit path, however, means we now really will have the read/discard point in the stream out of sync with seq_ack_from_tap. Signed-off-by: David Gibson <david(a)gibson.dropbear.id.au> --- tcp.c | 1 - 1 file changed, 1 deletion(-) diff --git a/tcp.c b/tcp.c index 905d26f6..2ab443d5 100644 --- a/tcp.c +++ b/tcp.c @@ -2365,7 +2365,6 @@ static int tcp_data_from_tap(struct ctx *c, struct tcp_tap_conn *conn, flow_trace(conn, "fast re-transmit, ACK: %u, previous sequence: %u", max_ack_seq, conn->seq_to_tap); - conn->seq_ack_from_tap = max_ack_seq; conn->seq_to_tap = max_ack_seq; tcp_data_from_sock(c, conn); } -- 2.43.0

2 1

[PATCH v2] tcp: add support for SO_PEEK_OFF
by Jon Maloy 08 Feb '24

08 Feb '24

When reading received messages from a socket with MSG_PEEK, we may want to read the contents with an offset, like we can do with pread/preadv() when reading files. Currently, it is not possible to do that. In this commit, we add support for the SO_PEEK_OFF socket option for TCP, in a similar way it is done for Unix Domain sockets. In the iperf3 log examples shown below, we can observe a throughput improvement of 15-20 % in the direction host->namespace when using the protocol splicer 'pasta' (https://passt.top). This is a consistent result. pasta(1) and passt(1) implement user-mode networking for network namespaces (containers) and virtual machines by means of a translation layer between Layer-2 network interface and native Layer-4 sockets (TCP, UDP, ICMP/ICMPv6 echo). Received, pending TCP data to the container/guest is kept in kernel buffers until acknowledged, so the tool routinely needs to fetch new data from socket, skipping data that was already sent. At the moment this is implemented using a dummy buffer passed to recvmsg(). With this change, we don't need a dummy buffer and the related buffer copy (copy_to_user()) anymore. passt and pasta are supported in KubeVirt and libvirt/qemu. jmaloy@freyr:~/passt$ perf record -g ./pasta --config-net -f SO_PEEK_OFF not supported by kernel. jmaloy@freyr:~/passt# iperf3 -s ----------------------------------------------------------- Server listening on 5201 (test #1) ----------------------------------------------------------- Accepted connection from 192.168.122.1, port 44822 [ 5] local 192.168.122.180 port 5201 connected to 192.168.122.1 port 44832 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 1.02 GBytes 8.78 Gbits/sec [ 5] 1.00-2.00 sec 1.06 GBytes 9.08 Gbits/sec [ 5] 2.00-3.00 sec 1.07 GBytes 9.15 Gbits/sec [ 5] 3.00-4.00 sec 1.10 GBytes 9.46 Gbits/sec [ 5] 4.00-5.00 sec 1.03 GBytes 8.85 Gbits/sec [ 5] 5.00-6.00 sec 1.10 GBytes 9.44 Gbits/sec [ 5] 6.00-7.00 sec 1.11 GBytes 9.56 Gbits/sec [ 5] 7.00-8.00 sec 1.07 GBytes 9.20 Gbits/sec [ 5] 8.00-9.00 sec 667 MBytes 5.59 Gbits/sec [ 5] 9.00-10.00 sec 1.03 GBytes 8.83 Gbits/sec [ 5] 10.00-10.04 sec 30.1 MBytes 6.36 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate [ 5] 0.00-10.04 sec 10.3 GBytes 8.78 Gbits/sec receiver ----------------------------------------------------------- Server listening on 5201 (test #2) ----------------------------------------------------------- ^Ciperf3: interrupt - the server has terminated jmaloy@freyr:~/passt# logout [ perf record: Woken up 23 times to write data ] [ perf record: Captured and wrote 5.696 MB perf.data (35580 samples) ] jmaloy@freyr:~/passt$ jmaloy@freyr:~/passt$ perf record -g ./pasta --config-net -f SO_PEEK_OFF supported by kernel. jmaloy@freyr:~/passt# iperf3 -s ----------------------------------------------------------- Server listening on 5201 (test #1) ----------------------------------------------------------- Accepted connection from 192.168.122.1, port 52084 [ 5] local 192.168.122.180 port 5201 connected to 192.168.122.1 port 52098 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 1.32 GBytes 11.3 Gbits/sec [ 5] 1.00-2.00 sec 1.19 GBytes 10.2 Gbits/sec [ 5] 2.00-3.00 sec 1.26 GBytes 10.8 Gbits/sec [ 5] 3.00-4.00 sec 1.36 GBytes 11.7 Gbits/sec [ 5] 4.00-5.00 sec 1.33 GBytes 11.4 Gbits/sec [ 5] 5.00-6.00 sec 1.21 GBytes 10.4 Gbits/sec [ 5] 6.00-7.00 sec 1.31 GBytes 11.2 Gbits/sec [ 5] 7.00-8.00 sec 1.25 GBytes 10.7 Gbits/sec [ 5] 8.00-9.00 sec 1.33 GBytes 11.5 Gbits/sec [ 5] 9.00-10.00 sec 1.24 GBytes 10.7 Gbits/sec [ 5] 10.00-10.04 sec 56.0 MBytes 12.1 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate [ 5] 0.00-10.04 sec 12.9 GBytes 11.0 Gbits/sec receiver ----------------------------------------------------------- Server listening on 5201 (test #2) ----------------------------------------------------------- ^Ciperf3: interrupt - the server has terminated logout [ perf record: Woken up 20 times to write data ] [ perf record: Captured and wrote 5.040 MB perf.data (33411 samples) ] jmaloy@freyr:~/passt$ The perf record confirms this result. Below, we can observe that the CPU spends significantly less time in the function ____sys_recvmsg() when we have offset support. Without offset support: ---------------------- jmaloy@freyr:~/passt$ perf report -q --symbol-filter=do_syscall_64 -p ____sys_recvmsg -x --stdio -i perf.data | head -1 46.32% 0.00% passt.avx2 [kernel.vmlinux] [k] do_syscall_64 ____sys_recvmsg With offset support: ---------------------- jmaloy@freyr:~/passt$ perf report -q --symbol-filter=do_syscall_64 -p ____sys_recvmsg -x --stdio -i perf.data | head -1 28.12% 0.00% passt.avx2 [kernel.vmlinux] [k] do_syscall_64 ____sys_recvmsg Suggested-by: Paolo Abeni <pabeni(a)redhat.com> Reviewed-by: David Gibson <dgibson(a)redhat.com> Signed-off-by: Jon Maloy <jmaloy(a)redhat.com> --- include/net/tcp.h | 1 + net/ipv4/af_inet.c | 1 + net/ipv4/tcp.c | 26 +++++++++++++++++++------- 3 files changed, 21 insertions(+), 7 deletions(-) diff --git a/include/net/tcp.h b/include/net/tcp.h index 87f0e6c2e1f2..7eca7f2ac102 100644 --- a/include/net/tcp.h +++ b/include/net/tcp.h @@ -357,6 +357,7 @@ void tcp_twsk_purge(struct list_head *net_exit_list, int family); ssize_t tcp_splice_read(struct socket *sk, loff_t *ppos, struct pipe_inode_info *pipe, size_t len, unsigned int flags); +int tcp_set_peek_offset(struct sock *sk, int val); struct sk_buff *tcp_stream_alloc_skb(struct sock *sk, gfp_t gfp, bool force_schedule); diff --git a/net/ipv4/af_inet.c b/net/ipv4/af_inet.c index fb81de10d332..7a8b3a91257f 100644 --- a/net/ipv4/af_inet.c +++ b/net/ipv4/af_inet.c @@ -1068,6 +1068,7 @@ const struct proto_ops inet_stream_ops = { #endif .splice_eof = inet_splice_eof, .splice_read = tcp_splice_read, + .set_peek_off = tcp_set_peek_offset, .read_sock = tcp_read_sock, .read_skb = tcp_read_skb, .sendmsg_locked = tcp_sendmsg_locked, diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c index fce5668a6a3d..ad03e0cee3c1 100644 --- a/net/ipv4/tcp.c +++ b/net/ipv4/tcp.c @@ -863,6 +863,14 @@ ssize_t tcp_splice_read(struct socket *sock, loff_t *ppos, } EXPORT_SYMBOL(tcp_splice_read); +int tcp_set_peek_offset(struct sock *sk, int val) +{ + WRITE_ONCE(sk->sk_peek_off, val); + + return 0; +} +EXPORT_SYMBOL(tcp_set_peek_offset); + struct sk_buff *tcp_stream_alloc_skb(struct sock *sk, gfp_t gfp, bool force_schedule) { @@ -1414,8 +1422,6 @@ static int tcp_peek_sndq(struct sock *sk, struct msghdr *msg, int len) struct sk_buff *skb; int copied = 0, err = 0; - /* XXX -- need to support SO_PEEK_OFF */ - skb_rbtree_walk(skb, &sk->tcp_rtx_queue) { err = skb_copy_datagram_msg(skb, 0, msg, skb->len); if (err) @@ -2317,6 +2323,7 @@ static int tcp_recvmsg_locked(struct sock *sk, struct msghdr *msg, size_t len, long timeo; struct sk_buff *skb, *last; u32 urg_hole = 0; + u32 peek_offset = 0; err = -ENOTCONN; if (sk->sk_state == TCP_LISTEN) @@ -2349,7 +2356,8 @@ static int tcp_recvmsg_locked(struct sock *sk, struct msghdr *msg, size_t len, seq = &tp->copied_seq; if (flags & MSG_PEEK) { - peek_seq = tp->copied_seq; + peek_offset = max(sk_peek_offset(sk, flags), 0); + peek_seq = tp->copied_seq + peek_offset; seq = &peek_seq; } @@ -2452,11 +2460,11 @@ static int tcp_recvmsg_locked(struct sock *sk, struct msghdr *msg, size_t len, } if ((flags & MSG_PEEK) && - (peek_seq - copied - urg_hole != tp->copied_seq)) { + (peek_seq - peek_offset - copied - urg_hole != tp->copied_seq)) { net_dbg_ratelimited("TCP(%s:%d): Application bug, race in MSG_PEEK\n", current->comm, task_pid_nr(current)); - peek_seq = tp->copied_seq; + peek_seq = tp->copied_seq + peek_offset; } continue; @@ -2497,7 +2505,10 @@ static int tcp_recvmsg_locked(struct sock *sk, struct msghdr *msg, size_t len, WRITE_ONCE(*seq, *seq + used); copied += used; len -= used; - + if (flags & MSG_PEEK) + sk_peek_offset_fwd(sk, used); + else + sk_peek_offset_bwd(sk, used); tcp_rcv_space_adjust(sk); skip_copy: @@ -2774,6 +2785,7 @@ void __tcp_close(struct sock *sk, long timeout) data_was_unread += len; __kfree_skb(skb); } + sk_set_peek_off(sk, -1); /* If socket has been already reset (e.g. in tcp_reset()) - kill it. */ if (sk->sk_state == TCP_CLOSE) @@ -4492,7 +4504,7 @@ void tcp_done(struct sock *sk) reqsk_fastopen_remove(sk, req, false); WRITE_ONCE(sk->sk_shutdown, SHUTDOWN_MASK); - + sk_set_peek_off(sk, -1); if (!sock_flag(sk, SOCK_DEAD)) sk->sk_state_change(sk); else -- 2.42.0

3 3

[RFC net-next] tcp: add support for SO_PEEK_OFF
by jmaloy＠redhat.com 07 Feb '24

07 Feb '24

From: Jon Maloy <jmaloy(a)redhat.com> When reading received messages from a socket with MSG_PEEK, we may want to read the contents with an offset, like we can do with pread/preadv() when reading files. Currently, it is not possible to do that. In this commit, we add support for the SO_PEEK_OFF socket option for TCP, in a similar way it is done for Unix Domain sockets. In the iperf3 log examples shown below, we can observe a throughput improvement of 15-20 % in the direction host->namespace when using the protocol splicer 'pasta' (https://passt.top). This is a consistent result. pasta(1) and passt(1) implement user-mode networking for network namespaces (containers) and virtual machines by means of a translation layer between Layer-2 network interface and native Layer-4 sockets (TCP, UDP, ICMP/ICMPv6 echo). Received, pending TCP data to the container/guest is kept in kernel buffers until acknowledged, so the tool routinely needs to fetch new data from socket, skipping data that was already sent. At the moment this is implemented using a dummy buffer passed to recvmsg(). With this change, we don't need a dummy buffer and the related buffer copy (copy_to_user()) anymore. passt and pasta are supported in KubeVirt and libvirt/qemu. jmaloy@freyr:~/passt$ perf record -g ./pasta --config-net -f SO_PEEK_OFF not supported by kernel. jmaloy@freyr:~/passt# iperf3 -s ----------------------------------------------------------- Server listening on 5201 (test #1) ----------------------------------------------------------- Accepted connection from 192.168.122.1, port 44822 [ 5] local 192.168.122.180 port 5201 connected to 192.168.122.1 port 44832 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 1.02 GBytes 8.78 Gbits/sec [ 5] 1.00-2.00 sec 1.06 GBytes 9.08 Gbits/sec [ 5] 2.00-3.00 sec 1.07 GBytes 9.15 Gbits/sec [ 5] 3.00-4.00 sec 1.10 GBytes 9.46 Gbits/sec [ 5] 4.00-5.00 sec 1.03 GBytes 8.85 Gbits/sec [ 5] 5.00-6.00 sec 1.10 GBytes 9.44 Gbits/sec [ 5] 6.00-7.00 sec 1.11 GBytes 9.56 Gbits/sec [ 5] 7.00-8.00 sec 1.07 GBytes 9.20 Gbits/sec [ 5] 8.00-9.00 sec 667 MBytes 5.59 Gbits/sec [ 5] 9.00-10.00 sec 1.03 GBytes 8.83 Gbits/sec [ 5] 10.00-10.04 sec 30.1 MBytes 6.36 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate [ 5] 0.00-10.04 sec 10.3 GBytes 8.78 Gbits/sec receiver ----------------------------------------------------------- Server listening on 5201 (test #2) ----------------------------------------------------------- ^Ciperf3: interrupt - the server has terminated jmaloy@freyr:~/passt# logout [ perf record: Woken up 23 times to write data ] [ perf record: Captured and wrote 5.696 MB perf.data (35580 samples) ] jmaloy@freyr:~/passt$ jmaloy@freyr:~/passt$ perf record -g ./pasta --config-net -f SO_PEEK_OFF supported by kernel. jmaloy@freyr:~/passt# iperf3 -s ----------------------------------------------------------- Server listening on 5201 (test #1) ----------------------------------------------------------- Accepted connection from 192.168.122.1, port 52084 [ 5] local 192.168.122.180 port 5201 connected to 192.168.122.1 port 52098 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 1.32 GBytes 11.3 Gbits/sec [ 5] 1.00-2.00 sec 1.19 GBytes 10.2 Gbits/sec [ 5] 2.00-3.00 sec 1.26 GBytes 10.8 Gbits/sec [ 5] 3.00-4.00 sec 1.36 GBytes 11.7 Gbits/sec [ 5] 4.00-5.00 sec 1.33 GBytes 11.4 Gbits/sec [ 5] 5.00-6.00 sec 1.21 GBytes 10.4 Gbits/sec [ 5] 6.00-7.00 sec 1.31 GBytes 11.2 Gbits/sec [ 5] 7.00-8.00 sec 1.25 GBytes 10.7 Gbits/sec [ 5] 8.00-9.00 sec 1.33 GBytes 11.5 Gbits/sec [ 5] 9.00-10.00 sec 1.24 GBytes 10.7 Gbits/sec [ 5] 10.00-10.04 sec 56.0 MBytes 12.1 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate [ 5] 0.00-10.04 sec 12.9 GBytes 11.0 Gbits/sec receiver ----------------------------------------------------------- Server listening on 5201 (test #2) ----------------------------------------------------------- ^Ciperf3: interrupt - the server has terminated logout [ perf record: Woken up 20 times to write data ] [ perf record: Captured and wrote 5.040 MB perf.data (33411 samples) ] jmaloy@freyr:~/passt$ The perf record confirms this result. Below, we can observe that the CPU spends significantly less time in the function ____sys_recvmsg() when we have offset support. Without offset support: ---------------------- jmaloy@freyr:~/passt$ perf report -q --symbol-filter=do_syscall_64 -p ____sys_recvmsg -x --stdio -i perf.data | head -1 46.32% 0.00% passt.avx2 [kernel.vmlinux] [k] do_syscall_64 ____sys_recvmsg With offset support: ---------------------- jmaloy@freyr:~/passt$ perf report -q --symbol-filter=do_syscall_64 -p ____sys_recvmsg -x --stdio -i perf.data | head -1 28.12% 0.00% passt.avx2 [kernel.vmlinux] [k] do_syscall_64 ____sys_recvmsg Signed-off-by: Jon Maloy <jmaloy(a)redhat.com> --- include/net/tcp.h | 1 + net/ipv4/af_inet.c | 1 + net/ipv4/tcp.c | 25 +++++++++++++++++++------ 3 files changed, 21 insertions(+), 6 deletions(-) diff --git a/include/net/tcp.h b/include/net/tcp.h index 87f0e6c2e1f2..7eca7f2ac102 100644 --- a/include/net/tcp.h +++ b/include/net/tcp.h @@ -357,6 +357,7 @@ void tcp_twsk_purge(struct list_head *net_exit_list, int family); ssize_t tcp_splice_read(struct socket *sk, loff_t *ppos, struct pipe_inode_info *pipe, size_t len, unsigned int flags); +int tcp_set_peek_offset(struct sock *sk, int val); struct sk_buff *tcp_stream_alloc_skb(struct sock *sk, gfp_t gfp, bool force_schedule); diff --git a/net/ipv4/af_inet.c b/net/ipv4/af_inet.c index fb81de10d332..7a8b3a91257f 100644 --- a/net/ipv4/af_inet.c +++ b/net/ipv4/af_inet.c @@ -1068,6 +1068,7 @@ const struct proto_ops inet_stream_ops = { #endif .splice_eof = inet_splice_eof, .splice_read = tcp_splice_read, + .set_peek_off = tcp_set_peek_offset, .read_sock = tcp_read_sock, .read_skb = tcp_read_skb, .sendmsg_locked = tcp_sendmsg_locked, diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c index fce5668a6a3d..33ade88633de 100644 --- a/net/ipv4/tcp.c +++ b/net/ipv4/tcp.c @@ -863,6 +863,14 @@ ssize_t tcp_splice_read(struct socket *sock, loff_t *ppos, } EXPORT_SYMBOL(tcp_splice_read); +int tcp_set_peek_offset(struct sock *sk, int val) +{ + WRITE_ONCE(sk->sk_peek_off, val); + + return 0; +} +EXPORT_SYMBOL(tcp_set_peek_offset); + struct sk_buff *tcp_stream_alloc_skb(struct sock *sk, gfp_t gfp, bool force_schedule) { @@ -2302,7 +2310,6 @@ static int tcp_inq_hint(struct sock *sk) * tricks with *seq access order and skb->users are not required. * Probably, code can be easily improved even more. */ - static int tcp_recvmsg_locked(struct sock *sk, struct msghdr *msg, size_t len, int flags, struct scm_timestamping_internal *tss, int *cmsg_flags) @@ -2317,6 +2324,7 @@ static int tcp_recvmsg_locked(struct sock *sk, struct msghdr *msg, size_t len, long timeo; struct sk_buff *skb, *last; u32 urg_hole = 0; + u32 peek_offset = 0; err = -ENOTCONN; if (sk->sk_state == TCP_LISTEN) @@ -2349,7 +2357,8 @@ static int tcp_recvmsg_locked(struct sock *sk, struct msghdr *msg, size_t len, seq = &tp->copied_seq; if (flags & MSG_PEEK) { - peek_seq = tp->copied_seq; + peek_offset = max(sk_peek_offset(sk, flags), 0); + peek_seq = tp->copied_seq + peek_offset; seq = &peek_seq; } @@ -2452,11 +2461,11 @@ static int tcp_recvmsg_locked(struct sock *sk, struct msghdr *msg, size_t len, } if ((flags & MSG_PEEK) && - (peek_seq - copied - urg_hole != tp->copied_seq)) { + (peek_seq - peek_offset - copied - urg_hole != tp->copied_seq)) { net_dbg_ratelimited("TCP(%s:%d): Application bug, race in MSG_PEEK\n", current->comm, task_pid_nr(current)); - peek_seq = tp->copied_seq; + peek_seq = tp->copied_seq + peek_offset; } continue; @@ -2497,7 +2506,10 @@ static int tcp_recvmsg_locked(struct sock *sk, struct msghdr *msg, size_t len, WRITE_ONCE(*seq, *seq + used); copied += used; len -= used; - + if (flags & MSG_PEEK) + sk_peek_offset_fwd(sk, used); + else + sk_peek_offset_bwd(sk, used); tcp_rcv_space_adjust(sk); skip_copy: @@ -2774,6 +2786,7 @@ void __tcp_close(struct sock *sk, long timeout) data_was_unread += len; __kfree_skb(skb); } + sk_set_peek_off(sk, -1); /* If socket has been already reset (e.g. in tcp_reset()) - kill it. */ if (sk->sk_state == TCP_CLOSE) @@ -4492,7 +4505,7 @@ void tcp_done(struct sock *sk) reqsk_fastopen_remove(sk, req, false); WRITE_ONCE(sk->sk_shutdown, SHUTDOWN_MASK); - + sk_set_peek_off(sk, -1); if (!sock_flag(sk, SOCK_DEAD)) sk->sk_state_change(sk); else -- 2.42.0

3 2

tcp: leverage support of SO_PEEK_OFF socket option when available
by Jon Maloy 05 Feb '24

05 Feb '24

The kernel may support recvmsg(MSG_PEEK), starting reading data from a given offset set by the SO_PEEK_OFF socket option. This makes it possible to avoid repeated reading of already read initial bytes of a received message, hence saving read cycles when forwarding TCP messages in the host->name space direction. In this commit, we add functionality to leverage this feature when available, while we fall back to the previous behavior when not. Measurements with iperf3 shows that throughput increases with 15-20 percent in the host->namespace direction when this feature is used. Signed-off-by: Jon Maloy <jmaloy(a)redhat.com> --- tcp.c | 35 +++++++++++++++++++++++++++++++++-- 1 file changed, 33 insertions(+), 2 deletions(-) diff --git a/tcp.c b/tcp.c index 905d26f..58674eb 100644 --- a/tcp.c +++ b/tcp.c @@ -505,6 +505,7 @@ static struct tcp_buf_seq_update tcp6_l2_buf_seq_update[TCP_FRAMES_MEM]; static unsigned int tcp6_l2_buf_used; /* recvmsg()/sendmsg() data for tap */ +static bool peek_offset_cap = false; static char tcp_buf_discard [MAX_WINDOW]; static struct iovec iov_sock [TCP_FRAMES_MEM + 1]; @@ -582,6 +583,14 @@ static_assert(ARRAY_SIZE(tc_hash) >= FLOW_MAX, int init_sock_pool4 [TCP_SOCK_POOL_SIZE]; int init_sock_pool6 [TCP_SOCK_POOL_SIZE]; +static void set_peek_offset(int s, int offset) +{ + if (!peek_offset_cap) + return; + if (setsockopt(s, SOL_SOCKET, SO_PEEK_OFF, &offset, sizeof(offset))) + perror("Failed to set SO_PEEK_OFF\n"); +} + /** * tcp_conn_epoll_events() - epoll events mask for given connection state * @events: Current connection events @@ -1951,7 +1960,7 @@ static void tcp_conn_from_tap(struct ctx *c, if (bind(s, (struct sockaddr *)&addr6_ll, sizeof(addr6_ll))) goto cancel; } - + set_peek_offset(s, 0); conn = &flow->tcp; conn->f.type = FLOW_TCP; conn->sock = s; @@ -2174,6 +2183,12 @@ static int tcp_data_from_sock(struct ctx *c, struct tcp_tap_conn *conn) if (iov_rem) iov_sock[fill_bufs].iov_len = iov_rem; + /* Don't use discard buffer if SO_PEEK_OFF is supported. */ + if (peek_offset_cap) { + mh_sock.msg_iov = &iov_sock[1]; + mh_sock.msg_iovlen -= 1; + } + /* Receive into buffers, don't dequeue until acknowledged by guest. */ do len = recvmsg(s, &mh_sock, MSG_PEEK); @@ -2195,7 +2210,9 @@ static int tcp_data_from_sock(struct ctx *c, struct tcp_tap_conn *conn) return 0; } - sendlen = len - already_sent; + sendlen = len; + if (!peek_offset_cap) + sendlen -= already_sent; if (sendlen <= 0) { conn_flag(c, conn, STALLED); return 0; @@ -2367,6 +2384,7 @@ static int tcp_data_from_tap(struct ctx *c, struct tcp_tap_conn *conn, max_ack_seq, conn->seq_to_tap); conn->seq_ack_from_tap = max_ack_seq; conn->seq_to_tap = max_ack_seq; + set_peek_offset(conn->sock, 0); tcp_data_from_sock(c, conn); } @@ -2718,6 +2736,7 @@ void tcp_listen_handler(struct ctx *c, union epoll_ref ref, tcp_splice_conn_from_sock(c, ref.tcp_listen, &flow->tcp_splice, s, (struct sockaddr *)&sa)) return; + set_peek_offset(s, 0); tcp_tap_conn_from_sock(c, ref.tcp_listen, &flow->tcp, s, (struct sockaddr *)&sa, now); @@ -3042,6 +3061,7 @@ static void tcp_sock_refill_init(const struct ctx *c) int tcp_init(struct ctx *c) { unsigned b; + int s; for (b = 0; b < TCP_HASH_TABLE_SIZE; b++) tc_hash[b] = FLOW_SIDX_NONE; @@ -3065,6 +3085,17 @@ int tcp_init(struct ctx *c) NS_CALL(tcp_ns_socks_init, c); } + /* Probe for SO_PEEK_OFF support */ + s = socket(AF_INET, SOCK_STREAM, IPPROTO_TCP); + if (s < 0) { + perror("Temporary tcp socket creation failed\n"); + } else { + if (!setsockopt(s, SOL_SOCKET, SO_PEEK_OFF, &(int){0}, sizeof(int))) { + peek_offset_cap = true; + } + close(s); + } + printf("SO_PEEK_OFF%ssupported\n", peek_offset_cap ? " " : " not "); return 0; } -- 2.42.0

2 1

[PATCH] netlink: Add support to fetch default gateway from multipath routes
by Stefano Brivio 05 Feb '24

05 Feb '24

If the default route for a given IP version is a multipath one, instead of refusing to start because there's no RTA_GATEWAY attribute in the set returned by the kernel, we can just pick one of the paths. To make this somewhat less arbitrary, pick the path with the highest weight, if weights differ. Reported-by: Ed Santiago <santiago(a)redhat.com> Link: https://github.com/containers/podman/issues/20927 Signed-off-by: Stefano Brivio <sbrivio(a)redhat.com> --- netlink.c | 51 ++++++++++++++++++++++++++++++++++++++++++++++++--- passt.1 | 6 ++++-- 2 files changed, 52 insertions(+), 5 deletions(-) diff --git a/netlink.c b/netlink.c index bf79dd4..f0b04cb 100644 --- a/netlink.c +++ b/netlink.c @@ -271,18 +271,60 @@ unsigned int nl_get_ext_if(int s, sa_family_t af) for (rta = RTM_RTA(rtm), na = RTM_PAYLOAD(nh); RTA_OK(rta, na); rta = RTA_NEXT(rta, na)) { - if (rta->rta_type != RTA_OIF) - continue; + if (rta->rta_type == RTA_OIF) { + ifi = *(unsigned int *)RTA_DATA(rta); + } else if (rta->rta_type == RTA_MULTIPATH) { + struct rtnexthop *rtnh; - ifi = *(unsigned int *)RTA_DATA(rta); + rtnh = (struct rtnexthop *)RTA_DATA(rta); + ifi = rtnh->rtnh_ifindex; + } } } + if (status < 0) warn("netlink: RTM_GETROUTE failed: %s", strerror(-status)); return ifi; } +/** + * nl_route_get_def_multipath() - Get lowest-weight route from nexthop list + * @rta: Routing netlink attribute with type RTA_MULTIPATH + * @gw: Default gateway to fill + * + * Return: true if a gateway was found, false otherwise + */ +bool nl_route_get_def_multipath(struct rtattr *rta, void *gw) +{ + struct rtnexthop *rtnh; + bool found = false; + int hops = -1; + + for (rtnh = (struct rtnexthop *)RTA_DATA(rta); + RTNH_OK(rtnh, RTA_PAYLOAD(rta)); rtnh = RTNH_NEXT(rtnh)) { + size_t len = rtnh->rtnh_len - sizeof(*rtnh); + struct rtattr *rta_inner; + + if (rtnh->rtnh_hops < hops) + continue; + + hops = rtnh->rtnh_hops; + + for (rta_inner = RTNH_DATA(rtnh); RTA_OK(rta_inner, len); + rta_inner = RTA_NEXT(rta_inner, len)) { + + if (rta_inner->rta_type != RTA_GATEWAY) + continue; + + memcpy(gw, RTA_DATA(rta_inner), RTA_PAYLOAD(rta_inner)); + found = true; + } + } + + return found; +} + /** * nl_route_get_def() - Get default route for given interface and address family * @s: Netlink socket @@ -326,6 +368,9 @@ int nl_route_get_def(int s, unsigned int ifi, sa_family_t af, void *gw) for (rta = RTM_RTA(rtm), na = RTM_PAYLOAD(nh); RTA_OK(rta, na); rta = RTA_NEXT(rta, na)) { + if (rta->rta_type == RTA_MULTIPATH) + found = nl_route_get_def_multipath(rta, gw); + if (rta->rta_type != RTA_GATEWAY) continue; diff --git a/passt.1 b/passt.1 index efd6bb7..cc678ed 100644 --- a/passt.1 +++ b/passt.1 @@ -171,8 +171,10 @@ Assign IPv4 \fIaddr\fR as default gateway via DHCP (option 3), or IPv6 \fIaddr\fR as source for NDP Router Advertisement and DHCPv6 messages. This option can be specified zero (for defaults) to two times (once for IPv4, once for IPv6). -By default, IPv4 and IPv6 addresses are taken from the host interface with the -first default route for the corresponding IP version. +By default, IPv4 and IPv6 gateways are taken from the host interface with the +first default route for the corresponding IP version. If the default route is a +multipath one, the gateway is the first nexthop router returned by the kernel +which has the highest weight in the set of paths. Note: these addresses are also used as source address for packets directed to the guest or to the target namespace having a loopback or local source address, -- 2.39.2

2 1

[PATCH v2] log: setlogmask(0) can actually result in a system call, don't use it
by Stefano Brivio 05 Feb '24

05 Feb '24

Before commit 32d07f5e59f2 ("passt, pasta: Completely avoid dynamic memory allocation"), we didn't store the current log mask in a variable, and we fetched it using setlogmask(0) wherever needed. But after that commit, we can use our log_mask copy instead. And we should: with recent glibc versions, setlogmask(0) actually results in a system call, which causes a substantial overhead with high transfer rates: we use setlogmask(0) even to decide we don't want to print debug messages. Now that we rely on log_mask in early stages, before setlogmask() is called, we need to initialise that variable to the special LOG_EMERG mask value right away: define LOG_EARLY to make this clearer, and, while at it, group conditions in vlogmsg() into something more terse. Signed-off-by: Stefano Brivio <sbrivio(a)redhat.com> --- v2: Initialise log_mask so that prints before the first __setlogmask() call still work. log.c | 18 ++++++++++-------- 1 file changed, 10 insertions(+), 8 deletions(-) diff --git a/log.c b/log.c index 4a70e29..eafaca2 100644 --- a/log.c +++ b/log.c @@ -30,9 +30,12 @@ #include "util.h" #include "passt.h" +/* LOG_EARLY means we don't know yet: log everything. LOG_EMERG is unused */ +#define LOG_EARLY LOG_MASK(LOG_EMERG) + static int log_sock = -1; /* Optional socket to system logger */ static char log_ident[BUFSIZ]; /* Identifier string for openlog() */ -static int log_mask; /* Current log priority mask */ +static int log_mask = LOG_EARLY; /* Current log priority mask */ static int log_opt; /* Options for openlog() */ static int log_file = -1; /* Optional log file descriptor */ @@ -45,34 +48,33 @@ static time_t log_start; /* Start timestamp */ int log_trace; /* --trace mode enabled */ int log_to_stdout; /* Print to stdout instead of stderr */ -#define BEFORE_DAEMON (setlogmask(0) == LOG_MASK(LOG_EMERG)) - void vlogmsg(int pri, const char *format, va_list ap) { + bool debug_print = (log_mask & LOG_MASK(LOG_DEBUG)) && log_file == -1; + bool early_print = LOG_PRI(log_mask) == LOG_EARLY; FILE *out = log_to_stdout ? stdout : stderr; struct timespec tp; - if (setlogmask(0) & LOG_MASK(LOG_DEBUG) && log_file == -1) { + if (debug_print) { clock_gettime(CLOCK_REALTIME, &tp); fprintf(out, "%lli.%04lli: ", (long long int)tp.tv_sec - log_start, (long long int)tp.tv_nsec / (100L * 1000)); } - if ((LOG_MASK(LOG_PRI(pri)) & log_mask) || BEFORE_DAEMON) { + if ((log_mask & LOG_MASK(LOG_PRI(pri))) || early_print) { va_list ap2; va_copy(ap2, ap); /* Don't clobber ap, we need it again */ if (log_file != -1) logfile_write(pri, format, ap2); - else if (!(setlogmask(0) & LOG_MASK(LOG_DEBUG))) + else if (!(log_mask & LOG_MASK(LOG_DEBUG))) passt_vsyslog(pri, format, ap2); va_end(ap2); } - if ((setlogmask(0) & LOG_MASK(LOG_DEBUG) && log_file == -1) || - (BEFORE_DAEMON && !(log_opt & LOG_PERROR))) { + if (debug_print || (early_print && !(log_opt & LOG_PERROR))) { (void)vfprintf(out, format, ap); if (format[strlen(format)] != '\n') fprintf(out, "\n"); -- 2.39.2

2 1

[PATCH] log: setlogmask(0) can actually result in a system call, don't use it
by Stefano Brivio 03 Feb '24

03 Feb '24

Before commit 32d07f5e59f2 ("passt, pasta: Completely avoid dynamic memory allocation"), we didn't store the current log mask in a variable, and we fetched it using setlogmask(0) wherever needed. But after that commit, we can just use our log_mask copy instead. And we should: with recent glibc versions, setlogmask(0) actually results in a system call, which causes a substantial overhead with high transfer rates: we use setlogmask(0) even to decide we don't want to print debug messages. Signed-off-by: Stefano Brivio <sbrivio(a)redhat.com> --- log.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/log.c b/log.c index 4a70e29..a66ecb7 100644 --- a/log.c +++ b/log.c @@ -45,33 +45,33 @@ static time_t log_start; /* Start timestamp */ int log_trace; /* --trace mode enabled */ int log_to_stdout; /* Print to stdout instead of stderr */ -#define BEFORE_DAEMON (setlogmask(0) == LOG_MASK(LOG_EMERG)) +#define BEFORE_DAEMON (log_mask == LOG_MASK(LOG_EMERG)) void vlogmsg(int pri, const char *format, va_list ap) { FILE *out = log_to_stdout ? stdout : stderr; struct timespec tp; - if (setlogmask(0) & LOG_MASK(LOG_DEBUG) && log_file == -1) { + if (log_mask & LOG_MASK(LOG_DEBUG) && log_file == -1) { clock_gettime(CLOCK_REALTIME, &tp); fprintf(out, "%lli.%04lli: ", (long long int)tp.tv_sec - log_start, (long long int)tp.tv_nsec / (100L * 1000)); } - if ((LOG_MASK(LOG_PRI(pri)) & log_mask) || BEFORE_DAEMON) { + if ((log_mask & LOG_MASK(LOG_PRI(pri))) || BEFORE_DAEMON) { va_list ap2; va_copy(ap2, ap); /* Don't clobber ap, we need it again */ if (log_file != -1) logfile_write(pri, format, ap2); - else if (!(setlogmask(0) & LOG_MASK(LOG_DEBUG))) + else if (!(log_mask & LOG_MASK(LOG_DEBUG))) passt_vsyslog(pri, format, ap2); va_end(ap2); } - if ((setlogmask(0) & LOG_MASK(LOG_DEBUG) && log_file == -1) || + if ((log_mask & LOG_MASK(LOG_DEBUG) && log_file == -1) || (BEFORE_DAEMON && !(log_opt & LOG_PERROR))) { (void)vfprintf(out, format, ap); if (format[strlen(format)] != '\n') -- 2.39.2

2 2

[PATCH 00/16] More flow table preliminaries: address handling improvements
by David Gibson 29 Jan '24

29 Jan '24

Here's another batch of cleanups and tweaks in preparation for the flow table. This set focuses on improved helpers for handling addresses, particularly in the TCP splice path. David Gibson (16): treewide: Use sa_family_t for address family variables tcp, udp: Don't precompute port remappings in epoll references flow: Add helper to determine a flow's protocol tcp_splice: Simplify clean up logic inany: Helper to test for IPv4 or IPv6 loopback address tcp, tcp_splice: Helpers for getting sockets from the pools tcp_splice: More specific variable names in new splice path tcp_splice: Fix incorrect parameter comment for tcp_splice_connect() tcp_splice: Merge tcp_splice_new() into its caller tcp_splice: Improve error reporting on connect path inany: Add inany_ntop() helper tcp_splice: Improve logic deciding when to splice util: Provide global constants for IPv4 loopback and unspecified address inany: Introduce union sockaddr_inany tcp, tcp_splice: Better construction of IPv4 or IPv6 sockaddrs inany: Extend inany_from_af to easily set unspecified addresses Makefile | 6 +- flow.c | 7 ++ flow.h | 4 + icmp.c | 24 +++--- icmp.h | 4 +- inany.c | 34 ++++++++ inany.h | 96 ++++++++++++++++++---- tcp.c | 106 ++++++++++++------------ tcp.h | 4 +- tcp_conn.h | 4 +- tcp_splice.c | 223 ++++++++++++++++++++++++++------------------------- tcp_splice.h | 5 +- udp.c | 21 ++--- udp.h | 2 +- util.c | 5 +- util.h | 4 +- 16 files changed, 335 insertions(+), 214 deletions(-) create mode 100644 inany.c -- 2.43.0

1 17

[PATCH v3 00/11] RFC: ICMP reworks preliminary to flow table integration
by David Gibson 23 Jan '24

23 Jan '24

As with TCP, it turns out that there are a bunch of clean ups and reworks to the ICMP code which will make integration with the flow table easier, even before introducing a non-trivial version of the flow table itself. Based on the flow based dispatch/allocation, and bind/addressing cleanup series. Changes since v2: * Rebased on current main and revised flow dispatch series * Standardise on the name 'id_sock' instead of 'id_map' for pointers to specific entries in the id_map * Avoid confusing usage of the word "sequence" to mean a flow of ping packets as opposed to the ping sequence number. * Use packet_get() rather than explicit comparisons to validate packet lengths Changes since v1: * Rebased on newer version of flow dispatch & allocation series * Added 12/12 splitting out close and new sequence functions David Gibson (11): icmp: Don't set "port" on destination sockaddr for ping sockets icmp: Remove redundant initialisation of sendto() address icmp: Don't attempt to handle "wrong direction" ping socket traffic icmp: Don't attempt to match host IDs to guest IDs icmp: Use -1 to represent "missing" sockets icmp: Simplify socket expiry scanning icmp: Share more between IPv4 and IPv6 paths in icmp_tap_handler() icmp: Consolidate icmp_sock_handler() with icmpv6_sock_handler() icmp: Warn on receive errors from ping sockets icmp: Validate packets received on ping sockets icmp: Dedicated functions for starting and closing ping sequences icmp.c | 329 ++++++++++++++++++++++++++++---------------------------- icmp.h | 5 +- passt.c | 4 +- 3 files changed, 166 insertions(+), 172 deletions(-) -- 2.43.0

2 12

[PATCH v4 00/13] Manage more flow related things from generic flow code
by David Gibson 23 Jan '24

23 Jan '24

There are a number of things that are more-or-less general to flows which are still explicitly handled in tcp.c and tcp_splice.c including allocation and freeing of flow entries, and dispatch of deferred and timer functions. Even without adding more fields to the common flow structure, we can handle a number of these in a more flow-centric way. Unlike v1 this version is based on the hash table rework series. Changes since v3: * Rebased on current tree, plus patches for some test failures * Renamed "free blocks" to "free clusters" in flow allocation scheme * Fixed bug where we wouldn't properly merge adjacent free clusters together during deferred handling * Improved theory of operation comments about flow allocation scheme Changes since v2: * Realised the prealloc/commit functions where confusing and worked poorly for some future stuff. Replaced with alloc/alloc_cancel * Fixed a bug where newly allocated flow entries might not be 0-filled, because of the free tracking information in there. This could cause very subtle problems. Changes since v1: * Store the timestamp of last flow timers run in a global, rather than a ctx field * Rebased on the TCP hash table rework * Add patches 9..13/13 with changes to allocation and freeing of flow entries. David Gibson (13): flow: Make flow_table.h #include the protocol specific headers it needs treewide: Standardise on 'now' for current timestamp variables tcp, tcp_splice: Remove redundant handling from tcp_timer() tcp, tcp_splice: Move per-type cleanup logic into per-type helpers flow, tcp: Add flow-centric dispatch for deferred flow handling flow, tcp: Add handling for per-flow timers epoll: Better handling of number of epoll types tcp, tcp_splice: Avoid double layered dispatch for connected TCP sockets flow: Move flow_log_() to near top of flow.c flow: Move flow_count from context structure to a global flow: Abstract allocation of new flows with helper function flow: Enforce that freeing of closed flows must happen in deferred handlers flow: Avoid moving flow entries to compact table flow.c | 235 +++++++++++++++++++++++++++++++++++++++++++-------- flow.h | 5 +- flow_table.h | 20 +++++ icmp.c | 12 +-- icmp.h | 2 +- log.c | 34 ++++---- passt.c | 20 +++-- passt.h | 9 +- tcp.c | 143 +++++++++---------------------- tcp.h | 2 +- tcp_conn.h | 8 +- tcp_splice.c | 49 +++++------ tcp_splice.h | 4 +- udp.c | 16 ++-- udp.h | 2 +- 15 files changed, 339 insertions(+), 222 deletions(-) -- 2.43.0

2 14