Re: pasta: UDP flow sockets use the default receive buffer, dropping datagrams of a WireGuard client
Hi Tobias,
On Wed, 18 Nov 2026 20:04:15 +0100
Tobias Wiesing
Setup: passt 2026_10_02.cba3570, Podman 6.1.4, kernel 7.3.4 (openSUSE MicroOS), rootless; a WireGuard client (gluetun, kernel WireGuard) in a container on a Podman bridge network, so its UDP leaves through pasta. Host: net.core.rmem_default = 212992, rmem_max = 4194304.
Observed: downloads through the tunnel reach 4-8 MB/s on a line that serves ~30 MB/s without it. During a 15 s download the host counts 300-15000 UdpRcvbufErrors (nstat). Counting the tunnel's datagrams at the host NIC, pasta's tap in the rootless netns, the container's eth0 and the WireGuard interface shows no loss after pasta once the buffer is large enough.
With net.core.rmem_default = 4194304 (nothing else changed, container restarted so pasta opens new sockets): 0 UdpRcvbufErrors, 24-27 MB/s. The same tunnel under rootful Docker on another host reaches line rate with the kernel defaults, as no socket is involved there.
Source: udp.c / udp_flow.c / sock_l4() set no SO_RCVBUF on UDP sockets; SO_RCVBUF/SO_SNDBUF are set only on the tap fd (tap.c), after sock_probe_features(). Unlike TCP there is no autotuning for UDP, so the socket keeps rmem_default.
Question: could pasta set SO_RCVBUF on its UDP sockets (e.g. up to rmem_max when sock_probe_features() allows large buffers, as for the tap fd), or offer an option for it? Raising rmem_default host-wide works but affects every socket on the host.
Thanks for the report and for the suggestion. I think it needs a bit of research. For context, for TCP, we used to set large values, and then we could finally drop those settings (and let auto-tuning do its course) after a number of optimisations and fixes: https://passt.top/passt/commit/?id=71249ef3f9bcf1dbb2d6c13cdbc41ba88c794f06 and the topic is tracked at https://bugs.passt.top/show_bug.cgi?id=138 in general. For UDP, we never really looked into it. I had no idea it mattered until your report, actually. For maximum throughput, of course, we could simply try to set SO_RCVBUF to rmem_max on all sockets, and the buffer allocation in the kernel should only happen as that memory is actually used, so we don't _necessarily_ waste memory by doing that, but I suppose there might be a risk (especially in case the whole buffer is used at some point, and then buffer usage decreases). I see three possibilities, broadly speaking: 1. implement some kind of auto-tuning ourselves. For example udp_sock_recv() (or callers) could check how many bytes are dequeued at once, and periodically adjust the buffer size for a specific socket with some hysteresis (say, expand if we use more than 50 % of the buffer, shrink if we use less than 20 %) 2. add a command line option, or two (what about SO_SNDBUF?), that would apply to all the UDP sockets we use. Safe, but not really convenient as most users wouldn't know about it, and by default we might impose a performance penalty on them 3. just go with the maximum size, maybe it's fine and we don't actually waste any memory (but it needs to be checked) One further thought before trying to evaluate one of these, though: this is UDP, where latency might matter more than throughput, especially if you carry TCP on top of it, and just making buffers larger might by itself contribute to bufferbloat. Could your issue with Wireguard come from the fact that we dequeue "too many" frames at once? What happens if you decrease UDP_MAX_FRAMES in udp.c to, say, 8? Should we try to (dynamically?) tune that value instead? All in all, I think any of those alternatives would be viable, so feel free to send changes, but they need to be supported by some evidence and testing. -- Stefano
participants (1)
-
Stefano Brivio