November 2025 - user

Help with pasta usage
by Ayon T 05 Dec '25

05 Dec '25

Hi, I've been using pasta as a network driver for rootless docker and I've been running into a couple of issues for a while now. I hope this is where I can find some help troubleshooting. The issue is that when I use pasta as the network driver as opposed to slirp4netns, I'm unable to access the internet through rootless docker or use ping (or traceroute) through its containers. So if I run "docker pull <image-name>" I get a timeout error: > Using default tag: latest Error response from daemon: Get "https://registry-1.docker.io/v2/": dial tcp: lookup registry-1.docker.io on 10.0.2.3:53: read udp 169.254.2.1:58905 ->10.0.2.3:53: i/o timeout I'm running pasta version 0.0~git20250217.a1e48a0-1 on Ubuntu 24.04.2 LTS with docker v27.3.1 build ce12230. Please let me know what other details I could provide to troubleshoot this issue. Thanks and regards, Ayon Tarafdar

3 10

Port Forwarding between host-guest via passt using QEMU-libvirt
by James Sinton 04 Dec '25

04 Dec '25

Hi All, I'm having trouble getting host-to-guest port forwarding to work via passt for a userspace connection: Please see my post on Superuser for more details: https://superuser.com/q/1927306/3115375?sem=2 Thank you. Best, James

3 6

Auto forwarding ports, but only to localhost
by Jan Wrobel 01 Dec '25

01 Dec '25

Hi, For pasta, would you consider an option to enable automatic forwarding of ports bound in a namespace, but make the forwarded ports available only via localhost, not all addresses? I'm working on a sandboxing program which uses pasta. The option -t "auto" is super convenient, but requires extra care, without proper firewall setup bound ports become automatically available to outside world. For a sandboxing program like mine, it is not a safe default to run with, because the program shouldn't assume the user will have a firewall configured. If something like "localhost/auto" was supported, it would match the convenience of "auto", no manual port mapping config would be needed, but would be safer for uses cases where exposing ports to outside world is problematic. Cheers, Jan

3 7

Re: pasta slow at HTTP upload
by David Gibson 23 Nov '25

23 Nov '25

On Wed, Nov 12, 2025 at 05:30:35PM -0700, Max Chernoff via user wrote: > Date: Wed, 12 Nov 2025 17:30:35 -0700 > From: Max Chernoff <git(a)maxchernoff.ca> > To: Stefano Brivio <sbrivio(a)redhat.com> > CC: passt-user(a)passt.top > Subject: Re: pasta slow at HTTP upload > List-Id: "For passt users: support, questions and answers" > <passt-user.passt.top> > > Hi Stefano, > > On Wed, 2025-11-12 at 13:53 +0100, Stefano Brivio wrote: > > As a further hack, you could probably do something like this on top: > > > > --- > > diff --git a/tcp.c b/tcp.c > > index 697f80d..8c50ee0 100644 > > --- a/tcp.c > > +++ b/tcp.c > > @@ -339,7 +339,7 @@ enum { > > #define MSS_DEFAULT 536 > > #define WINDOW_DEFAULT 14600 /* > > RFC 6928 */ > > > > -#define ACK_INTERVAL 1 /* ms */ > > +#define ACK_INTERVAL 200 /* us */ > > #define SYN_TIMEOUT 10 /* s */ > > #define ACK_TIMEOUT 2 > > #define FIN_TIMEOUT 60 > > @@ -582,7 +582,7 @@ static void tcp_timer_ctl(struct tcp_tap_conn *conn) > > } > > > > if (conn->flags & ACK_TO_TAP_DUE) { > > - it.it_value.tv_nsec = (long)ACK_INTERVAL * 1000 * 1000; > > + it.it_value.tv_nsec = (long)ACK_INTERVAL * 1000; > > } else if (conn->flags & ACK_FROM_TAP_DUE) { > > if (!(conn->events & ESTABLISHED)) > > it.it_value.tv_sec = SYN_TIMEOUT; > > --- > > That actually makes it worse again, about as bad as before the patch. > But I've just tried rebuilding with original patch again, and also with > the exact same binary that I used yesterday, and that's slow now too. > I've verified with pgrep that Podman is using the correct pasta version, > so I have no idea what's happening. > > However, I do remember that for the past few months, some uploads would > randomly go really quickly, so maybe the problem happens sporadically, > and when I was testing the patched version I just happened to get > (un)lucky? > > > > net.core.wmem_max=7500000 > > > net.core.rmem_max=7500000 > > > > Those were settings we recommended for KubeVirt until > > https://github.com/kubevirt/user-guide/pull/933, but they don't seem to > > necessarily make sense as we seem to have made peace with the TCP > > auto-tuning mechanism in Linux meanwhile. > > > > See also https://bugs.passt.top/show_bug.cgi?id=138 and commit > > 71249ef3f9bc ("tcp, tcp_splice: Don't set SO_SNDBUF and SO_RCVBUF to > > maximum values"). > > > > As the issue here is about socket (kernel) buffers being "too small" for > > a while, I guess that those settings plus reverting that commit would > > "fix" the issue entirely for you. But it's impractical to rely on users > > to set those, that's why I'm looking for something adaptive which still > > plays nicely with TCP auto-tuning instead. > > Ah, I didn't know that those (used to be) recommended for pasta; I set > those for Caddy since it complains on startup if those aren't set > > https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes > > > > net.ipv4.tcp_notsent_lowat=131072 > > > net.core.default_qdisc=cake > > > net.ipv4.tcp_congestion_control=bbr > > I set those as a general performance tuning thing (not for pasta > specifically) based off of > > https://blog.cloudflare.com/optimizing-tcp-for-high-throughput-and-low-late… > > https://grapheneos.org/articles/server-traffic-shaping > > > I'm not sure if those really matter for pasta, but I haven't really > > thought about them. > > Aha, those do seem to be the issue. Using the original (unpatched) > pasta: Huh, interesting. > (Set everything to my original settings) > $ sudo sysctl -w net.core.wmem_max=7500000 net.core.rmem_max=7500000 net.ipv4.tcp_notsent_lowat=131072 net.core.default_qdisc=cake net.ipv4.tcp_congestion_control=bbr > > (Test with --network=host) > $ podman run --rm --pull=newer --volume="$(realpath .):/srv/:Z" --workdir=/srv/ --network=host quay.io/fedora/fedora-minimal curl --output /dev/null --progress-meter --form file=@./test.tar.gz "https://www.ctan.org/submit/validate" > % Total % Received % Xferd Average Speed Time Time Time Current > Dload Upload Total Spent Left Speed > 100 100M 0 345 100 100M 69 20.0M 0:00:04 0:00:04 --:--:-- 20.4M > > (Test with --network=pasta) > $ podman run --rm --pull=newer --volume="$(realpath .):/srv/:Z" --workdir=/srv/ --network=pasta quay.io/fedora/fedora-minimal curl --output /dev/null --progress-meter --form file=@./test.tar.gz "https://www.ctan.org/submit/validate" > % Total % Received % Xferd Average Speed Time Time Time Current > Dload Upload Total Spent Left Speed > 4 100M 0 0 4 5120k 0 42440 0:41:10 0:02:03 0:39:07 26188 > (Stopped early since I got sick of waiting) > > (Set everything to the kernel defaults) > $ sudo sysctl -w net.core.wmem_max=212992 net.core.rmem_max=212992 net.ipv4.tcp_notsent_lowat=4294967295 net.core.default_qdisc=fq_codel net.ipv4.tcp_congestion_control=cubic > $ podman run --rm --pull=newer --volume="$(realpath .):/srv/:Z" --workdir=/srv/ --network=pasta quay.io/fedora/fedora-minimal curl --output /dev/null --progress-meter --form file=@./test.tar.gz "https://www.ctan.org/submit/validate" > % Total % Received % Xferd Average Speed Time Time Time Current > Dload Upload Total Spent Left Speed > 100 100M 0 345 100 100M 4 1431k 0:01:11 0:01:11 --:--:-- 1399k > > (tcp_congestion_control=default, tcp_notsent_lowat=custom, [rw]mem_max=default) > $ sudo sysctl -w net.ipv4.tcp_congestion_control=cubic net.ipv4.tcp_notsent_lowat=131072 net.core.rmem_max=212992 net.core.wmem_max=212992 > $ podman run --rm --pull=newer --volume="$(realpath .):/srv/:Z" --workdir=/srv/ --network=pasta quay.io/fedora/fedora-minimal curl --output /dev/null --progress-meter --form file=@./test.tar.gz "https://www.ctan.org/submit/validate" > % Total % Received % Xferd Average Speed Time Time Time Current > Dload Upload Total Spent Left Speed > 100 100M 0 345 100 100M 5 1484k 0:01:08 0:01:08 --:--:-- 1400k > > (tcp_congestion_control=custom, tcp_notsent_lowat=default, [rw]mem_max=default) > $ sudo sysctl -w net.ipv4.tcp_congestion_control=bbr net.ipv4.tcp_notsent_lowat=4294967295 net.core.rmem_max=212992 net.core.wmem_max=212992 > $ podman run --rm --pull=newer --volume="$(realpath .):/srv/:Z" --workdir=/srv/ --network=pasta quay.io/fedora/fedora-minimal curl --output /dev/null --progress-meter --form file=@./test.tar.gz "https://www.ctan.org/submit/validate" > % Total % Received % Xferd Average Speed Time Time Time Current > Dload Upload Total Spent Left Speed > 100 100M 0 345 100 100M 12 3576k 0:00:28 0:00:28 --:--:-- 14.9M > > (tcp_congestion_control=custom, tcp_notsent_lowat=custom, [rw]mem_max=default) > $ sudo sysctl -w net.ipv4.tcp_congestion_control=bbr net.ipv4.tcp_notsent_lowat=131072 net.core.rmem_max=212992 net.core.wmem_max=212992 > $ podman run --rm --pull=newer --volume="$(realpath .):/srv/:Z" --workdir=/srv/ --network=pasta quay.io/fedora/fedora-minimal curl --output /dev/null --progress-meter --form file=@./test.tar.gz "https://www.ctan.org/submit/validate" > % Total % Received % Xferd Average Speed Time Time Time Current > Dload Upload Total Spent Left Speed > 100 100M 0 345 100 100M 8 2595k 0:00:39 0:00:39 --:--:-- 17.1M > > (tcp_congestion_control=default, tcp_notsent_lowat=default, [rw]mem_max=custom) > $ sudo sysctl -w net.ipv4.tcp_congestion_control=cubic net.ipv4.tcp_notsent_lowat=4294967295 net.core.rmem_max=7500000 net.core.wmem_max=7500000 > $ podman run --rm --pull=newer --volume="$(realpath .):/srv/:Z" --workdir=/srv/ --network=pasta quay.io/fedora/fedora-minimal curl --output /dev/null --progress-meter --form file=@./test.tar.gz "https://www.ctan.org/submit/validate" > % Total % Received % Xferd Average Speed Time Time Time Current > Dload Upload Total Spent Left Speed > 100 100M 0 345 100 100M 7 2107k 0:00:48 0:00:48 --:--:-- 8310k > > (tcp_congestion_control=custom, tcp_notsent_lowat=default, [rw]mem_max=custom) > $ sudo sysctl -w net.ipv4.tcp_congestion_control=bbr net.ipv4.tcp_notsent_lowat=4294967295 net.core.rmem_max=7500000 net.core.wmem_max=7500000 > $ podman run --rm --pull=newer --volume="$(realpath .):/srv/:Z" --workdir=/srv/ --network=pasta quay.io/fedora/fedora-minimal curl --output /dev/null --progress-meter --form file=@./test.tar.gz "https://www.ctan.org/submit/validate" > % Total % Received % Xferd Average Speed Time Time Time Current > Dload Upload Total Spent Left Speed > 100 100M 0 345 100 100M 15 4620k 0:00:22 0:00:22 --:--:-- 12.1M > > (tcp_congestion_control=custom, tcp_notsent_lowat=custom, [rw]mem_max=custom) > $ sudo sysctl -w net.ipv4.tcp_congestion_control=bbr net.ipv4.tcp_notsent_lowat=131072 net.core.rmem_max=7500000 net.core.wmem_max=7500000 > $ podman run --rm --pull=newer --volume="$(realpath .):/srv/:Z" --workdir=/srv/ --network=pasta quay.io/fedora/fedora-minimal curl --output /dev/null --progress-meter --form file=@./test.tar.gz "https://www.ctan.org/submit/validate" > % Total % Received % Xferd Average Speed Time Time Time Current > Dload Upload Total Spent Left Speed > 5 100M 0 0 5 5376k 0 29899 0:58:27 0:03:04 0:55:23 0 > (Stopped early since I got sick of waiting) > > (Test with --network=host again) > $ podman run --rm --pull=newer --volume="$(realpath .):/srv/:Z" --workdir=/srv/ --network=host quay.io/fedora/fedora-minimal curl --output /dev/null --progress-meter --form file=@./test.tar.gz "https://www.ctan.org/submit/validate" > % Total % Received % Xferd Average Speed Time Time Time Current > Dload Upload Total Spent Left Speed > 100 100M 0 345 100 100M 69 20.0M 0:00:04 0:00:04 --:--:-- 24.3M Well, that sure is a confusing pattern. Looks like tcp_notsent_lowat is the biggest culprit... except for the outlier where it seems ok with both custom notsent_lowat and custom congestion control. How repeatable are each of these results? > I know fairly little about networking and the kernel, so if the answer > is just "don't set those sysctls together", I'd be okay with that. But I > haven't changed these sysctls since February, my upload speeds via pasta > were fine up until a few months ago, and the upload speeds are still > okay with --network=host, so I suspect that this is a bug somewhere. I have a vague theory why tcp_notsent_lowat might be bad for pasta but not --network=host. I'm far from certain about it though, and I don't have a good idea what change on the pasta side might have changed. Actually, is it just pasta that's changed in that time? Or could it be a new kernel version or something else? I'm having some trouble wrapping my head around what tcp_notsent_lowat does, but I _think_ the upshot of it will be to make the socket send path more "bursty": we'll fill the socket send buffer (which could be very large), then stop. We won't wake up again to refill until the amount in the socket buffer drops below tcp_notsend_lowat. Assuming we've been getting plenty of data from the other side, we'll then refill the buffer very quickly. Less wakups sounds like a good thing in the abstract, but I'm wondering if this burstiness is messing with the flow control on the other side, causing the guest side peer to slow down to nothing. If that is the case, it would make sense that it gets even worse with large socket buffer sizes, since that would make the "bursts" even bigger. Likewise I was wondering if having different congestion control algorithms on each side might somehow conflict causing a slowdown. I don't think the data you have above really suggest that though. > I also find it quite interesting that setting any of the sysctls > individually or in pairs improves the upload speeds, but setting all 3 > at once slows it down drastically. I'm curious about the case with default congestion control, but custom lowat and mem_max. I think that one's missing from the table above. > I bisected a kernel a few weeks ago, > so I can try that here if you think that this is a kernel bug and not a > pasta bug. Could be either, I have no real intuition either way so far. -- David Gibson (he or they) | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you, not the other way | around. http://www.ozlabs.org/~dgibson

2 1

Re: pasta slow at HTTP upload
by Stefano Brivio 13 Nov '25

13 Nov '25

Hi Max, On Tue, 11 Nov 2025 23:11:17 -0700 Max Chernoff <git(a)maxchernoff.ca> wrote: > Hi, > > For the past few months, I've noticed that my HTTP uploads from > containers are really slow. Reproduction: > > [...] > > - The host that I ran this from is a VM in a datacenter, supposedly with > a symmetrical 1Gb/s connection. Pinging github.com shows an RTT of > 2ms, and pinging www.ctan.org shows an RTT of 100ms. Thanks for reporting this. Coincidentally, I'm currently debugging an issue that looks similar to this, and I should have some patches for you to test in a couple of days, I'll keep you posted. -- Stefano

2 4

pasta slow at HTTP upload
by Max Chernoff 12 Nov '25

12 Nov '25

Hi, For the past few months, I've noticed that my HTTP uploads from containers are really slow. Reproduction: $ cd "$(mktemp -d)" $ head --bytes=100M /dev/urandom > test.bin $ tar cf ./test.tar.gz ./test.bin $ chmod -R a+rx . $ curl --output /dev/null --progress-meter --form file=@./test.tar.gz "https://www.ctan.org/submit/validate" % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 100M 0 345 100 100M 69 20.0M 0:00:04 0:00:04 --:--:-- 23.9M $ podman run --rm --pull=newer --volume="$(realpath .):/srv/:Z" --workdir=/srv/ --network=pasta quay.io/fedora/fedora-minimal curl --output /dev/null --progress-meter --form file=@./test.tar.gz "https://www.ctan.org/submit/validate" % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 6 100M 0 0 6 6912k 0 9517 3:03:39 0:12:23 2:51:16 0 (Stopped after 10 minutes since I got sick of waiting) $ podman run --rm --pull=newer --volume="$(realpath .):/srv/:Z" --workdir=/srv/ --network=host quay.io/fedora/fedora-minimal curl --output /dev/null --progress-meter --form file=@./test.tar.gz "https://www.ctan.org/submit/validate" % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 100M 0 345 100 100M 64 18.7M 0:00:05 0:00:05 --:--:-- 23.8M Some comments about the above: - "www.ctan.org" is a web server running Apache; it isn't my server, but it's where I first noticed this issue, and I have permission from its owners to use it for testing. - The upload speed is constant on the host and with --network=host, but with --network=pasta, it starts high-ish, but quickly and continuously decreases. But the "bytes transferred" number keeps increasing, so the upload hasn't stalled out completely. - I've also had problems with uploads to GitHub from a custom Go binary---it takes ~3 minutes to upload ~600MB, split across 5 files, which is much slower than I would expect, but still faster than the curl/CTAN upload speeds. So I don't think that this is a bug with curl or this specific web server. - The host that I ran this from is a VM in a datacenter, supposedly with a symmetrical 1Gb/s connection. Pinging github.com shows an RTT of 2ms, and pinging www.ctan.org shows an RTT of 100ms. - This problem seems to have started a few months ago. I don't have an exact date (sorry), but it seems to have been fine before then. - The host is running Fedora 43 IoT on x86_64, and I'm using the following software versions: $ rpm --query --queryformat='%{NAME}\t%{VERSION}\n' passt podman containers-common netavark | sort | column --table containers-common 0.64.2 netavark 1.16.1 passt 0^20250919.g623dbf6 podman 5.6.2 I had the same issue when the host/containers were using Fedora 42. - The host has fairly little CPU/network load when I ran these tests, and pasta's CPU usage was low enough that it didn't show up on the first page of htop. Does anybody have any advice on troubleshooting this? I'm willing to test with other servers/commands, compile/test any patches, or provide any other information that might be helpful. Thanks, -- Max

1 0