Pasta-networked rootless Podman container gets Connection Refused with the host's public IP
Hi; I previously asked this on the Podman mailing list, but I'm not sure if the issue in question is a feature of Podman or Passt (or both), and I got no replies from the Podman list, so I figured I'd try here as well. We're running some rootless Podman containers set up to use Pasta 2023_03_29.b10b983 for networking. One of the containers needs to access the host machine port 443 with its public IP address, but this causes a Connection Refused error. Any other public IP is accessible normally. This is specific to the containers; the host has no problem accessing itself with the public IP. The containers are set up with systemd generators (quadlet), with networking configured very simply: "Network=pasta:-t,auto,-T,auto" Podman has a --map-gw option useable with Pasta that seemed like it might help, but it didn't. "Network=pasta:--map-gw,-t,auto,-T,auto" fails like this at container startup: Error: failed to start pasta: Port forwarding mode 'none' conflicts with previous mode "Network=pasta:-t,auto,-T,auto,--map-gw" started the container fine, but did not fix the Connection Refused error. Apparently --map-gw just isn't the right option here. I don't know if the inability to contact the public IP is a feature of Podman or Pasta, but I'm hoping you're able to at least narrow it down for me. Is there a workaround on the Pasta side? Thanks in advance! - JK Laiho
On Wed, Jun 28, 2023 at 11:02:55AM +0300, jklaiho@iki.fi wrote:
Hi; I previously asked this on the Podman mailing list, but I'm not sure if the issue in question is a feature of Podman or Passt (or both), and I got no replies from the Podman list, so I figured I'd try here as well.
This behaviour is a property of passt. It's a consequence of a tradeoff that we make differently from slirp or kernel masquerading approaches. Pasta (usually) avoids NAT, which can avoid a number of problems, but the way it does this is by giving the container the host's IP address (or one of them, if the host has multiple). The tradeoff is that that implies the container can't contact the host by that IP address.
We're running some rootless Podman containers set up to use Pasta 2023_03_29.b10b983 for networking. One of the containers needs to access the host machine port 443 with its public IP address, but this causes a Connection Refused error. Any other public IP is accessible normally.
Right, this is expected. Because the container itself also has that IP, it will route that traffic to itself. It will never even reach pasta, let alone the host. I'm assuming there's no server on that port in the container, hence, connection refused.
This is specific to the containers; the host has no problem accessing itself with the public IP.
The containers are set up with systemd generators (quadlet), with networking configured very simply:
"Network=pasta:-t,auto,-T,auto"
Podman has a --map-gw option useable with Pasta that seemed like it might help, but it didn't.
"Network=pasta:--map-gw,-t,auto,-T,auto" fails like this at container startup:
Error: failed to start pasta: Port forwarding mode 'none' conflicts with previous mode
"Network=pasta:-t,auto,-T,auto,--map-gw" started the container fine, but did not fix the Connection Refused error. Apparently --map-gw just isn't the right option here.
It's odd that those last two approaches gave different results, AFAIK just changing the order of options shouldn't make a difference here. But in any case, --map-gw (which from pasta's point of view is removing the --no-map-gw option) will not do what you want for two reasons. 1. map-gw does provide a means for the container to access the host, but it's not via the host's normal public IP (that's impossible if the guest has it). Instead it repurposes the IP of the default gateway to refer to the host when used from the container. So to use this you'd need to change the address that your clients in the container use, which I gather isn't possible. 2. With map-gw, traffic forwaded appears on the host side to both come from and go to the loopback interface. That is from the servier's point of view the connection will go to 127.0.0.1, not the host's public IP. AIUI that won't work for your situation.
I don't know if the inability to contact the public IP is a feature of Podman or Pasta, but I'm hoping you're able to at least narrow it down for me.
Is there a workaround on the Pasta side?
Maybe. You can add the -a <address> option to the pasta command line which will tell it to assign the given address to the guest instead of the host's address. This should make the host contactable using it's public IP. However, it may cause other issues, since the container's IP as it sees itself will no longer be the same as the container's IP as things outside see it. You'll obviously also have to pick an address which won't conflict with anything else the guest needs to contact. -- David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson
Hi JK,
On Wed, 28 Jun 2023 11:02:55 +0300
"jklaiho@iki.fi"
Hi; I previously asked this on the Podman mailing list, but I'm not sure if the issue in question is a feature of Podman or Passt (or both), and I got no replies from the Podman list, so I figured I'd try here as well.
I actually saw an answer to that, did you miss this perhaps? https://lists.podman.io/archives/list/podman@lists.podman.io/message/AAL6OMI... Anyway, in more detail:
We're running some rootless Podman containers set up to use Pasta 2023_03_29.b10b983 for networking. One of the containers needs to access the host machine port 443 with its public IP address, but this causes a Connection Refused error. Any other public IP is accessible normally.
This is specific to the containers; the host has no problem accessing itself with the public IP.
The containers are set up with systemd generators (quadlet), with networking configured very simply:
"Network=pasta:-t,auto,-T,auto"
Podman has a --map-gw option useable with Pasta that seemed like it might help, but it didn't.
That would help you (minus the issue you hit) if you wanted to connect to the parent network namespace ("host") using another address, namely the address of the default gateway (only option at the moment, we plan to make it configurable) -- but not the public address.
"Network=pasta:--map-gw,-t,auto,-T,auto" fails like this at container startup:
Error: failed to start pasta: Port forwarding mode 'none' conflicts with previous mode
The problem here comes from the fact that the Podman integration already passes "-t none" by default, and the override here doesn't quite work, because pasta doesn't accept overriding options on its command line. Paul (Cc'ed) plans to send a patchset to fix this in pasta itself.
"Network=pasta:-t,auto,-T,auto,--map-gw" started the container fine, but did not fix the Connection Refused error. Apparently --map-gw just isn't the right option here.
"-T auto" is yet a different thing: that would map ports from your container to the "host" via loopback addresses (127.0.0.1 or ::1).
I don't know if the inability to contact the public IP is a feature of Podman or Pasta, but I'm hoping you're able to at least narrow it down for me.
Is there a workaround on the Pasta side?
I think the issue you're facing is that, by default, your container gets the same set of addresses as _one interface_ (the first one with a default route) on the host. Then you use one of those addresses to connect... but you already have that address in the container, so the container will try to connect to itself (hence the connection refused). There are two options I see: - you could change the address you use to connect to the HTTPS service on the host -- if you're in the container, you can use the address of the default gateway as reported by ip -4 route show / ip -6 route show (with --map-gw), or 127.0.0.1/::1 (with -T auto, or -T 443) - you could change the address of your container. You can specify address and default gateway with -a / -g (currently, only once for IPv4, and once for IPv6). Or you can select another interface on the host which pasta uses to source addresses and routes. For that, on the version of the passt package you're using, you're limited to "-i", which simply specifies the interface from which address and default gateways are sourced. Later versions implement a distinction between that "template" interface and outbound interfaces (--outbound-if4, --outbound-if6). I hope this helps. If you're facing issues with this, it would help if you could share addressing and routing information from your host and container (you can obfuscate by translating addresses consistently to TEST-NET subnets for IPv4, such as 192.0.2.0/24 and 198.51.100.0/24, or 2001:db8::/32 for IPv6). By the way of package versions, I requested just yesterday the first upload of an updated version after the Debian 12 release (I'm a maintainer without upload rights): https://qa.debian.org/cgi-bin/vcswatch?package=passt which should be synchronised to Ubuntu in a while (I think it's a couple of weeks after that). -- Stefano
participants (3)
-
David Gibson
-
jklaiho＠iki.fi
-
Stefano Brivio