Re: Pesto Protocol Proposals, imProved

On Fri, Mar 06, 2026 at 11:58:14AM +0100, Stefano Brivio wrote:

On Fri, 6 Mar 2026 12:08:07 +1100 David Gibson wrote:

...

Stefano convinced me that my earlier proposal for the dynamic update protocol was unnecessarily complex. Plus, I saw a much better way of handling socket continuity in the context of a "whole table" replacement. So here's an entirely revised protocol suggestion.

# Outline

I suggest that each connection to the control socket handles a single transaction.

1. Server hello - Server sends magic number, version - Possibly feature flags / limits (e.g. max number of rules allowed)

Feature flags and limits could be fixed depending on the version, for simplicity.

We could, but it's a tradeoff between simplicity right now, and simplicity of later alterations. Fwiw, I was thinking of feature flags for things that already fit into this general scheme, but might not be immediately implemented. For example, allowing rules for protocols other than TCP & UDP.

If pifs are unexpected (somebody trying to forward ports to a container and touching passt instead) we should find out as part of 3. I can't think of other substantial types of mismatches.

If we extend the table to control the target side address (which we probably want, at least once we have a table for tap packets) we might have some options for different mapping nodes (many to one, identity. If we don't implement all those immediately, feature flags could be useful.

...

2. Client hello - Client sends magic number - Do we need anything else?

As long as we have a version reported by the server, we should be fine. We'll just increase it if we need something else.

The model I had in mind was that the server just advertises a version, and it's up to the client to match it. Again, reducing server complexity at the cost of client complexity. I was thinking the server could advertise a min_compat_version, like with migration, for cases where the new version is strictly backwards compatible with the old one (e.g. identical layout, but some things that weren't permitted before now are).

Do we want a client version too?

I don't think we need one. I think a client magic number is a good idea though. We have a binary protocol that will largely be made up of IP addresses and u16s, where all (or at least most) possible bit patterns are valid. A client magic number removes the risk that some random unrelated program could connect to the socket, sends some garbage to passt and have it do anything other than just reject the transaction with an error.

...

3. Server lists pifs - Server sends the number of pifs, their indices and names

Up to here, I guess we can skip all this for an initial Podman-side-complete implementation.

Not really sure what you mean.

...

4. Server lists rules - Server sends the list of rules, one pif at a time

Could this be a fixed-size blob with up to, say, 16 pifs?

It could, but I don't think it's a great idea. Allowing a variable number isn't that hard to implement, and decouples the protocol from the specific internal values of PIF_NUM_TYPES and MAX_FWD_RULES. I guess we could do it for a PoC, but I don't think we should release something with a fixed blob.

We'll need to generalise pifs at some point. I'm not sure if it makes things simpler. I would defer this to the implementation.

...

5. Client gives new rules - Client sends the new list of rules, one pif at a time - Server loads them into the shadow table, and validates (no socket operations)

Is it one shadow table per pif or one with everything?

Conceptually, one shadow table globally. Internally I guess it would be one per pif, but I was intending that the whole shebang be switched as a unit.

If it's one per pif, do we want to have the whole exchange prepended by "load table for pif x" or "store table for pif y" commands?

I would suggest not, at the moment, as it looks slightly complicated, but eventually in a later version we could switch to that.

We could, but I don't really see a reason we'd want to.

...

6. Server acknowledges - Either reports an error and disconnects, or acks waiting for client 7. Client signals apply - Server swaps shadow and active tables, and syncs sockets with new active table 8. Server gives error summary - Server reports bind/listen/whatever errors 9a. Client signals commit - Shadow table (now the old table) discarded or 9b. Client signals rollback - Shadow and active tables swapped back, syncs sockets - Discard shadow table (now the "new" table again) - New bind error report?

Do we need these as five separate steps?

Couldn't the server simply apply or try to apply as soon as the client is done, and acknowledge or return error once everything is done?

What about this instead:

5. Client sends new rules (blob of known size)

6. Server receives, loads into shadow table, swaps tables and syncs socket, with rollback to old table on error.

8. Server sends error / success summary (single byte, at least in this version)

...

10. Server closes control connection

Uh.. yeah, I think that works. I still think a client magic number is worthwhile, but that could be considered part of (5). In this case I do think the error code needs at least 3 values: success, "safe" fail (before we attempted any socket operations), socket operation fail. We might also want one if something went wrong during the rollback (e.g. we removed a mapping, and some other program bound the port before we rolled back).

...if we keep my 8. above, it would be more logical that the client closes the connection.

I think having the server close the connection will simplify state management inside passt a little: we don't have to have handle events for closing our side once the client has closed, or deal with the client incorrectly sending garbage after the transaction. Once we've sent the error code, we're signalling unambiguously: the transaction is over now, if you want to do something else, the client must start again.

...

# Client disconnects

A client disconnect before step (7) is straightforward: discard the shadow table, nothing has changed.

A client disconnect between (7) and (9) triggers a rollback, same as (9b).

In my modified version, a client disconnect during 5. would trigger discarding of the shadow table that's being filled (kind of no-op really).

Right.

A disconnect after that doesn't affect the following steps instead, but the server won't report error or success.

That's one option. We could also force a rollback in this case (essentially, roll back if we get an error sending the status back to the client). I have no strong preference.

...

# Error reporting

Error reporting at step (6) is fairly straightforward: we can send an error code and/or an error message.

Error reporting at (8) is trickier. As a first cut, we could just report "yes" or "no" - taking into account the FWD_WEAK flag. But the client might be able to make better decisions or at least better messages to the user if we report more detailed information. Exactly how detailed is an open question: number of bind failures? number of failures per rule? specific ports which failed?

For the moment I would report a single byte.

Later, we could probably send back the list of rules with a success / error type version for each one of them. Think of just sending the same type of fixed-size table back and forth.

Right. We could put an error count / status field in each rule and consider it part of the table.

...

# Interim steps

I propose these steps toward implementing this:

i. Merge TCP and UDP rule tables. The protocol above assumes a single rule table per-pif, which I think is an easier model to understand and more extensible for future protocol support. ii. Read-only client. Implement steps (1) to (4). Client can query and list the current rules, but not change them. iii. Rule updates. Implement remaining protocol steps, but with a "close and re-open" approach on the server, so unaltered listening sockets might briefly disappear iv. Socket continuity. Have the socket sync "steal" sockets from the old table in preference to re-opening them.

If you have any time to work on (ii) while I work on (i), those should be parallelizable.

Yes, I'll start adapting the existing draft as soon as possible. I think ii. could go in parallel with all the other steps, I can just call some stubs meanwhile.

Ok, great.

...

# Concurrent updates

Server guarantees that a single transaction as above is atomic in the sense that nothing else is allowed to change the rules between (4) and (9). The easiest way to do that initially is probably to only allow a single client connection at a time.

I would call this a feature...

I tend to agree.

...

If there's a reason to, we could alter that so that concurrent connections are allowed, but if another client changed anything after step (4), then we give an error on the next op (or maybe just close the control socket from the server side).

...even if we go for my modified version.

Yes.

...

# Tweaks / variants

- I'm not sure that step (2) is necessary

I would skip it. The only reason why we might want it is to send a client version, but we can also implement sending of a client version only starting from a newer server version.

Again, I think the magic number is important, but we could fold that into a later step.

...

- I'm not certain that step (7) is necessary, although I do kind of prefer the client getting a chance to see a "so far, so good" before any socket operations happen.

I think it's quite unrealistic that we'll ever manage to build some sensible logic to decide what to do depending on partial failures.

Eh, maybe. An error count per rule would be enough to move the implementation of FWD_WEAK to the client.

If so far is not good, the server should just abort, and the user will have to fix mistakes and try again.

Yeah, that makes sense. -- David Gibson (he or they) | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you, not the other way | around. http://www.ozlabs.org/~dgibson