February 2025 - dev

[PATCH v21 0/5] State migration
by David Gibson 13 Feb '25

13 Feb '25

As discussed on IRC, I think we finally squashed, or at least worked around, the bugs with restoring the sendq. David Gibson (3): fixup: TCP_REPAIR_WINDOW before send unsent fixup?: Don't fail whole migration for errors restoring sndq debug Stefano Brivio (2): migrate: Migrate TCP flows test: Add migration tests contrib/selinux/passt.te | 4 +- flow.c | 260 ++++++++++ flow.h | 8 + migrate.c | 19 + passt.c | 6 +- repair.c | 1 - tcp.c | 919 ++++++++++++++++++++++++++++++++++++ tcp_conn.h | 103 ++++ test/lib/layout | 55 ++- test/lib/setup | 138 +++++- test/lib/test | 48 ++ test/migrate/basic | 59 +++ test/migrate/bidirectional | 64 +++ test/migrate/iperf3_bidir6 | 58 +++ test/migrate/iperf3_in4 | 50 ++ test/migrate/iperf3_in6 | 58 +++ test/migrate/iperf3_out4 | 50 ++ test/migrate/iperf3_out6 | 58 +++ test/migrate/rampstream_in | 10 +- test/migrate/rampstream_out | 6 +- test/run | 46 +- 21 files changed, 1991 insertions(+), 29 deletions(-) create mode 100644 test/migrate/basic create mode 100644 test/migrate/bidirectional create mode 100644 test/migrate/iperf3_bidir6 create mode 100644 test/migrate/iperf3_in4 create mode 100644 test/migrate/iperf3_in6 create mode 100644 test/migrate/iperf3_out4 create mode 100644 test/migrate/iperf3_out6 -- 2.48.1

1 5

[PATCH v20 0/5] State migration
by David Gibson 13 Feb '25

13 Feb '25

Has some temporary hacks for debugging problems with migrate/iperf3_bidir6. Has a couple of tentative fixes for possible bugs, but it's proving difficult to characterize the errors. David Gibson (3): fixup?: loop on EGAIN in repair queue depause fixup: TCP_REPAIR_WINDOW before send unsent Stefano Brivio (2): migrate: Migrate TCP flows test: Add migration tests contrib/selinux/passt.te | 4 +- flow.c | 260 ++++++++++ flow.h | 8 + migrate.c | 19 + passt.c | 6 +- repair.c | 1 - tcp.c | 915 ++++++++++++++++++++++++++++++++++++ tcp_conn.h | 103 ++++ test/lib/layout | 55 ++- test/lib/setup | 138 +++++- test/lib/test | 48 ++ test/migrate/basic | 59 +++ test/migrate/bidirectional | 64 +++ test/migrate/iperf3_bidir6 | 58 +++ test/migrate/iperf3_in4 | 50 ++ test/migrate/iperf3_in6 | 58 +++ test/migrate/iperf3_out4 | 50 ++ test/migrate/iperf3_out6 | 58 +++ test/migrate/rampstream_in | 10 +- test/migrate/rampstream_out | 6 +- test/run | 46 +- 21 files changed, 1987 insertions(+), 29 deletions(-) create mode 100644 test/migrate/basic create mode 100644 test/migrate/bidirectional create mode 100644 test/migrate/iperf3_bidir6 create mode 100644 test/migrate/iperf3_in4 create mode 100644 test/migrate/iperf3_in6 create mode 100644 test/migrate/iperf3_out4 create mode 100644 test/migrate/iperf3_out6 -- 2.48.1

1 5

[PATCH] selinux: Add rules needed to run tests
by Stefano Brivio 13 Feb '25

13 Feb '25

...other than being convenient, they might be reasonably representative of typical stand-alone usage. Signed-off-by: Stefano Brivio <sbrivio(a)redhat.com> --- contrib/selinux/passt.te | 4 ++++ contrib/selinux/pasta.te | 14 ++++++++++++-- 2 files changed, 16 insertions(+), 2 deletions(-) diff --git a/contrib/selinux/passt.te b/contrib/selinux/passt.te index c6cea34..6e7a4cb 100644 --- a/contrib/selinux/passt.te +++ b/contrib/selinux/passt.te @@ -20,6 +20,7 @@ require { type fs_t; type tmp_t; type user_tmp_t; + type user_home_t; type tmpfs_t; type root_t; @@ -80,6 +81,9 @@ allow passt_t root_t:dir mounton; allow passt_t tmp_t:dir { add_name mounton remove_name write }; allow passt_t tmpfs_t:filesystem mount; allow passt_t fs_t:filesystem unmount; +allow passt_t user_home_t:dir search; +allow passt_t user_tmp_t:fifo_file append; +allow passt_t user_tmp_t:file map; manage_files_pattern(passt_t, user_tmp_t, user_tmp_t) files_pid_filetrans(passt_t, user_tmp_t, file) diff --git a/contrib/selinux/pasta.te b/contrib/selinux/pasta.te index d0ff0cc..89c8043 100644 --- a/contrib/selinux/pasta.te +++ b/contrib/selinux/pasta.te @@ -18,6 +18,7 @@ require { type bin_t; type user_home_t; type user_home_dir_t; + type user_tmp_t; type fs_t; type tmp_t; type tmpfs_t; @@ -56,8 +57,10 @@ require { attribute port_type; type port_t; type http_port_t; + type http_cache_port_t; type ssh_port_t; type reserved_port_t; + type unreserved_port_t; type dns_port_t; type dhcpc_port_t; type chronyd_port_t; @@ -122,8 +125,8 @@ domain_auto_trans(pasta_t, ping_exec_t, ping_t); allow pasta_t nsfs_t:file { open read }; -allow pasta_t user_home_t:dir getattr; -allow pasta_t user_home_t:file { open read getattr setattr }; +allow pasta_t user_home_t:dir { getattr search }; +allow pasta_t user_home_t:file { open read getattr setattr execute execute_no_trans map}; allow pasta_t user_home_dir_t:dir { search getattr open add_name read write }; allow pasta_t user_home_dir_t:file { create open read write }; allow pasta_t tmp_t:dir { add_name mounton remove_name write }; @@ -133,6 +136,11 @@ allow pasta_t root_t:dir mounton; manage_files_pattern(pasta_t, pasta_pid_t, pasta_pid_t) files_pid_filetrans(pasta_t, pasta_pid_t, file) +allow pasta_t user_tmp_t:dir { add_name remove_name search write }; +allow pasta_t user_tmp_t:fifo_file append; +allow pasta_t user_tmp_t:file { create open write }; +allow pasta_t user_tmp_t:sock_file { create unlink }; + allow pasta_t console_device_t:chr_file { open write getattr ioctl }; allow pasta_t user_devpts_t:chr_file { getattr read write ioctl }; logging_send_syslog_msg(pasta_t) @@ -160,6 +168,8 @@ allow pasta_t self:udp_socket create_stream_socket_perms; allow pasta_t reserved_port_t:udp_socket name_bind; allow pasta_t llmnr_port_t:tcp_socket name_bind; allow pasta_t llmnr_port_t:udp_socket name_bind; +allow pasta_t http_cache_port_t:tcp_socket { name_bind name_connect }; +allow pasta_t unreserved_port_t:udp_socket name_bind; corenet_udp_sendrecv_generic_node(pasta_t) corenet_udp_bind_generic_node(pasta_t) allow pasta_t node_t:icmp_socket { name_bind node_bind }; -- 2.43.0

1 0

[PATCH v19 0/2] A-bit-less-draft state migration
by Stefano Brivio 13 Feb '25

13 Feb '25

Compared to v18: - add save and restore of permitted SACK and timestamps - return ENOBUFS if pending queues are too large, not ENOMEM - print SO_PEEK_OFF state from v18 9/9 - fix headers of rampstream_in and rampstream_out tests - filter out non-established TCP flows - fix SELinux rule for queue ioctl(): it's getattr, not ioctl Still missing: - reasonable error handling, and "empty flows" if target expects them but we fail to fetch them - tests with half-closed sockets Stefano Brivio (2): migrate: Migrate TCP flows test: Add migration tests contrib/selinux/passt.te | 4 +- flow.c | 260 ++++++++++ flow.h | 8 + migrate.c | 19 + passt.c | 6 +- repair.c | 1 - tcp.c | 916 ++++++++++++++++++++++++++++++++++++ tcp_conn.h | 103 ++++ test/lib/layout | 55 ++- test/lib/setup | 138 +++++- test/lib/test | 48 ++ test/migrate/basic | 59 +++ test/migrate/bidirectional | 64 +++ test/migrate/iperf3_bidir6 | 58 +++ test/migrate/iperf3_in4 | 50 ++ test/migrate/iperf3_in6 | 58 +++ test/migrate/iperf3_out4 | 50 ++ test/migrate/iperf3_out6 | 58 +++ test/migrate/rampstream_in | 10 +- test/migrate/rampstream_out | 6 +- test/run | 29 ++ 21 files changed, 1984 insertions(+), 16 deletions(-) create mode 100644 test/migrate/basic create mode 100644 test/migrate/bidirectional create mode 100644 test/migrate/iperf3_bidir6 create mode 100644 test/migrate/iperf3_in4 create mode 100644 test/migrate/iperf3_in6 create mode 100644 test/migrate/iperf3_out4 create mode 100644 test/migrate/iperf3_out6 -- 2.43.0

1 2

[PATCH v18 0/9] State migration (kinda draft again)
by David Gibson 12 Feb '25

12 Feb '25

More debugging of bugs found with the rampstream test today. * Send queue transfer bug I spotted by inspection a nasty bug which would mean we never properly transfer the send queue (in repair mode we read it into the the wrong buffer, then transferred the right one). I think we only didn't hit this because in each of the cases I've seen the send queue has been empty. I think that makes sense: with high speed local transfers, there's probably more than enough time between when the guest is stopped and when we dump the queue for the sndbuf to drain completely to the peer. * Queue transfer error checking I've made the error checking when extracting and reloading the queue a bit more robust. * rampstream corruption bug I found the cause of the stream corruption bug. I don't think repair mode fully supports SO_PEEK_OFF semantics, but apparently it shares enough code with the normal recv() path that the peek offset applied, which meant (I think) we skipped the "already sent" portion of the rcv queue when dumping it. I've fixed this by disabling SO_PEEK_OFF (setting it to -1) before migration on the source. This needs some fixing to deal correctly with the case of a failed migration which resumes on the source. * rampstream unexpected EOF bug Unfortunately that wasn't the only bug. With the peek offset fixed, I no longer get stream corruption, but I still get unexpected EOFs on the rampstream_in test (at least with 64M [rw]mem_max). The receiving rampstream is getting an EOF because passt is sending an RST to the guest 14.1750: Flow 0 (TCP connection): TCP reset at tcp_sock_handler:2270 That happens because we get an EPOLLERR on the socket at some point after migration. From some earlier debugging hacks, I think that's an ECONNRESET specifically, but I haven't debugged further because I was focused on the corruption bug. David Gibson (3): migrate: Migrate guest observed addresses rampstream: Add utility to test for corruption of data streams debug Stefano Brivio (6): migrate: Skeleton of live migration logic Add interfaces and configuration bits for passt-repair vhost_user: Make source quit after reporting migration state tcp: Get bound address for connected inbound sockets too migrate: Migrate TCP flows test: Add migration tests Makefile | 14 +- conf.c | 43 +- contrib/selinux/passt.te | 2 +- epoll_type.h | 6 +- flow.c | 259 +++++++++- flow.h | 8 + flow_table.h | 6 +- migrate.c | 309 ++++++++++++ migrate.h | 51 ++ passt.1 | 11 + passt.c | 21 +- passt.h | 15 + repair.c | 218 +++++++++ repair.h | 16 + tap.c | 65 +-- tcp.c | 921 +++++++++++++++++++++++++++++++++++- tcp_conn.h | 99 ++++ test/.gitignore | 1 + test/Makefile | 5 +- test/lib/layout | 55 ++- test/lib/setup | 140 +++++- test/lib/test | 48 ++ test/migrate/basic | 59 +++ test/migrate/bidirectional | 64 +++ test/migrate/iperf3_bidir6 | 58 +++ test/migrate/iperf3_in4 | 50 ++ test/migrate/iperf3_in6 | 58 +++ test/migrate/iperf3_out4 | 50 ++ test/migrate/iperf3_out6 | 58 +++ test/migrate/rampstream_in | 60 +++ test/migrate/rampstream_out | 55 +++ test/passt.mbuto | 5 +- test/rampstream-check.sh | 3 + test/rampstream.c | 143 ++++++ test/run | 29 ++ util.c | 62 +++ util.h | 30 ++ vhost_user.c | 67 +-- virtio.h | 4 - vu_common.c | 49 +- vu_common.h | 2 +- 41 files changed, 3014 insertions(+), 205 deletions(-) create mode 100644 migrate.c create mode 100644 migrate.h create mode 100644 repair.c create mode 100644 repair.h create mode 100644 test/migrate/basic create mode 100644 test/migrate/bidirectional create mode 100644 test/migrate/iperf3_bidir6 create mode 100644 test/migrate/iperf3_in4 create mode 100644 test/migrate/iperf3_in6 create mode 100644 test/migrate/iperf3_out4 create mode 100644 test/migrate/iperf3_out6 create mode 100644 test/migrate/rampstream_in create mode 100644 test/migrate/rampstream_out create mode 100755 test/rampstream-check.sh create mode 100644 test/rampstream.c -- 2.48.1

2 10

[PATCH] tcp_vu: Fix off-by one in header count array adjustment
by Stefano Brivio 12 Feb '25

12 Feb '25

From: Laurent Vivier <lvivier(a)redhat.com> Not entirely clear to me why, but Laurent proposed this patch to fix an issue were we would end up with a zero buf_cnt in tcp_vu_data_from_sock() and all sorts of weirdnesses. Reported-by: David Gibson <david(a)gibson.dropbear.id.au> [sbrivio: commit message, albeit not really descriptive] Signed-off-by: Stefano Brivio <sbrivio(a)redhat.com> --- tcp_vu.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tcp_vu.c b/tcp_vu.c index fad7065..0622f17 100644 --- a/tcp_vu.c +++ b/tcp_vu.c @@ -261,7 +261,7 @@ static ssize_t tcp_vu_sock_recv(const struct ctx *c, len -= iov->iov_len; } /* adjust head count */ - while (head_cnt > 0 && head[head_cnt - 1] > i) + while (head_cnt > 0 && head[head_cnt - 1] >= i) head_cnt--; /* mark end of array */ head[head_cnt] = i; -- 2.43.0

2 2

[PATCH] tcp: Don't discard window information on keep-alive segments
by Stefano Brivio 12 Feb '25

12 Feb '25

It looks like a detail, but it's critical if we're dealing with somebody, such as near-future self, using TCP_REPAIR to migrate TCP connections in the guest or container. The last packet sent from the 'source' process/guest/container typically reports a small window, or zero, because the guest/container hadn't been draining it for a while. The next packet, appearing as the target sets TCP_REPAIR_OFF on the migrated socket, is a keep-alive (also called "window probe" in CRIU or TCP_REPAIR-related code), and it comes with an updated window value, reflecting the pre-migration "regular" value. If we ignore it, it might take a while/forever before we realise we can actually restart sending. Fixes: 238c69f9af45 ("tcp: Acknowledge keep-alive segments, ignore them for the rest") Signed-off-by: Stefano Brivio <sbrivio(a)redhat.com> --- tcp.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/tcp.c b/tcp.c index af6bd95..2addf4a 100644 --- a/tcp.c +++ b/tcp.c @@ -1664,8 +1664,10 @@ static int tcp_data_from_tap(const struct ctx *c, struct tcp_tap_conn *conn, tcp_send_flag(c, conn, ACK); tcp_timer_ctl(c, conn); - if (p->count == 1) + if (p->count == 1) { + tcp_tap_window_update(conn, ntohs(th->window)); return 1; + } continue; } -- 2.43.0

2 2

[PATCH v17 0/8] Not-really-draft state migration
by Stefano Brivio 12 Feb '25

12 Feb '25

Still missing at this point: - filtering out non-established sockets - trivial: SACK_PERM and timestamps (I forgot, we don't support them guest-side but they're relevant for sockets) - reasonable error handling, including not failing a migration on the target when it's anyway too late, or not crashing if we can't bind() or connect() a socket - tests with half-closed sockets The new SELinux rule is still untested. David Gibson (2): migrate: Migrate guest observed addresses rampstream: Add utility to test for corruption of data streams Stefano Brivio (6): migrate: Skeleton of live migration logic Add interfaces and configuration bits for passt-repair vhost_user: Make source quit after reporting migration state tcp: Get bound address for connected inbound sockets too migrate: Migrate TCP flows test: Add migration tests Makefile | 14 +- conf.c | 43 +- contrib/selinux/passt.te | 2 +- epoll_type.h | 6 +- flow.c | 259 ++++++++++- flow.h | 8 + flow_table.h | 6 +- migrate.c | 309 +++++++++++++ migrate.h | 51 +++ passt.1 | 11 + passt.c | 21 +- passt.h | 15 + repair.c | 218 +++++++++ repair.h | 16 + tap.c | 65 +-- tcp.c | 886 +++++++++++++++++++++++++++++++++++- tcp_conn.h | 99 ++++ test/.gitignore | 1 + test/Makefile | 5 +- test/lib/layout | 55 ++- test/lib/setup | 138 +++++- test/lib/test | 48 ++ test/migrate/basic | 59 +++ test/migrate/bidirectional | 64 +++ test/migrate/iperf3_bidir6 | 58 +++ test/migrate/iperf3_in4 | 50 ++ test/migrate/iperf3_in6 | 58 +++ test/migrate/iperf3_out4 | 50 ++ test/migrate/iperf3_out6 | 58 +++ test/migrate/rampstream_in | 61 +++ test/migrate/rampstream_out | 55 +++ test/passt.mbuto | 5 +- test/rampstream-check.sh | 3 + test/rampstream.c | 142 ++++++ test/run | 29 ++ util.c | 62 +++ util.h | 30 ++ vhost_user.c | 67 +-- virtio.h | 4 - vu_common.c | 49 +- vu_common.h | 2 +- 41 files changed, 2977 insertions(+), 205 deletions(-) create mode 100644 migrate.c create mode 100644 migrate.h create mode 100644 repair.c create mode 100644 repair.h create mode 100644 test/migrate/basic create mode 100644 test/migrate/bidirectional create mode 100644 test/migrate/iperf3_bidir6 create mode 100644 test/migrate/iperf3_in4 create mode 100644 test/migrate/iperf3_in6 create mode 100644 test/migrate/iperf3_out4 create mode 100644 test/migrate/iperf3_out6 create mode 100644 test/migrate/rampstream_in create mode 100644 test/migrate/rampstream_out create mode 100755 test/rampstream-check.sh create mode 100644 test/rampstream.c -- 2.43.0

1 8

[PATCH] tcp: Implement conservative zero-window probe on ACK timeout
by Stefano Brivio 12 Feb '25

12 Feb '25

This probably doesn't cover all the cases where we should send a zero-window probe, but it's rather unobtrusive and obvious, so start from here, also because I just observed this case (without the fix from the previous patch, to take into account window information from keep-alive segments). If we hit the ACK timeout, and try re-sending data from the socket, if the window is zero, we'll just fail again, go back to the timer, and so on, until we hit the maximum number of re-transmissions and reset the connection. Don't do that: forcibly try to send something by implementing the equivalent of a zero-window probe in this case. Signed-off-by: Stefano Brivio <sbrivio(a)redhat.com> --- tcp.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/tcp.c b/tcp.c index 2addf4a..b87478f 100644 --- a/tcp.c +++ b/tcp.c @@ -2175,6 +2175,8 @@ void tcp_timer_handler(const struct ctx *c, union epoll_ref ref) flow_dbg(conn, "ACK timeout, retry"); conn->retrans++; conn->seq_to_tap = conn->seq_ack_from_tap; + if (!conn->wnd_from_tap) + conn->wnd_from_tap = 1; /* Zero-window probe */ if (tcp_set_peek_offset(conn->sock, 0)) { tcp_rst(c, conn); } else { -- 2.43.0

2 1

[PATCH] passt-repair: Fix off-by-one in check for number of file descriptors
by Stefano Brivio 12 Feb '25

12 Feb '25

Actually, 254 is too many, but 253 isn't. Signed-off-by: Stefano Brivio <sbrivio(a)redhat.com> --- passt-repair.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/passt-repair.c b/passt-repair.c index 0f03506..ba0c48b 100644 --- a/passt-repair.c +++ b/passt-repair.c @@ -132,7 +132,7 @@ loop: /* No inverse formula for CMSG_LEN(x), and building one with CMSG_LEN(0) * works but there's no guarantee it does. Search the whole domain. */ - for (i = 1; i < SCM_MAX_FD; i++) { + for (i = 1; i <= SCM_MAX_FD; i++) { if (CMSG_LEN(sizeof(int) * i) == cmsg->cmsg_len) { n = i; break; -- 2.43.0

1 0