[PATCH v2 00/10] multithreading: Prepare data structures for concurrent queue pair workers
This series makes the passt data structures safe for concurrent access by multiple worker threads, each handling a different queue pair. It builds on the multiqueue series which threads the queue pair parameter through the network stack. The changes fall into four categories: 1. Per-queue-pair data isolation: - Convert packet pools (pool_tap4, pool_tap6) to per-queue-pair arrays - Convert L4 sequence batching arrays (tap4_l4, tap6_l4) similarly - Move flow and TCP timer state out of global/context structures into per-caller parameters 2. Stack-local buffers: - Move static iovec and virtqueue buffers onto the stack in tcp.c, tcp_vu.c, and udp_vu.c so each thread operates on its own copy 3. Locking for remaining shared state: - pthread_mutex on pre-opened socket pools (init_sock_pool4/6) - pthread_rwlock on the flow table, hash table, and free list, with read locks for lookups and write locks for mutations - _Thread_local for flow_new_entry so each thread independently tracks its in-progress allocation - Per-qpair filtering in flow_defer_handler() so each worker only processes its own flows 4. Lock-free flow migration between queue pairs: - When guest steers traffic to a different TX queue, flows migrate lazily using a to_migrate[] array - flow_migrate_mark() records target qpair from tap path (write-once, no lock needed) - flow_migrate_epollfd() completes migration on next socket/timer event by moving fd and updating qpair on the old thread - TCP timers migrate the same way using new tcp_timer_epoll_add() helper Global operations that don't need per-queue-pair handling (tcp_payload_flush, socket pool refills) are guarded to run only on queue pair 0. v2: - Rebase on latest master and multiqueue v6 - Split v1's "flow: Add mutex and per-qpair filtering..." into two patches: - "flow: Add locking, per-qpair filtering, and intermediate state handling" (locking only) - "flow: Add lazy, lock-free flow migration between queue pairs" (migration logic separated) - Changed from simple mutex to pthread_rwlock for flow table (read locks for lookups, write locks for mutations and free-list rebuild) - Updated flow_foreach() and flow_defer_handler() to handle intermediate flow states (NEW, INI, TGT, TYPED) that can be observed between flow_alloc() and FLOW_ACTIVATE() when lock is released - New patch: "tcp: Extract tcp_timer_epoll_add() helper" - factors out epoll registration for timer migration support - flow_migrate_epollfd() now uses tcp_timer_epoll_add() for timer migration v1: - Initial submission Based-on: 20260616125130.1324274-1-lvivier@redhat.com Laurent Vivier (10): tap: Convert packet pools to per-queue-pair arrays for multiqueue tap: Make L4 sequence pools per-qpair for thread safety tcp: Make static buffers stack-local for thread safety udp_vu: Make virtqueue buffers stack-local for thread safety flow: Make flow timer per-caller for thread safety tcp: Make TCP timer state per-caller and guard global tasks tcp: Protect init socket pools with mutex for thread safety tcp: Extract tcp_timer_epoll_add() helper flow: Add locking, per-qpair filtering, and intermediate state handling flow: Add lazy, lock-free flow migration between queue pairs flow.c | 222 ++++++++++++++++++++++++++++++++++++++++++++++----- flow.h | 5 +- flow_table.h | 2 +- icmp.c | 10 ++- passt.c | 37 +++++---- tap.c | 109 ++++++++++++++----------- tap.h | 2 +- tcp.c | 108 ++++++++++++++++++------- tcp.h | 9 +-- tcp_vu.c | 33 +++++--- udp.c | 3 + udp_flow.c | 4 + udp_vu.c | 4 +- vu_common.c | 2 +- 14 files changed, 409 insertions(+), 141 deletions(-) -- 2.54.0
Convert the global pool_tap4 and pool_tap6 packet pools from single
pools to arrays of pools, one for each queue pair. This change is
necessary to support multiqueue operation in vhost-user mode, where
multiple queue pairs may be processing packets concurrently.
The pool storage structures (pool_tap4_storage and pool_tap6_storage)
are now arrays of VHOST_USER_MAX_VQS/2 elements, with corresponding
pointer arrays (pool_tap4 and pool_tap6) for accessing them.
Add a qpair parameter to tap_flush_pools() so it flushes the correct
pool. tap4_handler() and tap6_handler() now use the qpair they
already receive to index into the pool arrays. Add bounds checking
assertions in tap_handler() and tap_add_packet().
In passt and pasta modes, all operations use QPAIR_DEFAULT. In
vhost-user mode, the queue pair is derived from the virtqueue index
via QPAIR_FROM_QUEUE().
All pools within the array share the same buffer pointer:
- In vhost-user mode: Points to the vhost-user memory structure, which
is safe as packet data remains in guest memory and pools only track
iovecs
- In passt/pasta mode: Points to pkt_buf, which is safe as only queue
pair 0 is used
Signed-off-by: Laurent Vivier
The L4 sequence arrays tap4_l4[] and tap6_l4[] are used to batch
packets with the same L4 tuple within a single tap_handler() call.
They are global, but tap_handler() can be called concurrently from
different worker threads with different qpairs in vhost-user mode.
Make these arrays per-qpair by adding a VHOST_USER_MAX_VQS/2 first
dimension, indexed by the qpair parameter already available in
tap4_handler() and tap6_handler().
Update tap_sock_update_pool() to initialize all qpair*seq entries.
Signed-off-by: Laurent Vivier
Static buffers shared across all call sites are not safe when multiple
worker threads handle TCP connections concurrently.
In tcp.c, move tcp_iov[] from file scope into tcp_data_from_tap() where
it is exclusively used. At UIO_MAXIOV (1024) entries of struct iovec
(16 bytes each), this adds 16 KiB to the stack frame.
In tcp_vu.c, move iov_vu[], elem[], and frame[] from file scope into
tcp_vu_data_from_sock() and pass them to tcp_vu_sock_recv() as
parameters. Also make iov_msg[] in tcp_vu_sock_recv() a local variable
instead of static, as it is only used within a single call. Combined,
these add roughly 80 KiB across the nested stack frames, which is
acceptable for per-thread stacks.
Signed-off-by: Laurent Vivier
The function-local static buffers elem[] and iov_vu[] in
udp_vu_sock_to_tap() are shared across all threads. When multiple
worker threads process UDP vhost-user data concurrently, they would
stomp on each other's buffers.
Remove the static qualifier so each call gets its own stack-allocated
arrays, eliminating cross-thread sharing.
Signed-off-by: Laurent Vivier
Move the static flow_timer_run variable out of flow.c and pass it as a
parameter to flow_defer_handler(). This allows each caller to maintain
its own timer state: each vhost-user queue pair worker uses the per-qpair
context.
Signed-off-by: Laurent Vivier
tcp_defer_handler() uses c->tcp.timer_run, c->tcp.keepalive_run, and
c->tcp.inactivity_run as global timer gates shared across all callers.
In multiqueue mode, multiple qpair workers will call tcp_defer_handler()
concurrently, causing races on these fields. It also unconditionally
runs tcp_payload_flush(), tcp_sock_refill_init(), and tcp_splice_refill()
which operate on global state.
Add timer_run, keepalive_run, and inactivity_run as parameters so each
caller provides its own per-qpair timer state. Remove the now-unused
fields from struct tcp_ctx and drop timer_init() which only initialised
c->tcp.timer_run.
Guard tcp_payload_flush() and socket pool refills with qpair == 0 since
they operate on global buffers shared across all queue pairs.
Signed-off-by: Laurent Vivier
The pre-opened socket pools init_sock_pool4/6 are consumed by
tcp_conn_pool_sock() when creating new connections from any worker
thread, and refilled by tcp_sock_refill_pool() from tcp_timer() in
post_handler(). These can run concurrently on different threads.
Add a mutex protecting both operations in tcp_conn_sock() and
tcp_sock_refill_init(), where init namespace pools are accessed.
Signed-off-by: Laurent Vivier
Factor out the epoll registration of TCP timers from tcp_timer_ctl()
into a separate tcp_timer_epoll_add() helper.
Signed-off-by: Laurent Vivier
Protect flow table shared state (free list, hash table) with a
pthread_rwlock_t. Make flow_new_entry _Thread_local.
Filter flow_defer_handler() by qpair so each thread only processes
its own flows.
Hold the write lock during flow_defer_handler()'s second pass
(free-list rebuild), since the free list is shared with flow_alloc().
Since the lock is released between flow_alloc() and FLOW_ACTIVATE(),
other threads can observe intermediate flow states (NEW, INI, TGT,
TYPED) during traversal. Adapt flow_foreach() to skip them silently
instead of logging an error, and change flow_defer_handler()'s
free-list rebuild to break cluster merging across them instead of
asserting.
Signed-off-by: Laurent Vivier
When the guest steers traffic for an existing flow to a different TX
queue, the flow must migrate to the new queue pair so that socket
events are handled by the correct worker thread.
Use a to_migrate[] array for lazy, lock-free migration:
flow_migrate_mark() records the target qpair from the tap path, and
flow_migrate_epollfd() completes the migration on the next socket
event by moving the fd and updating qpair on the old thread.
TCP timers are migrated the same way in tcp_timer_handler() using
tcp_timer_epoll_add().
Signed-off-by: Laurent Vivier
participants (1)
-
Laurent Vivier