[PATCH 0/8] multithreading: Prepare data structures for concurrent queue pair workers
This series makes the passt data structures safe for concurrent access by multiple worker threads, each handling a different queue pair. It builds on the multiqueue series which threads the queue pair parameter through the network stack. The changes fall into three categories: 1. Per-queue-pair data isolation: - Convert packet pools (pool_tap4, pool_tap6) to per-queue-pair arrays - Convert L4 sequence batching arrays (tap4_l4, tap6_l4) similarly - Move flow and TCP timer state out of global/context structures into per-caller parameters 2. Stack-local buffers: - Move static iovec and virtqueue buffers onto the stack in tcp.c, tcp_vu.c, and udp_vu.c so each thread operates on its own copy 3. Locking for remaining shared state: - pthread_mutex on pre-opened socket pools (init_sock_pool4/6) - pthread_rwlock on the flow table, hash table, and free list, with read locks for lookups and write locks for mutations - _Thread_local for flow_new_entry so each thread independently tracks its in-progress allocation - Per-qpair filtering in flow_defer_handler() so each worker only processes its own flows Global operations that don't need per-queue-pair handling (tcp_payload_flush, socket pool refills) are guarded to run only on queue pair 0. Based-on: 20260616125130.1324274-1-lvivier@redhat.com Laurent Vivier (8): tap: Convert packet pools to per-queue-pair arrays for multiqueue tap: Make L4 sequence pools per-qpair for thread safety tcp: Make static buffers stack-local for thread safety udp_vu: Make virtqueue buffers stack-local for thread safety flow: Make flow timer per-caller for thread safety tcp: Make TCP timer state per-caller and guard global tasks tcp: Protect init socket pools with mutex for thread safety flow: Add mutex and per-qpair filtering to flow table operations flow.c | 68 +++++++++++++++++++++++++------- flow.h | 2 +- flow_table.h | 2 +- passt.c | 37 +++++++++-------- tap.c | 109 +++++++++++++++++++++++++++++---------------------- tap.h | 2 +- tcp.c | 64 +++++++++++++++++++----------- tcp.h | 9 +---- tcp_vu.c | 33 ++++++++++------ udp_vu.c | 4 +- vu_common.c | 2 +- 11 files changed, 203 insertions(+), 129 deletions(-) -- 2.54.0
Convert the global pool_tap4 and pool_tap6 packet pools from single
pools to arrays of pools, one for each queue pair. This change is
necessary to support multiqueue operation in vhost-user mode, where
multiple queue pairs may be processing packets concurrently.
The pool storage structures (pool_tap4_storage and pool_tap6_storage)
are now arrays of VHOST_USER_MAX_VQS/2 elements, with corresponding
pointer arrays (pool_tap4 and pool_tap6) for accessing them.
Add a qpair parameter to tap_flush_pools() so it flushes the correct
pool. tap4_handler() and tap6_handler() now use the qpair they
already receive to index into the pool arrays. Add bounds checking
assertions in tap_handler() and tap_add_packet().
In passt and pasta modes, all operations use QPAIR_DEFAULT. In
vhost-user mode, the queue pair is derived from the virtqueue index
via QPAIR_FROM_QUEUE().
All pools within the array share the same buffer pointer:
- In vhost-user mode: Points to the vhost-user memory structure, which
is safe as packet data remains in guest memory and pools only track
iovecs
- In passt/pasta mode: Points to pkt_buf, which is safe as only queue
pair 0 is used
Signed-off-by: Laurent Vivier
The L4 sequence arrays tap4_l4[] and tap6_l4[] are used to batch
packets with the same L4 tuple within a single tap_handler() call.
They are global, but tap_handler() can be called concurrently from
different worker threads with different qpairs in vhost-user mode.
Make these arrays per-qpair by adding a VHOST_USER_MAX_VQS/2 first
dimension, indexed by the qpair parameter already available in
tap4_handler() and tap6_handler().
Update tap_sock_update_pool() to initialize all qpair*seq entries.
Signed-off-by: Laurent Vivier
The function-local static buffers elem[] and iov_vu[] in
udp_vu_sock_to_tap() are shared across all threads. When multiple
worker threads process UDP vhost-user data concurrently, they would
stomp on each other's buffers.
Remove the static qualifier so each call gets its own stack-allocated
arrays, eliminating cross-thread sharing.
Signed-off-by: Laurent Vivier
Move the static flow_timer_run variable out of flow.c and pass it as a
parameter to flow_defer_handler(). This allows each caller to maintain
its own timer state: each vhost-user queue pair worker uses the per-qpair
context.
Signed-off-by: Laurent Vivier
Static buffers shared across all call sites are not safe when multiple
worker threads handle TCP connections concurrently.
In tcp.c, move tcp_iov[] from file scope into tcp_data_from_tap() where
it is exclusively used. At UIO_MAXIOV (1024) entries of struct iovec
(16 bytes each), this adds 16 KiB to the stack frame.
In tcp_vu.c, move iov_vu[], elem[], and frame[] from file scope into
tcp_vu_data_from_sock() and pass them to tcp_vu_sock_recv() as
parameters. Also make iov_msg[] in tcp_vu_sock_recv() a local variable
instead of static, as it is only used within a single call. Combined,
these add roughly 80 KiB across the nested stack frames, which is
acceptable for per-thread stacks.
Signed-off-by: Laurent Vivier
The pre-opened socket pools init_sock_pool4/6 are consumed by
tcp_conn_pool_sock() when creating new connections from any worker
thread, and refilled by tcp_sock_refill_pool() from tcp_timer() in
post_handler(). These can run concurrently on different threads.
Add a mutex protecting both operations in tcp_conn_sock() and
tcp_sock_refill_init(), where init namespace pools are accessed.
Signed-off-by: Laurent Vivier
tcp_defer_handler() uses c->tcp.timer_run, c->tcp.keepalive_run, and
c->tcp.inactivity_run as global timer gates shared across all callers.
In multiqueue mode, multiple qpair workers will call tcp_defer_handler()
concurrently, causing races on these fields. It also unconditionally
runs tcp_payload_flush(), tcp_sock_refill_init(), and tcp_splice_refill()
which operate on global state.
Add timer_run, keepalive_run, and inactivity_run as parameters so each
caller provides its own per-qpair timer state. Remove the now-unused
fields from struct tcp_ctx and drop timer_init() which only initialised
c->tcp.timer_run.
Guard tcp_payload_flush() and socket pool refills with qpair == 0 since
they operate on global buffers shared across all queue pairs.
Signed-off-by: Laurent Vivier
The flow table free list, hash table, and flow_new_entry are global
shared state accessed from multiple threads.
Protect flow_alloc(), flow_alloc_cancel(), flow_hash_insert(),
flow_hash_remove(), and the free list rebuild in flow_defer_handler()
with a pthread_rwlock_t: writers for mutations, readers for lookups.
Make flow_new_entry _Thread_local so each thread independently tracks
its own in-progress allocation.
Since the lock is released between flow_alloc() and flow_activate(),
other threads can observe intermediate flow states (NEW, INI, TGT,
TYPED) during traversal. Adapt flow_foreach() and flow_defer_handler()
accordingly: skip these entries silently rather than treating them as
errors, and break free-list cluster merging across them.
Filter flow_defer_handler()'s first loop by qpair, so each thread
only processes its own flows.
Signed-off-by: Laurent Vivier
participants (1)
-
Laurent Vivier