Skip to content

UCP/FLUSH: Flush the other lanes after the pending lane - #12266

Open
dmitrygx wants to merge 6 commits into
openucx:masterfrom
dmitrygx:topic/ucx/ep-flush-pending-lane
Open

dmitrygx wants to merge 6 commits into
openucx:masterfrom
dmitrygx:topic/ucx/ep-flush-pending-lane

Conversation

@dmitrygx

@dmitrygx dmitrygx commented Nov 5, 2026 •

Copy link
Copy Markdown
Member

What?

Fix a regression from #11923 that lets an endpoint flush and a worker flush complete before puts which were posted before them. When ucp_ep_flush_progress() adds the flush request to the pending queue of a busy lane, it now stops there, and the lanes after it are flushed only from the pending callback of that lane, after the requests queued there before the flush are sent.

This is the cause of the flaky tcp/test_ucp_rma.put_nonblocking failures in CI: Assertion `!(region->flags & UCS_RCACHE_REGION_FLAG_PGTABLE)' failed in ucs_mem_region_destroy_internal() with a user memh (put_nonblocking/2), and data validation failed without it (see the CI failures below).

Why?

How the problem happens:

  1. A put on an endpoint with two RMA lanes is split between the lanes. With tcp over two IPoIB interfaces the put protocol is put/offload/zcopy, "50% on tcp/ib0 and 50% on tcp/ib3": one fragment on each lane.
  2. If lane 0 has no TX resources, the put waits in the pending queue of lane 0. When that queue is dispatched, the put sends its remaining fragments round robin, so also on lane 1.
  3. The flush gets UCS_ERR_NO_RESOURCE from lane 0 and adds itself to the pending queue of lane 0, behind the put. This is the same for ucp_ep_flush_nbx() and for ucp_worker_flush_nbx(), which flushes each endpoint and completes from ucp_worker_flush_ep_flushed_cb().
  4. Before UCP/FLUSH: Add topology-safe selective endpoint flush #11923, the loop in ucp_ep_flush_progress() picked the lowest unstarted lane again, got UCS_ERR_NO_RESOURCE from lane 0 and stopped, so lane 1 was flushed only from the pending callback, after the put. UCP/FLUSH: Add topology-safe selective endpoint flush #11923 (UCP/FLUSH: Add topology-safe selective endpoint flush) changed the loop to iterate a precomputed next_lanes mask without the pending lane, so after uct_ep_pending_add() the loop goes on and flushes lane 1 right away.
  5. The put then sends its last fragment on lane 1 after lane 1 was flushed, and the flush completes while that fragment is in flight.
  6. test_ucp_memheap::test_xfer() unmaps the user memh right after the worker flush, and the late completion of the put (ucp_proto_request_zcopy_completion() -> ucp_datatype_iter_cleanup() -> ucp_memh_put()) uses the freed memh. Its memory is already reused for the memh of an rcache region (parent == NULL), so ucp_memh_put() drops a refcount that the put never took, and the region refcount reaches 0 while the region is still in the page table. The assertion fires either in the late completion itself (build 141709) or later, in the ucp_mem_unmap() of the region owner (the other builds). Without a user memh the same race shows up as data validation failed.

The resume callback of the flush has the same problem with three or more lanes: when the flush of the pending lane fails, the pending callback adds the request to the pending queue of the next lane with no resources, and it still schedules the resume callback, which then flushed the lanes after that lane before the requests queued there were sent.

CI history (failed builds of the "UCX PR" pipeline): in the month before #11923 was merged (2026-09-25 .. 10-25), none of the 87 failed builds has a tcp put_nonblocking failure. In the 12 days after it, 17 of 81 failed builds have one: 7 crash with the assertion above, the first one 3 hours after the merge, and 10 fail the test without a crash.

How?

  • UCP/FLUSH: ucp_ep_flush_progress() breaks out of the lane loop after it adds the flush request to the pending queue of a lane, as before UCP/FLUSH: Add topology-safe selective endpoint flush #11923. ucp_ep_flush_progress_pending() flushes its lane and then calls ucp_ep_flush_progress(), which flushes the remaining lanes after the requests queued before the flush were sent.
  • UCP/FLUSH: ucp_ep_flush_resume_slow_path_callback() does nothing while req->send.lane is set, that is, after the request was added to the pending queue of a lane. The pending callback of that lane, or the purge of its queue, continues the flush.
  • GTEST/UCP/FLUSH: the pending lane tests share flush_with_pending_lane() and test_flush_complete(), and the flush mock counts the calls of each lane.
    • pending_lane_defers_other_lanes: lane 0 returns UCS_ERR_NO_RESOURCE, and lane 1 is not flushed until the pending callback of lane 0 runs.
    • pending_lane_flushes_lanes_once: lane 0 is flushed and lane 1 returns UCS_ERR_NO_RESOURCE; the pending callback of lane 1 does not flush lane 0 again.
    • pending_lane_resume_skips_next_lanes (MAX_EAGER_LANES=3): the flush of lane 0 fails in its pending callback and the request goes to the pending queue of lane 1; the resume callback does not flush lane 2 before the pending callback of lane 1.
    • Update pending_error_lane_replacement, which expected lane 1 to be flushed in the first call; it now takes that completion after the pending callback.

Not in this PR: the lanes before the pending lane are already flushed when the flush request goes to the pending queue, and a request queued before the flush with more fragments than lanes (e.g. an IOV put) can wrap around to them. This gap is older than #11923 and is not the case of the CI failures (one fragment per lane). An earlier revision of this PR flushed the started lanes again from the pending callback, but then the flush also waited for the requests posted after it, and with traffic on two lanes it could move between their pending queues without end. Covering the gap needs UCP to track the sends posted before the flush.

Testing

On a node with two tcp lanes (IPoIB), devel build like in CI, UCX_PROTO_REQUEST_RESET=y:

Test Before the fix With the fix
tcp/test_ucp_rma.put_nonblocking/2 (user_memh) crashed in 5-6 of 60 runs 60 of 60 passed with the first fix (the break only), 40 of 40 with the final change
tcp/test_ucp_rma.put_nonblocking/0 data validation failed in 2 of 60 runs 60 of 60 passed with the first fix, 40 of 40 with the final change
test_ucp_flush.pending_lane_defers_other_lanes fails (log below) passes
test_ucp_flush.pending_lane_flushes_lanes_once fails with the re-flush of the previous revision (log below) passes
test_ucp_flush.pending_lane_resume_skips_next_lanes fails (log below) passes
*test_ucp_flush* (with test_ucp_flush_failover) 31 passed
*test_ucp_fault_tolerance*, *test_ucp_ep_reconfig*, *test_proto_reset.*, tcp/rcx test_ucp_rma.*, dcx/shm_ib test_ucp_rma.put*, rcx/tcp test_ucp_peer_failure.*, *test_ucp_ep_close* 938 passed

Test failures

CI

Build PR Date Job Aborts in
141668 #12111 2026-10-29 Tests new on worker 1 ucp_mem_unmap()
141709 #12180 2026-10-30 Tests new on worker 3 late put completion
141739 #12107 2026-10-30 Tests new on worker 0 ucp_mem_unmap()
141892 #12135 2026-11-01 Tests new on worker 3 ucp_mem_unmap()
142125 #12209 2026-11-02 Tests new on worker 3 ucp_mem_unmap()
142723 #12205 2026-11-04 Tests new on worker 0 ucp_mem_unmap()

Other failed builds with a tcp put_nonblocking failure since #11923: the same crash in 140518 (#11805, 2026-10-25), 142636 (#12234, 2026-11-05); a failure without a crash in 140693 (/2, #12162), 140878 (/0, #12180), 141265 (/2, #12162), 141689 (/2, #12201), 141923 (/0, #11529), 142018 (/1, #12206), 142100 (/1, #12195), 142189 (/0, #12215), 142321 (/0, #12165), 142554 (/0, #12200).

Build 141668 (#12111, "Tests new on worker 1", 2026-10-29)
[ RUN      ] tcp/test_ucp_rma.put_nonblocking/2 <tcp,cuda_copy,rocm_copy/user_memh>
[swx-rdmz-ucx-new-01:1545643:0:1545643]      rcache.c:396  Assertion `!(region->flags & UCS_RCACHE_REGION_FLAG_PGTABLE)' failed
==== backtrace (tid:1545643) ====
 0 0x000000000004d698 ucs_mem_region_destroy_internal()  src/ucs/memory/rcache.c:396
 1 0x000000000004d698 ucs_rcache_region_lru_get()  src/ucs/memory/rcache.inl:40
 2 0x000000000004d698 ucs_mem_region_destroy_internal()  src/ucs/memory/rcache.c:420
 3 0x000000000006a39d ucs_rcache_region_put_unsafe()  src/ucs/memory/rcache.inl:128
 4 0x000000000006a39d ucp_memh_put_rcache()  src/ucp/core/ucp_mm.c:441
 5 0x000000000006f95f ucp_memh_cleanup()  src/ucp/core/ucp_mm.c:470
 6 0x000000000006f95f ucp_mem_unmap()  src/ucp/core/ucp_mm.c:1291
 7 0x00000000009edbbc test_ucp_memheap::test_xfer()  test/gtest/ucp/test_ucp_memheap.cc:120
 8 0x0000000000b3af03 test_ucp_rma::test_message_sizes()  test/gtest/ucp/test_ucp_rma.cc:194
 9 0x0000000000b64a53 test_ucp_rma::test_mem_types()  test/gtest/ucp/test_ucp_rma.cc:157
10 0x0000000000b64a53 test_ucp_rma::test_mem_types()  test/gtest/ucp/test_ucp_rma.cc:150
11 0x0000000000593fe5 ucs::test_base::run()  test/gtest/common/test.cc:468
Build 141709 (#12180, "Tests new on worker 3", 2026-10-30)
[ RUN      ] tcp/test_ucp_rma.put_nonblocking/2 <tcp,cuda_copy,rocm_copy/user_memh>
test/gtest/ucp/test_ucp_memheap.cc:115: Failure
Failed
data validation failed
[     INFO ] host->host 100x128 100x259 100x1571 100x6672 100x2201 100x89092 
[swx-rdmz-ucx-new-01:2686166:0:2686166]      rcache.c:396  Assertion `!(region->flags & UCS_RCACHE_REGION_FLAG_PGTABLE)' failed
==== backtrace (tid:2686166) ====
 0 0x000000000004d698 ucs_mem_region_destroy_internal()  src/ucs/memory/rcache.c:396
 1 0x000000000004d698 ucs_rcache_region_lru_get()  src/ucs/memory/rcache.inl:40
 2 0x000000000004d698 ucs_mem_region_destroy_internal()  src/ucs/memory/rcache.c:420
 3 0x000000000009a62b ucs_rcache_region_put_unsafe()  src/ucs/memory/rcache.inl:128
 4 0x000000000009a62b ucp_memh_put()  src/ucp/core/ucp_mm.inl:103
 5 0x000000000009a62b ucp_datatype_iter_mem_dereg_single()  src/ucp/dt/datatype_iter.inl:310
 6 0x000000000009a62b ucp_datatype_iter_cleanup()  src/ucp/dt/datatype_iter.inl:349
 7 0x000000000009a62b ucp_proto_request_zcopy_complete_cb()  src/ucp/proto/proto_common.inl:120
 8 0x000000000009a62b ucp_proto_request_zcopy_complete()  src/ucp/proto/proto_common.inl:134
 9 0x000000000009a62b ucp_proto_request_zcopy_completion()  src/ucp/proto/proto_common.c:774
10 0x000000000002c0c4 uct_invoke_completion()  src/uct/base/uct_iface.h:1018
11 0x000000000002c0c4 uct_tcp_ep_handle_put_ack()  src/uct/tcp/tcp_ep.c:1032
12 0x000000000002c0c4 uct_tcp_ep_progress_am_rx()  src/uct/tcp/tcp_ep.c:1540
13 0x000000000002c0c4 uct_tcp_ep_progress_data_rx()  src/uct/tcp/tcp_ep.c:1621
14 0x000000000003034a uct_tcp_iface_handle_events()  src/uct/tcp/tcp_iface.c:435
15 0x00000000000563a1 ucs_event_set_wait()  src/ucs/sys/event_set.c:215
16 0x000000000002f267 uct_tcp_iface_progress()  src/uct/tcp/tcp_iface.c:452
17 0x0000000000088132 ucs_callbackq_dispatch()  src/ucs/datastruct/callbackq.h:215
18 0x0000000000088132 uct_worker_progress()  src/uct/api/uct.h:2841
19 0x0000000000088132 ucp_worker_progress()  src/ucp/core/ucp_worker.c:3314
20 0x0000000000ea5122 ucp_test_base::entity::progress()  test/gtest/ucp/ucp_test.cc:1066
21 0x0000000000ea5122 ucp_test::progress()  test/gtest/ucp/ucp_test.cc:182
22 0x0000000000ea5122 ucp_test::check_events()  test/gtest/ucp/ucp_test.cc:260
23 0x0000000000ea539e ucp_test::request_process()  test/gtest/ucp/ucp_test.cc:303
24 0x00000000009ed189 test_ucp_memheap::test_xfer()  test/gtest/ucp/test_ucp_memheap.cc:106
25 0x00000000009ed189 test_ucp_memheap::test_xfer()  test/gtest/ucp/test_ucp_memheap.cc:106
Build 141739 (#12107, "Tests new on worker 0", 2026-10-30)
[ RUN      ] tcp/test_ucp_rma.put_nonblocking/2 <tcp,cuda_copy,rocm_copy/user_memh>
[swx-rdmz-ucx-new-01:3233122:0:3233122]      rcache.c:396  Assertion `!(region->flags & UCS_RCACHE_REGION_FLAG_PGTABLE)' failed
==== backtrace (tid:3233122) ====
 0 0x000000000004d688 ucs_mem_region_destroy_internal()  src/ucs/memory/rcache.c:396
 1 0x000000000004d688 ucs_rcache_region_lru_get()  src/ucs/memory/rcache.inl:40
 2 0x000000000004d688 ucs_mem_region_destroy_internal()  src/ucs/memory/rcache.c:420
 3 0x000000000006a3bd ucs_rcache_region_put_unsafe()  src/ucs/memory/rcache.inl:128
 4 0x000000000006a3bd ucp_memh_put_rcache()  src/ucp/core/ucp_mm.c:441
 5 0x000000000006f97f ucp_memh_cleanup()  src/ucp/core/ucp_mm.c:470
 6 0x000000000006f97f ucp_mem_unmap()  src/ucp/core/ucp_mm.c:1291
 7 0x00000000009ee88c test_ucp_memheap::test_xfer()  test/gtest/ucp/test_ucp_memheap.cc:120
 8 0x0000000000b40ea3 test_ucp_rma::test_message_sizes()  test/gtest/ucp/test_ucp_rma.cc:194
 9 0x0000000000b699f3 test_ucp_rma::test_mem_types()  test/gtest/ucp/test_ucp_rma.cc:157
10 0x0000000000b699f3 test_ucp_rma::test_mem_types()  test/gtest/ucp/test_ucp_rma.cc:150
11 0x0000000000594cb5 ucs::test_base::run()  test/gtest/common/test.cc:468
Build 141892 (#12135, "Tests new on worker 3", 2026-11-01)
[ RUN      ] tcp/test_ucp_rma.put_nonblocking/2 <tcp,cuda_copy,rocm_copy/user_memh>
[swx-rdmz-ucx-new-02:635811:0:635811]      rcache.c:396  Assertion `!(region->flags & UCS_RCACHE_REGION_FLAG_PGTABLE)' failed
==== backtrace (tid: 635811) ====
 0 0x000000000004d6a8 ucs_mem_region_destroy_internal()  src/ucs/memory/rcache.c:396
 1 0x000000000004d6a8 ucs_rcache_region_lru_get()  src/ucs/memory/rcache.inl:40
 2 0x000000000004d6a8 ucs_mem_region_destroy_internal()  src/ucs/memory/rcache.c:420
 3 0x000000000006a39d ucs_rcache_region_put_unsafe()  src/ucs/memory/rcache.inl:128
 4 0x000000000006a39d ucp_memh_put_rcache()  src/ucp/core/ucp_mm.c:441
 5 0x000000000006f95f ucp_memh_cleanup()  src/ucp/core/ucp_mm.c:470
 6 0x000000000006f95f ucp_mem_unmap()  src/ucp/core/ucp_mm.c:1291
 7 0x00000000009f130c test_ucp_memheap::test_xfer()  test/gtest/ucp/test_ucp_memheap.cc:120
 8 0x0000000000b438e3 test_ucp_rma::test_message_sizes()  test/gtest/ucp/test_ucp_rma.cc:194
 9 0x0000000000b6c433 test_ucp_rma::test_mem_types()  test/gtest/ucp/test_ucp_rma.cc:157
10 0x0000000000b6c433 test_ucp_rma::test_mem_types()  test/gtest/ucp/test_ucp_rma.cc:150
11 0x00000000005972d5 ucs::test_base::run()  test/gtest/common/test.cc:468
Build 142125 (#12209, "Tests new on worker 3", 2026-11-02)
[ RUN      ] tcp/test_ucp_rma.put_nonblocking/2 <tcp,cuda_copy,rocm_copy/user_memh>
[swx-rdmz-ucx-new-01:1733043:0:1733043]      rcache.c:396  Assertion `!(region->flags & UCS_RCACHE_REGION_FLAG_PGTABLE)' failed
==== backtrace (tid:1733043) ====
 0 0x000000000004d6b8 ucs_mem_region_destroy_internal()  src/ucs/memory/rcache.c:396
 1 0x000000000004d6b8 ucs_rcache_region_lru_get()  src/ucs/memory/rcache.inl:40
 2 0x000000000004d6b8 ucs_mem_region_destroy_internal()  src/ucs/memory/rcache.c:420
 3 0x000000000006b16d ucs_rcache_region_put_unsafe()  src/ucs/memory/rcache.inl:128
 4 0x000000000006b16d ucp_memh_put_rcache()  src/ucp/core/ucp_mm.c:441
 5 0x000000000007072f ucp_memh_cleanup()  src/ucp/core/ucp_mm.c:470
 6 0x000000000007072f ucp_mem_unmap()  src/ucp/core/ucp_mm.c:1291
 7 0x00000000009f130c test_ucp_memheap::test_xfer()  test/gtest/ucp/test_ucp_memheap.cc:120
 8 0x0000000000b438e3 test_ucp_rma::test_message_sizes()  test/gtest/ucp/test_ucp_rma.cc:194
 9 0x0000000000b6c433 test_ucp_rma::test_mem_types()  test/gtest/ucp/test_ucp_rma.cc:157
10 0x0000000000b6c433 test_ucp_rma::test_mem_types()  test/gtest/ucp/test_ucp_rma.cc:150
11 0x00000000005972d5 ucs::test_base::run()  test/gtest/common/test.cc:468
Build 142723 (#12205, "Tests new on worker 0", 2026-11-04)
[ RUN      ] tcp/test_ucp_rma.put_nonblocking/2 <tcp,cuda_copy,rocm_copy/user_memh>
[swx-rdmz-ucx-new-01:1591665:0:1591665]      rcache.c:396  Assertion `!(region->flags & UCS_RCACHE_REGION_FLAG_PGTABLE)' failed
==== backtrace (tid:1591665) ====
 0 0x000000000004d6a8 ucs_mem_region_destroy_internal()  src/ucs/memory/rcache.c:396
 1 0x000000000004d6a8 ucs_rcache_region_lru_get()  src/ucs/memory/rcache.inl:40
 2 0x000000000004d6a8 ucs_mem_region_destroy_internal()  src/ucs/memory/rcache.c:420
 3 0x000000000006c29d ucs_rcache_region_put_unsafe()  src/ucs/memory/rcache.inl:128
 4 0x000000000006c29d ucp_memh_put_rcache()  src/ucp/core/ucp_mm.c:441
 5 0x000000000007185f ucp_memh_cleanup()  src/ucp/core/ucp_mm.c:470
 6 0x000000000007185f ucp_mem_unmap()  src/ucp/core/ucp_mm.c:1291
 7 0x0000000000a24fdc test_ucp_memheap::test_xfer()  test/gtest/ucp/test_ucp_memheap.cc:120
 8 0x0000000000b7e7e3 test_ucp_rma::test_message_sizes()  test/gtest/ucp/test_ucp_rma.cc:194
 9 0x0000000000ba9583 test_ucp_rma::test_mem_types()  test/gtest/ucp/test_ucp_rma.cc:157
10 0x0000000000ba9583 test_ucp_rma::test_mem_types()  test/gtest/ucp/test_ucp_rma.cc:150
11 0x00000000005ac705 ucs::test_base::run()  test/gtest/common/test.cc:468

Reproduction without the fix

tcp/test_ucp_rma.put_nonblocking/0 (no user memh): the late put shows up as a data validation failure
[ RUN      ] tcp/test_ucp_rma.put_nonblocking/0 <tcp,cuda_copy,rocm_copy/flush_worker>
test/gtest/ucp/test_ucp_memheap.cc:115: Failure
Failed
data validation failed
[     INFO ] host->host 100x128 100x319 100x1249 100x7040 100x31047 100x47930 100x90601 100x222446 100x714442 6x16495898
tcp_ep.c:1262 UCX  ERROR tcp_ep 0x7f92dc000c10 (state=CONNECTED): recv(31) failed: Input/output error
[     INFO ] host->host map_nb 100x128
flush.c:110  UCX  ERROR req 0x3a1c380: error during flush: Connection reset by remote peer
flush.c:110  UCX  ERROR req 0x3a1c380: error during flush: Connection reset by remote peer
[  FAILED  ] tcp/test_ucp_rma.put_nonblocking/0, where GetParam() = tcp,cuda_copy,rocm_copy/flush_worker (704 ms)
Debug build (instrumentation, not part of this PR): the worker flush completes from the endpoint flush while the last fragment of a put is in flight on lane 1

The instrumentation tracks put/offload/zcopy requests from their init to their completion, and dumps the state when a worker flush completes while a put posted before it is not completed. In every early completion (13, 11 and 8 of 60 runs in three loops) the worker flush completed from ucp_worker_flush_ep_flushed_cb(), and the put had sent all its data (offset 56860/56860), with one fragment still in flight (comp.count 1) on the second lane (ib3 outstanding 1), after it was sent from the pending queue of lane 0 (lane 0).

[ RUN      ] tcp/test_ucp_rma.put_nonblocking/0 <tcp,cuda_copy,rocm_copy/flush_worker>
flush.c:726  UCX  WARN  DBG-EARLY-WFLUSH ep-flushed-cb: worker 0x4a87ca0 1 puts in flight, flush_ops_count 0
flush.c:739  UCX  WARN  DBG   req 0x4b14b80 ep 0x7f80c1f1d000 flags 0x86 lane 0 lane_idx 1 offset 56860/56860 comp.count 1 uct.func ucp_proto_put_offload_zcopy_progress
flush.c:746  UCX  WARN  DBG   ep 0x7f80c1f1d000 flags 0x51 cfg 0 num_lanes 2 am_lane 0
flush.c:755  UCX  WARN  DBG     lane 0 uct_ep 0x4abaf00 wireup_ep (nil) flags 0x0 next (nil) pending 0
flush.c:755  UCX  WARN  DBG     lane 1 uct_ep 0x4acfd80 wireup_ep (nil) flags 0x0 next (nil) pending 0
tcp_iface.c:463  UCX  WARN  DBG   tcp iface 0x4ad16d0 eno1 outstanding 0
tcp_iface.c:463  UCX  WARN  DBG   tcp iface 0x4ad25e0 ib0 outstanding 0
tcp_iface.c:463  UCX  WARN  DBG   tcp iface 0x4ad5320 ib3 outstanding 1
tcp_iface.c:463  UCX  WARN  DBG   tcp iface 0x4ad2f90 lo outstanding 0
`pending_lane_defers_other_lanes` on master: lane 1 is flushed in the first call
[ RUN      ] shm_ib/test_ucp_flush.pending_lane_defers_other_lanes/0 <shm,ib>
test/gtest/ucp/test_ucp_flush.cc:582: Failure
Expected equality of these values:
  0u
    Which is: 0
  test_flush_lane_calls[1]
    Which is: 1
test/gtest/ucp/test_ucp_flush.cc:583: Failure
Expected equality of these values:
  0
  req->send.flush.started_lanes
    Which is: 2
[  FAILED  ] shm_ib/test_ucp_flush.pending_lane_defers_other_lanes/0, where GetParam() = shm,ib (774 ms)
`pending_lane_flushes_lanes_once` with the re-flush of the previous revision (b2e79b2): the pending callback of lane 1 flushes lane 0 again
[ RUN      ] shm_ib/test_ucp_flush.pending_lane_flushes_lanes_once/0 <shm,ib>
test/gtest/ucp/test_ucp_flush.cc:615: Failure
Expected equality of these values:
  1u
    Which is: 1
  test_flush_lane_calls[0]
    Which is: 2
test/gtest/ucp/test_ucp_flush.cc:617: Failure
Expected equality of these values:
  (1ul << (0)) | (1ul << (1))
    Which is: 3
  req->send.flush.started_lanes
    Which is: 2
[  FAILED  ] shm_ib/test_ucp_flush.pending_lane_flushes_lanes_once/0, where GetParam() = shm,ib (612 ms)
`pending_lane_resume_skips_next_lanes` without the resume fix (b2e79b2): the resume callback flushes lane 2 while the request waits on lane 1
[ RUN      ] shm_ib/test_ucp_flush.pending_lane_resume_skips_next_lanes/0 <shm,ib>
test/gtest/ucp/test_ucp_flush.cc:648: Failure
Expected equality of these values:
  0u
    Which is: 0
  test_flush_lane_calls[2]
    Which is: 1
test/gtest/ucp/test_ucp_flush.cc:658: Failure
Expected equality of these values:
  1u
    Which is: 1
  test_flush_lane_calls[2]
    Which is: 2
[  FAILED  ] shm_ib/test_ucp_flush.pending_lane_resume_skips_next_lanes/0, where GetParam() = shm,ib (590 ms)

When an endpoint flush waits in the pending queue of a lane, the requests
queued there before it can still send on the other lanes, e.g. a put which
is split between the lanes. Check that the flush starts the other lanes
only from its pending callback, after these requests are sent.
Since the selective flush change (openucx#11923), ucp_ep_flush_progress() went on
flushing the remaining lanes after it added the flush request to the
pending queue of a busy lane. A put which was split between the lanes and
queued on that lane before the flush then sent its last fragment on a lane
which was already flushed, so the endpoint and the worker flush completed
before the put.

In tcp/test_ucp_rma.put_nonblocking/2 the test then unmapped the user memh,
and the late put completion released the freed memh, which corrupted the
refcount of an rcache region (assertion in
ucs_mem_region_destroy_internal()).

Stop after adding the flush request to the pending queue, as before: the
pending callback flushes the other lanes after the requests queued before
the flush are sent. Update the test which expected the other lanes to be
flushed right away.
@dmitrygx
dmitrygx requested a review from a team as a code owner November 5, 2026 18:51
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread src/ucp/rma/flush.c Resolved
Comment thread src/ucp/rma/flush.c Resolved
…again

A flush which waits in the pending queue of a lane may have flushed the
lanes before it already. The requests queued on that lane before the flush
can still send on these lanes, e.g. a put with more fragments than lanes,
so check that the pending callback flushes them again.
ucp_ep_flush_progress() flushes the lanes in increasing order, so the lanes
before the one where the flush request goes to the pending queue are already
flushed. A request queued on that lane before the flush can still send on
them, since a multi-lane protocol sends its fragments round robin, so the
flush could complete before these fragments.

When the pending callback runs, the requests queued before the flush are
sent. Flush the started lanes again at this point; their previous flush
still completes, so each of them adds one completion.
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread src/ucp/rma/flush.c Outdated
reflush_lanes = req->send.flush.started_lanes & ucp_ep_get_live_lanes(ep) &
~UCS_BIT(lane);
req->send.flush.started_lanes &= ~reflush_lanes;
ucp_ep_flush_request_update_uct_comp(req, ucs_popcount(reflush_lanes), 0);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pls move the re-flush after the uct_ep_flush() call below and do it only when it did not return UCS_ERR_NO_RESOURCE: on NO_RESOURCE the request stays in the pending queue, and every retry of the pending callback un-starts the other lanes again and posts one more flush with one more expected completion on each of them.

Impact: with the current order the lanes are re-flushed on every pending retry (confirmed: on NO_RESOURCE the function still calls ucp_ep_flush_progress() before returning, and next_lanes only excludes req->send.lane), so outstanding uct flushes on the other lanes accumulate while the pending lane has no resources. Also note the new tests clear test_flush_no_resource_ep before invoking ucp_ep_flush_progress_pending(), so this retry path is not covered.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The re-flush is removed in 7d2798b, so a retry of the pending callback no longer touches the started lanes or the completion count. I removed it instead of moving it, because it also made the flush wait for the requests posted after it, and under sustained traffic on two lanes the flush could move between their pending queues without end (details in #12266 (comment)).

Comment thread src/ucp/rma/flush.c Outdated
* more fragments than lanes. Flush these lanes again; their previous flush
* still completes, so each of them needs one more completion. */
reflush_lanes = req->send.flush.started_lanes & ucp_ep_get_live_lanes(ep) &
~UCS_BIT(lane);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should we also exclude lanes that already failed in ucp_ep_flush_error()? they stay in started_lanes and in ucp_ep_get_live_lanes() until the scheduled failover callback runs, so a lane that just returned a hard error is flushed again, which reports the same error twice and schedules the same failed lane twice.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not needed any more: the re-flush is removed in 7d2798b, so the pending callback does not flush a lane whose flush failed again.

Comment thread src/ucp/rma/flush.c Outdated
* still completes, so each of them needs one more completion. */
reflush_lanes = req->send.flush.started_lanes & ucp_ep_get_live_lanes(ep) &
~UCS_BIT(lane);
req->send.flush.started_lanes &= ~reflush_lanes;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minor: pls clear started_lanes inside ucp_ep_flush_request_update_uct_comp() - it is the only place that keeps the count and the mask in sync, and as written the trace prints the already-cleared mask so the log does not show which lanes were restarted.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed together with the re-flush in 7d2798b.

ucp_request_release(request);
}

UCS_TEST_P(test_ucp_flush, pending_lane_defers_other_lanes,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minor: the two tests are ~50 nearly identical lines (setup, mock, request creation, completion drain, release) and differ only in the pending lane and the expected counters; maybe extract a helper parameterized by the pending lane?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 22ff84a: the setup, the request creation and the completion are in flush_with_pending_lane() and test_flush_complete(), and the flush mock counts the calls of each lane, so each test only has its own steps and expectations.

Comment thread test/gtest/ucp/test_ucp_flush.cc Outdated
ucp_request_release(request);
}

UCS_TEST_P(test_ucp_flush, pending_lane_reflushes_started_lanes,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minor: this test duplicates the preceding one (~50 nearly identical lines of setup, mock, request creation, completion drain, release), differing only in the pending lane and the expected counters; consider sharing a parameterized helper.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 22ff84a, see above. Since the re-flush is removed, this test is replaced by pending_lane_flushes_lanes_once, which checks that the pending callback of lane 1 does not flush lane 0 again.

@svc-ucx

svc-ucx commented Nov 5, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR (Tests roce on worker 0) · commit b2e79b2e

TL;DR: The roce on worker 0 gtest run failed on a single, unrelated UCT-level test — dc_mlx5/uct_atomic_key_reg_rdma_mem_type.fadd64/0 (concurrent 64-bit atomic fetch-add over dc_mlx5/mlx5_0:1) — while all 9030 other tests passed; the PR only changes the UCP flush path, so this is an infrastructure/flaky UCT atomics failure, not a regression from #12266. Re-run the job and, if it reproduces, chase the dc_mlx5 atomic-reply path rather than the flush change.

Full analysis

Summary: make test in the gtest stage exited non-zero (make: *** [Makefile:4992: test] Error 1) because exactly one test failed: dc_mlx5/uct_atomic_key_reg_rdma_mem_type.fadd64/0, where GetParam() = dc_mlx5/mlx5_0:1.

Root cause: Evidence from the log: [==========] 9031 tests ... [ PASSED ] 9030 tests. [ FAILED ] 1 test ... dc_mlx5/uct_atomic_key_reg_rdma_mem_type.fadd64/0. This is a pure UCT transport-level atomic test: N sender threads issue concurrent uct_ep_atomic64_fetch(ADD) to one remote 8-byte buffer and the test asserts that every expected reply value appears exactly once and the final memory value matches (uct_amo_test::validate_replies() / EXPECT_EQ(value, *(T*)recvbuf.ptr())). It never goes through UCP endpoint flush logic, which is all that PR #12266 ("UCP/FLUSH: Flush the other lanes after the pending lane") modifies — so the failure is not explained by the PR's change. The usual failure mode for this test family is a lost/duplicated atomic fetch reply under concurrency on the DC/RoCE device (the UCS_TEST_ABORT("Reply ... appeared N times") path), i.e. a transport-level/environment flake. Note: the exact assertion text is not recoverable — the Azure log API only returns the tail (~500 lines) of the task log and the failure printout occurred much earlier in the ~49-minute run.

Implicated commit: unknown — not attributable to [REDACTED:Hex High Entropy String] (PR #12266) based on the log evidence. If it reproduces, the most plausible recent suspects touching the RC/DC atomic send path are 22cff74b "UCT/RC: Reserve TX credits for multi-credit sends (#11841)" (Michal Shalev) and 68cfcbc5 "UCT/IB/MLX5: uct comp marker (#11915)" (Zihao Zhao).

File: test/gtest/uct/test_amo.cc:106 (uct_amo_test::validate_replies abort) / test/gtest/uct/test_amo.cc:39 (final-value check in the fadd test body)

Suggested fix: Re-trigger the roce on worker 0 job to confirm flakiness; this single-test failure should not block #12266. To get a real diagnosis, grab the full gtest console output (or the per-job raw log, not just the tail) so the uct_atomic_key_reg_rdma_mem_type.fadd64 assertion message is visible, and if it reproduces, run that test standalone with -x UCX_LOG_LEVEL=debug --gtest_filter='dc_mlx5/uct_atomic_key_reg_rdma_mem_type.fadd64*' --gtest_repeat=50 on the RoCE node to determine whether an atomic fetch reply is lost (TX credit/completion handling in the dc_mlx5 atomic path) rather than raising timeouts or touching the flush code.

Related: PR under test: #12266; possibly relevant transport changes: #11841, #11915

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 5f14e072-436d-4c9d-a882-d4d029fb581b in the triage console for the audit trail.

Move the setup and the completion of the pending lane tests to
flush_with_pending_lane() and test_flush_complete(), and count the flushes
of each lane.

Replace pending_lane_reflushes_started_lanes with
pending_lane_flushes_lanes_once: when the pending callback of lane 1 runs
and lane 0 has no resources again, lane 0 must not be flushed again.

Add pending_lane_resume_skips_next_lanes: after the pending callback of
lane 0 adds the flush request to the pending queue of lane 1, the resume
callback must not flush lane 2.
The pending callback flushed the started lanes again, so the flush waited
for the requests posted after it on these lanes too. If such a lane had no
resources, the flush went to its pending queue, and the pending callback of
that lane flushed the other lanes again, so with traffic on two lanes the
flush could move between their pending queues without end. Remove it: the
case it covered, a request queued before the flush with more fragments than
lanes, is older than openucx#11923 and needs UCP to track the sends posted before
the flush.

When the flush of the pending lane fails, the pending callback can add the
request to the pending queue of the next lane with no resources, and it
still schedules the resume callback. The resume callback then flushed the
lanes after that lane before the requests queued there were sent. Do
nothing in the resume callback while req->send.lane is set: the pending
callback of that lane, or the purge of its queue, continues the flush.
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Review complete — no issues found.

@svc-ucx

svc-ucx commented Nov 6, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR (Tests roce on worker 0) · commit 7d2798b4

TL;DR: The roce on worker 0 gtest run failed on a single unrelated test, ib/test_md.alloc/0 (mlx5_0), out of 9032 — a UCT memory-domain allocation test that has nothing to do with this PR's UCP flush change; re-run the job (and capture the full gtest log, since the assertion text is truncated out of the retrievable tail).

Full analysis

Summary: make test in contrib/test_jenkins.sh exited with code 2 because 1 of 9032 gtests failed: [ FAILED ] ib/test_md.alloc/0, where GetParam() = mlx5_0 (log line at 01:23:05, make: *** [Makefile:4992: test] Error 1).

Root cause: Not a regression from PR #12266. The log shows continuous test activity to the end (no hang, no timeout), 9031 passing tests, and the only failure is in test_md.alloc — a pure UCT uct_mem_alloc()/uct_md_mkey_pack_v2() loop over the IB MD. PR #12266 ("UCP/FLUSH: Flush the other lanes after the pending lane") touches only UCP endpoint flush logic, which this test never exercises (no worker, no endpoint, no lanes). The test loops 300 MD allocations per memory type and tolerates UCS_ERR_NO_MEMORY only up to a 50% ratio (EXPECT_LT((double)num_alloc_failures / iterations, 0.5) << "Too many OUT_OF_RESOURCE failures"), so it is sensitive to memlock/memory pressure on the shared roce worker; it also evaluates EXPECT_GT(mem.length, 0ul) on an uninitialized mem before ASSERT_UCS_OK(status), so any non-NO_MEMORY error status yields a spurious failure. The exact assertion line could not be confirmed: the Azure log API only returns the tail of this job's log, and the per-test failure message was emitted much earlier in the ~49-minute run.

Implicated commit: unknown (no commit in the implicated path is recent; last change to test/gtest/uct/test_md.cc is 4cdc171, Tristan Rice, and it only touches CUDA VMM paths)

File: test/gtest/uct/test_md.cc:479-561 (UCS_TEST_SKIP_COND_P(test_md, alloc); failure checks at lines 537-544 and 559-560)

Suggested fix: Re-trigger the roce on worker 0 job — this is almost certainly an environment-sensitive flake unrelated to the PR. To make it actionable if it recurs: (1) publish/attach the full gtest log for this job so the test_md.alloc assertion text is visible; (2) harden the test by moving ASSERT_UCS_OK(status) (line 541) before EXPECT_GT(mem.length, 0ul) (line 537) so a non-UCS_ERR_NO_MEMORY status doesn't read uninitialized mem, and print ucs_status_string(status) plus num_alloc_failures in the failure message at line 559 to distinguish memory pressure from a real allocation bug.

Related: PR #12266 (#12266) — the PR under test; no existing issue found for ib/test_md.alloc flakiness.

@dmitrygx

dmitrygx commented Nov 6, 2026

Copy link
Copy Markdown
Member Author

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 4 pipeline(s).

@dmitrygx

dmitrygx commented Nov 6, 2026

Copy link
Copy Markdown
Member Author

@tvegas1 @evgeny-leksikov could you review pls? it fixes CI failure seen in many PRs.

@svc-ucx

svc-ucx commented Nov 6, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR (Cuda Test Cuda Docker ubuntu22_cuda_12_0) · commit 7d2798b4

TL;DR: The ubuntu22_cuda_12_0 CUDA job failed because cudaMalloc() in test_cuda_hook_dynamic returned CUDA error 807 (cudaErrorMpsServerNotReady) — a GPU/MPS-state problem on the CI agent, not a code defect from PR #12266. Re-run the job on a healthy GPU agent and harden the pre-test GPU check (and add step retry) so an unusable GPU is reported as infrastructure failure rather than a test failure.

Full analysis

Summary: Azure job "Test Cuda Docker ubuntu22_cuda_12_0" (build 143206) failed in the Memory hooks test step: ./test/apps/test_cuda_hook_dynamic printed cudaMalloc() failed: 807 / got 0/2 memory events and exited non-zero (Bash exited with code '255').

Root cause: Environment, not code. The build completed cleanly (continuous compile output 17:55:34 → 18:04:03, no gaps, no hang). The script's own guards passed — check_gpu 'gpu' found at least one GPU via nvidia-smi -L, and check_cuda_driver confirmed /proc/driver/nvidia/version exists — yet the very first CUDA allocation in the test failed with runtime error 807 = cudaErrorMpsServerNotReady ("the MPS server is not ready to accept new client requests / is recovering from a fatal failure"). UCX's hook layer only forwards the real return value (src/ucm/cuda/cudamem.c:27-45, UCM_CUDA_ALLOC_FUNC returns ret from ucm_orig_cudaMalloc verbatim), so 807 was produced by the CUDA runtime/driver on the agent, not by UCM. got 0/2 memory events is just the downstream consequence of the allocation never succeeding. Note the CUDA containers run with --ipc=host --gpus all (buildlib/pr/main.yml:4), so a broken/restarting MPS control daemon or wedged GPU on the host leaks straight into the container. Nothing in PR #12266 ("UCP/FLUSH: Flush the other lanes after the pending lane") touches CUDA, UCM hooks, or memory-event code, and ucp_init() in the test succeeded — so the PR cannot explain this failure.

Implicated commit: unknown — not a code regression; the tested commit [REDACTED:Hex High Entropy String] (PR #12266, UCP flush) is unrelated to the failing CUDA path.

File: buildlib/pr/cuda/test_malloc_hook.sh:43 (invocation) → failure surfaces at test/apps/test_cuda_hook.c:74-77

Suggested fix:

  1. Re-run the ubuntu22_cuda_12_0 matrix leg; check the agent's GPU health first (nvidia-smi -q, nvidia-cuda-mps-control -d status, GPU compute mode, Xid errors in dmesg). If MPS is running on that host, stop/restart it or exclude the host from the ucx_gpu_test pool.
  2. Harden the pre-flight check in buildlib/pr/cuda/test_malloc_hook.sh: extend check_cuda_driver into a real CUDA smoke probe (a trivial cudaMalloc/cuMemAlloc or nvidia-smi -q -d MEMORY) before the hook tests, and call azure_log_error "GPU unusable on $(hostname -s)" so the failure is attributed to infrastructure instead of looking like a UCX regression.
  3. Add retryCountOnTaskFailure: 2 to the Memory hooks test step in buildlib/pr/cuda/cuda.yml:116-121 (the other steps in that job already use retries) to absorb transient GPU/MPS states.
  4. Minor triage improvement: in test/apps/test_cuda_hook.c:76, print cudaGetErrorName/cudaGetErrorString(res) alongside the numeric code so logs say cudaErrorMpsServerNotReady instead of bare 807.

Related: PR #12266 (#12266) — the PR under test; no existing issue found tracking cudaMalloc() failed: 807 in the CUDA docker memory-hooks job, so consider opening one to track GPU-agent flakiness.

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 64cfcdbf-d013-4aa3-b65d-a7687af437ae in the triage console for the audit trail.

@dmitrygx

dmitrygx commented Nov 6, 2026

Copy link
Copy Markdown
Member Author

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 4 pipeline(s).

@tvegas1

tvegas1 commented Nov 10, 2026

Copy link
Copy Markdown
Contributor

Confirmed this fixes the tcp/test_ucp_rma.put_nonblocking/2 failure reported in #11923 (comment).

Repro, debug build with CUDA:

make -C test/gtest test GTEST_FILTER='tcp/test_ucp_rma.put_nonblocking/2' \
     GTEST_EXTRA_ARGS='--gtest_repeat=100 -s 22013' TEST_ENV='UCX_HANDLE_ERRORS=bt'

@dmitrygx

Copy link
Copy Markdown
Member Author

I thought on how to get ridd of https://github.com/openucx/ucx/pull/12266/changes#diff-5ac4f6dbe05b73e43bbad42c1f3e00b2e6b936054efe49fefdf5f669eab5c750L190, but there are cases which could lead to this branch gets true.

Comment on lines +119 to +138
static uct_ep_h test_flush_lane_eps[3];
static ucs_status_t test_flush_lane_status[3];
static unsigned test_flush_lane_calls[3];

static ucs_status_t
test_flush_by_lane(uct_ep_h ep, unsigned, uct_completion_t *)
{
unsigned lane;

for (lane = 0; lane < ucs_static_array_size(test_flush_lane_eps);
++lane) {
if (ep == test_flush_lane_eps[lane]) {
++test_flush_lane_calls[lane];
return test_flush_lane_status[lane];
}
}

return UCS_OK;
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it make sense to extend test_flush_inprogress_except_ep instead?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, with test_flush_no_resource_ep and the order in test_flush_eps the new arrays aren't needed. the first two tests could also be one test looping over the pending lane.

}

/* Complete the outstanding lane flushes of the request and release it */
static ucs_status_t test_flush_complete(ucp_request_t *req)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe rename? It is close to test_flush_completion

test_flush_completion,
"flush_with_pending_lane",
UCT_FLUSH_FLAG_LOCAL,
UCS_MASK(num_lanes));

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe assert that num_lanes fits test_flush_lane_eps?

Comment thread src/ucp/rma/flush.c
* Flush the other lanes from the pending callback, after
* these requests are sent. */
req->send.lane = lane;
break;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need this break for UCT_FLUSH_FLAG_CANCEL?

Comment thread src/ucp/rma/flush.c
ucp_trace_req(req, "resume slow path callback comp %d lane %d",
req->send.state.uct_comp.count, req->send.lane);

if (req->send.lane != UCP_NULL_LANE) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we avoid scheduling this callback at all? The pending callback schedules it on every NO_RESOURCE retry and after a hard error and in both cases it returns here because send.lane is still set

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, should the pending callback skip ucp_ep_flush_progress() when its own lane returned NO_RESOURCE? Why flush the other lanes before this lane is flushed?

Comment thread src/ucp/rma/flush.c
/* The request was added to the pending queue of this lane, and the
* requests queued there before the flush may send on the remaining
* lanes too, so do not flush them from here. The pending callback, or
* the purge of that queue, continues the flush. */

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"continues the flush"
Does the purge continue the flush? ucp_ep_flush_request_ff() completes it with an error instead?

@tvegas1 tvegas1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

approving, this fixes the CI regression.

the remaining gap is a proto multi request that wraps onto lanes the flush already flushed, and it also hits eager multi and rndv bulk. proposals if a future follow-up:

  • per-ep list: proto multi requests that run out of resources join a per-ep list, and a flush issued after them waits until they posted their last fragment before it flushes any lane. covers every proto multi user except requests with an indirect id.
  • lane pinning: a request that ran out of resources sends its remaining fragments on that lane only, so the deferral in this PR covers it. smaller, but no striping under load.

not enough on their own:

  • re-flushing the started lanes catches only one more wrap.
  • uct can't queue a request on lanes that still have resources.
  • a counter of stalled requests makes the flush wait for requests posted after it.

Comment on lines +119 to +138
static uct_ep_h test_flush_lane_eps[3];
static ucs_status_t test_flush_lane_status[3];
static unsigned test_flush_lane_calls[3];

static ucs_status_t
test_flush_by_lane(uct_ep_h ep, unsigned, uct_completion_t *)
{
unsigned lane;

for (lane = 0; lane < ucs_static_array_size(test_flush_lane_eps);
++lane) {
if (ep == test_flush_lane_eps[lane]) {
++test_flush_lane_calls[lane];
return test_flush_lane_status[lane];
}
}

return UCS_OK;
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, with test_flush_no_resource_ep and the order in test_flush_eps the new arrays aren't needed. the first two tests could also be one test looping over the pending lane.

EXPECT_EQ(2u, test_flush_lane_calls[1]);
EXPECT_EQ(1u, test_flush_lane_calls[2]);

/* The flush error on lane 0 is not reported, since the completion

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is it expected that the lane 0 error is lost here? ucp_ep_flush_completion() overwrites the req->status set by ucp_ep_flush_error(), so the flush returns UCS_OK after a lane failed. maybe need to fix instead of documenting it as expected in the test, or we could add a TODO?

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants