Repository navigation
Conversation
When an endpoint flush waits in the pending queue of a lane, the requests queued there before it can still send on the other lanes, e.g. a put which is split between the lanes. Check that the flush starts the other lanes only from its pending callback, after these requests are sent.
Since the selective flush change (openucx#11923), ucp_ep_flush_progress() went on flushing the remaining lanes after it added the flush request to the pending queue of a busy lane. A put which was split between the lanes and queued on that lane before the flush then sent its last fragment on a lane which was already flushed, so the endpoint and the worker flush completed before the put. In tcp/test_ucp_rma.put_nonblocking/2 the test then unmapped the user memh, and the late put completion released the freed memh, which corrupted the refcount of an rcache region (assertion in ucs_mem_region_destroy_internal()). Stop after adding the flush request to the pending queue, as before: the pending callback flushes the other lanes after the requests queued before the flush are sent. Update the test which expected the other lanes to be flushed right away.
|
🤖 Starting review — findings will be posted here when done. |
…again A flush which waits in the pending queue of a lane may have flushed the lanes before it already. The requests queued on that lane before the flush can still send on these lanes, e.g. a put with more fragments than lanes, so check that the pending callback flushes them again.
ucp_ep_flush_progress() flushes the lanes in increasing order, so the lanes before the one where the flush request goes to the pending queue are already flushed. A request queued on that lane before the flush can still send on them, since a multi-lane protocol sends its fragments round robin, so the flush could complete before these fragments. When the pending callback runs, the requests queued before the flush are sent. Flush the started lanes again at this point; their previous flush still completes, so each of them adds one completion.
|
🤖 Starting review — findings will be posted here when done. |
| reflush_lanes = req->send.flush.started_lanes & ucp_ep_get_live_lanes(ep) & | ||
| ~UCS_BIT(lane); | ||
| req->send.flush.started_lanes &= ~reflush_lanes; | ||
| ucp_ep_flush_request_update_uct_comp(req, ucs_popcount(reflush_lanes), 0); |
There was a problem hiding this comment.
pls move the re-flush after the uct_ep_flush() call below and do it only when it did not return UCS_ERR_NO_RESOURCE: on NO_RESOURCE the request stays in the pending queue, and every retry of the pending callback un-starts the other lanes again and posts one more flush with one more expected completion on each of them.
Impact: with the current order the lanes are re-flushed on every pending retry (confirmed: on NO_RESOURCE the function still calls ucp_ep_flush_progress() before returning, and next_lanes only excludes req->send.lane), so outstanding uct flushes on the other lanes accumulate while the pending lane has no resources. Also note the new tests clear test_flush_no_resource_ep before invoking ucp_ep_flush_progress_pending(), so this retry path is not covered.
There was a problem hiding this comment.
The re-flush is removed in 7d2798b, so a retry of the pending callback no longer touches the started lanes or the completion count. I removed it instead of moving it, because it also made the flush wait for the requests posted after it, and under sustained traffic on two lanes the flush could move between their pending queues without end (details in #12266 (comment)).
| * more fragments than lanes. Flush these lanes again; their previous flush | ||
| * still completes, so each of them needs one more completion. */ | ||
| reflush_lanes = req->send.flush.started_lanes & ucp_ep_get_live_lanes(ep) & | ||
| ~UCS_BIT(lane); |
There was a problem hiding this comment.
should we also exclude lanes that already failed in ucp_ep_flush_error()? they stay in started_lanes and in ucp_ep_get_live_lanes() until the scheduled failover callback runs, so a lane that just returned a hard error is flushed again, which reports the same error twice and schedules the same failed lane twice.
There was a problem hiding this comment.
Not needed any more: the re-flush is removed in 7d2798b, so the pending callback does not flush a lane whose flush failed again.
| * still completes, so each of them needs one more completion. */ | ||
| reflush_lanes = req->send.flush.started_lanes & ucp_ep_get_live_lanes(ep) & | ||
| ~UCS_BIT(lane); | ||
| req->send.flush.started_lanes &= ~reflush_lanes; |
There was a problem hiding this comment.
minor: pls clear started_lanes inside ucp_ep_flush_request_update_uct_comp() - it is the only place that keeps the count and the mask in sync, and as written the trace prints the already-cleared mask so the log does not show which lanes were restarted.
There was a problem hiding this comment.
Removed together with the re-flush in 7d2798b.
| ucp_request_release(request); | ||
| } | ||
|
|
||
| UCS_TEST_P(test_ucp_flush, pending_lane_defers_other_lanes, |
There was a problem hiding this comment.
minor: the two tests are ~50 nearly identical lines (setup, mock, request creation, completion drain, release) and differ only in the pending lane and the expected counters; maybe extract a helper parameterized by the pending lane?
There was a problem hiding this comment.
Done in 22ff84a: the setup, the request creation and the completion are in flush_with_pending_lane() and test_flush_complete(), and the flush mock counts the calls of each lane, so each test only has its own steps and expectations.
| ucp_request_release(request); | ||
| } | ||
|
|
||
| UCS_TEST_P(test_ucp_flush, pending_lane_reflushes_started_lanes, |
There was a problem hiding this comment.
minor: this test duplicates the preceding one (~50 nearly identical lines of setup, mock, request creation, completion drain, release), differing only in the pending lane and the expected counters; consider sharing a parameterized helper.
There was a problem hiding this comment.
Done in 22ff84a, see above. Since the re-flush is removed, this test is replaced by pending_lane_flushes_lanes_once, which checks that the pending callback of lane 1 does not flush lane 0 again.
|
🤖 CI Triage Agent — TL;DR: The Full analysisSummary: Root cause: Evidence from the log: Implicated commit: unknown — not attributable to File: test/gtest/uct/test_amo.cc:106 ( Suggested fix: Re-trigger the Related: PR under test: #12266; possibly relevant transport changes: #11841, #11915
|
Move the setup and the completion of the pending lane tests to flush_with_pending_lane() and test_flush_complete(), and count the flushes of each lane. Replace pending_lane_reflushes_started_lanes with pending_lane_flushes_lanes_once: when the pending callback of lane 1 runs and lane 0 has no resources again, lane 0 must not be flushed again. Add pending_lane_resume_skips_next_lanes: after the pending callback of lane 0 adds the flush request to the pending queue of lane 1, the resume callback must not flush lane 2.
The pending callback flushed the started lanes again, so the flush waited for the requests posted after it on these lanes too. If such a lane had no resources, the flush went to its pending queue, and the pending callback of that lane flushed the other lanes again, so with traffic on two lanes the flush could move between their pending queues without end. Remove it: the case it covered, a request queued before the flush with more fragments than lanes, is older than openucx#11923 and needs UCP to track the sends posted before the flush. When the flush of the pending lane fails, the pending callback can add the request to the pending queue of the next lane with no resources, and it still schedules the resume callback. The resume callback then flushed the lanes after that lane before the requests queued there were sent. Do nothing in the resume callback while req->send.lane is set: the pending callback of that lane, or the purge of its queue, continues the flush.
|
🤖 Starting review — findings will be posted here when done. |
|
🤖 Review complete — no issues found. |
|
🤖 CI Triage Agent — TL;DR: The Full analysisSummary: Root cause: Not a regression from PR #12266. The log shows continuous test activity to the end (no hang, no timeout), 9031 passing tests, and the only failure is in Implicated commit: unknown (no commit in the implicated path is recent; last change to File: test/gtest/uct/test_md.cc:479-561 ( Suggested fix: Re-trigger the Related: PR #12266 (#12266) — the PR under test; no existing issue found for |
|
/azp run |
|
Azure Pipelines successfully started running 4 pipeline(s). |
|
@tvegas1 @evgeny-leksikov could you review pls? it fixes CI failure seen in many PRs. |
|
🤖 CI Triage Agent — TL;DR: The Full analysisSummary: Azure job "Test Cuda Docker ubuntu22_cuda_12_0" (build 143206) failed in the Root cause: Environment, not code. The build completed cleanly (continuous compile output 17:55:34 → 18:04:03, no gaps, no hang). The script's own guards passed — Implicated commit: unknown — not a code regression; the tested commit [REDACTED:Hex High Entropy String] (PR #12266, UCP flush) is unrelated to the failing CUDA path. File: Suggested fix:
Related: PR #12266 (#12266) — the PR under test; no existing issue found tracking
|
|
/azp run |
|
Azure Pipelines successfully started running 4 pipeline(s). |
|
Confirmed this fixes the Repro, debug build with CUDA: make -C test/gtest test GTEST_FILTER='tcp/test_ucp_rma.put_nonblocking/2' \
GTEST_EXTRA_ARGS='--gtest_repeat=100 -s 22013' TEST_ENV='UCX_HANDLE_ERRORS=bt' |
|
I thought on how to get ridd of https://github.com/openucx/ucx/pull/12266/changes#diff-5ac4f6dbe05b73e43bbad42c1f3e00b2e6b936054efe49fefdf5f669eab5c750L190, but there are cases which could lead to this branch gets |
| static uct_ep_h test_flush_lane_eps[3]; | ||
| static ucs_status_t test_flush_lane_status[3]; | ||
| static unsigned test_flush_lane_calls[3]; | ||
|
|
||
| static ucs_status_t | ||
| test_flush_by_lane(uct_ep_h ep, unsigned, uct_completion_t *) | ||
| { | ||
| unsigned lane; | ||
|
|
||
| for (lane = 0; lane < ucs_static_array_size(test_flush_lane_eps); | ||
| ++lane) { | ||
| if (ep == test_flush_lane_eps[lane]) { | ||
| ++test_flush_lane_calls[lane]; | ||
| return test_flush_lane_status[lane]; | ||
| } | ||
| } | ||
|
|
||
| return UCS_OK; | ||
| } | ||
|
|
There was a problem hiding this comment.
Would it make sense to extend test_flush_inprogress_except_ep instead?
There was a problem hiding this comment.
+1, with test_flush_no_resource_ep and the order in test_flush_eps the new arrays aren't needed. the first two tests could also be one test looping over the pending lane.
| } | ||
|
|
||
| /* Complete the outstanding lane flushes of the request and release it */ | ||
| static ucs_status_t test_flush_complete(ucp_request_t *req) |
There was a problem hiding this comment.
Maybe rename? It is close to test_flush_completion
| test_flush_completion, | ||
| "flush_with_pending_lane", | ||
| UCT_FLUSH_FLAG_LOCAL, | ||
| UCS_MASK(num_lanes)); |
There was a problem hiding this comment.
Maybe assert that num_lanes fits test_flush_lane_eps?
| * Flush the other lanes from the pending callback, after | ||
| * these requests are sent. */ | ||
| req->send.lane = lane; | ||
| break; |
There was a problem hiding this comment.
Do we need this break for UCT_FLUSH_FLAG_CANCEL?
| ucp_trace_req(req, "resume slow path callback comp %d lane %d", | ||
| req->send.state.uct_comp.count, req->send.lane); | ||
|
|
||
| if (req->send.lane != UCP_NULL_LANE) { |
There was a problem hiding this comment.
Can we avoid scheduling this callback at all? The pending callback schedules it on every NO_RESOURCE retry and after a hard error and in both cases it returns here because send.lane is still set
There was a problem hiding this comment.
Also, should the pending callback skip ucp_ep_flush_progress() when its own lane returned NO_RESOURCE? Why flush the other lanes before this lane is flushed?
| /* The request was added to the pending queue of this lane, and the | ||
| * requests queued there before the flush may send on the remaining | ||
| * lanes too, so do not flush them from here. The pending callback, or | ||
| * the purge of that queue, continues the flush. */ |
There was a problem hiding this comment.
"continues the flush"
Does the purge continue the flush? ucp_ep_flush_request_ff() completes it with an error instead?
tvegas1
left a comment
There was a problem hiding this comment.
approving, this fixes the CI regression.
the remaining gap is a proto multi request that wraps onto lanes the flush already flushed, and it also hits eager multi and rndv bulk. proposals if a future follow-up:
- per-ep list: proto multi requests that run out of resources join a per-ep list, and a flush issued after them waits until they posted their last fragment before it flushes any lane. covers every proto multi user except requests with an indirect id.
- lane pinning: a request that ran out of resources sends its remaining fragments on that lane only, so the deferral in this PR covers it. smaller, but no striping under load.
not enough on their own:
- re-flushing the started lanes catches only one more wrap.
- uct can't queue a request on lanes that still have resources.
- a counter of stalled requests makes the flush wait for requests posted after it.
| static uct_ep_h test_flush_lane_eps[3]; | ||
| static ucs_status_t test_flush_lane_status[3]; | ||
| static unsigned test_flush_lane_calls[3]; | ||
|
|
||
| static ucs_status_t | ||
| test_flush_by_lane(uct_ep_h ep, unsigned, uct_completion_t *) | ||
| { | ||
| unsigned lane; | ||
|
|
||
| for (lane = 0; lane < ucs_static_array_size(test_flush_lane_eps); | ||
| ++lane) { | ||
| if (ep == test_flush_lane_eps[lane]) { | ||
| ++test_flush_lane_calls[lane]; | ||
| return test_flush_lane_status[lane]; | ||
| } | ||
| } | ||
|
|
||
| return UCS_OK; | ||
| } | ||
|
|
There was a problem hiding this comment.
+1, with test_flush_no_resource_ep and the order in test_flush_eps the new arrays aren't needed. the first two tests could also be one test looping over the pending lane.
| EXPECT_EQ(2u, test_flush_lane_calls[1]); | ||
| EXPECT_EQ(1u, test_flush_lane_calls[2]); | ||
|
|
||
| /* The flush error on lane 0 is not reported, since the completion |
There was a problem hiding this comment.
is it expected that the lane 0 error is lost here? ucp_ep_flush_completion() overwrites the req->status set by ucp_ep_flush_error(), so the flush returns UCS_OK after a lane failed. maybe need to fix instead of documenting it as expected in the test, or we could add a TODO?
What?
Fix a regression from #11923 that lets an endpoint flush and a worker flush complete before puts which were posted before them. When
ucp_ep_flush_progress()adds the flush request to the pending queue of a busy lane, it now stops there, and the lanes after it are flushed only from the pending callback of that lane, after the requests queued there before the flush are sent.This is the cause of the flaky
tcp/test_ucp_rma.put_nonblockingfailures in CI:Assertion `!(region->flags & UCS_RCACHE_REGION_FLAG_PGTABLE)' failedinucs_mem_region_destroy_internal()with a user memh (put_nonblocking/2), anddata validation failedwithout it (see the CI failures below).Why?
How the problem happens:
put/offload/zcopy, "50% on tcp/ib0 and 50% on tcp/ib3": one fragment on each lane.UCS_ERR_NO_RESOURCEfrom lane 0 and adds itself to the pending queue of lane 0, behind the put. This is the same forucp_ep_flush_nbx()and forucp_worker_flush_nbx(), which flushes each endpoint and completes fromucp_worker_flush_ep_flushed_cb().ucp_ep_flush_progress()picked the lowest unstarted lane again, gotUCS_ERR_NO_RESOURCEfrom lane 0 and stopped, so lane 1 was flushed only from the pending callback, after the put. UCP/FLUSH: Add topology-safe selective endpoint flush #11923 (UCP/FLUSH: Add topology-safe selective endpoint flush) changed the loop to iterate a precomputednext_lanesmask without the pending lane, so afteruct_ep_pending_add()the loop goes on and flushes lane 1 right away.test_ucp_memheap::test_xfer()unmaps the user memh right after the worker flush, and the late completion of the put (ucp_proto_request_zcopy_completion()->ucp_datatype_iter_cleanup()->ucp_memh_put()) uses the freed memh. Its memory is already reused for the memh of an rcache region (parent == NULL), soucp_memh_put()drops a refcount that the put never took, and the region refcount reaches 0 while the region is still in the page table. The assertion fires either in the late completion itself (build 141709) or later, in theucp_mem_unmap()of the region owner (the other builds). Without a user memh the same race shows up asdata validation failed.The resume callback of the flush has the same problem with three or more lanes: when the flush of the pending lane fails, the pending callback adds the request to the pending queue of the next lane with no resources, and it still schedules the resume callback, which then flushed the lanes after that lane before the requests queued there were sent.
CI history (failed builds of the "UCX PR" pipeline): in the month before #11923 was merged (2026-09-25 .. 10-25), none of the 87 failed builds has a tcp
put_nonblockingfailure. In the 12 days after it, 17 of 81 failed builds have one: 7 crash with the assertion above, the first one 3 hours after the merge, and 10 fail the test without a crash.How?
ucp_ep_flush_progress()breaks out of the lane loop after it adds the flush request to the pending queue of a lane, as before UCP/FLUSH: Add topology-safe selective endpoint flush #11923.ucp_ep_flush_progress_pending()flushes its lane and then callsucp_ep_flush_progress(), which flushes the remaining lanes after the requests queued before the flush were sent.ucp_ep_flush_resume_slow_path_callback()does nothing whilereq->send.laneis set, that is, after the request was added to the pending queue of a lane. The pending callback of that lane, or the purge of its queue, continues the flush.flush_with_pending_lane()andtest_flush_complete(), and the flush mock counts the calls of each lane.pending_lane_defers_other_lanes: lane 0 returnsUCS_ERR_NO_RESOURCE, and lane 1 is not flushed until the pending callback of lane 0 runs.pending_lane_flushes_lanes_once: lane 0 is flushed and lane 1 returnsUCS_ERR_NO_RESOURCE; the pending callback of lane 1 does not flush lane 0 again.pending_lane_resume_skips_next_lanes(MAX_EAGER_LANES=3): the flush of lane 0 fails in its pending callback and the request goes to the pending queue of lane 1; the resume callback does not flush lane 2 before the pending callback of lane 1.pending_error_lane_replacement, which expected lane 1 to be flushed in the first call; it now takes that completion after the pending callback.Not in this PR: the lanes before the pending lane are already flushed when the flush request goes to the pending queue, and a request queued before the flush with more fragments than lanes (e.g. an IOV put) can wrap around to them. This gap is older than #11923 and is not the case of the CI failures (one fragment per lane). An earlier revision of this PR flushed the started lanes again from the pending callback, but then the flush also waited for the requests posted after it, and with traffic on two lanes it could move between their pending queues without end. Covering the gap needs UCP to track the sends posted before the flush.
Testing
On a node with two tcp lanes (IPoIB), devel build like in CI,
UCX_PROTO_REQUEST_RESET=y:tcp/test_ucp_rma.put_nonblocking/2(user_memh)breakonly), 40 of 40 with the final changetcp/test_ucp_rma.put_nonblocking/0data validation failedin 2 of 60 runstest_ucp_flush.pending_lane_defers_other_lanestest_ucp_flush.pending_lane_flushes_lanes_oncetest_ucp_flush.pending_lane_resume_skips_next_lanes*test_ucp_flush*(withtest_ucp_flush_failover)*test_ucp_fault_tolerance*,*test_ucp_ep_reconfig*,*test_proto_reset.*,tcp/rcxtest_ucp_rma.*,dcx/shm_ibtest_ucp_rma.put*,rcx/tcptest_ucp_peer_failure.*,*test_ucp_ep_close*Test failures
CI
ucp_mem_unmap()ucp_mem_unmap()ucp_mem_unmap()ucp_mem_unmap()ucp_mem_unmap()Other failed builds with a tcp
put_nonblockingfailure since #11923: the same crash in 140518 (#11805, 2026-10-25), 142636 (#12234, 2026-11-05); a failure without a crash in 140693 (/2, #12162), 140878 (/0, #12180), 141265 (/2, #12162), 141689 (/2, #12201), 141923 (/0, #11529), 142018 (/1, #12206), 142100 (/1, #12195), 142189 (/0, #12215), 142321 (/0, #12165), 142554 (/0, #12200).Build 141668 (#12111, "Tests new on worker 1", 2026-10-29)
Build 141709 (#12180, "Tests new on worker 3", 2026-10-30)
Build 141739 (#12107, "Tests new on worker 0", 2026-10-30)
Build 141892 (#12135, "Tests new on worker 3", 2026-11-01)
Build 142125 (#12209, "Tests new on worker 3", 2026-11-02)
Build 142723 (#12205, "Tests new on worker 0", 2026-11-04)
Reproduction without the fix
tcp/test_ucp_rma.put_nonblocking/0 (no user memh): the late put shows up as a data validation failure
Debug build (instrumentation, not part of this PR): the worker flush completes from the endpoint flush while the last fragment of a put is in flight on lane 1
The instrumentation tracks
put/offload/zcopyrequests from their init to their completion, and dumps the state when a worker flush completes while a put posted before it is not completed. In every early completion (13, 11 and 8 of 60 runs in three loops) the worker flush completed fromucp_worker_flush_ep_flushed_cb(), and the put had sent all its data (offset 56860/56860), with one fragment still in flight (comp.count 1) on the second lane (ib3 outstanding 1), after it was sent from the pending queue of lane 0 (lane 0).`pending_lane_defers_other_lanes` on master: lane 1 is flushed in the first call
`pending_lane_flushes_lanes_once` with the re-flush of the previous revision (b2e79b2): the pending callback of lane 1 flushes lane 0 again
`pending_lane_resume_skips_next_lanes` without the resume fix (b2e79b2): the resume callback flushes lane 2 while the request waits on lane 1