Skip to content

UCP: Split GET by the put samples of the peer - #12335

Open
dmitrygx wants to merge 5 commits into
openucx:masterfrom
dmitrygx:topic/ucx/dlb-get-balance-v2
Open

dmitrygx wants to merge 5 commits into
openucx:masterfrom
dmitrygx:topic/ucx/dlb-get-balance-v2

Conversation

@dmitrygx

@dmitrygx dmitrygx commented Nov 14, 2026 •

Copy link
Copy Markdown
Member

What?

Split GET between the RMA lanes of an endpoint by the put samples of the peer. Part 3 of 3 of path bandwidth balancing: #12334 keeps the measured path bandwidth of each endpoint, #12362 splits PUT by it.

Based on #12362: its three commits come first, and this part is the last two commits (+654 / -41, of them +231 / -12 tests). It will be rebased on master after #12362 is merged.

Why?

On systems with two NICs per GPU, other traffic on one NIC slows down every GET of an endpoint to its lane on that NIC, as with PUT. But the GET side samples its paths poorly: its probes find no room in send queues full of its reads, which wait for read slots, and where they run, they also see its own reads. The data of GET crosses the paths on which the peer puts, and the peer samples these paths from endpoints without such work requests.

Two nodes, other traffic on one of the two NICs of the GPU (~75% of its bandwidth), balancing on vs off, single runs of the revision before the first review of #12334:

Under load Without load / after load
nixlbench READ 1.56x 1.00x / 1.00x
vLLM P/D, KV cache pulled by NIXL READ: tokens/s per GPU 1.44x (TTFT p50 688 vs 1653 ms) 0.99x / 1.00x

How?

  • After the put samples of an endpoint are updated, it sends them to the peer in UCP_AM_ID_EP_PATH_BW: per path, the ratio of the sample to the path estimate, with the memory domain and system device of both sides. The send runs from progress, after the update of all endpoints, since a failed send can invoke the error callback of the user, which may close endpoints. A degraded path is sent on every update, so the peer gets it again after its endpoint reconfiguration, and its recovery once. Only peers which handle the message get it (UCP_MIN_DST_VERSION_EP_PATH_BW).
  • The peer matches the paths to its lanes by memory domain and system device, scales its path estimate of GET (UCT_EP_OP_GET_ZCOPY) by the mean ratio of the matching paths, and keeps the result below the estimate as a second source of samples next to its own put samples. The path estimates of an interface are queried once per port speed.
  • get/offload/zcopy splits GET by these samples with the same lane adjustment as PUT in UCP/RMA: Balance PUT fragments by measured path bandwidth #12362. An endpoint keeps the sources of its degraded samples, so PUT and GET look up the samples only when their own source is degraded.

Not covered yet: RMA bcopy and rendezvous protocols.

Tests: test_ucp_proto_mock_path_bw* on rc_x and posix,cma: GET split by the put samples of the peer and its recovery, the split with two paths per device, GET without samples of the peer, a zero sample of the peer skips its lane, PUT and GET degraded apart, the samples sent after the update of all endpoints, and the path estimates queried once.

@dmitrygx
dmitrygx requested a review from a team as a code owner November 14, 2026 19:21
@dmitrygx dmitrygx added the WIP-DNM Work in progress / Do not review label Nov 14, 2026
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

@dmitrygx
dmitrygx force-pushed the topic/ucx/dlb-get-balance-v2 branch from f17305e to dbe6b8b Compare November 14, 2026 19:23
Comment thread src/ucp/core/ucp_worker.c Outdated
}
}

return ucp_proto_common_iface_bandwidth(context, &wiface->attr.bandwidth);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In ucp_worker_ep_path_bw_model, the UCP_EP_PATH_BW_OP_PUT fallback uses the full iface bandwidth, but uct_ep_query() reports a single path bandwidth (uct_ep_attr::bandwidth defaults to uct_perf_attr_t::path_bandwidth, and uct_ib_iface_estimate_path_bw() scales it by 1/dev_num_paths). So with dev_num_paths > 1 every lane is permanently degraded, and the ratio sent to the peer is ~1/num_paths, which the peer then applies again on top of its own get path bandwidth. Can we query UCT_PERF_ATTR_FIELD_PATH_BANDWIDTH with UCT_EP_OP_PUT_ZCOPY here too, like the peer-put branch does? The IB_NUM_PATHS=1 in the new test suggests this is already known.

Comment thread src/ucp/core/ucp_worker.c Outdated
goto out;
}

lane_map = ucp_worker_ep_path_bw_lanes(ep) &

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In ucp_worker_ep_path_bw_pack: the pack callback recomputes lane_map from the current state->valid[PUT], but the size was validated against max_bcopy back in ucp_worker_ep_path_bw_send(). If the request waits on the pending queue, another progress round can sample more lanes and add bits to valid[PUT], so the pack can write more than the validated size. pls bound the number of packed paths here (and use ucp_ep_get_max_bcopy() for the limit, since it also caps by the peer's seg_size).

return lpriv;
}

for (i = 0; i < mpriv->num_lanes; ++i) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In ucp_proto_rma_adjust_lane: maybe we could cache total_bw/kept_bw/max_frag_ratio (and the per-lane weight/max_frag) in ucp_ep_path_bw_t when the samples change? Right now every fragment of every RMA request on a degraded endpoint walks all lanes twice, while the inputs only change on a sample update (docs/OptimizationStyle.md: cache results that are stable).

{
const auto &tls = GetParam().transports;
ucs_assert(m_self == nullptr);
m_self = this;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
m_self = this;
m_path_bw_self = this;

minor: test_ucp_proto_mock_path_bw::m_self hides mock_iface::m_self, so this init()/dtor pair looks like it manages the base pointer but actually manages a second one; pls rename.

@svc-nvidia-pr-review

Copy link
Copy Markdown

The diff adds well over 500 lines (≈500 in ucp_worker.c, ≈150 in proto_multi.c, ≈660 in the gtest), which is above the limit in REVIEW.md. pls split it, e.g. (1) the proto_multi helper extraction (max_frag_ratio/lane_max_frag/lane_weight) as a pure refactor, (2) the worker sampling + UCP_AM_ID_EP_PATH_BW exchange, (3) the RMA lane adjustment and its tests.

Residual coverage gap: no test exercises dev_num_paths > 1 (the suite pins IB_NUM_PATHS=1), which is exactly the configuration affected by the put-model/path-bandwidth mismatch above.

Residual coverage gap: no coverage for the bcopy RMA protocols (put/offload/bcopy, get/bcopy) — they also use RMA_BW lanes but are not hooked into ucp_proto_rma_lane_priv(); if that is intentional, a short comment saying so would help.

Residual coverage gap: no test for the endpoint-close path while a UCP_AM_ID_EP_PATH_BW request is still on a lane's pending queue (ucp_ep_cleanup_lanes() purges with ucp_destroyed_ep_pending_purge, which is a ucs_bug). I believe flush ordering makes this unreachable, but it is untested.

@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread src/ucp/core/ucp_worker.c
return state;
}

ucp_worker_ep_path_bw_cleanup(ep);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In ucp_worker_ep_path_bw_get: when the state is stale, ucp_worker_ep_path_bw_cleanup() also removes the queued oneshot and the new state starts with valid[PUT] == 0 and pending == 0. On the AM-handler path nothing re-arms sampling: ucp_proto_rma_adjust_lane() calls refresh() only while the state is stale, and for op == PUT it returns early on !(state->valid[op] & mpriv->lane_map). Impact: if the peer's UCP_AM_ID_EP_PATH_BW is handled after the local reconfiguration (very likely, since both sides react to the same link slowdown), the local put samples are dropped and the degraded put path is used at full weight until the next port-speed event.

Can we call ucp_worker_ep_path_bw_refresh() for the newly allocated state here? The cleanup above also cancels a pending refresh, and in the AM handler path nothing re-arms sampling, so the local put samples are lost until the next port speed event.

return lpriv;
}

for (i = 0; i < mpriv->num_lanes; ++i) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In ucp_proto_rma_adjust_lane: both lane loops (total_bw/max_frag_ratio, then kept_bw) plus the kh_get() run per fragment, per lane, for as long as UCP_EP_FLAG_PATH_DEGRADED is set — including when only the opposite direction is degraded, where the function ends up returning lpriv unchanged after the full O(num_lanes) FP loop. Maybe cache the per-lane weight/max_frag (and kept_bw/max_frag_ratio) in ucp_ep_path_bw_t when the samples change? docs/OptimizationStyle.md asks to keep this path free of O(n) work and cache stable results.

Comment thread src/ucp/core/ucp_worker.c Outdated
goto out;
}

lane_map = ucp_worker_ep_path_bw_lanes(ep) &

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In ucp_worker_ep_path_bw_pack: pack recomputes the path list from the current state, while the max_bcopy bound was checked in ucp_worker_ep_path_bw_send() for a possibly smaller put_lanes. A request that waits on the pending queue can therefore pack more paths than were validated — e.g. the endpoint is reconfigured with more rma_bw lanes, the new _send() size check fails and returns, and the queued request then packs the larger list into the bcopy buffer. Maybe store the checked lane map in the request and pack only it?

Comment thread src/ucp/core/ucp_worker.c Outdated

req = ucp_request_get(ep->worker);
if (req == NULL) {
ucs_error("ep %p: failed to allocate path bandwidth request", ep);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor: request-pool exhaustion here only skips one best-effort sample message; the flow continues as usual, so error is the wrong level per docs/LoggingStyle.md (error = "could not continue as usual"). Suggestion:

Suggested change
ucs_error("ep %p: failed to allocate path bandwidth request", ep);
ucs_debug("ep %p: failed to allocate path bandwidth request", ep);

@svc-nvidia-pr-review

Copy link
Copy Markdown

Residual coverage gaps: (1) No test covers the interleaving in finding 1 (peer EP_PATH_BW message handled after a local reconfiguration while a refresh is pending) — reconfig_resets_samples and put_reconfig_refreshes_samples each exercise only one of the two paths. (2) No test drives ucp_worker_ep_path_bw_send() on a lane with no send resources, so the pack-vs-max_bcopy divergence in finding 3 is never exercised.

@svc-ucx

svc-ucx commented Nov 15, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR · commit dbe6b8b7

Summary: Unable to fetch logs — cannot diagnose without them.
Root cause: Unknown — log access failed.
Implicated commit: unknown
File: unknown
Suggested fix: Provide a direct link to the failed build log in a follow-up comment.
Log fetch attempts:

  • fetch_azure_logs({'build_url': 'https://dev.azure.com/ucfconsort/0b36e3f0-8ab9-4a48-b68b-[REDACTED:Hex High Entropy String]/_build/results?buildId=145146'}) → [fetch_azure_logs ERROR: no failed job found in timeline for build 145146]
  • fetch_jenkins_stages({'job': 'UCX PR', 'build': 145146}) → [fetch_jenkins_stages ERROR] build URL names Jenkins controller 'https://dev.azure.com/ucfconsort/0b36e3f0-8ab9-4a48-b68b-[REDACTED:Hex High Entropy String]/_build/results?buildId=145146', but this deployment is configured for 'https://nbuprod.blsm.nvidia.com/nb
  • fetch_jenkins_logs({'job': 'UCX PR', 'build': 145146}) → [fetch_jenkins_logs ERROR] build URL names Jenkins controller 'https://dev.azure.com/ucfconsort/0b36e3f0-8ab9-4a48-b68b-[REDACTED:Hex High Entropy String]/_build/results?buildId=145146', but this deployment is configured for 'https://nbuprod.blsm.nvidia.com/nbu-
  • fetch_azure_logs({'build_url': 'https://dev.azure.com/ucfconsort/ucx/_build/results?buildId=145146'}) → [fetch_azure_logs ERROR: no failed job found in timeline for build 145146]
  • fetch_azure_logs({'build_url': 'https://dev.azure.com/ucfconsort/0b36e3f0-8ab9-4a48-b68b-[REDACTED:Hex High Entropy String]/_build/results?buildId=145146&view=logs'}) → [fetch_azure_logs ERROR: no failed job found in timeline for build 145146]

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id dd82289f-3d28-406b-9e0d-8eeedde42125 in the triage console for the audit trail.

@dmitrygx
dmitrygx force-pushed the topic/ucx/dlb-get-balance-v2 branch 3 times, most recently from eaafe9a to c98f548 Compare November 16, 2026 11:01
@svc-ucx

svc-ucx commented Nov 16, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR (Tests roce on worker 2) · commit c98f5489

TL;DR: The gtest job failed on a single hung test — ud/test_ucp_sockaddr_protocols_err.tag_zcopy_64k_unexp/13 (ud_v,...,bidi_stop,mt) ran 180,864 ms (vs ~300 ms for every sibling variant) before the framework's deadlines failed it; the hang is in the force-close/error-handling flow and the prime suspect is this PR's new untracked internal path-bandwidth control request (ucp_worker_ep_path_bw_send()), which can be posted on an endpoint that is failing/closing and is not guarded or purged there.

Full analysis

Summary: make test (gtest, "roce on worker 2") exited 1 because 1 of 9118 tests failed: ud/test_ucp_sockaddr_protocols_err.tag_zcopy_64k_unexp/13, GetParam() = ud_v,cuda_copy,rocm_copy/bidi_stop,mt.

Root cause: From the log: all other test_ucp_sockaddr_* variants complete in 200–450 ms, while this one is listed as TOP-1 longest at 180,864 ms — i.e. it made no progress and was failed by ucp_test's own wait deadlines (request_process() → "request %p did not complete on time", test/gtest/ucp/ucp_test.cc:299-311), not by a data/assert mismatch. The bidi_stop variant force-closes both endpoints (UCP_EP_CLOSE_FLAG_FORCE) while a 64 KB multi-lane zcopy tag send is in flight (test/gtest/ucp/test_ucp_sockaddr.cc:2307-2313, 2402-2407), so a request/EP-close that never completes produces exactly this signature.
The branch under test (PR #12335, stacked on #12334) introduces per-endpoint path-bandwidth sampling: a new worker hash ep_path_bw, the new UCP_EP_FLAG_PATH_DEGRADED flag, and a new internal control AM UCP_AM_ID_EP_PATH_BW. ucp_worker_ep_path_bw_send() (src/ucp/core/ucp_worker.c:1228) allocates a bare ucp_request_t (flags = UCP_REQUEST_FLAG_RELEASED, send.uct.func = ucp_worker_ep_path_bw_progress_send) and calls ucp_request_send() on the EP's AM lane. Unlike the iface-event path (ucp_worker_iface_path_bw_progress, line 1331-1337, which skips UCP_EP_FLAG_FAILED|UCP_EP_FLAG_CLOSED EPs), this send path checks only lanes/dst_version/REMOTE_ID, and the receive handler (line 1444) checks only FAILED, not CLOSED. Such a request, once parked on a lane's pending queue of an EP that is being force-closed/failed, is not tracked in the EP's request list, so the close/flush can wait on it indefinitely — matching the observed 180 s stall in precisely the error-handling variant. (Note the previous incarnation of this feature had to be reverted: #12082 reverting #12002.)

Implicated commit: [REDACTED:Hex High Entropy String] — "UCP: Split GET by the put samples of the peer" (Dmitrii), on top of 0f401ed / [REDACTED:Hex High Entropy String] of the same series.

File: src/ucp/core/ucp_worker.c:1228-1268 (ucp_worker_ep_path_bw_send), also :1444 (ucp_worker_ep_path_bw_handler)

Suggested fix:

  1. Guard the sample send the same way as the iface progress loop: bail out if ep->flags & (UCP_EP_FLAG_FAILED | UCP_EP_FLAG_CLOSED) (and require UCP_EP_FLAG_REMOTE_CONNECTED), and add the CLOSED check in ucp_worker_ep_path_bw_handler too.
  2. Make the internal path-bw request purgeable: register it like other internal control messages so ucp_ep_purge_lanes()/ucp_ep_req_purge() completes it on force close, instead of leaving it on the lane pending queue with UCP_REQUEST_FLAG_RELEASED and no owner.
  3. To confirm before fixing, re-run this single variant with the full log (--gtest_filter='ud/test_ucp_sockaddr_protocols_err.tag_zcopy_64k_unexp/13' plus UCX_LOG_LEVEL=debug) and check whether an EP_PATH_BW request is pending on the closing EP at the moment of the stall; the truncated Azure tail does not contain the per-test error lines.

Related: PR #12335 (this change), PR #12334 "UCP: Keep measured path bandwidth per endpoint", prior reverted attempt: #12002 / #12015, reverted by #12082.

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 1dc807e4-af58-4f6b-889e-304693a6c899 in the triage console for the audit trail.

@dmitrygx
dmitrygx force-pushed the topic/ucx/dlb-get-balance-v2 branch 2 times, most recently from 3c466ab to d2feb04 Compare November 16, 2026 16:16
@svc-ucx

svc-ucx commented Nov 16, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR (Tests gpu on worker 2) · commit d2feb04d

TL;DR: The gtest run aborted on a debug assertion !req->send.flush.started_lanes in ucp_ep_flush_request_resched() (flush.c) while a partially-started flush request was being re-pended from the failover path (ucp_wireup_replay_pending_requests → ucp_ep_flush_progress_pending) in rcx/test_ucp_fault_tolerance.target_failure/7 <rc_x/GET|FLUSH>; the assertion wrongly assumes a blocked-flush EP can never have lanes already flushing, which is false after a failover lane reconfiguration — relax/adjust it (or reset started_lanes via the lane-generation update path) instead.

Full analysis

Summary: rcx/test_ucp_fault_tolerance.target_failure/7 <rc_x/GET|FLUSH> aborted with flush.c:406 Assertion '!req->send.flush.started_lanes' failed: req=0x56052c340140 flush started_lanes=0x5, causing make test to exit with "Aborted (core dumped)" and the job to fail with Bash code 2.

Root cause: Sequence from the log: the test injected a target failure on lane 1/2, then entered "Checking recovery status..."; during recovery the EP was reconfigured and marked UCP_EP_FLAG_BLOCK_FLUSH, and the in-flight flush request was replayed from the wireup pending queue. Backtrace: ucp_worker_progress → ucp_wireup_eps_progress → ucp_wireup_replay_pending_requests → ucp_ep_flush_progress_pending → (inlined ucp_ep_flush_request_resched) → ucs_fatal_error_format.

In ucp_ep_flush_progress_pending() (src/ucp/rma/flush.c:417-419), when UCP_EP_FLAG_BLOCK_FLUSH is set the request is rescheduled via ucp_ep_flush_request_resched(). That function (src/ucp/rma/flush.c:366-389) asserts, for EPs with a CM lane or p2p lanes + proto_request_reset, that no lane has started flushing (started_lanes == 0). The reported value started_lanes=0x5 shows lanes 0 and 2 had already begun uct_ep_flush() before the failover reconfiguration set BLOCK_FLUSH. The assertion encodes the initial-wireup assumption ("flush cannot have started while the EP is blocked"), which no longer holds for the failover/selective-flush flow where a partially-started flush request is detached from pending and re-scheduled. This is a debug-build-only assertion (release builds would continue), and it is in the FT/flush code, unrelated to this PR's content (topic/ucx/dlb-get-balance-v2, path-bandwidth bookkeeping) — i.e. a pre-existing race in the failover flush path, not a regression from commit d2feb04.

Implicated commit: d513f98 "UCP/FLUSH: Add topology-safe selective endpoint flush (#11923)" (nbellalou) introduced the partial started_lanes tracking that this assertion contradicts; interacts with the failover replay added in 7561177 / 86cfb87 / b3bf0e5 (Evgeny Leksikov). Not caused by the PR commit [REDACTED:Hex High Entropy String].

File: src/ucp/rma/flush.c:371-373 (assertion; reported as flush.c:406 in the PR branch build) reached from src/ucp/rma/flush.c:417-419

Suggested fix: Make the reschedule path tolerant of an already-started flush when the EP is blocked due to failover rather than initial wireup. Concretely, in ucp_ep_flush_request_resched() either (a) skip the !req->send.flush.started_lanes / p2p-lane assertions when the EP is in failover reconfiguration (e.g. ucp_ep_err_mode_eq(ep, UCP_ERR_HANDLING_MODE_FAILOVER) or req->send.flush.lane_generation != ep->ext->lane_generation), or (b) before rescheduling, run the lane-generation reconciliation (ucp_ep_flush_lane_state_update() with lane_generation_changed=1) so started_lanes is cleared and uct_comp.count is re-adjusted consistently, then assert. Also clear req->send.lane consistently in that path. Re-run CI on the PR afterwards; the PR itself should not be blocked by this failure.

Related: #12266 (UCP/FLUSH: Flush the other lanes after the pending lane — same area, open), #11923, #12274, #12183

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id f3c6eb5b-17c0-451c-ae2b-c9db0f153ef9 in the triage console for the audit trail.

@dmitrygx
dmitrygx force-pushed the topic/ucx/dlb-get-balance-v2 branch from d2feb04 to ac17d46 Compare November 17, 2026 09:04
@dmitrygx dmitrygx changed the title [WIP-DNM] UCP/RMA: Balance PUT and GET by measured path bandwidth UCP: Split GET by the put samples of the peer Nov 17, 2026
@dmitrygx
dmitrygx force-pushed the topic/ucx/dlb-get-balance-v2 branch from ac17d46 to c5b4e5c Compare November 17, 2026 09:22
Keep the samples of each lane per source, so that a source other than
the put of this side can split other operations. The put of this side,
which splits put, is the only source for now. Setting a sample and the
endpoint flag of a degraded path go to helpers shared by the sources.
@dmitrygx
dmitrygx force-pushed the topic/ucx/dlb-get-balance-v2 branch from c5b4e5c to 9c16b4f Compare November 17, 2026 10:37
@svc-ucx

svc-ucx commented Nov 17, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR (AddressSanitizer roce on worker 0) · commit 9c16b4fe

TL;DR: The ASan roce on worker 0 job never reached any test — it was still compiling the gtest sources when the Azure DevOps agent itself was shut down ("The agent has received a shutdown signal"), so this is an infrastructure/agent-loss cancellation, not a defect in PR #12335. Re-run the job; no code change is warranted.

Full analysis

Summary: Build of the AddressSanitizer roce job was aborted mid-compile by an agent shutdown/cancellation, reported as ##[error]The operation was canceled.

Root cause: The log shows a normal, continuously progressing ASan build (./contrib/test_jenkins.sh → make of test/gtest). Compilation lines flow steadily with no stalls: 13:21:55 mv -f uct/ib/.deps/gtest-test_cqe_zipping.Tpo, 13:22:11 mv -f ucp/.deps/gtest-test_ucp_sockaddr.Tpo, 13:22:12 mv -f uct/ib/.deps/gtest-test_dc.Tpo, then at 13:22:38 → ##[error]The agent has received a shutdown signal. This can happen when the agent service is stopped, or a manually started agent is canceled. followed by The Operation will be canceled / The operation was canceled / Finalize Job. The largest gap between consecutive log lines is ~26 s, so nothing hung and nothing timed out from the job's side — the hosted agent/container was reclaimed (service stop, node eviction, or a manual cancel) while g++ -fsanitize=address was still building gtest objects. No compiler error, no linker error, no gtest output, and no ASan report appears anywhere in the log; zero tests were executed.

Implicated commit: none — not a code failure. The tested commit [REDACTED:Hex High Entropy String] (branch topic/ucx/dlb-get-balance-v2) produced no build or test errors before the abort.

File: n/a (failure originates in the Azure DevOps agent lifecycle, during contrib/test_jenkins.sh's build step)

Suggested fix: Simply re-trigger the AddressSanitizer roce on worker 0 job for PR #12335 — there is no actionable code defect in this log. If this recurs on the same worker, have CI ops check that agent/node for eviction, OOM-kill of the agent process, or service restarts (the ASan build is memory-heavy: -fsanitize=address with -O3 -g on the full gtest tree, and it had been compiling for ~19 minutes when the agent died). As a robustness improvement, consider parallelizing/caching the ASan gtest build so the job spends less wall time in the compile phase and is less exposed to agent reclamation windows.

Related: none — searches for "agent has received a shutdown signal" and related terms returned only unrelated open PRs (#12317, #12272, #12325, #12194), no existing issue tracking this agent cancellation.

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id e5e8d058-9cc2-4a9f-b22d-2f8c039a829c in the triage console for the audit trail.

@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread src/ucp/core/ucp_worker.c
!(ep->flags & UCP_EP_FLAG_REMOTE_ID) ||
((sizeof(ucp_ep_path_bw_hdr_t) +
(ucs_popcount(lane_map) * sizeof(ucp_ep_path_bw_path_t))) >
ucp_ep_get_iface_attr(ep, key->am_lane)->cap.am.max_bcopy)) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pls bound the message by ucp_ep_get_max_bcopy(); the local cap.am.max_bcopy ignores the peer's seg_size (key->lanes[].seg_size is the remote limit, see ucp_ep_get_max_bcopy), so with a smaller remote segment size we would post an AM bigger than the receiver's buffer. All other AM bcopy senders use it, and proto_am.inl even asserts against it.

Suggested change
ucp_ep_get_iface_attr(ep, key->am_lane)->cap.am.max_bcopy)) {
((sizeof(ucp_ep_path_bw_hdr_t) +
(ucs_popcount(lane_map) * sizeof(ucp_ep_path_bw_path_t))) >
ucp_ep_get_max_bcopy(ep, key->am_lane))) {

Comment thread src/ucp/core/ucp_worker.c Outdated
req->send.ep = ep;
req->send.datatype = ucp_dt_make_contig(1);
req->send.uct.func = ucp_worker_ep_path_bw_progress_send;
state->sync_peer = degraded;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sync_peer is cleared here, before the send can fail: if uct_ep_am_bcopy() returns an error other than UCS_ERR_NO_RESOURCE, the recovery message is dropped and the peer keeps applying the stale degraded samples to its get until the next degradation. This also contradicts the comment in ucp_worker_ep_path_bw_progress_send() that the next update sends them again - can we clear it only once the message is actually packed?

@svc-nvidia-pr-review

Copy link
Copy Markdown

Residual coverage gaps (beyond those already reported): no test exercises a peer whose AM-lane segment size is smaller than the packed EP_PATH_BW message, nor an EP_PATH_BW send that fails with an error other than UCS_ERR_NO_RESOURCE (the lost-recovery path above).

@dmitrygx
dmitrygx force-pushed the topic/ucx/dlb-get-balance-v2 branch from 9c16b4f to 8ec967c Compare November 17, 2026 17:09
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread src/ucp/proto/proto_multi.c Outdated
ucp_proto_common_lane_priv_init(&params->super, mpriv->reg_md_map, lane,
&lpriv->super);
/* Path samples are compared with the bandwidth of one path */
lpriv->bandwidth = lane_perf->bandwidth * lane_perf->path_ratio;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lane_perf->bandwidth is already capped by the local/remote distance in ucp_proto_common_update_lane_perf_by_distance(), while path_ratio was computed from the uncapped iface bandwidth, so this double-counts the cap. The samples it is compared with come from uct_ep_query() and are not capped, so on a PCIe-limited lane (GPU buffers) a degraded path keeps a larger share than it should. Can we keep the absolute path bandwidth in ucp_proto_common_tl_perf_t instead of a ratio?

Comment thread src/ucp/core/ucp_worker.c Outdated
return;
}

req = ucp_request_get(ep->worker);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No guard against stacking path-bandwidth requests: ucp_worker_ep_path_bw_send() allocates and submits a new request on every update while the path is degraded. Please skip the send while a previous request is still outstanding (e.g. a flag in state); uct_ep_am_bcopy() returns UCS_ERR_NO_RESOURCE while the AM lane is a wireup proxy ep (ucp_wireup_ep_t rejects every id except UCP_AM_ID_WIREUP), and then every following path event adds another request to the lane pending queue, with nothing bounding it since these requests are not counted in worker->flush_ops_count, and they all pack identical state once resources free up.

double bandwidth, selected_bw, min_bw;
ucp_lane_index_t i;

for (i = 0; i < mpriv->num_lanes; ++i) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor: per-fragment O(num_lanes) recompute. ucp_proto_rma_split_lane() runs two passes over all lanes plus floating-point divisions for every fragment, but total_bw and max_frag_ratio do not depend on length and change only when the samples or the endpoint configuration change, so they could be computed once per samples update; docs/OptimizationStyle.md asks to keep scheduling O(1) in the large-message path.

@svc-nvidia-pr-review

Copy link
Copy Markdown

Residual coverage gaps: no test uses a non-host (GPU) buffer, which is the configuration where the distance cap in finding 1 changes the split; and no test drives ucp_worker_ep_path_bw_send() repeatedly while the AM lane cannot send, which is the stacking case in finding 2.

@dmitrygx
dmitrygx force-pushed the topic/ucx/dlb-get-balance-v2 branch from 8ec967c to 5f7f7a7 Compare November 17, 2026 19:22
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread src/ucp/core/ucp_worker.c

ucp_ep_update_flags(ep, 0, UCP_EP_FLAG_PATH_DEGRADED);
ucp_worker_ep_path_bw_update_flag(ep, state);
ucp_worker_ep_path_bw_send(ep, state);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this sends while ucp_worker_iface_path_bw_progress() iterates worker->all_eps, and uct_ep_am_bcopy() can report a failure inline (e.g. uct_tcp_ep_handle_send_err() -> uct_iface_handle_ep_err()), which ends in ucp_ep_invoke_err_cb(); a user callback that closes the endpoint removes it from all_eps and the loop then walks freed memory. keepalive avoids this with worker->keepalive.iter + ucp_worker_keepalive_remove_ep() - can we collect the endpoints and send after the loop instead?

Impact: potential use-after-free / list corruption on the worker's endpoint list during a path-bandwidth update.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

@svc-nvidia-pr-review

Copy link
Copy Markdown

Residual coverage gap: no test closes or fails an endpoint from the error callback while a path-bandwidth update is being delivered, which is exactly the interleaving described in the finding above.

Residual coverage gap: no test exercises an AM lane on a different transport than the one reporting the path event (the suite uses rcx and posix,cma, where the AM lane and the sampled lanes come from the same mocked transport).

@dmitrygx
dmitrygx force-pushed the topic/ucx/dlb-get-balance-v2 branch from 5f7f7a7 to 10b5f94 Compare November 17, 2026 22:44
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread src/ucp/core/ucp_worker.c

for (lane = 0; lane < ucp_ep_num_lanes(ep); ++lane) {
for (op = 0; op < UCP_EP_PATH_BW_OP_LAST; ++op) {
if (state->bandwidth[lane][op] != UCP_EP_PATH_BW_INVALID) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe we could keep the degraded state per source instead of one endpoint flag? as it is, an endpoint degraded only in UCP_EP_PATH_BW_OP_PEER_PUT makes every PUT fragment enter ucp_proto_rma_adjust_lane() (hash lookup + two loops over the lanes) only to find degraded == 0 and return lpriv, and that path never clears the flag, so the cost stays as long as the other direction is degraded (and vice versa for GET).

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

Comment thread src/ucp/core/ucp_worker.c
rsc = &context->tl_rscs[key->lanes[lane].rsc_index];
bandwidth = state->bandwidth[lane][UCP_EP_PATH_BW_OP_PUT];
if ((bandwidth == UCP_EP_PATH_BW_INVALID) ||
(ucp_worker_iface_path_bw_estimate(

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minor: ucp_worker_iface_path_bw_estimate() is re-queried here per degraded lane per endpoint, so one path event now does O(eps x degraded lanes) uct_iface_estimate_perf() calls under the async lock instead of the single call ucp_worker_iface_path_bw_progress() already made. The value is stable per iface and operation, so can we cache it on ucp_worker_iface_t (or pass down the estimate the caller already computed)? See "Cache expensive capability queries when the result is stable" in docs/OptimizationStyle.md. The same applies to the call in ucp_worker_ep_path_bw_peer_update().

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

@svc-nvidia-pr-review

Copy link
Copy Markdown

Residual coverage gap: no test covers an endpoint whose two directions are degraded independently (peer-put degraded while local put is healthy and the reverse within one test), which is the state that keeps UCP_EP_FLAG_PATH_DEGRADED set for the healthy direction.

Residual coverage gap: no test has an endpoint with RMA BW lanes spread over more than one ucp_worker_iface_t, so ucp_worker_ep_path_bw_pack() is never exercised with lanes whose path estimates come from different ifaces.

@dmitrygx
dmitrygx force-pushed the topic/ucx/dlb-get-balance-v2 branch from 10b5f94 to a6210be Compare November 17, 2026 23:52
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread src/ucp/core/ucp_worker.c
hdr->ep_id, hdr->num_paths);
}

UCP_DEFINE_AM_WITH_PROXY(UINT64_MAX, UCP_AM_ID_EP_PATH_BW,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why WITH_PROXY? the counting proxy makes this message count as interface activity, so receiving it activates and progress-enables an interface that was deactivated by adaptive progress, although it carries no data for that interface. UCP_AM_ID_WIREUP, the other UINT64_MAX control AM, uses UCP_DEFINE_AM.

Comment thread src/ucp/core/ucp_worker.c
rsc = &context->tl_rscs[key->lanes[lane].rsc_index];
bandwidth = state->bandwidth[lane][UCP_EP_PATH_BW_OP_PUT];
if ((bandwidth == UCP_EP_PATH_BW_INVALID) ||
(ucp_worker_iface_path_bw_estimate(

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe compute the path estimate per iface in ucp_worker_ep_path_bw_send() and pass it in the request? here it is a uct_iface_estimate_perf() call per lane from inside the uct_ep_am_bcopy() pack callback, and lanes on the same iface repeat the same query.

@svc-nvidia-pr-review

Copy link
Copy Markdown

Residual coverage gap: no test covers the AM-id/dst-version compatibility gate itself (key->dst_version < UCP_MIN_DST_VERSION_EP_PATH_BW), i.e. that no EP_PATH_BW message is sent to a peer that would not have a handler for AM id 29.

Residual coverage gap: no test exercises an endpoint whose AM lane is on a transport without UCT_IFACE_FLAG_CB_SYNC (the handler is registered with flags = 0), so the "sync handler not installed on this iface" path is never reached.

Residual coverage gap: ucp_worker_ep_path_bw_dump() is only reached with UCX_LOG_LEVEL=data; no test or CI job in the diff enables packet tracing for this AM.

@svc-ucx

svc-ucx commented Nov 18, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR (AddressSanitizer roce on worker 3) · commit a6210be3

TL;DR: The ASAN "roce on worker 3" job failed on a single gtest, ud/test_ucp_peer_failure.zcopy/3 <ud_v/rma_fail_imm> (9129/9130 passed); this is the long-standing timing-sensitive UD peer-failure RMA test (open issues #6762/#6203 report the identical test_ucp_peer_failure.zcopy/3 variant failing), not an obvious regression from the PR — rerun the job and, if it reproduces, loosen the strict status equality checks in do_test().

Full analysis

Summary: make test in the AddressSanitizer/roce gtest stage exited with code 1 because exactly one test failed: ud/test_ucp_peer_failure.zcopy/3, where GetParam() = ud_v/rma_fail_imm (build 145836, PR #12335).

Root cause: The Azure job log (only the tail is retrievable; the per-test [ FAILED ] detail scrolled out of the fetched window) shows a clean run of 9130 tests with one failure and no sanitizer report, no crash, and no hang — total runtime 2,298,722 ms with continuous activity, so this is a plain gtest expectation failure, not a timeout or an ASAN error. The failing case is the RMA + fail-immediately variant of the UD peer-failure test: do_test(UCS_KBYTE, 0, false, /*must_fail=*/true) with ZCOPY_THRESH=1023. That body contains several strict, timing-dependent expectations around an artificially shortened UD peer timeout (set_ib_ud_peer_timeout(3.)): EXPECT_EQ(UCS_OK, m_err_status) before the receiver is killed (test/gtest/ucp/test_ucp_peer_failure.cc:395), and the exact-match checks EXPECT_TRUE((m_err_status == status) || (UCS_ERR_CANCELED == status)) (:421) and EXPECT_EQ(m_err_status, status) (:432). Under AddressSanitizer the whole run is heavily slowed, so a 3 s UD peer timeout can expire (or fire on the "stable" pair / before the expected point), and the per-request status (UCS_ERR_ENDPOINT_TIMEOUT vs UCS_ERR_CONNECTION_RESET vs UCS_ERR_CANCELED) can legitimately differ from m_err_status, which breaks these equality expectations. Two still-open issues report exactly this test/variant index failing sporadically on other transports (udx, dcx), i.e. pre-existing flakiness rather than new breakage. The PR's own change (ucp_proto_rma_adjust_lane / ucp_proto_rma_split_lane in src/ucp/proto/proto_multi.c:53-170) is on the RMA multi-lane split path and is only active when mpriv->num_lanes >= 2 and UCP_EP_FLAG_PATH_DEGRADED is set (src/ucp/proto/proto_multi.inl:70), which this UD-only, SW-emulated-PUT configuration is unlikely to hit — so the evidence points away from the PR, but it cannot be 100 % excluded without the failure text.

Implicated commit: No regression commit identified; the PR head under test is [REDACTED:Hex High Entropy String] "UCP: Split GET by the put samples of the peer" (Dmitrii), which touches src/ucp/proto/proto_multi.c and src/ucp/core/ucp_worker.c but is gated off for this test's single/degraded-less lane configuration.

File: test/gtest/ucp/test_ucp_peer_failure.cc:395 and :421-432 (test body at :601-610); PR code under suspicion: src/ucp/proto/proto_multi.c:114-126

Suggested fix: 1) Re-run the "AddressSanitizer roce" job to confirm flakiness (a single failure out of 9130 with no sanitizer report strongly suggests a race, not a build regression). 2) If it reproduces on this branch, capture the [ FAILED ] block (full gtest output / set GTEST_FILTER=*test_ucp_peer_failure.zcopy* with UCX_LOG_LEVEL=debug) to see which EXPECT fired, and check whether ucp_proto_rma_adjust_lane is being entered (print UCP_EP_FLAG_PATH_DEGRADED and mpriv->num_lanes) — note that returning NULL from ucp_proto_rma_split_lane (proto_multi.c:116) to skip a lane must still allow the request to complete/abort on peer failure. 3) Independently harden the test: scale the UD peer timeout for sanitizer builds (e.g. keep the default or use a larger value when RUNNING_ON_VALGRIND/ASAN) and relax :432 to EXPECT_TRUE(UCS_STATUS_IS_ERR(status)) plus an allowed set {m_err_status, UCS_ERR_CANCELED, UCS_ERR_ENDPOINT_TIMEOUT} instead of exact equality with m_err_status.

Related: #6762 (udx/test_ucp_peer_failure.zcopy/3 failure), #6203 (dcx/test_ucp_peer_failure.zcopy/3 failure), #12335 (PR under test), #12334, #12362

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id ba808dc4-a045-40a0-83fe-eeab97b3d089 in the triage console for the audit trail.

@dmitrygx
dmitrygx force-pushed the topic/ucx/dlb-get-balance-v2 branch from a6210be to 8b37b28 Compare November 18, 2026 05:56
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

@svc-nvidia-pr-review

Copy link
Copy Markdown

src/ucp/core/ucp_worker.c

ucp_worker_ep_path_bw_get() frees the stale state and allocates a new one on any local configuration/lane-generation change (ucp_ep_set_lane() bumps lane_generation on every lane replacement, so a failover or port-speed change is enough). That also discards UCP_EP_PATH_BW_OP_PEER_PUT. Setting state->sync_peer = 1; covers the mirror direction (this side re-sends its own samples), but there is no pull: the peer only sends UCP_AM_ID_EP_PATH_BW from ucp_worker_ep_path_bw_send(), which runs only from ucp_worker_iface_path_bw_progress(), i.e. only on a UCT_EP_EVENT_PATH_SPEED_CHANGE on the peer. A path that is already degraded generates no new event, so after a local reconfiguration GET silently returns to the static split for as long as the peer's path stays degraded. Impact: the GET-side benefit of the feature is lost permanently after the first local lane change, which is exactly the situation (failover, speed change) where it matters. Can we request a refresh from the peer when the state is recreated?

src/ucp/proto/proto_multi.c

In ucp_proto_rma_split_lane() (first lane loop): this function runs from ucp_proto_put_offload_zcopy_send_func() / ucp_proto_get_offload_zcopy_send_func() / the SGL send funcs, i.e. once per fragment. It walks mpriv->num_lanes twice plus one extra ucp_proto_rma_lane_share(), and each of those does a floating-point division (state->bandwidth[lane][op] / lpriv->bandwidth), on top of ucp_proto_multi_max_frag_ratio()'s division per lane. With MAX_RMA_LANES lanes and small chunks (MIN_RMA_CHUNK_SIZE can be a few KB) this is a notable per-fragment cost on the send path, and total/kept/max_frag_ratio/min_share do not change between fragments unless new samples arrive. Can we cache total/kept/max_frag_ratio/min_share on the request and refresh them only when the samples change?

Residual coverage gaps: no test covers losing the peer's PEER_PUT samples across a local reconfiguration (put_reconfig_resets_samples only checks the local PUT samples), so finding 1 is not exercised; and no test measures the per-fragment cost of the degraded split, so finding 2 would not show up as a regression.

Split the next PUT fragment by the static shares of the lanes, each
scaled down by the ratio of the path bandwidth sample of its lane to the
path estimate, so a slower path gets a smaller share, a recovered path
gets it back without endpoint reconfiguration, and healthy lanes keep
their shares of the static lane model. A lane whose share of the message
or fragment per round is below the minimal fragment is skipped. After
the configuration or lanes of the endpoint change, the static lane model
applies until the next path events, and the first fragment which finds
no samples of the current lanes clears the endpoint flag of a degraded
path under the async lock, so the next fragments do not look up the
samples. The endpoint flag keeps the send path of other endpoints
unchanged. The lane private data keeps the bandwidth of one path of the
lane, limited by the system distances like the bandwidth of the lane,
since the samples are compared with the path estimate, which is a share
of the device when it has several paths. The maximal fragment and the
weight are computed by the same helpers as in the static lane model; the
weight helper gets the share of the lane as a fraction of the weight
range, since the shares of nearly stalled paths are far below one. A
protocol passes its multi-lane private data to the adjustment, since the
private data of other protocols, such as rendezvous, has another layout.
The PUT reconfiguration check runs with a single-thread worker, whose
send does not hold the async lock; the force close test moves to a
fixture of its own on top of the single-thread one. The number of paths
of a mock device can differ between devices, so a split checks that a
sample scales the share of its lane only, also when a system distance
limits the bandwidth of the device. Samples of a few B/s check the split
of nearly stalled paths.
The data of GET from an endpoint crosses the paths on which the peer
endpoint puts, so the put samples of the peer must split GET of this
side. Check the split and its recovery, the split with two paths per
device, a stalled path of the peer, that the put samples of this side
do not split GET and the two directions degrade apart, that the samples
go to the peer after all endpoints update, and that the path estimates
are queried once per port speed.
The data of GET from an endpoint crosses the paths on which the peer
puts. The GET side samples these paths poorly: its probes find no room
in send queues full of its reads, which wait for read slots, and where
they run, they also see its own reads. The peer samples the same paths
from endpoints without application work requests.

After the put samples of an endpoint are updated, send them to the peer
in a UCP_AM_ID_EP_PATH_BW message: per path, the ratio of the sample to
the path estimate, with the memory domains and system devices of both
sides. The peer scales the path estimate of get on its lanes of these
paths by the ratio, keeps the result as a second source of samples, and
splits GET fragments by it with the same lane adjustment as PUT. A
degraded path is sent on every update, so the peer gets it again after
its endpoint reconfiguration, and its recovery once. Only peers of a
release which handles the message get it.

The samples are sent from progress, after the update of all endpoints,
since a failed send can invoke the error callback of the user, which may
close endpoints. An endpoint keeps the sources of its degraded samples
next to its flag, so an operation looks up the samples only when its own
source is degraded. The path estimates of an interface are queried once
per port speed.
@dmitrygx
dmitrygx force-pushed the topic/ucx/dlb-get-balance-v2 branch from 8b37b28 to 9655ef9 Compare November 18, 2026 07:01
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

double path_ratio;

/* Single path bandwidth, without the limits of the system distances */
double path_bandwidth;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minor: pls also set path_bandwidth in ucp_proto_common_tl_perf_reset(). That function is the NULL-lane path of ucp_proto_common_get_lane_perf() and already misses path_ratio; now ucp_proto_multi_init_priv() reads path_bandwidth into lpriv->bandwidth, so the field should not be left uninitialized.

Comment thread src/ucp/rma/put_offload.c
uct_iov_t iov;
ucs_status_t status;

lpriv = ucp_proto_rma_lane_priv(req, req->send.proto_config->priv, lpriv,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the same adjust-and-skip block now appears 4 times (here, ucp_proto_put_sgl_offload_send_func, ucp_proto_put_sgl_offload_sw_send_func, ucp_proto_get_offload_zcopy_send_func). can we fold it into one helper/macro next to ucp_proto_rma_lane_priv()? the root AGENTS.md asks not to duplicate similar implementation patterns.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

WIP-DNM Work in progress / Do not review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants