Skip to content

UCT/IB/MLX5: support am zcopy in purge outstanding - #11873

Merged
gleon99 merged 4 commits into
openucx:masterfrom
jeynmann:failover_uct_rcx_am_zcopy
Oct 13, 2026
Merged

gleon99 merged 4 commits into
openucx:masterfrom
jeynmann:failover_uct_rcx_am_zcopy

Conversation

@jeynmann

@jeynmann jeynmann commented Sep 1, 2026 •

Copy link
Copy Markdown
Contributor

What?

Add wqe parse support for uct am zcopy.

Will rebase after PR#12058 is merged

Why?

FT need extract info from wqe before replay uct am zcopy.

How?

Add parser for uct am zcopy.

@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread src/uct/ib/mlx5/rc/rc_mlx5_common.c Resolved
Comment thread src/uct/ib/mlx5/rc/rc_mlx5_common.c Resolved
Comment thread src/uct/ib/mlx5/rc/rc_mlx5_common.c Resolved
@svc-ucx

svc-ucx commented Sep 1, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR (Codestyle commit title) · commit 88e5bc5a

TL;DR: The "Codestyle / commit title" check failed because one commit in PR #11873 is titled GTEST/UCT/IB: test am zcopy parser with two spaces after the colon, which the title regex rejects; amend that commit message to use a single space.

Full analysis

Summary: Azure Pipelines job "UCX PR (Codestyle commit title)" build 133712 exited with code 1 because codestyle_check_commit_title flagged one of the PR's four commit titles as malformed.

Root cause: The check script iterates over the PR commits and validates each subject line against the regex ^[0-9A-Z/_-]+:\ [[:alnum:]_] — i.e. an uppercase module prefix, a colon, exactly one space, then an alphanumeric/underscore character. Log output:

Good commit title: 'UCT/IB/MLX5: log level'
Good commit title: 'UCT/IB/MLX5: cleanup'
Bad  commit title: 'GTEST/UCT/IB:  test am zcopy parser'
Good commit title: 'UCT/IB/MLX5: am zcopy parser'
##[error]Bash exited with code '1'.

GTEST/UCT/IB: is a valid prefix, but there are two spaces after the colon, so the character following the single required space is another space rather than [[:alnum:]_], and the match fails. This is a legitimate style violation in the PR's commit history, not a CI infrastructure problem — the other three commits from the same branch pass.

Implicated commit: The PR commit titled GTEST/UCT/IB: test am zcopy parser (third of four on branch failover_uct_rcx_am_zcopy, PR head [REDACTED:Hex High Entropy String]). Its individual SHA is not printed in the log (the script only echoes titles). The check logic itself is unchanged and correct — nearest related infra commit is cfd9cd1 "BUILD/CI: Limit PR commit range checks (#11527)" by Yossi Itigin, which only narrowed the commit range.

File: buildlib/tools/codestyle.sh:14 (the validating regex); the defect is in the PR's commit message, not this file.

Suggested fix: Rewrite that commit's subject with a single space after the colon, e.g. GTEST/UCT/IB: test am zcopy parser. Since it is not the tip commit, use an interactive rebase and force-push:

git rebase -i HEAD~4        # mark 'GTEST/UCT/IB:  test am zcopy parser' as "reword"
# change title to: GTEST/UCT/IB: test am zcopy parser
git push --force-with-lease

No change to buildlib/tools/codestyle.sh is warranted. If maintainers want to be lenient about extra whitespace, the regex could be relaxed to ^[0-9A-Z/_-]+:[[:space:]]+[[:alnum:]_], but the current strictness appears intentional to keep titles uniform.

Related: PR #11873; commit-range scoping change #11527

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 85f770cf-bd82-4cad-bc56-c6f4fc932c78 in the triage console for the audit trail.

@jeynmann
jeynmann force-pushed the failover_uct_rcx_am_zcopy branch from 88e5bc5 to 3db1c48 Compare September 4, 2026 09:08
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

@svc-ucx

svc-ucx commented Sep 4, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR (Codestyle commit title) · commit 3db1c488

TL;DR: The "commit title" codestyle check failed because one commit in PR #11873 has a double space after the colon — GTEST/UCT/IB: test am zcopy parser — which fails the title regex; reword/amend that commit title to use a single space.

Full analysis

Summary: Azure Pipelines job "UCX PR (Codestyle commit title)" build 134323 exited with code 1 after reporting Bad commit title: 'GTEST/UCT/IB: test am zcopy parser'.

Root cause: The check iterates over the PR's commits and validates each subject line against the regex ^[0-9A-Z/_-]+:\ [[:alnum:]_] in codestyle_check_commit_title(). That pattern allows exactly one space after the <MODULE>: prefix and requires the next character to be alphanumeric or _. The commit title GTEST/UCT/IB: test am zcopy parser contains two spaces after the colon, so the character following the single matched space is another space, the match fails, err=1 is set, and the Bash task exits non-zero. All other six commits in the PR passed (Good commit title: ...), confirming this is a single malformed title rather than a tooling/infra problem. This is a real code-review issue in the PR branch, not a CI flake.

Implicated commit: The PR commit titled GTEST/UCT/IB: test am zcopy parser on branch failover_uct_rcx_am_zcopy (PR head [REDACTED:Hex High Entropy String], merge commit e52088c). Exact SHA of the offending commit is not printed by the script — author is the PR author.

File: buildlib/tools/codestyle.sh:14 (the validating regex); the defect itself is in the commit message, not in tracked source.

Suggested fix: Rewrite the offending commit title to remove the extra space, e.g. GTEST/UCT/IB: test am zcopy parser, then force-push:

git rebase -i [REDACTED:Hex High Entropy String]   # mark that commit as 'reword'
# change title to: GTEST/UCT/IB: test am zcopy parser
git push --force-with-lease

Optionally, to make this class of failure more obvious, extend the script to print why a title was rejected (e.g. echo the expected format ^[0-9A-Z/_-]+: <alnum> alongside the "Bad commit title" message) so contributors don't have to reverse-engineer the regex.

Related: #11873

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id ac569af1-88f3-4994-8b01-7c85b632ed04 in the triage console for the audit trail.

Comment thread src/uct/ib/mlx5/rc/rc_mlx5_common.c Resolved
Comment thread src/uct/ib/mlx5/rc/rc_mlx5_common.c Resolved
Comment thread src/uct/ib/mlx5/ib_mlx5.c Resolved
@svc-nvidia-pr-review

Copy link
Copy Markdown

Test coverage gap: the PR adds uct_rc_mlx5_op_info_fill and helpers but includes no caller and no test exercising the AM zcopy op-info path. If a caller/test lands in a follow-up, that's worth stating in the PR description; otherwise the new code is entirely untested/dead in this PR.

@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

@svc-ucx

svc-ucx commented Sep 4, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR (Codestyle commit title) · commit 85f2e7c2

TL;DR: This is not a code/test failure — the "commit title" codestyle check rejected one commit in PR #11873 because its title GTEST/UCT/IB: test am zcopy parser has a double space after the colon; reword that commit to use a single space and force-push.

Full analysis

Summary: Azure Pipelines job "UCX PR (Codestyle commit title)" build 134345 failed with ##[error]Bash exited with code '1' after the commit-title linter printed Bad commit title: 'GTEST/UCT/IB: test am zcopy parser'.

Root cause: The commit-title validation script iterates over every commit in the PR range and matches each subject against the UCX title convention (<MODULE>[/<SUBMODULE>]: <single space><description>). Every other commit in the branch passed:

Good commit title: 'UCT/IB/MLX5: address comments'
Good commit title: 'UCT/IB/MLX5: address comments'
Good commit title: 'GTEST/UCT/IB: rm overlap gtests'
Good commit title: 'UCT/IB/MLX5: cleanup'
Good commit title: 'UCT/IB/MLX5: log level'
Good commit title: 'UCT/IB/MLX5: cleanup'
Bad commit title: 'GTEST/UCT/IB:  test am zcopy parser'
Good commit title: 'UCT/IB/MLX5: am zcopy parser'

The only difference between the passing 'GTEST/UCT/IB: rm overlap gtests' and the failing 'GTEST/UCT/IB: test am zcopy parser' is the extra space after the : separator, which breaks the expected : + description pattern. The script exits non-zero on the first bad title, which surfaces as the Bash task failure. There is no build, compile, or test problem here — the checkout and the whole job take under 2 seconds of actual work (12:29:58.63 → 12:29:58.70), so this is a fast, deterministic lint rejection, not a hang or timeout.

Implicated commit: The offending commit is the one titled GTEST/UCT/IB: test am zcopy parser in branch failover_uct_rcx_am_zcopy (second-oldest commit in the PR series; PR head at the time was [REDACTED:Hex High Entropy String], tested as merge commit [REDACTED:Hex High Entropy String]). This is a PR-authored commit, not a regression from upstream.

File: No source file is at fault — the failure is in the git commit metadata of the PR branch. The check itself runs as the commit title Bash task in the Azure Codestyle stage (buildlib commit-title check; exact path not resolvable in this checkout, read_source_file returned "not found" for buildlib/tools/commit_title.sh, buildlib/commit-format.sh, buildlib/az-check-commit-title.yml, buildlib/pr/az-check-commit-title.yml, and contrib/check_commit_title.sh).

Suggested fix: Reword the single bad commit title to remove the duplicated space:

git rebase -i [REDACTED:Hex High Entropy String]
# mark the 'GTEST/UCT/IB:  test am zcopy parser' commit as 'reword'
# change the title to: GTEST/UCT/IB: test am zcopy parser
git push --force-with-lease origin failover_uct_rcx_am_zcopy

Alternatively, since this PR is marked [WIP][DNM] and has several cleanup / address comments fixup commits, squashing the series into one or two properly-titled commits will clear this check and tidy the history at the same time. No infrastructure change or retry is warranted — a rerun will fail identically until the title is fixed.

Related: PR #11873 — [WIP][DNM] UCT/IB/MLX5: wqe parser for uct am zcopy

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 799f1b51-9dee-46e3-840c-d0f1c9737343 in the triage console for the audit trail.

@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Review complete — no issues found.

@jeynmann
jeynmann force-pushed the failover_uct_rcx_am_zcopy branch from 85f2e7c to 539e89e Compare September 7, 2026 07:49
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread src/uct/ib/mlx5/rc/rc_mlx5_common.c Resolved
Comment thread src/uct/ib/mlx5/rc/rc_mlx5_common.c Resolved
@svc-nvidia-pr-review

Copy link
Copy Markdown

Test/scope (already reported, restating for completeness): there is still no caller or test for uct_rc_mlx5_op_info_fill in this PR, so the new AM-zcopy path is untested/dead here. This was raised in the prior review; if a caller/test is coming in a follow-up, that should be stated in the PR description.

@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Review complete — no issues found.

@svc-ucx

svc-ucx commented Sep 7, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR (Tests roce on worker 0) · commit 119f81b9

TL;DR: The roce gtest job did not fail an assertion first — it hung: rc/test_ucp_wireup_1sided.disconnect_nb_onesided/0 <rc_v/rma> posted 1000 ucp_put_nbs and then its ucp_ep_flush_nbx never completed (3-minute dead silence, repeated "did not complete on time"), cascading into the data-validation failure, the un-freed EP close request and finally Connection timed out - abort testing (SIGABRT). The stall is in the rc_verbs/RoCE RMA flush path, which this PR does not touch, so it should be re-run and, if reproducible, chased as an rc_v flush/pending stall rather than as a PR regression.

Full analysis

Summary: gtest make test aborted after rc/test_ucp_wireup_1sided.disconnect_nb_onesided/0 <rc_v/rma> hung waiting for RMA put/flush completions on the RoCE worker (swx-rain04), exit via test_helpers.cc:57 "Connection timed out - abort testing".

Root cause: A hang, not slowness. Log timestamps show continuous ~400 ms test cadence until 10:08:45.14 ([ RUN ] ... disconnect_nb_onesided/0 <rc_v/rma>), then a 3-minute gap with zero output to 10:11:45.75, where the first ucp_test.cc:300 UCX ERROR request 0x...298 did not complete on time appears. That request is the ucp_ep_flush_nbx issued in test_ucp_wireup::send_nb() right after the 1000 ucp_put_nb calls (test_ucp_wireup.cc:287-302). Every subsequent step then failed as a cascade, each burning another 3-minute deadline:

  • ucp_test.cc:897 — "free not completed EP close request" (the disconnect_nb() close request never completed).
  • test_ucp_wireup.cc:361 — recv_data vs data: 0, "invalid data at index 1000 (control)": the control word m_recv_data[length] was still 0, i.e. no RMA data ever landed.
  • Two more requests_wait() timeouts (10:17:56, 10:20:57), then the global test deadline aborted the binary (Caught signal 6, backtrace through ucp_test::requests_wait → test_ucp_wireup.cc:748).

Notably there is no send-completion-with-error, no IB async event, no retry-exceeded error in the log — the QP produced no completions at all, consistent with the puts sitting un-progressed (pending/wireup) rather than a fabric error being reported.

Relation to the PR: this variant is rc_v (rc_verbs + ud_verbs). All seven commits on the branch (4b8ed2d3 am zcopy parser, a6e856fd, 37cf9f65, 2dc0d149, 83944d53, 539e89e7, 119f81b9, all by Zihao Zhao) touch only src/uct/ib/mlx5/rc/rc_mlx5_common.c, src/uct/ib/mlx5/ib_mlx5.c and src/uct/ib/mlx5/ib_mlx5_log.c — mlx5-accelerated code (uct_rc_mlx5_op_info_fill_am_zcopy / WQE parsing for op-info extraction). Nothing in src/uct/ib/rc/verbs, src/uct/ib/rc/base or src/ucp is modified, and the mlx5 tests earlier in the same run (dc_mlx5, rcx, gga_mlx5) all passed. So the hung path is not exercised by the PR's changes.

Implicated commit: unknown — not 119f81b9 (PR commits are mlx5-only; the hang is on the rc_verbs RMA path). Candidate pre-existing suspects on master for an unfinished flush are the failover/flush work 75611772 (UCP/FT: EP flush failover) and db208ee4 (UCP/FT: probe-gated lane recovery via aux uct_ep_check, which also touched this very test file), but the log contains no direct evidence for either.

File: test/gtest/ucp/test_ucp_wireup.cc:300 (the hung ucp_ep_flush_nbx wait, reached from test_ucp_wireup.cc:737/748); timeout reported at test/gtest/ucp/ucp_test.cc:300

Suggested fix:

  1. Re-run the roce on worker 0 job on the same commit — this is almost certainly not a regression from the mlx5 AM-zcopy parser change, and a clean re-run will unblock the PR.
  2. If it reproduces, treat it as an rc_verbs RMA flush stall, not a timeout to be raised: re-run just that test with UCX_LOG_LEVEL=debug, UCX_PROTO_INFO=y and --gtest_filter='rc/test_ucp_wireup_1sided.disconnect_nb_onesided/0', and on the hang attach gdb / use ucs_ep_pending dumps to see whether the 1000 puts are stuck in the UCP pending queue (wireup lane not yet ready / flush state not advanced) or posted with no CQE.
  3. Also check swx-rain04's RoCE port for link/PFC problems around 10:08 UTC — the complete absence of any error CQE while 8 MB of puts never complete is equally consistent with a stuck NIC/fabric on that agent.
  4. Do not raise the gtest deadline: the largest inter-line gap (180 s, the request deadline itself) proves the process was idle, not making progress.

Related: PR #11873 ("[WIP][DNM] UCT/IB/MLX5: wqe parser for uct am zcopy"); no existing issue found for this hang (test_ucp_wireup_1sided.disconnect_nb_onesided / rc_v flush timeout) — worth opening one if it recurs.

@jeynmann jeynmann changed the title [WIP][DNM] UCT/IB/MLX5: wqe parser for uct am zcopy [DNM] UCT/IB/MLX5: wqe parser for uct am zcopy Sep 9, 2026
@jeynmann
jeynmann force-pushed the failover_uct_rcx_am_zcopy branch from 119f81b to 957f00e Compare September 11, 2026 03:13
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread src/uct/ib/mlx5/rc/rc_mlx5_common.c Resolved
Comment thread src/uct/ib/mlx5/rc/rc_mlx5_common.c Resolved
@svc-nvidia-pr-review

Copy link
Copy Markdown

test/gtest/uct/test_peer_failure.cc

blocker: test_purge_outstanding(am_zcopy, &ctx); passes purge_ctx* to a purge_ctx& parameter (the signature was changed to test_purge_outstanding(const send_func_t&, purge_ctx&) in the same hunk). This does not compile. Suggested fix: test_purge_outstanding(am_zcopy, ctx);

test/gtest/uct/test_peer_failure.cc

blocker: UCT_INSTANTIATE_TEST_CASE(test_uct_purge_outstanding, am_zcopy) — UCT_INSTANTIATE_TEST_CASE(_test_case) (test/gtest/uct/uct_test.h:528) takes exactly one argument; the two-argument form is _UCT_INSTANTIATE_TEST_CASE(_test_case, _tl_name) and the second argument is a TL name, not a test name. Preprocessor error → build break. Should be UCT_INSTANTIATE_TEST_CASE(test_uct_purge_outstanding).

test/gtest/uct/test_peer_failure.cc

minor: recvbuf (and therefore RECV_SEED) is never used by the AM zcopy test — AM needs no remote registration, pls remove.

Coverage note: the new gtest only exercises uct_rc_mlx5_op_info_fill when the ib_mlx5_ext plugin providing ep_outstanding_purge is present (rc_mlx5_iface.c:1242); in a plain in-tree build the test skips on the UCT_IFACE_FLAG_V2_QUERY_TOKEN check, so upstream CI will not cover the new AM-zcopy parsing path.

@svc-ucx

svc-ucx commented Sep 11, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR (Build build_aarch64 rhel100_aarch64) · commit 957f00e7

TL;DR: The aarch64 build failed on a plain C++ compile error in the gtest sources added by this PR: UCT_INSTANTIATE_TEST_CASE was invoked with two arguments at test/gtest/uct/test_peer_failure.cc:656, which leaves the macro unexpanded and destroys the parse of everything after it (hence the flood of errors at line 772). Change it to UCT_INSTANTIATE_TEST_CASE(test_uct_purge_outstanding) and fix the test_purge_outstanding(am_zcopy, &ctx) argument type.

Full analysis

Summary: make failed building test/gtest/uct/gtest-test_peer_failure.o (make[3]: *** [Makefile:2475: uct/gtest-test_peer_failure.o] Error 1) in job build_aarch64 rhel100_aarch64.

Root cause: Compilation error introduced by the PR's new gtest code, not an infrastructure/timeout issue:

  1. test/gtest/uct/test_peer_failure.cc:656 calls UCT_INSTANTIATE_TEST_CASE(test_uct_purge_outstanding, am_zcopy), but the macro is defined with exactly one parameter (test/gtest/uct/uct_test.h:528: #define UCT_INSTANTIATE_TEST_CASE(_test_case)). GCC reports the arity mismatch and leaves the invocation unexpanded, so UCT_INSTANTIATE_TEST_CASE(test_uct_purge_outstanding, am_zcopy) is parsed as a stray declaration at file scope with a missing ;. That aborts the parse of the immediately following class test_uct_peer_failure_multiple (line 658), which is exactly why the log shows, at line 772, ‘test_uct_peer_failure_multiple’ was not declared in this scope; did you mean ‘gen_tcptest_uct_peer_failure_multiple’?, parse error in template argument list, ‘info’ was not declared in this scope, and no matching function for call to ... GetTestSuitePatternHolder<<expression error>>. All the visible errors are downstream fallout of line 656.
  2. A second, independent type error in the same new test: line 653 calls test_purge_outstanding(am_zcopy, &ctx); but the method is declared as void test_purge_outstanding(const send_func_t &send_func, purge_ctx &ctx) (line 576) — a purge_ctx* cannot bind to purge_ctx&. (Contrast with purge_outstanding(&ctx) at line 603, which does take a pointer.)

Implicated commit: 30bb3b7e — "GTEST/UCT/IB: add test for am zcopy", Zihao Zhao (branch tip commit [REDACTED:Hex High Entropy String], PR #11873)

File: test/gtest/uct/test_peer_failure.cc:656 (primary) and test/gtest/uct/test_peer_failure.cc:653

Suggested fix:

  • Line 656: drop the extra argument — UCT_INSTANTIATE_TEST_CASE(test_uct_purge_outstanding). The test name (am_zcopy) is already bound by UCS_TEST_SKIP_COND_P(test_uct_purge_outstanding, am_zcopy, ...) at line 623; the instantiate macro only takes the test-case class. If the intent was to restrict instantiation to a transport subset, use one of the dedicated macros (UCT_INSTANTIATE_IB_TEST_CASE, UCT_INSTANTIATE_NO_SELF_TEST_CASE, etc. in uct_test.h:540-597), each of which also takes exactly one argument.
  • Line 653: pass by reference — test_purge_outstanding(am_zcopy, ctx); (or change the signature to purge_ctx *ctx for consistency with purge_outstanding).
  • Build test/gtest locally (make -C test/gtest) before re-pushing; this error is architecture-independent and will fail every build job, so aarch64 is just the first one reported.

Related: PR #11873 (this PR); base infrastructure from #11891 "GTEST/UCT: Add outstanding purge test base" (commit 61104007, Roie Danino) which added test_uct_purge_outstanding and the test_purge_outstanding(const send_func_t&, purge_ctx&) signature.

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id a761f83a-2a9e-42a8-a213-e72638fe99ab in the triage console for the audit trail.

@jeynmann
jeynmann force-pushed the failover_uct_rcx_am_zcopy branch from 957f00e to c061cf2 Compare September 11, 2026 04:29
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

@jeynmann jeynmann changed the title [DNM] UCT/IB/MLX5: wqe parser for uct am zcopy UCT/IB/MLX5: wqe parser for uct am zcopy Sep 11, 2026
Comment thread src/uct/ib/mlx5/rc/rc_mlx5_common.c Resolved
Comment thread src/uct/ib/mlx5/rc/rc_mlx5_common.c Resolved
Comment thread src/uct/ib/mlx5/rc/rc_mlx5_common.c Resolved
Comment thread src/uct/ib/mlx5/rc/rc_mlx5_common.c Resolved
@svc-nvidia-pr-review

Copy link
Copy Markdown

Residual coverage gap (already raised in earlier reviews, not repeated inline): the new gtest only exercises this code when the out-of-tree plugin providing ep_outstanding_purge is loaded, so upstream CI does not cover the AM-zcopy parsing path.

@svc-ucx

svc-ucx commented Sep 11, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR (Tests new on worker 1) · commit c061cf21

TL;DR: The gtest run failed on a single, unrelated test — ud_mlx5/test_ud.stale_private_ep_reuse/0 — a newly added, timing-sensitive UD private-EP keepalive test that is already known to be flaky (open PR #11920 "Hardening of UD stale private EP tests"). Re-run the job; the real fix is to land/merge the test-hardening in #11920, not to change anything in PR #11873.

Full analysis

Summary: make test in build-test/test/gtest exited with Error 1 because 1 of 8608 tests failed: [ FAILED ] ud_mlx5/test_ud.stale_private_ep_reuse/0, where GetParam() = ud_mlx5/mlx5_0:1.

Root cause: Evidence from the log: 8607 tests passed, exactly one failed, the run completed normally (no SIGTERM/timeout, continuous timestamps up to Global test environment tear-down at 06:05:52 → summary at 06:05:57), and make: *** [Makefile:4713: test] Error 1 is purely the gtest non-zero exit.

The failing test is test_ud.stale_private_ep_reuse (test/gtest/uct/ib/test_ud.cc:935-970). It is a wall-clock/keepalive-timing dependent test: it runs with UD_LINGER_TIMEOUT=1s / UD_TIMEOUT=1s and asserts that a stale private UD EP on m_e2 is garbage-collected after its peer EP on m_e1 disappears. The wait helper ud_base_test::wait_for_ep_destroyed() (test/gtest/uct/ib/ud_base.cc:98-104) polls with a 60 s budget and, critically, does not assert on timeout — it silently returns, so the subsequent EXPECT_FALSE(ucs_ptr_array_lookup(&iface(m_e2)->eps, private_index, ud_ep)) (line 960) / EXPECT_TRUE(find_private_ep(...)) (line 963) fail whenever keepalive-driven release hasn't converged yet on a heavily loaded CI worker. This whole test + the keepalive-based release path are brand new (commit 183a66db "UCT: Keep-alive for private UD EPs", Ilia Yastrebov, 2026-08-24, plus 0ee470a8 generation-ID and 8a38ae21 "Remove from CEP early"), and there is already an open PR acknowledging this exact instability: #11920 "UCT/GTEST: Hardening of UD stale private EP tests".

Crucially, PR #11873 is "UCT/IB/MLX5: wqe parser for uct am zcopy" — it touches the RC/mlx5 WQE parsing path and has no interaction with the UD transport's private-EP/keepalive lifecycle. So this is a pre-existing flaky failure, not a regression from the PR under test.

Caveat: the per-test gtest assertion text is not recoverable from this build — the fetched log is only the tail (~last 500 lines) of a ~60-minute shuffled run, and the test itself executed much earlier. The diagnosis above is based on the failure summary line, the test source, and the existing hardening PR.

Implicated commit: 183a66db — "UCT: Keep-alive for private UD EPs", Ilia Yastrebov (test added by 2d550625 / 183a66db; not c061cf21 from PR #11873)

File: test/gtest/uct/ib/test_ud.cc:935-963 (assertions at 956-963), with the non-asserting timeout helper at test/gtest/uct/ib/ud_base.cc:98-104

Suggested fix:

  1. Unblock PR UCT/IB/MLX5: support am zcopy in purge outstanding #11873 now: re-run the "Tests new on worker 1" job. This failure is unrelated to the WQE-parser change; do not modify the PR.
  2. Real fix — harden the test (align with / merge PR UCT/GTEST: Hardening of UD stale private EP tests #11920):
    • Make wait_for_ep_destroyed() fail loudly instead of silently timing out, e.g. ASSERT_TRUE(...) on the wait result, so a timeout is distinguishable from a genuine logic bug and the log names which wait expired.
    • Replace the fixed 1 s UD_LINGER_TIMEOUT/UD_TIMEOUT with values scaled by ucs::test_time_multiplier(), since these timeouts are too tight for a loaded/shared CI worker running 8608 tests.
    • Wait for m_e1's old EP to be destroyed before asserting on m_e2's stale private EP, so the keepalive on m_e2 is guaranteed to observe a dead peer.
  3. If the flake persists after hardening, consider gating the test on a non-shared/serialized worker rather than the general gtest sweep.

Related: #11920 (UCT/GTEST: Hardening of UD stale private EP tests — the actual fix); #11873 (PR under test, unrelated); #11800 (UCT/UD: generation ID in dest_id, same feature area)

@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

@svc-nvidia-pr-review

Copy link
Copy Markdown

src/uct/ib/mlx5/rc/rc_mlx5_ep.c

uct_ep_am_zcopy() with comp == NULL creates no send op, and a zero-length payload (iovcnt == 0 or all-zero-length iov entries) leaves no data segment, so such a WQE is parsed here as am_short - the reported header/payload come from the user's zcopy header, or the am_short path returns UNSUPPORTED and the purge loop fatals (ucs_fatal("failed to parse outstanding WQE")). put_zcopy avoids this ambiguity by posting an always-op (uct_rc_txqp_add_send_comp_always()); can we add an always-op for am_zcopy too, like uct_rc_mlx5_base_ep_put_zcopy() does?

test/gtest/uct/test_peer_failure.cc

minor: payload size does not leave room for the AM header. cap.am.max_zcopy is seg_size - sizeof(uct_rc_mlx5_hdr_t), while uct_ep_am_zcopy() validates header_length + length <= seg_size (UCT_RC_CHECK_ZCOPY_DATA). With the default 8K segment min(4096, max_zcopy) is fine, but for any seg_size <= ~4160 the send returns UCS_ERR_INVALID_PARAM and post_op()'s ASSERT_UCS_OK_OR_INPROGRESS fails. Suggestion:

    const size_t size            = ucs_min((size_t)4096,
                                           attr.cap.am.max_zcopy - hdr_size);

The same line also breaks the assignment-column rule (docs/CodeStyle.md, "Indent multiple consecutive assignments on the column").

@jeynmann
jeynmann force-pushed the failover_uct_rcx_am_zcopy branch from 2ebd0b8 to 4efd673 Compare October 8, 2026 03:02
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

@svc-ucx

svc-ucx commented Oct 8, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR (AddressSanitizer roce on worker 2) · commit 4efd6737

TL;DR: The ASan gtest shard aborted because ud_mlx5/uct_p2p_mix_test.mix_10000/0 hit a UD endpoint timeout (UCS_ERR_ENDPOINT_TIMEOUT, -80) after ~31 s — the default UD_TIMEOUT=30s elapsed while the test spun in a zcopy completion loop under AddressSanitizer on a shared RoCE node; because the test registers no error handler, the timeout escalated to ucs_fatal and a core dump. This is an environment/test-hardening issue in the UD path, unrelated to the PR's RC-mlx5 changes.

Full analysis

Summary: ud_mlx5/uct_p2p_mix_test.mix_10000/0 failed with "Error: Endpoint timeout" at test_p2p_mix.cc:33, followed by ud_ep.c:293 Fatal: UD endpoint ... unhandled timeout error, aborting make -C test/gtest test (shard 3 of 4) with Bash exited with code '2'.

Root cause: Evidence from the log:

  • [ RUN ] ud_mlx5/uct_p2p_mix_test.mix_10000/0 starts at 03:59:43.20, fails at 04:00:14.81 — ~31.6 s, matching the UD peer_timeout default of 30s (src/uct/ib/ud/base/ud_iface.c:682, UD_TIMEOUT "30s").
  • The completion callback uct_p2p_mix_test::completion_callback (test/gtest/uct/test_p2p_mix.cc:33) received _status = -80 (UCS_ERR_ENDPOINT_TIMEOUT), i.e. the zcopy completion was released by uct_ud_ep_purge(ep, UCS_ERR_ENDPOINT_TIMEOUT).
  • random_op() (test_p2p_mix.cc:162-177) blocks unbounded in while (comp.count > 0) progress();. The run() 10 s wall-clock guard (test_p2p_mix.cc:210) is only checked between ops, so a stalled UD ACK pins the test until the 30 s UD peer timeout fires rather than exiting cleanly.
  • uct_p2p_mix_test is constructed as uct_p2p_test(0) with err_handler = NULL (test_p2p_mix.cc:16, uct_p2p_test.h:17), so uct_iface_handle_ep_err() returns non-OK and uct_ud_ep_deferred_timeout_handler calls ucs_fatal("UD endpoint ... unhandled timeout error") (src/uct/ib/ud/base/ud_ep.c:289-296) — turning a recoverable timeout into a core dump that kills the whole shard.
  • Context amplifying the stall: the binary runs with LD_PRELOAD=/lib/x86_64-linux-gnu/libasan.so.6 (log line at 03:59:38.01), and the EP peer is unresolved in the message (to :0), consistent with ACKs not being progressed in time on the shared swx-rain04 RoCE node rather than a logic error.

Notably, the PR's commit 4efd6737 ("UCT/IB/MLX5: Add WQE parser for am zcopy") touches only src/uct/ib/mlx5/rc/rc_mlx5_ep.c (RC transport); the failing transport is ud_mlx5, which does not use that code path — src/uct/ib/mlx5/ib_mlx5.c, ib_mlx5.h and ib_mlx5.inl show no commit 4efd6737 in their history. So the failure is not caused by this PR's diff.

Implicated commit: unknown (not caused by [REDACTED:Hex High Entropy String] by Zihao Zhao, which only touches the RC mlx5 path); the fatal-on-timeout behavior predates it

File: test/gtest/uct/test_p2p_mix.cc:33 (and the unbounded loop at test/gtest/uct/test_p2p_mix.cc:162-177); escalation at src/uct/ib/ud/base/ud_ep.c:293

Suggested fix:

  1. Re-trigger the job — this is an infrastructure-induced UD timeout, not a code regression from this PR.
  2. To stop this flake from core-dumping the whole shard, harden the test rather than the library:
    • Bound the completion wait in random_op() with a deadline (e.g. reuse wait_for_cond/get_deadline) and fail/skip gracefully instead of spinning until the UD peer timeout; and/or
    • Pass a non-NULL uct_error_handler_t to the uct_p2p_test base in uct_p2p_mix_test so uct_iface_handle_ep_err() returns UCS_OK and the UD timeout surfaces as a test failure instead of ucs_fatal + core dump.
    • Optionally raise UCX_UD_TIMEOUT for ASan gtest runs (ASan makes progress loops several times slower), consistent with how other timeouts are scaled via ucs::test_time_multiplier().

Related: #11920 UCT/GTEST: Hardening of UD stale private EP tests, #11948 same for v1.23, #11873 (this PR)

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 8d395393-6531-42e5-855e-2a770726fcb1 in the triage console for the audit trail.

@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

@jeynmann jeynmann changed the title UCT/IB/MLX5: wqe parser for uct am zcopy UCT/IB/MLX5: suppor am zcopy in purge outstanding Oct 8, 2026
@jeynmann jeynmann changed the title UCT/IB/MLX5: suppor am zcopy in purge outstanding UCT/IB/MLX5: support am zcopy in purge outstanding Oct 8, 2026
UCT/IB/MLX5: cleanup

GTEST/UCT/IB: update gtest

GTEST/UCT/IB: update test

GTEST/UCT/IB: refactor test

UCT/IB/MXL5: cleanup

UCT/IB/MLX5: address comments

UCT/IB/MLX5: address comments

UCT/IB/MLX5: address comments

UCT/IB/MLX5: address comments

UCT/IB/MLX5: address comments

UCT/IB/MLX5: address comments

UCT/IB/MLX5: address comments

UCT/IB/MLX5: address comments

UCT/IB/MLX5: address comments

UCT/IB/MLX5: address comments

GTEST/UCT/FT: add zcopy test

UCT/IB/MLX5: address comments

UCT/IB/MLX5: const wrap

UCT/IB/MXL5: const wrap

UCT/IB/MXL5: const wrap

UCT/IB/MLX5: am zcopy parser

GTEST/UCT/IB: test am zcopy parser

UCT/IB/MLX5: cleanup

UCT/IB/MLX5: log level

UCT/IB/MLX5: cleanup

GTEST/UCT/IB: rm overlap gtests

UCT/IB/MLX5: address comments

UCT/IB/MLX5: address comments

UCT/IB/MLX5: address comments

GTEST/UCT/IB: add test for am zcopy

GTEST/UCT/IB: fix test

GTEST/UCT/IB: cleanup

UCT/IB/MLX5: align inline size

UCT/IB/MLX5: const wrap

UCT/IB/MLX5: add assertions

UCT/IB/MLX5: add doc for copy segs

UCT/IB/MLX5: copy segs

UCT/IB/MLX5: refactor am zcopy parser

UCT/IB/MLX5: address comments

UCT/IB/MLX5: cleanup

UCT/IB/MLX5: cleanup

UCT/IB/MLX5: unsupport tag

GTEST/UCT/IB: cleanup

UCT/IB/MLX5: fix num packets

UCT/IB/MLX5: refactor

UCT/IB/MLX5: format

UCT/IB/MLX5: format

UCT/IB/MLX5: comp status

UCT/IB/MLX5: am segs helper

UCT/IB/MLX5: format and cleanup
@jeynmann
jeynmann force-pushed the failover_uct_rcx_am_zcopy branch from 2c5ce2d to 30571d3 Compare October 9, 2026 07:50
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment on lines +1007 to +1010
static int uct_rc_mlx5_send_op_is_am_zcopy(const uct_rc_iface_send_op_t *op)
{
return op->handler == uct_rc_ep_send_op_completion_handler;
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks too generic and unsafe, maybe smth like this:

.inl:

void UCS_F_ALWAYS_INLINE
uct_rc_ep_send_op_completion_inl(uct_rc_iface_send_op_t *op,
                                 const void *resp)
{
    uct_invoke_completion(op->user_comp, UCS_OK);
    uct_rc_iface_put_send_op(op);
}

.c:

void uct_rc_ep_send_op_completion_handler(uct_rc_iface_send_op_t *op,
                                          const void *resp)
{
    uct_rc_ep_send_op_completion_inl(op, resp);
}
...
void uct_rc_ep_send_op_am_zcopy_comp_handler(uct_rc_iface_send_op_t *op,
                                             const void *resp)
{
    uct_rc_ep_send_op_completion_inl(op, resp);
}

@svc-ucx

svc-ucx commented Oct 9, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR (AddressSanitizer roce on worker 0) · commit 30571d39

TL;DR: The ASAN/RoCE gtest run failed on a single test, dc_mlx5/uct_atomic_key_reg_rdma_mem_type.fadd64/0 (device mlx5_0:1), which is the device-memory (MEMIC / UCS_MEMORY_TYPE_RDMA) atomic fetch-add test; the PR's own HEAD commit 30571d39 "GTEST/UCT/IB: fix mem check" relaxed the memory-capability gating so this test now executes on a RoCE device where the UCS_MEMORY_TYPE_RDMA allocation/atomic path isn't viable. Fix: make check_rdma_memory() actually verify that device memory can be allocated and used for atomics (and that it is consistent with the IB_DM_COUNT=0 set in init()), rather than only checking the alloc_mem_types bit.

Full analysis

Summary: make test in the roce on worker 0 AddressSanitizer job exited with code 2 because 1 of 8815 gtests failed: dc_mlx5/uct_atomic_key_reg_rdma_mem_type.fadd64/0, where GetParam() = dc_mlx5/mlx5_0:1.

Root cause: Evidence from the log: the run completed normally (8815 tests from 1265 test suites ran, PASSED 8814, FAILED 1), with no timeout, no SIGTERM, no ASAN report, and no large gaps in the timestamps — so this is a genuine single-test assertion failure, not a hang or an infrastructure kill. The failing test is uct_atomic_key_reg_rdma_mem_type.fadd64, which is unique among the atomic tests in that it allocates its target buffer in device memory:

  • test/gtest/uct/test_atomic_key_reg_rdma_mem_type.cc:33 — mapped_buffer recvbuf(sizeof(uint64_t), receiver(), 0UL, UCS_MEMORY_TYPE_RDMA, UCT_MD_MEM_ACCESS_ALL, 10) (10 retries, i.e. this allocation is already known to be fragile).
  • It is gated only by check_rdma_memory() (:17-26), which merely tests md_attr().alloc_mem_types & UCS_BIT(UCS_MEMORY_TYPE_RDMA) — a capability bit, not an actual ability to allocate MEMIC and perform atomics on it.
  • Its init() (:11-15) sets IB_DM_COUNT=0, so the interface has no device-memory pool while the MD still advertises the RDMA alloc type — the gate and the runtime configuration disagree.

Only the /0 instance (mlx5_0:1, the RoCE port) failed; all other dc_mlx5 tests in the log ran on mlx5_1:1 and passed, and rc_mlx5/test_uct_ib_roce.local_subnet_only/0 <rc_mlx5/mlx5_0:1> passed — so this is specific to the device-memory path on the RoCE device, not general DC/RoCE breakage. The reachable failure points in this test are all hard asserts: ASSERT_UCS_OK(status) after the retry loop in uct_test::entity::mem_alloc() (test/gtest/uct/uct_test.cc:1007), UCS_TEST_ABORT(ucs_status_string(status)) in uct_amo_test::worker::run() (test/gtest/uct/test_amo.cc:188), or ASSERT_UCS_OK(status) in entity::flush() (uct_test.cc:1449) via wait_for_remote().

The branch's HEAD commit, 30571d39 "GTEST/UCT/IB: fix mem check" (Zihao Zhao, 2026-10-08), is a test-side change to memory checking in test/gtest/uct on exactly this PR branch, landing on top of the purge-outstanding series. A loosened/incorrect memory check is precisely what would let this previously-skipped device-memory test start running — and fail — on mlx5_0:1.

Caveat on evidence: the retained console tail covers only the last ~45 seconds of a ~45-minute run and the tests execute in shuffled order, so the assertion text for this test scrolled out; the final summary block is what identifies it. The mechanism above is inferred from the test source and the commit on the branch, not from a quoted assertion line.

Implicated commit: 30571d39 — "GTEST/UCT/IB: fix mem check", Zihao Zhao (PR #11873 HEAD). Prior context in the same series: 67c43670 (WQE parser for AM zcopy), 757f9f9b (#11866), 7f839dd3 (#11874), 8b378b5e (#11836).

File: test/gtest/uct/test_atomic_key_reg_rdma_mem_type.cc:17-34 (check_rdma_memory() and the UCS_MEMORY_TYPE_RDMA mapped_buffer), with the failing assert surfacing in test/gtest/uct/uct_test.cc:1007 / test/gtest/uct/test_amo.cc:188.

Suggested fix:

  1. Revert or tighten 30571d39's memory check. check_rdma_memory() should not rely on the alloc_mem_types bit alone — it should also require that atomics are actually supported on device memory and that the MD/iface configuration is consistent with the IB_DM_COUNT=0 applied in init(). Concretely, attempt a trial UCS_MEMORY_TYPE_RDMA allocation under a scoped_log_handler(wrap_errors_logger) and UCS_TEST_SKIP_R("device memory not available") on failure, instead of letting ASSERT_UCS_OK abort the test.
  2. Reconcile the IB_DM_COUNT=0 in init() with the gate — if the test genuinely needs MEMIC, don't disable the device-memory pool; if it doesn't, drop UCS_MEMORY_TYPE_RDMA.
  3. To confirm before fixing, re-run just this case with the assertion visible: GTEST_FILTER='dc_mlx5/uct_atomic_key_reg_rdma_mem_type.fadd64*' make -C build-test/test/gtest test on the RoCE worker, and attach the full output. Also worth a quick check that the new AM-zcopy WQE parser from 67c43670 handles MLX5_OPCODE_ATOMIC_FA WQEs, since the atomic path shares the txwq and uct_rc_txqp_add_send_comp machinery touched by this series.

Related: PR #11873 (this PR, "UCT/IB/MLX5: support am zcopy in purge outstanding"); sibling series PRs #11866, #11874, #11836, #12019, #11915, #11969. Historical flakiness of this exact test due to MEMIC device-memory allocation: #10475, plus commits 69718f6d and [REDACTED:Hex High Entropy String] (retries + random sleep on MEMIC allocation failure) and d7d57e17 (limit MEMIC allocation to prevent OOM test failures).

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 63b7fd95-267a-40ba-ae35-8dbf2758cec7 in the triage console for the audit trail.

@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

@svc-ucx

svc-ucx commented Oct 9, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR (Tests althca on worker 0) · commit 54c22778

TL;DR: The test_ucm_hooks step in contrib/test_jenkins.sh died because ./test/apps/test_hooks -n 27 crashed with SIGTRAP (Trace/breakpoint trap) on iteration 15/30 — this is the known UCM bistro in-place code-patching race that the test deliberately provokes, not a regression from this PR (which touches RC mlx5 AM zcopy failover). Re-run the job; to harden, raise UCX_BISTRO_GRACE_DURATION for this test and make the harness capture/report the crash instead of printing an empty log.

Full analysis

Summary: Job "althca on worker 0" failed in test_ucm_hooks: test/apps/test_hooks -n 27 terminated with a Trace/breakpoint trap (SIGTRAP), so the script hit exit 1 at contrib/test_jenkins.sh:1119-1124.

Root cause: test_hooks intentionally runs N threads hammering mmap() while dlopen() triggers UCM bistro to hot-patch the mmap prologue (test/apps/test_hooks.c:83, :189). The patch sequence in ucm_bistro_apply_patch_atomic() installs a 2-byte self-jump lock (jmp %rip-2, ucm_bistro_patch_lock()), waits ucm_global_opts.bistro_grace_duration, memcpy's the payload behind the lock, then unlocks with a 16-bit atomic swap. A thread that had already prefetched/entered the prologue past the 2-byte lock can resume inside the half-written payload and execute a garbage/trap instruction → SIGTRAP. The grace window is a probabilistic mitigation, so the test is inherently flaky under load: 14 iterations passed in ~1.2s each and the 15th crashed immediately. Evidence it is not PR-related: the crash is in UCM memory-hook patching, while the branch (failover_uct_rcx_am_zcopy, commit 54c2277) changes uct/ib RC mlx5 AM zcopy paths, and the preceding gtest run (test_obj_size) passed.

Secondary observation from the same log: line gcc: error: test_ucp_config-test_ucp_config.o: No such file or directory at 10:22:02 shows the reused incremental build dir on agent-01 is inconsistent, and that failure was silently swallowed because the command is piped to tee without pipefail (contrib/test_jenkins.sh:1104). Worth cleaning the agent workspace. Also, test_hooks.log was empty in the error dump (stdout never flushed on the fatal signal), so the harness gave zero diagnostics.

Implicated commit: Not the PR commit. The patching mechanism is from 676b5792 (Thomas Vegas, "UCM/BISTRO: Add locking step while patching library functions"); the race-window tuning is c77bb48b (shasson5, "UCM/BISTRO: Increase grace time to reduce patch race", #11112).

File: src/ucm/bistro/bistro.c:83-97 (lock/grace/payload-copy window), src/ucm/bistro/bistro_x86_64.c:309-327 (ucm_bistro_patch_lock), harness at contrib/test_jenkins.sh:1112-1129

Suggested fix:

  1. Re-trigger CI on PR UCT/IB/MLX5: support am zcopy in purge outstanding #11873 — the failure is unrelated to the AM zcopy failover change and should pass on retry.
  2. Make the flake visible instead of silent: in test_ucm_hooks, run under stdbuf -o0 -e0, and on failure report the exit status/signal ($?, 128+n) plus ulimit -c unlimited + backtrace, so an empty test_hooks.log no longer hides the signal.
  3. Reduce flakiness: export a larger UCX_BISTRO_GRACE_DURATION for the test_hooks invocations (the knob c77bb48b added for exactly this race), and/or cap threads to NPROC instead of 2*NPROC so loaded CI agents don't widen the race window.
  4. Fix the agent-side build inconsistency: add set -o pipefail (or check PIPESTATUS) around the |& tee commands near contrib/test_jenkins.sh:1096-1109 so a broken incremental build fails loudly, and wipe the stale build-test dir on agent-01.

Related: #11112 (UCM/BISTRO: Increase grace time to reduce patch race); PR under test: #11873

@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

@gleon99
gleon99 merged commit 0f59999 into openucx:master Oct 13, 2026
162 checks passed
@jeynmann
jeynmann deleted the failover_uct_rcx_am_zcopy branch October 13, 2026 07:44
@svc-ucx svc-ucx mentioned this pull request Oct 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants