Repository navigation
UCT/IB/MLX5: support am zcopy in purge outstanding - #11873
Conversation
|
🤖 Starting review — findings will be posted here when done. |
|
🤖 CI Triage Agent — TL;DR: The "Codestyle / commit title" check failed because one commit in PR #11873 is titled Full analysisSummary: Azure Pipelines job "UCX PR (Codestyle commit title)" build 133712 exited with code 1 because Root cause: The check script iterates over the PR commits and validates each subject line against the regex
Implicated commit: The PR commit titled File: buildlib/tools/codestyle.sh:14 (the validating regex); the defect is in the PR's commit message, not this file. Suggested fix: Rewrite that commit's subject with a single space after the colon, e.g. No change to
|
88e5bc5 to
3db1c48
Compare
|
🤖 Starting review — findings will be posted here when done. |
|
🤖 CI Triage Agent — TL;DR: The "commit title" codestyle check failed because one commit in PR #11873 has a double space after the colon — Full analysisSummary: Azure Pipelines job "UCX PR (Codestyle commit title)" build 134323 exited with code 1 after reporting Root cause: The check iterates over the PR's commits and validates each subject line against the regex Implicated commit: The PR commit titled File: buildlib/tools/codestyle.sh:14 (the validating regex); the defect itself is in the commit message, not in tracked source. Suggested fix: Rewrite the offending commit title to remove the extra space, e.g. Optionally, to make this class of failure more obvious, extend the script to print why a title was rejected (e.g. echo the expected format Related: #11873
|
|
Test coverage gap: the PR adds |
|
🤖 Starting review — findings will be posted here when done. |
|
🤖 CI Triage Agent — TL;DR: This is not a code/test failure — the "commit title" codestyle check rejected one commit in PR #11873 because its title Full analysisSummary: Azure Pipelines job "UCX PR (Codestyle commit title)" build 134345 failed with Root cause: The commit-title validation script iterates over every commit in the PR range and matches each subject against the UCX title convention ( The only difference between the passing Implicated commit: The offending commit is the one titled File: No source file is at fault — the failure is in the git commit metadata of the PR branch. The check itself runs as the Suggested fix: Reword the single bad commit title to remove the duplicated space: git rebase -i [REDACTED:Hex High Entropy String]
# mark the 'GTEST/UCT/IB: test am zcopy parser' commit as 'reword'
# change the title to: GTEST/UCT/IB: test am zcopy parser
git push --force-with-lease origin failover_uct_rcx_am_zcopyAlternatively, since this PR is marked Related: PR #11873 — [WIP][DNM] UCT/IB/MLX5: wqe parser for uct am zcopy
|
|
🤖 Review complete — no issues found. |
85f2e7c to
539e89e
Compare
|
🤖 Starting review — findings will be posted here when done. |
|
Test/scope (already reported, restating for completeness): there is still no caller or test for uct_rc_mlx5_op_info_fill in this PR, so the new AM-zcopy path is untested/dead here. This was raised in the prior review; if a caller/test is coming in a follow-up, that should be stated in the PR description. |
|
🤖 Starting review — findings will be posted here when done. |
|
🤖 Review complete — no issues found. |
|
🤖 CI Triage Agent — TL;DR: The Full analysisSummary: gtest Root cause: A hang, not slowness. Log timestamps show continuous ~400 ms test cadence until
Notably there is no send-completion-with-error, no IB async event, no retry-exceeded error in the log — the QP produced no completions at all, consistent with the puts sitting un-progressed (pending/wireup) rather than a fabric error being reported. Relation to the PR: this variant is Implicated commit: unknown — not File: Suggested fix:
Related: PR #11873 ("[WIP][DNM] UCT/IB/MLX5: wqe parser for uct am zcopy"); no existing issue found for this hang ( |
119f81b to
957f00e
Compare
|
🤖 Starting review — findings will be posted here when done. |
|
blocker:
blocker:
minor: Coverage note: the new gtest only exercises |
|
🤖 CI Triage Agent — TL;DR: The aarch64 build failed on a plain C++ compile error in the gtest sources added by this PR: Full analysisSummary: Root cause: Compilation error introduced by the PR's new gtest code, not an infrastructure/timeout issue:
Implicated commit: File: Suggested fix:
Related: PR #11873 (this PR); base infrastructure from #11891 "GTEST/UCT: Add outstanding purge test base" (commit
|
957f00e to
c061cf2
Compare
|
🤖 Starting review — findings will be posted here when done. |
|
Residual coverage gap (already raised in earlier reviews, not repeated inline): the new gtest only exercises this code when the out-of-tree plugin providing |
|
🤖 CI Triage Agent — TL;DR: The gtest run failed on a single, unrelated test — Full analysisSummary: Root cause: Evidence from the log: 8607 tests passed, exactly one failed, the run completed normally (no SIGTERM/timeout, continuous timestamps up to The failing test is Crucially, PR #11873 is "UCT/IB/MLX5: wqe parser for uct am zcopy" — it touches the RC/mlx5 WQE parsing path and has no interaction with the UD transport's private-EP/keepalive lifecycle. So this is a pre-existing flaky failure, not a regression from the PR under test. Caveat: the per-test gtest assertion text is not recoverable from this build — the fetched log is only the tail (~last 500 lines) of a ~60-minute shuffled run, and the test itself executed much earlier. The diagnosis above is based on the failure summary line, the test source, and the existing hardening PR. Implicated commit: File: Suggested fix:
Related: #11920 (UCT/GTEST: Hardening of UD stale private EP tests — the actual fix); #11873 (PR under test, unrelated); #11800 (UCT/UD: generation ID in dest_id, same feature area) |
|
🤖 Starting review — findings will be posted here when done. |
|
minor: payload size does not leave room for the AM header. The same line also breaks the assignment-column rule ( |
2ebd0b8 to
4efd673
Compare
|
🤖 Starting review — findings will be posted here when done. |
|
🤖 CI Triage Agent — TL;DR: The ASan gtest shard aborted because Full analysisSummary: Root cause: Evidence from the log:
Notably, the PR's commit Implicated commit: unknown (not caused by File: Suggested fix:
Related: #11920 UCT/GTEST: Hardening of UD stale private EP tests, #11948 same for v1.23, #11873 (this PR)
|
|
🤖 Starting review — findings will be posted here when done. |
UCT/IB/MLX5: cleanup GTEST/UCT/IB: update gtest GTEST/UCT/IB: update test GTEST/UCT/IB: refactor test UCT/IB/MXL5: cleanup UCT/IB/MLX5: address comments UCT/IB/MLX5: address comments UCT/IB/MLX5: address comments UCT/IB/MLX5: address comments UCT/IB/MLX5: address comments UCT/IB/MLX5: address comments UCT/IB/MLX5: address comments UCT/IB/MLX5: address comments UCT/IB/MLX5: address comments UCT/IB/MLX5: address comments GTEST/UCT/FT: add zcopy test UCT/IB/MLX5: address comments UCT/IB/MLX5: const wrap UCT/IB/MXL5: const wrap UCT/IB/MXL5: const wrap UCT/IB/MLX5: am zcopy parser GTEST/UCT/IB: test am zcopy parser UCT/IB/MLX5: cleanup UCT/IB/MLX5: log level UCT/IB/MLX5: cleanup GTEST/UCT/IB: rm overlap gtests UCT/IB/MLX5: address comments UCT/IB/MLX5: address comments UCT/IB/MLX5: address comments GTEST/UCT/IB: add test for am zcopy GTEST/UCT/IB: fix test GTEST/UCT/IB: cleanup UCT/IB/MLX5: align inline size UCT/IB/MLX5: const wrap UCT/IB/MLX5: add assertions UCT/IB/MLX5: add doc for copy segs UCT/IB/MLX5: copy segs UCT/IB/MLX5: refactor am zcopy parser UCT/IB/MLX5: address comments UCT/IB/MLX5: cleanup UCT/IB/MLX5: cleanup UCT/IB/MLX5: unsupport tag GTEST/UCT/IB: cleanup UCT/IB/MLX5: fix num packets UCT/IB/MLX5: refactor UCT/IB/MLX5: format UCT/IB/MLX5: format UCT/IB/MLX5: comp status UCT/IB/MLX5: am segs helper UCT/IB/MLX5: format and cleanup
2c5ce2d to
30571d3
Compare
|
🤖 Starting review — findings will be posted here when done. |
| static int uct_rc_mlx5_send_op_is_am_zcopy(const uct_rc_iface_send_op_t *op) | ||
| { | ||
| return op->handler == uct_rc_ep_send_op_completion_handler; | ||
| } |
There was a problem hiding this comment.
looks too generic and unsafe, maybe smth like this:
.inl:
void UCS_F_ALWAYS_INLINE
uct_rc_ep_send_op_completion_inl(uct_rc_iface_send_op_t *op,
const void *resp)
{
uct_invoke_completion(op->user_comp, UCS_OK);
uct_rc_iface_put_send_op(op);
}
.c:
void uct_rc_ep_send_op_completion_handler(uct_rc_iface_send_op_t *op,
const void *resp)
{
uct_rc_ep_send_op_completion_inl(op, resp);
}
...
void uct_rc_ep_send_op_am_zcopy_comp_handler(uct_rc_iface_send_op_t *op,
const void *resp)
{
uct_rc_ep_send_op_completion_inl(op, resp);
}
|
🤖 CI Triage Agent — TL;DR: The ASAN/RoCE gtest run failed on a single test, Full analysisSummary: Root cause: Evidence from the log: the run completed normally (
Only the The branch's HEAD commit, Caveat on evidence: the retained console tail covers only the last ~45 seconds of a ~45-minute run and the tests execute in shuffled order, so the assertion text for this test scrolled out; the final summary block is what identifies it. The mechanism above is inferred from the test source and the commit on the branch, not from a quoted assertion line. Implicated commit: File: Suggested fix:
Related: PR #11873 (this PR, "UCT/IB/MLX5: support am zcopy in purge outstanding"); sibling series PRs #11866, #11874, #11836, #12019, #11915, #11969. Historical flakiness of this exact test due to MEMIC device-memory allocation: #10475, plus commits
|
|
🤖 Starting review — findings will be posted here when done. |
|
🤖 CI Triage Agent — TL;DR: The Full analysisSummary: Job "althca on worker 0" failed in Root cause: Secondary observation from the same log: line Implicated commit: Not the PR commit. The patching mechanism is from File: Suggested fix:
Related: #11112 (UCM/BISTRO: Increase grace time to reduce patch race); PR under test: #11873 |
|
🤖 Starting review — findings will be posted here when done. |
What?
Add wqe parse support for uct am zcopy.
Will rebase after PR#12058 is merged
Why?
FT need extract info from wqe before replay uct am zcopy.
How?
Add parser for uct am zcopy.