Repository navigation
UCT/GTEST: Hardening of UD stale private EP tests - v1.23 - #11948
Conversation
|
🤖 Starting review — findings will be posted here when done. |
|
🤖 CI Triage Agent — TL;DR: The AddressSanitizer/roce worker-3 job did not fail on a test — the Azure agent on Full analysisSummary: Job "AddressSanitizer roce on worker 3" (build 135997) was cancelled mid-gtest because the Azure Pipelines agent received a shutdown signal, not because of a test assertion or sanitizer report. Root cause: Evidence from the log:
There is no gap in the timestamps longer than ~4 seconds anywhere in the log, so nothing hung and nothing timed out (the 200-minute Implicated commit: none — commit [REDACTED:Hex High Entropy String] (PR #11948, "UCT/GTEST: Hardening of UD stale private EP tests - v1.23") is not implicated; the cancellation is environmental. File: n/a (no source file implicated; failure occurred at the agent level, last log lines at 14:43:54 of the "Run ./contrib/test_jenkins.sh" step) Suggested fix: Re-trigger the "AddressSanitizer roce on worker 3" job — no code change is warranted. In parallel, have CI admins check the Related: PR #11948 (#11948) — the PR under test, not the cause. No existing issue found tracking this agent-shutdown signature.
|
What?
Backport #11920 to 1.23
Raise UD_TIMEOUT from 1s to 3s in stale_private_ep_reuse and private_ep_keepalive.
Why?
CI hit
ud_ep.c: Fatal: UD endpoint ... unhandled timeout errorinstale_private_ep_reuseon the ASAN stage. The failing job stalled for ~2s (other UD tests in it ran 2s slower than in every other ASAN job). With a 1s UD timeout, a stall while a CREQ is in flight makes the UD timer declare a peer timeout on a user endpoint before the already arrived CREP is processed. The UD gtest fixture has no error handler, so that is fatal and kills the whole shard. 3s matchestest_ud_peer_failureand survives such stalls; the keepalive idle window grows accordingly.