Skip to content

CI: Trigger Blossom CI automatically on PR events - #12235

Open
Alexey-Rivkin wants to merge 2 commits into
openucx:masterfrom
Alexey-Rivkin:ci-blossom-auto-trigger
Open

Alexey-Rivkin wants to merge 2 commits into
openucx:masterfrom
Alexey-Rivkin:ci-blossom-auto-trigger

Conversation

@Alexey-Rivkin

@Alexey-Rivkin Alexey-Rivkin commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

What?

Start Blossom CI (Jenkins) automatically on PR open/update/reopen, no /build comment needed.

Why?

Today every PR needs a manual /build comment before Jenkins runs.

How?

  • Add pull_request_target trigger (opened, synchronize, reopened) and let Authorization run on it
  • Skip Vulnerability-scan when Authorization returns no args
  • Workflow permissions: contents, pull-requests, statuses set to write
  • Same file as CI: re-enable blossom CI auto trigger ai-dynamo/nixl#2581, already running there
  • Dispatcher: the stale-run abort also reads the PR number from push payloads (top-level number), as CI: handle PR push payloads in the dispatcher ai-dynamo/nixl#2595 does; otherwise each push leaves the previous run going
  • Jenkins checks stay non-mandatory: the only required check on master/release is UCX PR (Azure)

Blossom CI only starts on a "/build" comment, so every PR needs a manual
step before Jenkins runs. Add a pull_request_target trigger so it starts
on PR open, update and reopen. Blossom AUTH still gates who may run it;
Vulnerability-scan is skipped when AUTH returns no args. The workflow
matches the version NIXL runs, which Blossom accepts. Jenkins checks stay
non-mandatory: the only required check on master is "UCX PR".

Signed-off-by: Alexey Rivkin <arivkin@nvidia.com>
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

@Alexey-Rivkin
Alexey-Rivkin marked this pull request as ready for review November 4, 2026 08:36
@Alexey-Rivkin
Alexey-Rivkin requested a review from a team as a code owner November 4, 2026 08:36
Comment thread .github/workflows/blossom-ci.yml Resolved
Comment thread .github/workflows/blossom-ci.yml Resolved
Comment thread .github/workflows/blossom-ci.yml Resolved
Comment thread .github/workflows/blossom-ci.yml Resolved
@svc-nvidia-pr-review

Copy link
Copy Markdown

Suggested review mode: COMMENT. The main security concern (pull_request_target + untrusted checkout) depends on blossom-ci AUTH behavior that is not visible in this repo, so it is posed as a question rather than a blocker; if the author confirms AUTH can produce args for unauthorized fork PRs, it should be escalated to a blocker before merge.

Residual gap: no CI evidence in-repo that the new trigger behaves as intended; validation must come from a fork run or AUTH-side logic.

Residual gap: actions/checkout@v2 in Vulnerability-scan is outdated, but it is not touched by this diff, so it is out of scope here.

@svc-ucx

svc-ucx commented Nov 4, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR (AddressSanitizer gpu on worker 3) · commit 46d710a7

TL;DR: The ASAN GPU shard didn't fail an assertion — it hung: ib/test_ucp_rma_rndv_cuda_async.put_blocking/1 issued a blocking ucp_put_nbx forced onto the rendezvous PUT protocol with cudaMallocAsync CUDA buffers and the request never completed for 11 minutes, so the gtest watchdog aborted the whole shard; the fix is to debug the stuck rndv/put completion (ATP/flush never arriving) rather than raise any timeout.

Full analysis

Summary: ib/test_ucp_rma_rndv_cuda_async.put_blocking/1 <ib,cuda_copy,rocm_copy/flush_ep> hung; ucp_test.cc:310 request 0x62e00003a420 did not complete on time, then the watchdog fired Connection timed out - abort testing → SIGABRT (make: *** [Makefile:4992: test] Aborted (core dumped)).

Root cause: A hang, not a slow test. Timestamp analysis of the stage log:

  • 18:38:41.196 [ RUN ] ib/test_ucp_rma_rndv_cuda_async.put_blocking/1
  • gap of 11 m 15 s with zero output → 18:49:56.555 UCX ERROR request 0x62e00003a420 did not complete on time (the ucp_test request-wait timeout)
  • another 3 m 45 s gap → 18:53:41.192 test_helpers.cc:57: Failure / Connection timed out - abort testing (gtest watchdog), backtrace shows the main thread spinning in sched_yield inside the progress/wait loop.

So UCX progress was being driven continuously but the rendezvous PUT request never completed and no transport error was reported — a lost completion / stalled rendezvous handshake (RTS→RTR→PUT→ATP), not a time-limit problem. The test configuration is the unusual part: test_ucp_rma_rndv forces PROTOS=put/rndv,get/rndv,rndv/* (so rndv/put/zcopy or rndv/put/mtype is always selected instead of the normal zcopy fast path), and test_ucp_rma_rndv_cuda_async additionally forces UCX_IB_GPU_DIRECT_RDMA=y and allocates the CUDA buffers with mem_buffer::alloc_mode::ASYNC (cudaMallocAsync), sizes up to 100 MB, under AddressSanitizer. The first message-size iteration never finished, i.e. the stall happens on the very first forced-rndv PUT of a host↔cuda / cuda↔cuda pair.

Note the ASAN job does not export UCX_PROTO_REQUEST_RESET=y (build_for_tests() returns early for ASAN_CHECK=yes, before the export) and leaves UCX_RNDV_FRAG_WORKER_MAX_MEM=auto (unlimited), so the newly merged fragment-throttling queues are not active here — those can be ruled out as the trigger. The log contains no UCX debug/trace output, so the exact lane/stage where the completion was lost cannot be pinned from this build alone.

Implicated commit: unknown (no UCX trace in the log). Recent commits touching exactly this path, in order of suspicion: [REDACTED:Hex High Entropy String] (Yaser Afshar, 2026‑11‑02, "UCP/RNDV: Use local memory info for unsupported remote memory types" — changes rndv protocol selection for GPU/unsupported remote memtypes and touches this very test file), 1acfdd61 (amastbaum, 2026‑11‑03, rndv fragment throttling in rndv_put.c/rndv_mtype.inl), 86cfb87c (Evgeny Leksikov, 2026‑11‑02, keeps lane/peer request id until error completion).

File: test/gtest/ucp/test_ucp_rma.cc:845 (hanging test; class at 820‑843), stalled protocol code in src/ucp/rndv/rndv_put.c:394-491 (rndv/put/zcopy send → FLUSH → ATP stages) / src/ucp/rndv/rndv_mtype.inl.

Suggested fix:

  1. Don't raise the 200 m job timeout or the request-wait timeout — the process was idle, not busy.
  2. A core was dumped ("core dumped") on swx-rdmz-ucx-gpu-02; inspect it to confirm where the PUT is stuck: req->send.proto_config->proto->name (zcopy vs mtype), req->send.state.uct_comp.count, req->send.rndv.put.atp_count/flush_lane, and whether the request is still on a UCT pending queue. That single data point identifies the lost completion.
  3. Reproduce locally/CI with just this test and tracing, e.g. GTEST_FILTER='ib/test_ucp_rma_rndv_cuda_async.put_blocking/1' UCX_PROTO_INFO=y UCX_LOG_LEVEL=trace_req to see which rndv protocol is chosen and which stage stops.
  4. Short-term, to unblock the shard: skip test_ucp_rma_rndv_cuda_async.put_blocking for the ib + async‑CUDA combination (same way test_ucp_rma_rndv_gpu_nic already guards its fragile variant at lines 528‑535) and open a tracking issue, since one hung gtest aborts the entire 4‑shard ASAN run.
  5. PR CI: Trigger Blossom CI automatically on PR events #12235 is on the ci-blossom-auto-trigger branch (CI plumbing); nothing in it touches UCP rendezvous, so this is almost certainly a pre-existing hang on the merge base rather than a regression introduced by the PR — re-run to confirm reproducibility before blaming the PR.

Related: PR #11062 ("UCP/RNDV: Throttle rndv fragment requests", merged as 1acfdd61) and PR #12111 ([REDACTED:Hex High Entropy String]) are the recent changes in this code path; no existing issue found for this hang — one should be filed.

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 5f8322f1-a8cf-45e0-9def-ef427fd1d7d0 in the triage console for the audit trail.

With Blossom CI starting on every push, the dispatcher also receives
PR push payloads, which carry the PR number at the top level rather
than in issue.number as /build comments do. The stale-run filter only
read issue.number, so a push never matched the previous run and left
it and its leaf jobs running alongside the new one.

Signed-off-by: Alexey Rivkin <arivkin@nvidia.com>
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread .github/workflows/blossom-ci.yml Resolved
Comment thread .github/workflows/blossom-ci.yml Resolved
Comment thread .ci/jenkins/Jenkinsfile.dispatcher Resolved
@svc-ucx

svc-ucx commented Nov 5, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR (AddressSanitizer BlueField on worker 0) · commit 31ac7f70

TL;DR: The ASAN gtest shard was killed because its very first test, dc_mlx5/test_uct_ib_roce.local_subnet_only/0 <dc_mlx5/mlx5_0:1>, hung for exactly 900 s inside uct_test::flush() (the DC send never completed on the BlueField RoCE port) and the gtest watchdog aborted the process; the hang is unbounded because uct_test::flush() defaults to deadline = ULONG_MAX, so one stuck device wedges the whole shard instead of failing a single test.

Full analysis

Summary: make -C test/gtest test (shard 1 of 4, ASAN/aarch64 on swx-rain02-bf1) aborted with SIGABRT/core after the gtest watchdog timed out on dc_mlx5/test_uct_ib_roce.local_subnet_only/0.

Root cause: Log timestamps show the test started at 13:36:29.995 and the next line is at 13:51:29.996 — a single 900 s gap with zero output, i.e. a hang, not a slow test. The 900 s is watchdog_timeout in test/gtest/common/test_helpers.cc:31; the failure text Connection timed out - abort testing is ADD_FAILURE() << strerror(ETIMEDOUT) from the watchdog thread (test_helpers.cc:57), followed by pthread_kill(SIGTERM) on the test thread. The captured backtrace puts the test thread in uct_test::flush() called from test_uct_ib::send_recv_short() (test/gtest/uct/ib/test_ib.cc:75), meaning uct_iface_flush() kept returning UCS_INPROGRESS forever — the uct_ep_am_short() posted on the DC endpoint over mlx5_0:1 never produced a TX completion on this BlueField node. Two contributing facts:

  • void flush(ucs_time_t deadline = ULONG_MAX) (test/gtest/uct/uct_test.h:409) with the loop at test/gtest/uct/uct_test.cc:803-820 has effectively no timeout when called as flush(), so the test spins until the watchdog kills the entire process, losing the remaining ~8970 tests of the shard.
  • PR CI: Trigger Blossom CI automatically on PR events #12235 ("CI: Trigger Blossom CI automatically on PR events") changes only CI plumbing and touches no UCX source, so this is not a regression introduced by the tested commit — it is a device/driver-level TX stall on that BF worker, exposed by the unbounded flush.

Implicated commit: unknown — the tested commit 31ac7f70 / PR #12235 is CI-only. If the stall reproduces on that node, the most recent changes to the DC/IB address-handle send path are f53da7f0 (Thomas Vegas, "UCT/IB: Add refcounted, TTL-based address handle cache", #11845) and c8495100 (Thomas Vegas, "UCT/IB: Add configurable TTL knob for the AH cache", #11847), both of which touch src/uct/ib/mlx5/dc/dc_mlx5_ep.c.

File: test/gtest/uct/ib/test_ib.cc:75 (hang point) and test/gtest/uct/uct_test.h:409 (flush(deadline = ULONG_MAX))

Suggested fix:

  1. Triage the worker first: check mlx5_0:1 port state / RoCE GID / ibv_devinfo on swx-rain02-bf1 and re-run the shard. A first-test DC flush that never completes usually means the port is down/misconfigured or a wedged DCI on that BlueField, not a code bug in the PR.
  2. Harden the test so a stuck device fails one test instead of the whole shard: in test_uct_ib::send_recv_short() pass an explicit deadline, e.g. flush(ucs::get_deadline(DEFAULT_TIMEOUT_SEC)); (the loop already does EXPECT_TRUE(flushed) << "Timed out"), and/or change the default of uct_test::flush() from ULONG_MAX to a real deadline so no caller can spin forever.
  3. Optionally add a uct_ib_iface_is_roce() skip guard to test_uct_ib_roce so IB_ROCE_REACHABILITY_MODE=local_subnet is only exercised on actual RoCE ports.

Related: #11845, #11847 (AH cache changes in the DC send path); #12235 (the CI-only PR under test)

@Alexey-Rivkin

Copy link
Copy Markdown
Contributor Author

Residual gap: no CI evidence in-repo that the new trigger behaves as intended; validation must come from a fork run or AUTH-side logic.

The same file runs on pull_request_target in ai-dynamo/nixl: AUTH gives args only to authorized authors (ran, skipped).

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants