Skip to content

ci: raise dynamo-runtime parallel CPU-only test timeout to 45m - #18338

Merged
nv-tusharma merged 2 commits into
mainfrom
devin/dyn-5487-raise-arm64-cpu-timeout
Nov 6, 2026
Merged

nv-tusharma merged 2 commits into
mainfrom
devin/dyn-5487-raise-arm64-cpu-timeout

Conversation

@devin-ai-integration

Copy link
Copy Markdown
Contributor

Overview

Since ae5d9f2 (#18121) the CPU-only step of dynamo-runtime / test / parallel cuda13.0, arm64 runs 1770–1800s, and 10 of 14 sampled Post-Merge runs since then were killed at exactly 1800s. Before that commit the step took 1410–1600s. Example: https://github.com/ai-dynamo/dynamo/actions/runs/42180752216 (killed at 99%, during the wheel smoke tests).

Key Changes

  • .github/workflows/dynamo-pipeline.yml: raise the runtime parallel job's cpu_only_test_timeout_minutes from 30 to 45. This is a stopgap. The input is shared by amd64 and arm64, and pre-merge, post-merge and nightly all call this workflow, so the higher limit applies to all of them. It only raises the upper limit and doesn't change what runs.
  • Adds TODO(DYN-5487): move the slow planner/replay tests (test_unified_cli_dynamo.py recommend round-trips, 87–153s each on arm64) out of this lane, then restore 30.

Validation

  • The YAML loads and git diff --check is clean.
  • pre-commit was not run. Its hooks could not be downloaded in this environment (HTTP 403 from the hook mirror).

Issues and Ongoing Work

Pull Request Checklist
  • I used Dynamo's /pr-checklist skill.

Link to Devin session: https://nvidia-cloud.devinenterprise.com/sessions/360339eecc2144ecab7b299f91291b9b
Open in Devin Desktop: https://nvidia-cloud.devinenterprise.com/desktop/session/360339eecc2144ecab7b299f91291b9b?variant=devin

Signed-off-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@devin-ai-integration
devin-ai-integration Bot requested a review from a team as a code owner November 6, 2026 19:01
@devin-ai-integration

Copy link
Copy Markdown
Contributor Author

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

  • Disable automatic comment, CI, and merge conflict monitoring

@copy-pr-bot

copy-pr-bot Bot commented Nov 6, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@github-actions github-actions Bot added ci Issues/PRs that reference CI build/test actions labels Nov 6, 2026
@nv-tusharma

Copy link
Copy Markdown
Collaborator

/ok to test cc70cee

@devin-ai-integration

devin-ai-integration Bot commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor Author

⏹️ Dynamo PR CI cancelled — run 42296280795 (attempt 1) on 200242b60f

Gate checks: ✅ backend-status-check · ❌ deploy-status-check · ✅ dynamo-status-check

Framework Deploy
vLLM ⏹️ 1 ✅ 3
SGLang ⏹️ 1 ✅ 2

Failure details

No job failed. 2 deploy tests were cancelled, and deploy-status-check fails only because of them. Nothing failed before them, so this was not a fail-fast cancel. Both were cancelled at the same point, about 23 minutes into Run Dynamo Deploy Test, which looks like a timeout or a manual cancel. Up to then, their logs show the deploy worker pods waiting on GPUs (FailedScheduling - 0/14 nodes are available: ... Insufficient nvidia.com/gpu), so the cluster had no GPU capacity free. Rerunning the failed jobs is probably enough.

  • ⏹️ vllm Deploy Test / agg · step Run Dynamo Deploy Test · gh run view --job 128173396574 -R ai-dynamo/dynamo --log
  • ⏹️ sglang Deploy Test / agg · step Run Dynamo Deploy Test · gh run view --job 128173648449 -R ai-dynamo/dynamo --log
For agents
{"pr": 18338, "run_id": 42296280795, "run_attempt": 1, "head_sha": "200242b60f088f7ed7ac08536a37b46297692ef5",
 "failures": [],
 "cancelled": [{"job": "vllm Deploy Test / agg", "job_id": 128173396574, "step": "Run Dynamo Deploy Test", "signature": "cancelled after ~23m; pods FailedScheduling Insufficient nvidia.com/gpu", "log_cmd": "gh run view --job 128173396574 -R ai-dynamo/dynamo --log"},
               {"job": "sglang Deploy Test / agg", "job_id": 128173648449, "step": "Run Dynamo Deploy Test", "signature": "cancelled after ~23m; pods FailedScheduling Insufficient nvidia.com/gpu", "log_cmd": "gh run view --job 128173648449 -R ai-dynamo/dynamo --log"}]}

Posted automatically by Devin for run 42296280795. Updated on every full-CI run of this PR.

@nv-anants

Copy link
Copy Markdown
Member

/ok to test 200242b

@nv-tusharma
nv-tusharma enabled auto-merge (squash) November 6, 2026 21:41
@nv-tusharma
nv-tusharma merged commit 1d6e9e0 into main Nov 6, 2026
215 of 218 checks passed
@nv-tusharma
nv-tusharma deleted the devin/dyn-5487-raise-arm64-cpu-timeout branch November 6, 2026 22:59
ishandhanani added a commit to ishandhanani/dynamo that referenced this pull request Nov 6, 2026
Absorbs one upstream commit, no conflicts, no router-owned path:

- ai-dynamo#18338 ci: raise dynamo-runtime parallel CPU-only test timeout to 45m:
  .github/workflows/dynamo-pipeline.yml only. Outside the router;
  auto-merged.

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
devivasudevan pushed a commit to devivasudevan/dynamo that referenced this pull request Nov 7, 2026
…namo#18338)

Signed-off-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Tushar Sharma <tusharma@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

actions ci Issues/PRs that reference CI build/test size/XS

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants