Repository navigation
ci: raise dynamo-runtime parallel CPU-only test timeout to 45m - #18338
Conversation
Signed-off-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
|
I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".
|
|
/ok to test cc70cee |
⏹️ Dynamo PR CI cancelled — run 42296280795 (attempt 1) on
|
| Framework | Deploy |
|---|---|
| vLLM | ⏹️ 1 ✅ 3 |
| SGLang | ⏹️ 1 ✅ 2 |
Failure details
No job failed. 2 deploy tests were cancelled, and deploy-status-check fails only because of them. Nothing failed before them, so this was not a fail-fast cancel. Both were cancelled at the same point, about 23 minutes into Run Dynamo Deploy Test, which looks like a timeout or a manual cancel. Up to then, their logs show the deploy worker pods waiting on GPUs (FailedScheduling - 0/14 nodes are available: ... Insufficient nvidia.com/gpu), so the cluster had no GPU capacity free. Rerunning the failed jobs is probably enough.
- ⏹️ vllm Deploy Test / agg · step
Run Dynamo Deploy Test·gh run view --job 128173396574 -R ai-dynamo/dynamo --log - ⏹️ sglang Deploy Test / agg · step
Run Dynamo Deploy Test·gh run view --job 128173648449 -R ai-dynamo/dynamo --log
For agents
{"pr": 18338, "run_id": 42296280795, "run_attempt": 1, "head_sha": "200242b60f088f7ed7ac08536a37b46297692ef5",
"failures": [],
"cancelled": [{"job": "vllm Deploy Test / agg", "job_id": 128173396574, "step": "Run Dynamo Deploy Test", "signature": "cancelled after ~23m; pods FailedScheduling Insufficient nvidia.com/gpu", "log_cmd": "gh run view --job 128173396574 -R ai-dynamo/dynamo --log"},
{"job": "sglang Deploy Test / agg", "job_id": 128173648449, "step": "Run Dynamo Deploy Test", "signature": "cancelled after ~23m; pods FailedScheduling Insufficient nvidia.com/gpu", "log_cmd": "gh run view --job 128173648449 -R ai-dynamo/dynamo --log"}]}Posted automatically by Devin for run 42296280795. Updated on every full-CI run of this PR.
|
/ok to test 200242b |
Absorbs one upstream commit, no conflicts, no router-owned path: - ai-dynamo#18338 ci: raise dynamo-runtime parallel CPU-only test timeout to 45m: .github/workflows/dynamo-pipeline.yml only. Outside the router; auto-merged. Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
…namo#18338) Signed-off-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Tushar Sharma <tusharma@nvidia.com>
Overview
Since ae5d9f2 (#18121) the CPU-only step of
dynamo-runtime / test / parallel cuda13.0, arm64runs 1770–1800s, and 10 of 14 sampled Post-Merge runs since then were killed at exactly 1800s. Before that commit the step took 1410–1600s. Example: https://github.com/ai-dynamo/dynamo/actions/runs/42180752216 (killed at 99%, during the wheel smoke tests).Key Changes
.github/workflows/dynamo-pipeline.yml: raise the runtimeparalleljob'scpu_only_test_timeout_minutesfrom 30 to 45. This is a stopgap. The input is shared by amd64 and arm64, and pre-merge, post-merge and nightly all call this workflow, so the higher limit applies to all of them. It only raises the upper limit and doesn't change what runs.TODO(DYN-5487): move the slow planner/replay tests (test_unified_cli_dynamo.pyrecommend round-trips, 87–153s each on arm64) out of this lane, then restore 30.Validation
git diff --checkis clean.Issues and Ongoing Work
Pull Request Checklist
/pr-checklistskill.Link to Devin session: https://nvidia-cloud.devinenterprise.com/sessions/360339eecc2144ecab7b299f91291b9b
Open in Devin Desktop: https://nvidia-cloud.devinenterprise.com/desktop/session/360339eecc2144ecab7b299f91291b9b?variant=devin