Skip to content

ci: move Planner tests out of runtime CPU lanes - #18351

Merged
tedzhouhk merged 2 commits into
mainfrom
hzhou-codex/dyn-5487-planner-ci
Nov 10, 2026
Merged

tedzhouhk merged 2 commits into
mainfrom
hzhou-codex/dyn-5487-planner-ci

Conversation

@tedzhouhk

@tedzhouhk tedzhouhk commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

Overview

Planner-family CPU tests also run in the generic Dynamo runtime lanes, duplicating coverage and pushing the arm64 parallel lane against its 30-minute timeout. Run these tests in the existing Planner jobs and add equivalent nightly coverage, while retaining runtime bindings and dependency checks.

Key Changes

  • 0 lines of production runtime logic. Both runtime CPU lanes exclude planner tests unless also marked core; the three replay bindings modules and AISimulate dependency contracts retain runtime coverage.
  • Repair three missing Planner markers and copy the AgentX, retune replay, and router retune contract modules into the Planner test image so moving them does not lose coverage. Unmarked Profiler suites remain in runtime.
  • Add amd64/arm64 nightly Planner tests using the matching nightly image and source SHA, include them in nightly status reporting, and preserve the moved tests' release gate. Planner build/test success therefore becomes required for nightly publication. Existing runtime timeout values are unchanged.

Validation

Behavior Evidence
Slow CLI work leaves runtime In PR #18121's arm64 CI logs, the same live-retune CLI case ran in runtime (144s) and Planner (139s). The runtime selector now excludes it; Planner still selects it. These are baseline measurements, not a measured post-change speedup.
Moved tests remain selected An authored-marker audit using pytest's expression evaluator checked tracked test definitions against old/new selectors: 2,987 definitions across 181 files move to Planner, with no selector-level coverage loss. Counts do not expand parameterized cases or execute tests.
Missing Planner markers repaired All 50 tests in test_advisory_mode.py, test_aic_interpolation.py, and test_perf_metrics_priority.py passed under the Planner selector with the pinned AISimulate wheel.
Runtime dependency contracts retained Four source dependency consistency tests passed under the new runtime selector; the installed-wheel check was excluded locally.
Independent review Two agentic reviews checked workflow wiring and test/image coverage. Fixed the discovered three-file image coverage gap; re-review found no remaining blocking issue.

Full runtime bindings collection could not complete locally: the shared venv's older _core extension lacks AisPerfConfig. No bindings rebuild, Planner image build, full relocated suite, or GitHub Actions run was performed locally; these remain CI validation boundaries. Workflow lint passed with existing unregistered runner-label warnings filtered out.

Issues and Ongoing Work

Pull Request Checklist
  • I used Dynamo's /pr-checklist skill.

Summary by CodeRabbit

  • Tests
    • Expanded planner test coverage to include additional replay, scaling, smoke, interpolation, advisory-mode, and performance-priority checks.
    • CPU-only planner tests are now selected more consistently across parallel and sequential test runs.
  • Chores
    • Nightly release checks now include planner tests, and publishing waits for those tests to pass.

Signed-off-by: hongkuanz <hongkuanz@nvidia.com>
@tedzhouhk
tedzhouhk requested review from a team as code owners November 6, 2026 22:42
@github-actions github-actions Bot added ci Issues/PRs that reference CI build/test planner container actions labels Nov 6, 2026
@coderabbitai

coderabbitai Bot commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

Explain Change →

No actionable comments were generated in the recent review.

Walkthrough

The change updates planner and core test markers, adjusts CPU-only test selection in the Dynamo pipeline, and adds a nightly planner test job that gates release and is a dependency of Slack notification.

Changes

Planner CPU testing

Layer / File(s) Summary
Test markers and planner image
components/src/dynamo/planner/tests/unit/*, lib/bindings/python/tests/replay/*, tests/dependencies/test_aisimulate_consistency.py, container/Dockerfile.test
Planner unit tests receive the planner marker. Replay and dependency tests receive the core marker. The planner test image copies additional replay and router test files.
CPU test selection
.github/workflows/dynamo-pipeline.yml
The parallel and sequential jobs restrict planner tests to those marked core while retaining non-planner tests under the supplied CPU-only markers.
Nightly planner test and release gate
.github/workflows/nightly-ci.yml
Nightly CI runs planner CPU tests for AMD and ARM. The release job requires the planner test to succeed, and Slack notification waits for the planner test job.

Estimated code review effort: 2 (Simple) | ~10 minutes

Merge Risk: Minimal · up to 60353

The scheduling change appears mergeable, subject to normal CI validation of the Planner image and nightly jobs.

Pre-merge checks: Passed 5

Check name Status Explanation
Title check Passed The title clearly describes the main test scheduling change.
Docstring Coverage Passed No eligible functions found in the changed files. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentat…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Description check Passed The description includes all required sections, validation results and limits, issue context, and the checklist.
Full details: Docstring Coverage

Explanation

No eligible functions found in the changed files. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 0 functions across 7 files. (3 skipped: 3 unsupported.)

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Tip

🎯 Customize your risk assessments

Guide how CodeRabbit assesses merge risk and security architecture risk. Highlight sensitive changes and explain your security boundaries using Merge Risk Instructions and Security Risk Instructions in your review settings. Security architecture risk assessments require the Advanced plan or above.

✅ Automatic approvals, with your rules

Let CodeRabbit approve pull requests once required threads are resolved, pre-merge checks pass, and your risk limits are met. Enable Automatic approvals in review settings, with overrides for teams or file paths.

Recent review info

Run configuration

Commits

Reviewing files that changed from the base of the PR and between dec6c22 and 603539a.

Files selected for processing: 10

  • .github/workflows/dynamo-pipeline.yml
  • .github/workflows/nightly-ci.yml
  • components/src/dynamo/planner/tests/unit/test_advisory_mode.py
  • components/src/dynamo/planner/tests/unit/test_aic_interpolation.py
  • components/src/dynamo/planner/tests/unit/test_perf_metrics_priority.py
  • container/Dockerfile.test
  • lib/bindings/python/tests/replay/test_replay_planner_load_modes.py
  • lib/bindings/python/tests/replay/test_replay_planner_scaling.py
  • lib/bindings/python/tests/replay/test_replay_smoke.py
  • tests/dependencies/test_aisimulate_consistency.py

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

@tedzhouhk
tedzhouhk enabled auto-merge (squash) November 6, 2026 23:04
Signed-off-by: hongkuanz <hongkuanz@nvidia.com>
@devin-ai-integration

Copy link
Copy Markdown
Contributor

✅ Dynamo PR CI passed — run 42309505310 (attempt 1) on 357432b99d

Gate checks: ✅ backend-status-check · ✅ deploy-status-check · ✅ dynamo-status-check

Posted automatically by Devin for run 42309505310. Updated on every full-CI run of this PR.

@tedzhouhk
tedzhouhk disabled auto-merge November 10, 2026 18:24
@tedzhouhk
tedzhouhk merged commit f4a437e into main Nov 10, 2026
147 of 149 checks passed
@tedzhouhk
tedzhouhk deleted the hzhou-codex/dyn-5487-planner-ci branch November 10, 2026 18:24
ishandhanani added a commit to ishandhanani/dynamo that referenced this pull request Nov 10, 2026
- 84c3ecd fix(mocker): admit sequential disagg decode with transferred prompt KV (ai-dynamo#18384): one file, lib/llm/src/mocker.rs (+56), auto-merged and taken whole: the decode-side branch that submits through LiveEngine::submit_decode when the request carries a prefill_result, plus the test sequential_decode_starts_from_transferred_prompt_kv. The mocker frontend stays in lib/llm; the handoff path it calls stays in lib/mocker. No router-owned path.
- 227025e fix(bindings): keep mm-routing opt-in for the default wheel (ai-dynamo#18199): three manifests. lib/bindings/python/Cargo.toml auto-merged (the mm-routing feature now also enables the sglang and trtllm sidecars' mm-routing; both sidecar dependencies gain default-features = false; the split's router dependency list and feature wiring untouched). lib/sidecar/sglang/Cargo.toml auto-merged (upstream's [features] block with default = ["mm-routing"]; the split's dynamo-kv-events dev-dependency stays). lib/sidecar/trtllm/Cargo.toml conflicted on the [dependencies] hunk: resolved as upstream's [features] block and the bare dynamo-llm = { path = "../../llm", default-features = false } line with the SBOM comment dropped as upstream did, keeping the split's dynamo-kv-events = { workspace = true } in place of the pre-split dynamo-kv-router edge; the dev-dependencies dynamo-kv-hashing and dynamo-router-protocol are unchanged. No lockfile change; all four lockfiles pass --locked. The default bindings wheel resolves no dynamo-llm/mm-routing edge and --features mm-routing resolves one, as upstream intends.
- f4a437e ci: move Planner tests out of runtime CPU lanes (ai-dynamo#18351): ten files, auto-merged and taken whole: .github/workflows/dynamo-pipeline.yml and nightly-ci.yml (the new planner-test lane, the "(not planner or core)" marker filter on the runtime CPU lanes), container/Dockerfile.test (COPY of the replay and router retune tests), and pytest.mark.planner or pytest.mark.core added to seven Python test files under components/src/dynamo/planner/tests/unit, lib/bindings/python/tests/replay and tests/dependencies. No Rust, no Cargo.toml, no router-owned path.
- f42e5b1 fix(frontend): keep declared Responses tools in the prompt when allowed_tools restricts calls (ai-dynamo#18429): three lib/llm files, auto-merged and taken whole: lib/llm/src/protocols/openai/responses/template_tools.rs (the renderer reads the full declared tool catalog through template_catalog(); allowed_tools narrows only the backend view and call enforcement), its tests/tool_adaptation.rs, and the new integration test lib/llm/tests/responses_http_replay.rs. No router-owned path, no Cargo.toml.

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

actions ci Issues/PRs that reference CI build/test container planner size/M

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants