Skip to content

ci: split feature-gated Rust builds into a parallel pre-merge job - #17958

Merged
nv-anants merged 1 commit into
mainfrom
anants/split-rust-feature-builds
Nov 3, 2026
Merged

nv-anants merged 1 commit into
mainfrom
anants/split-rust-feature-builds

Conversation

@nv-anants

@nv-anants nv-anants commented Nov 3, 2026 •

Copy link
Copy Markdown
Member

Move the isolated kv-router checks, standalone slot-tracker tests, canonical replay tests, and feature-gated benchmark compiles out of rust-tests into a new rust-feature-builds job. Each feature set builds a separate artifact tree, so running them after the full workspace test build exhausted runner disk; running them on their own runner removes that pressure and takes ~10 min off the rust-tests critical path.

Add a setup-rust-ci composite action that installs protoc, restores the cargo cache, and adds rustup components, replacing the inline copies in pre-merge rust-tests, rust-feature-builds, rust-clippy, and nightly rust-tests. Add the action to the rust change filter so edits to it run the Rust jobs.

Summary by CodeRabbit

  • Chores
    • Updated automated Rust checks to use a consistent setup across nightly and pre-merge workflows.
    • Added a separate set of checks for Rust feature builds, including isolated builds, tests, and benchmark compilation.
    • Pre-merge status now accounts for the feature-build checks.

Move the isolated kv-router checks, standalone slot-tracker tests,
canonical replay tests, and feature-gated benchmark compiles out of
rust-tests into a new rust-feature-builds job. Each feature set builds a
separate artifact tree, so running them after the full workspace test
build exhausted runner disk; running them on their own runner removes
that pressure and takes ~10 min off the rust-tests critical path.

Add a setup-rust-ci composite action that installs protoc, restores the
cargo cache, and adds rustup components, replacing the inline copies in
pre-merge rust-tests, rust-feature-builds, rust-clippy, and nightly
rust-tests. Add the action to the rust change filter so edits to it run
the Rust jobs.

Signed-off-by: Anant Sharma <anants@nvidia.com>
@nv-anants
nv-anants requested a review from a team as a code owner November 3, 2026 17:51
@coderabbitai

coderabbitai Bot commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

Walkthrough

The pull request adds a shared Rust CI setup action and uses it in nightly and pre-merge workflows. Pre-merge CI also adds a dedicated job for Rust feature checks and a Python AIS feature test.

Changes

Rust CI workflows

Layer / File(s) Summary
Shared Rust setup
.github/actions/setup-rust-ci/action.yml, .github/filters.yaml, .github/workflows/nightly-ci.yml, .github/workflows/pre-merge.yml
The composite action installs and verifies protoc, caches Cargo data, and adds optional rustup components. Nightly tests, pre-merge tests, and Clippy use the action. The Rust file filter includes the action.
Pre-merge feature checks
.github/workflows/pre-merge.yml
The pre-merge status job requires rust-feature-builds. That job runs the kv-router, slot-tracker, canonical replay, and benchmark checks. The rust-tests job adds the Python AIS feature test.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Merge Risk: Minimal · up to 96cb8

The Rust feature checks now run separately; no material merge-blocking risk is established.

Pre-merge checks: Passed 5
Check name Status Explanation
Title check Passed The title names the main change: feature-gated Rust builds are split into a separate job that runs in parallel in pre-merge. The raw summary confirms the new rust-feature-builds job and the removal of…
Description check Passed The description covers the main parts of the template. It explains the key changes: the new rust-feature-builds job, the reason for the split (runner disk exhaustion and the roughly 10 minute reductio…
Docstring Coverage Passed No eligible functions found in the changed files. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentat…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Title check

Explanation

The title names the main change: feature-gated Rust builds are split into a separate job that runs in parallel in pre-merge. The raw summary confirms the new rust-feature-builds job and the removal of those checks from rust-tests. The title does not mention the setup-rust-ci composite action, but that is a secondary change.

Full details: Description check

Explanation

The description covers the main parts of the template. It explains the key changes: the new rust-feature-builds job, the reason for the split (runner disk exhaustion and the roughly 10 minute reduction on the rust-tests critical path), and the new setup-rust-ci composite action with its filter change. It does not include the template's Overview, Validation, or Issues sections as headings, and it has no validation steps, no linked issue (the Closes #XXXX line is missing), and no Pull Request Checklist entry. These gaps are non-critical. The description is mostly complete.

Full details: Docstring Coverage

Explanation

No eligible functions found in the changed files. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 0 functions across 0 files. (4 skipped: 4 unsupported.)

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Tip

🎯 Customize your risk assessments

Guide how CodeRabbit assesses merge risk and security architecture risk. Highlight sensitive changes and explain your security boundaries using Merge Risk Instructions and Security Risk Instructions in your review settings. Security architecture risk assessments require the Advanced plan or above.

✅ Automatic approvals, with your rules

Let CodeRabbit approve pull requests once required threads are resolved, pre-merge checks pass, and your risk limits are met. Enable Automatic approvals in review settings, with overrides for teams or file paths.

CodeRabbit Verify

  • Run CodeRabbit Verify to test the changes in this pull request.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Additional comments: 1
.github/workflows/pre-merge.yml-499-501: 1

499-501: Minor · Security & Privacy · Quick win

Unverified finding
Verification ran but could not confirm this finding. It is shown for review, not as a verified issue.

Disable persisted checkout credentials in the new build job.

When rust-feature-builds runs Cargo, dependency build scripts can use the checkout credential because persist-credentials remains enabled. The job needs only contents: read, but it does not need authenticated Git commands after checkout. Set persist-credentials: false. (github.com)

Based on learnings, jobs that run dependency build code should disable persisted checkout credentials.

Sources: Learnings, Linters/SAST tools

Review info
Run configuration
Commits

Reviewing files that changed from the base of the PR and between 3aafbef and 96cb8d2.

Files selected for processing: 4
  • .github/actions/setup-rust-ci/action.yml
  • .github/filters.yaml
  • .github/workflows/nightly-ci.yml
  • .github/workflows/pre-merge.yml

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

@github-actions github-actions Bot added ci Issues/PRs that reference CI build/test actions labels Nov 3, 2026
@nv-rinig

nv-rinig commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

from claude:

Datadog CI (rust-tests (.), last 7 days: 728 success / 149 error / 296 canceled):

  • 24 of 149 errors mention "space" (e.g. "No space left on device. Free space left: 62 MB").
  • Median success is ~45 min. This PR removes ~650s (benchmark compile 501s + feature tests ~145s) from it.

Questions:

  1. Do the seven benchmark builds in rust-feature-builds share one target/ dir? If so, could it hit the same disk limit on Fastchecker (which doesn't seem to be in Velonix, so can't check disk limit)?

@devin-ai-integration

Copy link
Copy Markdown
Contributor

⏹️ Dynamo PR CI cancelled — run 41688205406 (attempt 1) on 96cb8d2e72

Gate checks: ✅ backend-status-check · ❌ deploy-status-check · ✅ dynamo-status-check

Framework Deploy
vLLM ⏹️ 2 ✅ 2

Failure details

No jobs failed. 2 vLLM deploy tests were cancelled because they hit the job's 25-minute limit, not by fail-fast or a manual cancel. This made deploy-status-check fail.

  • ⏹️ vllm Deploy Test / agg_router: cancelled in Run Dynamo Deploy Test after "The job has exceeded the maximum execution time of 25m0s". Just before that, all 3 connection tests to frontend pod vllm-agg-router-0-frontend-* failed with Read timed out. (read timeout=5).
  • ⏹️ vllm Deploy Test / agg: same pattern. The job hit the 25-minute limit in Run Dynamo Deploy Test after all 3 connection tests to vllm-agg-0-frontend-* timed out.
For agents
{"pr": 17958, "run_id": 41688205406, "run_attempt": 1, "head_sha": "96cb8d2e72822da2272e48d80f96b9185aee907f",
 "failures": [],
 "cancelled": [{"job": "vllm Deploy Test / agg_router", "job_id": 126125835071, "step": "Run Dynamo Deploy Test", "signature": "job exceeded 25m timeout; frontend connection test read timed out", "log_cmd": "gh api repos/ai-dynamo/dynamo/actions/jobs/126125835071/logs"},
               {"job": "vllm Deploy Test / agg", "job_id": 126125835170, "step": "Run Dynamo Deploy Test", "signature": "job exceeded 25m timeout; frontend connection test read timed out", "log_cmd": "gh api repos/ai-dynamo/dynamo/actions/jobs/126125835170/logs"}]}

Posted automatically by Devin for run 41688205406. Updated on every full-CI run of this PR.

@nv-anants

Copy link
Copy Markdown
Member Author
  1. Do the seven benchmark builds in rust-feature-builds share one target/ dir? If so, could it hit the same disk limit on Fastchecker (which doesn't seem to be in Velonix, so can't check disk limit)?

Yes. All the cargo invocations in rust-feature-builds run in one job with the default target/, so they share one target dir. But that dir is now much smaller than it used to be. Before, these builds stacked on top of the full workspace test build in rust-tests (.) (cargo test --no-run + docs + --all-targets), and that full build is what used most of the disk. In the new job, target/ holds only the dynamo-kv-router and dynamo-bench feature builds plus kv_router_bench.

@nv-anants
nv-anants enabled auto-merge (squash) November 3, 2026 19:31
@nv-anants
nv-anants merged commit 23ff72c into main Nov 3, 2026
228 of 232 checks passed
@nv-anants
nv-anants deleted the anants/split-rust-feature-builds branch November 3, 2026 19:44
ishandhanani added a commit to ishandhanani/dynamo that referenced this pull request Nov 4, 2026
Absorbs the 95 upstream commits since 464ff29. Each upstream change lands in the crate that owns that code at the stack head; the stack's moves stay intact. Router-touching PRs and where they landed:

- ai-dynamo#17712 host-path routing modes for worker-selection plugins: &self + Send + Sync plugin traits and WorkerInputs::ACTIVE_REQUESTS (deprecated OCCUPANCY alias) in dynamo-worker-selection and dynamo-router-protocol; HostedWorkerPicker and HostedSelectionRequest in dynamo-worker-selection (hosted.rs), reporting SelectionFailure, which the kernel converts into KvSchedulerError; the InstanceSelector adapter in lib/llm (HostedInstanceSelector); the compat test's hosted coverage in kv-router's prelude test.
- ai-dynamo#17713 random/round-robin/least-loaded as builtin plugins: router_modes in dynamo-router-policies (renamed from dynamo-custom-policy-builtin) importing the prelude; RouterPluginRegistry::resolve_type in kv-router; the frontend's hosted path through dynamo_worker_selection::host.
- ai-dynamo#17899 1.7.0 removals: taken whole; the alias and the setter were gone at the head; power-of-two never arrives.
- ai-dynamo#17657 single-source fetch continuations: KvTransferSourceInterval and source_intervals in dynamo-kv-indexer, the bucketed summary in dynamo-kv-hints, the interval filter in the kernel's resolver; tests in the kernel.
- ai-dynamo#17856 KVCR hints in offline replay: mocker and lib/llm bodies; supports_kv_transfer_chain_retention on the flat config; its indexer-vs-config test in kv-router's core host tests.
- ai-dynamo#17794 vLLM partial-tail filter and ai-dynamo#17126 SGLang engine-owned multimodal pads: dynamo-kv-events zmq_wire (mm_exclusion.rs); the codec-plus-indexer tests in dynamo-kv-indexer; the engine_owned_mm_identity worker flag through the kernel's WorkerRequest, kv-router's indexer registry and run config, and the bindings.
- ai-dynamo#17653 SGLang 0.5.23: pad constant docs in dynamo-multimodal.
- ai-dynamo#15333, ai-dynamo#17874, ai-dynamo#17484, ai-dynamo#17688, ai-dynamo#17237, ai-dynamo#17894, ai-dynamo#17958: host-side changes merged in place (bindings, lib/llm, CI).

Lockfiles regenerated; no shim reintroduced.

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
sachalmalick added a commit that referenced this pull request Nov 4, 2026
One conflict, in the `needs` list of `pre-merge-status-check` in
`.github/workflows/pre-merge.yml`: main added `rust-feature-builds`
(#17958) and this branch had added `rust-affinity-store-tests`, both
before `rust-clippy`. Both are kept here, in job order; the job blocks
themselves merged cleanly.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Sachal Malick <s@chal.ai>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

actions ci Issues/PRs that reference CI build/test size/L

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants