Skip to content

[CI] Enroll 12 steps in automatic sharding - #68257

Open
Thangnguyenvn98 wants to merge 5 commits into
vllm-project:mainfrom
Thangnguyenvn98:agent/automatic-shard-new-steps-1117
Open

Thangnguyenvn98 wants to merge 5 commits into
vllm-project:mainfrom
Thangnguyenvn98:agent/automatic-shard-new-steps-1117

Conversation

@Thangnguyenvn98

@Thangnguyenvn98 Thangnguyenvn98 commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Overview

Enrolls twelve steps that have never been enrolled, most added to .buildkite/test_areas/ since the last enrollment sweep, with the AMD mirrors of the four that have one, so they don't stay on the not-enrolled list:

Step Added in Planned shards
ShardQ Metadata #67338 1
Weight Reload Refit #65347 1
[Nightly] Weight Reload Refit Extra Cases #65347 1
[Nightly] Weight Reload Refit (2 GPUs) #65347 1
[Nightly] Multimodal FlashInfer Autotuning #65901 1
Mistral Large 4 (+ AMD mirror) #64534 1 (+1)
ShardQ Sequence Parallel #66549 1
ReplaySSM E2E TP2 (+ AMD mirror) #67553 1 (+1)
[Nightly] LM Eval ReplaySSM (+ AMD mirror) #55688 1 (+1)
Rust Frontend Chat Completions (+ AMD mirror) #66781 1 (+1)
[Nightly] DeepSeek-V4.1 SM80 and SM90 Decode (8 GPUs) #66497 1
[Nightly] Elastic EP Scaling from #61476 2

Elastic EP Scaling is also in the stale draft #61476, whose other four steps are already enrolled on main; no other open PR enrolls these. Every other NVIDIA job the sharding page lists as not enrolled is in an open PR, runs on several nodes, or has commands that aren't plain pytest (scripts, torchrun), which the generator can't shard.

Claims

  • Elastic EP Scaling (about 23 minutes as one job on main) runs as 2 shards, each test exactly once; the rest fit in one job today and still run as one job with the same tests, sharding on their own once their tests outgrow the 20-minute target.
  • No test is added, removed or changed.

Validation

  • Planner run on this branch with main's timings: one shard each, except Elastic EP Scaling at 2 (13.6 and 9.5 min, +3% machine time). Sharding harness for Elastic EP Scaling, as a PR build and as a main build: each test runs in exactly one shard. Each step and mirror passed on its recent main runs and the last nightly.
  • /ci run 103643 on the first five (head 4f9cf81a21) passed with no retries: same node IDs as main 103629 and nightly 103594, 0 missing, added, changed or duplicated, machine time within noise.
  • New /ci run on this head, plus a scoped build for any mirror it leaves out: results in a comment.

Details

  • Timeouts are unchanged.

Pull Request Checklist
  • I used vLLM's /pr-checklist skill. (Mandatory for agents, optional for humans).

  • AI assistance was used during the creation of this PR.

  • Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.

  • Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.

  • Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.

  • Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.

🤖 Generated with Claude Code

@Thangnguyenvn98

Copy link
Copy Markdown
Contributor Author

/ci run

@github-actions

Copy link
Copy Markdown

✅ Triggered Buildkite CI #103643 for commit 4f9cf81a2186.

@mergify mergify Bot added the ci/build label Nov 17, 2026
@Thangnguyenvn98

Copy link
Copy Markdown
Contributor Author

Build note: #103643 (/ci run, head 4f9cf81a21)

Passed, no automatic retries. The selector picked all five steps, nightly ones included, so no scoped build was needed. Each ran as one job, as planned.

Step Slowest Node IDs vs main Machine
shardq-metadata-b200 0.7 min 1 = 1 vs 103629, 0 diff +0.2 min
weight-reload-refit 13.7 min 83 = 83 vs 103629, 0 diff -1%
weight-reload-refit-nightly 8.3 min 4 = 4 vs nightly 103594, 0 diff -1%
weight-reload-refit-2gpu 18.4 min 5 = 5 vs nightly 103594, 0 diff -3%
multi-modal-flashinfer-autotuning 5.8 min 3 = 3 vs nightly 103594, 0 diff -5%
  • ShardQ Metadata's one test is skipped on this runner, as on main.

@Thangnguyenvn98
Thangnguyenvn98 marked this pull request as ready for review November 18, 2026 00:17
@Thangnguyenvn98
Thangnguyenvn98 requested a review from a team November 18, 2026 00:17

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@Thangnguyenvn98
Thangnguyenvn98 force-pushed the agent/automatic-shard-new-steps-1117 branch from 4f9cf81 to 2bea5cd Compare November 18, 2026 04:22
@Thangnguyenvn98 Thangnguyenvn98 changed the title [CI] Enroll 5 newly added steps in automatic sharding [CI] Enroll 10 newly added steps in automatic sharding Nov 18, 2026
@Thangnguyenvn98

Copy link
Copy Markdown
Contributor Author

/ci run

@mergify mergify Bot added the rust label Nov 18, 2026
@github-actions

Copy link
Copy Markdown

✅ Triggered Buildkite CI #103741 for commit 0a5b24684424.

ℹ️ This PR is 1 commit behind upstream main. Merged upstream main into this branch as 0a5b24684424 before starting CI. Pull the branch before pushing more changes.

Thangnguyenvn98 and others added 3 commits November 17, 2026 23:23
ShardQ Metadata, the three Weight Reload Refit steps and Multimodal FlashInfer
Autotuning are new on main and each fits in one job today, so the generator
plans one shard for each; they shard on their own when their tests grow.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
… E2E TP2, LM Eval ReplaySSM and Rust Frontend Chat Completions

Added to main between 11-04 and 11-13 and never enrolled. Each fits in one
job today; the AMD mirrors of the four that have one go with them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
…atic sharding

Nightly-only, plain pytest on one node; fits in one job today.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
@Thangnguyenvn98
Thangnguyenvn98 force-pushed the agent/automatic-shard-new-steps-1117 branch from 0a5b246 to bb9038b Compare November 18, 2026 04:23
@Thangnguyenvn98 Thangnguyenvn98 changed the title [CI] Enroll 10 newly added steps in automatic sharding [CI] Enroll 11 newly added steps in automatic sharding Nov 18, 2026
@Thangnguyenvn98

Copy link
Copy Markdown
Contributor Author

/ci run

@github-actions

Copy link
Copy Markdown

✅ Triggered Buildkite CI #103743 for commit bb9038b5ec35.

Runs about 23 minutes as one job on main; planned as 2 shards from main's
timings, splitting test_elastic_ep.py by test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
@Thangnguyenvn98

Copy link
Copy Markdown
Contributor Author

/ci run

@Thangnguyenvn98 Thangnguyenvn98 changed the title [CI] Enroll 11 newly added steps in automatic sharding [CI] Enroll 12 steps in automatic sharding Nov 18, 2026
@github-actions

Copy link
Copy Markdown

✅ Triggered Buildkite CI #103757 for commit e095525202e4.

ℹ️ This PR is 6 commits behind upstream main. Merged upstream main into this branch as e095525202e4 before starting CI. Pull the branch before pushing more changes.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants