Skip to content

perf(prefill-router): a successful decode terminal no longer waits for the prefill finish - #17484

Merged
ishandhanani merged 1 commit into
ai-dynamo:mainfrom
liqiangxl:perf/forward-decode-terminal-early
Nov 3, 2026
Merged

ishandhanani merged 1 commit into
ai-dynamo:mainfrom
liqiangxl:perf/forward-decode-terminal-early

Conversation

@liqiangxl

@liqiangxl liqiangxl commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Issue

Today, in disaggregated serving with a bootstrap handoff (SGLang), decode is dispatched before the prefill request completes. PrefillTask::forward_decode requires both legs to succeed, so it holds a successful decode terminal (the item with finish_reason, which usually also carries the last tokens) until the prefill request returns. That way a late prefill failure can still replace it.

That hold is costly. The SGLang prefill worker replies about 1.5 prefill scheduler iterations after the KV transfer. Short answers finish inside that window, so their last tokens wait for the prefill reply. In an agentic multi-turn benchmark these requests were 94 % of the slowest 10 %, and the per-user output token rate (P90) was 360 tok/s instead of about 470 (with the proposed fix in this PR).

Fix

Don't wait for prefill success.
A prefill failure is no longer a request failure once decode has delivered a successful terminal. Decode cannot succeed without its KV, so a successful decode terminal means the answer is complete and valid. A prefill failure after that point, for example in the prefill reply path, is logged instead of failing the request.

With the new rule there is nothing left to hold for: every decode item, including the terminal, is forwarded as soon as it arrives. The stream still ends only after both legs have finished: decode is read to its end, then the prefill completion is awaited and a failure is discarded. So the request lifetime, cancellation and router bookkeeping stay as they are now; only the delivery of the last tokens and finish_reason moves earlier.

Reading decode to its end before closing also keeps the frontend from killing a worker that is still publishing its completion message. (Dropping the decode stream right after the terminal made the worker log Failed to publish complete final for most requests.)

Tests

  • cargo test -p dynamo-llm --lib prefill_router::handoff: 13 passed. Two existing tests now state the new behaviour: tokens_flow_while_prefill_is_pending_but_close_waits checks that the terminal is available while prefill is pending, that later decode items are forwarded, and that the stream ends only after decode EOF and prefill completion. late_prefill_failure_does_not_follow_decode_success checks that a prefill failure after the terminal is not yielded. The BeforeLast test now also reads the stream to its end.
  • cargo test -p dynamo-llm --lib prefill_router -- --test-threads=1: 56 passed, 4 failed. The 4 failures are environment-blocked (inotify limit, activation timeouts) and also fail on main.
  • cargo clippy -p dynamo-llm --tests: no warnings. cargo fmt --check -p dynamo-llm: clean.

Influence on Benchmark (AIPerf)

AIPerf takes a request's latency from its last content response, not from the end of the stream: request_latency = last content response timestamp - request start. Usage-only chunks, empty chunks and data: [DONE] are excluded. Inter-token latency and per-user output throughput (output_token_throughput_per_user, the basis of P90) are derived from request_latency, so they also stop at the last content chunk.

  • With this change, the last content chunk, which comes from the decode terminal, arrives without waiting for the prefill reply. request_latency, inter-token latency and per-user throughput (P90) get the benefit.
  • data: [DONE], a trailing usage chunk and the HTTP stream end still wait for the prefill reply. This affects AIPerf's full_response_* metrics, the start of a multi-turn session's next request (sent after the previous stream ends) and non-streaming requests.
  • End-to-end, disaggregated SGLang (1 prefill + 6 decode workers, agentic multi-turn workload, one-hour run):
    • P90: 476.0 tok/s, against 359.9 with the hold
    • 0 request errors; golden acceptance length unchanged.
    • Worker Failed to publish complete final: 0 (about 10, before this PR). Prefill background task failed: 0.

Summary by CodeRabbit

  • Bug Fixes
    • Successful decode results can now be delivered before prefill finishes when prefill metadata isn’t requested.
    • When prefill metadata is requested, results still wait for prefill completion. End-of-stream responses continue to report prefill failures.

@liqiangxl
liqiangxl requested a review from a team as a code owner October 28, 2026 03:37
@copy-pr-bot

copy-pr-bot Bot commented Oct 28, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@liqiangxl
liqiangxl temporarily deployed to external_collaborator October 28, 2026 03:37 — with GitHub Actions Inactive
@liqiangxl
liqiangxl temporarily deployed to external_collaborator October 28, 2026 03:37 — with GitHub Actions Inactive
@github-actions github-actions Bot added the perf label Oct 28, 2026
@github-actions

Copy link
Copy Markdown
Contributor

👋 Hi liqiangxl! Thank you for contributing to ai-dynamo/dynamo.

Just a reminder: The NVIDIA Test Github Validation CI runs an essential subset of the testing framework to quickly catch errors.Your PR reviewers may elect to test the changes comprehensively before approving your changes.

🚀

@github-actions github-actions Bot added external-contribution Pull request is from an external contributor router Relates to routing, KV-aware routing, etc. labels Oct 28, 2026
@coderabbitai

coderabbitai Bot commented Oct 28, 2026

Copy link
Copy Markdown
Contributor

Review in Change Stack →

No actionable comments were generated in the recent review.

Recent review info
Run configuration
Commits

Reviewing files that changed from the base of the PR and between 37f2f0a and 96dcd36.

Files selected for processing: 1
  • lib/llm/src/kv_router/prefill_router/handoff.rs

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

Walkthrough

PrefillTask::forward_decode now delivers a successful decode terminal before prefill completes when prefill metadata is not requested. It drains remaining decode items before closing the response and does not forward a late prefill failure. Metadata requests and bare decode EOF still wait for prefill.

Changes

Prefill/decode handoff

Layer / File(s) Summary
Terminal delivery and post-terminal cleanup
lib/llm/src/kv_router/prefill_router/handoff.rs
Successful terminals wait for prefill when metadata is requested, but can be delivered while prefill is pending when it is not. After a terminal, the response drains remaining decode items and awaits prefill before closing. A late prefill failure is not forwarded after success. Tests cover these timing conditions and response closure.

Priority: Normal

Estimated code review effort: 3 (Moderate) | ~20 minutes

Merge Risk: Minimal · up to 96dcd

Early terminal delivery preserves cleanup during normal response consumption. No merge-blocking issue was identified.

Pre-merge checks: Passed 5
Check name Status Explanation
Title check Passed The title clearly and concisely describes the main performance change: successful decode terminals no longer wait for prefill completion.
Description check Passed The description is detailed and relevant. It explains the issue, fix, tests, benchmark impact, and known test failures. It does not use the template headings or include the requested checklist and iss…
Docstring Coverage Passed No eligible functions found in the changed files. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentat…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Description check

Explanation

The description is detailed and relevant. It explains the issue, fix, tests, benchmark impact, and known test failures. It does not use the template headings or include the requested checklist and issue-link section, but the required technical information is mostly present.

Full details: Docstring Coverage

Explanation

No eligible functions found in the changed files. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 0 functions across 1 files.

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

@richardhuo-nv

Copy link
Copy Markdown
Contributor

/ok to test 96dcd36

@richardhuo-nv

Copy link
Copy Markdown
Contributor

LGTM for the first pass. But I will let @ishandhanani , the PIC for router, take a look as well.

@devin-ai-integration

devin-ai-integration Bot commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

✅ Dynamo PR CI passed — run 40916575389 (attempt 1) on ca29b09d6c

Gate checks: ✅ backend-status-check · ✅ deploy-status-check · ✅ dynamo-status-check

Posted automatically by Devin for run 40916575389. Updated on every full-CI run of this PR.

…r the prefill reply

A prefill failure is no longer a request failure once decode has
delivered a successful terminal: decode cannot succeed without its KV,
so the answer is complete and valid, and a later prefill failure (for
example in the prefill reply path) is only logged.

In disaggregated serving with a bootstrap handoff (SGLang), decode is
dispatched before prefill completes, and forward_decode held a
successful decode terminal until the prefill request returned so that a
late prefill failure could replace it. The SGLang prefill reply trails
the KV transfer by about 1.5 prefill iterations, so the last tokens of
short answers waited for it.

Forward every decode item, including the terminal, as it arrives. After
the terminal, read decode to its end and wait for prefill before closing
the stream, discarding a prefill failure, so the frontend does not kill
a worker that is still publishing its completion message. BeforeFirst
and BeforeLast still order prefill output before the terminal, and a
decode EOF without a terminal still returns a prefill failure.

Signed-off-by: Liqiang Lu <liqiangl@nvidia.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@liqiangxl
liqiangxl force-pushed the perf/forward-decode-terminal-early branch from 96dcd36 to ca29b09 Compare October 29, 2026 16:36
@liqiangxl
liqiangxl temporarily deployed to external_collaborator October 29, 2026 16:36 — with GitHub Actions Inactive
@richardhuo-nv

Copy link
Copy Markdown
Contributor

/ok to test ca29b09

@weireweire

Copy link
Copy Markdown
Contributor

could we prioritize this? It will affect infX perf.

@ishandhanani
ishandhanani merged commit 2ec0b11 into ai-dynamo:main Nov 3, 2026
117 checks passed
ishandhanani added a commit to ishandhanani/dynamo that referenced this pull request Nov 4, 2026
Absorbs the 95 upstream commits since 464ff29. Each upstream change lands in the crate that owns that code at the stack head; the stack's moves stay intact. Router-touching PRs and where they landed:

- ai-dynamo#17712 host-path routing modes for worker-selection plugins: &self + Send + Sync plugin traits and WorkerInputs::ACTIVE_REQUESTS (deprecated OCCUPANCY alias) in dynamo-worker-selection and dynamo-router-protocol; HostedWorkerPicker and HostedSelectionRequest in dynamo-worker-selection (hosted.rs), reporting SelectionFailure, which the kernel converts into KvSchedulerError; the InstanceSelector adapter in lib/llm (HostedInstanceSelector); the compat test's hosted coverage in kv-router's prelude test.
- ai-dynamo#17713 random/round-robin/least-loaded as builtin plugins: router_modes in dynamo-router-policies (renamed from dynamo-custom-policy-builtin) importing the prelude; RouterPluginRegistry::resolve_type in kv-router; the frontend's hosted path through dynamo_worker_selection::host.
- ai-dynamo#17899 1.7.0 removals: taken whole; the alias and the setter were gone at the head; power-of-two never arrives.
- ai-dynamo#17657 single-source fetch continuations: KvTransferSourceInterval and source_intervals in dynamo-kv-indexer, the bucketed summary in dynamo-kv-hints, the interval filter in the kernel's resolver; tests in the kernel.
- ai-dynamo#17856 KVCR hints in offline replay: mocker and lib/llm bodies; supports_kv_transfer_chain_retention on the flat config; its indexer-vs-config test in kv-router's core host tests.
- ai-dynamo#17794 vLLM partial-tail filter and ai-dynamo#17126 SGLang engine-owned multimodal pads: dynamo-kv-events zmq_wire (mm_exclusion.rs); the codec-plus-indexer tests in dynamo-kv-indexer; the engine_owned_mm_identity worker flag through the kernel's WorkerRequest, kv-router's indexer registry and run config, and the bindings.
- ai-dynamo#17653 SGLang 0.5.23: pad constant docs in dynamo-multimodal.
- ai-dynamo#15333, ai-dynamo#17874, ai-dynamo#17484, ai-dynamo#17688, ai-dynamo#17237, ai-dynamo#17894, ai-dynamo#17958: host-side changes merged in place (bindings, lib/llm, CI).

Lockfiles regenerated; no shim reintroduced.

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>

This branch was previously deployed

1 inactive deployment
external_collaborator — ca29b09d Deployed Oct 29, 2026 by liqiangxl via ok-to-test #41471
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

external-contribution Pull request is from an external contributor perf router Relates to routing, KV-aware routing, etc. size/M

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants