Repository navigation
perf(prefill-router): a successful decode terminal no longer waits for the prefill finish - #17484
Conversation
|
👋 Hi liqiangxl! Thank you for contributing to ai-dynamo/dynamo. Just a reminder: The 🚀 |
|
No actionable comments were generated in the recent review.
|
| Layer / File(s) | Summary |
|---|---|
Terminal delivery and post-terminal cleanup lib/llm/src/kv_router/prefill_router/handoff.rs |
Successful terminals wait for prefill when metadata is requested, but can be delivered while prefill is pending when it is not. After a terminal, the response drains remaining decode items and awaits prefill before closing. A late prefill failure is not forwarded after success. Tests cover these timing conditions and response closure. |
Priority:
Estimated code review effort: 3 (Moderate) | ~20 minutes
Merge Risk: 
Minimal · up to 96dcd
-
Early terminal delivery preserves cleanup during normal response consumption. No merge-blocking issue was identified.
Pre-merge checks: 
5
-
Full details: Description check
-
Explanation
The description is detailed and relevant. It explains the issue, fix, tests, benchmark impact, and known test failures. It does not use the template headings or include the requested checklist and issue-link section, but the required technical information is mostly present.
Full details: Docstring Coverage
-
Explanation
No eligible functions found in the changed files. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 0 functions across 1 files.
-
- Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Comment @coderabbitai help to get the list of available commands.
|
/ok to test 96dcd36 |
|
LGTM for the first pass. But I will let @ishandhanani , the PIC for router, take a look as well. |
✅ Dynamo PR CI passed — run 40916575389 (attempt 1) on
|
…r the prefill reply A prefill failure is no longer a request failure once decode has delivered a successful terminal: decode cannot succeed without its KV, so the answer is complete and valid, and a later prefill failure (for example in the prefill reply path) is only logged. In disaggregated serving with a bootstrap handoff (SGLang), decode is dispatched before prefill completes, and forward_decode held a successful decode terminal until the prefill request returned so that a late prefill failure could replace it. The SGLang prefill reply trails the KV transfer by about 1.5 prefill iterations, so the last tokens of short answers waited for it. Forward every decode item, including the terminal, as it arrives. After the terminal, read decode to its end and wait for prefill before closing the stream, discarding a prefill failure, so the frontend does not kill a worker that is still publishing its completion message. BeforeFirst and BeforeLast still order prefill output before the terminal, and a decode EOF without a terminal still returns a prefill failure. Signed-off-by: Liqiang Lu <liqiangl@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
96dcd36 to
ca29b09
Compare
|
/ok to test ca29b09 |
|
could we prioritize this? It will affect infX perf. |
Absorbs the 95 upstream commits since 464ff29. Each upstream change lands in the crate that owns that code at the stack head; the stack's moves stay intact. Router-touching PRs and where they landed: - ai-dynamo#17712 host-path routing modes for worker-selection plugins: &self + Send + Sync plugin traits and WorkerInputs::ACTIVE_REQUESTS (deprecated OCCUPANCY alias) in dynamo-worker-selection and dynamo-router-protocol; HostedWorkerPicker and HostedSelectionRequest in dynamo-worker-selection (hosted.rs), reporting SelectionFailure, which the kernel converts into KvSchedulerError; the InstanceSelector adapter in lib/llm (HostedInstanceSelector); the compat test's hosted coverage in kv-router's prelude test. - ai-dynamo#17713 random/round-robin/least-loaded as builtin plugins: router_modes in dynamo-router-policies (renamed from dynamo-custom-policy-builtin) importing the prelude; RouterPluginRegistry::resolve_type in kv-router; the frontend's hosted path through dynamo_worker_selection::host. - ai-dynamo#17899 1.7.0 removals: taken whole; the alias and the setter were gone at the head; power-of-two never arrives. - ai-dynamo#17657 single-source fetch continuations: KvTransferSourceInterval and source_intervals in dynamo-kv-indexer, the bucketed summary in dynamo-kv-hints, the interval filter in the kernel's resolver; tests in the kernel. - ai-dynamo#17856 KVCR hints in offline replay: mocker and lib/llm bodies; supports_kv_transfer_chain_retention on the flat config; its indexer-vs-config test in kv-router's core host tests. - ai-dynamo#17794 vLLM partial-tail filter and ai-dynamo#17126 SGLang engine-owned multimodal pads: dynamo-kv-events zmq_wire (mm_exclusion.rs); the codec-plus-indexer tests in dynamo-kv-indexer; the engine_owned_mm_identity worker flag through the kernel's WorkerRequest, kv-router's indexer registry and run config, and the bindings. - ai-dynamo#17653 SGLang 0.5.23: pad constant docs in dynamo-multimodal. - ai-dynamo#15333, ai-dynamo#17874, ai-dynamo#17484, ai-dynamo#17688, ai-dynamo#17237, ai-dynamo#17894, ai-dynamo#17958: host-side changes merged in place (bindings, lib/llm, CI). Lockfiles regenerated; no shim reintroduced. Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Issue
Today, in disaggregated serving with a bootstrap handoff (SGLang), decode is dispatched before the prefill request completes.
PrefillTask::forward_decoderequires both legs to succeed, so it holds a successful decode terminal (the item withfinish_reason, which usually also carries the last tokens) until the prefill request returns. That way a late prefill failure can still replace it.That hold is costly. The SGLang prefill worker replies about 1.5 prefill scheduler iterations after the KV transfer. Short answers finish inside that window, so their last tokens wait for the prefill reply. In an agentic multi-turn benchmark these requests were 94 % of the slowest 10 %, and the per-user output token rate (P90) was 360 tok/s instead of about 470 (with the proposed fix in this PR).
Fix
Don't wait for prefill success.
A prefill failure is no longer a request failure once decode has delivered a successful terminal. Decode cannot succeed without its KV, so a successful decode terminal means the answer is complete and valid. A prefill failure after that point, for example in the prefill reply path, is logged instead of failing the request.
With the new rule there is nothing left to hold for: every decode item, including the terminal, is forwarded as soon as it arrives. The stream still ends only after both legs have finished: decode is read to its end, then the prefill completion is awaited and a failure is discarded. So the request lifetime, cancellation and router bookkeeping stay as they are now; only the delivery of the last tokens and
finish_reasonmoves earlier.Reading decode to its end before closing also keeps the frontend from killing a worker that is still publishing its completion message. (Dropping the decode stream right after the terminal made the worker log
Failed to publish complete finalfor most requests.)Tests
cargo test -p dynamo-llm --lib prefill_router::handoff: 13 passed. Two existing tests now state the new behaviour:tokens_flow_while_prefill_is_pending_but_close_waitschecks that the terminal is available while prefill is pending, that later decode items are forwarded, and that the stream ends only after decode EOF and prefill completion.late_prefill_failure_does_not_follow_decode_successchecks that a prefill failure after the terminal is not yielded. TheBeforeLasttest now also reads the stream to its end.cargo test -p dynamo-llm --lib prefill_router -- --test-threads=1: 56 passed, 4 failed. The 4 failures are environment-blocked (inotify limit, activation timeouts) and also fail on main.cargo clippy -p dynamo-llm --tests: no warnings.cargo fmt --check -p dynamo-llm: clean.Influence on Benchmark (AIPerf)
AIPerf takes a request's latency from its last content response, not from the end of the stream:
request_latency = last content response timestamp - request start. Usage-only chunks, empty chunks anddata: [DONE]are excluded. Inter-token latency and per-user output throughput (output_token_throughput_per_user, the basis of P90) are derived fromrequest_latency, so they also stop at the last content chunk.request_latency, inter-token latency and per-user throughput (P90) get the benefit.data: [DONE], a trailing usage chunk and the HTTP stream end still wait for the prefill reply. This affects AIPerf'sfull_response_*metrics, the start of a multi-turn session's next request (sent after the previous stream ends) and non-streaming requests.Failed to publish complete final: 0 (about 10, before this PR).Prefill background task failed: 0.Summary by CodeRabbit