Skip to content

feat(sglang): join native generate responses across P/D and stream chunks - #1

Draft
Aphoh wants to merge 3 commits into
mainfrom
warnold/sglang-generate-merge
Draft

Aphoh wants to merge 3 commits into
mainfrom
warnold/sglang-generate-merge

Conversation

@Aphoh

@Aphoh Aphoh commented Oct 13, 2026 •

Copy link
Copy Markdown
Owner

Summary

SGLang /generate through Dynamo now supports stream=false. In P/D, it also returns the outputs that only the prefill worker computes. The change uses the existing /generate path (the token pipeline). It adds no second HTTP path.

What problem this fixes

  • Unary requests: before this change, stream=false returned HTTP 501.
  • P/D prompt logprobs: the decode worker never runs the prompt, so P/D responses had empty prompt logprobs.
  • P/D routed experts: the prompt rows in routed_experts came from decode and were wrong. The blob length was correct, so the error was not visible.

How it works

  1. One trait for both joins. ResponseMerger (lib/llm/src/protocols/response_merge.rs) joins the pieces of one response:
    • join_step joins the streamed chunks of one request.
    • join_prefill and restore_prefill join prefill's output into a decode response or chunk.
    • prefill_need reports whether a request needs prefill's output, and how early.
  2. Rules are data. RuleSet implements the trait from a list of FieldRule values (path, op, drop_if_missing). This list is the intended plugin format. There are three join operations:
    • append joins strings or lists.
    • take_prefill uses prefill's value.
    • overlay_base64_prefix puts prefill's bytes at the start of decode's bytes.
  3. SGLang rules. lib/llm/src/protocols/sglang/merge.rs holds the SGLang rules. They follow SGLang's own merge code.
    • Steps: append text, output_ids, and the five output-side lists. Every other field takes the value from the last chunk.
    • P/D: take prefill's prompt logprob fields. Put prefill's prompt rows into routed_experts and indexer_topk. If prefill sends no rows, remove the field, so that wrong rows never reach the client.
  4. Unary requests. The worker still streams from SGLang. The frontend joins the chunks into the response that SGLang returns for stream=false.
  5. How prefill's output reaches the frontend.
    • Frontend: it adds the prefill_output annotation only when the request needs prefill's output. It adds prefill_output_first only for streamed requests where every chunk needs prefill's values (prompt logprobs).
    • Prefill worker: when it sees the annotation, the Rust sidecar keeps its final native response. The Python worker is not changed.
    • Prefill router: it moves that response into the decode stream as one item. With prefill_output_first, the item comes before the first decode output. Otherwise it comes as soon as prefill finishes, and always before the final decode output.

Effect on TTFT

  • Most requests do not wait for prefill. This includes plain requests and requests with return_logprob only, which gives output logprobs.
  • Prompt logprobs wait for prefill. These are requests with return_logprob and a non-negative logprob_start_len. Every streamed chunk must carry the prompt logprobs, so the first chunk waits for prefill.
  • Routed experts do not hold decode tokens. Only the final chunk carries routed experts, so decode tokens are sent as they arrive.

Host overhead

  • Merges do not copy values. They move or swap them. Streaming P/D swaps prefill's values into each chunk, serializes the chunk, and then swaps them back.
  • The overlay works in place. It changes decode's base64 text directly and decodes only the 4-character group where prefill's bytes end. The code has no unsafe.
  • Streaming uses one buffer. Streaming /generate serializes every chunk into one buffer that it reuses. This also makes requests that do not use these fields faster than on main.

Known cost: long prompts with routed experts

routed_experts grows with the prompt length × layers × top-k × 4 bytes. For Qwen3-30B-A3B, that is about 2 KB of base64 for each token. These costs happen once for each request, after the last token:

  • Overlay: about 0.5 ms for a 5 MB blob, which is a 2,048-token prompt. The cost grows linearly with the blob size.
  • Transport: the prefill response is sent once more over the request plane. The final response is serialized once.
  • Size limit: the sidecar reads SSE events of up to 64 MiB (MAX_EVENT_BYTES). With very long prompts, the routed_experts event can be larger than this limit. This limit also applies on main to decode's final chunk.

Compatibility

Validation

Unit tests

  • cargo test -p dynamo-llm --lib: 88 targeted tests pass. They cover the rules, the overlay for every boundary length, the stream adapter, unary dispatch, and the router's two prefill output modes.
  • cargo test -p dynamo-sglang-sidecar --lib: 109 pass.
  • cargo clippy -D warnings passes for dynamo-llm (lib, tests, benches) and for the sidecar. cargo fmt and pre-commit pass.

CPU benchmark

cargo bench -p dynamo-llm --bench response_merge measures time and allocations for each scenario. It uses Qwen3-30B-A3B shapes: a 2,048-token prompt, 512 output tokens, and a 5 MB routed_experts blob. "Work it cannot avoid" means freeing the input chunks, or serializing bytes that must be sent anyway.

Scenario Merge Work it cannot avoid Allocations
Unary aggregated, one chunk 68 µs 69 µs (freeing the chunk) none added
Unary P/D, one chunk 0.95 µs n/a 0
Unary P/D, after the last token (incl. overlay) 0.48 ms n/a 0
Streaming P/D, whole request 45.9 ms 49.7 ms (serializing joined chunks) same as baseline
Short streaming request, no merged fields 35.7 µs 49.2 µs on main fewer than main

End-to-end test on GPU

  • Setup: Qwen3-30B-A3B, one B200 GPU, --moe-runner-backend triton --enable-return-routed-experts --enable-deterministic-inference. SGLang is the nightly build 7224fce1 with #44002 applied. The Rust sidecar and the Dynamo frontend are built from this branch.
  • Correctness: 148 checks pass and 0 fail. The test compares Dynamo stream=false and Dynamo stream=true (joined by the test) with stock SGLang stream=false, in aggregated and in P/D mode. The output ids, text, finish reason, token counts, and logprobs (largest difference 4.4e-16) match. routed_experts is byte-identical in both modes.
  • TTFT: median of 20 streaming requests, 2,048-token prompts.
Request Aggregated P/D
plain 54.3 ms 61.5 ms
output logprobs only +0.1 ms +1.5 ms
prompt logprobs +10.2 ms +11.0 ms
routed experts +0.2 ms +2.3 ms
  • Prompt logprobs: the extra 10 ms is the same in both modes. SGLang uses this time to compute the prompt logprobs.
  • Routed experts in P/D: before the fix in the second commit, the first decode token waited for prefill, which added +72 ms. The cause of the remaining +2.3 ms is not known yet.

Not covered yet

  • No repository e2e test covers unary /generate, P/D /generate, or routed experts. A pytest e2e test with Qwen/Qwen3-0.6B is next.
  • Client cancellation of unary requests was not tested end to end.
  • P/D routed experts without deterministic inference were not tested.

Generated with Devin

…unks

Native /generate could only stream, and in P/D it dropped every output that
only prefill computes: decode never runs the prompt, so input logprobs came
back empty and routed experts came back with stale prompt rows.

Add an engine-neutral ResponseMerger trait with one merge operation for both
seams where a response arrives in pieces: inter-token chunks (Steps) and
prefill/decode (PrefillDecode). RuleSet is the declarative implementation
(seam, JSON pointer, join op, on-missing policy) and the intended plugin
format; SglangMerger carries SGLang's rules.

- stream=false is served by streaming from SGLang and joining the chunks in
  the frontend, following SGLang's own _coalesce_streaming_chunks.
- Requests that ask for prompt-side output carry a prefill_output annotation.
  Prefill workers (Rust sidecar and Python handler) then return their final
  native response, and the prefill router sends it ahead of the decode stream
  as an annotation event, so no new codepath or wire field is needed.
- routed_experts and indexer_topk take prefill's prompt rows; when prefill
  sends none, the field is removed instead of returning stale rows.

lib/sidecar/sglang/e2e holds the test container used to compare against stock
SGLang on Qwen3-30B-A3B.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Signed-off-by: William Arnold <7565007+Aphoh@users.noreply.github.com>
Rework the response merge after measuring it:

- Merges move or swap values instead of cloning. Streaming P/D swaps
  prefill's values into each chunk for serialization and back out.
- The routed_experts overlay patches decode's base64 in place and decodes
  only the boundary group, without unsafe.
- Requests wait for prefill only when they need its output, and only as
  early as needed: prompt logprobs (a non-negative logprob_start_len) hold
  the first streamed chunk; routed experts only need it before the final
  chunk, so decode tokens are not held. Plain return_logprob never waits.
- Prefill output reaches the frontend as a stream item, not a re-parsed
  JSON annotation, and the router moves engine_data instead of cloning it.
- Streaming /generate serializes into one buffer reused across chunks.
- The Python prefill handler is back to main; the sidecar owns this path.
- Add benches/response_merge.rs (time and allocations per scenario) and
  drop the ad-hoc e2e harness from the repository.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Signed-off-by: William Arnold <7565007+Aphoh@users.noreply.github.com>
@Aphoh
Aphoh had a problem deploying to external_collaborator October 13, 2026 13:27 — with GitHub Actions Failure
/// Dropping the observer detaches the task: client disconnects must not abort a
/// prefill whose KV transfer may still be needed by the decode worker.
pub(super) struct PrefillTask(JoinHandle<Result<(), PrefillError>>);
pub(super) struct PrefillTask(JoinHandle<Result<PrefillOutput, PrefillError>>);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

prolly gate behind enable rl because we dont care about anything coming from prefill usually

model: state.manager().metric_model_for(&model).to_string(),
endpoint: super::metrics::Endpoint::Generate.to_string(),
request_type: "streaming".to_string(),
request_type: if streaming { "streaming" } else { "unary" }.to_string(),

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this language used elsewhere? unary is new to me

…tests

The join rules copy SGLang internals, so an SGLang bump can silently break
them. Keep the rules in one data file and test them from both sides:

- Move the rules into merge_rules.json, loaded by merge.rs and read by a new
  SGLang contract test. merge.rs names the upstream code they mirror.
- The contract test (pre_merge, sglang, gpu_0) fails on unclassified response
  fields, new or renamed request flags, and changes to SGLang's own chunk
  join. Only the chunk-join check imports SGLang (~7s); the rest parse source.
- Replace the example merge tests with proptests: chunk joining, prefill swap,
  unary equals joined stream, P/D equals aggregated, and the base64 overlay.
- The overlay now refuses invalid base64 as documented; a proptest found it
  copying an invalid prefill blob into the response.
- Drop the response_merge bench and its stats_alloc dependency.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Signed-off-by: William Arnold <7565007+Aphoh@users.noreply.github.com>
@Aphoh
Aphoh had a problem deploying to external_collaborator October 14, 2026 01:55 — with GitHub Actions Failure
@github-actions github-actions Bot added the documentation Improvements or additions to documentation label Oct 14, 2026

@ishandhanani ishandhanani left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Full and deep review at e7a46233. Test and router evidence was gathered at d6d70211e7; the later commit changed the rules loader, overlay validation, and tests, and removed the bench. I re-read those changes but did not re-run them.

Verdict: no blocking correctness defect found. Two medium findings (one perf/design, one wrong compatibility claim) and three low ones are attached inline, plus one optional note.

Validation at d6d70211e7

  • cargo test -p dynamo-llm --lib -- prefill_router response_merge sglang: 89 passed.
  • cargo test -p dynamo-sglang-sidecar --lib -- native_http: 17 passed.
  • Bench compiled in the dev profile.
  • Three extra probe tests: polling past a decode terminal held for prefill ends the stream for PrefillNeed::None and BeforeLast; with no prefill item, decode's stale routed_experts is kept in unary and streaming; a malformed prefill blob removes decode's field with no log.
  • Not run: GPU e2e, criterion numbers, clippy --all-targets, the Python prefill worker path.

Checked and clean

  • Append set equals SGLang's _INCREMENTAL_STREAMING_META_INFO_KEYS plus text/output_ids, and the sidecar refuses non-incremental servers, so last-snapshot-plus-deltas is sound.
  • return_logprob without logprob_start_len: SGLang sets start_len to the prompt length (scheduler.py), so input logprobs are empty in both modes and skipping the prefill wait preserves parity.
  • Both overlaid fields are raw tobytes() base64 in tokenizer_manager.py, so the byte-prefix overlay is layout-valid.
  • The prefill item passes Migration::track_response with zero tokens and never reaches the routing-host request guard (chained outside PrefillRouter).
  • Conditional-disagg bypass and an inactive prefill router emit no item, so decode's own prompt-side outputs are kept.
  • Requests without prompt-side flags carry no annotation; the router path equals main.

Description drift: the CPU benchmark section and the clippy ... benches line still describe benches/response_merge.rs, which e7a46233 removed. The "Older prefill workers" bullet is addressed inline.

"/meta_info/input_top_logprobs_null_prefix"
],
"overlay_base64_prefix": [
"/meta_info/routed_experts",

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Medium, perf/design. sgl-project/sglang#44002, which this PR requires, documents the intended frontend protocol: send routed_experts_start_len=prompt_len to decode so decode skips gathering and serializing its stale [0, prompt_len) rows, and the frontend concatenates prefill + decode. This PR leaves decode's request untouched, so decode still gathers, base64-encodes, and ships about 4 MB of garbage rows per 2048-token prompt at Qwen3-30B-A3B shapes through the sidecar's SSE reader and the request plane before the frontend overwrites them. It also pushes the final decode event toward MAX_EVENT_BYTES (64 MiB) near 30K prompt tokens, which the description already flags.

Suggested: in the sidecar's request() for decode mode, insert routed_experts_start_len = token_ids.len() when return_routed_experts is set (it already rewrites prefill's sampling params there), and make this rule a base64 concat. indexer_topk has no start_len in SGLang, so it would keep the overlay.


/// Prefill's output as a stream item tagged with [`PREFILL_OUTPUT_ANNOTATION`].
fn output_item(output: PrefillOutput) -> Option<Annotated<LLMEngineOutput>> {
let engine_data = Some(output?);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Medium, contract vs. description. The description says older prefill workers cause routed_experts to be removed instead of returned stale. With a pre-PR Rust sidecar, prefill's terminal has engine_data: None, so admission.rs resolves None, this ? returns None, no item is emitted, and the assembler never calls join_prefill: decode's stale rows reach the client exactly as on main. A probe test confirms this for unary and streaming. The Python prefill worker only emits engine_data when kv_cache_hit is non-empty (prefill_handler.py:293-294), so the drop is incidental there too.

Smallest fix: emit the item even when output is None (engine_data: None). stream.rs:37 already treats a prefill item without sglang_response as an empty prefill and drops the drop_if_missing fields. Otherwise correct the description.

}
_ => false,
};
if join && !joined && rule.drop_if_missing {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Low, observability. This removes routed_experts/indexer_topk that the client asked for when prefill has no value or overlay_base64_prefix refuses the input (odd length, non-alphabet byte), with no trace. join_step warns on mismatched appends at line 183; a tracing::warn!(key = %rule.key, ...) here would make a silently missing field diagnosable.

let context = response.context();
let stream_context = context.clone();
let stream = async_stream::stream! {
if output == PrefillNeed::BeforeFirst {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Low, error latency. In BeforeFirst this awaits prefill before polling response at all, so a decode error item (worker failure, invalid argument) is held until prefill's stream ends. main and BeforeLast yield error items immediately through the early return in the loop below (lines 160-172). Bounded by SGLang's transfer timeout, so not a hang, but it is a behavior change the description does not name. A select! over completion and response.next() that yields only error items while waiting would keep the hold for data chunks alone.

const PAYLOAD_KEY: &str = "sglang_tito";
const MAX_EVENT_BYTES: usize = 64 * 1024 * 1024;
/// `dynamo_llm::protocols::response_merge::PREFILL_OUTPUT_ANNOTATION`.
const PREFILL_OUTPUT_ANNOTATION: &str = "prefill_output";

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Low, maintainability. This re-declares the wire constant with a doc comment pointing at dynamo_llm, but the crate already depends on dynamo-llm (Cargo.toml line 24; headless.rs imports from it). use dynamo_llm::protocols::response_merge::PREFILL_OUTPUT_ANNOTATION; keeps the two from drifting.

BeforeFirst,
}

pub trait ResponseMerger: Send + Sync {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Optional. Arc<dyn ResponseMerger> has one implementation, and FieldRule's Serialize/Deserialize derive is exercised only by a test (merge.rs deserializes path lists into RulesFile, not FieldRule). Until a second merger exists, RuleSet could be used directly and the derive dropped. Not blocking.

@github-actions

Copy link
Copy Markdown

This PR is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

@github-actions github-actions Bot added the Stale label Nov 13, 2026

This branch had an error being deployed

1 failed deployment
external_collaborator — e7a46233 Deployed Oct 14, 2026 by Aphoh via ok-to-test #3
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backend::sglang documentation Improvements or additions to documentation frontend router Stale

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants