Repository navigation
Conversation
…unks Native /generate could only stream, and in P/D it dropped every output that only prefill computes: decode never runs the prompt, so input logprobs came back empty and routed experts came back with stale prompt rows. Add an engine-neutral ResponseMerger trait with one merge operation for both seams where a response arrives in pieces: inter-token chunks (Steps) and prefill/decode (PrefillDecode). RuleSet is the declarative implementation (seam, JSON pointer, join op, on-missing policy) and the intended plugin format; SglangMerger carries SGLang's rules. - stream=false is served by streaming from SGLang and joining the chunks in the frontend, following SGLang's own _coalesce_streaming_chunks. - Requests that ask for prompt-side output carry a prefill_output annotation. Prefill workers (Rust sidecar and Python handler) then return their final native response, and the prefill router sends it ahead of the decode stream as an annotation event, so no new codepath or wire field is needed. - routed_experts and indexer_topk take prefill's prompt rows; when prefill sends none, the field is removed instead of returning stale rows. lib/sidecar/sglang/e2e holds the test container used to compare against stock SGLang on Qwen3-30B-A3B. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> Signed-off-by: William Arnold <7565007+Aphoh@users.noreply.github.com>
Rework the response merge after measuring it: - Merges move or swap values instead of cloning. Streaming P/D swaps prefill's values into each chunk for serialization and back out. - The routed_experts overlay patches decode's base64 in place and decodes only the boundary group, without unsafe. - Requests wait for prefill only when they need its output, and only as early as needed: prompt logprobs (a non-negative logprob_start_len) hold the first streamed chunk; routed experts only need it before the final chunk, so decode tokens are not held. Plain return_logprob never waits. - Prefill output reaches the frontend as a stream item, not a re-parsed JSON annotation, and the router moves engine_data instead of cloning it. - Streaming /generate serializes into one buffer reused across chunks. - The Python prefill handler is back to main; the sidecar owns this path. - Add benches/response_merge.rs (time and allocations per scenario) and drop the ad-hoc e2e harness from the repository. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> Signed-off-by: William Arnold <7565007+Aphoh@users.noreply.github.com>
| /// Dropping the observer detaches the task: client disconnects must not abort a | ||
| /// prefill whose KV transfer may still be needed by the decode worker. | ||
| pub(super) struct PrefillTask(JoinHandle<Result<(), PrefillError>>); | ||
| pub(super) struct PrefillTask(JoinHandle<Result<PrefillOutput, PrefillError>>); |
There was a problem hiding this comment.
prolly gate behind enable rl because we dont care about anything coming from prefill usually
| model: state.manager().metric_model_for(&model).to_string(), | ||
| endpoint: super::metrics::Endpoint::Generate.to_string(), | ||
| request_type: "streaming".to_string(), | ||
| request_type: if streaming { "streaming" } else { "unary" }.to_string(), |
There was a problem hiding this comment.
is this language used elsewhere? unary is new to me
…tests The join rules copy SGLang internals, so an SGLang bump can silently break them. Keep the rules in one data file and test them from both sides: - Move the rules into merge_rules.json, loaded by merge.rs and read by a new SGLang contract test. merge.rs names the upstream code they mirror. - The contract test (pre_merge, sglang, gpu_0) fails on unclassified response fields, new or renamed request flags, and changes to SGLang's own chunk join. Only the chunk-join check imports SGLang (~7s); the rest parse source. - Replace the example merge tests with proptests: chunk joining, prefill swap, unary equals joined stream, P/D equals aggregated, and the base64 overlay. - The overlay now refuses invalid base64 as documented; a proptest found it copying an invalid prefill blob into the response. - Drop the response_merge bench and its stats_alloc dependency. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> Signed-off-by: William Arnold <7565007+Aphoh@users.noreply.github.com>
ishandhanani
left a comment
There was a problem hiding this comment.
Full and deep review at e7a46233. Test and router evidence was gathered at d6d70211e7; the later commit changed the rules loader, overlay validation, and tests, and removed the bench. I re-read those changes but did not re-run them.
Verdict: no blocking correctness defect found. Two medium findings (one perf/design, one wrong compatibility claim) and three low ones are attached inline, plus one optional note.
Validation at d6d70211e7
cargo test -p dynamo-llm --lib -- prefill_router response_merge sglang: 89 passed.cargo test -p dynamo-sglang-sidecar --lib -- native_http: 17 passed.- Bench compiled in the dev profile.
- Three extra probe tests: polling past a decode terminal held for prefill ends the stream for
PrefillNeed::NoneandBeforeLast; with no prefill item, decode's stalerouted_expertsis kept in unary and streaming; a malformed prefill blob removes decode's field with no log. - Not run: GPU e2e, criterion numbers,
clippy --all-targets, the Python prefill worker path.
Checked and clean
- Append set equals SGLang's
_INCREMENTAL_STREAMING_META_INFO_KEYSplustext/output_ids, and the sidecar refuses non-incremental servers, so last-snapshot-plus-deltas is sound. return_logprobwithoutlogprob_start_len: SGLang sets start_len to the prompt length (scheduler.py), so input logprobs are empty in both modes and skipping the prefill wait preserves parity.- Both overlaid fields are raw
tobytes()base64 intokenizer_manager.py, so the byte-prefix overlay is layout-valid. - The prefill item passes
Migration::track_responsewith zero tokens and never reaches the routing-host request guard (chained outsidePrefillRouter). - Conditional-disagg bypass and an inactive prefill router emit no item, so decode's own prompt-side outputs are kept.
- Requests without prompt-side flags carry no annotation; the router path equals
main.
Description drift: the CPU benchmark section and the clippy ... benches line still describe benches/response_merge.rs, which e7a46233 removed. The "Older prefill workers" bullet is addressed inline.
| "/meta_info/input_top_logprobs_null_prefix" | ||
| ], | ||
| "overlay_base64_prefix": [ | ||
| "/meta_info/routed_experts", |
There was a problem hiding this comment.
Medium, perf/design. sgl-project/sglang#44002, which this PR requires, documents the intended frontend protocol: send routed_experts_start_len=prompt_len to decode so decode skips gathering and serializing its stale [0, prompt_len) rows, and the frontend concatenates prefill + decode. This PR leaves decode's request untouched, so decode still gathers, base64-encodes, and ships about 4 MB of garbage rows per 2048-token prompt at Qwen3-30B-A3B shapes through the sidecar's SSE reader and the request plane before the frontend overwrites them. It also pushes the final decode event toward MAX_EVENT_BYTES (64 MiB) near 30K prompt tokens, which the description already flags.
Suggested: in the sidecar's request() for decode mode, insert routed_experts_start_len = token_ids.len() when return_routed_experts is set (it already rewrites prefill's sampling params there), and make this rule a base64 concat. indexer_topk has no start_len in SGLang, so it would keep the overlay.
|
|
||
| /// Prefill's output as a stream item tagged with [`PREFILL_OUTPUT_ANNOTATION`]. | ||
| fn output_item(output: PrefillOutput) -> Option<Annotated<LLMEngineOutput>> { | ||
| let engine_data = Some(output?); |
There was a problem hiding this comment.
Medium, contract vs. description. The description says older prefill workers cause routed_experts to be removed instead of returned stale. With a pre-PR Rust sidecar, prefill's terminal has engine_data: None, so admission.rs resolves None, this ? returns None, no item is emitted, and the assembler never calls join_prefill: decode's stale rows reach the client exactly as on main. A probe test confirms this for unary and streaming. The Python prefill worker only emits engine_data when kv_cache_hit is non-empty (prefill_handler.py:293-294), so the drop is incidental there too.
Smallest fix: emit the item even when output is None (engine_data: None). stream.rs:37 already treats a prefill item without sglang_response as an empty prefill and drops the drop_if_missing fields. Otherwise correct the description.
| } | ||
| _ => false, | ||
| }; | ||
| if join && !joined && rule.drop_if_missing { |
There was a problem hiding this comment.
Low, observability. This removes routed_experts/indexer_topk that the client asked for when prefill has no value or overlay_base64_prefix refuses the input (odd length, non-alphabet byte), with no trace. join_step warns on mismatched appends at line 183; a tracing::warn!(key = %rule.key, ...) here would make a silently missing field diagnosable.
| let context = response.context(); | ||
| let stream_context = context.clone(); | ||
| let stream = async_stream::stream! { | ||
| if output == PrefillNeed::BeforeFirst { |
There was a problem hiding this comment.
Low, error latency. In BeforeFirst this awaits prefill before polling response at all, so a decode error item (worker failure, invalid argument) is held until prefill's stream ends. main and BeforeLast yield error items immediately through the early return in the loop below (lines 160-172). Bounded by SGLang's transfer timeout, so not a hang, but it is a behavior change the description does not name. A select! over completion and response.next() that yields only error items while waiting would keep the hold for data chunks alone.
| const PAYLOAD_KEY: &str = "sglang_tito"; | ||
| const MAX_EVENT_BYTES: usize = 64 * 1024 * 1024; | ||
| /// `dynamo_llm::protocols::response_merge::PREFILL_OUTPUT_ANNOTATION`. | ||
| const PREFILL_OUTPUT_ANNOTATION: &str = "prefill_output"; |
There was a problem hiding this comment.
Low, maintainability. This re-declares the wire constant with a doc comment pointing at dynamo_llm, but the crate already depends on dynamo-llm (Cargo.toml line 24; headless.rs imports from it). use dynamo_llm::protocols::response_merge::PREFILL_OUTPUT_ANNOTATION; keeps the two from drifting.
| BeforeFirst, | ||
| } | ||
|
|
||
| pub trait ResponseMerger: Send + Sync { |
There was a problem hiding this comment.
Optional. Arc<dyn ResponseMerger> has one implementation, and FieldRule's Serialize/Deserialize derive is exercised only by a test (merge.rs deserializes path lists into RulesFile, not FieldRule). Until a second merger exists, RuleSet could be used directly and the derive dropped. Not blocking.
|
This PR is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days. |
Summary
SGLang
/generatethrough Dynamo now supportsstream=false. In P/D, it also returns the outputs that only the prefill worker computes. The change uses the existing/generatepath (the token pipeline). It adds no second HTTP path.What problem this fixes
stream=falsereturned HTTP 501.routed_expertscame from decode and were wrong. The blob length was correct, so the error was not visible.How it works
ResponseMerger(lib/llm/src/protocols/response_merge.rs) joins the pieces of one response:join_stepjoins the streamed chunks of one request.join_prefillandrestore_prefilljoin prefill's output into a decode response or chunk.prefill_needreports whether a request needs prefill's output, and how early.RuleSetimplements the trait from a list ofFieldRulevalues (path,op,drop_if_missing). This list is the intended plugin format. There are three join operations:appendjoins strings or lists.take_prefilluses prefill's value.overlay_base64_prefixputs prefill's bytes at the start of decode's bytes.lib/llm/src/protocols/sglang/merge.rsholds the SGLang rules. They follow SGLang's own merge code.text,output_ids, and the five output-side lists. Every other field takes the value from the last chunk.routed_expertsandindexer_topk. If prefill sends no rows, remove the field, so that wrong rows never reach the client.stream=false.prefill_outputannotation only when the request needs prefill's output. It addsprefill_output_firstonly for streamed requests where every chunk needs prefill's values (prompt logprobs).prefill_output_first, the item comes before the first decode output. Otherwise it comes as soon as prefill finishes, and always before the final decode output.Effect on TTFT
return_logprobonly, which gives output logprobs.return_logproband a non-negativelogprob_start_len. Every streamed chunk must carry the prompt logprobs, so the first chunk waits for prefill.Host overhead
unsafe./generateserializes every chunk into one buffer that it reuses. This also makes requests that do not use these fields faster than onmain.Known cost: long prompts with routed experts
routed_expertsgrows with the prompt length × layers × top-k × 4 bytes. For Qwen3-30B-A3B, that is about 2 KB of base64 for each token. These costs happen once for each request, after the last token:MAX_EVENT_BYTES). With very long prompts, therouted_expertsevent can be larger than this limit. This limit also applies onmainto decode's final chunk.Compatibility
main.routed_expertsis removed instead of returned with wrong rows.main's prefill returns no routed experts, so Dynamo removes the field.Validation
Unit tests
cargo test -p dynamo-llm --lib: 88 targeted tests pass. They cover the rules, the overlay for every boundary length, the stream adapter, unary dispatch, and the router's two prefill output modes.cargo test -p dynamo-sglang-sidecar --lib: 109 pass.cargo clippy -D warningspasses for dynamo-llm (lib, tests, benches) and for the sidecar.cargo fmtand pre-commit pass.CPU benchmark
cargo bench -p dynamo-llm --bench response_mergemeasures time and allocations for each scenario. It uses Qwen3-30B-A3B shapes: a 2,048-token prompt, 512 output tokens, and a 5 MBrouted_expertsblob. "Work it cannot avoid" means freeing the input chunks, or serializing bytes that must be sent anyway.mainmainEnd-to-end test on GPU
--moe-runner-backend triton --enable-return-routed-experts --enable-deterministic-inference. SGLang is the nightly build 7224fce1 with #44002 applied. The Rust sidecar and the Dynamo frontend are built from this branch.stream=falseand Dynamostream=true(joined by the test) with stock SGLangstream=false, in aggregated and in P/D mode. The output ids, text, finish reason, token counts, and logprobs (largest difference 4.4e-16) match.routed_expertsis byte-identical in both modes.Not covered yet
/generate, P/D/generate, or routed experts. A pytest e2e test withQwen/Qwen3-0.6Bis next.Generated with Devin