Skip to content

perf(sglang): batch streaming responses natively - #13679

Closed
jthomson04 wants to merge 13 commits into
mainfrom
jthomson04/perf-sglang-native-batch-fanout
Closed

jthomson04 wants to merge 13 commits into
mainfrom
jthomson04/perf-sglang-native-batch-fanout

Conversation

@jthomson04

@jthomson04 jthomson04 commented Aug 22, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Use the Engine-wide SGLang response handler from Add engine streaming response handler sgl-project/sglang#36015 when the installed SGLang version supports it.
  • Register each external RID with its Dynamo context, conversion state, and Rust ResponseSender before generation starts.
  • Convert token, text, and native SGLang intermediate outputs in one handler call and invoke ResponseSender.send_batch() once for the batch.
  • Ignore unregistered internal probe and warmup RIDs.
  • Keep terminal conversion, sender closure, errors, cancellation, and registry cleanup on the existing iterator adapter.
  • Keep the current iterator-based push path on older SGLang versions.
  • Remove the prototype monkeypatch, fake events, request-state interception, first-output arming, and request eligibility callback.

This draft keeps its current jthomson04/quic-response-fixed-lanes base. The matching upstream interface is draft SGLang PR #36015.

Validation

  • Focused SGLang response-handler tests: 5 passed.
  • Existing push-egress tests: 41 passed.
  • isort, Ruff, Black, py_compile, cargo fmt --check, and git diff --check passed for the changed files.

Prior prototype evidence was 45,157 output tok/s versus 15,450 output tok/s for the matched Dynamo control. That result used the earlier prototype interception path; it is evidence for the batching boundary, not a new benchmark of this revision. No Tyche allocation or benchmark was run.

Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
@github-actions github-actions Bot added perf backend::sglang Relates to the sglang backend backend::trtllm Relates to the trtllm backend labels Aug 22, 2026
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
@jthomson04
jthomson04 force-pushed the jthomson04/perf-sglang-native-batch-fanout branch from 37023de to 29fb560 Compare August 26, 2026 00:23
@jthomson04 jthomson04 changed the title perf(sglang): fan out scheduler batches natively perf(sglang): batch streaming responses natively Aug 26, 2026
Base automatically changed from jthomson04/quic-response-fixed-lanes to main September 8, 2026 17:56
@jthomson04 jthomson04 closed this Sep 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backend::sglang Relates to the sglang backend backend::trtllm Relates to the trtllm backend perf size/XXL

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant