Repository navigation
perf(server): streaming hot-path optimizations + uvloop; fix gpt-oss embed param naming - #1378
Merged
Merged
Conversation
- Reuse engine-computed num_prompt_tokens in the stream response generators instead of re-encoding the prompt on the event loop at stream start (drops a redundant per-request tokenize). - Run multimodal input prep (image download + HF processor) in a worker thread instead of synchronously on the event loop. - Batch-decode a whole step's buffered stream chunks with one tokenizer.batch_decode in flush_stream_batch instead of one decode per seq on the output thread (one GIL-released call instead of N). - Coalesce each request's finalization SSE messages (content/finish + usage + [DONE]) into a single send to cut socket-write syscalls when many requests finish simultaneously.
uvloop: - Run uvicorn on uvloop (libuv) instead of the stdlib asyncio selector loop, with graceful fallback to the default loop if uvloop is absent. Under high streaming concurrency this cuts the event-loop cost of SSE socket I/O (sock.send / selector register-unregister): steady-state TPOT P99 8.50ms -> 8.18ms and frontend loop-scheduling delay roughly halved. Adds uvloop to dependencies. gpt-oss: - Register `embed_tokens` first (with `embedding` as the shared-storage alias) so it stays the primary, non-deduped name in named_parameters(). The checkpoint stores `model.embed_tokens.weight`; with `embedding` as the primary name the load-completeness check falsely flagged `model.embedding.weight` as unloaded even though the weight is loaded via the alias. Byte-identical weights (GSM8K 0.8832, unchanged); the spurious "parameters were NOT loaded" warning is gone.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two commits reducing event-loop work in the OpenAI streaming serving path, plus a gpt-oss weight-loading false-positive fix.
perf(server): cut event-loop work in streaming hot path(55dcb4a)At high streaming concurrency the single API event loop is the bottleneck. This commit removes redundant per-token / per-request work from it:
flush_stream_batchdoes onetokenizer.batch_decodefor the whole step (one GIL acquire/release instead of one decode per seq). Element-wise identical output.num_prompt_tokens(=Sequence.num_prompt_tokens) instead of re-runningtokenizer.encode(prompt)on the loop.[DONE](and per-sibling finishes in fan-out) collapse from 3–4 socket writes per request to 1 — fewersock.sendsyscalls at wave boundaries.perf(server): enable uvloop event loop; fix gpt-oss embed param naming(9b8f451)uvloopto dependencies.embed_tokensfirst (withembeddingas the shared-storage alias) so it stays the primary, non-deduped name innamed_parameters(). The checkpoint storesmodel.embed_tokens.weight; withembeddingprimary the load-completeness check falsely flaggedmodel.embedding.weightas unloaded even though it loads via the alias. Byte-identical weights; the spurious "parameters were NOT loaded" warning is gone.Test plan
ruff check+black --checkcleanpytest tests/— identical to baseline (no new failures; pre-existing env import errors only)local-chat-completions --apply_chat_template max_gen_toks=2048) = 0.8832 (threshold 0.88), unchanged by either commitloop=uvloop), fallback path verified (loop=auto)