Skip to content

perf(server): streaming hot-path optimizations + uvloop; fix gpt-oss embed param naming - #1378

Merged
valarLip merged 2 commits into
mainfrom
perf-streaming-serving-optims
Jun 27, 2026
Merged

valarLip merged 2 commits into
mainfrom
perf-streaming-serving-optims

Conversation

@valarLip

Copy link
Copy Markdown
Collaborator

Summary

Two commits reducing event-loop work in the OpenAI streaming serving path, plus a gpt-oss weight-loading false-positive fix.

perf(server): cut event-loop work in streaming hot path (55dcb4a)

At high streaming concurrency the single API event loop is the bottleneck. This commit removes redundant per-token / per-request work from it:

  • Batched step decode (fix the example code cmd #5): the engine output callback now only buffers raw token-ids; flush_stream_batch does one tokenizer.batch_decode for the whole step (one GIL acquire/release instead of one decode per seq). Element-wise identical output.
  • Reuse engine prompt length (update the link to the repo #3): stream generators take num_prompt_tokens (= Sequence.num_prompt_tokens) instead of re-running tokenizer.encode(prompt) on the loop.
  • Multimodal preprocessing off-loop (Update logo #2): image load (blocking urlopen up to 30s) + processor run move to a worker thread; the processor is warmed on-loop first to avoid a cold-start race.
  • SSE finalization coalescing: finish + usage + [DONE] (and per-sibling finishes in fan-out) collapse from 3–4 socket writes per request to 1 — fewer sock.send syscalls at wave boundaries.

perf(server): enable uvloop event loop; fix gpt-oss embed param naming (9b8f451)

  • uvloop: run uvicorn on uvloop (libuv) with graceful fallback to the default loop. Cuts the event-loop cost of SSE socket I/O; measured steady-state TPOT P99 8.50ms → 8.18ms and frontend loop-scheduling delay roughly halved. Frontend process drops from CPU-hot to ~1% CPU under 64 concurrent streams. Adds uvloop to dependencies.
  • gpt-oss embed fix: register embed_tokens first (with embedding as the shared-storage alias) so it stays the primary, non-deduped name in named_parameters(). The checkpoint stores model.embed_tokens.weight; with embedding primary the load-completeness check falsely flagged model.embedding.weight as unloaded even though it loads via the alias. Byte-identical weights; the spurious "parameters were NOT loaded" warning is gone.

Test plan

  • ruff check + black --check clean
  • pytest tests/ — identical to baseline (no new failures; pre-existing env import errors only)
  • gpt-oss-120b tp1 GSM8K (CI method: local-chat-completions --apply_chat_template max_gen_toks=2048) = 0.8832 (threshold 0.88), unchanged by either commit
  • uvloop A/B (gpt-oss-120b tp1, 1k/1k, conc=64, rr=1.0): Total tok/s +1.7%, TPOT P99 −3.8%, TTFT P99 −13%
  • uvloop load-startup verified at runtime (loop=uvloop), fallback path verified (loop=auto)

valarLip added 2 commits June 27, 2026 04:05
- Reuse engine-computed num_prompt_tokens in the stream response
  generators instead of re-encoding the prompt on the event loop at
  stream start (drops a redundant per-request tokenize).
- Run multimodal input prep (image download + HF processor) in a worker
  thread instead of synchronously on the event loop.
- Batch-decode a whole step's buffered stream chunks with one
  tokenizer.batch_decode in flush_stream_batch instead of one decode per
  seq on the output thread (one GIL-released call instead of N).
- Coalesce each request's finalization SSE messages (content/finish +
  usage + [DONE]) into a single send to cut socket-write syscalls when
  many requests finish simultaneously.
uvloop:
- Run uvicorn on uvloop (libuv) instead of the stdlib asyncio selector
  loop, with graceful fallback to the default loop if uvloop is absent.
  Under high streaming concurrency this cuts the event-loop cost of SSE
  socket I/O (sock.send / selector register-unregister): steady-state
  TPOT P99 8.50ms -> 8.18ms and frontend loop-scheduling delay roughly
  halved. Adds uvloop to dependencies.

gpt-oss:
- Register `embed_tokens` first (with `embedding` as the shared-storage
  alias) so it stays the primary, non-deduped name in named_parameters().
  The checkpoint stores `model.embed_tokens.weight`; with `embedding` as
  the primary name the load-completeness check falsely flagged
  `model.embedding.weight` as unloaded even though the weight is loaded
  via the alias. Byte-identical weights (GSM8K 0.8832, unchanged); the
  spurious "parameters were NOT loaded" warning is gone.
@valarLip
valarLip merged commit d7c0a44 into main Jun 27, 2026
24 of 33 checks passed
@valarLip
valarLip deleted the perf-streaming-serving-optims branch June 27, 2026 07:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant