Skip to content

[Perf] openai: Skip decoding text-only prompts sent as token IDs - #44110

Open
bvolpato wants to merge 2 commits into
sgl-project:mainfrom
bvolpato:openai-text-only-skip-prompt-decode
Open

bvolpato wants to merge 2 commits into
sgl-project:mainfrom
bvolpato:openai-text-only-skip-prompt-decode

Conversation

@bvolpato

@bvolpato bvolpato commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

Motivation

On multimodal models, a new text-only chat turn can miss the template cache, encode the full history, and decode it back to text even though the engine receives the token IDs. The unused decode costs API-process time, and the cache retains both the string and a Python integer per token. Requests that already arrive as IDs bypass this path.

Modifications

The template cache stores text-only prompts as packed token IDs, with a tuple fallback for IDs outside the unsigned 32-bit range. If a caller needs text, as Responses and Moss-VL can, the cache decodes it on first use and reuses it on an exact resend. An assistant continuation is not part of the cache key, so its decoded text stays on that request. Media requests keep their original tuple of IDs and decoded text. Neither the engine input nor the cache key changes.

Related work: #30876 also skips the decode but predates the template cache. #46796 targets requests with media. #27298 sends text-only requests as text, and #47028 caches the encode step.

Accuracy Tests

The OpenAI Chat, Responses, tokenization, prompt round-trip, and Anthropic serving unit suites passed on CPU. The resend and large-token-ID regression cases failed on cada55ca before the fix. Pre-commit passed on the changed files.

Earlier B200 runs on 96eb1d47bc matched main 4ffc9e1b7d for prompt IDs and temperature-zero output on Qwen3-VL-2B and LFM2.5-VL. They covered text, tools, assistant continuations, images, streaming, tokenization, Responses, and Anthropic requests. Qwen3-VL-2B also produced the same GSM8K predictions and score (0.815). These GPU runs were not repeated on the current head.

Speed Tests and Profiling

I measured request conversion on one ARM64 CPU core with the Qwen3-VL-2B-Instruct tokenizer and template, without a model server. One user message containing A long chat history. repeated 22,400 times produced 89,609 tokens. Values are median times per request across five batches of three cold requests or ten warm resends. The token IDs and decoded text matched across all three revisions.

Request Main (ab84df61) Previous PR head (cada55ca) This revision (4a7bc95c)
Chat sent as IDs, template cache cleared 360 ms 305 ms 297 ms
Exact resend sent as text, warm cache 0.66 ms 48.76 ms 3.96 ms

The resend no longer pays for a second decode. It still costs more than main because the compact cache must rebuild a Python list of IDs.

Earlier server runs used Qwen3-VL-2B on one B200 with --context-length 128000 --mem-fraction-static 0.65. Throughput compares 96eb1d47bc merged with 582168dd7c against that main revision, alternating runs with a shared cached prefix and a unique new turn per request. The largest gain was in this API-process-limited setup, not a general GPU throughput gain.

ID-only chat workload Main Earlier PR revision
96.9K-token prompt, concurrency 16, max_tokens=1 23.5–23.7 req/s 51.3–51.6 req/s
Same workload, max_tokens=32 15.5–15.7 req/s 20.6–21.0 req/s
RSS growth after 160 distinct prompts of about 116K tokens 586–609 MB 186–191 MB

Live Moss-VL and mistral-common servers, router behavior, and production resend rates were not tested.

Checklist


CI States

Latest PR Test (Base): ⊘ Skipped -- add the run-ci label to run CI tests.
Latest PR Test (Extra): ⊘ Skipped -- add the run-ci label to run CI tests.
Latest PR Test (AMD ROCm 10): ⊘ Skipped -- add the run-ci label to run CI tests.

Tip

Base CI only covers the core cases, and its workload may not exercise your change. Use /rerun-test to run the tests your change touches, or copy this prompt for an agent:

Find the registered tests this PR affects and run them with /rerun-test, following https://github.com/sgl-project/sglang/blob/main/.agents/skills/rerun-test/SKILL.md

@bvolpato bvolpato changed the title [OpenAI] Skip the prompt decode for text-only chats on multimodal models [OpenAI] Skip unused prompt decoding for text-only chats Oct 13, 2026
@bvolpato
bvolpato marked this pull request as ready for review October 13, 2026 22:52
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 13, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-13T22:56:08.883159Z f9a9d2b Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@bvolpato bvolpato changed the title [OpenAI] Skip unused prompt decoding for text-only chats [Perf] openai: Skip unused prompt decoding Oct 15, 2026
@bvolpato
bvolpato force-pushed the openai-text-only-skip-prompt-decode branch 4 times, most recently from bd3ff91 to 0511ec4 Compare October 18, 2026 23:25
@bvolpato
bvolpato force-pushed the openai-text-only-skip-prompt-decode branch from 0511ec4 to 27dcc39 Compare October 27, 2026 04:32
@bvolpato
bvolpato force-pushed the openai-text-only-skip-prompt-decode branch from 96eb1d4 to 2b49108 Compare November 3, 2026 18:32
@bvolpato

bvolpato commented Nov 9, 2026

Copy link
Copy Markdown
Contributor Author

Could someone with CI permissions run /run-ci nv on 4a7bc95c? The Base gate stopped at the missing run-ci label, before it ran any tests. The affected OpenAI and Anthropic CPU suites and the changed-file pre-commit checks passed locally.

@bvolpato bvolpato changed the title [Perf] openai: Skip unused prompt decoding [Perf] openai: Skip decoding text-only prompts sent as token IDs Nov 9, 2026
Store text-only chat template cache entries as packed token IDs (with a
tuple fallback outside the unsigned 32-bit range) and defer decoding prompt
text until a caller actually needs it (such as media requests, Moss-VL, or
Responses text paths), reusing the decoded string on exact resends while
keeping assistant continuations out of the cached prompt.
@bvolpato
bvolpato force-pushed the openai-text-only-skip-prompt-decode branch from 4a7bc95 to 2c6c7db Compare November 14, 2026 23:57
@bvolpato

Copy link
Copy Markdown
Contributor Author

rebased and squashed onto main (2c6c7dbb61), updated the new test_serving_chat.py case to use _use_chat_encoding, and verified it merges cleanly with #48180.

could someone with CI permissions run /tag-and-rerun-ci on 2c6c7dbb61?

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant