Repository navigation
Conversation
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
bd3ff91 to
0511ec4
Compare
0511ec4 to
27dcc39
Compare
96eb1d4 to
2b49108
Compare
|
Could someone with CI permissions run |
Store text-only chat template cache entries as packed token IDs (with a tuple fallback outside the unsigned 32-bit range) and defer decoding prompt text until a caller actually needs it (such as media requests, Moss-VL, or Responses text paths), reusing the decoded string on exact resends while keeping assistant continuations out of the cached prompt.
4a7bc95 to
2c6c7db
Compare
|
rebased and squashed onto could someone with CI permissions run |
Motivation
On multimodal models, a new text-only chat turn can miss the template cache, encode the full history, and decode it back to text even though the engine receives the token IDs. The unused decode costs API-process time, and the cache retains both the string and a Python integer per token. Requests that already arrive as IDs bypass this path.
Modifications
The template cache stores text-only prompts as packed token IDs, with a tuple fallback for IDs outside the unsigned 32-bit range. If a caller needs text, as Responses and Moss-VL can, the cache decodes it on first use and reuses it on an exact resend. An assistant continuation is not part of the cache key, so its decoded text stays on that request. Media requests keep their original tuple of IDs and decoded text. Neither the engine input nor the cache key changes.
Related work: #30876 also skips the decode but predates the template cache. #46796 targets requests with media. #27298 sends text-only requests as text, and #47028 caches the encode step.
Accuracy Tests
The OpenAI Chat, Responses, tokenization, prompt round-trip, and Anthropic serving unit suites passed on CPU. The resend and large-token-ID regression cases failed on
cada55cabefore the fix. Pre-commit passed on the changed files.Earlier B200 runs on
96eb1d47bcmatched main4ffc9e1b7dfor prompt IDs and temperature-zero output on Qwen3-VL-2B and LFM2.5-VL. They covered text, tools, assistant continuations, images, streaming, tokenization, Responses, and Anthropic requests. Qwen3-VL-2B also produced the same GSM8K predictions and score (0.815). These GPU runs were not repeated on the current head.Speed Tests and Profiling
I measured request conversion on one ARM64 CPU core with the Qwen3-VL-2B-Instruct tokenizer and template, without a model server. One user message containing
A long chat history.repeated 22,400 times produced 89,609 tokens. Values are median times per request across five batches of three cold requests or ten warm resends. The token IDs and decoded text matched across all three revisions.ab84df61)cada55ca)4a7bc95c)The resend no longer pays for a second decode. It still costs more than main because the compact cache must rebuild a Python list of IDs.
Earlier server runs used Qwen3-VL-2B on one B200 with
--context-length 128000 --mem-fraction-static 0.65. Throughput compares96eb1d47bcmerged with582168dd7cagainst that main revision, alternating runs with a shared cached prefix and a unique new turn per request. The largest gain was in this API-process-limited setup, not a general GPU throughput gain.max_tokens=1max_tokens=32Live Moss-VL and mistral-common servers, router behavior, and production resend rates were not tested.
Checklist
CI States
Latest PR Test (Base): ⊘ Skipped -- add the
run-cilabel to run CI tests.Latest PR Test (Extra): ⊘ Skipped -- add the
run-cilabel to run CI tests.Latest PR Test (AMD ROCm 10): ⊘ Skipped -- add the
run-cilabel to run CI tests.Tip
Base CI only covers the core cases, and its workload may not exercise your change. Use
/rerun-testto run the tests your change touches, or copy this prompt for an agent: