Repository navigation
Fix latency benchmark - #557
Merged
Merged
Conversation
timethink
pushed a commit
to timethink/sglang
that referenced
this pull request
Mar 9, 2025
vschandramourya
pushed a commit
to vschandramourya/sglang
that referenced
this pull request
Feb 3, 2026
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 17, 2026
…and cache_read_input_tokens without the absent-vs-0 ambiguity
GAP 1 -- chat_template_kwargs were SILENTLY DROPPED. The Anthropic adapter
converts into an OpenAI ChatCompletionRequest and delegates
(serving.py:_convert_to_chat_completion_request -> self.openai_serving_chat).
The OpenAI front has carried chat_template_kwargs all along
(openai/protocol.py:844), but AnthropicMessagesRequest declared no such field
and sets no model_config, so pydantic's default extra="ignore" applied: the
caller's kwargs vanished with no error. Rejecting would at least have been
visible.
Fix: declare the field, and set request_data["chat_template_kwargs"] only when
present -- an always-present {} reads downstream as "the caller specified no
kwargs" rather than "the caller said nothing", and templates that branch on
the dict's presence tell those apart.
SEMANTICS MIRRORED, NOT INVENTED, as briefed. apply_reasoning_enabled
(openai/serving_chat.py:1823-1825) already MERGES into whatever the request
carries and overrides only the reasoning toggle, so a caller's unrelated keys
survive while Anthropic's typed `thinking` stays authoritative over the toggle
itself. The stamp is placed BEFORE construction so that merge sees it. The
merge shape is pinned BY SOURCE so a future change to the OpenAI write side
cannot silently start clobbering Anthropic callers' kwargs.
GAP 2 -- cache_read_input_tokens was already mapped, and from the right source:
_cached_prompt_tokens reads usage.prompt_tokens_details.cached_tokens, the real
count, NOT the broken cache_hit_rate. The defect was narrower: the field was
written under `if cached_tokens:`, falsy at 0, so a request that simply got no
cache hit omitted it and a client could not tell "no cache hit" from "this
server does not report cache usage".
AND THE CURE IS NOT "ALWAYS EMIT A NUMBER". Fixing it that way broke an
existing pin (anthropic/test_serving.py:580) which requires the field ABSENT
when the usage object carries no prompt_tokens_details -- and that pin is
right. A backend that sent no details reported nothing; answering 0 would
publish a measurement never taken and convert "unknown" into "definitely no
cache". That is the defaulted-measurement defect (sgl-project#606), which this session
has already refused twice elsewhere.
So the mapping is three-way, via a new _reported_cached_tokens:
details present, cached > 0 -> that number
details present, cached == 0 -> 0 (a MEASURED miss; the ambiguity the
brief targets, removed)
no details at all -> absent (nothing was reported)
_cached_prompt_tokens keeps returning an int because the input-token
arithmetic needs one; the reporting side is where unknown and zero differ.
sgl-project#710 ADJACENCY (item 3): DISJOINT BY FILE, and sgl-project#540 is not touched or worked
around. This diff is entrypoints/anthropic/{protocol.py,serving.py} only; the
sgl-project#710-deferred adapter half is srt/function_call/qwen3_coder_detector.py
(landed guards 7a9e478, 0e0f31b; signature proof blocked on sgl-project#540). No
overlap in file or function.
Tests, hermetic (CUDA_VISIBLE_DEVICES="", no server contacted, fake engine
usage objects, --color=no):
test/registered/unit/entrypoints/test_anthropic_557.py 11 passed
RED FIRST: 5 of 10 failed before the fixes
MUTATION PROOF: reverting both fixes turns 3 pins red
test_anthropic_557 + entrypoints/anthropic/ 124 passed, 9 subtests
test/registered/unit/entrypoints/ 4 failed / 436 passed / 3 errors --
IDENTICAL failure set to the baseline WITHOUT my file (4 failed / 425
passed / 3 errors), i.e. no regressions; the CORS failures are ordering
pollution that pre-exists (that file passes 29/29 alone).
ruff + codespell clean.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.