Skip to content

Fix latency benchmark - #557

Merged
hnyls2002 merged 1 commit into
mainfrom
fix-latency
Jun 22, 2024
Merged

hnyls2002 merged 1 commit into
mainfrom
fix-latency

Conversation

@hnyls2002

Copy link
Copy Markdown
Collaborator

No description provided.

@hnyls2002
hnyls2002 merged commit 92cb93f into main Jun 22, 2024
@hnyls2002
hnyls2002 deleted the fix-latency branch June 22, 2024 07:11
timethink pushed a commit to timethink/sglang that referenced this pull request Mar 9, 2025
vschandramourya pushed a commit to vschandramourya/sglang that referenced this pull request Feb 3, 2026
efschu added a commit to efschu/htsglang that referenced this pull request Aug 17, 2026
…and cache_read_input_tokens without the absent-vs-0 ambiguity

GAP 1 -- chat_template_kwargs were SILENTLY DROPPED. The Anthropic adapter
converts into an OpenAI ChatCompletionRequest and delegates
(serving.py:_convert_to_chat_completion_request -> self.openai_serving_chat).
The OpenAI front has carried chat_template_kwargs all along
(openai/protocol.py:844), but AnthropicMessagesRequest declared no such field
and sets no model_config, so pydantic's default extra="ignore" applied: the
caller's kwargs vanished with no error. Rejecting would at least have been
visible.

Fix: declare the field, and set request_data["chat_template_kwargs"] only when
present -- an always-present {} reads downstream as "the caller specified no
kwargs" rather than "the caller said nothing", and templates that branch on
the dict's presence tell those apart.

SEMANTICS MIRRORED, NOT INVENTED, as briefed. apply_reasoning_enabled
(openai/serving_chat.py:1823-1825) already MERGES into whatever the request
carries and overrides only the reasoning toggle, so a caller's unrelated keys
survive while Anthropic's typed `thinking` stays authoritative over the toggle
itself. The stamp is placed BEFORE construction so that merge sees it. The
merge shape is pinned BY SOURCE so a future change to the OpenAI write side
cannot silently start clobbering Anthropic callers' kwargs.

GAP 2 -- cache_read_input_tokens was already mapped, and from the right source:
_cached_prompt_tokens reads usage.prompt_tokens_details.cached_tokens, the real
count, NOT the broken cache_hit_rate. The defect was narrower: the field was
written under `if cached_tokens:`, falsy at 0, so a request that simply got no
cache hit omitted it and a client could not tell "no cache hit" from "this
server does not report cache usage".

AND THE CURE IS NOT "ALWAYS EMIT A NUMBER". Fixing it that way broke an
existing pin (anthropic/test_serving.py:580) which requires the field ABSENT
when the usage object carries no prompt_tokens_details -- and that pin is
right. A backend that sent no details reported nothing; answering 0 would
publish a measurement never taken and convert "unknown" into "definitely no
cache". That is the defaulted-measurement defect (sgl-project#606), which this session
has already refused twice elsewhere.

So the mapping is three-way, via a new _reported_cached_tokens:
  details present, cached > 0  -> that number
  details present, cached == 0 -> 0        (a MEASURED miss; the ambiguity the
                                            brief targets, removed)
  no details at all            -> absent   (nothing was reported)
_cached_prompt_tokens keeps returning an int because the input-token
arithmetic needs one; the reporting side is where unknown and zero differ.

sgl-project#710 ADJACENCY (item 3): DISJOINT BY FILE, and sgl-project#540 is not touched or worked
around. This diff is entrypoints/anthropic/{protocol.py,serving.py} only; the
sgl-project#710-deferred adapter half is srt/function_call/qwen3_coder_detector.py
(landed guards 7a9e478, 0e0f31b; signature proof blocked on sgl-project#540). No
overlap in file or function.

Tests, hermetic (CUDA_VISIBLE_DEVICES="", no server contacted, fake engine
usage objects, --color=no):
  test/registered/unit/entrypoints/test_anthropic_557.py  11 passed
    RED FIRST: 5 of 10 failed before the fixes
    MUTATION PROOF: reverting both fixes turns 3 pins red
  test_anthropic_557 + entrypoints/anthropic/  124 passed, 9 subtests
  test/registered/unit/entrypoints/  4 failed / 436 passed / 3 errors --
    IDENTICAL failure set to the baseline WITHOUT my file (4 failed / 425
    passed / 3 errors), i.e. no regressions; the CORS failures are ordering
    pollution that pre-exists (that file passes 29/29 alone).
  ruff + codespell clean.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant