Repository navigation
fix(agentic): calculate TRT-LLM cache hits from prefill counters / fix(agentic):使用 prefill 计数修正 TRT-LLM 缓存命中率 - #5112
Conversation
Co-Authored-By: Zachary Patel <22306219+zbpatel@users.noreply.github.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Generally most InferenceX submissions don't need to add changes to the infx package. On the off chance they do, generally the diff is small. Please prompt your agent to clean your PR to keep your diff to only necessary & minimal changes too. We have an increased code quality bar for changes to the infx package. 中文大多数 InferenceX 提交通常不需要修改 infx 包。即使确实需要,改动通常也很小。请让你的 agent 清理 PR,使改动只包含必要且最小的内容。infx 包的改动有更高的代码质量要求。 |
| hit_rate, cached_tokens, queried_tokens = _prefix_hit_rate( | ||
| metrics, scope, None if scope else prompt_total | ||
| ) | ||
|
|
||
| flat["total_prompt_tokens"] = counter_int(prompt_total) | ||
| flat["total_generation_tokens"] = counter_int(generation_total) | ||
| flat["server_gpu_cache_hit_rate"] = normalize_fraction( | ||
| gauge_stat( | ||
| metrics, | ||
| "trtllm_kv_cache_hit_rate", | ||
| preferred_keys=("avg", "max", "total"), | ||
| combine="avg", | ||
| ) | ||
| ) | ||
| if flat["server_gpu_cache_hit_rate"] is None: | ||
| flat["server_gpu_cache_hit_rate"] = rate(cached_tokens, prompt_total) | ||
| flat["server_overall_cache_hit_rate"] = flat["server_gpu_cache_hit_rate"] | ||
| flat["server_gpu_cache_hit_rate"] = hit_rate | ||
| flat["server_overall_cache_hit_rate"] = hit_rate | ||
|
|
||
| flat["gpu_kv_cache_usage_pct"] = normalize_fraction( | ||
| gauge_stat( |
There was a problem hiding this comment.
🔴 Disaggregated TRT-LLM runs with no prefill role labels now report inflated computed prompt tokens. The decode worker's re-reported prompts are counted, and the breakdown no longer adds up to prompt_total. When scope is None, _prefix_hit_rate sums trtllm_prompt_tokens over every endpoint before it tries fallback_queried (the frontend total). computed_tokens is now that sum minus cached, not prompt_total - cached. Fix: when no prefill endpoint is identified and dynamo_frontend_input_sequence_tokens exists, use it as the queried total so computed and hit rate match prompt_total, as on base.
Why this was flagged
Trigger: a disaggregated run that exports dynamo_frontend_input_sequence_tokens but whose labels do not normalize to prefill. _endpoint_roles then leaves every endpoint as worker, and scope at trtllm.py:38 is None. Example: frontend 1000, prefill trtllm_prompt_tokens 1000, decode re-report 1000, cached 970. prompt_total is 1000 (trtllm.py:39). _prefix_hit_rate (trtllm.py:222-224) takes the unscoped counter sum of 2000, because fallback_queried applies only when that counter is missing. trtllm.py:82 then sets computed to 1030, and prefix_cache_queries to 2000. On base, computed was prompt_total - cached = 30. Now gpu_cache_hit + computed = 2000, while tokens.prompt_total = 1000 in the same artifact. The PR says aggregate behavior is kept without prefill identification, but this field changes in that case.
Verification: In a disaggregated TRT-LLM run with no prefill role label, scope is None (trtllm.py:37-38). trtllm.py:219-220 uses fallback_queried only if queried is None, so the frontend total is ignored whenever any per-worker prompt counter exists. On the base branch, computed = prompt_total - cached = 30, not 1030. So this is a regression on that path.
| roles = _endpoint_roles(metrics) | ||
| prefill_endpoints = {endpoint for endpoint, role in roles.items() if role == "prefill"} | ||
| scope = _endpoint_filter(prefill_endpoints) if prefill_endpoints else None | ||
| prompt_total = sum_stat(metrics, "dynamo_frontend_input_sequence_tokens") |
There was a problem hiding this comment.
🟡 (optional) Runs where a prefill role label sits on an endpoint without trtllm_ series would report no TRT-LLM cache hit rate. The base branch reported one. prefill_endpoints takes every endpoint with a prefill label, whether or not it exports trtllm_ counters. If none of those endpoints has trtllm_ series, every scoped lookup returns None. Fix: build the scope only from prefill endpoints that export trtllm_ series. When that set is empty, fall back to the unscoped path.
Why this was flagged
The trigger is a scrape where the Dynamo series that carry dynamo_component/disaggregation_mode (for example a router or system-status port) come from a different endpoint_url than the worker's trtllm_ series. I could not enumerate the scraped URLs: srtctl sets AIPERF_SERVER_METRICS_URLS (replay.py:119), not this repo. Only a code comment states the same-endpoint assumption (trtllm.py:190). _endpoint_roles (trtllm.py:176-199) tags every endpoint seen in any series, and trtllm.py:37-38 scopes to every prefill-tagged one. _counter(..., scope) at trtllm.py:41 then returns None. In _prefix_hit_rate (trtllm.py:217-244), the token, block and gauge lookups are all filtered by endpoint, so server_gpu_cache_hit_rate, prefix_cache_hits and prefix_cache_queries come out None. The base used the unfiltered trtllm_kv_cache_hit_rate gauge and the unfiltered counters, so it still reported a rate. Nothing checks that the scope contains any trtllm_ series.
Verification: trtllm.py:37-38 builds scope from every endpoint with role == "prefill" and does not check that those endpoints export trtllm_ series. _prefix_hit_rate(metrics, scope, None if scope else prompt_total) (line 52-54) then runs every lookup through that scope, so server_gpu_cache_hit_rate/server_overall_cache_hit_rate become None (58-59). The base branch read trtllm_kv_cache_hit_rate unscoped, so it reported a value for the same scrape.
[by Codex]
Disaggregated TRT-LLM reports prompt tokens on both prefill and decode workers, while the cache-hit gauge is a process-lifetime ratio. Averaging those gauges or summing both workers understates prompt-prefix reuse. Compute the hit rate from window token counters on identified prefill endpoints, falling back to block counters and then the gauge, and accept both
_totaland unsuffixed counter names.Keep frontend request totals when present, scope fallback prompt totals to prefill, and retain each worker's own metrics. A concrete case with 970 cached tokens out of 1,000 prefill tokens now reports 97%, even when decode re-reports another 1,000 prompt tokens. Endpoint role detection preserves known prefill/decode roles when other metrics use generic worker labels.
Related dashboard fix: SemiAnalysisAI/InferenceX-app#1797.
The harness infers prefill scope from exported role labels; without positive prefill identification it preserves aggregate behavior. This changes result processing and artifacts, so it adds no performance changelog entry or benchmark sweep label.
AI model disclosure
The original source commit credits Claude Opus 5.5 as a co-author. Codex reviewed and revised the implementation; two delegated Codex agents reviewed correctness and standards and contributed harness and backfill corrections. The exact model identifiers for the current Codex session and its agents could not be verified from the runtime.
中文
分离式 TRT-LLM 的 prefill 和 decode 节点都会上报 prompt token,而缓存命中率 gauge 反映的是进程生命周期内的比率。此修改优先使用已识别 prefill 端点在测量窗口内的 token 计数计算命中率,缺失时依次使用 block 计数和 gauge,并兼容带
_total后缀及不带后缀的指标名。保留各节点自身的指标;有 frontend 总量时继续优先使用,否则仅统计 prefill 的 prompt 总量。对于 1,000 个 prompt token 中有 970 个命中的情况,即使 decode 再次上报这 1,000 个 token,聚合命中率仍为 97%。此修改仅影响结果处理及产物,不改变基准测试执行内容。prefill 范围依赖导出的角色标签;未识别到 prefill 时保留原有聚合行为。配套仪表板修改:https://github.com/SemiAnalysisAI/InferenceX-app/pull/1797。