[Metrics] Expose cached prompt tokens by cache tier by cquil11 · Pull Request #56318 · vllm-project/vllm

cquil11 · 2026-09-10T17:06:35Z

Changes

Adds vllm:prompt_tokens_cached_by_source_total{source} with five values: device, host, p2p, disk, external_unspecified.
Counts cache hits at admission. Source counts sum to local and external prefix-cache hits. Existing prompt-token counters are unchanged.
Records attribution through the existing PrefixCacheStats path. Connectors return typed counts through get_external_cache_hit_sources. Incorrect totals fall back to external_unspecified with a one-time warning.
Simple offload reports host or disk. NIXL, Mooncake transfer and MoRIIO report p2p. Mooncake Store and uninstrumented connectors report external_unspecified. MultiConnector delegates to the selected connector.
Native offload uses the keys selected for loading. Mixed KV-group dependencies use the outermost required tier, so host + disk reports disk. Missing attribution reports external_unspecified. Promotion preserves the initiating request's original tier.
Supports Python and Rust frontends.

Validation

Passed Run Sweep: DeepSeek-V4-Pro, one-hour AIPerf AgentX profiles, evaluations disabled. AIPerf scraped Prometheus. JSON/CSV source counts and accounting checks passed.

B200: DRAM c8, Simple NVMe c14, native DRAM+NVMe c14.
GB300: NIXL + Mooncake Store c256. Four request errors (0.197%) and 430 end-of-run cancellations, within the configured pass criteria.

GPU tests used bd57138f with the separate promotion-policy backport. The subsequent rebase changed no attribution logic.

claude

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

mergify · 2026-09-11T01:39:46Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @cquil11.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

mergify · 2026-09-15T07:08:23Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @cquil11.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

orozery · 2026-09-15T09:49:07Z

FYI we already have:

vllm:kv_offload_tiering_chunk_queries
vllm:kv_offload_tiering_chunk_hits

These are per tier type labeled.

cquil11 · 2026-09-15T18:56:18Z

FYI we already have:
vllm:kv_offload_tiering_chunk_queries
vllm:kv_offload_tiering_chunk_hits
These are per tier type labeled.

@orozery Yes, I saw this. Afaik this is only for tiered and doesn't exactly answer the question of "how many cached prompt tokens came from each tier" rather how many qere queried in each tier.

I am trying to add a simple way to declare which abstract source a cached token came from across all connectors.

mergify · 2026-09-17T01:11:46Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @cquil11.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

cquil11 · 2026-10-05T21:19:36Z

/ci run

github-actions · 2026-10-05T21:19:58Z

❌ @cquil11, A reviewer with write access must run /ci run, approve the PR, or add the ready label first.

github-actions · 2026-10-05T21:42:13Z

✅ @cquil11, CI is now available for this PR.

/ci run starts upstream CI; /amd-ci run starts AMD CI only.
Your branch must contain every commit currently on its upstream target branch. Merge or rebase onto the latest target branch, then rerun the command. Append --allow-stale to a run command to test an outdated branch at your own risk.
/ci retry retries failed jobs in the CI build for the current PR head. If the current head has no CI build, it starts a new CI build for the current head containing only jobs that failed in the latest earlier CI build for this PR.
/amd-ci retry retries failed jobs in AMD CI for the current PR head. Use /amd-ci run when the current head has no AMD CI build.
/ci cancel cancels scheduled or running CI builds for this PR branch; /amd-ci cancel does the same for AMD CI only.

ivanium · 2026-10-05T21:42:35Z

/ci run

github-actions · 2026-10-05T21:44:07Z

✅ Triggered Buildkite CI #92966 for commit 72f7fb8e2bf0.

cquil11 · 2026-10-05T22:20:09Z

/ci retry

github-actions · 2026-10-05T22:20:30Z

✅ Queued 1 failed job(s) for retry in Buildkite CI #92966.

cquil11 · 2026-10-05T22:23:33Z

/ci retry

github-actions · 2026-10-05T22:23:53Z

✅ Queued 1 failed job(s) for retry in Buildkite CI #92966.

cquil11 · 2026-10-05T22:36:51Z

/ci retry

github-actions · 2026-10-05T22:37:51Z

✅ Queued 1 failed job(s) for retry in Buildkite CI #92966.

cquil11 · 2026-10-06T00:20:40Z

/ci retry

github-actions · 2026-10-06T00:21:00Z

✅ Queued 1 failed job(s) for retry in Buildkite CI #92966.

robertgshaw2-redhat

metrics side and scheduler side lgtm. I dont know enough about offloadingconnector to really review it in detail

robertgshaw2-redhat · 2026-10-06T18:52:12Z

thanks for simplifying this PR, it looks really good now

…et_load_source in offloading connector test mock Root cause: vllm#56318 wraps manager.get_load_source() in CacheHitSource(); our MagicMock manager returns a Mock, so ValueError Upstream: vllm-project/vllm#56318 Fix: MockOffloadingSpec manager returns CacheHitSource.EXTERNAL_UNSPECIFIED from get_load_source, mirroring upstream utils.py Signed-off-by: Paweł Olejniczak <pawelx.olejniczak@intel.com>

… chat_utils helpers, Drop HPU SeedOss (+5 more) (#1880) This PR consolidates 7 hourly-CI fixes against vllm@`8d7b93fec2eb72ce01f2217472ac1f51b9bce600`. ## Bug 1: Import chat_utils helpers from vllm.renderers.chat_utils - **State machine id**: missing_module_vllm_entrypoints_chat_utils - **Commit**: 6607e1b ### Root cause vllm#60522 (c1692c85f1) moved vllm/entrypoints/chat_utils.py to vllm/renderers/chat_utils.py; two vllm-gaudi imports still use the old path ### Culprit Probable culprit: [PR #60522](vllm-project/vllm#60522) - pinned by symbol archaeology, bisect not run. Candidate range: [compare](vllm-project/vllm@ac7f3e1...8d7b93f). ### Fix switch load_chat_template and make_tool_call_id imports to vllm.renderers.chat_utils (both defined at pinned sha) ## Bug 2: Drop HPU SeedOss override and kernel_block_sizes removed upstream - **State machine id**: missing_module_vllm_model_executor_models_seed_oss - **Commit**: d4aece7 ### Root cause vllm#59701 (6e4efafc6e) deleted vllm.model_executor.models.seed_oss, which vllm_gaudi.models.seed_oss subclassed, so register_model() fails; with that fixed, engine init fails because vllm#57169 (02e55d2af3) removed kernel_block_sizes and the KV-to-kernel block split from MultiGroupBlockTable ### Culprit Probable culprit: [PR #59701](vllm-project/vllm#59701) - pinned by symbol archaeology, bisect not run. Candidate range: [compare](vllm-project/vllm@ac7f3e1...8d7b93f). ### Fix delete pass-through HpuSeedOssForCausalLM and its registration (upstream serves SeedOss via the Transformers backend); stop passing kernel_block_sizes, and raise NotImplementedError when a group's kernel block size differs from its KV block size, the split upstream no longer performs ## Bug 3: Stub get_load_source in offloading connector test mock - **State machine id**: hpu_unit_tests - **Commit**: 56ef9c0 ### Root cause vllm#56318 wraps manager.get_load_source() in CacheHitSource(); our MagicMock manager returns a Mock, so ValueError ### Culprit Probable culprit: [PR #56318](vllm-project/vllm#56318) - pinned by symbol archaeology, bisect not run. ### Fix MockOffloadingSpec manager returns CacheHitSource.EXTERNAL_UNSPECIFIED from get_load_source, mirroring upstream utils.py ## Bug 4: Use vLLM LayerNorm in the regional compilation unit test - **State machine id**: hpu_unit_tests_2 - **Commit**: aa162a6 ### Root cause vllm#57172 replaced torch nn.LayerNorm in OPT with vllm's CustomOp LayerNorm, so the test's torch LayerNorm filter matches nothing ### Culprit Regression introduced by [PR #57172](vllm-project/vllm#57172). ### Fix import LayerNorm from vllm.model_executor.layers.layernorm, which OPT now builds its norms from ## Bug 5: Split KV blocks into kernel blocks in an HPU block table - **State machine id**: multigroup_block_table_kernel_block_sizes_removed - **Commit**: 31c5fa1 ### Root cause vllm#57169 dropped kernel_block_sizes and virtual block splitting from MultiGroupBlockTable/BlockTable; HPU InputBatch still passes it ### Culprit Probable culprit: [PR #57169](vllm-project/vllm#57169) - pinned by symbol archaeology, bisect not run. ### Fix add HPUMultiGroupBlockTable: tables built at kernel block size, manager block ids expanded to kernel ids on append/add_row, replacing the NotImplementedError split guard ## Bug 6: Accept list block ids in the HPU block table - **State machine id**: pre_commit - **Commit**: 8abe42f ### Root cause HPUMultiGroupBlockTable typed add_row/append_row block_ids as tuple[list[int], ...] like upstream, but HPU InputBatch passes the request's list[list[int]], which mypy rejects now that the subclass is local ### Culprit Upstream culprit: not identified - no candidate range recovered (see investigation notes). ### Fix type block_ids as Sequence[list[int]] in add_row, append_row and _to_kernel_blocks; runtime is unchanged ## Bug 7: Accept kv_cache_spec in HPU get_supported_kernel_block_sizes - **State machine id**: hpu_pd_tests - **Commit**: c78bc81 ### Root cause vllm#53175 added kv_cache_spec to get_supported_kernel_block_sizes; attention.py passes it positionally, HPU overrides take none ### Culprit Probable culprit: [PR #53175](vllm-project/vllm#53175) - pinned by symbol archaeology, bisect not run. ### Fix add optional kv_cache_spec=None parameter to both HPU backend overrides, matching the upstream base signature --------- Signed-off-by: Paweł Olejniczak <pawelx.olejniczak@intel.com>

Signed-off-by: Cam Quilici <cjquilici@gmail.com> Signed-off-by: Cam Quilici <cameron@semianalysis.com> Co-authored-by: Cam Quilici <cameron@semianalysis.com> Co-authored-by: Nick Hill <nickhill123@gmail.com> Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai>

cquil11 requested review from ApostaC, BugenZhao, NickLucche, WoosukKwon, alexm-redhat, heheda12345, ivanium, njhill, orozery, robertgshaw2-redhat, xuechendi and ywang96 as code owners September 10, 2026 17:06

claude Bot reviewed Sep 10, 2026

View reviewed changes

mergify Bot added rust kv-connector scheduler labels Sep 10, 2026

mergify Bot added the needs-rebase label Sep 11, 2026

cquil11 force-pushed the feat/prompt-token-tier-source branch from 798e721 to 2f4cd9d Compare September 15, 2026 04:00

mergify Bot removed the needs-rebase label Sep 15, 2026

mergify Bot added the needs-rebase label Sep 15, 2026

cquil11 force-pushed the feat/prompt-token-tier-source branch from 2f4cd9d to 4072c5a Compare September 15, 2026 18:49

mergify Bot removed the needs-rebase label Sep 15, 2026

mergify Bot added the needs-rebase label Sep 17, 2026

robertgshaw2-redhat reviewed Sep 21, 2026

View reviewed changes

Comment thread vllm/v1/metrics/stats.py Resolved

robertgshaw2-redhat reviewed Sep 21, 2026

View reviewed changes

Comment thread vllm/distributed/kv_transfer/kv_connector/v1/base.py Resolved

ivanium added the ready ONLY add when PR is ready to merge/full CI is needed label Oct 5, 2026

Merge branch 'main' into feat/prompt-token-tier-source

72f7fb8

ivanium enabled auto-merge (squash) October 6, 2026 03:39

robertgshaw2-redhat approved these changes Oct 6, 2026

View reviewed changes

ivanium merged commit e37e51d into vllm-project:main Oct 6, 2026
201 checks passed

markmc added this to Prometheus Metrics Oct 7, 2026

pawel-olejniczak mentioned this pull request Oct 14, 2026

[FIX_FOR_VLLM_CUSTOM=8d7b93fec2eb72ce01f2217472ac1f51b9bce600] Import chat_utils helpers, Drop HPU SeedOss (+5 more) vllm-project/vllm-gaudi#1880

Merged

harsh543 mentioned this pull request Oct 31, 2026

[Feature]: Per-request and per-session KV cache tier attribution in OpenTelemetry traces #53926

Open

1 task done

AdvayMonga mentioned this pull request Nov 13, 2026

[Perf][KV Offload] Split external hit sources in one pass #67530

Open

6 tasks done

Conversation

cquil11 commented Sep 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Validation

Uh oh!

claude Bot left a comment

Choose a reason for hiding this comment

Claude Code Review

Uh oh!

mergify Bot commented Sep 11, 2026

Uh oh!

mergify Bot commented Sep 15, 2026

Uh oh!

orozery commented Sep 15, 2026

Uh oh!

cquil11 commented Sep 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mergify Bot commented Sep 17, 2026

Uh oh!

Uh oh!

Uh oh!

cquil11 commented Oct 5, 2026

Uh oh!

github-actions Bot commented Oct 5, 2026

Uh oh!

github-actions Bot commented Oct 5, 2026

Uh oh!

ivanium commented Oct 5, 2026

Uh oh!

github-actions Bot commented Oct 5, 2026

Uh oh!

cquil11 commented Oct 5, 2026

Uh oh!

github-actions Bot commented Oct 5, 2026

Uh oh!

cquil11 commented Oct 5, 2026

Uh oh!

github-actions Bot commented Oct 5, 2026

Uh oh!

cquil11 commented Oct 5, 2026

Uh oh!

github-actions Bot commented Oct 5, 2026

Uh oh!

cquil11 commented Oct 6, 2026

Uh oh!

github-actions Bot commented Oct 6, 2026

Uh oh!

robertgshaw2-redhat left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

robertgshaw2-redhat commented Oct 6, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

cquil11 commented Sep 10, 2026 •

edited

Loading

cquil11 commented Sep 15, 2026 •

edited

Loading