Skip to content

fix(dspark): keep the GQA draft's per-tensor KV scale out of kv_cache_data so LMCache offload starts - #3336

Open
rbrugaro-amd wants to merge 3 commits into
ROCm:mainfrom
rbrugaro-amd:rbrugaro/dspark-gqa-lmcache
Open

rbrugaro-amd wants to merge 3 commits into
ROCm:mainfrom
rbrugaro-amd:rbrugaro/dspark-gqa-lmcache

Conversation

@rbrugaro-amd

@rbrugaro-amd rbrugaro-amd commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

Problem

With the Kimi-K3 DSpark GQA draft (#2581, RadixArk/Kimi-K3-DSpark) and LMCache offload (--kv-transfer-config '{"kv_connector":"lmcache_offload",...}'), the server fails at startup:

ValueError: DenseKVByteCodec: KV tensor size 1 not divisible by num_blocks=13081 (shape=(1,))

The GQA draft declares a per-tensor KV scale (DraftKvPoolSpec.kv_scale), so DraftKvBuilder.build_kv_cache_tensor binds one shared 1-element scale tensor to every draft layer, and also published it as the layer's KVCacheTensor.k_scale / v_scale in kv_cache_data. The dense offload codec reads those fields as per-block data and requires every tensor to divide by the block count. #2581 was validated without a KV connector, and no CI case combines the GQA draft with LMCache, so this was not caught; LMCache with the MLA (Inferact) draft is unaffected.

Fix

The published KVCacheTensor carries only the draft's K/V when the pool declares a per-tensor scale. The scale stays on the layer's own binding, which is where the draft's attention reads it (self.kv.k_scale); nothing reads it off kv_cache_data. A pool's own per-block scale planes (MiniMax-M3.1, Eagle3) are still published.

Validation

  • tests/test_draft_kv_builder.py on one MI355X GPU, rocm/atom-dev:nightly_202611131540: 35 passed. Two new tests, both failing with the fix reverted:
    • test_a_per_tensor_scale_stays_off_the_blockwise_kv_cache_data: the draft's kv_cache_data entries carry no scale, the layers keep theirs, and DenseKVByteCodec accepts them.
    • test_an_offload_round_trip_restores_the_draft_rows_at_new_block_ids: the codec's tier save (gpu_to_chunk_major_device_buffer) and load (chunk_major_device_buffer_to_gpu) move the draft's K/V rows byte for byte to new block ids, beside a target-like block-major cache, and write nothing else.
  • End to end, moonshotai/Kimi-K3 + online ptpc_fp8 with the RadixArk draft, TP8/DCP8, fp8 KV, prefix caching and lmcache_offload (with fix(dspark): pass aiter's new _UAParams fields in the Kimi-K3 DSpark GQA draft #3333): on main the server fails at startup; with this fix it serves, GSM8K 5-shot (full 1319) 96.21% (strict 96.21%), acceptance 61.79%, and saves the draft's rows to the tier.
  • Tier load, end to end: the same deployment with a KV pool smaller than the workload (--gpu-memory-utilization 0.762 --max-num-seqs 16, 1,082 blocks), state checkpoints on (the default 8,192), and a probe that sends 96 prompts of ~14.6K tokens (1.41M tokens), each opening with a unique passcode, then the same 96 again. Pass 1 evicts its own early prefixes from HBM, so pass 2 restores them from LMCache: all 96 requests load 14,336 of their tokens from the tier (768 Retrieved 14336 out of 14336 lines, load_failures: 0). Passcode recall is 96/96 on both passes, and draft acceptance on the tier-loaded pass (29.4%, 27.0% per stats window) matches the pass before any load (28.2%), so the draft's restored rows serve as freshly written ones do.
  • Note: with --state-checkpoint-interval-tokens -1, as the production env sets it, Kimi-K3 finds tier hits but cannot load them (skip_per_req_cache_state_boundary), with any draft; a hit needs a state checkpoint at its boundary.
  • CI: both new tests are in tests/test_draft_kv_builder.py, which the GPU unit-test job proposed in ci: cover the Kimi-K3 DSpark GQA draft (accuracy entry + GPU unit-test job) #3338 runs; the CPU unit-test job skips them.

🤖 Generated with Claude Code

rbrugaro-amd and others added 2 commits November 13, 2026 23:18
…_data

A draft pool that declares a per-tensor KV scale (the Kimi-K3 DSpark GQA
draft) binds one shared 1-element scale tensor to every layer, and
DraftKvBuilder also published it as the layer's KVCacheTensor k_scale /
v_scale. Block-wise walkers of kv_cache_data take those fields as per-block
data: the LMCache offload codec refused the server at startup with
"DenseKVByteCodec: KV tensor size 1 not divisible by num_blocks".

The draft's attention reads the scale off its own binding, so the published
entry now carries only the draft's K/V; a pool's own per-block scale planes
(MiniMax-M3.1, Eagle3) are still published.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…new block ids

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown
Contributor

🏷️ CI Guide

Runs automatically on every eligible PR before approval:

  • ✅ Pre Checkin: Black, Ruff, catalog schema validation, non-GPU unit tests

Heavy model tests:

  • ✅ Run after the PR is approved and Pre Checkin passes
  • ✅ Run immediately when an approval review is submitted
  • ✅ Can be requested before approval with labels
Label Tests
ci:full Run all heavy PR model tests: native ATOM, vLLM, and SGLang
ci:atom Run native ATOM model accuracy tests
ci:vllm Run ATOM vLLM OOT model accuracy tests
ci:sglang Run ATOM SGLang model accuracy tests

Heavy jobs are skipped when the PR is not approved and no matching ci:* label is present.
Add labels via the sidebar or gh pr edit 3336 --add-label <label>

@valar-cop valar-cop left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

ci green.

@valarLip valarLip left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Action required before merge (2)

The fix is in the right place: the startup error comes from ATOM's own builder publishing a 1-element shared scale where the dense offload codec expects per-block data. A CPU probe reproduces KV tensor size 1 not divisible by num_blocks on main and shows it passing at this head, with the fp8 scale-plane path (MiniMax-M3.1 / Eagle3) still publishing its planes.

  1. Return the scales from locals instead of keeping getattr(module, "k_scale", None). atom/spec_decode/draft_kv.py:342-348. The getattr only exists because the test stub _PagedAttention (tests/test_draft_kv_builder.py:36) has no k_scale. Every real binding sets it to None (dspark_draft.py:370, attention_mha.py:87, paged_attention.py:93). Set k_scale = v_scale = None before the if, assign k_scale, v_scale = pool.scale_views(idx) in the elif pool.scale_fields branch (then bind them to the module), and pass the locals to KVCacheTensor. That drops both per_tensor and the getattr, with no change in behaviour.
  2. Assert that the fp8 scale-plane path still publishes its planes. _bind_first_layer (tests/test_draft_kv_builder.py:553-563) throws away the returned KVCacheTensor, so nothing checks the "still published" half of this fix. Return the entry and, in test_a_default_pool_binds_shuffle_views_and_its_own_scale_planes, assert entry.k_scale.data_ptr() == k_scale.data_ptr() (and the same for v_scale).

Smaller, optional:

  • Add a one-line comment to KVCacheTensor.k_scale / v_scale (atom/config.py:57-58) saying they must be block-major when set, as the neighbouring fields already do.
  • The three flash-pool tests (:425, :458, :492) copy the same pool, runner and builder setup; a small helper would remove the copies.
  • In the GPU round trip, zero_() runs before the restore, so not t[untouched].any() can't catch a stray write of zeros. Fill the untouched blocks with a non-zero sentinel instead.

Pre Checkin and the offline smoke test are green. Heavy accuracy CI hasn't run yet because the PR isn't approved. The GPU test only runs once #3338 lands.


Reviewed at 2c9dac1.

@valar-cop

valar-cop commented Nov 14, 2026 •

Copy link
Copy Markdown

…halves

Review follow-up: the published KVCacheTensor takes its scales from locals,
set only in the scale-plane branch, instead of reading them back off the
module with getattr. The default-pool test now checks the planes are still
published (MiniMax-M3.1 / Eagle3), the flash-pool test that a declared
per-tensor scale is not; the three flash-pool tests share one setup helper,
and the offload round trip fills untouched blocks with a non-zero sentinel.
KVCacheTensor documents k_scale / v_scale as block-major.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@rbrugaro-amd

Copy link
Copy Markdown
Contributor Author

Thanks @valarLip. All five done in 155a1a5fd:

  1. The published KVCacheTensor takes its scales from locals: k_scale = v_scale = None before the branch, k_scale, v_scale = pool.scale_views(idx) in the pool.scale_fields branch (then bound to the module), so per_tensor and the getattr are gone. No change in behaviour.
  2. _bind_first_layer returns the published entry. test_a_default_pool_binds_shuffle_views_and_its_own_scale_planes now asserts entry.k_scale / entry.v_scale are the pool's scale planes (same data_ptr), and the flash-pool test that a declared per-tensor scale is not published.
  • KVCacheTensor.k_scale / v_scale (atom/config.py) say they are block-major, or None when the layer has none.
  • The three flash-pool tests share one _gqa_draft(blocks, device) setup helper.
  • The round trip fills every cache with a non-zero sentinel before the restore, so a stray write of zeros fails it too.

tests/test_draft_kv_builder.py on one MI355X GPU, rocm/atom-dev:nightly_202611131540: 35 passed (round trip included). Publishing the per-tensor scale fails 4 tests; not publishing the scale planes fails the new assertion.

@valarLip valarLip left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Action required before merge (1)

All five r18 items are fixed at this head (checked in the code, not just the reply). The product change is now three lines with no getattr, and a CPU probe still reproduces KV tensor size 1 not divisible by num_blocks on main and passes here.

  1. Run the per-tensor-scale test on the production fp8 KV and assert the exact byte count. tests/test_draft_kv_builder.py:427-477. _gqa_draft builds a torch.bfloat16 pool, but the deployment that failed is Kimi-K3 with fp8 KV, and test_a_per_tensor_scale_stays_off_the_blockwise_kv_cache_data only checks codec.bytes_per_block > 0. That check would still pass if a stray segment leaked into the codec. Give _gqa_draft a kv_dtype argument, parametrize the test over torch.bfloat16 and the fp8 dtype, and assert codec.bytes_per_block * 4 == sum(t.nbytes for e in entries.values() for t in (e.k_cache, e.v_cache)) (this is 2 * k_cache.numel() per layer for fp8). These are the two pieces of the closed duplicate #3015 worth keeping. The probe confirms the equality for bf16, float8_e4m3fn and float8_e4m3fnuz at this head.

Optional follow-up (not in this diff): the new KVCacheTensor comment (atom/config.py:57-58) says a per-tensor scale stays off the entry, but atom/plugin/sglang/glm52_dsa_bridge.py:933-934 still publishes MLA's 0-dim _k_scale as both k_scale and v_scale. Either sweep it to None here, the same way as the draft builder, or track it separately.

CI: Pre Checkin, the non-GPU unit tests and the offline smoke test are green at this head. Heavy ATOM / vLLM / SGLang passed at 2c9dac1 (valar-cop triage). Since then the product code has only changed from getattr to locals, with no behaviour change, so no ci:* label is needed. The GPU round-trip test still runs only once #3338 lands.


Re-reviewed at 155a1a5. Already fixed: scales published from locals with no getattr (#1); the fp8 scale-plane entry is asserted by data_ptr (#2); KVCacheTensor scale comment; shared _gqa_draft helper; non-zero sentinel in the round trip.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants