Repository navigation
fix(dspark): keep the GQA draft's per-tensor KV scale out of kv_cache_data so LMCache offload starts - #3336
fix(dspark): keep the GQA draft's per-tensor KV scale out of kv_cache_data so LMCache offload starts#3336rbrugaro-amd wants to merge 3 commits into
Conversation
…_data A draft pool that declares a per-tensor KV scale (the Kimi-K3 DSpark GQA draft) binds one shared 1-element scale tensor to every layer, and DraftKvBuilder also published it as the layer's KVCacheTensor k_scale / v_scale. Block-wise walkers of kv_cache_data take those fields as per-block data: the LMCache offload codec refused the server at startup with "DenseKVByteCodec: KV tensor size 1 not divisible by num_blocks". The draft's attention reads the scale off its own binding, so the published entry now carries only the draft's K/V; a pool's own per-block scale planes (MiniMax-M3.1, Eagle3) are still published. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…new block ids Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
🏷️ CI GuideRuns automatically on every eligible PR before approval:
Heavy model tests:
|
valarLip
left a comment
There was a problem hiding this comment.
⚠️ Action required before merge (2)
The fix is in the right place: the startup error comes from ATOM's own builder publishing a 1-element shared scale where the dense offload codec expects per-block data. A CPU probe reproduces KV tensor size 1 not divisible by num_blocks on main and shows it passing at this head, with the fp8 scale-plane path (MiniMax-M3.1 / Eagle3) still publishing its planes.
- Return the scales from locals instead of keeping
getattr(module, "k_scale", None).atom/spec_decode/draft_kv.py:342-348. Thegetattronly exists because the test stub_PagedAttention(tests/test_draft_kv_builder.py:36) has nok_scale. Every real binding sets it to None (dspark_draft.py:370,attention_mha.py:87,paged_attention.py:93). Setk_scale = v_scale = Nonebefore theif, assignk_scale, v_scale = pool.scale_views(idx)in theelif pool.scale_fieldsbranch (then bind them to the module), and pass the locals toKVCacheTensor. That drops bothper_tensorand thegetattr, with no change in behaviour. - Assert that the fp8 scale-plane path still publishes its planes.
_bind_first_layer(tests/test_draft_kv_builder.py:553-563) throws away the returnedKVCacheTensor, so nothing checks the "still published" half of this fix. Return the entry and, intest_a_default_pool_binds_shuffle_views_and_its_own_scale_planes, assertentry.k_scale.data_ptr() == k_scale.data_ptr()(and the same forv_scale).
Smaller, optional:
- Add a one-line comment to
KVCacheTensor.k_scale/v_scale(atom/config.py:57-58) saying they must be block-major when set, as the neighbouring fields already do. - The three flash-pool tests (
:425,:458,:492) copy the same pool, runner and builder setup; a small helper would remove the copies. - In the GPU round trip,
zero_()runs before the restore, sonot t[untouched].any()can't catch a stray write of zeros. Fill the untouched blocks with a non-zero sentinel instead.
Pre Checkin and the offline smoke test are green. Heavy accuracy CI hasn't run yet because the PR isn't approved. The GPU test only runs once #3338 lands.
Reviewed at 2c9dac1.
|
ci triage for 155a1a5:
|
…halves Review follow-up: the published KVCacheTensor takes its scales from locals, set only in the scale-plane branch, instead of reading them back off the module with getattr. The default-pool test now checks the planes are still published (MiniMax-M3.1 / Eagle3), the flash-pool test that a declared per-tensor scale is not; the three flash-pool tests share one setup helper, and the offload round trip fills untouched blocks with a non-zero sentinel. KVCacheTensor documents k_scale / v_scale as block-major. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
|
Thanks @valarLip. All five done in
|
valarLip
left a comment
There was a problem hiding this comment.
⚠️ Action required before merge (1)
All five r18 items are fixed at this head (checked in the code, not just the reply). The product change is now three lines with no getattr, and a CPU probe still reproduces KV tensor size 1 not divisible by num_blocks on main and passes here.
- Run the per-tensor-scale test on the production fp8 KV and assert the exact byte count.
tests/test_draft_kv_builder.py:427-477._gqa_draftbuilds atorch.bfloat16pool, but the deployment that failed is Kimi-K3 with fp8 KV, andtest_a_per_tensor_scale_stays_off_the_blockwise_kv_cache_dataonly checkscodec.bytes_per_block > 0. That check would still pass if a stray segment leaked into the codec. Give_gqa_draftakv_dtypeargument, parametrize the test overtorch.bfloat16and the fp8 dtype, and assertcodec.bytes_per_block * 4 == sum(t.nbytes for e in entries.values() for t in (e.k_cache, e.v_cache))(this is2 * k_cache.numel()per layer for fp8). These are the two pieces of the closed duplicate #3015 worth keeping. The probe confirms the equality for bf16,float8_e4m3fnandfloat8_e4m3fnuzat this head.
Optional follow-up (not in this diff): the new KVCacheTensor comment (atom/config.py:57-58) says a per-tensor scale stays off the entry, but atom/plugin/sglang/glm52_dsa_bridge.py:933-934 still publishes MLA's 0-dim _k_scale as both k_scale and v_scale. Either sweep it to None here, the same way as the draft builder, or track it separately.
CI: Pre Checkin, the non-GPU unit tests and the offline smoke test are green at this head. Heavy ATOM / vLLM / SGLang passed at 2c9dac1 (valar-cop triage). Since then the product code has only changed from getattr to locals, with no behaviour change, so no ci:* label is needed. The GPU round-trip test still runs only once #3338 lands.
Re-reviewed at 155a1a5. Already fixed: scales published from locals with no getattr (#1); the fp8 scale-plane entry is asserted by data_ptr (#2); KVCacheTensor scale comment; shared _gqa_draft helper; non-zero sentinel in the round trip.
Problem
With the Kimi-K3 DSpark GQA draft (#2581,
RadixArk/Kimi-K3-DSpark) and LMCache offload (--kv-transfer-config '{"kv_connector":"lmcache_offload",...}'), the server fails at startup:The GQA draft declares a per-tensor KV scale (
DraftKvPoolSpec.kv_scale), soDraftKvBuilder.build_kv_cache_tensorbinds one shared 1-element scale tensor to every draft layer, and also published it as the layer'sKVCacheTensor.k_scale/v_scaleinkv_cache_data. The dense offload codec reads those fields as per-block data and requires every tensor to divide by the block count. #2581 was validated without a KV connector, and no CI case combines the GQA draft with LMCache, so this was not caught; LMCache with the MLA (Inferact) draft is unaffected.Fix
The published
KVCacheTensorcarries only the draft's K/V when the pool declares a per-tensor scale. The scale stays on the layer's own binding, which is where the draft's attention reads it (self.kv.k_scale); nothing reads it offkv_cache_data. A pool's own per-block scale planes (MiniMax-M3.1, Eagle3) are still published.Validation
tests/test_draft_kv_builder.pyon one MI355X GPU,rocm/atom-dev:nightly_202611131540: 35 passed. Two new tests, both failing with the fix reverted:test_a_per_tensor_scale_stays_off_the_blockwise_kv_cache_data: the draft'skv_cache_dataentries carry no scale, the layers keep theirs, andDenseKVByteCodecaccepts them.test_an_offload_round_trip_restores_the_draft_rows_at_new_block_ids: the codec's tier save (gpu_to_chunk_major_device_buffer) and load (chunk_major_device_buffer_to_gpu) move the draft's K/V rows byte for byte to new block ids, beside a target-like block-major cache, and write nothing else.moonshotai/Kimi-K3+ onlineptpc_fp8with the RadixArk draft, TP8/DCP8, fp8 KV, prefix caching andlmcache_offload(with fix(dspark): pass aiter's new _UAParams fields in the Kimi-K3 DSpark GQA draft #3333): onmainthe server fails at startup; with this fix it serves, GSM8K 5-shot (full 1319) 96.21% (strict 96.21%), acceptance 61.79%, and saves the draft's rows to the tier.--gpu-memory-utilization 0.762 --max-num-seqs 16, 1,082 blocks), state checkpoints on (the default 8,192), and a probe that sends 96 prompts of ~14.6K tokens (1.41M tokens), each opening with a unique passcode, then the same 96 again. Pass 1 evicts its own early prefixes from HBM, so pass 2 restores them from LMCache: all 96 requests load 14,336 of their tokens from the tier (768Retrieved 14336 out of 14336lines,load_failures: 0). Passcode recall is 96/96 on both passes, and draft acceptance on the tier-loaded pass (29.4%, 27.0% per stats window) matches the pass before any load (28.2%), so the draft's restored rows serve as freshly written ones do.--state-checkpoint-interval-tokens -1, as the production env sets it, Kimi-K3 finds tier hits but cannot load them (skip_per_req_cache_state_boundary), with any draft; a hit needs a state checkpoint at its boundary.tests/test_draft_kv_builder.py, which the GPU unit-test job proposed in ci: cover the Kimi-K3 DSpark GQA draft (accuracy entry + GPU unit-test job) #3338 runs; the CPU unit-test job skips them.🤖 Generated with Claude Code