Skip to content

[Kimi-K3][LMCache] LMCache offload for Kimi-K3 - #1960

Closed
zejunchen-zejun wants to merge 7 commits into
mainfrom
zejun/lmcache
Closed

zejunchen-zejun wants to merge 7 commits into
mainfrom
zejun/lmcache

Conversation

@zejunchen-zejun

Copy link
Copy Markdown
Collaborator

No description provided.

@github-actions

Copy link
Copy Markdown
Contributor

🏷️ CI Guide

Runs automatically on every eligible PR before approval:

  • ✅ Pre Checkin: Black, Ruff, catalog schema validation, non-GPU unit tests

Heavy model tests:

  • ✅ Run after the PR is approved and Pre Checkin passes
  • ✅ Run immediately when an approval review is submitted
  • ✅ Can be requested before approval with labels
Label Tests
ci:full Run all heavy PR model tests: native ATOM, vLLM, and SGLang
ci:atom Run native ATOM model accuracy tests
ci:vllm Run ATOM vLLM OOT model accuracy tests
ci:sglang Run ATOM SGLang model accuracy tests

Heavy jobs are skipped when the PR is not approved and no matching ci:* label is present.
Add labels via the sidebar or gh pr edit 1960 --add-label <label>

@zejunchen-zejun
zejunchen-zejun force-pushed the zejun/lmcache branch 2 times, most recently from 6a33858 to b0f98df Compare August 20, 2026 03:54
Comment on lines +11 to +35
from __future__ import annotations

import logging
import os
import time

from atom.kv_transfer.disaggregation.types import ConnectorCompletion
from atom.kv_transfer.offload import config as offcfg
from atom.kv_transfer.offload._offload_common import (
max_pending_saves,
pp_aware_rank_and_world,
)
from atom.kv_transfer.offload.dense.connector import (
DenseOffloadConnector,
DenseOffloadScheduler,
)
from atom.kv_transfer.offload.hybrid.kimi_k3.staging import StagedTransfer
from atom.kv_transfer.offload.hybrid.kimi_k3.state_object import StateByteCodec
from atom.kv_transfer.offload.hybrid.kimi_k3.state_tier import (
_JointPark,
StateOffloadTier,
)
from atom.kv_transfer.offload.metadata import LMCacheOffloadMetadata

logger = logging.getLogger("atom")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ [ruff] <I001> reported by reviewdog 🐶
Import block is un-sorted or un-formatted

Suggested change
from __future__ import annotations
import logging
import os
import time
from atom.kv_transfer.disaggregation.types import ConnectorCompletion
from atom.kv_transfer.offload import config as offcfg
from atom.kv_transfer.offload._offload_common import (
max_pending_saves,
pp_aware_rank_and_world,
)
from atom.kv_transfer.offload.dense.connector import (
DenseOffloadConnector,
DenseOffloadScheduler,
)
from atom.kv_transfer.offload.hybrid.kimi_k3.staging import StagedTransfer
from atom.kv_transfer.offload.hybrid.kimi_k3.state_object import StateByteCodec
from atom.kv_transfer.offload.hybrid.kimi_k3.state_tier import (
_JointPark,
StateOffloadTier,
)
from atom.kv_transfer.offload.metadata import LMCacheOffloadMetadata
logger = logging.getLogger("atom")
from __future__ import annotations
import logging
import os
import time
from atom.kv_transfer.disaggregation.types import ConnectorCompletion
from atom.kv_transfer.offload import config as offcfg
from atom.kv_transfer.offload._offload_common import (
max_pending_saves,
pp_aware_rank_and_world,
)
from atom.kv_transfer.offload.dense.connector import (
DenseOffloadConnector,
DenseOffloadScheduler,
)
from atom.kv_transfer.offload.hybrid.kimi_k3.staging import StagedTransfer
from atom.kv_transfer.offload.hybrid.kimi_k3.state_object import StateByteCodec
from atom.kv_transfer.offload.hybrid.kimi_k3.state_tier import (
StateOffloadTier,
_JointPark,
)
from atom.kv_transfer.offload.metadata import LMCacheOffloadMetadata
logger = logging.getLogger("atom")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ [ruff] <I001> reported by reviewdog 🐶
Import block is un-sorted or un-formatted

from __future__ import annotations
import logging
import os
import threading
import time
from contextlib import nullcontext
from dataclasses import dataclass
from math import isfinite
from numbers import Integral
import torch
from atom.kv_transfer.disaggregation.base import (
KVConnectorBase,
KVConnectorSchedulerBase,
)
from atom.kv_transfer.disaggregation.types import (
ConnectorCompletion,
KVConnectorOutput,
LoadOperationId,
ReqId,
SaveCompletionId,
SaveOperationId,
)
from atom.kv_transfer.offload import config as offcfg
from atom.kv_transfer.offload._block_gpu_connector import BlockGPUConnector
from atom.kv_transfer.offload._offload_common import (
max_pending_saves,
OffloadSchedulerMixin,
OffloadWorkerMixin,
build_offload_engine,
pp_aware_rank_and_world,
validated_kv_role,
)
from atom.kv_transfer.offload.hybrid.dsv4.codec import (
HEADER_BYTES,
DSV4CheckpointCodec,
DSV4CheckpointCorruptionError,
DSV4CheckpointError,
DSV4CheckpointKey,
DSV4CheckpointStore,
DSV4PageSlotCodec,
)
from atom.kv_transfer.offload.hybrid.dsv4.policy import (
DSV4StagingAdmission,
_BoundedLRUSet,
_chained_prefix_hashes,
_committed_sidecar_capacity,
_compute_slot_fingerprint,
build_dsv4_profile,
select_pending_sidecar_boundary,
sidecar_boundary_tokens,
)
from atom.kv_transfer.offload.metadata import (
LMCacheOffloadMetadata,
LMCacheReqMeta,
LoadSpec,
SaveSpec,
SlotLoadSpec,
SlotSaveSpec,
)

@zejunchen-zejun zejunchen-zejun changed the title [Kimi-K3][LMCache] LMCache offload for hybrid attention model(paged mixed with state), like Kimi-K3 [Kimi-K3][LMCache] LMCache offload for Kimi-K3 Aug 20, 2026
@zejunchen-zejun
zejunchen-zejun force-pushed the zejun/lmcache branch 3 times, most recently from 6729761 to bd8462c Compare August 26, 2026 07:52
Zejun Chen and others added 7 commits August 27, 2026 09:19
Rebuilds the K3 work on main's restructured offload package instead of
merging the old branch: `connector.py` is now a dispatch shell selecting a
layout from config alone, so K3 becomes a third layout (`kimi_k3`) beside
`dense` and `hybrid`, sharing dense's KV path and adding only the KDA
per-request state tier.

  atom/kv_transfer/offload/hybrid/kimi_k3/
    connector.py    KimiK3OffloadConnector (worker) / KimiK3OffloadScheduler
    state_tier.py   the tier and its joint park
    state_object.py the entry codec
    staging.py      single-entry staged transfer

The scheduler half is six small overrides on `DenseOffloadScheduler` rather
than a fork of it:

  * `_decide_load_after_alloc` clamps a hybrid's KV leg to the boundary the
    state leg is aimed at, and refuses the load outright when there is none.
    A hybrid's state is the compressed history of exactly `[0, hbm)`, so
    raising the KV-loaded length past that has the forward skip tokens the
    recurrence never saw -- wrong output, no exception.
  * `should_defer_free` releases the blocks of a save that was never handed
    out once the backend stops draining. Holding them is what turned a
    stopped backend into a stopped engine.
  * `_may_emit_save` bounds how many requests may have a save outstanding,
    since each one pins its blocks. Dense gains the seam (unbounded there,
    it does not pin); `max_pending_saves` moves to `_offload_common` so both
    it and DSV4 read the same `OFFLOAD_MAX_PENDING_SAVES`.
  * `build_connector_meta` drains the state-load queue onto the metadata.
  * `connector_completion` routes the tier's two channels.

Reports ride main's generic `ConnectorCompletion` channels rather than extra
`KVConnectorOutput` fields, so TP quorum is the aggregator's existing job and
`multi_connector` needs only the tier re-exposure and `enqueue_state_loads`.

Env vars: four are gone. `OFFLOAD_STATE` and `OFFLOAD_KV_FOR_HYBRID` because
a resumable prefix needs both legs -- state is offloaded exactly when KV is,
and the connector being configured is the only switch. `OFFLOAD_SAVE_MAX_INFLIGHT`
and `OFFLOAD_SAVE_STALL_S` fold into `OFFLOAD_MAX_PENDING_SAVES` and a
constant. `OFFLOAD_STATE_MIN_LOAD_TOKENS` defaulted to 0 and a token floor is
the wrong unit for a flat-cost entry. The staging ring is gated on the hosting
connector (`state_pool(..., offload_hosted=)`) so a deployment without one
does not pay for a ring nothing can drain. `docs/environment_variables.md`
gains a full offload section covering main's existing vars as well.

`STATE_CKPT_EXTRA_ENTRIES` is deliberately NOT reintroduced: the old branch
predates #1894, which deleted it, and it is a state-pool sizing knob with no
bearing on offload. `state_pool` keeps `extra_entries` as the backend-declared
parameter main left it as.

Also removed as dead: `STATE_OFFLOAD_LOADS_WIRED` (permanently True),
`clamp_state_boundary` (no caller), `StateOffloadIndex.kv_offload_enabled`
(always True once the index is only built for a hosting connector), a
duplicate `_submit_state_spills` override, and ~150 lines of `staging.py`
that re-implemented `atom_lmcache_staging`. Its `_env_flag` fix -- strip, and
read empty as off, so `VAR=` does not turn a flag ON -- moves to the shared
copy.

Measured on c10 (Kimi-K3, TP8, 3600s): P90 interactivity 31.19 -> 41.03,
per-GPU throughput 4313 -> 4784 tok/s, combined hit rate 91.62% -> 93.21%.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…s request forever

`connector_metadata_has_work` decides whether a step's metadata reaches the
worker at all, and it decided from a hardcoded field list in
`disaggregation/types.py` shared by three connector families.
`MultiConnectorMetadata` holds none of the work itself -- everything lives in
`metas` -- so to satisfy that list it mirrors the subs' fields as aggregating
properties. It mirrors `requests` and `lookup_requests_in_step`, and misses
`state_loads`:

    sub = LMCacheOffloadMetadata(); sub.state_loads = [("req-1", h, group)]
    connector_metadata_has_work(sub)                            # True
    connector_metadata_has_work(MultiConnectorMetadata([sub]))  # False

So under `multi`, every step whose only work is a state load is discarded, and
the request parked on that load waits for a report nobody was asked to
produce. A state-only load is not a corner: it is what a K3 admission does
whenever the boundary's checkpoint is in the tier and the KV prefix needs no
transfer, which the joint counters put at the overwhelming majority.

Fixed by asking the metadata instead of inspecting it. Each class declares its
own `WORK_FIELDS` next to the fields it defines, `has_work()` accumulates down
the hierarchy, and `MultiConnectorMetadata` delegates to its subs rather than
mirroring their fields -- so a field added to a sub can no longer be missed
here. The old union list stays only as the fallback for duck-typed doubles,
marked as not the place to register anything new.

The failure mode this replaces is the reason it is worth the indirection: a
forgotten entry is invisible at review, silent at runtime, and surfaces hours
later as a request that never completes.

Also drops a dead `_env_flag` import in `kimi_k3/staging.py` (ruff).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…kept two copies

`can_allocate` returns one number and the request needs two. What it may call
*cached* is gated on a resumable state sitting behind it; what it may point its
block table at is every block the prefix walk matched. `_gated_hit` cuts the
first, and the second was following it down.

So with a 4096-token prefix resident in HBM, no checkpoint anywhere inside it,
and the boundary at 5120, the request claimed nothing, and the KV leg was aimed
at `[0, 5120)` -- 4096 tokens of which the pool was already holding. Two costs,
both silent:

  * ~64 MB/rank/request of H2D that moved bytes the GPU had.
  * A second copy of them in HBM. `publish_loaded_prefix` keeps the existing
    canonical mapping, so the freshly written blocks are never indexed: they
    stay private to the request and occupy the pool until it finishes.

Two things were wrong. `_joint_kv_boundary` never recorded how far the walk had
actually matched, so `allocate` had no way to claim past the gated hit; and the
K3 `_decide_load_after_alloc` never wrote `hbm_cached_tokens` back onto the load
spec, so the transfer started wherever the *lookup* had left it -- which is
before `can_allocate` runs, i.e. 0. The second one was costing the refetch even
when the hit was not gated at all.

Now `_joint_kv_boundary` records `state_joint_claim_tokens` (the matched extent,
floored to the chunk grid), `allocate` claims to it while still calling only the
gated hit cached, and the KV leg starts there. On the case above the leg moves
`[4096, 5120)` -- 1024 tokens instead of 5120, with no second copy.

Claiming past the hit is safe precisely because a joint boundary exists: those
blocks sit below the boundary the forward will start from, so nobody writes
them. Without one the claim stays at the hit, where widening it would pin blocks
the forward is about to overwrite.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…e boundary

`KimiK3OffloadScheduler._claim_after_load` was written and never called. All
four completion sites inlined `max(hbm, lmc)` instead, and for k3 that `lmc` is
the chunk *covering* the state boundary, not the boundary:

    LMCACHE_CHUNK_SIZE=256, --block-size 128, boundary at 5120
    covering chunk ends at 5376  ->  the request claimed 5376

The forward then starts 256 tokens past what the recurrent state covers. The
KDA layers never saw that range, and nothing raises -- the output is just
wrong. Invisible today only because the recipe pins chunk == block_size, where
the covering chunk *is* the boundary.

Made it a real seam: `OffloadSchedulerMixin._claim_after_load` returns
`max(hbm, lmc)` -- correct for every layout whose claim and transfer share a
boundary, which is dense and dsv4 -- and k3's override clamps to its own. Four
call sites now go through it.

Scoped deliberately to the claim. `chunk != block_size` costs dense and dsv4
nothing but `unaligned_hbm_prefill` refusals, which fail closed, and it costs
k3 transfer overshoot into blocks the forward rewrites plus roughly half its
joint boundaries. All efficiency, no correctness, so there is nothing here to
refuse at startup.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Rebasing onto main brought #1989/#1990, which turned `Sequence.token_ids`
and `Sequence.block_table` into `array("i")` for the GC pause it buys at
long context. An `array("i")` never compares equal to a list, so the two
assertions this branch added in `TestJointClaimReusesResidentBlocks` --
correct when written against a `list` block table -- started failing.

Same fix main applied to its own test in #1990
(`assert list(seq.output_tokens) == [...]`): convert the array side.

  assert list(seq.block_table[:4]) == resident

`seq.block_table = [0]` in `test_scheduler.py` is deliberately left alone:
main has two of its own at lines 469 and 489 of that file, so a lone
converted third would read as the odd one out, and it passes.

The formatting half is `black .` over the nine source files and three
tests this branch had left non-compliant; main is black-clean, so every
one of them was ours. Three of the reformats join a wrapped expression
onto one line and the rest are blank lines -- `git diff -w
--ignore-blank-lines` is empty outside those three.

Verified: `pytest tests/ --ignore=tests/plugin` -> 3780 passed, 192
skipped, 3 xfailed. The two remaining collection errors (msgpack,
aiter) are on main as well. `tests/plugin` is excluded because
`test_rtpllm_forward_context_semantics.py` and
`test_rtpllm_glm5_sparse_backend_contract.py` replace
`sys.modules["atom.utils.forward_context"]` without restoring it, which
breaks every later test that builds a Scheduler -- 66 of main's own
included. Pre-existing and not this branch's to fix.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…'t hang

The K3 state-offload tier assumes every dispatched offload op eventually
reports completion, but LMCache's pin_monitor force-unpins a stalled
transfer's memory after pin_timeout_sec WITHOUT emitting that report. Two
counters then never clear:

  - a KV save's deferred blocks (deferred_free_blocks) are freed only on
    finished_saving, so has_pending_kv_work() stays True forever and the
    engine busy-loops with every GPU idle -- a hard hang under a tight HBM
    pool, a slow block leak under a large one (reproduced: J4 util=0.84
    hangs ~7min after READY, GPUs 0%, no preemption, no recovery);

  - a state spill's staging-ring slot comes home only via release_staging
    on the worker's report, so a lost report pins the slot until every
    further spill drops (spills_dropped climbs, ring starves).

Add two self-throttled reconcilers that reclaim after an abandon window:

  - Scheduler._reconcile_stalled_deferred_saves(), wired into the KV
    progress poll, frees blocks whose save has been deferred past
    OFFLOAD_SAVE_ABANDON_TIMEOUT_S (default 330s);

  - StateOffloadIndex.reclaim_stale_slots(), wired into the state-report
    drain, returns slots reserved longer than the same window and drops
    their undrained pending entries.

The abandon window is deliberately larger than LMCache's pin_timeout_sec
(default 300s): by the time we reclaim, upstream has already force-unpinned
and released the source bytes, so returning blocks/slots cannot race an
in-flight copy. Tests shrink OFFLOAD_SAVE_ABANDON_TIMEOUT_S and
LMCACHE_EC_PIN_TIMEOUT_SEC together to preserve that ordering; <= 0
disables reclamation and restores the wait-forever behaviour.

Also harden the producer finished_sending path: a save reclaimed by the
reconciler leaves nothing to free, so a late completion report is a no-op
(continue) rather than an assertion failure.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ming slots a batch still owes

Follow-up on the reconcilers, three changes.

**The window is no longer an ATOM knob.** `OFFLOAD_SAVE_ABANDON_TIMEOUT_S`
defaulted to 330s to sit above LMCache's 300s `pin_timeout_sec`, but that
ordering IS the safety argument and nothing enforced it: two independent
env vars, and setting ours below theirs would reclaim blocks a copy was
still reading, silently. It now reads LMCache's own
`LMCACHE_EC_PIN_TIMEOUT_SEC` and adds a 30s margin, so raising the pin
timeout carries the window with it and there is one number to get right
instead of two. Unset gives 330s, exactly the previous default. Memoised,
because both reconcilers ask for it on a 1ms poll
(`KV_IDLE_DRAIN_INTERVAL_S`) and the answer is a process constant.

The docstring now states the argument that actually holds. The timing
claim on its own did not: our clock starts when the block free is
deferred, LMCache's when it pins, and nothing aligns them. What makes
reclamation safe is that `OffloadWorkerMixin._guard` reports
`finished_saving` on both the success and the exception path, so a report
is lost only when `store()` neither returns nor raises. Two cases follow
and they are exhaustive: the parked save is not copying (LMCache
force-unpins its source and stops reading), and a save queued behind a
parked one never reaches `store()` at all.

**`reclaim_stale_slots` no longer touches `_pending`.** It used to drop
the stale slots' queued entries and return the slots. But a slot in
`_pending` has not been drained by `BlockManager.state_spills_for_batch`,
and that method drains `take_pending()` and the pool's
`take_spill_copies()` in ONE call -- so such a slot still has a
`(group, slot)` copy queued on `StateGroupPool`. Returning it let a new
spill reserve the same slot, and the two copies then raced for the single
`{slot: hash}` entry the drain builds: the loser logs "a copy with no
pending hash" and calls `release_staging`, freeing the slot out from under
the winner. Skipping queued slots removes the race and the `_pending`
surgery with it -- an undrained spill means no batch has run, which is
what the save reconciler is busy fixing, and the slot becomes eligible on
the next scan.

**`slots_reclaimed` reaches `stats()`.** The counter's own comment said it
existed so the symptom was visible rather than a silent throughput cliff,
but it was never added to the dict `StateGroupPool.checkpoint_fates`
merges, so it only appeared in the one-shot warning.

Also cross-referenced the two stall mechanisms in both docstrings. The K3
connector's `SAVE_STALL_SECONDS` escape releases the blocks of a save the
backend never took; this reconciler covers the save it took and never
answered. Neither subsumes the other and the pair is what leaves no
deferred save without a way out.

Verified: pytest tests/ --ignore=tests/plugin -> 3780 passed, 192 skipped,
3 xfailed (unchanged); black clean; ruff 33, same as main.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Comment on lines +388 to +389
if self._save_stalled and sid not in self._save_inflight:
if self._has_pending_save(seq):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ [ruff] <SIM102> reported by reviewdog 🐶
Use a single if statement instead of nested if statements

@zejunchen-zejun

Copy link
Copy Markdown
Collaborator Author

need to redesign based on PR: #2045

@zejunchen-zejun

Copy link
Copy Markdown
Collaborator Author

close as it is out-of-data, new PR is here: #2053

zejunchen-zejun added a commit that referenced this pull request Aug 28, 2026
Split out of PR #1960 so they can land on their own. None of them is about
state offload, and none needs #2045; blocking them behind either is pure loss,
and pulling them out narrows what is left to review by a few hundred lines.

--- a worker that outlives the join keeps the node

`EngineCore.exit` joined each worker with a 5s timeout and did nothing after
it. A worker that outlives that join keeps its VRAM slice and its all-reduce
IPC handles, and `multiprocessing`'s atexit handler then joins the same
process again with NO timeout -- so an engine already on its way out parks in
`_exit_function -> join` forever while /metrics keeps answering 200. Seen
twice on the k3-dev line; only a manual kill recovered the node.
`enable_orphan_reaping` cannot help: `PR_SET_PDEATHSIG` fires when the parent
*dies*, and this parent never does. Now escalates terminate -> kill.

--- a stalled offload save hung the engine with every GPU idle

A deferred save's blocks are freed only on `finished_saving`, but LMCache's
pin monitor force-unpins a stalled transfer after `pin_timeout_sec` WITHOUT
emitting that report. `has_pending_kv_work()` then never clears and the engine
busy-loops with every GPU idle -- a hard hang under a tight pool, a slow block
leak under a large one.

`Scheduler._reconcile_stalled_deferred_saves`, wired into the KV progress
poll, reclaims a save deferred past an abandon window and self-throttles to
5s so a 1ms poll costs nothing.

The window reads LMCache's own `LMCACHE_EC_PIN_TIMEOUT_SEC` and adds 30s
rather than taking an ATOM knob of its own, because that ordering IS the
safety argument: two independent env vars would let ours sit below the timeout
it must exceed, and nothing would say so. Unset gives 330s. Non-positive
disables reclamation and restores the wait-forever behaviour.

What makes reclaiming safe is not the clock -- our clock starts when the free
is deferred, LMCache's when it pins, and nothing aligns them. It is that
`OffloadWorkerMixin._guard` reports `finished_saving` on both the success and
the exception path, so a report is lost only when `store()` neither returns
nor raises. Two cases follow and they are exhaustive: the parked save is not
copying (LMCache force-unpinned its source and stopped reading), and a save
queued behind a parked one never reached `store()` at all.

`finished_sending`'s `assert seq is not None` becomes a `continue`: after a
reclaim, a late completion report has nothing left to free.

--- a state-only step under `multi` parked its request forever

`connector_metadata_has_work` decides whether a step's metadata reaches the
worker, and it decided from a hardcoded field list shared by three connector
families. `MultiConnectorMetadata` holds none of the work itself -- everything
is in `metas` -- so it mirrored the subs' fields as aggregating properties,
and missed one. Every step whose only work was in an unmirrored field was
discarded, and the request parked on it waited for a report nobody was asked
to produce.

Fixed by asking the metadata instead of inspecting it: each class declares its
own `WORK_FIELDS` next to the fields it defines, `has_work()` accumulates down
the hierarchy, and `MultiConnectorMetadata` delegates to its subs rather than
mirroring them -- so a field added to a sub can no longer be missed here. The
old union list stays only as the fallback for duck-typed doubles, marked as
not the place to register anything new.

--- two behaviour-neutral seams and one marker

`OffloadSchedulerMixin._claim_after_load` returns `max(hbm, lmc)`, which is
what all four inlined call sites did. A seam rather than four `max`es because
a layout whose claim and transfer aim at different boundaries has to override
it, and getting that wrong is silent: the request starts further along than
its state supports and the output is wrong with no exception.

`DenseOffloadScheduler._may_emit_save` returns True. It matters for a layout
whose `should_defer_free` pins a finished request's blocks until its save
drains -- an unbounded queue then lets a slow backend hold an unbounded slice
of the pool. `max_pending_saves` moves to `_offload_common` so every layout
reads the same `OFFLOAD_MAX_PENDING_SAVES`.

`KVCacheTensor.per_request_state` marks a hybrid's slot-addressed recurrent
state. It must be registered in `kv_cache_data` -- the linear-attention
forward reads it from there -- but no block-addressed mover may touch it, and
`DenseKVByteCodec` now skips it: including it either fails the divisibility
check or, if the slot count happens to divide `num_blocks`, inflates
`bytes_per_block` past what `block_regions` describes.

Verified on top of #2045 (`4df58434`): pytest tests/ (less tests/plugin and
the two msgpack/aiter collection errors that fail on #2045 too) -> 3871
passed, 326 skipped, 3 xfailed, zero failures. black clean; ruff 36, the same
count as #2045.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
zejunchen-zejun added a commit that referenced this pull request Aug 28, 2026
Phase 1 of remounting PR #1960 on #2045. Deliberately a RENAME, not a redesign:
the tier still hangs off `StateSlotPool` and still spills on eviction through
the staging ring. Phase 2 deletes the ring; Phase 3 moves the mount point to
`PageUnitCheckpointStore`. Keeping those apart is what lets a reviewer tell
rebase noise from design decisions.

Squashed rather than replayed for the reason #2045 gives for its own squash:
other's resolutions. #1960 keeps the original history.

--- what #2045 renamed underneath the branch

  StateGroupPool                -> StateSlotPool
  lookup_group(h)               -> lookup(h)
  seq.per_req_cache_group       -> seq.state_slots (list) + seq.state_slot
  num_per_req_cache_groups      -> num_state_slots + state_slots_per_req
  pop() / release()             -> pop_many(width) / release_many()
  has_free()                    -> has_free(n)
  _attach_state_group()         -> _attach_state_slots()

The semantic change hides in that table: **the "group" concept is gone.** Slots
are allocated one at a time, `pop_many` states outright that the set is not
adjacent and nothing may assume it is, and a checkpoint occupies exactly ONE
slot -- speculation scratch is not state anybody resumes into. Every
`group * span` address computation in #1960 lost its basis, so the tier's
`_spill` / `_resumable_from` / `has_pending_spill` / `take_spill_copies` are
re-expressed over slots, and `(group, staging)` becomes `(slot, staging)`.

--- the three places the two PRs actually collide

`_attach_state_slots` now returns bool. #2045 rewrote its body for per-slot
allocation; #1960 needs it to be able to say "the state behind this boundary is
not really here", because with the tier voting a hash can be accepted by
`_resumable_from` and still not be in HBM. The two compose: #2045's width logic
runs, and only the committed slot (element 0) is the load's destination, so
only it takes `pop(spill=False)` -- the rollback scratch is this request's own
and has nothing to forgo.

`allocate` keeps #1960's two extents (`num_cached_blocks` vs `claim_blocks`)
on top of #2045's per-slot attach, and the fresh-block loop starts from
`len(seq.block_table)` rather than `num_cached_blocks`, since the widened claim
already put blocks there.

`CacheStats` gains #1960's `total_offload_tokens` alongside #2045's
`total_reusable_tokens`. They are not the same series and neither subsumes the
other: `reusable` is the honest denominator for the HBM walk, while tier tokens
sit ABOVE that walk by design, so folding them in would make `cached > wanted`.
`update()` therefore takes five required arguments and one defaulted.

`deallocate` keeps #2045's midstep cancellation and `release_many`, and #1960's
in-flight-load hold: the committed slot is parked in `_orphan_load_slots` until
`settle_state_load`, because a worker is writing it; the scratch goes back
immediately.

`checkpoint_funnel` reports through `state_checkpoint_fates()` (a sum across
`state_caches`) rather than the single `_state_checkpoint_cache` call. Today
`state_caches` is a 1-tuple and the two are equivalent -- kept so Phase 3's
tier counters, which arrive from a second class, are not silently dropped.

--- functionality is NOT verified by this commit

Three gates all take the else branch for K3 under #2045, so the tier is inert:

  1. `isinstance(cache, StateSlotPool)` -- K3's `state_caches` is
     `(PagedStateCheckpointCoordinator,)`, so `cache.offload` is never set.
  2. `_joint_kv_boundary` returns `_no_joint("not_hybrid")` whenever
     `paged_state_checkpoints is not None`, which for K3 is always.
  3. `_attach_state_slots` returns from its PAGE branch before reaching
     `_tier_can_serve`.

That is deliberate and is what Phase 3/5 undo. Do not read the green suite
below as evidence the feature works -- the exit criterion for the tier is a
nonzero `state_tier` counter in the logs, which no unit test can produce.

Verified: pytest tests/ (less tests/plugin and the two msgpack/aiter collection
errors that fail on #2045 too) -> 3915 passed, 326 skipped, 3 xfailed, zero
failures. black clean; ruff 36, the same count as #2045.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
zejunchen-zejun added a commit that referenced this pull request Aug 28, 2026
…to rescue

Phase 2. About 1000 lines go, and none of them for tidiness: the ring existed
to solve exactly one problem, and #2045 removed the problem.

--- why the ring existed at all

A checkpoint used to live in a state slot, and `StateSlotPool.pop` handed that
slot to the next request the instant it was evicted. The bytes therefore had to
be rescued somewhere before they were overwritten, which is what the ring was:
K entries allocated inside the state arena, never leased to a request, and
addressed as `num_slots + staging` so `state_entry_views` could reach them with
no second scheme. Everything else -- `staging_entries`, `admission_entries`, the
drain protocol, the starvation detector, the reclaimer, `pop(spill=)`,
`spills_forgone`, the spill-before-relocate ordering in
`CommonAttentionBuilder.build` -- was a consequence.

Under #2045 a K3 checkpoint is a PAGE image in the KV pool whose 127 units are
*reserved*, and reserved units are structurally un-takeable: `_take_free` draws
only from `_free` and they sit in `_used`, `BlockPool.free` raises on one, and
`retire_top` refuses outright (`if top in self._raw_unit_owner: return None`).
The source holds still on its own, so a transfer needs no rescue -- exactly the
property dense KV has always had, where `should_defer_free` alone carries a save
through its D2H and no ring was ever needed.

So this is not a simplification, it is the removal of a workaround whose premise
is gone. `state_pool.py` and `sub_pool_spec.py` now have an EMPTY diff against

--- what stays, and why

`state_entry_views` stays (re-indexed from group to slot, since a checkpoint is
one slot wide now). Phase 4's load writes the committed slot directly, so this
is its destination -- the plan's earlier draft had it deleted, which was wrong.

`StagedTransfer` / `_StagingBuffer` stays. It shares a word with the ring and is
a different thing: a per-thread pinned host bounce buffer that D2H and H2D both
need, sized in bytes and unrelated to slot count.

`StateOffloadIndex` keeps its load half (`request_load` / `complete_load` /
`fail_load` / `abandon_load`) and its optimistic hash set. `confirm_spill`
becomes `note_stored`, which is what it always did.

`StateOffloadTier.submit_spill` becomes `submit_store(h, slot)` and loses its
`ready_event`: there is no compute-stream producer to fence against once the
packer gathers straight from a source that is not moving.

--- what this leaves inert, deliberately

**Nothing calls `submit_store` after this commit.** Phase 2 removed the ring
that used to drive it and Phase 3 gives it its new caller, whose source is the
checkpoint's PAGE units rather than a slot. Nothing populates
`StateOffloadIndex.hashes` in between, so `_tier_can_serve` is false and no load
is ever offered -- the tier is fully inert, on top of the three gates Phase 1
already documented. That is a WIP state, not a shippable one.

--- one dead-code find, pre-existing on #1960

A method between `_tier_can_serve` and `_request_state_load` had lost its `def`
line in an earlier rebase, leaving its docstring and `return` as unreachable
statements after `_tier_can_serve`'s own return. It reads like a live invariant
("Whether `hit_hash`'s state can be produced at all, from either tier") and has
no caller: the tier's vote actually reached `_gated_hit` through
`StateSlotPool._resumable_from`. Removed rather than restored, since Phase 4
puts the vote on the coordinator side.

Verified: pytest tests/ (less tests/plugin and the two msgpack/aiter collection
errors that fail on #2045 too) -> 3906 passed, 326 skipped, 3 xfailed, zero
failures. black clean; ruff 36, the same count as #2045. `git diff` against

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
zejunchen-zejun added a commit that referenced this pull request Aug 28, 2026
Split out of PR #1960 so they can land on their own. None of them is about
state offload, and none needs #2045; blocking them behind either is pure loss,
and pulling them out narrows what is left to review by a few hundred lines.

--- a worker that outlives the join keeps the node

`EngineCore.exit` joined each worker with a 5s timeout and did nothing after
it. A worker that outlives that join keeps its VRAM slice and its all-reduce
IPC handles, and `multiprocessing`'s atexit handler then joins the same
process again with NO timeout -- so an engine already on its way out parks in
`_exit_function -> join` forever while /metrics keeps answering 200. Seen
twice on the k3-dev line; only a manual kill recovered the node.
`enable_orphan_reaping` cannot help: `PR_SET_PDEATHSIG` fires when the parent
*dies*, and this parent never does. Now escalates terminate -> kill.

--- a stalled offload save hung the engine with every GPU idle

A deferred save's blocks are freed only on `finished_saving`, but LMCache's
pin monitor force-unpins a stalled transfer after `pin_timeout_sec` WITHOUT
emitting that report. `has_pending_kv_work()` then never clears and the engine
busy-loops with every GPU idle -- a hard hang under a tight pool, a slow block
leak under a large one.

`Scheduler._reconcile_stalled_deferred_saves`, wired into the KV progress
poll, reclaims a save deferred past an abandon window and self-throttles to
5s so a 1ms poll costs nothing.

The window reads LMCache's own `LMCACHE_EC_PIN_TIMEOUT_SEC` and adds 30s
rather than taking an ATOM knob of its own, because that ordering IS the
safety argument: two independent env vars would let ours sit below the timeout
it must exceed, and nothing would say so. Unset gives 330s. Non-positive
disables reclamation and restores the wait-forever behaviour.

What makes reclaiming safe is not the clock -- our clock starts when the free
is deferred, LMCache's when it pins, and nothing aligns them. It is that
`OffloadWorkerMixin._guard` reports `finished_saving` on both the success and
the exception path, so a report is lost only when `store()` neither returns
nor raises. Two cases follow and they are exhaustive: the parked save is not
copying (LMCache force-unpinned its source and stopped reading), and a save
queued behind a parked one never reached `store()` at all.

`finished_sending`'s `assert seq is not None` becomes a `continue`: after a
reclaim, a late completion report has nothing left to free.

--- a state-only step under `multi` parked its request forever

`connector_metadata_has_work` decides whether a step's metadata reaches the
worker, and it decided from a hardcoded field list shared by three connector
families. `MultiConnectorMetadata` holds none of the work itself -- everything
is in `metas` -- so it mirrored the subs' fields as aggregating properties,
and missed one. Every step whose only work was in an unmirrored field was
discarded, and the request parked on it waited for a report nobody was asked
to produce.

Fixed by asking the metadata instead of inspecting it: each class declares its
own `WORK_FIELDS` next to the fields it defines, `has_work()` accumulates down
the hierarchy, and `MultiConnectorMetadata` delegates to its subs rather than
mirroring them -- so a field added to a sub can no longer be missed here. The
old union list stays only as the fallback for duck-typed doubles, marked as
not the place to register anything new.

--- two behaviour-neutral seams and one marker

`OffloadSchedulerMixin._claim_after_load` returns `max(hbm, lmc)`, which is
what all four inlined call sites did. A seam rather than four `max`es because
a layout whose claim and transfer aim at different boundaries has to override
it, and getting that wrong is silent: the request starts further along than
its state supports and the output is wrong with no exception.

`DenseOffloadScheduler._may_emit_save` returns True. It matters for a layout
whose `should_defer_free` pins a finished request's blocks until its save
drains -- an unbounded queue then lets a slow backend hold an unbounded slice
of the pool. `max_pending_saves` moves to `_offload_common` so every layout
reads the same `OFFLOAD_MAX_PENDING_SAVES`.

`KVCacheTensor.per_request_state` marks a hybrid's slot-addressed recurrent
state. It must be registered in `kv_cache_data` -- the linear-attention
forward reads it from there -- but no block-addressed mover may touch it, and
`DenseKVByteCodec` now skips it: including it either fails the divisibility
check or, if the slot count happens to divide `num_blocks`, inflates
`bytes_per_block` past what `block_regions` describes.

Verified on top of #2045 (`4df58434`): pytest tests/ (less tests/plugin and
the two msgpack/aiter collection errors that fail on #2045 too) -> 3871
passed, 326 skipped, 3 xfailed, zero failures. black clean; ruff 36, the same
count as #2045.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
zejunchen-zejun added a commit that referenced this pull request Aug 28, 2026
Phase 1 of remounting PR #1960 on #2045. Deliberately a RENAME, not a redesign:
the tier still hangs off `StateSlotPool` and still spills on eviction through
the staging ring. Phase 2 deletes the ring; Phase 3 moves the mount point to
`PageUnitCheckpointStore`. Keeping those apart is what lets a reviewer tell
rebase noise from design decisions.

Squashed rather than replayed for the reason #2045 gives for its own squash:
other's resolutions. #1960 keeps the original history.

--- what #2045 renamed underneath the branch

  StateGroupPool                -> StateSlotPool
  lookup_group(h)               -> lookup(h)
  seq.per_req_cache_group       -> seq.state_slots (list) + seq.state_slot
  num_per_req_cache_groups      -> num_state_slots + state_slots_per_req
  pop() / release()             -> pop_many(width) / release_many()
  has_free()                    -> has_free(n)
  _attach_state_group()         -> _attach_state_slots()

The semantic change hides in that table: **the "group" concept is gone.** Slots
are allocated one at a time, `pop_many` states outright that the set is not
adjacent and nothing may assume it is, and a checkpoint occupies exactly ONE
slot -- speculation scratch is not state anybody resumes into. Every
`group * span` address computation in #1960 lost its basis, so the tier's
`_spill` / `_resumable_from` / `has_pending_spill` / `take_spill_copies` are
re-expressed over slots, and `(group, staging)` becomes `(slot, staging)`.

--- the three places the two PRs actually collide

`_attach_state_slots` now returns bool. #2045 rewrote its body for per-slot
allocation; #1960 needs it to be able to say "the state behind this boundary is
not really here", because with the tier voting a hash can be accepted by
`_resumable_from` and still not be in HBM. The two compose: #2045's width logic
runs, and only the committed slot (element 0) is the load's destination, so
only it takes `pop(spill=False)` -- the rollback scratch is this request's own
and has nothing to forgo.

`allocate` keeps #1960's two extents (`num_cached_blocks` vs `claim_blocks`)
on top of #2045's per-slot attach, and the fresh-block loop starts from
`len(seq.block_table)` rather than `num_cached_blocks`, since the widened claim
already put blocks there.

`CacheStats` gains #1960's `total_offload_tokens` alongside #2045's
`total_reusable_tokens`. They are not the same series and neither subsumes the
other: `reusable` is the honest denominator for the HBM walk, while tier tokens
sit ABOVE that walk by design, so folding them in would make `cached > wanted`.
`update()` therefore takes five required arguments and one defaulted.

`deallocate` keeps #2045's midstep cancellation and `release_many`, and #1960's
in-flight-load hold: the committed slot is parked in `_orphan_load_slots` until
`settle_state_load`, because a worker is writing it; the scratch goes back
immediately.

`checkpoint_funnel` reports through `state_checkpoint_fates()` (a sum across
`state_caches`) rather than the single `_state_checkpoint_cache` call. Today
`state_caches` is a 1-tuple and the two are equivalent -- kept so Phase 3's
tier counters, which arrive from a second class, are not silently dropped.

--- functionality is NOT verified by this commit

Three gates all take the else branch for K3 under #2045, so the tier is inert:

  1. `isinstance(cache, StateSlotPool)` -- K3's `state_caches` is
     `(PagedStateCheckpointCoordinator,)`, so `cache.offload` is never set.
  2. `_joint_kv_boundary` returns `_no_joint("not_hybrid")` whenever
     `paged_state_checkpoints is not None`, which for K3 is always.
  3. `_attach_state_slots` returns from its PAGE branch before reaching
     `_tier_can_serve`.

That is deliberate and is what Phase 3/5 undo. Do not read the green suite
below as evidence the feature works -- the exit criterion for the tier is a
nonzero `state_tier` counter in the logs, which no unit test can produce.

Verified: pytest tests/ (less tests/plugin and the two msgpack/aiter collection
errors that fail on #2045 too) -> 3915 passed, 326 skipped, 3 xfailed, zero
failures. black clean; ruff 36, the same count as #2045.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
zejunchen-zejun added a commit that referenced this pull request Aug 28, 2026
…to rescue

Phase 2. About 1000 lines go, and none of them for tidiness: the ring existed
to solve exactly one problem, and #2045 removed the problem.

--- why the ring existed at all

A checkpoint used to live in a state slot, and `StateSlotPool.pop` handed that
slot to the next request the instant it was evicted. The bytes therefore had to
be rescued somewhere before they were overwritten, which is what the ring was:
K entries allocated inside the state arena, never leased to a request, and
addressed as `num_slots + staging` so `state_entry_views` could reach them with
no second scheme. Everything else -- `staging_entries`, `admission_entries`, the
drain protocol, the starvation detector, the reclaimer, `pop(spill=)`,
`spills_forgone`, the spill-before-relocate ordering in
`CommonAttentionBuilder.build` -- was a consequence.

Under #2045 a K3 checkpoint is a PAGE image in the KV pool whose 127 units are
*reserved*, and reserved units are structurally un-takeable: `_take_free` draws
only from `_free` and they sit in `_used`, `BlockPool.free` raises on one, and
`retire_top` refuses outright (`if top in self._raw_unit_owner: return None`).
The source holds still on its own, so a transfer needs no rescue -- exactly the
property dense KV has always had, where `should_defer_free` alone carries a save
through its D2H and no ring was ever needed.

So this is not a simplification, it is the removal of a workaround whose premise
is gone. `state_pool.py` and `sub_pool_spec.py` now have an EMPTY diff against

--- what stays, and why

`state_entry_views` stays (re-indexed from group to slot, since a checkpoint is
one slot wide now). Phase 4's load writes the committed slot directly, so this
is its destination -- the plan's earlier draft had it deleted, which was wrong.

`StagedTransfer` / `_StagingBuffer` stays. It shares a word with the ring and is
a different thing: a per-thread pinned host bounce buffer that D2H and H2D both
need, sized in bytes and unrelated to slot count.

`StateOffloadIndex` keeps its load half (`request_load` / `complete_load` /
`fail_load` / `abandon_load`) and its optimistic hash set. `confirm_spill`
becomes `note_stored`, which is what it always did.

`StateOffloadTier.submit_spill` becomes `submit_store(h, slot)` and loses its
`ready_event`: there is no compute-stream producer to fence against once the
packer gathers straight from a source that is not moving.

--- what this leaves inert, deliberately

**Nothing calls `submit_store` after this commit.** Phase 2 removed the ring
that used to drive it and Phase 3 gives it its new caller, whose source is the
checkpoint's PAGE units rather than a slot. Nothing populates
`StateOffloadIndex.hashes` in between, so `_tier_can_serve` is false and no load
is ever offered -- the tier is fully inert, on top of the three gates Phase 1
already documented. That is a WIP state, not a shippable one.

--- one dead-code find, pre-existing on #1960

A method between `_tier_can_serve` and `_request_state_load` had lost its `def`
line in an earlier rebase, leaving its docstring and `return` as unreachable
statements after `_tier_can_serve`'s own return. It reads like a live invariant
("Whether `hit_hash`'s state can be produced at all, from either tier") and has
no caller: the tier's vote actually reached `_gated_hit` through
`StateSlotPool._resumable_from`. Removed rather than restored, since Phase 4
puts the vote on the coordinator side.

Verified: pytest tests/ (less tests/plugin and the two msgpack/aiter collection
errors that fail on #2045 too) -> 3906 passed, 326 skipped, 3 xfailed, zero
failures. black clean; ruff 36, the same count as #2045. `git diff` against

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
valarLip pushed a commit that referenced this pull request Sep 3, 2026
* fix(offload): the generic fixes #1960 was holding behind the state tier

Split out of PR #1960 so they can land on their own. None of them is about
state offload, and none needs #2045; blocking them behind either is pure loss,
and pulling them out narrows what is left to review by a few hundred lines.

--- a worker that outlives the join keeps the node

`EngineCore.exit` joined each worker with a 5s timeout and did nothing after
it. A worker that outlives that join keeps its VRAM slice and its all-reduce
IPC handles, and `multiprocessing`'s atexit handler then joins the same
process again with NO timeout -- so an engine already on its way out parks in
`_exit_function -> join` forever while /metrics keeps answering 200. Seen
twice on the k3-dev line; only a manual kill recovered the node.
`enable_orphan_reaping` cannot help: `PR_SET_PDEATHSIG` fires when the parent
*dies*, and this parent never does. Now escalates terminate -> kill.

--- a stalled offload save hung the engine with every GPU idle

A deferred save's blocks are freed only on `finished_saving`, but LMCache's
pin monitor force-unpins a stalled transfer after `pin_timeout_sec` WITHOUT
emitting that report. `has_pending_kv_work()` then never clears and the engine
busy-loops with every GPU idle -- a hard hang under a tight pool, a slow block
leak under a large one.

`Scheduler._reconcile_stalled_deferred_saves`, wired into the KV progress
poll, reclaims a save deferred past an abandon window and self-throttles to
5s so a 1ms poll costs nothing.

The window reads LMCache's own `LMCACHE_EC_PIN_TIMEOUT_SEC` and adds 30s
rather than taking an ATOM knob of its own, because that ordering IS the
safety argument: two independent env vars would let ours sit below the timeout
it must exceed, and nothing would say so. Unset gives 330s. Non-positive
disables reclamation and restores the wait-forever behaviour.

What makes reclaiming safe is not the clock -- our clock starts when the free
is deferred, LMCache's when it pins, and nothing aligns them. It is that
`OffloadWorkerMixin._guard` reports `finished_saving` on both the success and
the exception path, so a report is lost only when `store()` neither returns
nor raises. Two cases follow and they are exhaustive: the parked save is not
copying (LMCache force-unpinned its source and stopped reading), and a save
queued behind a parked one never reached `store()` at all.

`finished_sending`'s `assert seq is not None` becomes a `continue`: after a
reclaim, a late completion report has nothing left to free.

--- a state-only step under `multi` parked its request forever

`connector_metadata_has_work` decides whether a step's metadata reaches the
worker, and it decided from a hardcoded field list shared by three connector
families. `MultiConnectorMetadata` holds none of the work itself -- everything
is in `metas` -- so it mirrored the subs' fields as aggregating properties,
and missed one. Every step whose only work was in an unmirrored field was
discarded, and the request parked on it waited for a report nobody was asked
to produce.

Fixed by asking the metadata instead of inspecting it: each class declares its
own `WORK_FIELDS` next to the fields it defines, `has_work()` accumulates down
the hierarchy, and `MultiConnectorMetadata` delegates to its subs rather than
mirroring them -- so a field added to a sub can no longer be missed here. The
old union list stays only as the fallback for duck-typed doubles, marked as
not the place to register anything new.

--- two behaviour-neutral seams and one marker

`OffloadSchedulerMixin._claim_after_load` returns `max(hbm, lmc)`, which is
what all four inlined call sites did. A seam rather than four `max`es because
a layout whose claim and transfer aim at different boundaries has to override
it, and getting that wrong is silent: the request starts further along than
its state supports and the output is wrong with no exception.

`DenseOffloadScheduler._may_emit_save` returns True. It matters for a layout
whose `should_defer_free` pins a finished request's blocks until its save
drains -- an unbounded queue then lets a slow backend hold an unbounded slice
of the pool. `max_pending_saves` moves to `_offload_common` so every layout
reads the same `OFFLOAD_MAX_PENDING_SAVES`.

`KVCacheTensor.per_request_state` marks a hybrid's slot-addressed recurrent
state. It must be registered in `kv_cache_data` -- the linear-attention
forward reads it from there -- but no block-addressed mover may touch it, and
`DenseKVByteCodec` now skips it: including it either fails the divisibility
check or, if the slot count happens to divide `num_blocks`, inflates
`bytes_per_block` past what `block_regions` describes.

Verified on top of #2045 (`4df58434`): pytest tests/ (less tests/plugin and
the two msgpack/aiter collection errors that fail on #2045 too) -> 3871
passed, 326 skipped, 3 xfailed, zero failures. black clean; ruff 36, the same
count as #2045.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* lmcache(k3): port the state offload tier onto #2045's per-slot pool

Phase 1 of remounting PR #1960 on #2045. Deliberately a RENAME, not a redesign:
the tier still hangs off `StateSlotPool` and still spills on eviction through
the staging ring. Phase 2 deletes the ring; Phase 3 moves the mount point to
`PageUnitCheckpointStore`. Keeping those apart is what lets a reviewer tell
rebase noise from design decisions.

Squashed rather than replayed for the reason #2045 gives for its own squash:
other's resolutions. #1960 keeps the original history.

--- what #2045 renamed underneath the branch

  StateGroupPool                -> StateSlotPool
  lookup_group(h)               -> lookup(h)
  seq.per_req_cache_group       -> seq.state_slots (list) + seq.state_slot
  num_per_req_cache_groups      -> num_state_slots + state_slots_per_req
  pop() / release()             -> pop_many(width) / release_many()
  has_free()                    -> has_free(n)
  _attach_state_group()         -> _attach_state_slots()

The semantic change hides in that table: **the "group" concept is gone.** Slots
are allocated one at a time, `pop_many` states outright that the set is not
adjacent and nothing may assume it is, and a checkpoint occupies exactly ONE
slot -- speculation scratch is not state anybody resumes into. Every
`group * span` address computation in #1960 lost its basis, so the tier's
`_spill` / `_resumable_from` / `has_pending_spill` / `take_spill_copies` are
re-expressed over slots, and `(group, staging)` becomes `(slot, staging)`.

--- the three places the two PRs actually collide

`_attach_state_slots` now returns bool. #2045 rewrote its body for per-slot
allocation; #1960 needs it to be able to say "the state behind this boundary is
not really here", because with the tier voting a hash can be accepted by
`_resumable_from` and still not be in HBM. The two compose: #2045's width logic
runs, and only the committed slot (element 0) is the load's destination, so
only it takes `pop(spill=False)` -- the rollback scratch is this request's own
and has nothing to forgo.

`allocate` keeps #1960's two extents (`num_cached_blocks` vs `claim_blocks`)
on top of #2045's per-slot attach, and the fresh-block loop starts from
`len(seq.block_table)` rather than `num_cached_blocks`, since the widened claim
already put blocks there.

`CacheStats` gains #1960's `total_offload_tokens` alongside #2045's
`total_reusable_tokens`. They are not the same series and neither subsumes the
other: `reusable` is the honest denominator for the HBM walk, while tier tokens
sit ABOVE that walk by design, so folding them in would make `cached > wanted`.
`update()` therefore takes five required arguments and one defaulted.

`deallocate` keeps #2045's midstep cancellation and `release_many`, and #1960's
in-flight-load hold: the committed slot is parked in `_orphan_load_slots` until
`settle_state_load`, because a worker is writing it; the scratch goes back
immediately.

`checkpoint_funnel` reports through `state_checkpoint_fates()` (a sum across
`state_caches`) rather than the single `_state_checkpoint_cache` call. Today
`state_caches` is a 1-tuple and the two are equivalent -- kept so Phase 3's
tier counters, which arrive from a second class, are not silently dropped.

--- functionality is NOT verified by this commit

Three gates all take the else branch for K3 under #2045, so the tier is inert:

  1. `isinstance(cache, StateSlotPool)` -- K3's `state_caches` is
     `(PagedStateCheckpointCoordinator,)`, so `cache.offload` is never set.
  2. `_joint_kv_boundary` returns `_no_joint("not_hybrid")` whenever
     `paged_state_checkpoints is not None`, which for K3 is always.
  3. `_attach_state_slots` returns from its PAGE branch before reaching
     `_tier_can_serve`.

That is deliberate and is what Phase 3/5 undo. Do not read the green suite
below as evidence the feature works -- the exit criterion for the tier is a
nonzero `state_tier` counter in the logs, which no unit test can produce.

Verified: pytest tests/ (less tests/plugin and the two msgpack/aiter collection
errors that fail on #2045 too) -> 3915 passed, 326 skipped, 3 xfailed, zero
failures. black clean; ruff 36, the same count as #2045.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* refactor(offload): delete the staging ring, which #2045 left nothing to rescue

Phase 2. About 1000 lines go, and none of them for tidiness: the ring existed
to solve exactly one problem, and #2045 removed the problem.

--- why the ring existed at all

A checkpoint used to live in a state slot, and `StateSlotPool.pop` handed that
slot to the next request the instant it was evicted. The bytes therefore had to
be rescued somewhere before they were overwritten, which is what the ring was:
K entries allocated inside the state arena, never leased to a request, and
addressed as `num_slots + staging` so `state_entry_views` could reach them with
no second scheme. Everything else -- `staging_entries`, `admission_entries`, the
drain protocol, the starvation detector, the reclaimer, `pop(spill=)`,
`spills_forgone`, the spill-before-relocate ordering in
`CommonAttentionBuilder.build` -- was a consequence.

Under #2045 a K3 checkpoint is a PAGE image in the KV pool whose 127 units are
*reserved*, and reserved units are structurally un-takeable: `_take_free` draws
only from `_free` and they sit in `_used`, `BlockPool.free` raises on one, and
`retire_top` refuses outright (`if top in self._raw_unit_owner: return None`).
The source holds still on its own, so a transfer needs no rescue -- exactly the
property dense KV has always had, where `should_defer_free` alone carries a save
through its D2H and no ring was ever needed.

So this is not a simplification, it is the removal of a workaround whose premise
is gone. `state_pool.py` and `sub_pool_spec.py` now have an EMPTY diff against

--- what stays, and why

`state_entry_views` stays (re-indexed from group to slot, since a checkpoint is
one slot wide now). Phase 4's load writes the committed slot directly, so this
is its destination -- the plan's earlier draft had it deleted, which was wrong.

`StagedTransfer` / `_StagingBuffer` stays. It shares a word with the ring and is
a different thing: a per-thread pinned host bounce buffer that D2H and H2D both
need, sized in bytes and unrelated to slot count.

`StateOffloadIndex` keeps its load half (`request_load` / `complete_load` /
`fail_load` / `abandon_load`) and its optimistic hash set. `confirm_spill`
becomes `note_stored`, which is what it always did.

`StateOffloadTier.submit_spill` becomes `submit_store(h, slot)` and loses its
`ready_event`: there is no compute-stream producer to fence against once the
packer gathers straight from a source that is not moving.

--- what this leaves inert, deliberately

**Nothing calls `submit_store` after this commit.** Phase 2 removed the ring
that used to drive it and Phase 3 gives it its new caller, whose source is the
checkpoint's PAGE units rather than a slot. Nothing populates
`StateOffloadIndex.hashes` in between, so `_tier_can_serve` is false and no load
is ever offered -- the tier is fully inert, on top of the three gates Phase 1
already documented. That is a WIP state, not a shippable one.

--- one dead-code find, pre-existing on #1960

A method between `_tier_can_serve` and `_request_state_load` had lost its `def`
line in an earlier rebase, leaving its docstring and `return` as unreachable
statements after `_tier_can_serve`'s own return. It reads like a live invariant
("Whether `hit_hash`'s state can be produced at all, from either tier") and has
no caller: the tier's vote actually reached `_gated_hit` through
`StateSlotPool._resumable_from`. Removed rather than restored, since Phase 4
puts the vote on the coordinator side.

Verified: pytest tests/ (less tests/plugin and the two msgpack/aiter collection
errors that fail on #2045 too) -> 3906 passed, 326 skipped, 3 xfailed, zero
failures. black clean; ruff 36, the same count as #2045. `git diff` against

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* lmcache(k3): one CPU pool for KV and state, not two

Phase 3a. Deletes the tier's separate `atom-state-{rank}` LMCache engine, its
`OFFLOAD_STATE_CPU_SIZE` knob, and its `cache_policy = "FIFO"` override. The
codec now binds the same `storage_manager` the KV leg uses.

--- what changed

  - `_state_storage_manager()` is gone (~50 lines). `bind_storage_manager` is
    handed `self._engine.storage_manager` directly. The `gib <= 0` branch that
    already did this was the whole implementation; what goes is the `> 0` half.
  - `OFFLOAD_STATE_CPU_SIZE` removed, from the code and from
    `docs/environment_variables.md`.
  - `cache_policy = "FIFO"` removed; state inherits the KV pool's LRU.

--- why the two-pool argument does not hold

It rested on "the KV write stream is several times the state volume, so it
evicts the checkpoints". Measured against the code rather than assumed:
`page_unit_bytes = num_MLA_layers * block_size * entry * itemsize = 442,368`
with `--block-size 128` pinned by the recipe, so KV costs 3,456 B/token and a
53.6 MiB state image is worth ~16k tokens of it -- 13% of a 117k-token prompt's
KV. (Read the MLA layer count the other way, as a quarter of 93 layers, and it
is 3.3%. The conclusion is the same either way, which is why it did not need
settling first.)

And the byte ratio is not what decides an LRU eviction anyway. A request writes
its ~457 KV chunks and its one state object inside the same prefill window, so
both enter the LRU at the same position and cool at the same rate. State is
retired ALONGSIDE its own KV -- which is the behaviour we want, because a
boundary whose KV is gone is worthless.

Two pools cost what one does not: two eviction policies that must drift, while
a joint boundary needs both legs to survive together. #2045 made this same
argument one tier up -- state moved out of its reserved slots into the KV pool,
81.85% -> 93.60% -- and `OFFLOAD_STATE_CPU_SIZE` is the same reserved capacity
one tier down, so keeping it repeats the thing #2045 disproved.
`LMCACHE_MAX_LOCAL_CPU_SIZE` is now the one size to tune.

--- the cost, stated rather than hidden

`cache_policy` is per-engine, so one pool means one policy for both legs. That
is a real loss of freedom and it is currently free, because LRU is right for
both. The FIFO override's premise was "a state entry is written once and read
once"; #2045's own numbers refute it (~4,808 resumes over ~1,508 checkpoints is
~3 reads each), and re-access is bursty -- branches and retries on one boundary,
then superseded -- which is LRU's home ground. The genuine want, if it shows up,
is "stickier than LRU gives it", which FIFO does not provide either; the lever
then is touching the state key on a KV hit, not splitting the pool back apart.

Nothing yet measures whether that is needed. The counter to add with the store
path (Phase 3d) is `joint_boundary_state_miss`: a boundary whose KV leg landed
and whose state leg missed. Zero means the two pools never needed separating.

Verified: pytest tests/test_lmcache_offload_connector.py -> 186 passed. black
clean; ruff unchanged.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* lmcache(k3): bind the state key to its layout, and refuse a misfit read

Phase 3b. `StateByteCodec.key()` stops being the bare ATOM hash; it becomes
`xxh64(hash || layout_id)`. `get()` gains a size check on what comes back.

--- what changed

  - `StateByteCodec.__init__` takes a required `layout_id` and refuses an empty
    one. `_build_state_tier` reads it off `state_runtime.checkpoint_spec` and
    declines to build the tier at all when it is absent, rather than falling
    back to an unqualified key.
  - `key(h)` hashes `h` and the layout id together with `xxh64`.
  - `get(h, ...)` measures the returned `MemoryObj` and, on a size other than
    `entry_bytes`, discharges its reference, counts `_misfit_reads`, warns, and
    returns a miss.
  - `_object_bytes()` reads `get_size()` with the tensor as fallback, and
    returns None when neither answers.

--- why the key needed both

*Build safety.* The same prefix hash names a completely different image under a
different `num_spec`, TP size, or conv/ssm dtype. A size mismatch throws at
`entry_bytes`, but **the same size with a different order or meaning reads back
silently wrong state** -- and a request resumed onto wrong state produces wrong
output with nothing raised anywhere. #2045 already encodes all of it in
`layout_id` (layers / conv shape+dtype / ssm shape+dtype / order / tp / spec /
carry) and enforces it HBM-side in `_validate_paged_state_op`. This is the CPU
side of the same check, and it was missing.

*Namespace separation.* `CacheEngineKey` is
`(model_name, world_size, worker_id, chunk_hash, dtype)` -- no field says what an
entry IS. KV chunk keys carry `ChunkedTokenDatabase`'s chunk hash and state keys
carry ATOM's chained block hash: two different hash functions writing plain
integers into one space. Until Phase 3a they at least lived in different pools;
now they share one, and the only thing keeping them apart was accidental -- this
side hard-codes `torch.uint8` while KV carries the KV dtype. `MemoryFormat` is
not available as a discriminator either: the tier's own comment records that its
`KV_2LTD` is inert, chosen only to pass the allocator's `raise ValueError`.
Folding the layout in makes the two spaces disjoint by construction, since no KV
key can carry a K3 layout id.

`xxh64` rather than Python's `hash((h, layout_id))`: `hash` of a str is salted
per process, so a restart would silently orphan every entry the previous run
wrote -- a cache that quietly starts cold and never says so. `xxh64` is what the
block hashes themselves use.

--- why the size check, if the key already prevents it

It is unreachable, and that is the point: reaching it means two things collided
in the shared pool, and unpacking a colliding object writes another entry's
bytes over a request's live state. Degrading to a miss costs one recompute; not
checking costs silent wrong output. `_misfit_reads` is the counter that says the
key stopped doing its job.

None from `_object_bytes` means "cannot measure", never "size 0" -- refusing
what cannot be measured would turn an unknown into a guaranteed miss.

--- tests

Five, in `tests/test_lmcache_offload_connector.py`, on a `CacheEngineKey`
fixture rather than the full lmcache stub (these are about two keys being equal
or not, which the real dataclass answers by value): a layout change makes a
different key; the layout does not collapse distinct hashes; the key survives a
restart; a state key cannot equal a raw-hash KV key; a wrong-sized hit is a miss
with its reference discharged; an object that will not report a size is still
loaded.

Verified: pytest tests/ (less tests/plugin and the two msgpack/aiter collection
errors that fail on #2045 too) -> 3911 passed, zero failures. black clean;
ruff 36, unchanged.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* k3: name a checkpoint's PAGE units as tensors, for the CPU tier's packer

Phase 3c. Adds `_KimiMLAGDNCommon.page_unit_views(unit_ids)` -- the tensor-view
counterpart of #2045's `_page_unit_regions`. No behaviour changes; nothing calls
it until Phase 3d.

--- what changed

  - `page_unit_views(unit_ids) -> list[torch.Tensor]`, one view per (unit, MLA
    row), unit major and row minor.
  - Seven tests in `tests/test_kda_checkpoint_slot_copy.py`, on the harness
    #2045 already built for `_page_unit_bases`.

--- why a second way to name the same bytes

`_page_unit_regions` hands the Triton descriptor raw int64 addresses, which is
what a D2D copy the GPU issues wants. The LMCache staging packer takes
`list[torch.Tensor]` -- `fused_pack_chunk_major` is already a fully
parameterised gather over `segment_ptrs[] + segment_block_bytes[]`, so it needs
no kernel change, only its segments named the way it names them.

This is the ONLY new seam the tier needs. A load writes the Active Slot directly
and reuses `state_entry_views`, so the asymmetry is deliberate: the save gathers
from units, the load scatters to a slot, and the two compose because #2045's
`plan_segmented_copy` intersects two *ordered byte streams* -- the slot's ranges
and the units' regions are two cuts of the same logical image, so a blob
gathered in one order is byte-identical to one gathered in the other.

--- the two traps, both asserted rather than assumed

*Order is the contract.* Unit major, row minor -- the same ravel as
`_page_unit_bases`, which is the order `_checkpoint_copy_plan` builds the
destination stream in. The packer indexes segments positionally, so this order
IS the blob layout. Self-consistency would be enough for one build; what makes
it a contract across builds is `layout_id` in the key (Phase 3b), so a build
that reordered cannot read another's blob.

*`block_ratio`.* `unit_ids` carries **logical** block ids while `kv_cache` is
shaped in **physical** ones, and K3's ratio is 128. Flattening the two block
axes together and indexing by `unit * block_size` is the conversion -- the same
arithmetic `_page_unit_bases` does in addresses, so the two cannot drift into
disagreeing about which bytes a unit owns. The range check is against the
*logical* count for the same reason: the physical count is `ratio` times larger
in rows, so checking the wrong axis admits ids that read past the end. Its test
computes the bound from the tensor rather than from a constant, because taking
the harness's label would let the very confusion under test pass unnoticed.

`_page_unit_regions` is called first for its side effect: the contiguity and
granularity checks live there, and they have to run before any view is handed
out rather than after.

--- verification

The unit tests skip on a non-GPU runner (the module imports aiter at load), so
the geometry was checked directly by extracting the four methods and comparing
`page_unit_views` against `_page_unit_bases` byte for byte: 9 segments for 3
units x 3 rows, identical `data_ptr()`s, lengths equal to
`_page_unit_stream_sizes`, every view contiguous, no two units sharing a byte,
an out-of-range id refused, and a granularity mismatch refused before any view
is produced.

pytest tests/ (less tests/plugin and the two msgpack/aiter collection errors
that fail on #2045 too) -> 3911 passed, zero failures. black clean; ruff 36,
unchanged.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* lmcache(k3): store a checkpoint when it becomes READY, not when it is evicted

Phase 3d, the last of Phase 3. This is what turns the tier back on for saves:
a checkpoint that reaches READY is offered to the CPU tier, and the whole
engine->worker->engine round trip is wired. Loads are still off (Phase 4).

--- what changed

`page_unit_checkpoint.py`
  - `_offload_ready`: checkpoints nominated at the READY transition. NOT pinned.
  - `take_offload_stores(max_inflight)`: hands `(hash, unit_ids)` over and takes
    the pin at that moment, up to the cap.
  - `settle_offload_store(hash)` / `reclaim_stale_offload_pins(timeout_s)` /
    `offload_pins_reclaimed`, exposed on the coordinator and reported in
    `checkpoint_fates`.
  - `PageUnitCheckpointStore(..., offload_sink=)` -- False nominates nothing.

`block_manager.py`
  - `take_state_stores` / `settle_state_store(hash, ok)` /
    `reclaim_stale_state_store_pins`. The switch (`kv_connector_hosts_state_tier`)
    moves above the coordinator's construction, which now needs it.

`scheduler.py`
  - `_publish_state_stores()`, beside `_publish_state_loads()` and on the same
    two call sites plus the idle path.
  - `_offload_max_pending_saves()`, reading the KV leg's own
    `OFFLOAD_MAX_PENDING_SAVES`.
  - The report drain settles the pin on both outcomes and runs the reconciler.

`kimi_k3/connector.py` -- `enqueue_state_stores`, `meta.state_stores`,
`_start_state_stores`, and `_state_store_failed_locally` for stores a worker
with no tier cannot even attempt.
`kimi_k3/state_tier.py` -- `submit_store(h, unit_ids)`, no `ready_event`.
`kimi_k3/state_object.py` -- `put(h, unit_ids)` gathers `page_unit_views`;
`get(h, slot)` still scatters into `state_entry_views`; `puts_refused`.
`metadata.py` / `types.py` -- `state_stores` declared as work.

--- why READY, and why the pin is not taken there

Three candidate trigger points and only one works. `begin_store` is too early:
the scatter has not ridden a batch, so the units do not hold the image yet.
`_evict` / `unindex` are too late -- they fire when the pool wants those units
*now*, so a multi-millisecond D2H started there holds 127 of the most-wanted
blocks at the worst possible moment. READY is when the bytes first exist and
when the record is least wanted: it has just entered the LRU at the cold end.

The pin is a separate question, and getting it wrong broke a real invariant --
`test_a_ready_unpinned_checkpoint_is_available_to_live_kv` caught the first
draft. A pin lives in this process while the copy runs in the worker, so it
spans several scheduler passes. Pinning at READY would make EVERY checkpoint
un-evictable for that window, which is exactly what #2045's admission argument
forbids. So READY only *nominates*: a nominee stays unpinned and spendable, and
`take_offload_stores` pins only the few actually in flight. A checkpoint the
pool needed more than the tier did simply loses its copy, which is the right
price for never making the pool wait on the CPU.

The cap is `OFFLOAD_MAX_PENDING_SAVES`, shared with the KV leg rather than given
a knob of its own: a KV save and a state store both hold bytes out of the same
pool while they run, so the depth is one question about one resource, and two
numbers would let an operator raise one and unknowingly double the other.

--- write volume, which is the objection this trigger invites

Storing every checkpoint rather than only evicted ones is more traffic, and it
is affordable: per request state adds 53.6 MiB on top of ~404 MB of KV, **13%**
on a path already in flight (~12 MB/s against the KV leg's ~90 MB/s over a
3600s run). What it buys is the trigger being at the cheapest possible moment
and dedup for free -- `begin_store` already refuses a hash it holds or has
pending, so one hash is offered once.

--- unindex during a copy does not cancel it

`unindex` means the boundary's KV block was spent, so the image is unreachable
in HBM from here on. That is precisely when the CPU copy is worth having, so
the pin holds the units to the end of the D2H and `_release_record` runs after.
The CPU copy outliving the HBM one is the entire point of the tier.

--- the lost report, and why 20 lines is enough

`reclaim_stale_offload_pins` takes the same window as
`_reconcile_stalled_deferred_saves`, derived from LMCache's own pin timeout for
the same reason. Recovery is total: a leaked pin breaks no `BlockPool`
invariant and leaves no half-released state, so zeroing the count restores the
record exactly -- which is why this needs a counter and a timer rather than any
repair logic. `offload_pins_reclaimed` reaches `checkpoint_fates`, because
otherwise the symptom is a pool that has quietly shrunk.

--- still off

Loads. Nothing calls `_request_state_load` for K3 yet: `_attach_state_slots`
returns from its PAGE branch before reaching `_tier_can_serve`, and the
`begin_restore` raise above it has to become an issue-a-load branch first
(Phase 4). `StateOffloadIndex.hashes` now fills, so the tier is half live: it
writes, and nothing reads yet.

--- tests

11 new in `tests/test_page_unit_checkpoint.py` -- nomination does not pin, the
hand-off does, a nominee stays spendable and may be lost, no sink nominates
nothing, offered once, the cap makes the rest wait rather than drop, a waiting
nominee stays evictable, both outcomes release, a report for a hash never sent
is a no-op, `unindex` mid-copy holds the units to the end, and a lost report is
reclaimed with the record left unbroken. Two more in
`tests/test_lmcache_offload_connector.py`: stores drain exactly once, and a
step whose only work is a store still reaches the worker.

Verified: pytest tests/ (less tests/plugin and the two msgpack/aiter collection
errors that fail on #2045 too) -> 3924 passed, zero failures. black clean;
ruff 36, unchanged.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(state-cache): a gated boundary neither tier has is disowned, not raised

Phase 4a, and it has to be first: with nothing else changed this commit is
behaviour-preserving, but without it the first request that actually uses the
CPU tier takes the engine down.

--- what changed

`BlockManager._attach_state_slots`, PAGE branch. Was:

    if hit_hash != -1 and not begin_restore(hit_hash, seq.state_slot):
        release_many(seq.state_slots); seq.state_slots = []
        raise RuntimeError("gated PAGE checkpoint disappeared before state attach")

Now: cold start returns early; `begin_restore` success returns as before; a
miss falls through to `_request_state_load`, and only if that also declines is
the boundary disowned (`return False`, which `allocate` turns into
`num_cached_tokens = 0`). New counter `state_gate_lost_boundary`, reported in
`checkpoint_funnel`.

--- why the raise could not stay

It asserted that the gate and the HBM store agree. That held while
`can_allocate` only ever accepted boundaries the HBM index carried. Phase 4b
gives the CPU tier a vote, so a hash the gate accepted may live only there --
and every such request would hit this raise. Turning it into a load branch has
to land before the vote does, not after, or the two commits are only safe as a
pair.

--- why disowning is reachable rather than defensive

Two ways to arrive with neither tier able to produce it:

  * the tier's index is optimistic by construction -- `hashes` means "was
    stored once", never "is still there", because LMCache's own LRU can drop
    bytes at any time;
  * an HBM checkpoint can be unindexed between `can_allocate` and `allocate`
    by another seq's `_fresh_block` in the same pass.

So this is a normal path with a counter, not an assertion. A large fraction of
`joint_boundaries` landing here would mean the gate is accepting boundaries
that do not survive to attach.

--- the slots are kept, unlike the old abort

The old code released the slots because it was aborting. Disowning is not an
abort: the request is about to recompute its whole prefix, and it writes that
state into these very slots. Releasing them would hand the next request a
buffer this one is still filling.

`test_missing_gated_checkpoint_releases_the_new_slot_and_raises` becomes
`test_a_gated_boundary_neither_tier_has_is_disowned_not_raised`, and now pins
all three: returns False, keeps its slots, counts one.

Verified: pytest tests/ (less tests/plugin and the two msgpack/aiter collection
errors that fail on #2045 too) -> zero failures. black clean; ruff unchanged.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* lmcache(k3): give the CPU tier a vote, and let a PAGE resume come from it

Phase 4b. This is the commit that makes the tier reachable: until now K3 wrote
53.6 MiB per checkpoint to LMCache and nothing ever read it.

--- what changed

`page_unit_checkpoint.py`
  - `PagedStateCheckpointCoordinator.offload` + `attach_offload(index)`.
  - `resumable_hit` now asks `_reachable(h)` instead of `store.contains(h)`:
    HBM first, then `offload.hashes`.
  - The constructor's `offload_sink: bool` is gone -- `attach_offload` sets the
    vote and the sink together (see below).

`block_manager.py`
  - Attaches the index to the coordinator once both exist. The coordinator is
    built from `state_runtime.checkpoint_spec` and the index from the connector
    config, so they are constructed in the wrong order for a ctor argument.

`kimi_k3/connector.py`
  - Refuses to build the tier when `image_bytes != entry_bytes`.

--- why the vote is the whole feature

`can_allocate` walks the HBM prefix and asks every `Pool.STATE` class "the
rightmost boundary <= X that you accept". With the coordinator answering only
from `store.contains`, a hash whose image went to LMCache is invisible: the
walk stops short, nothing ever asks for it back, and the store path is pure
cost. `_reachable` is where the tier gets to say "I have that one".

It stops at the first boundary it accepts, scanning right to left, so a hash
accepted here that nothing can deliver does not cost a wasted lookup -- it
costs the whole walk-back, hiding every shorter checkpoint still resident in
HBM. That is why the disown path (Phase 4a) had to land first.

No preference rule between the tiers is needed: both are keyed by the same
content hash, so the scan takes the rightmost boundary wherever it lives, and
`_attach_state_slots` tries `begin_restore` before `_request_state_load`
regardless. A resident image never pays a park.

--- why the vote is optimistic, deliberately

`hashes` means "was stored once", never "is still there" -- LMCache's own LRU
can drop bytes under it at any moment. A false positive costs one park plus a
recompute, and retracts itself: `fail_load` calls `forget(h)`, so the next
request over that prefix is not sent down the same hole. Being certain instead
would mean a synchronous cross-process lookup on the admission path.

--- why attaching is one call, not two fields

Half-attached is worse than off in both directions. A vote with no sink accepts
hashes nothing will ever store, so every accepting request parks and
recomputes. A sink with no vote pins 127 units per checkpoint for a transfer
nobody will read. `attach_offload` sets both, so they cannot diverge.

--- the load's destination is the Active Slot

Not 127 freshly reserved units. The two are interchangeable because #2045's
copy plan intersects two *ordered byte streams* -- the slot's ranges and the
units' regions are two cuts of one image -- so a blob gathered in unit order
unpacks correctly in slot order. Writing the slot directly means no
`ensure_free_units` on the admission path, no refusal branch, no units pinned
for the length of an H2D, and no cap needed on concurrent loads.

It does mean the store reads `image_bytes` and the load writes a whole slot, so
the connector now refuses to build a tier where those differ, at the one point
both numbers are in scope. They are equal for K3; a model where they are not
would silently truncate the store or over-read the load.

--- tests

Six in `tests/test_page_unit_checkpoint.py`: a tier-only hash is accepted, no
tier means no acceptance, the scan still takes the rightmost, attaching turns
on both halves together, and the optimistic half retracts on `forget`.

Three in `tests/test_state_checkpoint.py`, end to end through
`_attach_state_slots`: a tier-only boundary becomes a queued load with no
restore op and the slot as its destination; a boundary in both is served from
HBM with no park; a tier that declines disowns and counts.

Verified: pytest tests/ (less tests/plugin and the two msgpack/aiter collection
errors that fail on #2045 too) -> zero failures. black clean; ruff unchanged.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* lmcache(k3): the joint boundary now serves PAGE, which is its only reason to exist

Phase 5, the last one. `_joint_kv_boundary` refused every PAGE sequence; it now
serves only PAGE. With this the tier is reachable end to end.

--- what changed

`block_manager.py`
  - The gate inverts:
      -  if not seq.has_per_req_cache or self.paged_state_checkpoints is not None:
      -      return self._no_joint("not_hybrid")
      +  if not seq.has_per_req_cache or self.paged_state_checkpoints is None:
      +      return self._no_joint("no_paged_checkpoints")
  - The hbm/tier split reads the coordinator instead of the slot pool.
  - `joint_boundaries_hbm` / `_tier` become `state_hbm` / `state_tier`.

`page_unit_checkpoint.py` -- `contains(h)` delegation, so the split does not
reach into `.store`.

`scheduler.py` / `llm_engine.py` -- the log line and the Prometheus tuple follow
the rename.

--- why the old refusal was right, and why it stopped being right

It read `is not None` and refused, because a K3 checkpoint was an Active Slot
that the tier spilled out of `StateSlotPool` -- a PAGE image lived in the KV
pool, which the KV connector already offloads on its own, so there was no
second boundary to agree on.

#2045 moved K3's image into the KV pool and with it the entire reason the joint
path exists. HBM now enforces `state ⊆ KV`: `_record_evicted` unindexes a
checkpoint the instant its boundary block is spent, so a checkpoint can no
longer outlive its KV there. **When LMCache hands the KV back, nothing hands
the state back unless the two are fetched together.** Leaving this refusal in
place makes the tier dead weight -- it writes 53.6 MiB per checkpoint and
nothing ever reads it -- and no other counter in the system would say so.

--- why the counters were renamed rather than left alone

`joint_boundaries_hbm/_tier` described the boundary as a whole, which was true
when the KV leg had no tier of its own to be served from. It has one now
(`_decide_load_after_alloc` decides it separately), so the old names read as if
they covered both legs. `state_hbm` / `state_tier` say what they measure.

The split also means something new. It used to say "the state pool is too small
for this concurrency". Under #2045 checkpoints share the paged pool with the KV
write stream, so a ratio dominated by `state_tier` says **the paged pool** is
too small -- checkpoints are being squeezed out by the KV they now live beside.
That diagnosis did not exist before.

`state_tier` is also the only counter here that cannot be non-zero with the CPU
tier switched off, which makes it the honest test of "did this feature run".
No passing unit test can produce it in production.

--- tests

Three in `tests/test_state_checkpoint.py`, on a harness that had to reproduce
the shape the joint path is for: publish a prompt, place ONE rung at an
interior boundary (a checkpoint under the last block is one no scan can look
up, since `can_allocate` matches over `range(n_hash_blocks - 1)`), then evict a
KV block below it so the walk stops short. A PAGE seq gets a joint boundary at
the rung with real work for the KV leg; the split reports `hbm` while the
checkpoint is resident and `tier` once it is not; no tier means no boundary.

Verified: pytest tests/ (less tests/plugin and the two msgpack/aiter collection
errors that fail on #2045 too) -> 3935 passed, zero failures. black clean;
ruff unchanged.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(offload): the delegating shell never forwarded `enqueue_state_stores`

Found by the first end-to-end run of the K3 CPU state tier on this branch: the
tier reported healthy on both sides and stored exactly nothing. 94 checkpoints
were nominated, handed to `_publish_state_stores`, refused, and had their PAGE
units released again.

--- the symptom

Server log, 94 times over a 6-minute agentic replay at concurrency 8:

    state offload: LMCacheOffloadConnectorScheduler did not carry 1 state
    store(s); releasing their units now.

with everything else looking correct:

    lmcache_offload: worker family=kimi_k3
    kimi_k3 offload: state tier up, entry=55.39 MiB rank=0, sharing the
      paged-KV CPU pool, layout=kda-paged-state-v1:layers=69:...
    state checkpoints: checkpoints_kept=112 ... offload_pins_reclaimed=0

So the worker half built the tier, the coordinator nominated checkpoints at
READY, and `take_offload_stores` pinned and handed them over -- and the
scheduler half dropped every one.

--- the cause

`LMCacheOffloadConnectorScheduler` is a shell that owns no behaviour: it holds
`self._impl = _build_scheduler(config)` and forwards each method by hand.
`7b6b58e0` added `enqueue_state_stores` to `KimiK3OffloadScheduler`
(kimi_k3/connector.py:352) but did not add the matching forwarder to the shell,
so the method existed on the implementation and was invisible from outside:

    $ git grep -c enqueue_state_stores atom/kv_transfer/offload/connector.py
    0

`Scheduler._publish_state_stores` probes with
`getattr(self.kv_connector, "enqueue_state_stores", None)`, and
`self.kv_connector` is the shell, so the probe returned None unconditionally
and the store path took its "nothing will carry these" branch on every pass.
This is not a race or a config error -- it could never have worked.

The load leg was unaffected only by luck: `enqueue_state_loads` and
`take_state_reports` were both given forwarders when they were added. Loads
still moved no bytes, because with nothing ever stored `offload.hashes` stayed
empty and the tier had nothing to vote for.

--- the fix

Add the missing forwarder, in the same shape as its `enqueue_state_loads`
neighbour so the two cannot drift again.

--- the second change, and why it is in this commit

The warning that reported the failure named a cause that was not the one:

    The tier's index should not have been installed against this connector.

That reads as a configuration mistake, and it sent the first pass of this
diagnosis at `kv_connector_hosts_state_tier` and the layout selector -- both of
which were behaving correctly. The class name printed is the *shell*, which
tells you nothing about whether `_impl` supports the call. Both warnings now
name the two possible causes and say which class is being reported.

--- verification

Static: the probe is `getattr` on a method the shell does not define, so the
refusal is unconditional; adding the forwarder is the whole of the repair, and
it is additive -- no existing path changes.

Runtime: re-running the same workload with this commit applied, the
`did not carry` warning no longer appears.

Lines conform to black's 88-column limit; black and ruff are not installed in
the benchmark image, so formatting was checked by inspection against the
surrounding style rather than by running them.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* feat(offload): counters for the store leg, and stop reporting a pool that holds none

Three observability defects, found while trying to judge whether the K3 CPU
state tier was working. Together they meant the tier could store nothing at all
and no number anywhere would say so -- which is exactly what happened in
8a5364ab, where 94 refused stores produced one warning line and four counters
that all read zero for unrelated reasons.

--- 1. the store leg had no counters at all

`StateOffloadIndex.stats()` reported `loads_attempted / completed / failed` and
`indexed`. On a K3 run the load counters stay zero until something has been
stored, so the only store-side signal was `indexed` growing -- and `indexed` is
a set size that also grows for reasons unrelated to this pass.

Added `stores_attempted` (handed to the connector, counted in
`take_state_stores`), `stores_completed` / `stores_failed` (split in
`settle_state_store` by the worker's report), and `stores_refused`.

`stores_refused` is deliberately apart from `stores_failed`, and it is the one
that matters most: failed means the worker tried and could not, refused means
nobody tried. The shell-forwarding bug fixed in 8a5364ab was a pure `refused`
condition, and with this counter it would have been one glance at the periodic
line instead of a hunt through the connector factory.

The worker's own `puts_refused` needs no separate channel: a refused `put`
already returns False and settles as `ok=False`, so it lands in `stores_failed`.

--- 2. `[Pool Pressure]` read the pool that holds no checkpoints

`BlockManager.pool_pressure()` folded in `self.state.checkpoint_fates()`. Under
#2045 a K3 checkpoint is a PAGE image the coordinator owns and the slot pool
holds none, so the line printed

    [Checkpoint Fates] kept: 0, dropped: 0, evicted: 0, orphaned: 0

on the same run whose other line said `checkpoints_kept=112`. Two counters for
one fact, disagreeing in one log, with the wrong one printed in the section a
reader goes to for pool pressure.

Now reads `self._state_checkpoint_cache`, which is already the coordinator when
there is one and the slot pool otherwise -- the same object every other
checkpoint question is asked of. `occupancy()` stays on the slot pool, which is
the thing that has slots.

--- 3. the tier's counters vanished when K3's checkpoints moved to PAGE

`StateGroupPool.checkpoint_fates` used to fold the tier's stats in under a
`state_offload_` prefix. `PagedStateCheckpointCoordinator.checkpoint_fates` was
written without that, so once #2045 moved K3's checkpoints into the coordinator
the whole `state_offload_*` family disappeared from the logs -- a silent
regression, since nothing errors when a key stops being emitted.

Restored on the coordinator. The fold is `getattr`-guarded because
`attach_offload` accepts anything answering `hashes`, and the tests attach a
double with no counters; a missing `stats` means "nothing to fold", not an
error. (`test_a_tier_that_declines_disowns_rather_than_parking` caught this on
the first draft.)

--- also added: `state_offload_store_backlog`

A gauge rather than a counter: `len(_offload_ready)`, the nominations waiting
for a slot under `OFFLOAD_MAX_PENDING_SAVES`. That queue is unbounded and
drained oldest-first, so checkpoints reaching READY faster than the tier drains
them shows up as a value that climbs and stays high, with fresh nominations
queueing behind stale ones. There is no other way to see it -- the stale
entries are skipped cheaply on the way out, so nothing else registers the
backlog.

--- how to read the new line

    state_offload_stores_refused  > 0  -- wiring fault, the connector never took it
    stores_attempted - completed - failed  -- in flight
    stores_completed vs checkpoints_kept   -- how much of what HBM keeps the tier got
    loads_failed / loads_attempted         -- the index's false-positive rate
    store_backlog climbing                 -- READY outruns the drain

Verified: pytest tests/test_page_unit_checkpoint.py tests/test_state_checkpoint.py
tests/test_lmcache_offload_connector.py tests/test_block_manager.py -> 466 passed.
Lines conform to black's 88-column limit; black and ruff are absent from the
benchmark image, so formatting was checked by inspection.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(offload): the K3 state packer imported a module that does not exist

Every state store on this branch has failed since the store path was wired.
Found by the first run where stores were actually carried (8a5364ab): 3,904
tracebacks, all identical, and not one checkpoint reached LMCache.

--- the symptom

    Traceback (most recent call last):
      File "kimi_k3/state_tier.py", line 109, in _do_store
        stored = bool(self.codec.put(h, unit_ids))
      File "kimi_k3/state_object.py", line 141, in put
        self._staged.pack(self._backend.page_unit_views(unit_ids), obj)
      File "kimi_k3/staging.py", line 175, in pack
        from atom.kv_transfer.offload.triton_kv_staging import fused_pack_chunk_major
    ModuleNotFoundError: No module named 'atom.kv_transfer.offload.triton_kv_staging'

    state offload: store of hash 10785477046942485201 failed

3,904 of them over a 50-minute concurrency-16 agentic replay -- one per store
attempt, on every rank. `stores_failed` would have counted all of them, and
`indexed` never moved.

--- cause 1: the module is one package deeper

`fused_pack_chunk_major` lives in `atom.kv_transfer.offload.dense.triton_kv_staging`.
The dense codec, which has used it all along, imports it from there
(`dense/kv_byte_codec.py:127`). The K3 stager's import dropped the `.dense`.

Nothing caught it because the import is function-local, inside `pack`, so it is
only attempted when a store actually runs -- and until 8a5364ab no store ever
reached the worker. The unit tests do not reach it either: they exercise the
codec against doubles that never call the real packer.

--- cause 2: a failed `put` leaked its allocation, once per failure

    obj = self._allocate(self.entry_bytes)     # ref_count = 1
    self._staged.pack(...)                     # raises
    self._storage.batched_put([key], [obj])    # never reached

`batched_put` discharges the reference it is handed
(`StorageManager.batched_put` ends in `memory_obj.ref_count_down()`), so the
success path correctly owes nothing -- but only if it is reached. With `pack`
throwing, the allocation was stranded at ref_count=1 and LMCache reported it
much later:

    MemoryObj at ... is being garbage collected with ref_count=1, pin_count=0.
    This indicates ref_count_down()/unpin() was not called

3,768 such warnings against 3,904 failed stores: one leak per failure. Left
alone this shrinks the shared CPU pool by one 53.6 MiB entry per failed store
until nothing can be allocated at all -- so the second-order effect of the
import bug was a slow starvation of the pool the KV leg shares.

`get` already guarded its own reference with `finally` for exactly this reason;
`put` did not. It does now, on the exception path only, matching what the DSV4
codec does (`hybrid/dsv4/codec.py:1293`).

--- why one commit

The leak is only reachable through a throwing `pack`, and the import was the
only thing throwing. Fixing the import alone would hide the leak rather than
remove it, and the next exception from that line -- an out-of-memory in the
staging buffer, say -- would strand allocations again with no traceback to
explain the pool shrinking.

--- verification

    python -c "from atom.kv_transfer.offload.dense.triton_kv_staging import
               fused_pack_chunk_major"      -> OK

    pytest tests/test_lmcache_offload_connector.py tests/test_state_checkpoint.py
           tests/test_page_unit_checkpoint.py  -> 412 passed

The import is still function-local, so this remains untested by anything that
does not run a real store. A unit test that calls `StagedTransfer.pack` against
real CUDA tensors would have caught it; that is worth adding and is not in this
commit.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(offload): the same two wiring gaps, in the places the fixes did not reach

Fix-then-sweep over `dfab0fa0` and `8a5364ab`. Both of those repaired one
instance of a bug that had two, and neither instance the fix reached is covered
by any test.

--- 1. `unpack` imports the same module that does not exist

`dfab0fa0` corrected `StagedTransfer.pack`'s import of
`fused_pack_chunk_major` (the module is one package deeper, under `.dense`).
`unpack` imports `fused_unpack_chunk_major` from the same wrong path, eleven
lines further down, and was left alone.

It has never fired because no state load has ever run -- with nothing stored,
`offload.hashes` stayed empty and the tier had nothing to vote for. The first
successful store makes the first load reachable, and it would have raised
`ModuleNotFoundError` on that load, on every rank.

Both kernels live in `atom.kv_transfer.offload.dense.triton_kv_staging`;
verified by importing both.

--- 2. `chunk_size` is not on the delegating shell either

`8a5364ab` added the missing `enqueue_state_stores` forwarder. Sweeping every
member `Scheduler` reads off `self.kv_connector` against what
`LMCacheOffloadConnectorScheduler` defines turns up one more: `chunk_size`.

    scheduler.py:2284   seq.offload_kv_chunk_tokens = int(
                            getattr(self.kv_connector, "chunk_size", 0) or 0)

`self.kv_connector` is the shell, the shell has no `chunk_size`, so this has
always stamped 0. It is latent rather than live: `_joint_kv_boundary` reads
`self._joint_chunk_tokens` first and only falls back to this, and that primary
comes from `BlockManager.__init__` reading the LMCache config directly. But the
fallback exists precisely for the case where that read failed -- and it was
dead, so a build that could not read the config at startup would refuse every
joint boundary with `no_chunk_size` and never say why.

Forwarded as a property, with the reason in its docstring.

--- 3. the class of bug, closed rather than fixed twice

Both are the same shape: the shell forwards by hand, so a member added to an
implementation is invisible from outside, and every reader is a
`getattr(..., default)` that does not raise -- it takes the default forever.
Two instances in two days says the next one is a matter of time.

`test_every_member_the_scheduler_reads_is_reachable_through_the_shell` scans
`scheduler.py`'s source for `getattr(self.kv_connector, "x")`,
`self.kv_connector.x`, and `_connector_flag("x")`, and asserts every name is
reachable on the shell class. It reads the source rather than a hand-kept list,
because a hand-kept list is the same failure one level up. The scan asserts it
found something, so a regex that stops matching fails loudly instead of passing
vacuously.

Falsified: renaming the new `chunk_size` property makes it fail and name the
missing member.

Verified: pytest tests/ (less tests/plugin and the two msgpack/aiter collection
errors) -> 3936 passed, zero failures. black clean; ruff 36, unchanged.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(state-cache): the CPU tier gathered a checkpoint's padding as if it were image

Fourth and last blocker in the chain that kept the K3 CPU tier from storing
anything. With the previous three fixed, a 25-minute concurrency-16 replay
still ended at

    state_offload_stores_attempted=462
    state_offload_stores_completed=0
    state_offload_stores_failed=450
    state_offload_indexed=0

and 3,696 identical tracebacks:

    File "kimi_k3/state_object.py", line 149, in put
      self._staged.pack(self._backend.page_unit_views(unit_ids), obj)
    File "kimi_k3/staging.py", line 182, in pack
      dst_tensor = self.memory_tensor(dst, nbytes)
    ValueError: ATOM LMCache connector: MemoryObj tensor is too small for
                59867136 bytes; got 58079232

--- the geometry

From the run's own startup line:

    unit_bytes=2138112  image_bytes=58079232  units_per_checkpoint=28

    ceil(58079232 / 2138112) = ceil(27.163) = 28
    28 x 2138112            = 59867136      <- what page_unit_views gathered
    58079232                                <- what put() allocated
    difference               = 1787904      <- padding in the 28th unit

An image occupies whole units, and the last one is 83.6% padding here. Nothing
owns those bytes.

--- why only the CPU side hit it

The HBM path never had the problem because it already passes the image size
down: `_checkpoint_copy_plan` calls

    plan_segmented_copy(src_sizes, self._page_unit_stream_sizes(units),
                        spec.image_bytes)

so the D2D copy writes exactly `image_bytes` into the unit stream and leaves
the tail untouched. The image is, by construction, the LEADING `image_bytes`
of that stream. `page_unit_views` was written as the tensor-view counterpart of
the same addressing but did not carry the third argument across, so it named
whole units to a consumer that sizes its destination at `entry_bytes` -- which
for K3 is exactly `image_bytes`.

The connector's `image_bytes != entry_bytes` guard cannot catch this: both are
58079232 and agree. The quantity that disagrees is a third one,
`sum(page_unit_views)`, which no check compared against either.

--- the fix

Trim the gathered stream to `image_bytes`, dropping whole views past the
budget and slicing the one that straddles it.

The slice is byte-exact rather than view-aligned because the image does not end
on a view boundary: 58079232 leaves 350208 B of the last unit in use, which is
20.96 of its rows. That view is reinterpreted as `uint8` and sliced; the packer
sizes segments as `numel * element_size`, so a uint8 segment costs it nothing.

The load side needs no counterpart: it scatters into `state_entry_views`, which
sums to `entry_bytes` exactly, so both ends of the round trip are now the same
58079232 bytes.

A budget left unspent raises rather than storing a short blob -- that would be
a real geometry disagreement, and a truncated image read back as valid is the
one outcome worth crashing over.

--- two notes on shape

Inline rather than a helper method: the existing tests drive `page_unit_views`
with a `types.SimpleNamespace` stand-in for `self`, and a second attribute would
have to be stubbed in every one of them. For the same reason the spec is read
with `getattr(spec, "image_bytes", 0)` -- a fork build carries no spec at all,
and there the whole-unit stream is the right answer.

--- verification

    pytest tests/test_kda_checkpoint_slot_copy.py
           tests/test_lmcache_offload_connector.py
           tests/test_state_checkpoint.py
           tests/test_page_unit_checkpoint.py     -> 441 passed

Also confirmed from the same run that the previous commit's leak guard works:
`MemoryObj ... garbage collected with ref_count=1` went from 3,768 occurrences
to 0, even though every store still failed.

None of the four bugs in this chain was reachable by the unit tests, because
all of them live past the first line that only a real store executes. A test
that packs real CUDA tensors through `StagedTransfer` would have caught three
of the four; it is still worth adding.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(offload): the state key treated an unsigned block hash as signed

About half of every K3 state store failed. Measured over a concurrency-16
agentic replay with the four earlier blockers fixed:

    state_offload_stores_attempted=46
    state_offload_stores_completed=19
    state_offload_stores_failed=27      <- 59%

with 296 identical tracebacks:

    File "kimi_k3/state_object.py", line 153, in put
      self._storage.batched_put([self.key(h)], [obj])
                                 ^^^^^^^^^^^
    File "kimi_k3/state_object.py", line 107, in key
      digest.update(int(h).to_bytes(8, "little", signed=True))
    OverflowError: int too big to convert

    state offload: store of hash 12983297547834104159 failed

--- cause 1: the hash is unsigned and the key said otherwise

12983297547834104159 > 2**63-1, so `to_bytes(8, signed=True)` cannot hold it.

An ATOM block hash is `xxhash.xxh64().intdigest()`, which spans the whole
0..2**64-1, and `BlockManager.compute_hash` chains one into the next with
`prefix.to_bytes(8, "little")` -- unsigned, no keyword. The state key was the
only place in the chain that read the same value as signed, so it rejected
every hash with the top bit set. That is half the key space, and the observed
59% is that with sampling noise.

Nothing downstream noticed because the failure is inside a `try` in
`_do_store`, which counts a failed store and moves on. Before the counters
added in a8045614 it produced no number at all.

--- cause 2: the raise stranded a MemoryObj, in a window the last guard missed

`dfab0fa0` wrapped `pack` so a throwing gather would discharge the allocation.
`key(h)` is evaluated in `batched_put`'s argument list, which is after the
allocation and before `batched_put` can take ownership -- outside that guard.
So each of the 296 overflows also leaked a 53.6 MiB entry, and the LMCache
warning reappeared next to the traceback:

    MemoryObj at 136839168 is being garbage collected with ref_count=1

Fixed by computing the key *before* allocating. `key` is pure, so hoisting it
removes the window rather than guarding it, and the `except` now covers
`batched_put` itself as well -- matching what the DSV4 codec does.

--- what this unblocks

The same `key` is used by `get`, so roughly half of all loads would have missed
in a way indistinguishable from an LRU eviction -- `fail_load` would then
`forget` a hash whose bytes were never stored under a name it could find, and
the index would decay for a reason nothing reported.

Verified: pytest over the state and offload suites -> 442 passed. The overflow
itself is confirmed directly:

    >>> (12983297547834104159).to_bytes(8, "little", signed=True)
    OverflowError: int too big to convert
    >>> (12983297547834104159).to_bytes(8, "little", signed=False).hex()
    '5f19dacf54f02db4'

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* test(state-cache): pin the image trim, which nothing reached

`961f80d6` fixed `page_unit_views` gathering a checkpoint's padding, and the
suite it ran stayed green before and after — because no test reaches the trim
at all.

The addressing tests build their stub spec as

    checkpoint_spec=SimpleNamespace(page_unit_bytes=...)

with no `image_bytes`, so `budget` is 0 and every one of them takes the
`return views` early path. The trim shipped untested, which is how a bug that
failed 3,696 stores in a row got there in the first place.

--- what is pinned

Seven cases in `TestPageUnitViewsStopAtTheImage`, on the harness the addressing
tests already use (`build` gains optional `image_bytes` and `dtype`; both stay
off by default so the existing tests keep exercising whole units):

  * the gathered stream is exactly `image_bytes` — the invariant `put`'s
    allocation depends on, and the one that was broken;
  * a whole view past the budget is dropped, not truncated to zero length;
  * an image that lands on a view boundary is not sliced at all;
  * **the kept bytes are the leading bytes and nothing else** — trimming
    truncates, it does not reorder. The blob is read back by scattering into
    the slot in the same order, so a byte that moved here would land in the
    wrong layer there with nothing raised;
  * a multi-byte dtype is sliced by BYTES, not elements — the straddling view
    is reinterpreted as `uint8` first, and on bf16 an element-wise slice would
    keep twice what it should. The production cache is 1 byte per element, so
    only a test with a wider dtype can hold this;
  * a spec with no image size keeps whole units (the fork build, which has no
    spec to trim against);
  * units that cannot cover the image raise rather than storing a short blob.

--- verification

`tests/test_kda_checkpoint_slot_copy.py` skips wholesale on a non-GPU runner
(the module imports aiter at load), so the seven cases were additionally run by
extracting `page_unit_views` and its three neighbours out of the source and
driving them directly. All seven hold, including the bf16 one.

pytest tests/ (less tests/plugin and the two msgpack/aiter collection errors)
-> 3936 passed, zero failures. black clean; ruff 36, unchanged.

--- one note on `put`'s exception shape, checked not changed

`2c5c3457` moved `batched_put` inside the `try` whose `except` calls
`ref_count_down`. That is safe only if `batched_put` takes ownership just on
success — and it is the pattern the DSV4 codec already uses
(`hybrid/dsv4/codec.py:1284`, "StorageManager owns the MemoryObj after a
successful batched_put"). Consistent with the repo rather than a new
assumption, so left alone.

The window between `_allocate` and `batched_put` is now fully covered:
`page_unit_views` can raise (the geometry check above) and is inside the try;
`key(h)` is computed before the allocation.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* style: clear the ruff findings CI reports on this PR

CI's reviewdog run on #2053 flagged five findings, all introduced by this
branch. Every one is real and still present.

  atom/kv_transfer/offload/hybrid/dsv4/connector.py:27  I001
  atom/kv_transfer/offload/hybrid/kimi_k3/connector.py:11   I001
  atom/kv_transfer/offload/hybrid/kimi_k3/connector.py:425  SIM102
  tests/test_scheduler.py:1466  I001
  tests/test_scheduler.py:1498  I001

plus one more the bot has not reported yet because reviewdog filters to diff
context and that line has not changed since:

  atom/model_engine/page_unit_checkpoint.py:6  I001

--- why the local checks said clean

`ruff check` with no `--select` runs the default rule set (E4/E7/E9/F), and
neither `I` (isort) nor `SIM` is in it. There is no `[tool.ruff]` in
`pyproject.toml` and no `ruff.toml`, so every local run in this branch's
history measured the default set and reported "36, unchanged" against the base.

That number was true and I reported it as if it meant CI would be clean, which
it does not. Checked from now on with `--select I,SIM` as well.

--- scope

`--fix` over the whole tree r…
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant