Skip to content

[Kimi-K3][LMCache] LMCache offload for Kimi-K3 - #2053

Merged
valarLip merged 155 commits into
mainfrom
zejun/lmcache_with_2045
Sep 3, 2026
Merged

valarLip merged 155 commits into
mainfrom
zejun/lmcache_with_2045

Conversation

@zejunchen-zejun

@zejunchen-zejun zejunchen-zejun commented Aug 27, 2026 •

Copy link
Copy Markdown
Collaborator

[Kimi-K3][LMCache] LMCache offload for Kimi-K3

Adds a CPU offload tier for K3's KDA recurrent state, alongside the KV tier
lmcache_offload already provides. A prefix whose KV came back from LMCache is
useless without the recurrent state that goes with it, so the two are fetched
together or not at all.

Builds on #2045, which turned a K3 checkpoint from a whole Active Slot into a
PAGE image living in ordinary KV blocks. That is what makes this tier cheap:
the source of a save is a set of reserved blocks that nobody can take away,
so nothing has to be rescued anywhere before the D2H.


Design

A third layout, inheriting dense rather than written beside it.
K3's paged KV is ordinary dense MLA — block-addressed, sliceable, moved by
chunk — so KimiK3OffloadConnector / KimiK3OffloadScheduler extend
DenseOffloadConnector / DenseOffloadScheduler and add one extra leg for the
69 layers of KDA state. (DSV4 is written in parallel instead because its PAGE
leg carries an indexer and has a different shape.) Every override wraps a dense
call rather than replacing it; the KV leg's behaviour is unchanged.

Layout selection is config-only (select_offload_layout), because the
scheduler process has no model.

Both legs save when the bytes land in HBM, into one pool.

KV state
trigger a prefill chunk completes a checkpoint reaches READY
source the request's blocks, held by should_defer_free the checkpoint's PAGE units, held by a pin
pool one LMCache engine, one allocator, one LRU same pool, same LRU

No second engine and no OFFLOAD_STATE_CPU_SIZE. A request writes its KV
chunks and its one state object inside the same prefill window, so both enter
the LRU together and cool at the same rate — state is retired alongside its own
KV, which is what we want, since a boundary whose KV is gone is worthless. Two
pools would buy two eviction policies that must drift, while a joint boundary
needs both legs to survive together.

Loads pick one boundary for both legs. The ceiling comes from KV, the
landing point from state; the KV leg fetches into blocks the request already
holds, and the state leg's H2D writes the request's Active Slot directly — no
units are reserved for it.

Everything is asynchronous. submit() returns immediately; the Triton
gather runs on its own stream, the D2H on another, and the engine advances on
the worker's report. Nothing waits on the compute stream.


Why the boundary has to be joint

KDA state at position B is the compressed history of exactly [0, B). Raise
the KV-loaded length past B and the linear layers never see the tokens in
between: wrong output, no exception. So the two legs cannot be decided
separately.


Example

A request whose prompt is 12 hash blocks, A B C D E F G H I J K L
(--block-size 128, LMCACHE_CHUNK_SIZE 256). Block C has been evicted from
HBM; LMCache still holds the KV for A..K and the state checkpoint filed at
H.

HBM   KV:  A B  ✗C evicted✗
HBM ckpt:  (gone with C's neighbourhood)
CPU   KV:  A B C D E F G H I J K
CPU ckpt:                  H
  1. can_allocate walks the HBM prefix and stops at the first miss, C.
    Two blocks matched.
  2. No checkpoint at or below B, so the request may claim 0 tokens as
    cached.
  3. _joint_kv_boundary:
    • ceiling from KV — LMCache covers 11 blocks, floored to the chunk grid
      gives 10; capped by n_hash_blocks - 1 (prefill must forward one block to
      produce logits) → 10;
    • the chained hash is a function of the prompt alone, so the chain is
      continued past where HBM stopped, which is what makes a boundary that
      exists only in LMCache addressable at all;
    • the rightmost reachable checkpoint on that chain is H → B = 1024;
    • KV leg target = the chunk covering B = 1024;
    • block table may claim what the walk matched, floored to the grid = 256.
  4. allocate points the block table at A B — free, since those blocks sit
    below where the forward will start — and leaves num_cached_tokens at 0.
  5. State: HBM misses, the tier votes, a load is issued for H.
  6. KV: transfer [256, 1024) = C D E F G H. The already-resident A B is
    not re-fetched; doing so would move bytes the GPU holds and leave an
    unindexed second copy of them in HBM.
  7. _claim_after_load → 1024, the state boundary, not the transfer end.
  8. Both legs report; the request unparks with num_cached_tokens = 1024 and
    the forward computes I J K L.

Three positions, all different, and each is needed: 0 is what may be called
cached, 256 is what the block table may point at, 1024 is where both
legs are aimed. Claiming past the first is safe only because a joint boundary
exists — those blocks are below the forward's start, so nobody writes them.
Transferring past B costs one chunk the forward immediately rewrites;
claiming past B is silent wrong output.


Notes for reviewers

  • state_pool.py and sub_pool_spec.py have an empty diff against main.
    The tier does not reach into the slot pool at all.
  • The state key folds in layout_id. CacheEngineKey has no field saying what
    an entry is, and KV and state keys come from different hash functions into one
    integer space — now sharing one pool. Folding the layout in makes the two
    spaces disjoint by construction, and separately stops a build that changed the
    state geometry from reading another's image back as valid.
  • A checkpoint is nominated at READY but pinned only when handed to the
    worker, bounded by OFFLOAD_MAX_PENDING_SAVES. Pinning at READY would make
    every checkpoint un-evictable for a window that spans several scheduler
    passes, breaking the admission rule that a READY unpinned checkpoint counts
    as space available to live KV.
  • Both the KV and the state save gather through the same Triton kernel
    (fused_pack_chunk_major), unmodified.
  • Removed env vars: OFFLOAD_STATE, OFFLOAD_KV_FOR_HYBRID,
    OFFLOAD_SAVE_MAX_INFLIGHT, OFFLOAD_SAVE_STALL_S,
    OFFLOAD_STATE_MIN_LOAD_TOKENS, OFFLOAD_STATE_CPU_SIZE,
    OFFLOAD_STATE_STAGING_GROUPS. A resumable prefix needs both legs, so the
    connector being configured is the only switch.

Verification

CON16
image

CON32

@github-actions

Copy link
Copy Markdown
Contributor

🏷️ CI Guide

Runs automatically on every eligible PR before approval:

  • ✅ Pre Checkin: Black, Ruff, catalog schema validation, non-GPU unit tests

Heavy model tests:

  • ✅ Run after the PR is approved and Pre Checkin passes
  • ✅ Run immediately when an approval review is submitted
  • ✅ Can be requested before approval with labels
Label Tests
ci:full Run all heavy PR model tests: native ATOM, vLLM, and SGLang
ci:atom Run native ATOM model accuracy tests
ci:vllm Run ATOM vLLM OOT model accuracy tests
ci:sglang Run ATOM SGLang model accuracy tests

Heavy jobs are skipped when the PR is not approved and no matching ci:* label is present.
Add labels via the sidebar or gh pr edit 2053 --add-label <label>

zejunchen-zejun added a commit that referenced this pull request Aug 28, 2026
CI's reviewdog run on #2053 flagged five findings, all introduced by this
branch. Every one is real and still present.

  atom/kv_transfer/offload/hybrid/dsv4/connector.py:27  I001
  atom/kv_transfer/offload/hybrid/kimi_k3/connector.py:11   I001
  atom/kv_transfer/offload/hybrid/kimi_k3/connector.py:425  SIM102
  tests/test_scheduler.py:1466  I001
  tests/test_scheduler.py:1498  I001

plus one more the bot has not reported yet because reviewdog filters to diff
context and that line has not changed since:

  atom/model_engine/page_unit_checkpoint.py:6  I001

--- why the local checks said clean

`ruff check` with no `--select` runs the default rule set (E4/E7/E9/F), and
neither `I` (isort) nor `SIM` is in it. There is no `[tool.ruff]` in
`pyproject.toml` and no `ruff.toml`, so every local run in this branch's
history measured the default set and reported "36, unchanged" against the base.

That number was true and I reported it as if it meant CI would be clean, which
it does not. Checked from now on with `--select I,SIM` as well.

--- scope

`--fix` over the whole tree repairs 100 findings in files this branch never
touched; that was reverted. Fixed here are only the findings this branch
introduced, established per file by running the same rules against the same
file on the base:

    for f in $(git diff --name-only <base> HEAD -- '*.py'); do
        compare ruff --select I,SIM on <base>:$f against $f
    done

Six files come back with more findings than their base; the six above are them.
After this commit that comparison returns nothing. The 14 findings that remain
across files this branch touches are all pre-existing on untouched lines, which
is also why reviewdog does not report them.

SIM102 is `should_defer_free`'s stalled-save escape: two nested `if`s collapsed
into one condition, same short-circuit order, no behaviour change.

Verified: pytest tests/ (less tests/plugin and the two msgpack/aiter collection
errors) -> 3936 passed, zero failures. black clean; ruff default set 36,
unchanged; ruff `--select I,SIM` shows zero findings introduced by this branch.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@zejunchen-zejun
zejunchen-zejun force-pushed the zejun/lmcache_with_2045 branch from 1fe7ce5 to 76a10fa Compare August 28, 2026 07:53
zejunchen-zejun added a commit that referenced this pull request Aug 28, 2026
CI's reviewdog run on #2053 flagged five findings, all introduced by this
branch. Every one is real and still present.

  atom/kv_transfer/offload/hybrid/dsv4/connector.py:27  I001
  atom/kv_transfer/offload/hybrid/kimi_k3/connector.py:11   I001
  atom/kv_transfer/offload/hybrid/kimi_k3/connector.py:425  SIM102
  tests/test_scheduler.py:1466  I001
  tests/test_scheduler.py:1498  I001

plus one more the bot has not reported yet because reviewdog filters to diff
context and that line has not changed since:

  atom/model_engine/page_unit_checkpoint.py:6  I001

--- why the local checks said clean

`ruff check` with no `--select` runs the default rule set (E4/E7/E9/F), and
neither `I` (isort) nor `SIM` is in it. There is no `[tool.ruff]` in
`pyproject.toml` and no `ruff.toml`, so every local run in this branch's
history measured the default set and reported "36, unchanged" against the base.

That number was true and I reported it as if it meant CI would be clean, which
it does not. Checked from now on with `--select I,SIM` as well.

--- scope

`--fix` over the whole tree repairs 100 findings in files this branch never
touched; that was reverted. Fixed here are only the findings this branch
introduced, established per file by running the same rules against the same
file on the base:

    for f in $(git diff --name-only <base> HEAD -- '*.py'); do
        compare ruff --select I,SIM on <base>:$f against $f
    done

Six files come back with more findings than their base; the six above are them.
After this commit that comparison returns nothing. The 14 findings that remain
across files this branch touches are all pre-existing on untouched lines, which
is also why reviewdog does not report them.

SIM102 is `should_defer_free`'s stalled-save escape: two nested `if`s collapsed
into one condition, same short-circuit order, no behaviour change.

Verified: pytest tests/ (less tests/plugin and the two msgpack/aiter collection
errors) -> 3936 passed, zero failures. black clean; ruff default set 36,
unchanged; ruff `--select I,SIM` shows zero findings introduced by this branch.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@zejunchen-zejun
zejunchen-zejun marked this pull request as ready for review August 28, 2026 09:14
Copilot AI lite review requested due to automatic review settings August 28, 2026 09:14

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds Kimi-K3 support to the lmcache_offload connector by introducing a CPU offload tier for K3’s per-request recurrent (KDA) state and making KV+state resumes joint (both legs must agree on a single safe boundary). This extends the existing dense offload path with K3-specific state spill/load plumbing, new engine-side bookkeeping, and substantial test coverage.

Changes:

  • Introduce kimi_k3 offload layout: worker-side state tier (store/load), scheduler-side joint-boundary policy, and shared-pool keying with build-safety (layout_id) separation.
  • Add engine-side StateOffloadIndex + BlockManager/Scheduler wiring to enqueue state loads/stores, settle reports, and expose new cache stats counters.
  • Register per-request state tensors via KVCacheTensor.per_request_state and exclude them from block-addressed KV movers; add backend hooks (state_entry_views, page_unit_views) to name state bytes for packing.

Reviewed changes

Copilot reviewed 39 out of 39 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
tests/test_state_checkpoint.py Adds extensive tests for tier load behavior and joint boundary semantics.
tests/test_scheduler.py Tests offload-resume accounting and stalled deferred-save reclamation behavior.
tests/test_page_unit_checkpoint.py Tests PAGE checkpoint nomination/pinning/offload vote behavior.
tests/test_multi_connector.py Ensures multi-connector routes state loads and exposes _state_tier.
tests/test_lmcache_offload_v4_page_slot.py Updates tests to use shared max_pending_saves helper.
tests/test_lmcache_offload_connector.py Adds K3-specific scheduler contract tests (joint boundary, state channels, shell forwarding).
tests/test_kda_checkpoint_slot_copy.py Adds coverage for page_unit_views correctness and image trimming.
tests/test_block_manager.py Tests widened claim behavior + state-tier index installation gating.
docs/environment_variables.md Documents offload connector behavior and layout selection; introduces a new section.
atom/plugin/sglang/attention_backend/attention_gdn.py Marks recurrent per-request state tensors as per_request_state=True.
atom/plugin/rtpllm/utils/forward_context.py Same per_request_state marking for RTP plugin path.
atom/model_ops/attentions/kimi_mla_gdn_attn.py Implements page_unit_views + marks per-request state tensors.
atom/model_ops/attentions/gdn_attn.py Adds state_entry_views and marks recurrent state tensors as per-request state.
atom/model_ops/attentions/deepseek_v4_attn.py Adds state_entry_views and clarifies state-copy docstring/comments.
atom/model_ops/attentions/backends.py Defines new state_entry_views abstract hook for backends owning per-request state.
atom/model_engine/state_offload.py New engine-side state offload index + connector-host gating helper.
atom/model_engine/sequence.py Adds fields to track state load intent and joint-boundary extents.
atom/model_engine/page_unit_checkpoint.py Adds offload nomination/pinning lifecycle + vote through CPU-tier reachability.
atom/model_engine/model_runner.py Publishes attention backend to transfer tensors so tier can access state_entry_views.
atom/model_engine/llm_engine.py Exposes new joint/tier counters in cache statistics.
atom/model_engine/engine_core.py Adds worker termination hardening + periodic stalled-save reconciliation + dispatch of state load/store publishing.
atom/model_engine/block_manager.py Implements joint boundary selection + widened claim + state-tier load/store APIs and funnel aggregation changes.
atom/kv_transfer/offload/metadata.py Adds state_loads / state_stores to offload metadata and work-field declaration.
atom/kv_transfer/offload/hybrid/kimi_k3/state_tier.py New worker-side tier executor for state store/load plus joint-park helper.
atom/kv_transfer/offload/hybrid/kimi_k3/state_object.py New codec for state objects (stable keying + packing/unpacking).
atom/kv_transfer/offload/hybrid/kimi_k3/staging.py New bounded staging helper for whole-entry pack/unpack.
atom/kv_transfer/offload/hybrid/kimi_k3/connector.py New K3-specific connector/scheduler implementation (state tier + joint KV clamp + stall guards).
atom/kv_transfer/offload/hybrid/kimi_k3/init.py New package marker for K3 offload layout.
atom/kv_transfer/offload/hybrid/dsv4/connector.py Uses shared max_pending_saves + routes claim computation through _claim_after_load.
atom/kv_transfer/offload/dense/kv_byte_codec.py Skips per_request_state tensors in dense KV packing.
atom/kv_transfer/offload/dense/connector.py Adds _may_emit_save seam + uses _claim_after_load for post-load claim.
atom/kv_transfer/offload/connector.py Adds kimi_k3 variant selection and strengthens shell docs/forwarding.
atom/kv_transfer/offload/config.py Adds kimi_k3 alias and selection rule based on HF model_type.
atom/kv_transfer/offload/atom_lmcache_staging.py Makes _env_flag robust to whitespace/empty values.
atom/kv_transfer/offload/_offload_common.py Adds _claim_after_load seam + factors max_pending_saves helper.
atom/kv_transfer/disaggregation/types.py Adds ConnectorMetadata.WORK_FIELDS + has_work() and uses it in work detection.
atom/kv_transfer/disaggregation/multi/multi_connector.py Delegates has_work() to subs, re-exports _state_tier, and routes state loads.
atom/config.py Adds KVCacheTensor.per_request_state flag to separate per-request state from block-addressed KV.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment on lines +50 to +56
def __init__(self, config) -> None:
super().__init__(config)
self._state_tier = None
# Inert until a request has both legs, which only a joint boundary
# produces; costs one dict lookup per report otherwise.
self._joint_park = _JointPark()

Comment on lines +249 to +253
def get_finished(self):
out = super().get_finished()
if self._state_tier is None:
return out
indexed, index_failed = self._state_tier.take_store_reports()
Comment on lines +232 to +233
| Variable | Type | Default | Description |
|----------|------|---------|-------------|
zejunchen-zejun and others added 17 commits August 28, 2026 18:46
Split out of PR #1960 so they can land on their own. None of them is about
state offload, and none needs #2045; blocking them behind either is pure loss,
and pulling them out narrows what is left to review by a few hundred lines.

--- a worker that outlives the join keeps the node

`EngineCore.exit` joined each worker with a 5s timeout and did nothing after
it. A worker that outlives that join keeps its VRAM slice and its all-reduce
IPC handles, and `multiprocessing`'s atexit handler then joins the same
process again with NO timeout -- so an engine already on its way out parks in
`_exit_function -> join` forever while /metrics keeps answering 200. Seen
twice on the k3-dev line; only a manual kill recovered the node.
`enable_orphan_reaping` cannot help: `PR_SET_PDEATHSIG` fires when the parent
*dies*, and this parent never does. Now escalates terminate -> kill.

--- a stalled offload save hung the engine with every GPU idle

A deferred save's blocks are freed only on `finished_saving`, but LMCache's
pin monitor force-unpins a stalled transfer after `pin_timeout_sec` WITHOUT
emitting that report. `has_pending_kv_work()` then never clears and the engine
busy-loops with every GPU idle -- a hard hang under a tight pool, a slow block
leak under a large one.

`Scheduler._reconcile_stalled_deferred_saves`, wired into the KV progress
poll, reclaims a save deferred past an abandon window and self-throttles to
5s so a 1ms poll costs nothing.

The window reads LMCache's own `LMCACHE_EC_PIN_TIMEOUT_SEC` and adds 30s
rather than taking an ATOM knob of its own, because that ordering IS the
safety argument: two independent env vars would let ours sit below the timeout
it must exceed, and nothing would say so. Unset gives 330s. Non-positive
disables reclamation and restores the wait-forever behaviour.

What makes reclaiming safe is not the clock -- our clock starts when the free
is deferred, LMCache's when it pins, and nothing aligns them. It is that
`OffloadWorkerMixin._guard` reports `finished_saving` on both the success and
the exception path, so a report is lost only when `store()` neither returns
nor raises. Two cases follow and they are exhaustive: the parked save is not
copying (LMCache force-unpinned its source and stopped reading), and a save
queued behind a parked one never reached `store()` at all.

`finished_sending`'s `assert seq is not None` becomes a `continue`: after a
reclaim, a late completion report has nothing left to free.

--- a state-only step under `multi` parked its request forever

`connector_metadata_has_work` decides whether a step's metadata reaches the
worker, and it decided from a hardcoded field list shared by three connector
families. `MultiConnectorMetadata` holds none of the work itself -- everything
is in `metas` -- so it mirrored the subs' fields as aggregating properties,
and missed one. Every step whose only work was in an unmirrored field was
discarded, and the request parked on it waited for a report nobody was asked
to produce.

Fixed by asking the metadata instead of inspecting it: each class declares its
own `WORK_FIELDS` next to the fields it defines, `has_work()` accumulates down
the hierarchy, and `MultiConnectorMetadata` delegates to its subs rather than
mirroring them -- so a field added to a sub can no longer be missed here. The
old union list stays only as the fallback for duck-typed doubles, marked as
not the place to register anything new.

--- two behaviour-neutral seams and one marker

`OffloadSchedulerMixin._claim_after_load` returns `max(hbm, lmc)`, which is
what all four inlined call sites did. A seam rather than four `max`es because
a layout whose claim and transfer aim at different boundaries has to override
it, and getting that wrong is silent: the request starts further along than
its state supports and the output is wrong with no exception.

`DenseOffloadScheduler._may_emit_save` returns True. It matters for a layout
whose `should_defer_free` pins a finished request's blocks until its save
drains -- an unbounded queue then lets a slow backend hold an unbounded slice
of the pool. `max_pending_saves` moves to `_offload_common` so every layout
reads the same `OFFLOAD_MAX_PENDING_SAVES`.

`KVCacheTensor.per_request_state` marks a hybrid's slot-addressed recurrent
state. It must be registered in `kv_cache_data` -- the linear-attention
forward reads it from there -- but no block-addressed mover may touch it, and
`DenseKVByteCodec` now skips it: including it either fails the divisibility
check or, if the slot count happens to divide `num_blocks`, inflates
`bytes_per_block` past what `block_regions` describes.

Verified on top of #2045 (`4df58434`): pytest tests/ (less tests/plugin and
the two msgpack/aiter collection errors that fail on #2045 too) -> 3871
passed, 326 skipped, 3 xfailed, zero failures. black clean; ruff 36, the same
count as #2045.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Phase 1 of remounting PR #1960 on #2045. Deliberately a RENAME, not a redesign:
the tier still hangs off `StateSlotPool` and still spills on eviction through
the staging ring. Phase 2 deletes the ring; Phase 3 moves the mount point to
`PageUnitCheckpointStore`. Keeping those apart is what lets a reviewer tell
rebase noise from design decisions.

Squashed rather than replayed for the reason #2045 gives for its own squash:
other's resolutions. #1960 keeps the original history.

--- what #2045 renamed underneath the branch

  StateGroupPool                -> StateSlotPool
  lookup_group(h)               -> lookup(h)
  seq.per_req_cache_group       -> seq.state_slots (list) + seq.state_slot
  num_per_req_cache_groups      -> num_state_slots + state_slots_per_req
  pop() / release()             -> pop_many(width) / release_many()
  has_free()                    -> has_free(n)
  _attach_state_group()         -> _attach_state_slots()

The semantic change hides in that table: **the "group" concept is gone.** Slots
are allocated one at a time, `pop_many` states outright that the set is not
adjacent and nothing may assume it is, and a checkpoint occupies exactly ONE
slot -- speculation scratch is not state anybody resumes into. Every
`group * span` address computation in #1960 lost its basis, so the tier's
`_spill` / `_resumable_from` / `has_pending_spill` / `take_spill_copies` are
re-expressed over slots, and `(group, staging)` becomes `(slot, staging)`.

--- the three places the two PRs actually collide

`_attach_state_slots` now returns bool. #2045 rewrote its body for per-slot
allocation; #1960 needs it to be able to say "the state behind this boundary is
not really here", because with the tier voting a hash can be accepted by
`_resumable_from` and still not be in HBM. The two compose: #2045's width logic
runs, and only the committed slot (element 0) is the load's destination, so
only it takes `pop(spill=False)` -- the rollback scratch is this request's own
and has nothing to forgo.

`allocate` keeps #1960's two extents (`num_cached_blocks` vs `claim_blocks`)
on top of #2045's per-slot attach, and the fresh-block loop starts from
`len(seq.block_table)` rather than `num_cached_blocks`, since the widened claim
already put blocks there.

`CacheStats` gains #1960's `total_offload_tokens` alongside #2045's
`total_reusable_tokens`. They are not the same series and neither subsumes the
other: `reusable` is the honest denominator for the HBM walk, while tier tokens
sit ABOVE that walk by design, so folding them in would make `cached > wanted`.
`update()` therefore takes five required arguments and one defaulted.

`deallocate` keeps #2045's midstep cancellation and `release_many`, and #1960's
in-flight-load hold: the committed slot is parked in `_orphan_load_slots` until
`settle_state_load`, because a worker is writing it; the scratch goes back
immediately.

`checkpoint_funnel` reports through `state_checkpoint_fates()` (a sum across
`state_caches`) rather than the single `_state_checkpoint_cache` call. Today
`state_caches` is a 1-tuple and the two are equivalent -- kept so Phase 3's
tier counters, which arrive from a second class, are not silently dropped.

--- functionality is NOT verified by this commit

Three gates all take the else branch for K3 under #2045, so the tier is inert:

  1. `isinstance(cache, StateSlotPool)` -- K3's `state_caches` is
     `(PagedStateCheckpointCoordinator,)`, so `cache.offload` is never set.
  2. `_joint_kv_boundary` returns `_no_joint("not_hybrid")` whenever
     `paged_state_checkpoints is not None`, which for K3 is always.
  3. `_attach_state_slots` returns from its PAGE branch before reaching
     `_tier_can_serve`.

That is deliberate and is what Phase 3/5 undo. Do not read the green suite
below as evidence the feature works -- the exit criterion for the tier is a
nonzero `state_tier` counter in the logs, which no unit test can produce.

Verified: pytest tests/ (less tests/plugin and the two msgpack/aiter collection
errors that fail on #2045 too) -> 3915 passed, 326 skipped, 3 xfailed, zero
failures. black clean; ruff 36, the same count as #2045.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…to rescue

Phase 2. About 1000 lines go, and none of them for tidiness: the ring existed
to solve exactly one problem, and #2045 removed the problem.

--- why the ring existed at all

A checkpoint used to live in a state slot, and `StateSlotPool.pop` handed that
slot to the next request the instant it was evicted. The bytes therefore had to
be rescued somewhere before they were overwritten, which is what the ring was:
K entries allocated inside the state arena, never leased to a request, and
addressed as `num_slots + staging` so `state_entry_views` could reach them with
no second scheme. Everything else -- `staging_entries`, `admission_entries`, the
drain protocol, the starvation detector, the reclaimer, `pop(spill=)`,
`spills_forgone`, the spill-before-relocate ordering in
`CommonAttentionBuilder.build` -- was a consequence.

Under #2045 a K3 checkpoint is a PAGE image in the KV pool whose 127 units are
*reserved*, and reserved units are structurally un-takeable: `_take_free` draws
only from `_free` and they sit in `_used`, `BlockPool.free` raises on one, and
`retire_top` refuses outright (`if top in self._raw_unit_owner: return None`).
The source holds still on its own, so a transfer needs no rescue -- exactly the
property dense KV has always had, where `should_defer_free` alone carries a save
through its D2H and no ring was ever needed.

So this is not a simplification, it is the removal of a workaround whose premise
is gone. `state_pool.py` and `sub_pool_spec.py` now have an EMPTY diff against

--- what stays, and why

`state_entry_views` stays (re-indexed from group to slot, since a checkpoint is
one slot wide now). Phase 4's load writes the committed slot directly, so this
is its destination -- the plan's earlier draft had it deleted, which was wrong.

`StagedTransfer` / `_StagingBuffer` stays. It shares a word with the ring and is
a different thing: a per-thread pinned host bounce buffer that D2H and H2D both
need, sized in bytes and unrelated to slot count.

`StateOffloadIndex` keeps its load half (`request_load` / `complete_load` /
`fail_load` / `abandon_load`) and its optimistic hash set. `confirm_spill`
becomes `note_stored`, which is what it always did.

`StateOffloadTier.submit_spill` becomes `submit_store(h, slot)` and loses its
`ready_event`: there is no compute-stream producer to fence against once the
packer gathers straight from a source that is not moving.

--- what this leaves inert, deliberately

**Nothing calls `submit_store` after this commit.** Phase 2 removed the ring
that used to drive it and Phase 3 gives it its new caller, whose source is the
checkpoint's PAGE units rather than a slot. Nothing populates
`StateOffloadIndex.hashes` in between, so `_tier_can_serve` is false and no load
is ever offered -- the tier is fully inert, on top of the three gates Phase 1
already documented. That is a WIP state, not a shippable one.

--- one dead-code find, pre-existing on #1960

A method between `_tier_can_serve` and `_request_state_load` had lost its `def`
line in an earlier rebase, leaving its docstring and `return` as unreachable
statements after `_tier_can_serve`'s own return. It reads like a live invariant
("Whether `hit_hash`'s state can be produced at all, from either tier") and has
no caller: the tier's vote actually reached `_gated_hit` through
`StateSlotPool._resumable_from`. Removed rather than restored, since Phase 4
puts the vote on the coordinator side.

Verified: pytest tests/ (less tests/plugin and the two msgpack/aiter collection
errors that fail on #2045 too) -> 3906 passed, 326 skipped, 3 xfailed, zero
failures. black clean; ruff 36, the same count as #2045. `git diff` against

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Phase 3a. Deletes the tier's separate `atom-state-{rank}` LMCache engine, its
`OFFLOAD_STATE_CPU_SIZE` knob, and its `cache_policy = "FIFO"` override. The
codec now binds the same `storage_manager` the KV leg uses.

--- what changed

  - `_state_storage_manager()` is gone (~50 lines). `bind_storage_manager` is
    handed `self._engine.storage_manager` directly. The `gib <= 0` branch that
    already did this was the whole implementation; what goes is the `> 0` half.
  - `OFFLOAD_STATE_CPU_SIZE` removed, from the code and from
    `docs/environment_variables.md`.
  - `cache_policy = "FIFO"` removed; state inherits the KV pool's LRU.

--- why the two-pool argument does not hold

It rested on "the KV write stream is several times the state volume, so it
evicts the checkpoints". Measured against the code rather than assumed:
`page_unit_bytes = num_MLA_layers * block_size * entry * itemsize = 442,368`
with `--block-size 128` pinned by the recipe, so KV costs 3,456 B/token and a
53.6 MiB state image is worth ~16k tokens of it -- 13% of a 117k-token prompt's
KV. (Read the MLA layer count the other way, as a quarter of 93 layers, and it
is 3.3%. The conclusion is the same either way, which is why it did not need
settling first.)

And the byte ratio is not what decides an LRU eviction anyway. A request writes
its ~457 KV chunks and its one state object inside the same prefill window, so
both enter the LRU at the same position and cool at the same rate. State is
retired ALONGSIDE its own KV -- which is the behaviour we want, because a
boundary whose KV is gone is worthless.

Two pools cost what one does not: two eviction policies that must drift, while
a joint boundary needs both legs to survive together. #2045 made this same
argument one tier up -- state moved out of its reserved slots into the KV pool,
81.85% -> 93.60% -- and `OFFLOAD_STATE_CPU_SIZE` is the same reserved capacity
one tier down, so keeping it repeats the thing #2045 disproved.
`LMCACHE_MAX_LOCAL_CPU_SIZE` is now the one size to tune.

--- the cost, stated rather than hidden

`cache_policy` is per-engine, so one pool means one policy for both legs. That
is a real loss of freedom and it is currently free, because LRU is right for
both. The FIFO override's premise was "a state entry is written once and read
once"; #2045's own numbers refute it (~4,808 resumes over ~1,508 checkpoints is
~3 reads each), and re-access is bursty -- branches and retries on one boundary,
then superseded -- which is LRU's home ground. The genuine want, if it shows up,
is "stickier than LRU gives it", which FIFO does not provide either; the lever
then is touching the state key on a KV hit, not splitting the pool back apart.

Nothing yet measures whether that is needed. The counter to add with the store
path (Phase 3d) is `joint_boundary_state_miss`: a boundary whose KV leg landed
and whose state leg missed. Zero means the two pools never needed separating.

Verified: pytest tests/test_lmcache_offload_connector.py -> 186 passed. black
clean; ruff unchanged.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Phase 3b. `StateByteCodec.key()` stops being the bare ATOM hash; it becomes
`xxh64(hash || layout_id)`. `get()` gains a size check on what comes back.

--- what changed

  - `StateByteCodec.__init__` takes a required `layout_id` and refuses an empty
    one. `_build_state_tier` reads it off `state_runtime.checkpoint_spec` and
    declines to build the tier at all when it is absent, rather than falling
    back to an unqualified key.
  - `key(h)` hashes `h` and the layout id together with `xxh64`.
  - `get(h, ...)` measures the returned `MemoryObj` and, on a size other than
    `entry_bytes`, discharges its reference, counts `_misfit_reads`, warns, and
    returns a miss.
  - `_object_bytes()` reads `get_size()` with the tensor as fallback, and
    returns None when neither answers.

--- why the key needed both

*Build safety.* The same prefix hash names a completely different image under a
different `num_spec`, TP size, or conv/ssm dtype. A size mismatch throws at
`entry_bytes`, but **the same size with a different order or meaning reads back
silently wrong state** -- and a request resumed onto wrong state produces wrong
output with nothing raised anywhere. #2045 already encodes all of it in
`layout_id` (layers / conv shape+dtype / ssm shape+dtype / order / tp / spec /
carry) and enforces it HBM-side in `_validate_paged_state_op`. This is the CPU
side of the same check, and it was missing.

*Namespace separation.* `CacheEngineKey` is
`(model_name, world_size, worker_id, chunk_hash, dtype)` -- no field says what an
entry IS. KV chunk keys carry `ChunkedTokenDatabase`'s chunk hash and state keys
carry ATOM's chained block hash: two different hash functions writing plain
integers into one space. Until Phase 3a they at least lived in different pools;
now they share one, and the only thing keeping them apart was accidental -- this
side hard-codes `torch.uint8` while KV carries the KV dtype. `MemoryFormat` is
not available as a discriminator either: the tier's own comment records that its
`KV_2LTD` is inert, chosen only to pass the allocator's `raise ValueError`.
Folding the layout in makes the two spaces disjoint by construction, since no KV
key can carry a K3 layout id.

`xxh64` rather than Python's `hash((h, layout_id))`: `hash` of a str is salted
per process, so a restart would silently orphan every entry the previous run
wrote -- a cache that quietly starts cold and never says so. `xxh64` is what the
block hashes themselves use.

--- why the size check, if the key already prevents it

It is unreachable, and that is the point: reaching it means two things collided
in the shared pool, and unpacking a colliding object writes another entry's
bytes over a request's live state. Degrading to a miss costs one recompute; not
checking costs silent wrong output. `_misfit_reads` is the counter that says the
key stopped doing its job.

None from `_object_bytes` means "cannot measure", never "size 0" -- refusing
what cannot be measured would turn an unknown into a guaranteed miss.

--- tests

Five, in `tests/test_lmcache_offload_connector.py`, on a `CacheEngineKey`
fixture rather than the full lmcache stub (these are about two keys being equal
or not, which the real dataclass answers by value): a layout change makes a
different key; the layout does not collapse distinct hashes; the key survives a
restart; a state key cannot equal a raw-hash KV key; a wrong-sized hit is a miss
with its reference discharged; an object that will not report a size is still
loaded.

Verified: pytest tests/ (less tests/plugin and the two msgpack/aiter collection
errors that fail on #2045 too) -> 3911 passed, zero failures. black clean;
ruff 36, unchanged.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Phase 3c. Adds `_KimiMLAGDNCommon.page_unit_views(unit_ids)` -- the tensor-view
counterpart of #2045's `_page_unit_regions`. No behaviour changes; nothing calls
it until Phase 3d.

--- what changed

  - `page_unit_views(unit_ids) -> list[torch.Tensor]`, one view per (unit, MLA
    row), unit major and row minor.
  - Seven tests in `tests/test_kda_checkpoint_slot_copy.py`, on the harness
    #2045 already built for `_page_unit_bases`.

--- why a second way to name the same bytes

`_page_unit_regions` hands the Triton descriptor raw int64 addresses, which is
what a D2D copy the GPU issues wants. The LMCache staging packer takes
`list[torch.Tensor]` -- `fused_pack_chunk_major` is already a fully
parameterised gather over `segment_ptrs[] + segment_block_bytes[]`, so it needs
no kernel change, only its segments named the way it names them.

This is the ONLY new seam the tier needs. A load writes the Active Slot directly
and reuses `state_entry_views`, so the asymmetry is deliberate: the save gathers
from units, the load scatters to a slot, and the two compose because #2045's
`plan_segmented_copy` intersects two *ordered byte streams* -- the slot's ranges
and the units' regions are two cuts of the same logical image, so a blob
gathered in one order is byte-identical to one gathered in the other.

--- the two traps, both asserted rather than assumed

*Order is the contract.* Unit major, row minor -- the same ravel as
`_page_unit_bases`, which is the order `_checkpoint_copy_plan` builds the
destination stream in. The packer indexes segments positionally, so this order
IS the blob layout. Self-consistency would be enough for one build; what makes
it a contract across builds is `layout_id` in the key (Phase 3b), so a build
that reordered cannot read another's blob.

*`block_ratio`.* `unit_ids` carries **logical** block ids while `kv_cache` is
shaped in **physical** ones, and K3's ratio is 128. Flattening the two block
axes together and indexing by `unit * block_size` is the conversion -- the same
arithmetic `_page_unit_bases` does in addresses, so the two cannot drift into
disagreeing about which bytes a unit owns. The range check is against the
*logical* count for the same reason: the physical count is `ratio` times larger
in rows, so checking the wrong axis admits ids that read past the end. Its test
computes the bound from the tensor rather than from a constant, because taking
the harness's label would let the very confusion under test pass unnoticed.

`_page_unit_regions` is called first for its side effect: the contiguity and
granularity checks live there, and they have to run before any view is handed
out rather than after.

--- verification

The unit tests skip on a non-GPU runner (the module imports aiter at load), so
the geometry was checked directly by extracting the four methods and comparing
`page_unit_views` against `_page_unit_bases` byte for byte: 9 segments for 3
units x 3 rows, identical `data_ptr()`s, lengths equal to
`_page_unit_stream_sizes`, every view contiguous, no two units sharing a byte,
an out-of-range id refused, and a granularity mismatch refused before any view
is produced.

pytest tests/ (less tests/plugin and the two msgpack/aiter collection errors
that fail on #2045 too) -> 3911 passed, zero failures. black clean; ruff 36,
unchanged.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… evicted

Phase 3d, the last of Phase 3. This is what turns the tier back on for saves:
a checkpoint that reaches READY is offered to the CPU tier, and the whole
engine->worker->engine round trip is wired. Loads are still off (Phase 4).

--- what changed

`page_unit_checkpoint.py`
  - `_offload_ready`: checkpoints nominated at the READY transition. NOT pinned.
  - `take_offload_stores(max_inflight)`: hands `(hash, unit_ids)` over and takes
    the pin at that moment, up to the cap.
  - `settle_offload_store(hash)` / `reclaim_stale_offload_pins(timeout_s)` /
    `offload_pins_reclaimed`, exposed on the coordinator and reported in
    `checkpoint_fates`.
  - `PageUnitCheckpointStore(..., offload_sink=)` -- False nominates nothing.

`block_manager.py`
  - `take_state_stores` / `settle_state_store(hash, ok)` /
    `reclaim_stale_state_store_pins`. The switch (`kv_connector_hosts_state_tier`)
    moves above the coordinator's construction, which now needs it.

`scheduler.py`
  - `_publish_state_stores()`, beside `_publish_state_loads()` and on the same
    two call sites plus the idle path.
  - `_offload_max_pending_saves()`, reading the KV leg's own
    `OFFLOAD_MAX_PENDING_SAVES`.
  - The report drain settles the pin on both outcomes and runs the reconciler.

`kimi_k3/connector.py` -- `enqueue_state_stores`, `meta.state_stores`,
`_start_state_stores`, and `_state_store_failed_locally` for stores a worker
with no tier cannot even attempt.
`kimi_k3/state_tier.py` -- `submit_store(h, unit_ids)`, no `ready_event`.
`kimi_k3/state_object.py` -- `put(h, unit_ids)` gathers `page_unit_views`;
`get(h, slot)` still scatters into `state_entry_views`; `puts_refused`.
`metadata.py` / `types.py` -- `state_stores` declared as work.

--- why READY, and why the pin is not taken there

Three candidate trigger points and only one works. `begin_store` is too early:
the scatter has not ridden a batch, so the units do not hold the image yet.
`_evict` / `unindex` are too late -- they fire when the pool wants those units
*now*, so a multi-millisecond D2H started there holds 127 of the most-wanted
blocks at the worst possible moment. READY is when the bytes first exist and
when the record is least wanted: it has just entered the LRU at the cold end.

The pin is a separate question, and getting it wrong broke a real invariant --
`test_a_ready_unpinned_checkpoint_is_available_to_live_kv` caught the first
draft. A pin lives in this process while the copy runs in the worker, so it
spans several scheduler passes. Pinning at READY would make EVERY checkpoint
un-evictable for that window, which is exactly what #2045's admission argument
forbids. So READY only *nominates*: a nominee stays unpinned and spendable, and
`take_offload_stores` pins only the few actually in flight. A checkpoint the
pool needed more than the tier did simply loses its copy, which is the right
price for never making the pool wait on the CPU.

The cap is `OFFLOAD_MAX_PENDING_SAVES`, shared with the KV leg rather than given
a knob of its own: a KV save and a state store both hold bytes out of the same
pool while they run, so the depth is one question about one resource, and two
numbers would let an operator raise one and unknowingly double the other.

--- write volume, which is the objection this trigger invites

Storing every checkpoint rather than only evicted ones is more traffic, and it
is affordable: per request state adds 53.6 MiB on top of ~404 MB of KV, **13%**
on a path already in flight (~12 MB/s against the KV leg's ~90 MB/s over a
3600s run). What it buys is the trigger being at the cheapest possible moment
and dedup for free -- `begin_store` already refuses a hash it holds or has
pending, so one hash is offered once.

--- unindex during a copy does not cancel it

`unindex` means the boundary's KV block was spent, so the image is unreachable
in HBM from here on. That is precisely when the CPU copy is worth having, so
the pin holds the units to the end of the D2H and `_release_record` runs after.
The CPU copy outliving the HBM one is the entire point of the tier.

--- the lost report, and why 20 lines is enough

`reclaim_stale_offload_pins` takes the same window as
`_reconcile_stalled_deferred_saves`, derived from LMCache's own pin timeout for
the same reason. Recovery is total: a leaked pin breaks no `BlockPool`
invariant and leaves no half-released state, so zeroing the count restores the
record exactly -- which is why this needs a counter and a timer rather than any
repair logic. `offload_pins_reclaimed` reaches `checkpoint_fates`, because
otherwise the symptom is a pool that has quietly shrunk.

--- still off

Loads. Nothing calls `_request_state_load` for K3 yet: `_attach_state_slots`
returns from its PAGE branch before reaching `_tier_can_serve`, and the
`begin_restore` raise above it has to become an issue-a-load branch first
(Phase 4). `StateOffloadIndex.hashes` now fills, so the tier is half live: it
writes, and nothing reads yet.

--- tests

11 new in `tests/test_page_unit_checkpoint.py` -- nomination does not pin, the
hand-off does, a nominee stays spendable and may be lost, no sink nominates
nothing, offered once, the cap makes the rest wait rather than drop, a waiting
nominee stays evictable, both outcomes release, a report for a hash never sent
is a no-op, `unindex` mid-copy holds the units to the end, and a lost report is
reclaimed with the record left unbroken. Two more in
`tests/test_lmcache_offload_connector.py`: stores drain exactly once, and a
step whose only work is a store still reaches the worker.

Verified: pytest tests/ (less tests/plugin and the two msgpack/aiter collection
errors that fail on #2045 too) -> 3924 passed, zero failures. black clean;
ruff 36, unchanged.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…raised

Phase 4a, and it has to be first: with nothing else changed this commit is
behaviour-preserving, but without it the first request that actually uses the
CPU tier takes the engine down.

--- what changed

`BlockManager._attach_state_slots`, PAGE branch. Was:

    if hit_hash != -1 and not begin_restore(hit_hash, seq.state_slot):
        release_many(seq.state_slots); seq.state_slots = []
        raise RuntimeError("gated PAGE checkpoint disappeared before state attach")

Now: cold start returns early; `begin_restore` success returns as before; a
miss falls through to `_request_state_load`, and only if that also declines is
the boundary disowned (`return False`, which `allocate` turns into
`num_cached_tokens = 0`). New counter `state_gate_lost_boundary`, reported in
`checkpoint_funnel`.

--- why the raise could not stay

It asserted that the gate and the HBM store agree. That held while
`can_allocate` only ever accepted boundaries the HBM index carried. Phase 4b
gives the CPU tier a vote, so a hash the gate accepted may live only there --
and every such request would hit this raise. Turning it into a load branch has
to land before the vote does, not after, or the two commits are only safe as a
pair.

--- why disowning is reachable rather than defensive

Two ways to arrive with neither tier able to produce it:

  * the tier's index is optimistic by construction -- `hashes` means "was
    stored once", never "is still there", because LMCache's own LRU can drop
    bytes at any time;
  * an HBM checkpoint can be unindexed between `can_allocate` and `allocate`
    by another seq's `_fresh_block` in the same pass.

So this is a normal path with a counter, not an assertion. A large fraction of
`joint_boundaries` landing here would mean the gate is accepting boundaries
that do not survive to attach.

--- the slots are kept, unlike the old abort

The old code released the slots because it was aborting. Disowning is not an
abort: the request is about to recompute its whole prefix, and it writes that
state into these very slots. Releasing them would hand the next request a
buffer this one is still filling.

`test_missing_gated_checkpoint_releases_the_new_slot_and_raises` becomes
`test_a_gated_boundary_neither_tier_has_is_disowned_not_raised`, and now pins
all three: returns False, keeps its slots, counts one.

Verified: pytest tests/ (less tests/plugin and the two msgpack/aiter collection
errors that fail on #2045 too) -> zero failures. black clean; ruff unchanged.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…m it

Phase 4b. This is the commit that makes the tier reachable: until now K3 wrote
53.6 MiB per checkpoint to LMCache and nothing ever read it.

--- what changed

`page_unit_checkpoint.py`
  - `PagedStateCheckpointCoordinator.offload` + `attach_offload(index)`.
  - `resumable_hit` now asks `_reachable(h)` instead of `store.contains(h)`:
    HBM first, then `offload.hashes`.
  - The constructor's `offload_sink: bool` is gone -- `attach_offload` sets the
    vote and the sink together (see below).

`block_manager.py`
  - Attaches the index to the coordinator once both exist. The coordinator is
    built from `state_runtime.checkpoint_spec` and the index from the connector
    config, so they are constructed in the wrong order for a ctor argument.

`kimi_k3/connector.py`
  - Refuses to build the tier when `image_bytes != entry_bytes`.

--- why the vote is the whole feature

`can_allocate` walks the HBM prefix and asks every `Pool.STATE` class "the
rightmost boundary <= X that you accept". With the coordinator answering only
from `store.contains`, a hash whose image went to LMCache is invisible: the
walk stops short, nothing ever asks for it back, and the store path is pure
cost. `_reachable` is where the tier gets to say "I have that one".

It stops at the first boundary it accepts, scanning right to left, so a hash
accepted here that nothing can deliver does not cost a wasted lookup -- it
costs the whole walk-back, hiding every shorter checkpoint still resident in
HBM. That is why the disown path (Phase 4a) had to land first.

No preference rule between the tiers is needed: both are keyed by the same
content hash, so the scan takes the rightmost boundary wherever it lives, and
`_attach_state_slots` tries `begin_restore` before `_request_state_load`
regardless. A resident image never pays a park.

--- why the vote is optimistic, deliberately

`hashes` means "was stored once", never "is still there" -- LMCache's own LRU
can drop bytes under it at any moment. A false positive costs one park plus a
recompute, and retracts itself: `fail_load` calls `forget(h)`, so the next
request over that prefix is not sent down the same hole. Being certain instead
would mean a synchronous cross-process lookup on the admission path.

--- why attaching is one call, not two fields

Half-attached is worse than off in both directions. A vote with no sink accepts
hashes nothing will ever store, so every accepting request parks and
recomputes. A sink with no vote pins 127 units per checkpoint for a transfer
nobody will read. `attach_offload` sets both, so they cannot diverge.

--- the load's destination is the Active Slot

Not 127 freshly reserved units. The two are interchangeable because #2045's
copy plan intersects two *ordered byte streams* -- the slot's ranges and the
units' regions are two cuts of one image -- so a blob gathered in unit order
unpacks correctly in slot order. Writing the slot directly means no
`ensure_free_units` on the admission path, no refusal branch, no units pinned
for the length of an H2D, and no cap needed on concurrent loads.

It does mean the store reads `image_bytes` and the load writes a whole slot, so
the connector now refuses to build a tier where those differ, at the one point
both numbers are in scope. They are equal for K3; a model where they are not
would silently truncate the store or over-read the load.

--- tests

Six in `tests/test_page_unit_checkpoint.py`: a tier-only hash is accepted, no
tier means no acceptance, the scan still takes the rightmost, attaching turns
on both halves together, and the optimistic half retracts on `forget`.

Three in `tests/test_state_checkpoint.py`, end to end through
`_attach_state_slots`: a tier-only boundary becomes a queued load with no
restore op and the slot as its destination; a boundary in both is served from
HBM with no park; a tier that declines disowns and counts.

Verified: pytest tests/ (less tests/plugin and the two msgpack/aiter collection
errors that fail on #2045 too) -> zero failures. black clean; ruff unchanged.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ason to exist

Phase 5, the last one. `_joint_kv_boundary` refused every PAGE sequence; it now
serves only PAGE. With this the tier is reachable end to end.

--- what changed

`block_manager.py`
  - The gate inverts:
      -  if not seq.has_per_req_cache or self.paged_state_checkpoints is not None:
      -      return self._no_joint("not_hybrid")
      +  if not seq.has_per_req_cache or self.paged_state_checkpoints is None:
      +      return self._no_joint("no_paged_checkpoints")
  - The hbm/tier split reads the coordinator instead of the slot pool.
  - `joint_boundaries_hbm` / `_tier` become `state_hbm` / `state_tier`.

`page_unit_checkpoint.py` -- `contains(h)` delegation, so the split does not
reach into `.store`.

`scheduler.py` / `llm_engine.py` -- the log line and the Prometheus tuple follow
the rename.

--- why the old refusal was right, and why it stopped being right

It read `is not None` and refused, because a K3 checkpoint was an Active Slot
that the tier spilled out of `StateSlotPool` -- a PAGE image lived in the KV
pool, which the KV connector already offloads on its own, so there was no
second boundary to agree on.

#2045 moved K3's image into the KV pool and with it the entire reason the joint
path exists. HBM now enforces `state ⊆ KV`: `_record_evicted` unindexes a
checkpoint the instant its boundary block is spent, so a checkpoint can no
longer outlive its KV there. **When LMCache hands the KV back, nothing hands
the state back unless the two are fetched together.** Leaving this refusal in
place makes the tier dead weight -- it writes 53.6 MiB per checkpoint and
nothing ever reads it -- and no other counter in the system would say so.

--- why the counters were renamed rather than left alone

`joint_boundaries_hbm/_tier` described the boundary as a whole, which was true
when the KV leg had no tier of its own to be served from. It has one now
(`_decide_load_after_alloc` decides it separately), so the old names read as if
they covered both legs. `state_hbm` / `state_tier` say what they measure.

The split also means something new. It used to say "the state pool is too small
for this concurrency". Under #2045 checkpoints share the paged pool with the KV
write stream, so a ratio dominated by `state_tier` says **the paged pool** is
too small -- checkpoints are being squeezed out by the KV they now live beside.
That diagnosis did not exist before.

`state_tier` is also the only counter here that cannot be non-zero with the CPU
tier switched off, which makes it the honest test of "did this feature run".
No passing unit test can produce it in production.

--- tests

Three in `tests/test_state_checkpoint.py`, on a harness that had to reproduce
the shape the joint path is for: publish a prompt, place ONE rung at an
interior boundary (a checkpoint under the last block is one no scan can look
up, since `can_allocate` matches over `range(n_hash_blocks - 1)`), then evict a
KV block below it so the walk stops short. A PAGE seq gets a joint boundary at
the rung with real work for the KV leg; the split reports `hbm` while the
checkpoint is resident and `tier` once it is not; no tier means no boundary.

Verified: pytest tests/ (less tests/plugin and the two msgpack/aiter collection
errors that fail on #2045 too) -> 3935 passed, zero failures. black clean;
ruff unchanged.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…res`

Found by the first end-to-end run of the K3 CPU state tier on this branch: the
tier reported healthy on both sides and stored exactly nothing. 94 checkpoints
were nominated, handed to `_publish_state_stores`, refused, and had their PAGE
units released again.

--- the symptom

Server log, 94 times over a 6-minute agentic replay at concurrency 8:

    state offload: LMCacheOffloadConnectorScheduler did not carry 1 state
    store(s); releasing their units now.

with everything else looking correct:

    lmcache_offload: worker family=kimi_k3
    kimi_k3 offload: state tier up, entry=55.39 MiB rank=0, sharing the
      paged-KV CPU pool, layout=kda-paged-state-v1:layers=69:...
    state checkpoints: checkpoints_kept=112 ... offload_pins_reclaimed=0

So the worker half built the tier, the coordinator nominated checkpoints at
READY, and `take_offload_stores` pinned and handed them over -- and the
scheduler half dropped every one.

--- the cause

`LMCacheOffloadConnectorScheduler` is a shell that owns no behaviour: it holds
`self._impl = _build_scheduler(config)` and forwards each method by hand.
`7b6b58e0` added `enqueue_state_stores` to `KimiK3OffloadScheduler`
(kimi_k3/connector.py:352) but did not add the matching forwarder to the shell,
so the method existed on the implementation and was invisible from outside:

    $ git grep -c enqueue_state_stores atom/kv_transfer/offload/connector.py
    0

`Scheduler._publish_state_stores` probes with
`getattr(self.kv_connector, "enqueue_state_stores", None)`, and
`self.kv_connector` is the shell, so the probe returned None unconditionally
and the store path took its "nothing will carry these" branch on every pass.
This is not a race or a config error -- it could never have worked.

The load leg was unaffected only by luck: `enqueue_state_loads` and
`take_state_reports` were both given forwarders when they were added. Loads
still moved no bytes, because with nothing ever stored `offload.hashes` stayed
empty and the tier had nothing to vote for.

--- the fix

Add the missing forwarder, in the same shape as its `enqueue_state_loads`
neighbour so the two cannot drift again.

--- the second change, and why it is in this commit

The warning that reported the failure named a cause that was not the one:

    The tier's index should not have been installed against this connector.

That reads as a configuration mistake, and it sent the first pass of this
diagnosis at `kv_connector_hosts_state_tier` and the layout selector -- both of
which were behaving correctly. The class name printed is the *shell*, which
tells you nothing about whether `_impl` supports the call. Both warnings now
name the two possible causes and say which class is being reported.

--- verification

Static: the probe is `getattr` on a method the shell does not define, so the
refusal is unconditional; adding the forwarder is the whole of the repair, and
it is additive -- no existing path changes.

Runtime: re-running the same workload with this commit applied, the
`did not carry` warning no longer appears.

Lines conform to black's 88-column limit; black and ruff are not installed in
the benchmark image, so formatting was checked by inspection against the
surrounding style rather than by running them.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…that holds none

Three observability defects, found while trying to judge whether the K3 CPU
state tier was working. Together they meant the tier could store nothing at all
and no number anywhere would say so -- which is exactly what happened in
8a5364a, where 94 refused stores produced one warning line and four counters
that all read zero for unrelated reasons.

--- 1. the store leg had no counters at all

`StateOffloadIndex.stats()` reported `loads_attempted / completed / failed` and
`indexed`. On a K3 run the load counters stay zero until something has been
stored, so the only store-side signal was `indexed` growing -- and `indexed` is
a set size that also grows for reasons unrelated to this pass.

Added `stores_attempted` (handed to the connector, counted in
`take_state_stores`), `stores_completed` / `stores_failed` (split in
`settle_state_store` by the worker's report), and `stores_refused`.

`stores_refused` is deliberately apart from `stores_failed`, and it is the one
that matters most: failed means the worker tried and could not, refused means
nobody tried. The shell-forwarding bug fixed in 8a5364a was a pure `refused`
condition, and with this counter it would have been one glance at the periodic
line instead of a hunt through the connector factory.

The worker's own `puts_refused` needs no separate channel: a refused `put`
already returns False and settles as `ok=False`, so it lands in `stores_failed`.

--- 2. `[Pool Pressure]` read the pool that holds no checkpoints

`BlockManager.pool_pressure()` folded in `self.state.checkpoint_fates()`. Under
#2045 a K3 checkpoint is a PAGE image the coordinator owns and the slot pool
holds none, so the line printed

    [Checkpoint Fates] kept: 0, dropped: 0, evicted: 0, orphaned: 0

on the same run whose other line said `checkpoints_kept=112`. Two counters for
one fact, disagreeing in one log, with the wrong one printed in the section a
reader goes to for pool pressure.

Now reads `self._state_checkpoint_cache`, which is already the coordinator when
there is one and the slot pool otherwise -- the same object every other
checkpoint question is asked of. `occupancy()` stays on the slot pool, which is
the thing that has slots.

--- 3. the tier's counters vanished when K3's checkpoints moved to PAGE

`StateGroupPool.checkpoint_fates` used to fold the tier's stats in under a
`state_offload_` prefix. `PagedStateCheckpointCoordinator.checkpoint_fates` was
written without that, so once #2045 moved K3's checkpoints into the coordinator
the whole `state_offload_*` family disappeared from the logs -- a silent
regression, since nothing errors when a key stops being emitted.

Restored on the coordinator. The fold is `getattr`-guarded because
`attach_offload` accepts anything answering `hashes`, and the tests attach a
double with no counters; a missing `stats` means "nothing to fold", not an
error. (`test_a_tier_that_declines_disowns_rather_than_parking` caught this on
the first draft.)

--- also added: `state_offload_store_backlog`

A gauge rather than a counter: `len(_offload_ready)`, the nominations waiting
for a slot under `OFFLOAD_MAX_PENDING_SAVES`. That queue is unbounded and
drained oldest-first, so checkpoints reaching READY faster than the tier drains
them shows up as a value that climbs and stays high, with fresh nominations
queueing behind stale ones. There is no other way to see it -- the stale
entries are skipped cheaply on the way out, so nothing else registers the
backlog.

--- how to read the new line

    state_offload_stores_refused  > 0  -- wiring fault, the connector never took it
    stores_attempted - completed - failed  -- in flight
    stores_completed vs checkpoints_kept   -- how much of what HBM keeps the tier got
    loads_failed / loads_attempted         -- the index's false-positive rate
    store_backlog climbing                 -- READY outruns the drain

Verified: pytest tests/test_page_unit_checkpoint.py tests/test_state_checkpoint.py
tests/test_lmcache_offload_connector.py tests/test_block_manager.py -> 466 passed.
Lines conform to black's 88-column limit; black and ruff are absent from the
benchmark image, so formatting was checked by inspection.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every state store on this branch has failed since the store path was wired.
Found by the first run where stores were actually carried (8a5364a): 3,904
tracebacks, all identical, and not one checkpoint reached LMCache.

--- the symptom

    Traceback (most recent call last):
      File "kimi_k3/state_tier.py", line 109, in _do_store
        stored = bool(self.codec.put(h, unit_ids))
      File "kimi_k3/state_object.py", line 141, in put
        self._staged.pack(self._backend.page_unit_views(unit_ids), obj)
      File "kimi_k3/staging.py", line 175, in pack
        from atom.kv_transfer.offload.triton_kv_staging import fused_pack_chunk_major
    ModuleNotFoundError: No module named 'atom.kv_transfer.offload.triton_kv_staging'

    state offload: store of hash 10785477046942485201 failed

3,904 of them over a 50-minute concurrency-16 agentic replay -- one per store
attempt, on every rank. `stores_failed` would have counted all of them, and
`indexed` never moved.

--- cause 1: the module is one package deeper

`fused_pack_chunk_major` lives in `atom.kv_transfer.offload.dense.triton_kv_staging`.
The dense codec, which has used it all along, imports it from there
(`dense/kv_byte_codec.py:127`). The K3 stager's import dropped the `.dense`.

Nothing caught it because the import is function-local, inside `pack`, so it is
only attempted when a store actually runs -- and until 8a5364a no store ever
reached the worker. The unit tests do not reach it either: they exercise the
codec against doubles that never call the real packer.

--- cause 2: a failed `put` leaked its allocation, once per failure

    obj = self._allocate(self.entry_bytes)     # ref_count = 1
    self._staged.pack(...)                     # raises
    self._storage.batched_put([key], [obj])    # never reached

`batched_put` discharges the reference it is handed
(`StorageManager.batched_put` ends in `memory_obj.ref_count_down()`), so the
success path correctly owes nothing -- but only if it is reached. With `pack`
throwing, the allocation was stranded at ref_count=1 and LMCache reported it
much later:

    MemoryObj at ... is being garbage collected with ref_count=1, pin_count=0.
    This indicates ref_count_down()/unpin() was not called

3,768 such warnings against 3,904 failed stores: one leak per failure. Left
alone this shrinks the shared CPU pool by one 53.6 MiB entry per failed store
until nothing can be allocated at all -- so the second-order effect of the
import bug was a slow starvation of the pool the KV leg shares.

`get` already guarded its own reference with `finally` for exactly this reason;
`put` did not. It does now, on the exception path only, matching what the DSV4
codec does (`hybrid/dsv4/codec.py:1293`).

--- why one commit

The leak is only reachable through a throwing `pack`, and the import was the
only thing throwing. Fixing the import alone would hide the leak rather than
remove it, and the next exception from that line -- an out-of-memory in the
staging buffer, say -- would strand allocations again with no traceback to
explain the pool shrinking.

--- verification

    python -c "from atom.kv_transfer.offload.dense.triton_kv_staging import
               fused_pack_chunk_major"      -> OK

    pytest tests/test_lmcache_offload_connector.py tests/test_state_checkpoint.py
           tests/test_page_unit_checkpoint.py  -> 412 passed

The import is still function-local, so this remains untested by anything that
does not run a real store. A unit test that calls `StagedTransfer.pack` against
real CUDA tensors would have caught it; that is worth adding and is not in this
commit.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ot reach

Fix-then-sweep over `dfab0fa0` and `8a5364ab`. Both of those repaired one
instance of a bug that had two, and neither instance the fix reached is covered
by any test.

--- 1. `unpack` imports the same module that does not exist

`dfab0fa0` corrected `StagedTransfer.pack`'s import of
`fused_pack_chunk_major` (the module is one package deeper, under `.dense`).
`unpack` imports `fused_unpack_chunk_major` from the same wrong path, eleven
lines further down, and was left alone.

It has never fired because no state load has ever run -- with nothing stored,
`offload.hashes` stayed empty and the tier had nothing to vote for. The first
successful store makes the first load reachable, and it would have raised
`ModuleNotFoundError` on that load, on every rank.

Both kernels live in `atom.kv_transfer.offload.dense.triton_kv_staging`;
verified by importing both.

--- 2. `chunk_size` is not on the delegating shell either

`8a5364ab` added the missing `enqueue_state_stores` forwarder. Sweeping every
member `Scheduler` reads off `self.kv_connector` against what
`LMCacheOffloadConnectorScheduler` defines turns up one more: `chunk_size`.

    scheduler.py:2284   seq.offload_kv_chunk_tokens = int(
                            getattr(self.kv_connector, "chunk_size", 0) or 0)

`self.kv_connector` is the shell, the shell has no `chunk_size`, so this has
always stamped 0. It is latent rather than live: `_joint_kv_boundary` reads
`self._joint_chunk_tokens` first and only falls back to this, and that primary
comes from `BlockManager.__init__` reading the LMCache config directly. But the
fallback exists precisely for the case where that read failed -- and it was
dead, so a build that could not read the config at startup would refuse every
joint boundary with `no_chunk_size` and never say why.

Forwarded as a property, with the reason in its docstring.

--- 3. the class of bug, closed rather than fixed twice

Both are the same shape: the shell forwards by hand, so a member added to an
implementation is invisible from outside, and every reader is a
`getattr(..., default)` that does not raise -- it takes the default forever.
Two instances in two days says the next one is a matter of time.

`test_every_member_the_scheduler_reads_is_reachable_through_the_shell` scans
`scheduler.py`'s source for `getattr(self.kv_connector, "x")`,
`self.kv_connector.x`, and `_connector_flag("x")`, and asserts every name is
reachable on the shell class. It reads the source rather than a hand-kept list,
because a hand-kept list is the same failure one level up. The scan asserts it
found something, so a regex that stops matching fails loudly instead of passing
vacuously.

Falsified: renaming the new `chunk_size` property makes it fail and name the
missing member.

Verified: pytest tests/ (less tests/plugin and the two msgpack/aiter collection
errors) -> 3936 passed, zero failures. black clean; ruff 36, unchanged.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…it were image

Fourth and last blocker in the chain that kept the K3 CPU tier from storing
anything. With the previous three fixed, a 25-minute concurrency-16 replay
still ended at

    state_offload_stores_attempted=462
    state_offload_stores_completed=0
    state_offload_stores_failed=450
    state_offload_indexed=0

and 3,696 identical tracebacks:

    File "kimi_k3/state_object.py", line 149, in put
      self._staged.pack(self._backend.page_unit_views(unit_ids), obj)
    File "kimi_k3/staging.py", line 182, in pack
      dst_tensor = self.memory_tensor(dst, nbytes)
    ValueError: ATOM LMCache connector: MemoryObj tensor is too small for
                59867136 bytes; got 58079232

--- the geometry

From the run's own startup line:

    unit_bytes=2138112  image_bytes=58079232  units_per_checkpoint=28

    ceil(58079232 / 2138112) = ceil(27.163) = 28
    28 x 2138112            = 59867136      <- what page_unit_views gathered
    58079232                                <- what put() allocated
    difference               = 1787904      <- padding in the 28th unit

An image occupies whole units, and the last one is 83.6% padding here. Nothing
owns those bytes.

--- why only the CPU side hit it

The HBM path never had the problem because it already passes the image size
down: `_checkpoint_copy_plan` calls

    plan_segmented_copy(src_sizes, self._page_unit_stream_sizes(units),
                        spec.image_bytes)

so the D2D copy writes exactly `image_bytes` into the unit stream and leaves
the tail untouched. The image is, by construction, the LEADING `image_bytes`
of that stream. `page_unit_views` was written as the tensor-view counterpart of
the same addressing but did not carry the third argument across, so it named
whole units to a consumer that sizes its destination at `entry_bytes` -- which
for K3 is exactly `image_bytes`.

The connector's `image_bytes != entry_bytes` guard cannot catch this: both are
58079232 and agree. The quantity that disagrees is a third one,
`sum(page_unit_views)`, which no check compared against either.

--- the fix

Trim the gathered stream to `image_bytes`, dropping whole views past the
budget and slicing the one that straddles it.

The slice is byte-exact rather than view-aligned because the image does not end
on a view boundary: 58079232 leaves 350208 B of the last unit in use, which is
20.96 of its rows. That view is reinterpreted as `uint8` and sliced; the packer
sizes segments as `numel * element_size`, so a uint8 segment costs it nothing.

The load side needs no counterpart: it scatters into `state_entry_views`, which
sums to `entry_bytes` exactly, so both ends of the round trip are now the same
58079232 bytes.

A budget left unspent raises rather than storing a short blob -- that would be
a real geometry disagreement, and a truncated image read back as valid is the
one outcome worth crashing over.

--- two notes on shape

Inline rather than a helper method: the existing tests drive `page_unit_views`
with a `types.SimpleNamespace` stand-in for `self`, and a second attribute would
have to be stubbed in every one of them. For the same reason the spec is read
with `getattr(spec, "image_bytes", 0)` -- a fork build carries no spec at all,
and there the whole-unit stream is the right answer.

--- verification

    pytest tests/test_kda_checkpoint_slot_copy.py
           tests/test_lmcache_offload_connector.py
           tests/test_state_checkpoint.py
           tests/test_page_unit_checkpoint.py     -> 441 passed

Also confirmed from the same run that the previous commit's leak guard works:
`MemoryObj ... garbage collected with ref_count=1` went from 3,768 occurrences
to 0, even though every store still failed.

None of the four bugs in this chain was reachable by the unit tests, because
all of them live past the first line that only a real store executes. A test
that packs real CUDA tensors through `StagedTransfer` would have caught three
of the four; it is still worth adding.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
About half of every K3 state store failed. Measured over a concurrency-16
agentic replay with the four earlier blockers fixed:

    state_offload_stores_attempted=46
    state_offload_stores_completed=19
    state_offload_stores_failed=27      <- 59%

with 296 identical tracebacks:

    File "kimi_k3/state_object.py", line 153, in put
      self._storage.batched_put([self.key(h)], [obj])
                                 ^^^^^^^^^^^
    File "kimi_k3/state_object.py", line 107, in key
      digest.update(int(h).to_bytes(8, "little", signed=True))
    OverflowError: int too big to convert

    state offload: store of hash 12983297547834104159 failed

--- cause 1: the hash is unsigned and the key said otherwise

12983297547834104159 > 2**63-1, so `to_bytes(8, signed=True)` cannot hold it.

An ATOM block hash is `xxhash.xxh64().intdigest()`, which spans the whole
0..2**64-1, and `BlockManager.compute_hash` chains one into the next with
`prefix.to_bytes(8, "little")` -- unsigned, no keyword. The state key was the
only place in the chain that read the same value as signed, so it rejected
every hash with the top bit set. That is half the key space, and the observed
59% is that with sampling noise.

Nothing downstream noticed because the failure is inside a `try` in
`_do_store`, which counts a failed store and moves on. Before the counters
added in a804561 it produced no number at all.

--- cause 2: the raise stranded a MemoryObj, in a window the last guard missed

`dfab0fa0` wrapped `pack` so a throwing gather would discharge the allocation.
`key(h)` is evaluated in `batched_put`'s argument list, which is after the
allocation and before `batched_put` can take ownership -- outside that guard.
So each of the 296 overflows also leaked a 53.6 MiB entry, and the LMCache
warning reappeared next to the traceback:

    MemoryObj at 136839168 is being garbage collected with ref_count=1

Fixed by computing the key *before* allocating. `key` is pure, so hoisting it
removes the window rather than guarding it, and the `except` now covers
`batched_put` itself as well -- matching what the DSV4 codec does.

--- what this unblocks

The same `key` is used by `get`, so roughly half of all loads would have missed
in a way indistinguishable from an LRU eviction -- `fail_load` would then
`forget` a hash whose bytes were never stored under a name it could find, and
the index would decay for a reason nothing reported.

Verified: pytest over the state and offload suites -> 442 passed. The overflow
itself is confirmed directly:

    >>> (12983297547834104159).to_bytes(8, "little", signed=True)
    OverflowError: int too big to convert
    >>> (12983297547834104159).to_bytes(8, "little", signed=False).hex()
    '5f19dacf54f02db4'

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`961f80d6` fixed `page_unit_views` gathering a checkpoint's padding, and the
suite it ran stayed green before and after — because no test reaches the trim
at all.

The addressing tests build their stub spec as

    checkpoint_spec=SimpleNamespace(page_unit_bytes=...)

with no `image_bytes`, so `budget` is 0 and every one of them takes the
`return views` early path. The trim shipped untested, which is how a bug that
failed 3,696 stores in a row got there in the first place.

--- what is pinned

Seven cases in `TestPageUnitViewsStopAtTheImage`, on the harness the addressing
tests already use (`build` gains optional `image_bytes` and `dtype`; both stay
off by default so the existing tests keep exercising whole units):

  * the gathered stream is exactly `image_bytes` — the invariant `put`'s
    allocation depends on, and the one that was broken;
  * a whole view past the budget is dropped, not truncated to zero length;
  * an image that lands on a view boundary is not sliced at all;
  * **the kept bytes are the leading bytes and nothing else** — trimming
    truncates, it does not reorder. The blob is read back by scattering into
    the slot in the same order, so a byte that moved here would land in the
    wrong layer there with nothing raised;
  * a multi-byte dtype is sliced by BYTES, not elements — the straddling view
    is reinterpreted as `uint8` first, and on bf16 an element-wise slice would
    keep twice what it should. The production cache is 1 byte per element, so
    only a test with a wider dtype can hold this;
  * a spec with no image size keeps whole units (the fork build, which has no
    spec to trim against);
  * units that cannot cover the image raise rather than storing a short blob.

--- verification

`tests/test_kda_checkpoint_slot_copy.py` skips wholesale on a non-GPU runner
(the module imports aiter at load), so the seven cases were additionally run by
extracting `page_unit_views` and its three neighbours out of the source and
driving them directly. All seven hold, including the bf16 one.

pytest tests/ (less tests/plugin and the two msgpack/aiter collection errors)
-> 3936 passed, zero failures. black clean; ruff 36, unchanged.

--- one note on `put`'s exception shape, checked not changed

`2c5c3457` moved `batched_put` inside the `try` whose `except` calls
`ref_count_down`. That is safe only if `batched_put` takes ownership just on
success — and it is the pattern the DSV4 codec already uses
(`hybrid/dsv4/codec.py:1284`, "StorageManager owns the MemoryObj after a
successful batched_put"). Consistent with the repo rather than a new
assumption, so left alone.

The window between `_allocate` and `batched_put` is now fully covered:
`page_unit_views` can raise (the geometry check above) and is inside the try;
`key(h)` is computed before the allocation.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The two admission-path voters -- `PagedStateCheckpointCoordinator.
_reachable` and `BlockManager._tier_can_serve` -- tested `h in
offload.hashes`, dropping the `can_load` half that `request_load` itself
enforces. A store-only role (`kv_producer`) populates `hashes` from its
own stores but cannot load them back, so it voted a tier hit it would then
refuse: the right-to-left resumable scan stopped at the tier rung, skipped
a still-resident HBM rung, and `request_load` returned False -- the
boundary disowned and the HBM checkpoint forfeited to a full recompute.

Factor the membership+capability predicate onto the index as
`StateOffloadIndex.could_serve(h) = can_load and h in hashes` (the guard
of `request_load` minus its per-request in-flight check) and route both
voters and `request_load` through it, so all three test exactly the same
capability and cannot drift -- fix-then-sweep across the three read sites.
Test doubles for the index gain the mirror method.
Copilot AI review requested due to automatic review settings September 3, 2026 01:29
@zejunchen-zejun

Copy link
Copy Markdown
Collaborator Author

Thanks for the round-6 pass — every finding was actionable. Fixed each as its own commit on top of the review head (a19828e4), one commit per finding:

# Finding Commit
1 Joint loads armed from metadata.requests skip state-only loads → hang in WAITING_FOR_REMOTE_KVS 96b881b4
2 gdn_attn.state_entry_views guards only a negative slot; a too-large slot corrupts silently 4d18af02
3 StateByteCodec.store handed_off flag strands a ref on a pre-adoption batched_put raise fa2facbb
4 Three joint-park tests nested inside another test body, never collected 0513b720
5 PPEngineCoreProc override drops the _reconcile_stalled_deferred_saves call → PP save-stall wedge 930252aa
6 h in offload.hashes at three sites drops the can_load half of the load predicate cb179e2e

Detail per finding below.


1 — Arm joint loads from state ids, not metadata.requests (96b881b4)

You were right that the two sets have independent sources. state_loads arrives via BlockManager.take_state_loads() → Scheduler._publish_state_loads, while metadata.requests is populated by the base KV scheduler. A state-only load — KV resident, recurrent state not, the _park_for_remote_load path taken when needs_remote_load is False — carries no LMCacheReqMeta that step, so it is absent from metadata.requests. Iterating that set skipped exactly the state-only requests the else branch exists to arm; unarmed, their failure emitted nothing and they hung in WAITING_FOR_REMOTE_KVS forever (the orphan reclaimer never wakes a live parked request).

Now the loop is driven off state_ids, looking the KV leg up per id via a by_id = {req.req_id: req for req in metadata.requests} map. Both-legs still file under the KV identity; the state-only leg arms on the state channel alone.

This is exactly the path the three tests in 0513b720 cover (test_with_no_tier_the_joint_load_fails_for_recompute drives the state-only failure end to end), which is why I collected them first.

2 — Bound the state slot in both directions in gdn_attn (4d18af02)

Matched to the DSV4 twin as you asked. deepseek_v4_attn.py indexes a Python list (self._slot_views()[slot]), so it raises IndexError on a stray -1 or an out-of-range positive. Here cache[layer, slot : slot + 1] clamps silently instead: a -1 (the "no slot" sentinel) gathers/scatters the last slot under this request's identity, and a too-large slot yields a zero-element view (numel() == 0, no error) that the staging packer moves as 0 bytes — StateByteCodec.get then reports success on state never written and the request resumes on whatever the slot already held. Now rejects not 0 <= slot < num_slots (num_slots from mamba_k_cache.shape[1]). It guards the shape invariant rather than a demonstrated path — StateSlotPool.pop never returns negative and _attach_state_slots pops before requesting the load — so it should never fire; if it does, it fails loudly.

3 — Down the state ref on any store exception (fa2facbb)

Confirmed against the vendored LMCache storage_manager.py: batched_put discharges the reference in its terminal tail loop (one ref_count_down() per object), which runs after every step that can raise — the scheduler-role guard, get_allocator_backend, allocate_and_copy_objects, batched_submit_put_task. So there is no "adopted, then a later step raises" window that the handed_off flag was trying to protect. Setting handed_off = True before the call only suppressed the down on a pre-adoption batched_put raise, stranding the allocation at ref_count=1 (the later "garbage collected with ref_count=1, pin_count=0") and shrinking the CPU pool one entry per failure. Dropped the flag: on any exception we still own the reference and down it exactly once, matching the finally guard in get.

(For reference: the sibling _start_state_loads you flagged nearby already iterates loads directly, so it did not have the metadata.requests problem from #1 — no change needed there.)

4 — Collect the three joint-park tests that never ran (0513b720)

Good catch — they were nested inside the body of test_worker_close_is_a_noop_when_no_tier_was_built, so pytest never collected them; they were dead assertions. Dedented all three to module level and dropped the unused self params. They pass, and they are the acceptance gate for the no-tier / state-only joint-load paths (they gate #1).

5 — Reclaim stalled deferred saves in the PP engine core (930252aa)

PPEngineCoreProc._poll_kv_transfer_progress fully overrides the base EngineCore method, and the base version's getattr-guarded Scheduler._reconcile_stalled_deferred_saves call is that reclaimer's only caller repo-wide — so under pp_size > 1 it was silently dropped. A save whose completion never came (worker crash, dropped completion, force-unpin) then stayed in deferred_free_blocks forever, has_pending_kv_work() stayed True, and the engine busy-looped with every GPU idle (and _drain_kv_work_at_exit spun to timeout on shutdown). Mirrored the reclaimer call in the override, above the has_offload/pp_messages early-returns so a quiet poll still reclaims. It is self-throttled, so per-poll cost is negligible.

6 — Gate tier-hit votes on could_serve, not bare hashes (cb179e2e)

The two admission-path voters — PagedStateCheckpointCoordinator._reachable and BlockManager._tier_can_serve — tested h in offload.hashes, dropping the can_load half that request_load itself enforces. A store-only role (kv_producer) populates hashes from its own stores but cannot load them back, so it voted a tier hit it would then refuse: the right-to-left resumable scan stopped at the tier rung, skipped a still-resident HBM rung, and request_load returned False — boundary disowned, HBM checkpoint forfeited to a full recompute. Factored the predicate onto the index as StateOffloadIndex.could_serve(h) = can_load and h in hashes (the guard of request_load minus its per-request in-flight check) and routed both voters and request_load through it, so all three test the same capability and cannot drift — fix-then-sweep across the three read sites. The index test doubles gained the mirror method.


Verification. black --check clean and ruff check clean on all changed files. Affected unit suites green: test_lmcache_offload_connector.py, test_block_manager.py, test_scheduler.py, test_page_unit_checkpoint.py, test_state_checkpoint.py — 654 passed. (One block_manager case, TestJointChunkProbeIsGated::test_a_hosted_tier_still_probes_..., asserts _joint_chunk_tokens == 0 on the premise that lmcache is not importable; it passes in CI but fails in a container that bundles lmcache, where the probe succeeds and returns a real chunk size. No fix code touches that path.)

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

It introduces a new offload layout and cross-cutting scheduler/worker/lifecycle changes where subtle correctness regressions (silent wrong output or hangs) require careful human validation beyond automated review.

Review details

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

tests/test_multi_connector.py:64

  • FakeSchedSub defaults has_state_tier to True whenever offload_methods=True, but the real LMCacheOffloadConnectorScheduler.has_state_tier is only True for the kimi_k3 layout (dense/hybrid offload shells should report False). This makes the test double behave unlike production and can mask routing bugs where state calls should be rejected when no tier is present.

Consider defaulting has_state_tier to False (while still defining the attribute) and having individual tests opt-in with has_state_tier=True only when they explicitly model the K3 state tier.

  • Files reviewed: 51/51 changed files
  • Comments generated: 0 new
  • Review effort level: Lite

Brings in GLM-5.3-Flash (#2051), fuse prefill dcp (#2113), DSV4 PCP+MTP
fix (#2049). Two conflicts resolved: the page-unit geometry refactor
(PageUnitGeometryMixin) now carries main's sparse-indexer index-cache
region support; kept main's new kpool-tail checkpoint tests.
Copilot AI review requested due to automatic review settings September 3, 2026 03:52

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

It introduces a new cross-process joint KV+state offload path with broad engine/connector lifecycle changes where subtle correctness or liveness regressions are hard to rule out automatically.

Review details
  • Files reviewed: 51/51 changed files
  • Comments generated: 1
  • Review effort level: Lite

self._state_source_released = set()
return released

def take_state_reports(self) -> tuple[set[int], set[int]]:
@valarLip

valarLip commented Sep 3, 2026

Copy link
Copy Markdown
Collaborator

Sixth pass, re-reviewed from scratch at 84ed7d91a.

Round 5: seven of eight fixed, and fixed well. Not repeated below — _offload_subconfig now raises at startup instead of degrading (finding 0); _load_owner routes the three load-side questions to the sub that armed the load (finding 1); StateByteCodec.put's discharge is answered with an argument rather than a code change (finding 2, see the one note below); the orphan-slot pair became a single (slot, stamp) tuple and _state_loads is now filtered in deallocate (finding 4); release_offload_store_source / settle_offload_store is a real two-phase pin (finding 5); process_completions' partition problem is made unrepresentable by finding 0's raise (finding 6); and select_offload_layout resolves off get_hf_text_config (finding 7).


1. A multimodal prompt that spans a checkpoint rung livelocks the scheduler

scheduler.py:1454:

if atomic_prefill and chunk < seq.num_tokens - seq.num_cached_tokens:
    self.block_manager.deallocate(seq)
    self.waiting.appendleft(seq)
    break

The re-assert is correct in intent, but it differs from the pre-allocation guard in the one way that matters:

compares against driven by
:1342 (pre-alloc) num_new_tokens — the budget-limited size the budget, which changes as other requests drain → makes progress
:1454 (post-alloc) seq.num_tokens - seq.num_cached_tokens — the whole remaining prompt the checkpoint rung

And the adjuster that shortened the chunk is deterministic. _finalize_prefill_chunk (:2005):

target = bm.checkpoint_cut(seq, start, start + chunk)
if target:
    chunk = target - start

checkpoint_cut (block_manager.py:2023-2075) computes the earliest rung in (start, end] from end, checkpoint_limit(seq), state_checkpoint_interval_tokens, seq.checkpoint_demand_pos and seq.checkpoint_end_pos. None of those is a function of pool pressure or of how many times this request has been retried. Its only short-circuit is if all(c.readable_midstep for c in applicable): return 0, and the PAGE class declares readable_midstep = False (page_unit_checkpoint.py:740), so on K3 it does not fire.

So for KimiK3ForConditionalGeneration (kimi_k3.py:1917), which sets multimodal_data and therefore atomic_prefill (scheduler.py:1331): any image prompt longer than one checkpoint interval produces the same shortened chunk on every pass. deallocate → waiting.appendleft(seq) → break, forever. 0 tok/s with idle GPUs — and because it is appendleft plus break, every request behind it is head-of-line blocked too.

The pre-alloc guard at :1342 does not have this property: it requeues without having allocated, and its input moves.

One correction to how this was reported to me: the two adjusters are not equivalent here. _adjust_prefill_chunk_after_alloc (deferring part of a prefill for an offload load) is transient — once the load lands it stops shortening, so that path self-resolves. The livelock is specific to the checkpoint-rung path.

The new test cannot see it. test_multimodal_prefill_shortened_after_alloc_is_requeued_whole drives exactly one scheduling pass and asserts the requeue. A second pass over the same waiting queue is the whole finding.

Options, roughly in order of how much they preserve the invariant: suppress the rung cut when atomic_prefill (the chunk is whole by construction, so there is nothing to land on a grid); or admit whole and let checkpointers_at refuse the off-grid position; or, if neither is acceptable, at minimum detect the no-progress case rather than spinning.

2. The offload_layout override bypasses the new GDN refusal

offload/config.py:100-108. The override is resolved and returned before any model inspection:

mapped = _OFFLOAD_LAYOUT_ALIASES.get(override) if isinstance(override, str) else None
if mapped is not None:
    return mapped
raise ValueError(f"lmcache_offload: unknown offload_layout={override!r}; ...")

hf_config = getattr(config, "hf_config", None)   # <- everything below is unreachable with an override
...
if model_type in _GDN_LINEAR_MODEL_TYPES:
    raise ValueError(...)

So kv_transfer_config: {offload_layout: dense} on Qwen3-Next or a kimi_linear checkpoint returns "dense" without the refusal ever running — producing exactly the failure the refusal was written for: a KV prefix restored over stale recurrent state, silent wrong output.

The refusal is about a model capability, and the override is about which layout serves it; the former should gate the latter, not the other way round. Checking the model first and then applying the override (or refusing the combination explicitly) is a few lines.

3. _arm_joint_loads sits outside the try the invariant is stated over

kimi_k3/connector.py:274-279:

self._arm_joint_loads(metadata)
try:
    self._start_state_loads(metadata)
    self._start_state_stores(metadata)
finally:
    super().start_load_kv(metadata)

The comment above it says "each submit is isolated inside its helper ... and super() runs in a finally so a KV load or save is never dropped". That holds for two of the three helpers. _arm_joint_loads builds dicts and calls _joint_park.arm, so it does not submit and the window is narrow — but the stated invariant is broader than what the code guarantees, and this is precisely the ordering that a previous regression (recorded in the __init__ comment) came from. Moving the call inside the try costs one line and makes the comment true.


Still open from round 5

_JointPark has no abort, expiry or eviction path (state_tier.py: arm :325, settle_kv :414, settle_state :417, _release :442, take_ready :474 — and nothing else). A state load whose tier report is lost leaves _need / _alias / _alias_of populated for the process lifetime, _settle_joint swallows the KV leg's completion on every later get_finished, and the request sits in WAITING_FOR_REMOTE_KVS with its blocks held.

This is now load-bearing for a second reason. block_manager.py:2481's comment justifies holding an in-flight load's slot with:

Held until settle_state_load, which always comes: the worker reports every load either way.

The missing abort path is exactly what makes that premise false. Every other lost-report channel in this PR got a reclaimer (_reconcile_stalled_deferred_saves, reclaim_stale_state_store_pins, reconcile_orphan_load_slots); the park did not.

The orphan-slot park still overwrites. The two-dict desync is fixed — it is one (slot, monotonic()) tuple now — but block_manager.py:2484 is still a plain assignment:

self._orphan_load_slots[seq.id] = (seq.state_slot, monotonic())

seq.id is per-request, not per-admission, and abandon_load(seq.id) on the same path clears pending_loads, so the same id can be granted a second load after a preempt/re-admit. A second park under that id silently drops slot A, which is then off the StateSlotPool free list for the process lifetime and invisible to reconcile_orphan_load_slots.

One note on the finding-2 answer

The double-discharge concern was answered with a comment rather than a change — that batched_put's ref_count_down() runs in its terminal tail loop, after every step that can raise, so there is no window where the object is adopted and a later step throws. The argument is specific and names the LMCache source, and I cannot refute it from here.

Worth recording that it now rests on an LMCache implementation detail with nothing pinning it: a future release that moves the discharge earlier (or adds a step after it) reintroduces a silent wrong-bytes path, and this repo would not notice. A version floor, or an assertion on the object's refcount after a caught exception, converts the argument into something that fails loudly if it stops being true.

Reported to me but not independently verified

Listing these so they are not lost, explicitly marked: the state-store cap and the KV-save cap share a value but are enforced by two independent counters (scheduler.py:2130), so a slow backend could pin 2N; the assert seq is not None removed from the producer finished_sending loop (scheduler.py:3118) is justified by a reclaimer that cannot reach that channel; take_state_stores bumps stores_attempted at hand-out so a refusal is counted twice and the documented attempted - completed - failed in-flight derivation is permanently inflated (block_manager.py:1349); and OFFLOAD_MAX_PENDING_SAVES has two readers with different defaults and different malformed-value behaviour (scheduler.py:62 vs _offload_common.py:617), while docs/environment_variables.md:199 documents only one.

Already fixed — please ignore if re-raised

close() on the worker shell and on MultiConnector, save_abandon_timeout_s, release_stalled_save and get_statistics are all present (offload/connector.py:118; multi_connector.py:396, :751, :772, :788). They were the round-4 findings and landed before round 5. Two consecutive review passes have re-reported them from a stale checkout.

zejchen and others added 4 commits September 3, 2026 08:30
…rantee holds

Finding 3: start_load_kv() called self._arm_joint_loads(metadata) OUTSIDE the
try/finally that guarantees super().start_load_kv() runs. _arm_joint_loads builds
park state and can raise; if it did, the finally-super() KV-leg fallback was
skipped and the KV load was silently dropped. Move the arm inside the try next to
the two state helpers so every state-side helper is covered by the finally.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… state-owning model

Finding 2: select_offload_layout() mapped an explicit offload_layout override and
returned it BEFORE inspecting the model, so an override could (a) bypass the GDN
refusal and (b) downgrade a kimi_linear (kimi_k3) checkpoint to a layout that owns
no tier for its per-request KDA recurrent state -> a KV prefix restored over stale
state = silent wrong output.

Fix: always compute the natural layout via the extracted _layout_from_model()
first (which still refuses GDN), then, if an override is present, refuse mapping a
state-owning natural layout (_STATE_OWNING_LAYOUTS = {"kimi_k3"}) down to a
non-state layout. dense<->hybrid remains a legitimate operator choice (namespace
separation), so the guard is deliberately narrow.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…g a checkpoint-rung cut

Finding 1: _finalize_prefill_chunk() always aligned the prefill chunk to a
state-checkpoint rung via checkpoint_cut(). For a multimodal prompt (taken as one
atomic unit) whose length is off the rung grid, the deterministic cut kept shaving
the chunk to the same sub-rung length every pass -> the request could never make
prefill progress (livelock).

Fix: when seq.multimodal_data is present, return the whole chunk before
checkpoint_cut(). A checkpoint is only ever kept where a forward ends on a rung, so
an off-grid whole prompt simply keeps no mid-prompt checkpoint; forks do not arise
on a fresh multimodal request, so no correctness is lost.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… re-admission cannot drop a slot

Still-open 2: _orphan_load_slots mapped req id -> a single (slot, ts) tuple. If a
request was preempted and re-admitted while a state load was still in flight, the
second park overwrote the first entry and leaked the first slot; settle/abandon
also released the wrong (only) slot.

Fix: store a list of (slot, ts) per id, append on each park, and release FIFO via
_release_oldest_orphan_slot() (matching worker report order). settle_state_load and
abandon_state_load release the oldest; reconcile_orphan_load_slots expires stale
entries per-slot by age while keeping fresh ones. The entry is deleted when its
list empties.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Copilot AI review requested due to automatic review settings September 3, 2026 08:38
@zejunchen-zejun

zejunchen-zejun commented Sep 3, 2026 •

Copy link
Copy Markdown
Collaborator Author

Thanks for the round-6 pass — the three new findings were all real. Fixed one commit per finding on the merged tip (84ed7d91), plus the still-open orphan-slot leak. Summary first, details and anchors below.

# Finding Status Commit
1 Multimodal prompt spanning a checkpoint rung livelocks the scheduler fixed 92dcef82
2 offload_layout override bypasses GDN refusal / downgrades a state-owning model (silent wrong output) fixed 15ead445
3 _arm_joint_loads sits outside the try (KV-leg guarantee can be skipped) fixed 576a0b64
Still-open 1 _JointPark has no abort/expiry path already addressed at this tip — anchors below, no change (ca4708a0)
Still-open 2 orphan-slot park overwrites per-request (leaks slot A) fixed adb1e346

Finding 1 — multimodal prefill livelock — 92dcef82

Confirmed. _finalize_prefill_chunk always aligned the chunk to a state-checkpoint rung via checkpoint_cut(seq, start, start+chunk). A multimodal prompt is taken as one atomic unit (atomic_prefill); when its length is off the rung grid, the deterministic cut shaves the chunk to the same sub-rung length every pass, so the request never makes prefill progress — 0 tok/s with idle GPUs and head-of-line blocking, exactly as you described.

Fix = option (a): when seq.multimodal_data is not None, return the whole chunk before checkpoint_cut. A checkpoint is only ever kept where a forward ends on a rung, so an off-grid whole prompt simply keeps no mid-prompt checkpoint; forks don't arise on a fresh multimodal request, so nothing is lost on correctness.

You were right that the existing test drove only one pass and couldn't catch this. Added test_multimodal_prefill_spanning_checkpoint_rung_admits_whole which monkeypatches checkpoint_cut to a fixed sub-rung length and drives two passes, asserting the 8-token multimodal seq is admitted whole (num_scheduled_tokens == [8], not requeued) rather than re-shortened on the second pass. The pre-existing one-pass test is left intact.

Finding 2 — offload_layout override bypasses GDN refusal — 15ead445

Confirmed, and it was actually two holes in the same early-return: an override returned mapped before the model-type inspection, so (a) offload_layout: dense on a GDN checkpoint (Qwen3-Next / Qwen3.5) escaped the GDN refusal, and (b) the same path could downgrade a kimi_linear model to a layout that owns no tier for its per-request KDA recurrent state — a KV prefix restored over stale state = silent wrong output.

Fix: select_offload_layout now computes the natural layout via an extracted _layout_from_model(config) first (which still raises for GDN), then, only if an override is present, refuses mapping a state-owning natural layout down to a non-state layout:

natural = _layout_from_model(config)          # raises on GDN, regardless of override
if override is None:
    return natural
mapped = _map_override(override)               # raises "unknown offload_layout" on typo
if natural in _STATE_OWNING_LAYOUTS and mapped not in _STATE_OWNING_LAYOUTS:
    raise ValueError(... "owns no tier for that state ... drop the override, or disable offload")
return mapped

_STATE_OWNING_LAYOUTS = frozenset({"kimi_k3"}) deliberately, so the guard stays narrow: dense <-> hybrid remains a legitimate operator choice (namespace separation), which the existing test_page_namespace_separates_explicit_offload_layout_families relies on. Added three tests: GDN-refusal-not-bypassed-by-override (parametrized over the four GDN types), kimi_k3-cannot-be-downgraded-to-dense, and dense/hybrid-override-still-allowed.

Finding 3 — _arm_joint_loads outside the try — 576a0b64

Confirmed, one-line fix as suggested. _arm_joint_loads builds park state and can raise; it sat outside the try/finally that guarantees super().start_load_kv(), so a raise there would silently drop the KV leg. Moved it inside the try next to the two state helpers:

try:
    self._arm_joint_loads(metadata)
    self._start_state_loads(metadata)
    self._start_state_stores(metadata)
finally:
    super().start_load_kv(metadata)

Still-open 1 — _JointPark abort/expiry — already addressed at this tip

This one is already in the branch (commit ca4708a0, ancestor of both the round-6 tip and the merge), so no new change — flagging the anchors in case it was read from a stale checkout:

  • _JointPark.reclaim_stale_parks(max_age_s) -> int — state_tier.py:385, backed by a per-armed-key monotonic stamp at state_tier.py:316.
  • Wired via _reclaim_stale_parks() at connector.py:551, called from the get_finished path at connector.py:448, with the eviction window computed at connector.py:579.

This gives the joint park the same lost-report reclaimer parity as reconcile_orphan_load_slots / reclaim_stale_state_store_pins / _reconcile_stalled_deferred_saves. Happy to fold in an explicit max-age knob if you'd prefer it configurable.

Still-open 2 — orphan-slot per-request overwrite — adb1e346

Confirmed. _orphan_load_slots mapped seq.id -> (slot, ts), so a preempt+re-admit while a state load was still in flight overwrote the first entry and leaked slot A; settle/abandon then released the wrong (only) slot.

Fix = your "or append" alternative: _orphan_load_slots is now dict[id, list[(slot, ts)]], appended per park. Release is FIFO via a new _release_oldest_orphan_slot(req_id) (matching worker report order — settle_state_load only carries req_id, so FIFO is the right discipline without threading the slot through). abandon_state_load and settle_state_load both release the oldest; reconcile_orphan_load_slots now expires stale entries per-slot by age while keeping fresh ones; the id entry is deleted when its list empties. Added test_a_preempt_readmit_parks_the_same_id_twice_without_dropping_a_slot and test_reconcile_expires_one_admission_and_keeps_the_fresh_one; updated the five existing tests to the list shape.


On the double-discharge (Finding-2-answer rebuttal)

Fair point that the rebuttal rests on an unpinned LMCache internal (batched_put / ref_count_down() / StateByteCodec.put). I can add either a version floor on the LMCache dependency or a post-exception refcount assertion so a future LMCache change can't silently reintroduce the double-discharge — let me know which you'd prefer and I'll add it as a follow-up commit rather than bundling it here.

Reported-but-not-independently-verified

Still standing at the merge tip; anchors for your re-check: independent store/save counters scheduler.py:2130; removed assert seq is not None in the producer finished_sending loop scheduler.py:3118; take_state_stores attempted-completed-failed derivation block_manager.py:1349; OFFLOAD_MAX_PENDING_SAVES readers/defaults scheduler.py:62 / _offload_common.py:617 / docs/environment_variables.md:199.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

A couple of incorrect public type annotations for state-tier store reports (reported as int rather than StateStoreOperationId) should be corrected to avoid misuse and tooling confusion in a correctness-critical area.

Review details

Suppressed comments (2)

Previously missed (1) — in code that hasn't changed since the last review.

atom/kv_transfer/offload/_offload_common.py:293

  • StateOffloadFace.take_state_reports() is annotated as returning tuple[set[int], set[int]], but the scheduler consumes these as StateStoreOperationId objects (passed to BlockManager.settle_state_store(op, ...)). The current annotation is misleading and can cause incorrect assumptions in type-aware tooling.

atom/kv_transfer/offload/hybrid/kimi_k3/connector.py:898

  • take_state_reports() drains STATE_INDEX_CHANNEL completions whose operation_id is a StateStoreOperationId, not an int hash. The current return type tuple[set[int], set[int]] is incorrect and conflicts with how the scheduler calls BlockManager.settle_state_store(op, ...).
    def take_state_reports(self) -> tuple[set[int], set[int]]:
        """Drain this step's tier store reports for the engine-side index.

        Both real tier failures and no-tier worker failures land in
        `_state_index_failed` via STATE_INDEX_CHANNEL, so there is a single
        source of truth to drain.
        """
        indexed = self._state_indexed
        failed = self._state_index_failed
        self._state_indexed = set()
        self._state_index_failed = set()
        return indexed, failed
  • Files reviewed: 51/51 changed files
  • Comments generated: 0 new
  • Review effort level: Lite

@valarLip
valarLip merged commit e4e473f into main Sep 3, 2026
35 of 53 checks passed
@valarLip
valarLip deleted the zejun/lmcache_with_2045 branch September 3, 2026 09:26
zejunchen-zejun added a commit that referenced this pull request Sep 3, 2026
Reviewing what PR #2053's six rounds left behind turned up a gap in this branch
rather than in that one.

The shell-forwarding sweep pins a `must_be_seen` set so that a later refactor
hiding one of the scheduler's `getattr(self.kv_connector, ...)` reads shrinks
`probed` loudly instead of silently. Fusing the load leg removed
`enqueue_state_loads` from that surface but ADDED `take_missed_state_hashes`
(scheduler.py:3070) -- so the guarded surface did not shrink, and the one member
whose read is new was the one member not anchored. The regex matches it today;
nothing was pinning that it keeps doing so.

Also drops a comment naming `handed_off`, a flag removed before this branch
began. The reasoning it carried is the non-obvious half -- `batched_put`
discharges the caller's reference in its terminal tail loop, so there is no
adopted-then-raised window -- so it stays, rewritten as the constraint it is
("do not reintroduce a flag that skips the down") rather than as a story about
a symbol no longer in the tree.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
zejunchen-zejun added a commit that referenced this pull request Sep 3, 2026
Answering a fair challenge: if the debt was architecture-induced, why is none of
it redeemable? Auditing what #2053 changed in PRE-EXISTING code -- rather than
what its fixes added -- found one keep of mine that does not survive.

`MultiConnector._state_tier` was written twice and read never. The two-tier
refusal reads `_state_tier` off the SUBS; the four state forwarders route
through `_state_tier_sub()`, a different method that also probes the subs. The
comment justified the mirror as making a probe resolve, but no such probe
exists: `start_load_kv` fans each sub its own metadata, so the K3 sub uses its
own field. Its two tests asserted the mirror against a rationale -- "the K3
store/load path reads `_state_tier` off whatever connector the forward context
holds; under `multi` that is this object" -- which is simply not what the code
does.

The method's real and only product is the refusal, so it is renamed to say so.
The refusal, its ValueError and its test are untouched: a config listing two
offload subs would leave a hash reported indexed by one tier and fetched from
another, and that is worth a loud failure at model load.

Also drops `lmc-state-load` from a docstring listing the threads `close` must
join -- that executor went with the fusion.

4252 passed (4254 - 2 tests whose subject no longer exists).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
zejunchen-zejun added a commit that referenced this pull request Sep 3, 2026
Sweeping #2053's ~4,200 added test lines for tests whose subject the fusion
removed found none -- but it found the opposite: a test that looked like
coverage and was not.

`test_success_and_failure_release_identically` was `for _ in range(2):` around
five identical statements, both iterations taking the success path. It could not
have tested a failure: `PageUnitCheckpointStore.settle_offload_store` takes no
outcome argument, so at that level there is no distinction to make. Renamed to
what it actually pins, and it now points at where the distinction really lives.

That distinction is `BlockManager.settle_state_store(op, ok=..., attempted=...)`,
whose docstring states it precisely -- and `ok=False` was called by NO test in
the repo. `grep "ok=False\|attempted=False" tests/` returned nothing; all three
call sites passed `ok=True`. So the failure legs of a state store were untested:
that a failed store still releases its units but must NOT index its hash (voting
for a hash whose bytes never landed sends the next request over that prefix to a
`get` that must miss), and that a REFUSED store is not additionally counted as
failed and backs out its optimistic `stores_attempted` bump.

Both now have tests. Also renames `TestStateLoadsAndStoresRunInSeparateLanes`:
there are no lanes since the fusion made the load synchronous on the KV task's
own thread, and the class docstring now says the property is structural rather
than scheduled.

4254 passed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
zejunchen-zejun added a commit that referenced this pull request Sep 14, 2026
Reviewing what PR #2053's six rounds left behind turned up a gap in this branch
rather than in that one.

The shell-forwarding sweep pins a `must_be_seen` set so that a later refactor
hiding one of the scheduler's `getattr(self.kv_connector, ...)` reads shrinks
`probed` loudly instead of silently. Fusing the load leg removed
`enqueue_state_loads` from that surface but ADDED `take_missed_state_hashes`
(scheduler.py:3070) -- so the guarded surface did not shrink, and the one member
whose read is new was the one member not anchored. The regex matches it today;
nothing was pinning that it keeps doing so.

Also drops a comment naming `handed_off`, a flag removed before this branch
began. The reasoning it carried is the non-obvious half -- `batched_put`
discharges the caller's reference in its terminal tail loop, so there is no
adopted-then-raised window -- so it stays, rewritten as the constraint it is
("do not reintroduce a flag that skips the down") rather than as a story about
a symbol no longer in the tree.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
zejunchen-zejun added a commit that referenced this pull request Sep 14, 2026
Answering a fair challenge: if the debt was architecture-induced, why is none of
it redeemable? Auditing what #2053 changed in PRE-EXISTING code -- rather than
what its fixes added -- found one keep of mine that does not survive.

`MultiConnector._state_tier` was written twice and read never. The two-tier
refusal reads `_state_tier` off the SUBS; the four state forwarders route
through `_state_tier_sub()`, a different method that also probes the subs. The
comment justified the mirror as making a probe resolve, but no such probe
exists: `start_load_kv` fans each sub its own metadata, so the K3 sub uses its
own field. Its two tests asserted the mirror against a rationale -- "the K3
store/load path reads `_state_tier` off whatever connector the forward context
holds; under `multi` that is this object" -- which is simply not what the code
does.

The method's real and only product is the refusal, so it is renamed to say so.
The refusal, its ValueError and its test are untouched: a config listing two
offload subs would leave a hash reported indexed by one tier and fetched from
another, and that is worth a loud failure at model load.

Also drops `lmc-state-load` from a docstring listing the threads `close` must
join -- that executor went with the fusion.

4252 passed (4254 - 2 tests whose subject no longer exists).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
zejunchen-zejun added a commit that referenced this pull request Sep 14, 2026
Sweeping #2053's ~4,200 added test lines for tests whose subject the fusion
removed found none -- but it found the opposite: a test that looked like
coverage and was not.

`test_success_and_failure_release_identically` was `for _ in range(2):` around
five identical statements, both iterations taking the success path. It could not
have tested a failure: `PageUnitCheckpointStore.settle_offload_store` takes no
outcome argument, so at that level there is no distinction to make. Renamed to
what it actually pins, and it now points at where the distinction really lives.

That distinction is `BlockManager.settle_state_store(op, ok=..., attempted=...)`,
whose docstring states it precisely -- and `ok=False` was called by NO test in
the repo. `grep "ok=False\|attempted=False" tests/` returned nothing; all three
call sites passed `ok=True`. So the failure legs of a state store were untested:
that a failed store still releases its units but must NOT index its hash (voting
for a hash whose bytes never landed sends the next request over that prefix to a
`get` that must miss), and that a REFUSED store is not additionally counted as
failed and backs out its optimistic `stores_attempted` bump.

Both now have tests. Also renames `TestStateLoadsAndStoresRunInSeparateLanes`:
there are no lanes since the fusion made the load synchronous on the KV task's
own thread, and the class docstring now says the property is structural rather
than scheduled.

4254 passed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
zejunchen-zejun added a commit that referenced this pull request Sep 15, 2026
Reviewing what PR #2053's six rounds left behind turned up a gap in this branch
rather than in that one.

The shell-forwarding sweep pins a `must_be_seen` set so that a later refactor
hiding one of the scheduler's `getattr(self.kv_connector, ...)` reads shrinks
`probed` loudly instead of silently. Fusing the load leg removed
`enqueue_state_loads` from that surface but ADDED `take_missed_state_hashes`
(scheduler.py:3070) -- so the guarded surface did not shrink, and the one member
whose read is new was the one member not anchored. The regex matches it today;
nothing was pinning that it keeps doing so.

Also drops a comment naming `handed_off`, a flag removed before this branch
began. The reasoning it carried is the non-obvious half -- `batched_put`
discharges the caller's reference in its terminal tail loop, so there is no
adopted-then-raised window -- so it stays, rewritten as the constraint it is
("do not reintroduce a flag that skips the down") rather than as a story about
a symbol no longer in the tree.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants