Repository navigation
[Kimi-K3][LMCache] LMCache offload for Kimi-K3 - #1960
Closed
zejunchen-zejun wants to merge 7 commits into
Closed
zejunchen-zejun wants to merge 7 commits into
zejunchen-zejun wants to merge 7 commits into
Conversation
Contributor
🏷️ CI GuideRuns automatically on every eligible PR before approval:
Heavy model tests:
|
zejunchen-zejun
force-pushed
the
zejun/lmcache
branch
2 times, most recently
from
August 20, 2026 03:54
6a33858 to
b0f98df
Compare
Comment on lines
+11
to
+35
| from __future__ import annotations | ||
|
|
||
| import logging | ||
| import os | ||
| import time | ||
|
|
||
| from atom.kv_transfer.disaggregation.types import ConnectorCompletion | ||
| from atom.kv_transfer.offload import config as offcfg | ||
| from atom.kv_transfer.offload._offload_common import ( | ||
| max_pending_saves, | ||
| pp_aware_rank_and_world, | ||
| ) | ||
| from atom.kv_transfer.offload.dense.connector import ( | ||
| DenseOffloadConnector, | ||
| DenseOffloadScheduler, | ||
| ) | ||
| from atom.kv_transfer.offload.hybrid.kimi_k3.staging import StagedTransfer | ||
| from atom.kv_transfer.offload.hybrid.kimi_k3.state_object import StateByteCodec | ||
| from atom.kv_transfer.offload.hybrid.kimi_k3.state_tier import ( | ||
| _JointPark, | ||
| StateOffloadTier, | ||
| ) | ||
| from atom.kv_transfer.offload.metadata import LMCacheOffloadMetadata | ||
|
|
||
| logger = logging.getLogger("atom") |
Contributor
There was a problem hiding this comment.
Import block is un-sorted or un-formatted
Suggested change
| from __future__ import annotations | |
| import logging | |
| import os | |
| import time | |
| from atom.kv_transfer.disaggregation.types import ConnectorCompletion | |
| from atom.kv_transfer.offload import config as offcfg | |
| from atom.kv_transfer.offload._offload_common import ( | |
| max_pending_saves, | |
| pp_aware_rank_and_world, | |
| ) | |
| from atom.kv_transfer.offload.dense.connector import ( | |
| DenseOffloadConnector, | |
| DenseOffloadScheduler, | |
| ) | |
| from atom.kv_transfer.offload.hybrid.kimi_k3.staging import StagedTransfer | |
| from atom.kv_transfer.offload.hybrid.kimi_k3.state_object import StateByteCodec | |
| from atom.kv_transfer.offload.hybrid.kimi_k3.state_tier import ( | |
| _JointPark, | |
| StateOffloadTier, | |
| ) | |
| from atom.kv_transfer.offload.metadata import LMCacheOffloadMetadata | |
| logger = logging.getLogger("atom") | |
| from __future__ import annotations | |
| import logging | |
| import os | |
| import time | |
| from atom.kv_transfer.disaggregation.types import ConnectorCompletion | |
| from atom.kv_transfer.offload import config as offcfg | |
| from atom.kv_transfer.offload._offload_common import ( | |
| max_pending_saves, | |
| pp_aware_rank_and_world, | |
| ) | |
| from atom.kv_transfer.offload.dense.connector import ( | |
| DenseOffloadConnector, | |
| DenseOffloadScheduler, | |
| ) | |
| from atom.kv_transfer.offload.hybrid.kimi_k3.staging import StagedTransfer | |
| from atom.kv_transfer.offload.hybrid.kimi_k3.state_object import StateByteCodec | |
| from atom.kv_transfer.offload.hybrid.kimi_k3.state_tier import ( | |
| StateOffloadTier, | |
| _JointPark, | |
| ) | |
| from atom.kv_transfer.offload.metadata import LMCacheOffloadMetadata | |
| logger = logging.getLogger("atom") |
Contributor
There was a problem hiding this comment.
Import block is un-sorted or un-formatted
ATOM/atom/kv_transfer/offload/hybrid/dsv4/connector.py
Lines 27 to 88 in b0f98df
zejunchen-zejun
force-pushed
the
zejun/lmcache
branch
3 times, most recently
from
August 26, 2026 07:52
6729761 to
bd8462c
Compare
Rebuilds the K3 work on main's restructured offload package instead of
merging the old branch: `connector.py` is now a dispatch shell selecting a
layout from config alone, so K3 becomes a third layout (`kimi_k3`) beside
`dense` and `hybrid`, sharing dense's KV path and adding only the KDA
per-request state tier.
atom/kv_transfer/offload/hybrid/kimi_k3/
connector.py KimiK3OffloadConnector (worker) / KimiK3OffloadScheduler
state_tier.py the tier and its joint park
state_object.py the entry codec
staging.py single-entry staged transfer
The scheduler half is six small overrides on `DenseOffloadScheduler` rather
than a fork of it:
* `_decide_load_after_alloc` clamps a hybrid's KV leg to the boundary the
state leg is aimed at, and refuses the load outright when there is none.
A hybrid's state is the compressed history of exactly `[0, hbm)`, so
raising the KV-loaded length past that has the forward skip tokens the
recurrence never saw -- wrong output, no exception.
* `should_defer_free` releases the blocks of a save that was never handed
out once the backend stops draining. Holding them is what turned a
stopped backend into a stopped engine.
* `_may_emit_save` bounds how many requests may have a save outstanding,
since each one pins its blocks. Dense gains the seam (unbounded there,
it does not pin); `max_pending_saves` moves to `_offload_common` so both
it and DSV4 read the same `OFFLOAD_MAX_PENDING_SAVES`.
* `build_connector_meta` drains the state-load queue onto the metadata.
* `connector_completion` routes the tier's two channels.
Reports ride main's generic `ConnectorCompletion` channels rather than extra
`KVConnectorOutput` fields, so TP quorum is the aggregator's existing job and
`multi_connector` needs only the tier re-exposure and `enqueue_state_loads`.
Env vars: four are gone. `OFFLOAD_STATE` and `OFFLOAD_KV_FOR_HYBRID` because
a resumable prefix needs both legs -- state is offloaded exactly when KV is,
and the connector being configured is the only switch. `OFFLOAD_SAVE_MAX_INFLIGHT`
and `OFFLOAD_SAVE_STALL_S` fold into `OFFLOAD_MAX_PENDING_SAVES` and a
constant. `OFFLOAD_STATE_MIN_LOAD_TOKENS` defaulted to 0 and a token floor is
the wrong unit for a flat-cost entry. The staging ring is gated on the hosting
connector (`state_pool(..., offload_hosted=)`) so a deployment without one
does not pay for a ring nothing can drain. `docs/environment_variables.md`
gains a full offload section covering main's existing vars as well.
`STATE_CKPT_EXTRA_ENTRIES` is deliberately NOT reintroduced: the old branch
predates #1894, which deleted it, and it is a state-pool sizing knob with no
bearing on offload. `state_pool` keeps `extra_entries` as the backend-declared
parameter main left it as.
Also removed as dead: `STATE_OFFLOAD_LOADS_WIRED` (permanently True),
`clamp_state_boundary` (no caller), `StateOffloadIndex.kv_offload_enabled`
(always True once the index is only built for a hosting connector), a
duplicate `_submit_state_spills` override, and ~150 lines of `staging.py`
that re-implemented `atom_lmcache_staging`. Its `_env_flag` fix -- strip, and
read empty as off, so `VAR=` does not turn a flag ON -- moves to the shared
copy.
Measured on c10 (Kimi-K3, TP8, 3600s): P90 interactivity 31.19 -> 41.03,
per-GPU throughput 4313 -> 4784 tok/s, combined hit rate 91.62% -> 93.21%.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…s request forever
`connector_metadata_has_work` decides whether a step's metadata reaches the
worker at all, and it decided from a hardcoded field list in
`disaggregation/types.py` shared by three connector families.
`MultiConnectorMetadata` holds none of the work itself -- everything lives in
`metas` -- so to satisfy that list it mirrors the subs' fields as aggregating
properties. It mirrors `requests` and `lookup_requests_in_step`, and misses
`state_loads`:
sub = LMCacheOffloadMetadata(); sub.state_loads = [("req-1", h, group)]
connector_metadata_has_work(sub) # True
connector_metadata_has_work(MultiConnectorMetadata([sub])) # False
So under `multi`, every step whose only work is a state load is discarded, and
the request parked on that load waits for a report nobody was asked to
produce. A state-only load is not a corner: it is what a K3 admission does
whenever the boundary's checkpoint is in the tier and the KV prefix needs no
transfer, which the joint counters put at the overwhelming majority.
Fixed by asking the metadata instead of inspecting it. Each class declares its
own `WORK_FIELDS` next to the fields it defines, `has_work()` accumulates down
the hierarchy, and `MultiConnectorMetadata` delegates to its subs rather than
mirroring their fields -- so a field added to a sub can no longer be missed
here. The old union list stays only as the fallback for duck-typed doubles,
marked as not the place to register anything new.
The failure mode this replaces is the reason it is worth the indirection: a
forgotten entry is invisible at review, silent at runtime, and surfaces hours
later as a request that never completes.
Also drops a dead `_env_flag` import in `kimi_k3/staging.py` (ruff).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…kept two copies
`can_allocate` returns one number and the request needs two. What it may call
*cached* is gated on a resumable state sitting behind it; what it may point its
block table at is every block the prefix walk matched. `_gated_hit` cuts the
first, and the second was following it down.
So with a 4096-token prefix resident in HBM, no checkpoint anywhere inside it,
and the boundary at 5120, the request claimed nothing, and the KV leg was aimed
at `[0, 5120)` -- 4096 tokens of which the pool was already holding. Two costs,
both silent:
* ~64 MB/rank/request of H2D that moved bytes the GPU had.
* A second copy of them in HBM. `publish_loaded_prefix` keeps the existing
canonical mapping, so the freshly written blocks are never indexed: they
stay private to the request and occupy the pool until it finishes.
Two things were wrong. `_joint_kv_boundary` never recorded how far the walk had
actually matched, so `allocate` had no way to claim past the gated hit; and the
K3 `_decide_load_after_alloc` never wrote `hbm_cached_tokens` back onto the load
spec, so the transfer started wherever the *lookup* had left it -- which is
before `can_allocate` runs, i.e. 0. The second one was costing the refetch even
when the hit was not gated at all.
Now `_joint_kv_boundary` records `state_joint_claim_tokens` (the matched extent,
floored to the chunk grid), `allocate` claims to it while still calling only the
gated hit cached, and the KV leg starts there. On the case above the leg moves
`[4096, 5120)` -- 1024 tokens instead of 5120, with no second copy.
Claiming past the hit is safe precisely because a joint boundary exists: those
blocks sit below the boundary the forward will start from, so nobody writes
them. Without one the claim stays at the hit, where widening it would pin blocks
the forward is about to overwrite.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…e boundary
`KimiK3OffloadScheduler._claim_after_load` was written and never called. All
four completion sites inlined `max(hbm, lmc)` instead, and for k3 that `lmc` is
the chunk *covering* the state boundary, not the boundary:
LMCACHE_CHUNK_SIZE=256, --block-size 128, boundary at 5120
covering chunk ends at 5376 -> the request claimed 5376
The forward then starts 256 tokens past what the recurrent state covers. The
KDA layers never saw that range, and nothing raises -- the output is just
wrong. Invisible today only because the recipe pins chunk == block_size, where
the covering chunk *is* the boundary.
Made it a real seam: `OffloadSchedulerMixin._claim_after_load` returns
`max(hbm, lmc)` -- correct for every layout whose claim and transfer share a
boundary, which is dense and dsv4 -- and k3's override clamps to its own. Four
call sites now go through it.
Scoped deliberately to the claim. `chunk != block_size` costs dense and dsv4
nothing but `unaligned_hbm_prefill` refusals, which fail closed, and it costs
k3 transfer overshoot into blocks the forward rewrites plus roughly half its
joint boundaries. All efficiency, no correctness, so there is nothing here to
refuse at startup.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Rebasing onto main brought #1989/#1990, which turned `Sequence.token_ids` and `Sequence.block_table` into `array("i")` for the GC pause it buys at long context. An `array("i")` never compares equal to a list, so the two assertions this branch added in `TestJointClaimReusesResidentBlocks` -- correct when written against a `list` block table -- started failing. Same fix main applied to its own test in #1990 (`assert list(seq.output_tokens) == [...]`): convert the array side. assert list(seq.block_table[:4]) == resident `seq.block_table = [0]` in `test_scheduler.py` is deliberately left alone: main has two of its own at lines 469 and 489 of that file, so a lone converted third would read as the odd one out, and it passes. The formatting half is `black .` over the nine source files and three tests this branch had left non-compliant; main is black-clean, so every one of them was ours. Three of the reformats join a wrapped expression onto one line and the rest are blank lines -- `git diff -w --ignore-blank-lines` is empty outside those three. Verified: `pytest tests/ --ignore=tests/plugin` -> 3780 passed, 192 skipped, 3 xfailed. The two remaining collection errors (msgpack, aiter) are on main as well. `tests/plugin` is excluded because `test_rtpllm_forward_context_semantics.py` and `test_rtpllm_glm5_sparse_backend_contract.py` replace `sys.modules["atom.utils.forward_context"]` without restoring it, which breaks every later test that builds a Scheduler -- 66 of main's own included. Pre-existing and not this branch's to fix. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…'t hang
The K3 state-offload tier assumes every dispatched offload op eventually
reports completion, but LMCache's pin_monitor force-unpins a stalled
transfer's memory after pin_timeout_sec WITHOUT emitting that report. Two
counters then never clear:
- a KV save's deferred blocks (deferred_free_blocks) are freed only on
finished_saving, so has_pending_kv_work() stays True forever and the
engine busy-loops with every GPU idle -- a hard hang under a tight HBM
pool, a slow block leak under a large one (reproduced: J4 util=0.84
hangs ~7min after READY, GPUs 0%, no preemption, no recovery);
- a state spill's staging-ring slot comes home only via release_staging
on the worker's report, so a lost report pins the slot until every
further spill drops (spills_dropped climbs, ring starves).
Add two self-throttled reconcilers that reclaim after an abandon window:
- Scheduler._reconcile_stalled_deferred_saves(), wired into the KV
progress poll, frees blocks whose save has been deferred past
OFFLOAD_SAVE_ABANDON_TIMEOUT_S (default 330s);
- StateOffloadIndex.reclaim_stale_slots(), wired into the state-report
drain, returns slots reserved longer than the same window and drops
their undrained pending entries.
The abandon window is deliberately larger than LMCache's pin_timeout_sec
(default 300s): by the time we reclaim, upstream has already force-unpinned
and released the source bytes, so returning blocks/slots cannot race an
in-flight copy. Tests shrink OFFLOAD_SAVE_ABANDON_TIMEOUT_S and
LMCACHE_EC_PIN_TIMEOUT_SEC together to preserve that ordering; <= 0
disables reclamation and restores the wait-forever behaviour.
Also harden the producer finished_sending path: a save reclaimed by the
reconciler leaves nothing to free, so a late completion report is a no-op
(continue) rather than an assertion failure.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ming slots a batch still owes
Follow-up on the reconcilers, three changes.
**The window is no longer an ATOM knob.** `OFFLOAD_SAVE_ABANDON_TIMEOUT_S`
defaulted to 330s to sit above LMCache's 300s `pin_timeout_sec`, but that
ordering IS the safety argument and nothing enforced it: two independent
env vars, and setting ours below theirs would reclaim blocks a copy was
still reading, silently. It now reads LMCache's own
`LMCACHE_EC_PIN_TIMEOUT_SEC` and adds a 30s margin, so raising the pin
timeout carries the window with it and there is one number to get right
instead of two. Unset gives 330s, exactly the previous default. Memoised,
because both reconcilers ask for it on a 1ms poll
(`KV_IDLE_DRAIN_INTERVAL_S`) and the answer is a process constant.
The docstring now states the argument that actually holds. The timing
claim on its own did not: our clock starts when the block free is
deferred, LMCache's when it pins, and nothing aligns them. What makes
reclamation safe is that `OffloadWorkerMixin._guard` reports
`finished_saving` on both the success and the exception path, so a report
is lost only when `store()` neither returns nor raises. Two cases follow
and they are exhaustive: the parked save is not copying (LMCache
force-unpins its source and stops reading), and a save queued behind a
parked one never reaches `store()` at all.
**`reclaim_stale_slots` no longer touches `_pending`.** It used to drop
the stale slots' queued entries and return the slots. But a slot in
`_pending` has not been drained by `BlockManager.state_spills_for_batch`,
and that method drains `take_pending()` and the pool's
`take_spill_copies()` in ONE call -- so such a slot still has a
`(group, slot)` copy queued on `StateGroupPool`. Returning it let a new
spill reserve the same slot, and the two copies then raced for the single
`{slot: hash}` entry the drain builds: the loser logs "a copy with no
pending hash" and calls `release_staging`, freeing the slot out from under
the winner. Skipping queued slots removes the race and the `_pending`
surgery with it -- an undrained spill means no batch has run, which is
what the save reconciler is busy fixing, and the slot becomes eligible on
the next scan.
**`slots_reclaimed` reaches `stats()`.** The counter's own comment said it
existed so the symptom was visible rather than a silent throughput cliff,
but it was never added to the dict `StateGroupPool.checkpoint_fates`
merges, so it only appeared in the one-shot warning.
Also cross-referenced the two stall mechanisms in both docstrings. The K3
connector's `SAVE_STALL_SECONDS` escape releases the blocks of a save the
backend never took; this reconciler covers the save it took and never
answered. Neither subsumes the other and the pair is what leaves no
deferred save without a way out.
Verified: pytest tests/ --ignore=tests/plugin -> 3780 passed, 192 skipped,
3 xfailed (unchanged); black clean; ruff 33, same as main.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
zejunchen-zejun
force-pushed
the
zejun/lmcache
branch
from
August 27, 2026 01:32
bd8462c to
6feaaeb
Compare
Comment on lines
+388
to
+389
| if self._save_stalled and sid not in self._save_inflight: | ||
| if self._has_pending_save(seq): |
Contributor
Collaborator
Author
|
need to redesign based on PR: #2045 |
Collaborator
Author
|
close as it is out-of-data, new PR is here: #2053 |
zejunchen-zejun
added a commit
that referenced
this pull request
Aug 28, 2026
Split out of PR #1960 so they can land on their own. None of them is about state offload, and none needs #2045; blocking them behind either is pure loss, and pulling them out narrows what is left to review by a few hundred lines. --- a worker that outlives the join keeps the node `EngineCore.exit` joined each worker with a 5s timeout and did nothing after it. A worker that outlives that join keeps its VRAM slice and its all-reduce IPC handles, and `multiprocessing`'s atexit handler then joins the same process again with NO timeout -- so an engine already on its way out parks in `_exit_function -> join` forever while /metrics keeps answering 200. Seen twice on the k3-dev line; only a manual kill recovered the node. `enable_orphan_reaping` cannot help: `PR_SET_PDEATHSIG` fires when the parent *dies*, and this parent never does. Now escalates terminate -> kill. --- a stalled offload save hung the engine with every GPU idle A deferred save's blocks are freed only on `finished_saving`, but LMCache's pin monitor force-unpins a stalled transfer after `pin_timeout_sec` WITHOUT emitting that report. `has_pending_kv_work()` then never clears and the engine busy-loops with every GPU idle -- a hard hang under a tight pool, a slow block leak under a large one. `Scheduler._reconcile_stalled_deferred_saves`, wired into the KV progress poll, reclaims a save deferred past an abandon window and self-throttles to 5s so a 1ms poll costs nothing. The window reads LMCache's own `LMCACHE_EC_PIN_TIMEOUT_SEC` and adds 30s rather than taking an ATOM knob of its own, because that ordering IS the safety argument: two independent env vars would let ours sit below the timeout it must exceed, and nothing would say so. Unset gives 330s. Non-positive disables reclamation and restores the wait-forever behaviour. What makes reclaiming safe is not the clock -- our clock starts when the free is deferred, LMCache's when it pins, and nothing aligns them. It is that `OffloadWorkerMixin._guard` reports `finished_saving` on both the success and the exception path, so a report is lost only when `store()` neither returns nor raises. Two cases follow and they are exhaustive: the parked save is not copying (LMCache force-unpinned its source and stopped reading), and a save queued behind a parked one never reached `store()` at all. `finished_sending`'s `assert seq is not None` becomes a `continue`: after a reclaim, a late completion report has nothing left to free. --- a state-only step under `multi` parked its request forever `connector_metadata_has_work` decides whether a step's metadata reaches the worker, and it decided from a hardcoded field list shared by three connector families. `MultiConnectorMetadata` holds none of the work itself -- everything is in `metas` -- so it mirrored the subs' fields as aggregating properties, and missed one. Every step whose only work was in an unmirrored field was discarded, and the request parked on it waited for a report nobody was asked to produce. Fixed by asking the metadata instead of inspecting it: each class declares its own `WORK_FIELDS` next to the fields it defines, `has_work()` accumulates down the hierarchy, and `MultiConnectorMetadata` delegates to its subs rather than mirroring them -- so a field added to a sub can no longer be missed here. The old union list stays only as the fallback for duck-typed doubles, marked as not the place to register anything new. --- two behaviour-neutral seams and one marker `OffloadSchedulerMixin._claim_after_load` returns `max(hbm, lmc)`, which is what all four inlined call sites did. A seam rather than four `max`es because a layout whose claim and transfer aim at different boundaries has to override it, and getting that wrong is silent: the request starts further along than its state supports and the output is wrong with no exception. `DenseOffloadScheduler._may_emit_save` returns True. It matters for a layout whose `should_defer_free` pins a finished request's blocks until its save drains -- an unbounded queue then lets a slow backend hold an unbounded slice of the pool. `max_pending_saves` moves to `_offload_common` so every layout reads the same `OFFLOAD_MAX_PENDING_SAVES`. `KVCacheTensor.per_request_state` marks a hybrid's slot-addressed recurrent state. It must be registered in `kv_cache_data` -- the linear-attention forward reads it from there -- but no block-addressed mover may touch it, and `DenseKVByteCodec` now skips it: including it either fails the divisibility check or, if the slot count happens to divide `num_blocks`, inflates `bytes_per_block` past what `block_regions` describes. Verified on top of #2045 (`4df58434`): pytest tests/ (less tests/plugin and the two msgpack/aiter collection errors that fail on #2045 too) -> 3871 passed, 326 skipped, 3 xfailed, zero failures. black clean; ruff 36, the same count as #2045. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
zejunchen-zejun
added a commit
that referenced
this pull request
Aug 28, 2026
Phase 1 of remounting PR #1960 on #2045. Deliberately a RENAME, not a redesign: the tier still hangs off `StateSlotPool` and still spills on eviction through the staging ring. Phase 2 deletes the ring; Phase 3 moves the mount point to `PageUnitCheckpointStore`. Keeping those apart is what lets a reviewer tell rebase noise from design decisions. Squashed rather than replayed for the reason #2045 gives for its own squash: other's resolutions. #1960 keeps the original history. --- what #2045 renamed underneath the branch StateGroupPool -> StateSlotPool lookup_group(h) -> lookup(h) seq.per_req_cache_group -> seq.state_slots (list) + seq.state_slot num_per_req_cache_groups -> num_state_slots + state_slots_per_req pop() / release() -> pop_many(width) / release_many() has_free() -> has_free(n) _attach_state_group() -> _attach_state_slots() The semantic change hides in that table: **the "group" concept is gone.** Slots are allocated one at a time, `pop_many` states outright that the set is not adjacent and nothing may assume it is, and a checkpoint occupies exactly ONE slot -- speculation scratch is not state anybody resumes into. Every `group * span` address computation in #1960 lost its basis, so the tier's `_spill` / `_resumable_from` / `has_pending_spill` / `take_spill_copies` are re-expressed over slots, and `(group, staging)` becomes `(slot, staging)`. --- the three places the two PRs actually collide `_attach_state_slots` now returns bool. #2045 rewrote its body for per-slot allocation; #1960 needs it to be able to say "the state behind this boundary is not really here", because with the tier voting a hash can be accepted by `_resumable_from` and still not be in HBM. The two compose: #2045's width logic runs, and only the committed slot (element 0) is the load's destination, so only it takes `pop(spill=False)` -- the rollback scratch is this request's own and has nothing to forgo. `allocate` keeps #1960's two extents (`num_cached_blocks` vs `claim_blocks`) on top of #2045's per-slot attach, and the fresh-block loop starts from `len(seq.block_table)` rather than `num_cached_blocks`, since the widened claim already put blocks there. `CacheStats` gains #1960's `total_offload_tokens` alongside #2045's `total_reusable_tokens`. They are not the same series and neither subsumes the other: `reusable` is the honest denominator for the HBM walk, while tier tokens sit ABOVE that walk by design, so folding them in would make `cached > wanted`. `update()` therefore takes five required arguments and one defaulted. `deallocate` keeps #2045's midstep cancellation and `release_many`, and #1960's in-flight-load hold: the committed slot is parked in `_orphan_load_slots` until `settle_state_load`, because a worker is writing it; the scratch goes back immediately. `checkpoint_funnel` reports through `state_checkpoint_fates()` (a sum across `state_caches`) rather than the single `_state_checkpoint_cache` call. Today `state_caches` is a 1-tuple and the two are equivalent -- kept so Phase 3's tier counters, which arrive from a second class, are not silently dropped. --- functionality is NOT verified by this commit Three gates all take the else branch for K3 under #2045, so the tier is inert: 1. `isinstance(cache, StateSlotPool)` -- K3's `state_caches` is `(PagedStateCheckpointCoordinator,)`, so `cache.offload` is never set. 2. `_joint_kv_boundary` returns `_no_joint("not_hybrid")` whenever `paged_state_checkpoints is not None`, which for K3 is always. 3. `_attach_state_slots` returns from its PAGE branch before reaching `_tier_can_serve`. That is deliberate and is what Phase 3/5 undo. Do not read the green suite below as evidence the feature works -- the exit criterion for the tier is a nonzero `state_tier` counter in the logs, which no unit test can produce. Verified: pytest tests/ (less tests/plugin and the two msgpack/aiter collection errors that fail on #2045 too) -> 3915 passed, 326 skipped, 3 xfailed, zero failures. black clean; ruff 36, the same count as #2045. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
zejunchen-zejun
added a commit
that referenced
this pull request
Aug 28, 2026
…to rescue Phase 2. About 1000 lines go, and none of them for tidiness: the ring existed to solve exactly one problem, and #2045 removed the problem. --- why the ring existed at all A checkpoint used to live in a state slot, and `StateSlotPool.pop` handed that slot to the next request the instant it was evicted. The bytes therefore had to be rescued somewhere before they were overwritten, which is what the ring was: K entries allocated inside the state arena, never leased to a request, and addressed as `num_slots + staging` so `state_entry_views` could reach them with no second scheme. Everything else -- `staging_entries`, `admission_entries`, the drain protocol, the starvation detector, the reclaimer, `pop(spill=)`, `spills_forgone`, the spill-before-relocate ordering in `CommonAttentionBuilder.build` -- was a consequence. Under #2045 a K3 checkpoint is a PAGE image in the KV pool whose 127 units are *reserved*, and reserved units are structurally un-takeable: `_take_free` draws only from `_free` and they sit in `_used`, `BlockPool.free` raises on one, and `retire_top` refuses outright (`if top in self._raw_unit_owner: return None`). The source holds still on its own, so a transfer needs no rescue -- exactly the property dense KV has always had, where `should_defer_free` alone carries a save through its D2H and no ring was ever needed. So this is not a simplification, it is the removal of a workaround whose premise is gone. `state_pool.py` and `sub_pool_spec.py` now have an EMPTY diff against --- what stays, and why `state_entry_views` stays (re-indexed from group to slot, since a checkpoint is one slot wide now). Phase 4's load writes the committed slot directly, so this is its destination -- the plan's earlier draft had it deleted, which was wrong. `StagedTransfer` / `_StagingBuffer` stays. It shares a word with the ring and is a different thing: a per-thread pinned host bounce buffer that D2H and H2D both need, sized in bytes and unrelated to slot count. `StateOffloadIndex` keeps its load half (`request_load` / `complete_load` / `fail_load` / `abandon_load`) and its optimistic hash set. `confirm_spill` becomes `note_stored`, which is what it always did. `StateOffloadTier.submit_spill` becomes `submit_store(h, slot)` and loses its `ready_event`: there is no compute-stream producer to fence against once the packer gathers straight from a source that is not moving. --- what this leaves inert, deliberately **Nothing calls `submit_store` after this commit.** Phase 2 removed the ring that used to drive it and Phase 3 gives it its new caller, whose source is the checkpoint's PAGE units rather than a slot. Nothing populates `StateOffloadIndex.hashes` in between, so `_tier_can_serve` is false and no load is ever offered -- the tier is fully inert, on top of the three gates Phase 1 already documented. That is a WIP state, not a shippable one. --- one dead-code find, pre-existing on #1960 A method between `_tier_can_serve` and `_request_state_load` had lost its `def` line in an earlier rebase, leaving its docstring and `return` as unreachable statements after `_tier_can_serve`'s own return. It reads like a live invariant ("Whether `hit_hash`'s state can be produced at all, from either tier") and has no caller: the tier's vote actually reached `_gated_hit` through `StateSlotPool._resumable_from`. Removed rather than restored, since Phase 4 puts the vote on the coordinator side. Verified: pytest tests/ (less tests/plugin and the two msgpack/aiter collection errors that fail on #2045 too) -> 3906 passed, 326 skipped, 3 xfailed, zero failures. black clean; ruff 36, the same count as #2045. `git diff` against Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
zejunchen-zejun
added a commit
that referenced
this pull request
Aug 28, 2026
Split out of PR #1960 so they can land on their own. None of them is about state offload, and none needs #2045; blocking them behind either is pure loss, and pulling them out narrows what is left to review by a few hundred lines. --- a worker that outlives the join keeps the node `EngineCore.exit` joined each worker with a 5s timeout and did nothing after it. A worker that outlives that join keeps its VRAM slice and its all-reduce IPC handles, and `multiprocessing`'s atexit handler then joins the same process again with NO timeout -- so an engine already on its way out parks in `_exit_function -> join` forever while /metrics keeps answering 200. Seen twice on the k3-dev line; only a manual kill recovered the node. `enable_orphan_reaping` cannot help: `PR_SET_PDEATHSIG` fires when the parent *dies*, and this parent never does. Now escalates terminate -> kill. --- a stalled offload save hung the engine with every GPU idle A deferred save's blocks are freed only on `finished_saving`, but LMCache's pin monitor force-unpins a stalled transfer after `pin_timeout_sec` WITHOUT emitting that report. `has_pending_kv_work()` then never clears and the engine busy-loops with every GPU idle -- a hard hang under a tight pool, a slow block leak under a large one. `Scheduler._reconcile_stalled_deferred_saves`, wired into the KV progress poll, reclaims a save deferred past an abandon window and self-throttles to 5s so a 1ms poll costs nothing. The window reads LMCache's own `LMCACHE_EC_PIN_TIMEOUT_SEC` and adds 30s rather than taking an ATOM knob of its own, because that ordering IS the safety argument: two independent env vars would let ours sit below the timeout it must exceed, and nothing would say so. Unset gives 330s. Non-positive disables reclamation and restores the wait-forever behaviour. What makes reclaiming safe is not the clock -- our clock starts when the free is deferred, LMCache's when it pins, and nothing aligns them. It is that `OffloadWorkerMixin._guard` reports `finished_saving` on both the success and the exception path, so a report is lost only when `store()` neither returns nor raises. Two cases follow and they are exhaustive: the parked save is not copying (LMCache force-unpinned its source and stopped reading), and a save queued behind a parked one never reached `store()` at all. `finished_sending`'s `assert seq is not None` becomes a `continue`: after a reclaim, a late completion report has nothing left to free. --- a state-only step under `multi` parked its request forever `connector_metadata_has_work` decides whether a step's metadata reaches the worker, and it decided from a hardcoded field list shared by three connector families. `MultiConnectorMetadata` holds none of the work itself -- everything is in `metas` -- so it mirrored the subs' fields as aggregating properties, and missed one. Every step whose only work was in an unmirrored field was discarded, and the request parked on it waited for a report nobody was asked to produce. Fixed by asking the metadata instead of inspecting it: each class declares its own `WORK_FIELDS` next to the fields it defines, `has_work()` accumulates down the hierarchy, and `MultiConnectorMetadata` delegates to its subs rather than mirroring them -- so a field added to a sub can no longer be missed here. The old union list stays only as the fallback for duck-typed doubles, marked as not the place to register anything new. --- two behaviour-neutral seams and one marker `OffloadSchedulerMixin._claim_after_load` returns `max(hbm, lmc)`, which is what all four inlined call sites did. A seam rather than four `max`es because a layout whose claim and transfer aim at different boundaries has to override it, and getting that wrong is silent: the request starts further along than its state supports and the output is wrong with no exception. `DenseOffloadScheduler._may_emit_save` returns True. It matters for a layout whose `should_defer_free` pins a finished request's blocks until its save drains -- an unbounded queue then lets a slow backend hold an unbounded slice of the pool. `max_pending_saves` moves to `_offload_common` so every layout reads the same `OFFLOAD_MAX_PENDING_SAVES`. `KVCacheTensor.per_request_state` marks a hybrid's slot-addressed recurrent state. It must be registered in `kv_cache_data` -- the linear-attention forward reads it from there -- but no block-addressed mover may touch it, and `DenseKVByteCodec` now skips it: including it either fails the divisibility check or, if the slot count happens to divide `num_blocks`, inflates `bytes_per_block` past what `block_regions` describes. Verified on top of #2045 (`4df58434`): pytest tests/ (less tests/plugin and the two msgpack/aiter collection errors that fail on #2045 too) -> 3871 passed, 326 skipped, 3 xfailed, zero failures. black clean; ruff 36, the same count as #2045. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
zejunchen-zejun
added a commit
that referenced
this pull request
Aug 28, 2026
Phase 1 of remounting PR #1960 on #2045. Deliberately a RENAME, not a redesign: the tier still hangs off `StateSlotPool` and still spills on eviction through the staging ring. Phase 2 deletes the ring; Phase 3 moves the mount point to `PageUnitCheckpointStore`. Keeping those apart is what lets a reviewer tell rebase noise from design decisions. Squashed rather than replayed for the reason #2045 gives for its own squash: other's resolutions. #1960 keeps the original history. --- what #2045 renamed underneath the branch StateGroupPool -> StateSlotPool lookup_group(h) -> lookup(h) seq.per_req_cache_group -> seq.state_slots (list) + seq.state_slot num_per_req_cache_groups -> num_state_slots + state_slots_per_req pop() / release() -> pop_many(width) / release_many() has_free() -> has_free(n) _attach_state_group() -> _attach_state_slots() The semantic change hides in that table: **the "group" concept is gone.** Slots are allocated one at a time, `pop_many` states outright that the set is not adjacent and nothing may assume it is, and a checkpoint occupies exactly ONE slot -- speculation scratch is not state anybody resumes into. Every `group * span` address computation in #1960 lost its basis, so the tier's `_spill` / `_resumable_from` / `has_pending_spill` / `take_spill_copies` are re-expressed over slots, and `(group, staging)` becomes `(slot, staging)`. --- the three places the two PRs actually collide `_attach_state_slots` now returns bool. #2045 rewrote its body for per-slot allocation; #1960 needs it to be able to say "the state behind this boundary is not really here", because with the tier voting a hash can be accepted by `_resumable_from` and still not be in HBM. The two compose: #2045's width logic runs, and only the committed slot (element 0) is the load's destination, so only it takes `pop(spill=False)` -- the rollback scratch is this request's own and has nothing to forgo. `allocate` keeps #1960's two extents (`num_cached_blocks` vs `claim_blocks`) on top of #2045's per-slot attach, and the fresh-block loop starts from `len(seq.block_table)` rather than `num_cached_blocks`, since the widened claim already put blocks there. `CacheStats` gains #1960's `total_offload_tokens` alongside #2045's `total_reusable_tokens`. They are not the same series and neither subsumes the other: `reusable` is the honest denominator for the HBM walk, while tier tokens sit ABOVE that walk by design, so folding them in would make `cached > wanted`. `update()` therefore takes five required arguments and one defaulted. `deallocate` keeps #2045's midstep cancellation and `release_many`, and #1960's in-flight-load hold: the committed slot is parked in `_orphan_load_slots` until `settle_state_load`, because a worker is writing it; the scratch goes back immediately. `checkpoint_funnel` reports through `state_checkpoint_fates()` (a sum across `state_caches`) rather than the single `_state_checkpoint_cache` call. Today `state_caches` is a 1-tuple and the two are equivalent -- kept so Phase 3's tier counters, which arrive from a second class, are not silently dropped. --- functionality is NOT verified by this commit Three gates all take the else branch for K3 under #2045, so the tier is inert: 1. `isinstance(cache, StateSlotPool)` -- K3's `state_caches` is `(PagedStateCheckpointCoordinator,)`, so `cache.offload` is never set. 2. `_joint_kv_boundary` returns `_no_joint("not_hybrid")` whenever `paged_state_checkpoints is not None`, which for K3 is always. 3. `_attach_state_slots` returns from its PAGE branch before reaching `_tier_can_serve`. That is deliberate and is what Phase 3/5 undo. Do not read the green suite below as evidence the feature works -- the exit criterion for the tier is a nonzero `state_tier` counter in the logs, which no unit test can produce. Verified: pytest tests/ (less tests/plugin and the two msgpack/aiter collection errors that fail on #2045 too) -> 3915 passed, 326 skipped, 3 xfailed, zero failures. black clean; ruff 36, the same count as #2045. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
zejunchen-zejun
added a commit
that referenced
this pull request
Aug 28, 2026
…to rescue Phase 2. About 1000 lines go, and none of them for tidiness: the ring existed to solve exactly one problem, and #2045 removed the problem. --- why the ring existed at all A checkpoint used to live in a state slot, and `StateSlotPool.pop` handed that slot to the next request the instant it was evicted. The bytes therefore had to be rescued somewhere before they were overwritten, which is what the ring was: K entries allocated inside the state arena, never leased to a request, and addressed as `num_slots + staging` so `state_entry_views` could reach them with no second scheme. Everything else -- `staging_entries`, `admission_entries`, the drain protocol, the starvation detector, the reclaimer, `pop(spill=)`, `spills_forgone`, the spill-before-relocate ordering in `CommonAttentionBuilder.build` -- was a consequence. Under #2045 a K3 checkpoint is a PAGE image in the KV pool whose 127 units are *reserved*, and reserved units are structurally un-takeable: `_take_free` draws only from `_free` and they sit in `_used`, `BlockPool.free` raises on one, and `retire_top` refuses outright (`if top in self._raw_unit_owner: return None`). The source holds still on its own, so a transfer needs no rescue -- exactly the property dense KV has always had, where `should_defer_free` alone carries a save through its D2H and no ring was ever needed. So this is not a simplification, it is the removal of a workaround whose premise is gone. `state_pool.py` and `sub_pool_spec.py` now have an EMPTY diff against --- what stays, and why `state_entry_views` stays (re-indexed from group to slot, since a checkpoint is one slot wide now). Phase 4's load writes the committed slot directly, so this is its destination -- the plan's earlier draft had it deleted, which was wrong. `StagedTransfer` / `_StagingBuffer` stays. It shares a word with the ring and is a different thing: a per-thread pinned host bounce buffer that D2H and H2D both need, sized in bytes and unrelated to slot count. `StateOffloadIndex` keeps its load half (`request_load` / `complete_load` / `fail_load` / `abandon_load`) and its optimistic hash set. `confirm_spill` becomes `note_stored`, which is what it always did. `StateOffloadTier.submit_spill` becomes `submit_store(h, slot)` and loses its `ready_event`: there is no compute-stream producer to fence against once the packer gathers straight from a source that is not moving. --- what this leaves inert, deliberately **Nothing calls `submit_store` after this commit.** Phase 2 removed the ring that used to drive it and Phase 3 gives it its new caller, whose source is the checkpoint's PAGE units rather than a slot. Nothing populates `StateOffloadIndex.hashes` in between, so `_tier_can_serve` is false and no load is ever offered -- the tier is fully inert, on top of the three gates Phase 1 already documented. That is a WIP state, not a shippable one. --- one dead-code find, pre-existing on #1960 A method between `_tier_can_serve` and `_request_state_load` had lost its `def` line in an earlier rebase, leaving its docstring and `return` as unreachable statements after `_tier_can_serve`'s own return. It reads like a live invariant ("Whether `hit_hash`'s state can be produced at all, from either tier") and has no caller: the tier's vote actually reached `_gated_hit` through `StateSlotPool._resumable_from`. Removed rather than restored, since Phase 4 puts the vote on the coordinator side. Verified: pytest tests/ (less tests/plugin and the two msgpack/aiter collection errors that fail on #2045 too) -> 3906 passed, 326 skipped, 3 xfailed, zero failures. black clean; ruff 36, the same count as #2045. `git diff` against Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
valarLip
pushed a commit
that referenced
this pull request
Sep 3, 2026
* fix(offload): the generic fixes #1960 was holding behind the state tier
Split out of PR #1960 so they can land on their own. None of them is about
state offload, and none needs #2045; blocking them behind either is pure loss,
and pulling them out narrows what is left to review by a few hundred lines.
--- a worker that outlives the join keeps the node
`EngineCore.exit` joined each worker with a 5s timeout and did nothing after
it. A worker that outlives that join keeps its VRAM slice and its all-reduce
IPC handles, and `multiprocessing`'s atexit handler then joins the same
process again with NO timeout -- so an engine already on its way out parks in
`_exit_function -> join` forever while /metrics keeps answering 200. Seen
twice on the k3-dev line; only a manual kill recovered the node.
`enable_orphan_reaping` cannot help: `PR_SET_PDEATHSIG` fires when the parent
*dies*, and this parent never does. Now escalates terminate -> kill.
--- a stalled offload save hung the engine with every GPU idle
A deferred save's blocks are freed only on `finished_saving`, but LMCache's
pin monitor force-unpins a stalled transfer after `pin_timeout_sec` WITHOUT
emitting that report. `has_pending_kv_work()` then never clears and the engine
busy-loops with every GPU idle -- a hard hang under a tight pool, a slow block
leak under a large one.
`Scheduler._reconcile_stalled_deferred_saves`, wired into the KV progress
poll, reclaims a save deferred past an abandon window and self-throttles to
5s so a 1ms poll costs nothing.
The window reads LMCache's own `LMCACHE_EC_PIN_TIMEOUT_SEC` and adds 30s
rather than taking an ATOM knob of its own, because that ordering IS the
safety argument: two independent env vars would let ours sit below the timeout
it must exceed, and nothing would say so. Unset gives 330s. Non-positive
disables reclamation and restores the wait-forever behaviour.
What makes reclaiming safe is not the clock -- our clock starts when the free
is deferred, LMCache's when it pins, and nothing aligns them. It is that
`OffloadWorkerMixin._guard` reports `finished_saving` on both the success and
the exception path, so a report is lost only when `store()` neither returns
nor raises. Two cases follow and they are exhaustive: the parked save is not
copying (LMCache force-unpinned its source and stopped reading), and a save
queued behind a parked one never reached `store()` at all.
`finished_sending`'s `assert seq is not None` becomes a `continue`: after a
reclaim, a late completion report has nothing left to free.
--- a state-only step under `multi` parked its request forever
`connector_metadata_has_work` decides whether a step's metadata reaches the
worker, and it decided from a hardcoded field list shared by three connector
families. `MultiConnectorMetadata` holds none of the work itself -- everything
is in `metas` -- so it mirrored the subs' fields as aggregating properties,
and missed one. Every step whose only work was in an unmirrored field was
discarded, and the request parked on it waited for a report nobody was asked
to produce.
Fixed by asking the metadata instead of inspecting it: each class declares its
own `WORK_FIELDS` next to the fields it defines, `has_work()` accumulates down
the hierarchy, and `MultiConnectorMetadata` delegates to its subs rather than
mirroring them -- so a field added to a sub can no longer be missed here. The
old union list stays only as the fallback for duck-typed doubles, marked as
not the place to register anything new.
--- two behaviour-neutral seams and one marker
`OffloadSchedulerMixin._claim_after_load` returns `max(hbm, lmc)`, which is
what all four inlined call sites did. A seam rather than four `max`es because
a layout whose claim and transfer aim at different boundaries has to override
it, and getting that wrong is silent: the request starts further along than
its state supports and the output is wrong with no exception.
`DenseOffloadScheduler._may_emit_save` returns True. It matters for a layout
whose `should_defer_free` pins a finished request's blocks until its save
drains -- an unbounded queue then lets a slow backend hold an unbounded slice
of the pool. `max_pending_saves` moves to `_offload_common` so every layout
reads the same `OFFLOAD_MAX_PENDING_SAVES`.
`KVCacheTensor.per_request_state` marks a hybrid's slot-addressed recurrent
state. It must be registered in `kv_cache_data` -- the linear-attention
forward reads it from there -- but no block-addressed mover may touch it, and
`DenseKVByteCodec` now skips it: including it either fails the divisibility
check or, if the slot count happens to divide `num_blocks`, inflates
`bytes_per_block` past what `block_regions` describes.
Verified on top of #2045 (`4df58434`): pytest tests/ (less tests/plugin and
the two msgpack/aiter collection errors that fail on #2045 too) -> 3871
passed, 326 skipped, 3 xfailed, zero failures. black clean; ruff 36, the same
count as #2045.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* lmcache(k3): port the state offload tier onto #2045's per-slot pool
Phase 1 of remounting PR #1960 on #2045. Deliberately a RENAME, not a redesign:
the tier still hangs off `StateSlotPool` and still spills on eviction through
the staging ring. Phase 2 deletes the ring; Phase 3 moves the mount point to
`PageUnitCheckpointStore`. Keeping those apart is what lets a reviewer tell
rebase noise from design decisions.
Squashed rather than replayed for the reason #2045 gives for its own squash:
other's resolutions. #1960 keeps the original history.
--- what #2045 renamed underneath the branch
StateGroupPool -> StateSlotPool
lookup_group(h) -> lookup(h)
seq.per_req_cache_group -> seq.state_slots (list) + seq.state_slot
num_per_req_cache_groups -> num_state_slots + state_slots_per_req
pop() / release() -> pop_many(width) / release_many()
has_free() -> has_free(n)
_attach_state_group() -> _attach_state_slots()
The semantic change hides in that table: **the "group" concept is gone.** Slots
are allocated one at a time, `pop_many` states outright that the set is not
adjacent and nothing may assume it is, and a checkpoint occupies exactly ONE
slot -- speculation scratch is not state anybody resumes into. Every
`group * span` address computation in #1960 lost its basis, so the tier's
`_spill` / `_resumable_from` / `has_pending_spill` / `take_spill_copies` are
re-expressed over slots, and `(group, staging)` becomes `(slot, staging)`.
--- the three places the two PRs actually collide
`_attach_state_slots` now returns bool. #2045 rewrote its body for per-slot
allocation; #1960 needs it to be able to say "the state behind this boundary is
not really here", because with the tier voting a hash can be accepted by
`_resumable_from` and still not be in HBM. The two compose: #2045's width logic
runs, and only the committed slot (element 0) is the load's destination, so
only it takes `pop(spill=False)` -- the rollback scratch is this request's own
and has nothing to forgo.
`allocate` keeps #1960's two extents (`num_cached_blocks` vs `claim_blocks`)
on top of #2045's per-slot attach, and the fresh-block loop starts from
`len(seq.block_table)` rather than `num_cached_blocks`, since the widened claim
already put blocks there.
`CacheStats` gains #1960's `total_offload_tokens` alongside #2045's
`total_reusable_tokens`. They are not the same series and neither subsumes the
other: `reusable` is the honest denominator for the HBM walk, while tier tokens
sit ABOVE that walk by design, so folding them in would make `cached > wanted`.
`update()` therefore takes five required arguments and one defaulted.
`deallocate` keeps #2045's midstep cancellation and `release_many`, and #1960's
in-flight-load hold: the committed slot is parked in `_orphan_load_slots` until
`settle_state_load`, because a worker is writing it; the scratch goes back
immediately.
`checkpoint_funnel` reports through `state_checkpoint_fates()` (a sum across
`state_caches`) rather than the single `_state_checkpoint_cache` call. Today
`state_caches` is a 1-tuple and the two are equivalent -- kept so Phase 3's
tier counters, which arrive from a second class, are not silently dropped.
--- functionality is NOT verified by this commit
Three gates all take the else branch for K3 under #2045, so the tier is inert:
1. `isinstance(cache, StateSlotPool)` -- K3's `state_caches` is
`(PagedStateCheckpointCoordinator,)`, so `cache.offload` is never set.
2. `_joint_kv_boundary` returns `_no_joint("not_hybrid")` whenever
`paged_state_checkpoints is not None`, which for K3 is always.
3. `_attach_state_slots` returns from its PAGE branch before reaching
`_tier_can_serve`.
That is deliberate and is what Phase 3/5 undo. Do not read the green suite
below as evidence the feature works -- the exit criterion for the tier is a
nonzero `state_tier` counter in the logs, which no unit test can produce.
Verified: pytest tests/ (less tests/plugin and the two msgpack/aiter collection
errors that fail on #2045 too) -> 3915 passed, 326 skipped, 3 xfailed, zero
failures. black clean; ruff 36, the same count as #2045.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* refactor(offload): delete the staging ring, which #2045 left nothing to rescue
Phase 2. About 1000 lines go, and none of them for tidiness: the ring existed
to solve exactly one problem, and #2045 removed the problem.
--- why the ring existed at all
A checkpoint used to live in a state slot, and `StateSlotPool.pop` handed that
slot to the next request the instant it was evicted. The bytes therefore had to
be rescued somewhere before they were overwritten, which is what the ring was:
K entries allocated inside the state arena, never leased to a request, and
addressed as `num_slots + staging` so `state_entry_views` could reach them with
no second scheme. Everything else -- `staging_entries`, `admission_entries`, the
drain protocol, the starvation detector, the reclaimer, `pop(spill=)`,
`spills_forgone`, the spill-before-relocate ordering in
`CommonAttentionBuilder.build` -- was a consequence.
Under #2045 a K3 checkpoint is a PAGE image in the KV pool whose 127 units are
*reserved*, and reserved units are structurally un-takeable: `_take_free` draws
only from `_free` and they sit in `_used`, `BlockPool.free` raises on one, and
`retire_top` refuses outright (`if top in self._raw_unit_owner: return None`).
The source holds still on its own, so a transfer needs no rescue -- exactly the
property dense KV has always had, where `should_defer_free` alone carries a save
through its D2H and no ring was ever needed.
So this is not a simplification, it is the removal of a workaround whose premise
is gone. `state_pool.py` and `sub_pool_spec.py` now have an EMPTY diff against
--- what stays, and why
`state_entry_views` stays (re-indexed from group to slot, since a checkpoint is
one slot wide now). Phase 4's load writes the committed slot directly, so this
is its destination -- the plan's earlier draft had it deleted, which was wrong.
`StagedTransfer` / `_StagingBuffer` stays. It shares a word with the ring and is
a different thing: a per-thread pinned host bounce buffer that D2H and H2D both
need, sized in bytes and unrelated to slot count.
`StateOffloadIndex` keeps its load half (`request_load` / `complete_load` /
`fail_load` / `abandon_load`) and its optimistic hash set. `confirm_spill`
becomes `note_stored`, which is what it always did.
`StateOffloadTier.submit_spill` becomes `submit_store(h, slot)` and loses its
`ready_event`: there is no compute-stream producer to fence against once the
packer gathers straight from a source that is not moving.
--- what this leaves inert, deliberately
**Nothing calls `submit_store` after this commit.** Phase 2 removed the ring
that used to drive it and Phase 3 gives it its new caller, whose source is the
checkpoint's PAGE units rather than a slot. Nothing populates
`StateOffloadIndex.hashes` in between, so `_tier_can_serve` is false and no load
is ever offered -- the tier is fully inert, on top of the three gates Phase 1
already documented. That is a WIP state, not a shippable one.
--- one dead-code find, pre-existing on #1960
A method between `_tier_can_serve` and `_request_state_load` had lost its `def`
line in an earlier rebase, leaving its docstring and `return` as unreachable
statements after `_tier_can_serve`'s own return. It reads like a live invariant
("Whether `hit_hash`'s state can be produced at all, from either tier") and has
no caller: the tier's vote actually reached `_gated_hit` through
`StateSlotPool._resumable_from`. Removed rather than restored, since Phase 4
puts the vote on the coordinator side.
Verified: pytest tests/ (less tests/plugin and the two msgpack/aiter collection
errors that fail on #2045 too) -> 3906 passed, 326 skipped, 3 xfailed, zero
failures. black clean; ruff 36, the same count as #2045. `git diff` against
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* lmcache(k3): one CPU pool for KV and state, not two
Phase 3a. Deletes the tier's separate `atom-state-{rank}` LMCache engine, its
`OFFLOAD_STATE_CPU_SIZE` knob, and its `cache_policy = "FIFO"` override. The
codec now binds the same `storage_manager` the KV leg uses.
--- what changed
- `_state_storage_manager()` is gone (~50 lines). `bind_storage_manager` is
handed `self._engine.storage_manager` directly. The `gib <= 0` branch that
already did this was the whole implementation; what goes is the `> 0` half.
- `OFFLOAD_STATE_CPU_SIZE` removed, from the code and from
`docs/environment_variables.md`.
- `cache_policy = "FIFO"` removed; state inherits the KV pool's LRU.
--- why the two-pool argument does not hold
It rested on "the KV write stream is several times the state volume, so it
evicts the checkpoints". Measured against the code rather than assumed:
`page_unit_bytes = num_MLA_layers * block_size * entry * itemsize = 442,368`
with `--block-size 128` pinned by the recipe, so KV costs 3,456 B/token and a
53.6 MiB state image is worth ~16k tokens of it -- 13% of a 117k-token prompt's
KV. (Read the MLA layer count the other way, as a quarter of 93 layers, and it
is 3.3%. The conclusion is the same either way, which is why it did not need
settling first.)
And the byte ratio is not what decides an LRU eviction anyway. A request writes
its ~457 KV chunks and its one state object inside the same prefill window, so
both enter the LRU at the same position and cool at the same rate. State is
retired ALONGSIDE its own KV -- which is the behaviour we want, because a
boundary whose KV is gone is worthless.
Two pools cost what one does not: two eviction policies that must drift, while
a joint boundary needs both legs to survive together. #2045 made this same
argument one tier up -- state moved out of its reserved slots into the KV pool,
81.85% -> 93.60% -- and `OFFLOAD_STATE_CPU_SIZE` is the same reserved capacity
one tier down, so keeping it repeats the thing #2045 disproved.
`LMCACHE_MAX_LOCAL_CPU_SIZE` is now the one size to tune.
--- the cost, stated rather than hidden
`cache_policy` is per-engine, so one pool means one policy for both legs. That
is a real loss of freedom and it is currently free, because LRU is right for
both. The FIFO override's premise was "a state entry is written once and read
once"; #2045's own numbers refute it (~4,808 resumes over ~1,508 checkpoints is
~3 reads each), and re-access is bursty -- branches and retries on one boundary,
then superseded -- which is LRU's home ground. The genuine want, if it shows up,
is "stickier than LRU gives it", which FIFO does not provide either; the lever
then is touching the state key on a KV hit, not splitting the pool back apart.
Nothing yet measures whether that is needed. The counter to add with the store
path (Phase 3d) is `joint_boundary_state_miss`: a boundary whose KV leg landed
and whose state leg missed. Zero means the two pools never needed separating.
Verified: pytest tests/test_lmcache_offload_connector.py -> 186 passed. black
clean; ruff unchanged.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* lmcache(k3): bind the state key to its layout, and refuse a misfit read
Phase 3b. `StateByteCodec.key()` stops being the bare ATOM hash; it becomes
`xxh64(hash || layout_id)`. `get()` gains a size check on what comes back.
--- what changed
- `StateByteCodec.__init__` takes a required `layout_id` and refuses an empty
one. `_build_state_tier` reads it off `state_runtime.checkpoint_spec` and
declines to build the tier at all when it is absent, rather than falling
back to an unqualified key.
- `key(h)` hashes `h` and the layout id together with `xxh64`.
- `get(h, ...)` measures the returned `MemoryObj` and, on a size other than
`entry_bytes`, discharges its reference, counts `_misfit_reads`, warns, and
returns a miss.
- `_object_bytes()` reads `get_size()` with the tensor as fallback, and
returns None when neither answers.
--- why the key needed both
*Build safety.* The same prefix hash names a completely different image under a
different `num_spec`, TP size, or conv/ssm dtype. A size mismatch throws at
`entry_bytes`, but **the same size with a different order or meaning reads back
silently wrong state** -- and a request resumed onto wrong state produces wrong
output with nothing raised anywhere. #2045 already encodes all of it in
`layout_id` (layers / conv shape+dtype / ssm shape+dtype / order / tp / spec /
carry) and enforces it HBM-side in `_validate_paged_state_op`. This is the CPU
side of the same check, and it was missing.
*Namespace separation.* `CacheEngineKey` is
`(model_name, world_size, worker_id, chunk_hash, dtype)` -- no field says what an
entry IS. KV chunk keys carry `ChunkedTokenDatabase`'s chunk hash and state keys
carry ATOM's chained block hash: two different hash functions writing plain
integers into one space. Until Phase 3a they at least lived in different pools;
now they share one, and the only thing keeping them apart was accidental -- this
side hard-codes `torch.uint8` while KV carries the KV dtype. `MemoryFormat` is
not available as a discriminator either: the tier's own comment records that its
`KV_2LTD` is inert, chosen only to pass the allocator's `raise ValueError`.
Folding the layout in makes the two spaces disjoint by construction, since no KV
key can carry a K3 layout id.
`xxh64` rather than Python's `hash((h, layout_id))`: `hash` of a str is salted
per process, so a restart would silently orphan every entry the previous run
wrote -- a cache that quietly starts cold and never says so. `xxh64` is what the
block hashes themselves use.
--- why the size check, if the key already prevents it
It is unreachable, and that is the point: reaching it means two things collided
in the shared pool, and unpacking a colliding object writes another entry's
bytes over a request's live state. Degrading to a miss costs one recompute; not
checking costs silent wrong output. `_misfit_reads` is the counter that says the
key stopped doing its job.
None from `_object_bytes` means "cannot measure", never "size 0" -- refusing
what cannot be measured would turn an unknown into a guaranteed miss.
--- tests
Five, in `tests/test_lmcache_offload_connector.py`, on a `CacheEngineKey`
fixture rather than the full lmcache stub (these are about two keys being equal
or not, which the real dataclass answers by value): a layout change makes a
different key; the layout does not collapse distinct hashes; the key survives a
restart; a state key cannot equal a raw-hash KV key; a wrong-sized hit is a miss
with its reference discharged; an object that will not report a size is still
loaded.
Verified: pytest tests/ (less tests/plugin and the two msgpack/aiter collection
errors that fail on #2045 too) -> 3911 passed, zero failures. black clean;
ruff 36, unchanged.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* k3: name a checkpoint's PAGE units as tensors, for the CPU tier's packer
Phase 3c. Adds `_KimiMLAGDNCommon.page_unit_views(unit_ids)` -- the tensor-view
counterpart of #2045's `_page_unit_regions`. No behaviour changes; nothing calls
it until Phase 3d.
--- what changed
- `page_unit_views(unit_ids) -> list[torch.Tensor]`, one view per (unit, MLA
row), unit major and row minor.
- Seven tests in `tests/test_kda_checkpoint_slot_copy.py`, on the harness
#2045 already built for `_page_unit_bases`.
--- why a second way to name the same bytes
`_page_unit_regions` hands the Triton descriptor raw int64 addresses, which is
what a D2D copy the GPU issues wants. The LMCache staging packer takes
`list[torch.Tensor]` -- `fused_pack_chunk_major` is already a fully
parameterised gather over `segment_ptrs[] + segment_block_bytes[]`, so it needs
no kernel change, only its segments named the way it names them.
This is the ONLY new seam the tier needs. A load writes the Active Slot directly
and reuses `state_entry_views`, so the asymmetry is deliberate: the save gathers
from units, the load scatters to a slot, and the two compose because #2045's
`plan_segmented_copy` intersects two *ordered byte streams* -- the slot's ranges
and the units' regions are two cuts of the same logical image, so a blob
gathered in one order is byte-identical to one gathered in the other.
--- the two traps, both asserted rather than assumed
*Order is the contract.* Unit major, row minor -- the same ravel as
`_page_unit_bases`, which is the order `_checkpoint_copy_plan` builds the
destination stream in. The packer indexes segments positionally, so this order
IS the blob layout. Self-consistency would be enough for one build; what makes
it a contract across builds is `layout_id` in the key (Phase 3b), so a build
that reordered cannot read another's blob.
*`block_ratio`.* `unit_ids` carries **logical** block ids while `kv_cache` is
shaped in **physical** ones, and K3's ratio is 128. Flattening the two block
axes together and indexing by `unit * block_size` is the conversion -- the same
arithmetic `_page_unit_bases` does in addresses, so the two cannot drift into
disagreeing about which bytes a unit owns. The range check is against the
*logical* count for the same reason: the physical count is `ratio` times larger
in rows, so checking the wrong axis admits ids that read past the end. Its test
computes the bound from the tensor rather than from a constant, because taking
the harness's label would let the very confusion under test pass unnoticed.
`_page_unit_regions` is called first for its side effect: the contiguity and
granularity checks live there, and they have to run before any view is handed
out rather than after.
--- verification
The unit tests skip on a non-GPU runner (the module imports aiter at load), so
the geometry was checked directly by extracting the four methods and comparing
`page_unit_views` against `_page_unit_bases` byte for byte: 9 segments for 3
units x 3 rows, identical `data_ptr()`s, lengths equal to
`_page_unit_stream_sizes`, every view contiguous, no two units sharing a byte,
an out-of-range id refused, and a granularity mismatch refused before any view
is produced.
pytest tests/ (less tests/plugin and the two msgpack/aiter collection errors
that fail on #2045 too) -> 3911 passed, zero failures. black clean; ruff 36,
unchanged.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* lmcache(k3): store a checkpoint when it becomes READY, not when it is evicted
Phase 3d, the last of Phase 3. This is what turns the tier back on for saves:
a checkpoint that reaches READY is offered to the CPU tier, and the whole
engine->worker->engine round trip is wired. Loads are still off (Phase 4).
--- what changed
`page_unit_checkpoint.py`
- `_offload_ready`: checkpoints nominated at the READY transition. NOT pinned.
- `take_offload_stores(max_inflight)`: hands `(hash, unit_ids)` over and takes
the pin at that moment, up to the cap.
- `settle_offload_store(hash)` / `reclaim_stale_offload_pins(timeout_s)` /
`offload_pins_reclaimed`, exposed on the coordinator and reported in
`checkpoint_fates`.
- `PageUnitCheckpointStore(..., offload_sink=)` -- False nominates nothing.
`block_manager.py`
- `take_state_stores` / `settle_state_store(hash, ok)` /
`reclaim_stale_state_store_pins`. The switch (`kv_connector_hosts_state_tier`)
moves above the coordinator's construction, which now needs it.
`scheduler.py`
- `_publish_state_stores()`, beside `_publish_state_loads()` and on the same
two call sites plus the idle path.
- `_offload_max_pending_saves()`, reading the KV leg's own
`OFFLOAD_MAX_PENDING_SAVES`.
- The report drain settles the pin on both outcomes and runs the reconciler.
`kimi_k3/connector.py` -- `enqueue_state_stores`, `meta.state_stores`,
`_start_state_stores`, and `_state_store_failed_locally` for stores a worker
with no tier cannot even attempt.
`kimi_k3/state_tier.py` -- `submit_store(h, unit_ids)`, no `ready_event`.
`kimi_k3/state_object.py` -- `put(h, unit_ids)` gathers `page_unit_views`;
`get(h, slot)` still scatters into `state_entry_views`; `puts_refused`.
`metadata.py` / `types.py` -- `state_stores` declared as work.
--- why READY, and why the pin is not taken there
Three candidate trigger points and only one works. `begin_store` is too early:
the scatter has not ridden a batch, so the units do not hold the image yet.
`_evict` / `unindex` are too late -- they fire when the pool wants those units
*now*, so a multi-millisecond D2H started there holds 127 of the most-wanted
blocks at the worst possible moment. READY is when the bytes first exist and
when the record is least wanted: it has just entered the LRU at the cold end.
The pin is a separate question, and getting it wrong broke a real invariant --
`test_a_ready_unpinned_checkpoint_is_available_to_live_kv` caught the first
draft. A pin lives in this process while the copy runs in the worker, so it
spans several scheduler passes. Pinning at READY would make EVERY checkpoint
un-evictable for that window, which is exactly what #2045's admission argument
forbids. So READY only *nominates*: a nominee stays unpinned and spendable, and
`take_offload_stores` pins only the few actually in flight. A checkpoint the
pool needed more than the tier did simply loses its copy, which is the right
price for never making the pool wait on the CPU.
The cap is `OFFLOAD_MAX_PENDING_SAVES`, shared with the KV leg rather than given
a knob of its own: a KV save and a state store both hold bytes out of the same
pool while they run, so the depth is one question about one resource, and two
numbers would let an operator raise one and unknowingly double the other.
--- write volume, which is the objection this trigger invites
Storing every checkpoint rather than only evicted ones is more traffic, and it
is affordable: per request state adds 53.6 MiB on top of ~404 MB of KV, **13%**
on a path already in flight (~12 MB/s against the KV leg's ~90 MB/s over a
3600s run). What it buys is the trigger being at the cheapest possible moment
and dedup for free -- `begin_store` already refuses a hash it holds or has
pending, so one hash is offered once.
--- unindex during a copy does not cancel it
`unindex` means the boundary's KV block was spent, so the image is unreachable
in HBM from here on. That is precisely when the CPU copy is worth having, so
the pin holds the units to the end of the D2H and `_release_record` runs after.
The CPU copy outliving the HBM one is the entire point of the tier.
--- the lost report, and why 20 lines is enough
`reclaim_stale_offload_pins` takes the same window as
`_reconcile_stalled_deferred_saves`, derived from LMCache's own pin timeout for
the same reason. Recovery is total: a leaked pin breaks no `BlockPool`
invariant and leaves no half-released state, so zeroing the count restores the
record exactly -- which is why this needs a counter and a timer rather than any
repair logic. `offload_pins_reclaimed` reaches `checkpoint_fates`, because
otherwise the symptom is a pool that has quietly shrunk.
--- still off
Loads. Nothing calls `_request_state_load` for K3 yet: `_attach_state_slots`
returns from its PAGE branch before reaching `_tier_can_serve`, and the
`begin_restore` raise above it has to become an issue-a-load branch first
(Phase 4). `StateOffloadIndex.hashes` now fills, so the tier is half live: it
writes, and nothing reads yet.
--- tests
11 new in `tests/test_page_unit_checkpoint.py` -- nomination does not pin, the
hand-off does, a nominee stays spendable and may be lost, no sink nominates
nothing, offered once, the cap makes the rest wait rather than drop, a waiting
nominee stays evictable, both outcomes release, a report for a hash never sent
is a no-op, `unindex` mid-copy holds the units to the end, and a lost report is
reclaimed with the record left unbroken. Two more in
`tests/test_lmcache_offload_connector.py`: stores drain exactly once, and a
step whose only work is a store still reaches the worker.
Verified: pytest tests/ (less tests/plugin and the two msgpack/aiter collection
errors that fail on #2045 too) -> 3924 passed, zero failures. black clean;
ruff 36, unchanged.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(state-cache): a gated boundary neither tier has is disowned, not raised
Phase 4a, and it has to be first: with nothing else changed this commit is
behaviour-preserving, but without it the first request that actually uses the
CPU tier takes the engine down.
--- what changed
`BlockManager._attach_state_slots`, PAGE branch. Was:
if hit_hash != -1 and not begin_restore(hit_hash, seq.state_slot):
release_many(seq.state_slots); seq.state_slots = []
raise RuntimeError("gated PAGE checkpoint disappeared before state attach")
Now: cold start returns early; `begin_restore` success returns as before; a
miss falls through to `_request_state_load`, and only if that also declines is
the boundary disowned (`return False`, which `allocate` turns into
`num_cached_tokens = 0`). New counter `state_gate_lost_boundary`, reported in
`checkpoint_funnel`.
--- why the raise could not stay
It asserted that the gate and the HBM store agree. That held while
`can_allocate` only ever accepted boundaries the HBM index carried. Phase 4b
gives the CPU tier a vote, so a hash the gate accepted may live only there --
and every such request would hit this raise. Turning it into a load branch has
to land before the vote does, not after, or the two commits are only safe as a
pair.
--- why disowning is reachable rather than defensive
Two ways to arrive with neither tier able to produce it:
* the tier's index is optimistic by construction -- `hashes` means "was
stored once", never "is still there", because LMCache's own LRU can drop
bytes at any time;
* an HBM checkpoint can be unindexed between `can_allocate` and `allocate`
by another seq's `_fresh_block` in the same pass.
So this is a normal path with a counter, not an assertion. A large fraction of
`joint_boundaries` landing here would mean the gate is accepting boundaries
that do not survive to attach.
--- the slots are kept, unlike the old abort
The old code released the slots because it was aborting. Disowning is not an
abort: the request is about to recompute its whole prefix, and it writes that
state into these very slots. Releasing them would hand the next request a
buffer this one is still filling.
`test_missing_gated_checkpoint_releases_the_new_slot_and_raises` becomes
`test_a_gated_boundary_neither_tier_has_is_disowned_not_raised`, and now pins
all three: returns False, keeps its slots, counts one.
Verified: pytest tests/ (less tests/plugin and the two msgpack/aiter collection
errors that fail on #2045 too) -> zero failures. black clean; ruff unchanged.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* lmcache(k3): give the CPU tier a vote, and let a PAGE resume come from it
Phase 4b. This is the commit that makes the tier reachable: until now K3 wrote
53.6 MiB per checkpoint to LMCache and nothing ever read it.
--- what changed
`page_unit_checkpoint.py`
- `PagedStateCheckpointCoordinator.offload` + `attach_offload(index)`.
- `resumable_hit` now asks `_reachable(h)` instead of `store.contains(h)`:
HBM first, then `offload.hashes`.
- The constructor's `offload_sink: bool` is gone -- `attach_offload` sets the
vote and the sink together (see below).
`block_manager.py`
- Attaches the index to the coordinator once both exist. The coordinator is
built from `state_runtime.checkpoint_spec` and the index from the connector
config, so they are constructed in the wrong order for a ctor argument.
`kimi_k3/connector.py`
- Refuses to build the tier when `image_bytes != entry_bytes`.
--- why the vote is the whole feature
`can_allocate` walks the HBM prefix and asks every `Pool.STATE` class "the
rightmost boundary <= X that you accept". With the coordinator answering only
from `store.contains`, a hash whose image went to LMCache is invisible: the
walk stops short, nothing ever asks for it back, and the store path is pure
cost. `_reachable` is where the tier gets to say "I have that one".
It stops at the first boundary it accepts, scanning right to left, so a hash
accepted here that nothing can deliver does not cost a wasted lookup -- it
costs the whole walk-back, hiding every shorter checkpoint still resident in
HBM. That is why the disown path (Phase 4a) had to land first.
No preference rule between the tiers is needed: both are keyed by the same
content hash, so the scan takes the rightmost boundary wherever it lives, and
`_attach_state_slots` tries `begin_restore` before `_request_state_load`
regardless. A resident image never pays a park.
--- why the vote is optimistic, deliberately
`hashes` means "was stored once", never "is still there" -- LMCache's own LRU
can drop bytes under it at any moment. A false positive costs one park plus a
recompute, and retracts itself: `fail_load` calls `forget(h)`, so the next
request over that prefix is not sent down the same hole. Being certain instead
would mean a synchronous cross-process lookup on the admission path.
--- why attaching is one call, not two fields
Half-attached is worse than off in both directions. A vote with no sink accepts
hashes nothing will ever store, so every accepting request parks and
recomputes. A sink with no vote pins 127 units per checkpoint for a transfer
nobody will read. `attach_offload` sets both, so they cannot diverge.
--- the load's destination is the Active Slot
Not 127 freshly reserved units. The two are interchangeable because #2045's
copy plan intersects two *ordered byte streams* -- the slot's ranges and the
units' regions are two cuts of one image -- so a blob gathered in unit order
unpacks correctly in slot order. Writing the slot directly means no
`ensure_free_units` on the admission path, no refusal branch, no units pinned
for the length of an H2D, and no cap needed on concurrent loads.
It does mean the store reads `image_bytes` and the load writes a whole slot, so
the connector now refuses to build a tier where those differ, at the one point
both numbers are in scope. They are equal for K3; a model where they are not
would silently truncate the store or over-read the load.
--- tests
Six in `tests/test_page_unit_checkpoint.py`: a tier-only hash is accepted, no
tier means no acceptance, the scan still takes the rightmost, attaching turns
on both halves together, and the optimistic half retracts on `forget`.
Three in `tests/test_state_checkpoint.py`, end to end through
`_attach_state_slots`: a tier-only boundary becomes a queued load with no
restore op and the slot as its destination; a boundary in both is served from
HBM with no park; a tier that declines disowns and counts.
Verified: pytest tests/ (less tests/plugin and the two msgpack/aiter collection
errors that fail on #2045 too) -> zero failures. black clean; ruff unchanged.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* lmcache(k3): the joint boundary now serves PAGE, which is its only reason to exist
Phase 5, the last one. `_joint_kv_boundary` refused every PAGE sequence; it now
serves only PAGE. With this the tier is reachable end to end.
--- what changed
`block_manager.py`
- The gate inverts:
- if not seq.has_per_req_cache or self.paged_state_checkpoints is not None:
- return self._no_joint("not_hybrid")
+ if not seq.has_per_req_cache or self.paged_state_checkpoints is None:
+ return self._no_joint("no_paged_checkpoints")
- The hbm/tier split reads the coordinator instead of the slot pool.
- `joint_boundaries_hbm` / `_tier` become `state_hbm` / `state_tier`.
`page_unit_checkpoint.py` -- `contains(h)` delegation, so the split does not
reach into `.store`.
`scheduler.py` / `llm_engine.py` -- the log line and the Prometheus tuple follow
the rename.
--- why the old refusal was right, and why it stopped being right
It read `is not None` and refused, because a K3 checkpoint was an Active Slot
that the tier spilled out of `StateSlotPool` -- a PAGE image lived in the KV
pool, which the KV connector already offloads on its own, so there was no
second boundary to agree on.
#2045 moved K3's image into the KV pool and with it the entire reason the joint
path exists. HBM now enforces `state ⊆ KV`: `_record_evicted` unindexes a
checkpoint the instant its boundary block is spent, so a checkpoint can no
longer outlive its KV there. **When LMCache hands the KV back, nothing hands
the state back unless the two are fetched together.** Leaving this refusal in
place makes the tier dead weight -- it writes 53.6 MiB per checkpoint and
nothing ever reads it -- and no other counter in the system would say so.
--- why the counters were renamed rather than left alone
`joint_boundaries_hbm/_tier` described the boundary as a whole, which was true
when the KV leg had no tier of its own to be served from. It has one now
(`_decide_load_after_alloc` decides it separately), so the old names read as if
they covered both legs. `state_hbm` / `state_tier` say what they measure.
The split also means something new. It used to say "the state pool is too small
for this concurrency". Under #2045 checkpoints share the paged pool with the KV
write stream, so a ratio dominated by `state_tier` says **the paged pool** is
too small -- checkpoints are being squeezed out by the KV they now live beside.
That diagnosis did not exist before.
`state_tier` is also the only counter here that cannot be non-zero with the CPU
tier switched off, which makes it the honest test of "did this feature run".
No passing unit test can produce it in production.
--- tests
Three in `tests/test_state_checkpoint.py`, on a harness that had to reproduce
the shape the joint path is for: publish a prompt, place ONE rung at an
interior boundary (a checkpoint under the last block is one no scan can look
up, since `can_allocate` matches over `range(n_hash_blocks - 1)`), then evict a
KV block below it so the walk stops short. A PAGE seq gets a joint boundary at
the rung with real work for the KV leg; the split reports `hbm` while the
checkpoint is resident and `tier` once it is not; no tier means no boundary.
Verified: pytest tests/ (less tests/plugin and the two msgpack/aiter collection
errors that fail on #2045 too) -> 3935 passed, zero failures. black clean;
ruff unchanged.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(offload): the delegating shell never forwarded `enqueue_state_stores`
Found by the first end-to-end run of the K3 CPU state tier on this branch: the
tier reported healthy on both sides and stored exactly nothing. 94 checkpoints
were nominated, handed to `_publish_state_stores`, refused, and had their PAGE
units released again.
--- the symptom
Server log, 94 times over a 6-minute agentic replay at concurrency 8:
state offload: LMCacheOffloadConnectorScheduler did not carry 1 state
store(s); releasing their units now.
with everything else looking correct:
lmcache_offload: worker family=kimi_k3
kimi_k3 offload: state tier up, entry=55.39 MiB rank=0, sharing the
paged-KV CPU pool, layout=kda-paged-state-v1:layers=69:...
state checkpoints: checkpoints_kept=112 ... offload_pins_reclaimed=0
So the worker half built the tier, the coordinator nominated checkpoints at
READY, and `take_offload_stores` pinned and handed them over -- and the
scheduler half dropped every one.
--- the cause
`LMCacheOffloadConnectorScheduler` is a shell that owns no behaviour: it holds
`self._impl = _build_scheduler(config)` and forwards each method by hand.
`7b6b58e0` added `enqueue_state_stores` to `KimiK3OffloadScheduler`
(kimi_k3/connector.py:352) but did not add the matching forwarder to the shell,
so the method existed on the implementation and was invisible from outside:
$ git grep -c enqueue_state_stores atom/kv_transfer/offload/connector.py
0
`Scheduler._publish_state_stores` probes with
`getattr(self.kv_connector, "enqueue_state_stores", None)`, and
`self.kv_connector` is the shell, so the probe returned None unconditionally
and the store path took its "nothing will carry these" branch on every pass.
This is not a race or a config error -- it could never have worked.
The load leg was unaffected only by luck: `enqueue_state_loads` and
`take_state_reports` were both given forwarders when they were added. Loads
still moved no bytes, because with nothing ever stored `offload.hashes` stayed
empty and the tier had nothing to vote for.
--- the fix
Add the missing forwarder, in the same shape as its `enqueue_state_loads`
neighbour so the two cannot drift again.
--- the second change, and why it is in this commit
The warning that reported the failure named a cause that was not the one:
The tier's index should not have been installed against this connector.
That reads as a configuration mistake, and it sent the first pass of this
diagnosis at `kv_connector_hosts_state_tier` and the layout selector -- both of
which were behaving correctly. The class name printed is the *shell*, which
tells you nothing about whether `_impl` supports the call. Both warnings now
name the two possible causes and say which class is being reported.
--- verification
Static: the probe is `getattr` on a method the shell does not define, so the
refusal is unconditional; adding the forwarder is the whole of the repair, and
it is additive -- no existing path changes.
Runtime: re-running the same workload with this commit applied, the
`did not carry` warning no longer appears.
Lines conform to black's 88-column limit; black and ruff are not installed in
the benchmark image, so formatting was checked by inspection against the
surrounding style rather than by running them.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* feat(offload): counters for the store leg, and stop reporting a pool that holds none
Three observability defects, found while trying to judge whether the K3 CPU
state tier was working. Together they meant the tier could store nothing at all
and no number anywhere would say so -- which is exactly what happened in
8a5364ab, where 94 refused stores produced one warning line and four counters
that all read zero for unrelated reasons.
--- 1. the store leg had no counters at all
`StateOffloadIndex.stats()` reported `loads_attempted / completed / failed` and
`indexed`. On a K3 run the load counters stay zero until something has been
stored, so the only store-side signal was `indexed` growing -- and `indexed` is
a set size that also grows for reasons unrelated to this pass.
Added `stores_attempted` (handed to the connector, counted in
`take_state_stores`), `stores_completed` / `stores_failed` (split in
`settle_state_store` by the worker's report), and `stores_refused`.
`stores_refused` is deliberately apart from `stores_failed`, and it is the one
that matters most: failed means the worker tried and could not, refused means
nobody tried. The shell-forwarding bug fixed in 8a5364ab was a pure `refused`
condition, and with this counter it would have been one glance at the periodic
line instead of a hunt through the connector factory.
The worker's own `puts_refused` needs no separate channel: a refused `put`
already returns False and settles as `ok=False`, so it lands in `stores_failed`.
--- 2. `[Pool Pressure]` read the pool that holds no checkpoints
`BlockManager.pool_pressure()` folded in `self.state.checkpoint_fates()`. Under
#2045 a K3 checkpoint is a PAGE image the coordinator owns and the slot pool
holds none, so the line printed
[Checkpoint Fates] kept: 0, dropped: 0, evicted: 0, orphaned: 0
on the same run whose other line said `checkpoints_kept=112`. Two counters for
one fact, disagreeing in one log, with the wrong one printed in the section a
reader goes to for pool pressure.
Now reads `self._state_checkpoint_cache`, which is already the coordinator when
there is one and the slot pool otherwise -- the same object every other
checkpoint question is asked of. `occupancy()` stays on the slot pool, which is
the thing that has slots.
--- 3. the tier's counters vanished when K3's checkpoints moved to PAGE
`StateGroupPool.checkpoint_fates` used to fold the tier's stats in under a
`state_offload_` prefix. `PagedStateCheckpointCoordinator.checkpoint_fates` was
written without that, so once #2045 moved K3's checkpoints into the coordinator
the whole `state_offload_*` family disappeared from the logs -- a silent
regression, since nothing errors when a key stops being emitted.
Restored on the coordinator. The fold is `getattr`-guarded because
`attach_offload` accepts anything answering `hashes`, and the tests attach a
double with no counters; a missing `stats` means "nothing to fold", not an
error. (`test_a_tier_that_declines_disowns_rather_than_parking` caught this on
the first draft.)
--- also added: `state_offload_store_backlog`
A gauge rather than a counter: `len(_offload_ready)`, the nominations waiting
for a slot under `OFFLOAD_MAX_PENDING_SAVES`. That queue is unbounded and
drained oldest-first, so checkpoints reaching READY faster than the tier drains
them shows up as a value that climbs and stays high, with fresh nominations
queueing behind stale ones. There is no other way to see it -- the stale
entries are skipped cheaply on the way out, so nothing else registers the
backlog.
--- how to read the new line
state_offload_stores_refused > 0 -- wiring fault, the connector never took it
stores_attempted - completed - failed -- in flight
stores_completed vs checkpoints_kept -- how much of what HBM keeps the tier got
loads_failed / loads_attempted -- the index's false-positive rate
store_backlog climbing -- READY outruns the drain
Verified: pytest tests/test_page_unit_checkpoint.py tests/test_state_checkpoint.py
tests/test_lmcache_offload_connector.py tests/test_block_manager.py -> 466 passed.
Lines conform to black's 88-column limit; black and ruff are absent from the
benchmark image, so formatting was checked by inspection.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(offload): the K3 state packer imported a module that does not exist
Every state store on this branch has failed since the store path was wired.
Found by the first run where stores were actually carried (8a5364ab): 3,904
tracebacks, all identical, and not one checkpoint reached LMCache.
--- the symptom
Traceback (most recent call last):
File "kimi_k3/state_tier.py", line 109, in _do_store
stored = bool(self.codec.put(h, unit_ids))
File "kimi_k3/state_object.py", line 141, in put
self._staged.pack(self._backend.page_unit_views(unit_ids), obj)
File "kimi_k3/staging.py", line 175, in pack
from atom.kv_transfer.offload.triton_kv_staging import fused_pack_chunk_major
ModuleNotFoundError: No module named 'atom.kv_transfer.offload.triton_kv_staging'
state offload: store of hash 10785477046942485201 failed
3,904 of them over a 50-minute concurrency-16 agentic replay -- one per store
attempt, on every rank. `stores_failed` would have counted all of them, and
`indexed` never moved.
--- cause 1: the module is one package deeper
`fused_pack_chunk_major` lives in `atom.kv_transfer.offload.dense.triton_kv_staging`.
The dense codec, which has used it all along, imports it from there
(`dense/kv_byte_codec.py:127`). The K3 stager's import dropped the `.dense`.
Nothing caught it because the import is function-local, inside `pack`, so it is
only attempted when a store actually runs -- and until 8a5364ab no store ever
reached the worker. The unit tests do not reach it either: they exercise the
codec against doubles that never call the real packer.
--- cause 2: a failed `put` leaked its allocation, once per failure
obj = self._allocate(self.entry_bytes) # ref_count = 1
self._staged.pack(...) # raises
self._storage.batched_put([key], [obj]) # never reached
`batched_put` discharges the reference it is handed
(`StorageManager.batched_put` ends in `memory_obj.ref_count_down()`), so the
success path correctly owes nothing -- but only if it is reached. With `pack`
throwing, the allocation was stranded at ref_count=1 and LMCache reported it
much later:
MemoryObj at ... is being garbage collected with ref_count=1, pin_count=0.
This indicates ref_count_down()/unpin() was not called
3,768 such warnings against 3,904 failed stores: one leak per failure. Left
alone this shrinks the shared CPU pool by one 53.6 MiB entry per failed store
until nothing can be allocated at all -- so the second-order effect of the
import bug was a slow starvation of the pool the KV leg shares.
`get` already guarded its own reference with `finally` for exactly this reason;
`put` did not. It does now, on the exception path only, matching what the DSV4
codec does (`hybrid/dsv4/codec.py:1293`).
--- why one commit
The leak is only reachable through a throwing `pack`, and the import was the
only thing throwing. Fixing the import alone would hide the leak rather than
remove it, and the next exception from that line -- an out-of-memory in the
staging buffer, say -- would strand allocations again with no traceback to
explain the pool shrinking.
--- verification
python -c "from atom.kv_transfer.offload.dense.triton_kv_staging import
fused_pack_chunk_major" -> OK
pytest tests/test_lmcache_offload_connector.py tests/test_state_checkpoint.py
tests/test_page_unit_checkpoint.py -> 412 passed
The import is still function-local, so this remains untested by anything that
does not run a real store. A unit test that calls `StagedTransfer.pack` against
real CUDA tensors would have caught it; that is worth adding and is not in this
commit.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(offload): the same two wiring gaps, in the places the fixes did not reach
Fix-then-sweep over `dfab0fa0` and `8a5364ab`. Both of those repaired one
instance of a bug that had two, and neither instance the fix reached is covered
by any test.
--- 1. `unpack` imports the same module that does not exist
`dfab0fa0` corrected `StagedTransfer.pack`'s import of
`fused_pack_chunk_major` (the module is one package deeper, under `.dense`).
`unpack` imports `fused_unpack_chunk_major` from the same wrong path, eleven
lines further down, and was left alone.
It has never fired because no state load has ever run -- with nothing stored,
`offload.hashes` stayed empty and the tier had nothing to vote for. The first
successful store makes the first load reachable, and it would have raised
`ModuleNotFoundError` on that load, on every rank.
Both kernels live in `atom.kv_transfer.offload.dense.triton_kv_staging`;
verified by importing both.
--- 2. `chunk_size` is not on the delegating shell either
`8a5364ab` added the missing `enqueue_state_stores` forwarder. Sweeping every
member `Scheduler` reads off `self.kv_connector` against what
`LMCacheOffloadConnectorScheduler` defines turns up one more: `chunk_size`.
scheduler.py:2284 seq.offload_kv_chunk_tokens = int(
getattr(self.kv_connector, "chunk_size", 0) or 0)
`self.kv_connector` is the shell, the shell has no `chunk_size`, so this has
always stamped 0. It is latent rather than live: `_joint_kv_boundary` reads
`self._joint_chunk_tokens` first and only falls back to this, and that primary
comes from `BlockManager.__init__` reading the LMCache config directly. But the
fallback exists precisely for the case where that read failed -- and it was
dead, so a build that could not read the config at startup would refuse every
joint boundary with `no_chunk_size` and never say why.
Forwarded as a property, with the reason in its docstring.
--- 3. the class of bug, closed rather than fixed twice
Both are the same shape: the shell forwards by hand, so a member added to an
implementation is invisible from outside, and every reader is a
`getattr(..., default)` that does not raise -- it takes the default forever.
Two instances in two days says the next one is a matter of time.
`test_every_member_the_scheduler_reads_is_reachable_through_the_shell` scans
`scheduler.py`'s source for `getattr(self.kv_connector, "x")`,
`self.kv_connector.x`, and `_connector_flag("x")`, and asserts every name is
reachable on the shell class. It reads the source rather than a hand-kept list,
because a hand-kept list is the same failure one level up. The scan asserts it
found something, so a regex that stops matching fails loudly instead of passing
vacuously.
Falsified: renaming the new `chunk_size` property makes it fail and name the
missing member.
Verified: pytest tests/ (less tests/plugin and the two msgpack/aiter collection
errors) -> 3936 passed, zero failures. black clean; ruff 36, unchanged.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(state-cache): the CPU tier gathered a checkpoint's padding as if it were image
Fourth and last blocker in the chain that kept the K3 CPU tier from storing
anything. With the previous three fixed, a 25-minute concurrency-16 replay
still ended at
state_offload_stores_attempted=462
state_offload_stores_completed=0
state_offload_stores_failed=450
state_offload_indexed=0
and 3,696 identical tracebacks:
File "kimi_k3/state_object.py", line 149, in put
self._staged.pack(self._backend.page_unit_views(unit_ids), obj)
File "kimi_k3/staging.py", line 182, in pack
dst_tensor = self.memory_tensor(dst, nbytes)
ValueError: ATOM LMCache connector: MemoryObj tensor is too small for
59867136 bytes; got 58079232
--- the geometry
From the run's own startup line:
unit_bytes=2138112 image_bytes=58079232 units_per_checkpoint=28
ceil(58079232 / 2138112) = ceil(27.163) = 28
28 x 2138112 = 59867136 <- what page_unit_views gathered
58079232 <- what put() allocated
difference = 1787904 <- padding in the 28th unit
An image occupies whole units, and the last one is 83.6% padding here. Nothing
owns those bytes.
--- why only the CPU side hit it
The HBM path never had the problem because it already passes the image size
down: `_checkpoint_copy_plan` calls
plan_segmented_copy(src_sizes, self._page_unit_stream_sizes(units),
spec.image_bytes)
so the D2D copy writes exactly `image_bytes` into the unit stream and leaves
the tail untouched. The image is, by construction, the LEADING `image_bytes`
of that stream. `page_unit_views` was written as the tensor-view counterpart of
the same addressing but did not carry the third argument across, so it named
whole units to a consumer that sizes its destination at `entry_bytes` -- which
for K3 is exactly `image_bytes`.
The connector's `image_bytes != entry_bytes` guard cannot catch this: both are
58079232 and agree. The quantity that disagrees is a third one,
`sum(page_unit_views)`, which no check compared against either.
--- the fix
Trim the gathered stream to `image_bytes`, dropping whole views past the
budget and slicing the one that straddles it.
The slice is byte-exact rather than view-aligned because the image does not end
on a view boundary: 58079232 leaves 350208 B of the last unit in use, which is
20.96 of its rows. That view is reinterpreted as `uint8` and sliced; the packer
sizes segments as `numel * element_size`, so a uint8 segment costs it nothing.
The load side needs no counterpart: it scatters into `state_entry_views`, which
sums to `entry_bytes` exactly, so both ends of the round trip are now the same
58079232 bytes.
A budget left unspent raises rather than storing a short blob -- that would be
a real geometry disagreement, and a truncated image read back as valid is the
one outcome worth crashing over.
--- two notes on shape
Inline rather than a helper method: the existing tests drive `page_unit_views`
with a `types.SimpleNamespace` stand-in for `self`, and a second attribute would
have to be stubbed in every one of them. For the same reason the spec is read
with `getattr(spec, "image_bytes", 0)` -- a fork build carries no spec at all,
and there the whole-unit stream is the right answer.
--- verification
pytest tests/test_kda_checkpoint_slot_copy.py
tests/test_lmcache_offload_connector.py
tests/test_state_checkpoint.py
tests/test_page_unit_checkpoint.py -> 441 passed
Also confirmed from the same run that the previous commit's leak guard works:
`MemoryObj ... garbage collected with ref_count=1` went from 3,768 occurrences
to 0, even though every store still failed.
None of the four bugs in this chain was reachable by the unit tests, because
all of them live past the first line that only a real store executes. A test
that packs real CUDA tensors through `StagedTransfer` would have caught three
of the four; it is still worth adding.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(offload): the state key treated an unsigned block hash as signed
About half of every K3 state store failed. Measured over a concurrency-16
agentic replay with the four earlier blockers fixed:
state_offload_stores_attempted=46
state_offload_stores_completed=19
state_offload_stores_failed=27 <- 59%
with 296 identical tracebacks:
File "kimi_k3/state_object.py", line 153, in put
self._storage.batched_put([self.key(h)], [obj])
^^^^^^^^^^^
File "kimi_k3/state_object.py", line 107, in key
digest.update(int(h).to_bytes(8, "little", signed=True))
OverflowError: int too big to convert
state offload: store of hash 12983297547834104159 failed
--- cause 1: the hash is unsigned and the key said otherwise
12983297547834104159 > 2**63-1, so `to_bytes(8, signed=True)` cannot hold it.
An ATOM block hash is `xxhash.xxh64().intdigest()`, which spans the whole
0..2**64-1, and `BlockManager.compute_hash` chains one into the next with
`prefix.to_bytes(8, "little")` -- unsigned, no keyword. The state key was the
only place in the chain that read the same value as signed, so it rejected
every hash with the top bit set. That is half the key space, and the observed
59% is that with sampling noise.
Nothing downstream noticed because the failure is inside a `try` in
`_do_store`, which counts a failed store and moves on. Before the counters
added in a8045614 it produced no number at all.
--- cause 2: the raise stranded a MemoryObj, in a window the last guard missed
`dfab0fa0` wrapped `pack` so a throwing gather would discharge the allocation.
`key(h)` is evaluated in `batched_put`'s argument list, which is after the
allocation and before `batched_put` can take ownership -- outside that guard.
So each of the 296 overflows also leaked a 53.6 MiB entry, and the LMCache
warning reappeared next to the traceback:
MemoryObj at 136839168 is being garbage collected with ref_count=1
Fixed by computing the key *before* allocating. `key` is pure, so hoisting it
removes the window rather than guarding it, and the `except` now covers
`batched_put` itself as well -- matching what the DSV4 codec does.
--- what this unblocks
The same `key` is used by `get`, so roughly half of all loads would have missed
in a way indistinguishable from an LRU eviction -- `fail_load` would then
`forget` a hash whose bytes were never stored under a name it could find, and
the index would decay for a reason nothing reported.
Verified: pytest over the state and offload suites -> 442 passed. The overflow
itself is confirmed directly:
>>> (12983297547834104159).to_bytes(8, "little", signed=True)
OverflowError: int too big to convert
>>> (12983297547834104159).to_bytes(8, "little", signed=False).hex()
'5f19dacf54f02db4'
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* test(state-cache): pin the image trim, which nothing reached
`961f80d6` fixed `page_unit_views` gathering a checkpoint's padding, and the
suite it ran stayed green before and after — because no test reaches the trim
at all.
The addressing tests build their stub spec as
checkpoint_spec=SimpleNamespace(page_unit_bytes=...)
with no `image_bytes`, so `budget` is 0 and every one of them takes the
`return views` early path. The trim shipped untested, which is how a bug that
failed 3,696 stores in a row got there in the first place.
--- what is pinned
Seven cases in `TestPageUnitViewsStopAtTheImage`, on the harness the addressing
tests already use (`build` gains optional `image_bytes` and `dtype`; both stay
off by default so the existing tests keep exercising whole units):
* the gathered stream is exactly `image_bytes` — the invariant `put`'s
allocation depends on, and the one that was broken;
* a whole view past the budget is dropped, not truncated to zero length;
* an image that lands on a view boundary is not sliced at all;
* **the kept bytes are the leading bytes and nothing else** — trimming
truncates, it does not reorder. The blob is read back by scattering into
the slot in the same order, so a byte that moved here would land in the
wrong layer there with nothing raised;
* a multi-byte dtype is sliced by BYTES, not elements — the straddling view
is reinterpreted as `uint8` first, and on bf16 an element-wise slice would
keep twice what it should. The production cache is 1 byte per element, so
only a test with a wider dtype can hold this;
* a spec with no image size keeps whole units (the fork build, which has no
spec to trim against);
* units that cannot cover the image raise rather than storing a short blob.
--- verification
`tests/test_kda_checkpoint_slot_copy.py` skips wholesale on a non-GPU runner
(the module imports aiter at load), so the seven cases were additionally run by
extracting `page_unit_views` and its three neighbours out of the source and
driving them directly. All seven hold, including the bf16 one.
pytest tests/ (less tests/plugin and the two msgpack/aiter collection errors)
-> 3936 passed, zero failures. black clean; ruff 36, unchanged.
--- one note on `put`'s exception shape, checked not changed
`2c5c3457` moved `batched_put` inside the `try` whose `except` calls
`ref_count_down`. That is safe only if `batched_put` takes ownership just on
success — and it is the pattern the DSV4 codec already uses
(`hybrid/dsv4/codec.py:1284`, "StorageManager owns the MemoryObj after a
successful batched_put"). Consistent with the repo rather than a new
assumption, so left alone.
The window between `_allocate` and `batched_put` is now fully covered:
`page_unit_views` can raise (the geometry check above) and is inside the try;
`key(h)` is computed before the allocation.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* style: clear the ruff findings CI reports on this PR
CI's reviewdog run on #2053 flagged five findings, all introduced by this
branch. Every one is real and still present.
atom/kv_transfer/offload/hybrid/dsv4/connector.py:27 I001
atom/kv_transfer/offload/hybrid/kimi_k3/connector.py:11 I001
atom/kv_transfer/offload/hybrid/kimi_k3/connector.py:425 SIM102
tests/test_scheduler.py:1466 I001
tests/test_scheduler.py:1498 I001
plus one more the bot has not reported yet because reviewdog filters to diff
context and that line has not changed since:
atom/model_engine/page_unit_checkpoint.py:6 I001
--- why the local checks said clean
`ruff check` with no `--select` runs the default rule set (E4/E7/E9/F), and
neither `I` (isort) nor `SIM` is in it. There is no `[tool.ruff]` in
`pyproject.toml` and no `ruff.toml`, so every local run in this branch's
history measured the default set and reported "36, unchanged" against the base.
That number was true and I reported it as if it meant CI would be clean, which
it does not. Checked from now on with `--select I,SIM` as well.
--- scope
`--fix` over the whole tree r…
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.