From 5bb6c10aacd1c422bb855bf517640b5941ee79a2 Mon Sep 17 00:00:00 2001 From: zejunchen-zejun Date: Thu, 27 Aug 2026 15:52:15 +0800 Subject: [PATCH 001/151] fix(offload): the generic fixes #1960 was holding behind the state tier Split out of PR #1960 so they can land on their own. None of them is about state offload, and none needs #2045; blocking them behind either is pure loss, and pulling them out narrows what is left to review by a few hundred lines. --- a worker that outlives the join keeps the node `EngineCore.exit` joined each worker with a 5s timeout and did nothing after it. A worker that outlives that join keeps its VRAM slice and its all-reduce IPC handles, and `multiprocessing`'s atexit handler then joins the same process again with NO timeout -- so an engine already on its way out parks in `_exit_function -> join` forever while /metrics keeps answering 200. Seen twice on the k3-dev line; only a manual kill recovered the node. `enable_orphan_reaping` cannot help: `PR_SET_PDEATHSIG` fires when the parent *dies*, and this parent never does. Now escalates terminate -> kill. --- a stalled offload save hung the engine with every GPU idle A deferred save's blocks are freed only on `finished_saving`, but LMCache's pin monitor force-unpins a stalled transfer after `pin_timeout_sec` WITHOUT emitting that report. `has_pending_kv_work()` then never clears and the engine busy-loops with every GPU idle -- a hard hang under a tight pool, a slow block leak under a large one. `Scheduler._reconcile_stalled_deferred_saves`, wired into the KV progress poll, reclaims a save deferred past an abandon window and self-throttles to 5s so a 1ms poll costs nothing. The window reads LMCache's own `LMCACHE_EC_PIN_TIMEOUT_SEC` and adds 30s rather than taking an ATOM knob of its own, because that ordering IS the safety argument: two independent env vars would let ours sit below the timeout it must exceed, and nothing would say so. Unset gives 330s. Non-positive disables reclamation and restores the wait-forever behaviour. What makes reclaiming safe is not the clock -- our clock starts when the free is deferred, LMCache's when it pins, and nothing aligns them. It is that `OffloadWorkerMixin._guard` reports `finished_saving` on both the success and the exception path, so a report is lost only when `store()` neither returns nor raises. Two cases follow and they are exhaustive: the parked save is not copying (LMCache force-unpinned its source and stopped reading), and a save queued behind a parked one never reached `store()` at all. `finished_sending`'s `assert seq is not None` becomes a `continue`: after a reclaim, a late completion report has nothing left to free. --- a state-only step under `multi` parked its request forever `connector_metadata_has_work` decides whether a step's metadata reaches the worker, and it decided from a hardcoded field list shared by three connector families. `MultiConnectorMetadata` holds none of the work itself -- everything is in `metas` -- so it mirrored the subs' fields as aggregating properties, and missed one. Every step whose only work was in an unmirrored field was discarded, and the request parked on it waited for a report nobody was asked to produce. Fixed by asking the metadata instead of inspecting it: each class declares its own `WORK_FIELDS` next to the fields it defines, `has_work()` accumulates down the hierarchy, and `MultiConnectorMetadata` delegates to its subs rather than mirroring them -- so a field added to a sub can no longer be missed here. The old union list stays only as the fallback for duck-typed doubles, marked as not the place to register anything new. --- two behaviour-neutral seams and one marker `OffloadSchedulerMixin._claim_after_load` returns `max(hbm, lmc)`, which is what all four inlined call sites did. A seam rather than four `max`es because a layout whose claim and transfer aim at different boundaries has to override it, and getting that wrong is silent: the request starts further along than its state supports and the output is wrong with no exception. `DenseOffloadScheduler._may_emit_save` returns True. It matters for a layout whose `should_defer_free` pins a finished request's blocks until its save drains -- an unbounded queue then lets a slow backend hold an unbounded slice of the pool. `max_pending_saves` moves to `_offload_common` so every layout reads the same `OFFLOAD_MAX_PENDING_SAVES`. `KVCacheTensor.per_request_state` marks a hybrid's slot-addressed recurrent state. It must be registered in `kv_cache_data` -- the linear-attention forward reads it from there -- but no block-addressed mover may touch it, and `DenseKVByteCodec` now skips it: including it either fails the divisibility check or, if the slot count happens to divide `num_blocks`, inflates `bytes_per_block` past what `block_regions` describes. Verified on top of #2045 (`4df58434`): pytest tests/ (less tests/plugin and the two msgpack/aiter collection errors that fail on #2045 too) -> 3871 passed, 326 skipped, 3 xfailed, zero failures. black clean; ruff 36, the same count as #2045. Co-Authored-By: Claude Opus 5 (1M context) --- atom/config.py | 6 + .../disaggregation/multi/multi_connector.py | 15 +++ atom/kv_transfer/disaggregation/types.py | 59 +++++++-- atom/kv_transfer/offload/_offload_common.py | 42 +++++- atom/kv_transfer/offload/dense/connector.py | 13 +- .../offload/dense/kv_byte_codec.py | 9 ++ atom/kv_transfer/offload/metadata.py | 16 +++ atom/model_engine/engine_core.py | 35 +++++ atom/model_engine/scheduler.py | 125 +++++++++++++++++- atom/model_ops/attentions/gdn_attn.py | 4 + .../model_ops/attentions/kimi_mla_gdn_attn.py | 4 + atom/plugin/rtpllm/utils/forward_context.py | 3 + .../sglang/attention_backend/attention_gdn.py | 3 + tests/test_lmcache_offload_connector.py | 17 +++ tests/test_multi_connector.py | 41 ++++++ tests/test_scheduler.py | 90 +++++++++++++ 16 files changed, 465 insertions(+), 17 deletions(-) diff --git a/atom/config.py b/atom/config.py index 93c2686156..7d68b300af 100644 --- a/atom/config.py +++ b/atom/config.py @@ -54,6 +54,12 @@ class KVCacheTensor: replay_buf_k: torch.Tensor = None replay_buf_u: torch.Tensor = None replay_buf_g: torch.Tensor = None + # True when the tensors above are a hybrid's PER-REQUEST state (GDN/KDA + # recurrent state, the V4 compressor ring) rather than paged KV. They belong + # in ``kv_cache_data`` -- the linear-attention forward reads them from there + # -- but are addressed by request slot, so no block-addressed mover may + # touch them. The dense offload codec skips them for that reason. + per_request_state: bool = False @dataclass diff --git a/atom/kv_transfer/disaggregation/multi/multi_connector.py b/atom/kv_transfer/disaggregation/multi/multi_connector.py index d655d73650..3778f11d88 100644 --- a/atom/kv_transfer/disaggregation/multi/multi_connector.py +++ b/atom/kv_transfer/disaggregation/multi/multi_connector.py @@ -68,6 +68,7 @@ KVConnectorOutput, SaveCompletionId, completion_req_key, + connector_metadata_has_work, ) logger = logging.getLogger("atom") @@ -160,6 +161,20 @@ def __init__(self, metas: list) -> None: super().__init__() self.metas = list(metas) + def has_work(self) -> bool: + """Ask the subs; this wrapper holds none of the work itself. + + Its own base fields are always empty -- everything lives in `metas` -- + so answering from them alone drops every step whose only work belongs + to a sub. Delegating rather than mirroring the subs' fields is the + point: the aggregating properties below exist for the idle-dispatch + path and have to name each field, so one added to a sub and forgotten + here is a permanently parked request, not a wasted step. + """ + return super().has_work() or any( + connector_metadata_has_work(m) for m in self.metas + ) + @property def requests(self): """Aggregate of sub-metas' ``requests`` (offload uses this attribute). diff --git a/atom/kv_transfer/disaggregation/types.py b/atom/kv_transfer/disaggregation/types.py index 3f00478e8d..37894cb664 100644 --- a/atom/kv_transfer/disaggregation/types.py +++ b/atom/kv_transfer/disaggregation/types.py @@ -271,6 +271,22 @@ class ConnectorMetadata: and the worker-side connector consumes it in ``start_load_kv``. """ + #: Attributes whose truthiness means "the worker has something to do this + #: step". A subclass **extends** this rather than replacing it, and owns + #: only its own fields -- which is the point: the engine drops metadata + #: that reports no work, so a field added to a subclass and not listed + #: here leaves every request parked against it waiting for a report nobody + #: was asked to produce. Keeping the list next to the fields it names is + #: what makes that omission local instead of a shared table three + #: connector families have to remember to edit. + WORK_FIELDS: tuple[str, ...] = ( + "reqs_to_recv", + "reqs_to_save", + "reqs_to_send", + "reqs_in_batch", + "reqs_not_processed", + ) + def __init__(self) -> None: self.reqs_to_recv: dict[ReqId, ReqMeta] = {} self.reqs_to_save: dict[ReqId, ReqMeta] = {} @@ -279,6 +295,10 @@ def __init__(self) -> None: self.reqs_not_processed: set[ReqId] = set() self.request_id_to_transfer_id: dict[ReqId, int] = {} + def has_work(self) -> bool: + """Whether the worker has anything to do with this snapshot.""" + return any(bool(getattr(self, name, None)) for name in self.WORK_FIELDS) + @staticmethod def _build_req_meta( req_id: ReqId, @@ -343,19 +363,32 @@ def completion_req_key(completion: ConnectorCompletionId) -> str: return str(getattr(completion, "req_id", completion)) +#: Fallback for objects that are not `ConnectorMetadata` -- test doubles and +#: duck-typed sub-metas. Real metadata answers through `has_work`; this list is +#: deliberately not the place to register a new field. +_DUCK_TYPED_WORK_FIELDS = ( + "requests", + "state_loads", + "lookup_requests_in_step", + "reqs_to_recv", + "reqs_to_save", + "reqs_to_send", + "reqs_in_batch", + "reqs_not_processed", +) + + def connector_metadata_has_work(metadata: object | None) -> bool: - """Return whether connector metadata contains dispatchable work.""" + """Return whether connector metadata contains dispatchable work. + + Asks the metadata rather than inspecting it, so that each connector family + declares its own work fields next to where it defines them. The engine + drops a snapshot that reports nothing, so an unreported field is a + permanently parked request, not a wasted step. + """ if metadata is None: return False - return any( - bool(getattr(metadata, name, None)) - for name in ( - "requests", - "lookup_requests_in_step", - "reqs_to_recv", - "reqs_to_save", - "reqs_to_send", - "reqs_in_batch", - "reqs_not_processed", - ) - ) + probe = getattr(metadata, "has_work", None) + if callable(probe): + return bool(probe()) + return any(bool(getattr(metadata, name, None)) for name in _DUCK_TYPED_WORK_FIELDS) diff --git a/atom/kv_transfer/offload/_offload_common.py b/atom/kv_transfer/offload/_offload_common.py index 29b7fd9fc4..94856d4764 100644 --- a/atom/kv_transfer/offload/_offload_common.py +++ b/atom/kv_transfer/offload/_offload_common.py @@ -391,6 +391,21 @@ def _maybe_start_unaligned_handoff( ) return True + def _claim_after_load(self, seq, hbm: int, lmc: int) -> int: + """How far the request may call itself cached once the load lands. + + The transfer's end, for every layout whose claim and transfer share a + boundary. A layout that aims them at different places overrides this -- + `kimi_k3` transfers the chunk *covering* its state boundary and may only + claim the boundary, and claiming the rounded-up figure there would have + the forward skip tokens the recurrent state does not cover. + + A seam rather than four inlined `max`es because getting it wrong is + silent: the request simply starts further along than its state + supports, and the output is wrong with no exception anywhere. + """ + return max(int(hbm), int(lmc)) + def should_park_partial_prefill_for_load(self, seq) -> bool: if not self._do_load: return False @@ -417,7 +432,7 @@ def should_park_partial_prefill_for_load(self, seq) -> bool: load_spec.can_load = True self._reqs_need_recv[sid] = seq self._handoff_loads.discard(sid) - seq.offload_loaded_tokens = max(hbm, lmc) + seq.offload_loaded_tokens = self._claim_after_load(seq, hbm, lmc) logger.debug( "[OFFLOAD-LOAD-HANDOFF-READY] seq=%s hbm_cached=%d " "lmc_cached=%d offload_loaded=%d need=%d", @@ -471,7 +486,7 @@ def should_park_for_load_after_alloc(self, seq) -> bool: self._mark_load_skip(seq, reason, hbm, lmc, need, chunk) self._clear_pending_load(sid) return False - seq.offload_loaded_tokens = max(hbm, lmc) + seq.offload_loaded_tokens = self._claim_after_load(seq, hbm, lmc) return True def _save_frontier(self, seq) -> int: @@ -493,3 +508,26 @@ def _has_active_load(self, seq) -> bool: active = self._active_load_operations.get(str(seq.id)) return active is not None and active[0] is seq + + +def max_pending_saves(kvc, save_workers: int) -> int: + """Return the maximum running-plus-queued worker save operations.""" + + extra = (kvc or {}).get("kv_connector_extra_config", kvc or {}) or {} + configured = extra.get("max_pending_saves") + if configured is None: + configured = os.environ.get( + "OFFLOAD_MAX_PENDING_SAVES", + str(max(2, 2 * save_workers)), + ) + try: + capacity = int(configured) + except (TypeError, ValueError) as exc: + raise ValueError("max pending saves must be a positive integer") from exc + else: + if isinstance(configured, bool) or not isinstance(configured, int): + raise ValueError("max pending saves must be a positive integer") + capacity = configured + if capacity <= 0: + raise ValueError("max pending saves must be a positive integer") + return capacity diff --git a/atom/kv_transfer/offload/dense/connector.py b/atom/kv_transfer/offload/dense/connector.py index 070b6ff2b3..0e8a876a12 100644 --- a/atom/kv_transfer/offload/dense/connector.py +++ b/atom/kv_transfer/offload/dense/connector.py @@ -546,6 +546,15 @@ def adjust_prefill_chunk_after_alloc(self, seq, chunk: int) -> int: adjusted = min(int(chunk), limit) return max(1, adjusted) + def _may_emit_save(self) -> bool: + """Seam for a subclass to bound how many saves may be outstanding. + + Unbounded here. It matters for a layout whose `should_defer_free` pins + a finished request's blocks until its save drains -- an unbounded queue + then lets a slow backend hold an unbounded slice of the pool. + """ + return True + def build_connector_meta(self) -> LMCacheOffloadMetadata: meta = LMCacheOffloadMetadata() @@ -578,7 +587,7 @@ def build_connector_meta(self) -> LMCacheOffloadMetadata: self._clear_pending_load(sid) continue # num_cached after load = max(HBM, offload); never drop below HBM. - seq.offload_loaded_tokens = max(hbm, lmc) + seq.offload_loaded_tokens = self._claim_after_load(seq, hbm, lmc) # req_id MUST be the raw seq.id (the type the scheduler compares # against in _update_waiting_for_remote_kv); str(seq.id) is only for # LMCache's lookup/pin API. A str here silently never wakes the seq. @@ -617,6 +626,8 @@ def build_connector_meta(self) -> LMCacheOffloadMetadata: for sid, entry in self._save_tracker.items(): if not self._do_save: continue + if not self._may_emit_save(): + break seq, saved = entry if sid in self._reqs_need_recv or sid in loading_sids: continue # loading this step; defer its save diff --git a/atom/kv_transfer/offload/dense/kv_byte_codec.py b/atom/kv_transfer/offload/dense/kv_byte_codec.py index 3e6676d689..bf8e6a99c4 100644 --- a/atom/kv_transfer/offload/dense/kv_byte_codec.py +++ b/atom/kv_transfer/offload/dense/kv_byte_codec.py @@ -64,6 +64,15 @@ def __init__(self, kv_caches: dict, num_blocks: int | None = None) -> None: preserving the original non-MLA behaviour.""" self._segments: list[torch.Tensor] = [] for kvt in kv_caches.values(): + # A hybrid registers its per-request recurrent state in the same + # dict, because the linear-attention forward reads it from + # `kv_cache_data`. It is indexed by request slot, not by block, so + # including it either fails the divisibility check below or -- if + # the slot count happens to divide `num_blocks` -- inflates + # `bytes_per_block` past what `block_regions` describes. The tier + # reaches those bytes through `state_entry_views`. + if getattr(kvt, "per_request_state", False): + continue for t in ( getattr(kvt, "k_cache", None), getattr(kvt, "v_cache", None), diff --git a/atom/kv_transfer/offload/metadata.py b/atom/kv_transfer/offload/metadata.py index 81b46b8344..071ec0d7f6 100644 --- a/atom/kv_transfer/offload/metadata.py +++ b/atom/kv_transfer/offload/metadata.py @@ -156,11 +156,27 @@ class LMCacheOffloadMetadata(ConnectorMetadata): descriptors the worker consumes in ``start_load_kv``. """ + #: `state_loads` is the one that is easy to miss: it carries no + #: `LMCacheReqMeta`, so a step whose only work is a state load looks empty + #: to anything that only counts requests -- and the requests parked on + #: those loads are woken by nothing but the report they would never be + #: asked to produce. + WORK_FIELDS = ConnectorMetadata.WORK_FIELDS + ( + "requests", + "lookup_requests_in_step", + "state_loads", + ) + def __init__(self) -> None: super().__init__() self.requests: list[LMCacheReqMeta] = [] # req_ids whose worker-side lookup pin can be released this step. self.lookup_requests_in_step: list[str] = [] + # (req_id, state_hash, target_group) for the K3 state tier. A separate + # list because a state load shares no shape with a KV transfer -- no + # token ids, no block ids, no chunking -- only the park/report + # lifecycle, which is why it rides the metadata rather than the batch. + self.state_loads: list[tuple] = [] def add_request(self, meta: LMCacheReqMeta) -> None: self.requests.append(meta) diff --git a/atom/model_engine/engine_core.py b/atom/model_engine/engine_core.py index a43ab8123b..1dba21a4d5 100644 --- a/atom/model_engine/engine_core.py +++ b/atom/model_engine/engine_core.py @@ -267,6 +267,35 @@ def exit(self): continue # process object already closed by CoreManager if alive: proc.join(timeout=5) + # The join above has a timeout; nothing after it did. A worker + # that outlives it keeps its VRAM slice and its all-reduce IPC + # handles, and `multiprocessing`'s atexit handler then joins the + # same process again with NO timeout -- so an engine that is + # already on its way out hangs there forever. Seen twice on the + # k3-dev line: the MainThread parks in `_exit_function -> join`, + # all TP workers stay alive, /metrics keeps answering 200, and + # only a manual kill recovers the node. `enable_orphan_reaping` + # cannot help, because PR_SET_PDEATHSIG fires when the parent + # *dies* and this parent never does. + try: + if proc.is_alive(): + logger.warning( + "%s: worker pid=%s still alive after 5s; terminating", + self.label, + getattr(proc, "pid", "?"), + ) + proc.terminate() + proc.join(timeout=5) + if proc.is_alive(): + logger.error( + "%s: worker pid=%s ignored SIGTERM; killing", + self.label, + getattr(proc, "pid", "?"), + ) + proc.kill() + proc.join(timeout=5) + except (ValueError, OSError): + pass # process object already closed / already reaped self._send_engine_dead() logger.debug(f"{self.label}: model runner exit") @@ -487,6 +516,12 @@ def _poll_kv_transfer_progress(self) -> None: return kvoutput = self.runner_mgr.call_func_with_aggregation("async_proc_aggregation") self.scheduler._update_from_kv_xfer_finished(kvoutput) + # Reclaim any offload save whose completion report never came (LMCache + # force-unpinned it upstream). Self-throttled, so calling it on every + # poll is cheap; without it a stalled save hangs the engine forever. + reconcile = getattr(self.scheduler, "_reconcile_stalled_deferred_saves", None) + if callable(reconcile): + reconcile() def _dispatch_idle_offload_work(self) -> None: if not self.kv_transfer_enabled: diff --git a/atom/model_engine/scheduler.py b/atom/model_engine/scheduler.py index 2073d1c4e0..30afe89607 100644 --- a/atom/model_engine/scheduler.py +++ b/atom/model_engine/scheduler.py @@ -18,6 +18,7 @@ from __future__ import annotations import logging +import os import struct import threading import time @@ -46,6 +47,73 @@ logger = logging.getLogger("atom") +# How often the stalled-save reconciler actually scans deferred_free_blocks. The +# engine polls KV progress every millisecond (KV_IDLE_DRAIN_INTERVAL_S), so the +# reconciler no-ops until this interval has passed. +_SAVE_RECONCILE_INTERVAL_S = 5.0 + +# Seconds added on top of LMCache's own pin timeout before the engine reclaims a +# save that never reported. See `_offload_save_abandon_timeout_s` for why the sum +# and not the timeout itself is the safe window. +_SAVE_ABANDON_MARGIN_S = 30.0 + +# What LMCache's pin monitor uses when `LMCACHE_EC_PIN_TIMEOUT_SEC` is unset. +_LMCACHE_PIN_TIMEOUT_DEFAULT_S = 300.0 + +# Memoised result of `_offload_save_abandon_timeout_s`; the reconciler asks for +# it on a 1ms poll and the answer is a process constant. +_SAVE_ABANDON_TIMEOUT_S: float | None = None + + +def _offload_save_abandon_timeout_s() -> float: + """Seconds a finished request's blocks may sit deferred on an offload save + before the engine reclaims them itself. + + A deferred save's blocks are freed only when the connector reports + `finished_saving`, so a lost report leaves them deferred forever: + `has_pending_kv_work()` never clears and the engine busy-loops with every + GPU idle -- a hard hang under a tight pool, a slow block leak under a large + one. + + **Why reclaiming cannot race a live copy.** `OffloadWorkerMixin._guard` + reports `finished_saving` on both the success and the exception path, so a + report is lost only when `self._engine.store(...)` neither returns nor + raises -- it is parked inside LMCache. Two things follow, and together they + are exhaustive: + + * the save that is parked is not copying. LMCache's pin monitor + force-unpins its source after `pin_timeout_sec` and stops reading the + blocks; reclaiming after that window therefore frees blocks nobody + holds. + * a save queued behind a parked one never reaches `store()` at all, so + its blocks were never read to begin with. + + Derived from LMCache's own `LMCACHE_EC_PIN_TIMEOUT_SEC` rather than an ATOM + knob of its own, because the ordering above is the whole safety argument: a + second env var could be set below the timeout it must exceed, and nothing + would say so. Raising LMCache's pin timeout now carries this window with it. + A non-positive value disables reclamation and restores the wait-forever + behaviour. + """ + global _SAVE_ABANDON_TIMEOUT_S + if _SAVE_ABANDON_TIMEOUT_S is not None: + return _SAVE_ABANDON_TIMEOUT_S + pin = _LMCACHE_PIN_TIMEOUT_DEFAULT_S + raw = os.environ.get("LMCACHE_EC_PIN_TIMEOUT_SEC") + if raw is not None: + try: + pin = float(raw) + except ValueError: + logger.warning( + "invalid LMCACHE_EC_PIN_TIMEOUT_SEC=%r; assuming LMCache's %.0fs " + "default for the offload save abandon window", + raw, + _LMCACHE_PIN_TIMEOUT_DEFAULT_S, + ) + _SAVE_ABANDON_TIMEOUT_S = pin + _SAVE_ABANDON_MARGIN_S if pin > 0 else 0.0 + return _SAVE_ABANDON_TIMEOUT_S + + class SpecStats: """Tracks speculative decoding acceptance statistics.""" @@ -938,6 +1006,10 @@ def __init__( self.finished_recving_kv_req_ids: list[int] = [] self.failed_recving_kv_req_ids: list[int] = [] self.deferred_free_blocks: dict[int, Sequence] = {} + # Reclamation bookkeeping for offload saves whose completion is never + # reported (see `_reconcile_stalled_deferred_saves`). + self._abandoned_saves: int = 0 + self._next_save_reconcile_at: float = 0.0 # Scheduling delay for batching efficiency self.prev_time = 0.0 @@ -1265,6 +1337,51 @@ def _maybe_release_deferred(self, seq: Sequence) -> None: self.deferred_free_blocks.pop(seq.id, None) self.block_manager.deallocate(seq) + def _reconcile_stalled_deferred_saves(self) -> int: + """Reclaim blocks whose offload save has stalled past the abandon timeout. + + `_maybe_release_deferred` cannot do this: the connector still reports the + save as pending (`should_defer_free` stays True), which is exactly why + the blocks were never released. Once the save has been deferred longer + than `_offload_save_abandon_timeout_s()` -- set above LMCache's pin + timeout, so upstream has force-unpinned and released the source bytes -- + the blocks are safe to return to the pool. Without this, a single + never-reported save keeps `has_pending_kv_work()` True forever and the + engine busy-loops with every GPU idle. + + Mirrors the producer `finished_sending` reclaim (pop + deallocate), + deliberately not re-invoking `request_finished`: it was already called + when the request finished, before the block free was deferred. + """ + timeout = _offload_save_abandon_timeout_s() + if timeout <= 0 or not self.deferred_free_blocks: + return 0 + now = time.monotonic() + if now < self._next_save_reconcile_at: + return 0 + self._next_save_reconcile_at = now + _SAVE_RECONCILE_INTERVAL_S + stalled = [ + seq + for seq in list(self.deferred_free_blocks.values()) + if getattr(seq, "_deferred_save_at", None) is not None + and now - seq._deferred_save_at >= timeout + ] + for seq in stalled: + self.deferred_free_blocks.pop(seq.id, None) + self.block_manager.deallocate(seq) + self._abandoned_saves += 1 + if stalled: + logger.warning( + "Reclaimed %d offload save(s) still deferred after %.0fs with no " + "completion report (LMCache force-unpins a stalled save without " + "reporting it); freed their blocks so the engine does not stall. " + "total abandoned_saves=%d", + len(stalled), + timeout, + self._abandoned_saves, + ) + return len(stalled) + def _unschedulable_reason(self, seq: Sequence) -> str | None: """Return a human-readable reason if `seq` is permanently unschedulable. @@ -2783,6 +2900,9 @@ def postprocess( "Deferring block free for seq %s until KV save completes.", seq.id, ) + # Stamp when the save was deferred so the reconciler can + # reclaim it if the completion report never arrives. + seq._deferred_save_at = time.monotonic() self.deferred_free_blocks[seq.id] = seq else: self.block_manager.deallocate(seq) @@ -3047,7 +3167,10 @@ def _update_from_kv_xfer_finished(self, kv_connector_output: KVConnectorOutput): ), "Only producer should free blocks after sending KV" logger.debug("Finished sending KV transfer for request %s", req_id) seq = self._deferred_sequence(req_id) - assert seq is not None, f"req_id={req_id} not found in deferred_free_blocks" + if seq is None: + # Already reclaimed by `_reconcile_stalled_deferred_saves` after + # a stall; a late completion report has nothing left to free. + continue self.deferred_free_blocks.pop(seq.id, None) self.block_manager.deallocate(seq) diff --git a/atom/model_ops/attentions/gdn_attn.py b/atom/model_ops/attentions/gdn_attn.py index b88f7f5b65..17a72d70cf 100644 --- a/atom/model_ops/attentions/gdn_attn.py +++ b/atom/model_ops/attentions/gdn_attn.py @@ -1408,6 +1408,10 @@ def build_kv_cache_tensor(self, layer_id: int, module): replay_buf_g=( runner.replayssm_buf_g[gdn_idx] if self.replayssm else None ), + # Slot-addressed recurrent state, not paged KV. It has to be + # registered (the linear-attention forward reads its state out + # of `kv_cache_data`), but no block-addressed mover may touch it. + per_request_state=True, ) return super().build_kv_cache_tensor(layer_id, module) diff --git a/atom/model_ops/attentions/kimi_mla_gdn_attn.py b/atom/model_ops/attentions/kimi_mla_gdn_attn.py index 083c271b2f..e6e6cd4fb7 100644 --- a/atom/model_ops/attentions/kimi_mla_gdn_attn.py +++ b/atom/model_ops/attentions/kimi_mla_gdn_attn.py @@ -243,6 +243,10 @@ def build_kv_cache_tensor(self, layer_id: int, module): replay_buf_k=(runner.replayssm_buf_k[row] if self.replayssm else None), replay_buf_u=(runner.replayssm_buf_u[row] if self.replayssm else None), replay_buf_g=(runner.replayssm_buf_g[row] if self.replayssm else None), + # KDA recurrent state: slot-addressed, not paged. Registered + # because the forward reads it from `kv_cache_data`, but + # excluded from every block-addressed transfer. + per_request_state=True, ) if hasattr(module, "base_attention") and getattr(module, "use_mla", False): diff --git a/atom/plugin/rtpllm/utils/forward_context.py b/atom/plugin/rtpllm/utils/forward_context.py index 5d0d0861bd..a388b8ee14 100644 --- a/atom/plugin/rtpllm/utils/forward_context.py +++ b/atom/plugin/rtpllm/utils/forward_context.py @@ -1450,6 +1450,9 @@ def _build_kv_cache_tensors( v_cache=ssm_state, k_scale=None, v_scale=None, + # Slot-addressed recurrent state, not paged KV -- see + # `KVCacheTensor.per_request_state`. + per_request_state=True, ) # Build full-attn cache references from RTP LayerKVCache. diff --git a/atom/plugin/sglang/attention_backend/attention_gdn.py b/atom/plugin/sglang/attention_backend/attention_gdn.py index d65242c9af..15e7efd9cf 100644 --- a/atom/plugin/sglang/attention_backend/attention_gdn.py +++ b/atom/plugin/sglang/attention_backend/attention_gdn.py @@ -431,6 +431,9 @@ def _build_kv_cache_tensors( v_cache=layer_cache.temporal, k_scale=None, v_scale=None, + # Slot-addressed recurrent state, not paged KV -- see + # `KVCacheTensor.per_request_state`. + per_request_state=True, ) return out diff --git a/tests/test_lmcache_offload_connector.py b/tests/test_lmcache_offload_connector.py index d160b433b6..996313934a 100644 --- a/tests/test_lmcache_offload_connector.py +++ b/tests/test_lmcache_offload_connector.py @@ -4624,3 +4624,20 @@ def test_multi_metadata_exposes_sub_meta_unpins(): assert not connector_metadata_has_work( MultiConnectorMetadata(metas=[LMCacheOffloadMetadata()]) ) + + +def test_a_layout_whose_claim_and_transfer_share_a_boundary_claims_the_end(): + """The base seam: claim as far as the transfer reached, never below HBM. + + Correct for every layout that aims its claim and its transfer at the same + place, which is dense and dsv4. A layout that aims them apart overrides it; + the seam exists because getting it wrong is silent -- the request simply + starts further along than its state supports. + """ + from atom.kv_transfer.offload.dense.connector import DenseOffloadScheduler + + sched = object.__new__(DenseOffloadScheduler) + seq = SimpleNamespace(id=1) + assert sched._claim_after_load(seq, 4096, 8192) == 8192 + # Never below the HBM floor, whatever the transfer reported. + assert sched._claim_after_load(seq, 4096, 0) == 4096 diff --git a/tests/test_multi_connector.py b/tests/test_multi_connector.py index 62d8289715..0e54cf1a10 100644 --- a/tests/test_multi_connector.py +++ b/tests/test_multi_connector.py @@ -527,3 +527,44 @@ def test_real_constructor_populates_the_pairing_state(monkeypatch, pp_rank, hold w.start_load_kv(MultiConnectorMetadata([ConnectorMetadata(), _save_meta(9)])) out = w.get_finished() assert out.finished_sending == (set() if holds_send else {9}) + + +def test_a_state_only_step_under_multi_is_not_dropped(): + """A step whose only work is a state load must reach the worker. + + `state_loads` carries no `LMCacheReqMeta`, so a wrapper that answers from + its own (always empty) fields reports no work, the engine drops the + snapshot, and the request parked on that load is woken by nothing. + """ + from atom.kv_transfer.disaggregation.types import connector_metadata_has_work + from atom.kv_transfer.offload.metadata import LMCacheOffloadMetadata + + sub = LMCacheOffloadMetadata() + sub.state_loads = [("req-1", 12345, 7)] + + assert connector_metadata_has_work(sub) + assert connector_metadata_has_work(MultiConnectorMetadata([sub])) + + +def test_a_multi_wrapper_over_idle_subs_still_reports_no_work(): + from types import SimpleNamespace + + from atom.kv_transfer.disaggregation.types import connector_metadata_has_work + from atom.kv_transfer.offload.metadata import LMCacheOffloadMetadata + + assert not connector_metadata_has_work( + MultiConnectorMetadata([LMCacheOffloadMetadata(), SimpleNamespace()]) + ) + + +def test_every_metadata_field_a_subclass_adds_is_declared_work_or_not(): + """`WORK_FIELDS` must name real attributes, or a typo silences a field. + + A misspelled entry is invisible: `getattr` returns None, the field never + counts, and the only symptom is a parked request much later. + """ + from atom.kv_transfer.offload.metadata import LMCacheOffloadMetadata + + meta = LMCacheOffloadMetadata() + for name in LMCacheOffloadMetadata.WORK_FIELDS: + assert hasattr(meta, name), f"WORK_FIELDS names a missing attribute: {name}" diff --git a/tests/test_scheduler.py b/tests/test_scheduler.py index 4b8c1837ff..3a08d52f08 100644 --- a/tests/test_scheduler.py +++ b/tests/test_scheduler.py @@ -1367,3 +1367,93 @@ def test_decode_counts_scheduled_query_tokens(self): assert batch.detailed_sqsq == 9 # 3^2 assert batch.detailed_sqsk == 300 # 3 * 100 assert batch.detailed_sk == 100 + + +class TestStalledOffloadSaveReclaim: + """`_reconcile_stalled_deferred_saves`: the way out for a save nobody answers. + + LMCache's pin monitor force-unpins a stalled transfer without emitting a + completion, so `should_defer_free` stays True forever, `has_pending_kv_work()` + never clears, and the engine busy-loops with every GPU idle. Reproduced on + the k3-dev line as a hard hang under a tight pool. + """ + + @staticmethod + def _sched(monkeypatch, deferred): + import atom.model_engine.scheduler as sched_mod + + # The window is memoised as a process constant; tests choose their own. + monkeypatch.setattr(sched_mod, "_SAVE_ABANDON_TIMEOUT_S", 100.0) + s = object.__new__(sched_mod.Scheduler) + s.deferred_free_blocks = {seq.id: seq for seq in deferred} + s._abandoned_saves = 0 + s._next_save_reconcile_at = 0.0 + freed: list[int] = [] + s.block_manager = SimpleNamespace(deallocate=lambda q: freed.append(q.id)) + return s, freed + + def test_a_save_past_the_window_gets_its_blocks_back(self, monkeypatch): + import time as _time + + now = _time.monotonic() + stale = SimpleNamespace(id=1, _deferred_save_at=now - 500.0) + fresh = SimpleNamespace(id=2, _deferred_save_at=now) + s, freed = self._sched(monkeypatch, [stale, fresh]) + + assert s._reconcile_stalled_deferred_saves() == 1 + assert freed == [1], "only the stalled save is reclaimed" + assert 2 in s.deferred_free_blocks, "a save still inside its window is kept" + assert s._abandoned_saves == 1 + + def test_it_self_throttles_so_a_1ms_poll_is_cheap(self, monkeypatch): + import time as _time + + stale = SimpleNamespace(id=1, _deferred_save_at=_time.monotonic() - 500.0) + s, freed = self._sched(monkeypatch, [stale]) + + assert s._reconcile_stalled_deferred_saves() == 1 + # Second call inside the throttle interval must not rescan. + s.deferred_free_blocks = { + 9: SimpleNamespace(id=9, _deferred_save_at=_time.monotonic() - 500.0) + } + assert s._reconcile_stalled_deferred_saves() == 0 + assert freed == [1] + + def test_a_non_positive_window_restores_wait_forever(self, monkeypatch): + import atom.model_engine.scheduler as sched_mod + import time as _time + + monkeypatch.setattr(sched_mod, "_SAVE_ABANDON_TIMEOUT_S", 0.0) + s = object.__new__(sched_mod.Scheduler) + s.deferred_free_blocks = { + 1: SimpleNamespace(id=1, _deferred_save_at=_time.monotonic() - 1e6) + } + s._abandoned_saves = 0 + s._next_save_reconcile_at = 0.0 + s.block_manager = SimpleNamespace(deallocate=lambda q: pytest.fail("reclaimed")) + + assert s._reconcile_stalled_deferred_saves() == 0 + + def test_the_window_sits_above_lmcaches_own_pin_timeout(self, monkeypatch): + """Deriving it from LMCache's knob IS the safety argument. + + Two independent env vars would let ours be set below the timeout it has + to exceed, and nothing would say so. + """ + import atom.model_engine.scheduler as sched_mod + + monkeypatch.setattr(sched_mod, "_SAVE_ABANDON_TIMEOUT_S", None) + monkeypatch.setenv("LMCACHE_EC_PIN_TIMEOUT_SEC", "900") + assert sched_mod._offload_save_abandon_timeout_s() == 930.0 + + monkeypatch.setattr(sched_mod, "_SAVE_ABANDON_TIMEOUT_S", None) + monkeypatch.delenv("LMCACHE_EC_PIN_TIMEOUT_SEC", raising=False) + assert sched_mod._offload_save_abandon_timeout_s() == 330.0 + + def test_a_late_report_on_a_reclaimed_save_is_not_an_assertion(self): + """`finished_sending` used to assert the sequence was still deferred.""" + import atom.model_engine.scheduler as sched_mod + import inspect + + src = inspect.getsource(sched_mod.Scheduler._update_from_kv_xfer_finished) + assert "not found in deferred_free_blocks" not in src From 4208e3e1e9aa57e9118f667fea5f816b894a3c6a Mon Sep 17 00:00:00 2001 From: zejunchen-zejun Date: Thu, 27 Aug 2026 16:01:36 +0800 Subject: [PATCH 002/151] lmcache(k3): port the state offload tier onto #2045's per-slot pool Phase 1 of remounting PR #1960 on #2045. Deliberately a RENAME, not a redesign: the tier still hangs off `StateSlotPool` and still spills on eviction through the staging ring. Phase 2 deletes the ring; Phase 3 moves the mount point to `PageUnitCheckpointStore`. Keeping those apart is what lets a reviewer tell rebase noise from design decisions. Squashed rather than replayed for the reason #2045 gives for its own squash: other's resolutions. #1960 keeps the original history. --- what #2045 renamed underneath the branch StateGroupPool -> StateSlotPool lookup_group(h) -> lookup(h) seq.per_req_cache_group -> seq.state_slots (list) + seq.state_slot num_per_req_cache_groups -> num_state_slots + state_slots_per_req pop() / release() -> pop_many(width) / release_many() has_free() -> has_free(n) _attach_state_group() -> _attach_state_slots() The semantic change hides in that table: **the "group" concept is gone.** Slots are allocated one at a time, `pop_many` states outright that the set is not adjacent and nothing may assume it is, and a checkpoint occupies exactly ONE slot -- speculation scratch is not state anybody resumes into. Every `group * span` address computation in #1960 lost its basis, so the tier's `_spill` / `_resumable_from` / `has_pending_spill` / `take_spill_copies` are re-expressed over slots, and `(group, staging)` becomes `(slot, staging)`. --- the three places the two PRs actually collide `_attach_state_slots` now returns bool. #2045 rewrote its body for per-slot allocation; #1960 needs it to be able to say "the state behind this boundary is not really here", because with the tier voting a hash can be accepted by `_resumable_from` and still not be in HBM. The two compose: #2045's width logic runs, and only the committed slot (element 0) is the load's destination, so only it takes `pop(spill=False)` -- the rollback scratch is this request's own and has nothing to forgo. `allocate` keeps #1960's two extents (`num_cached_blocks` vs `claim_blocks`) on top of #2045's per-slot attach, and the fresh-block loop starts from `len(seq.block_table)` rather than `num_cached_blocks`, since the widened claim already put blocks there. `CacheStats` gains #1960's `total_offload_tokens` alongside #2045's `total_reusable_tokens`. They are not the same series and neither subsumes the other: `reusable` is the honest denominator for the HBM walk, while tier tokens sit ABOVE that walk by design, so folding them in would make `cached > wanted`. `update()` therefore takes five required arguments and one defaulted. `deallocate` keeps #2045's midstep cancellation and `release_many`, and #1960's in-flight-load hold: the committed slot is parked in `_orphan_load_slots` until `settle_state_load`, because a worker is writing it; the scratch goes back immediately. `checkpoint_funnel` reports through `state_checkpoint_fates()` (a sum across `state_caches`) rather than the single `_state_checkpoint_cache` call. Today `state_caches` is a 1-tuple and the two are equivalent -- kept so Phase 3's tier counters, which arrive from a second class, are not silently dropped. --- functionality is NOT verified by this commit Three gates all take the else branch for K3 under #2045, so the tier is inert: 1. `isinstance(cache, StateSlotPool)` -- K3's `state_caches` is `(PagedStateCheckpointCoordinator,)`, so `cache.offload` is never set. 2. `_joint_kv_boundary` returns `_no_joint("not_hybrid")` whenever `paged_state_checkpoints is not None`, which for K3 is always. 3. `_attach_state_slots` returns from its PAGE branch before reaching `_tier_can_serve`. That is deliberate and is what Phase 3/5 undo. Do not read the green suite below as evidence the feature works -- the exit criterion for the tier is a nonzero `state_tier` counter in the logs, which no unit test can produce. Verified: pytest tests/ (less tests/plugin and the two msgpack/aiter collection errors that fail on #2045 too) -> 3915 passed, 326 skipped, 3 xfailed, zero failures. black clean; ruff 36, the same count as #2045. Co-Authored-By: Claude Opus 5 (1M context) --- atom/config.py | 3 +- .../disaggregation/multi/multi_connector.py | 50 +- .../offload/atom_lmcache_staging.py | 6 +- atom/kv_transfer/offload/config.py | 9 +- atom/kv_transfer/offload/connector.py | 25 +- .../offload/hybrid/dsv4/connector.py | 28 +- .../offload/hybrid/kimi_k3/__init__.py | 4 + .../offload/hybrid/kimi_k3/connector.py | 484 +++++++++++++++++ .../offload/hybrid/kimi_k3/staging.py | 259 +++++++++ .../offload/hybrid/kimi_k3/state_object.py | 155 ++++++ .../offload/hybrid/kimi_k3/state_tier.py | 233 +++++++++ atom/model_engine/block_manager.py | 495 +++++++++++++++++- atom/model_engine/engine_core.py | 6 + atom/model_engine/llm_engine.py | 7 + atom/model_engine/model_runner.py | 25 +- atom/model_engine/scheduler.py | 263 +++++++++- atom/model_engine/sequence.py | 30 ++ atom/model_engine/state_offload.py | 369 +++++++++++++ atom/model_engine/state_pool.py | 135 ++++- atom/model_engine/state_runtime.py | 11 +- atom/model_ops/attentions/backends.py | 47 ++ atom/model_ops/attentions/deepseek_v4_attn.py | 56 +- atom/model_ops/attentions/gdn_attn.py | 20 + atom/model_ops/attentions/sub_pool_spec.py | 67 ++- docs/environment_variables.md | 51 ++ tests/test_block_manager.py | 152 ++++++ tests/test_lmcache_offload_connector.py | 214 +++++++- tests/test_lmcache_offload_v4_page_slot.py | 10 +- tests/test_multi_connector.py | 145 +++-- tests/test_scheduler.py | 43 ++ tests/test_state_checkpoint.py | 161 ++++++ tests/test_sub_pool_spec.py | 151 ++++++ 32 files changed, 3592 insertions(+), 122 deletions(-) create mode 100644 atom/kv_transfer/offload/hybrid/kimi_k3/__init__.py create mode 100644 atom/kv_transfer/offload/hybrid/kimi_k3/connector.py create mode 100644 atom/kv_transfer/offload/hybrid/kimi_k3/staging.py create mode 100644 atom/kv_transfer/offload/hybrid/kimi_k3/state_object.py create mode 100644 atom/kv_transfer/offload/hybrid/kimi_k3/state_tier.py create mode 100644 atom/model_engine/state_offload.py diff --git a/atom/config.py b/atom/config.py index 7d68b300af..b9c4422fd5 100644 --- a/atom/config.py +++ b/atom/config.py @@ -58,7 +58,8 @@ class KVCacheTensor: # recurrent state, the V4 compressor ring) rather than paged KV. They belong # in ``kv_cache_data`` -- the linear-attention forward reads them from there # -- but are addressed by request slot, so no block-addressed mover may - # touch them. The dense offload codec skips them for that reason. + # touch them. The dense codec skips them; the state tier reaches the same + # bytes through ``state_entry_views``. per_request_state: bool = False diff --git a/atom/kv_transfer/disaggregation/multi/multi_connector.py b/atom/kv_transfer/disaggregation/multi/multi_connector.py index 3778f11d88..ca8e869c74 100644 --- a/atom/kv_transfer/disaggregation/multi/multi_connector.py +++ b/atom/kv_transfer/disaggregation/multi/multi_connector.py @@ -38,6 +38,10 @@ index in ``start_load_kv``. * ``get_finished`` — union the completion sets, **but** see the send/save pairing below. +* ``_state_tier`` — the state offload tier, if a sub built one, re-exposed on + the composite. ``AttentionBackend._submit_state_spills`` reads this attribute + off whatever connector the forward context holds, so without it the engine's + spills are staged and never submitted. Send/save pairing (the one tricky correctness point) ---------------------------------------------------- @@ -168,8 +172,8 @@ def has_work(self) -> bool: so answering from them alone drops every step whose only work belongs to a sub. Delegating rather than mirroring the subs' fields is the point: the aggregating properties below exist for the idle-dispatch - path and have to name each field, so one added to a sub and forgotten - here is a permanently parked request, not a wasted step. + path and have to name each field, and `state_loads` was missed there, + which silently parked every state-only load run under `multi`. """ return super().has_work() or any( connector_metadata_has_work(m) for m in self.metas @@ -236,6 +240,10 @@ def __init__(self, config: Any) -> None: self._pending_save_ops: dict[str, set[SaveCompletionId]] = {} self._sent: dict[str, Any] = {} self._saved: dict[str, set[SaveCompletionId]] = {} + # The state tier of whichever sub owns one. Always defined: + # `_submit_state_spills` probes it every batch, and a composite without + # the attribute would swallow every spill and starve the ring. + self._state_tier = None @property def _pairs_send_and_save(self) -> bool: @@ -255,6 +263,29 @@ def register_kv_caches( ) -> None: for c in self._connectors: c.register_kv_caches(kv_caches, transfer_tensors, num_blocks) + self._adopt_state_tier() + + def _adopt_state_tier(self) -> None: + """Take over the one sub-connector's state tier, or refuse two. + + Nothing in ``_build_subconnectors`` stops a config from listing + ``lmcache_offload`` twice, which would leave two live tiers and no + answer to "which one packs this spill". Picking the first is wrong + rather than arbitrary: a hash could be reported indexed by a tier that + never stored it, then fetched from one that cannot produce it. Raising + at model load costs nothing and is loud. + """ + tiers = [ + c for c in self._connectors if getattr(c, "_state_tier", None) is not None + ] + if len(tiers) > 1: + names = [type(c).__name__ for c in tiers] + raise ValueError( + f"multi connector: {len(tiers)} sub-connectors built a state " + f"offload tier ({names}); exactly one may. List the offload " + "backend once in kv_transfer_config.connectors." + ) + self._state_tier = tiers[0]._state_tier if tiers else None def start_load_kv(self, metadata: ConnectorMetadata) -> None: metas = getattr(metadata, "metas", None) @@ -404,6 +435,21 @@ def adjust_prefill_chunk_after_alloc(self, seq: Any, chunk: int) -> int: c.adjust_prefill_chunk_after_alloc(seq, chunk) if c is not None else chunk ) + def enqueue_state_loads(self, loads) -> bool: + """First sub that can carry them owns them; False if none can. + + Only one sub may host the tier (the worker raises at model load when + two do), so "first" is also "only". + + The False is not a formality: every load here belongs to a parked + request only a report can wake, and the caller's `hasattr` guard cannot + catch a swallowed one because this method always exists. + """ + c = _first_with(self._connectors, "enqueue_state_loads") + if c is None: + return False + return bool(c.enqueue_state_loads(loads)) + def should_park_partial_prefill_for_load(self, seq: Any) -> bool: c = _first_with(self._connectors, "should_park_partial_prefill_for_load") return c.should_park_partial_prefill_for_load(seq) if c is not None else False diff --git a/atom/kv_transfer/offload/atom_lmcache_staging.py b/atom/kv_transfer/offload/atom_lmcache_staging.py index f6ce4ea82a..685651a8ad 100644 --- a/atom/kv_transfer/offload/atom_lmcache_staging.py +++ b/atom/kv_transfer/offload/atom_lmcache_staging.py @@ -33,7 +33,11 @@ def __init__(self, use_cuda: bool) -> None: def _env_flag(name: str, default: str = "0") -> bool: - return os.environ.get(name, default).lower() not in ("0", "false", "no", "off") + # Stripped, and empty reads as off: `VAR=` is how a shell script clears a + # flag inline, and a bare membership test reads the empty string as ON -- + # the opposite of what the operator wrote. `VAR="off "` did the same. + raw = os.environ.get(name, default).strip().lower() + return bool(raw) and raw not in ("0", "false", "no", "off") def _env_int(name: str, default: int, *, min_value: int = 1) -> int: diff --git a/atom/kv_transfer/offload/config.py b/atom/kv_transfer/offload/config.py index 6bf4f573de..17a68e1e6d 100644 --- a/atom/kv_transfer/offload/config.py +++ b/atom/kv_transfer/offload/config.py @@ -31,6 +31,7 @@ _OFFLOAD_LAYOUT_ALIASES = { "hybrid": "hybrid", "dense": "dense", + "kimi_k3": "kimi_k3", # Compatibility with the names used while the layout split was developed. "terminal_unit": "hybrid", "page_slot": "hybrid", @@ -92,7 +93,13 @@ def select_offload_layout(config) -> str: ) hf_config = getattr(config, "hf_config", None) - if hf_config is not None and getattr(hf_config, "compress_ratios", None): + if hf_config is None: + return "dense" + # K3's paged KV is ordinary dense MLA; what makes it its own family is the + # KDA per-request state that a reusable prefix also needs. + if getattr(hf_config, "model_type", None) == "kimi_linear": + return "kimi_k3" + if getattr(hf_config, "compress_ratios", None): return "hybrid" return "dense" diff --git a/atom/kv_transfer/offload/connector.py b/atom/kv_transfer/offload/connector.py index 78e8e2e213..c35ffaab1a 100644 --- a/atom/kv_transfer/offload/connector.py +++ b/atom/kv_transfer/offload/connector.py @@ -7,7 +7,8 @@ both the scheduler and worker from configuration alone: * ``dense`` stores ordinary token-indexed KV chunks; -* ``hybrid`` stores DSV4 compressed PAGE chunks plus complete SLOT sidecars. +* ``hybrid`` stores DSV4 compressed PAGE chunks plus complete SLOT sidecars; +* ``kimi_k3`` stores dense MLA KV plus a KDA per-request state tier. Keeping selection config-only is important because the scheduler process does not have access to the worker's transfer tensors. @@ -42,6 +43,13 @@ def _build_worker(config): return DSV4OffloadConnector(config) + if variant == "kimi_k3": + from atom.kv_transfer.offload.hybrid.kimi_k3.connector import ( + KimiK3OffloadConnector, + ) + + return KimiK3OffloadConnector(config) + from atom.kv_transfer.offload.dense.connector import DenseOffloadConnector return DenseOffloadConnector(config) @@ -57,6 +65,13 @@ def _build_scheduler(config): return DSV4OffloadScheduler(config) + if variant == "kimi_k3": + from atom.kv_transfer.offload.hybrid.kimi_k3.connector import ( + KimiK3OffloadScheduler, + ) + + return KimiK3OffloadScheduler(config) + from atom.kv_transfer.offload.dense.connector import DenseOffloadScheduler return DenseOffloadScheduler(config) @@ -141,6 +156,14 @@ def load_finished(self, req_id): def process_completions(self, output): return self._impl.process_completions(output) + def enqueue_state_loads(self, loads) -> bool: + callback = getattr(self._impl, "enqueue_state_loads", None) + return bool(callback(loads)) if callback is not None else False + + def take_state_reports(self): + callback = getattr(self._impl, "take_state_reports", None) + return callback() if callback is not None else (set(), set(), set()) + def get_statistics(self) -> dict[str, int]: return self._impl.get_statistics() diff --git a/atom/kv_transfer/offload/hybrid/dsv4/connector.py b/atom/kv_transfer/offload/hybrid/dsv4/connector.py index 77ae7a1570..00f8fa3fc1 100644 --- a/atom/kv_transfer/offload/hybrid/dsv4/connector.py +++ b/atom/kv_transfer/offload/hybrid/dsv4/connector.py @@ -52,6 +52,7 @@ from atom.kv_transfer.offload import config as offcfg from atom.kv_transfer.offload._block_gpu_connector import BlockGPUConnector from atom.kv_transfer.offload._offload_common import ( + max_pending_saves, OffloadSchedulerMixin, OffloadWorkerMixin, build_offload_engine, @@ -91,29 +92,6 @@ DSV4_CHECKPOINT_SAVE_CHANNEL = "atom.dsv4.checkpoint.save" -def _max_pending_saves(kvc, save_workers: int) -> int: - """Return the maximum running-plus-queued worker save operations.""" - - extra = (kvc or {}).get("kv_connector_extra_config", kvc or {}) or {} - configured = extra.get("max_pending_saves") - if configured is None: - configured = os.environ.get( - "OFFLOAD_MAX_PENDING_SAVES", - str(max(2, 2 * save_workers)), - ) - try: - capacity = int(configured) - except (TypeError, ValueError) as exc: - raise ValueError("max pending saves must be a positive integer") from exc - else: - if isinstance(configured, bool) or not isinstance(configured, int): - raise ValueError("max pending saves must be a positive integer") - capacity = configured - if capacity <= 0: - raise ValueError("max pending saves must be a positive integer") - return capacity - - def _wait_for_publication( probe, *, @@ -243,7 +221,7 @@ def __init__(self, config) -> None: # The ATOM LMCache GPU connector owns per-thread staging streams. # OFFLOAD_COPY_WORKERS tunes the SAVE pool only. n_save_workers = int(os.environ.get("OFFLOAD_COPY_WORKERS", "1")) - self._max_pending_saves = _max_pending_saves(kvc, n_save_workers) + self._max_pending_saves = max_pending_saves(kvc, n_save_workers) self._save_admission = threading.BoundedSemaphore(self._max_pending_saves) self._init_worker_common( config, @@ -2241,7 +2219,7 @@ def build_connector_meta(self) -> LMCacheOffloadMetadata: # combines it with offload_loaded_tokens after all TP workers # succeed to publish the restored GPU prefix. seq.offload_load_start_tokens = hbm - seq.offload_loaded_tokens = max(hbm, lmc) + seq.offload_loaded_tokens = self._claim_after_load(seq, hbm, lmc) # req_id MUST be the raw seq.id (the type the scheduler compares # against in _update_waiting_for_remote_kv); str(seq.id) is only for # LMCache's lookup/pin API. A str here silently never wakes the seq. diff --git a/atom/kv_transfer/offload/hybrid/kimi_k3/__init__.py b/atom/kv_transfer/offload/hybrid/kimi_k3/__init__.py new file mode 100644 index 0000000000..48a425078a --- /dev/null +++ b/atom/kv_transfer/offload/hybrid/kimi_k3/__init__.py @@ -0,0 +1,4 @@ +# SPDX-License-Identifier: MIT +# Copyright (C) 2024-2026, Advanced Micro Devices, Inc. All rights reserved. + +"""Kimi-K3 offload: dense paged KV plus the KDA per-request state tier.""" diff --git a/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py b/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py new file mode 100644 index 0000000000..b15a5dc84b --- /dev/null +++ b/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py @@ -0,0 +1,484 @@ +# SPDX-License-Identifier: MIT +# Copyright (C) 2024-2026, Advanced Micro Devices, Inc. All rights reserved. + +"""Kimi-K3 offload: dense paged KV plus the KDA per-request state tier. + +K3 keeps a recurrent (KDA) state alongside its paged KV, and a prefix is only +resumable where both are available. This variant is the dense connector plus a +CPU tier for that state: same paged-KV path, one extra leg. +""" + +from __future__ import annotations + +import logging +import os +import time + +from atom.kv_transfer.disaggregation.types import ConnectorCompletion +from atom.kv_transfer.offload import config as offcfg +from atom.kv_transfer.offload._offload_common import ( + max_pending_saves, + pp_aware_rank_and_world, +) +from atom.kv_transfer.offload.dense.connector import ( + DenseOffloadConnector, + DenseOffloadScheduler, +) +from atom.kv_transfer.offload.hybrid.kimi_k3.staging import StagedTransfer +from atom.kv_transfer.offload.hybrid.kimi_k3.state_object import StateByteCodec +from atom.kv_transfer.offload.hybrid.kimi_k3.state_tier import ( + _JointPark, + StateOffloadTier, +) +from atom.kv_transfer.offload.metadata import LMCacheOffloadMetadata + +logger = logging.getLogger("atom") + +#: Completion channels this variant owns. The generic aggregator transports +#: them opaquely and takes a failure-dominant TP quorum, which is what makes a +#: partial store resolve instead of pinning a key forever. +STATE_INDEX_CHANNEL = "k3_state_index" +STATE_STAGING_CHANNEL = "k3_state_staging" + +#: A save outstanding longer than this is a backend that stopped, not one that +#: is busy: a 4096-token store costs ~65ms. +SAVE_STALL_SECONDS = 120.0 + + +class KimiK3OffloadConnector(DenseOffloadConnector): + """Worker side: dense KV, plus spill/load of the per-request state.""" + + def __init__(self, config) -> None: + super().__init__(config) + self._state_tier = None + self._state_engine = None + # Inert until a request has both legs, which only a joint boundary + # produces; costs one dict lookup per report otherwise. + self._joint_park = _JointPark() + + def register_kv_caches( + self, kv_caches: dict, transfer_tensors=None, num_blocks: int | None = None + ) -> None: + super().register_kv_caches(kv_caches, transfer_tensors, num_blocks) + self._build_state_tier(transfer_tensors) + + # -- tier construction ------------------------------------------------- + def _build_state_tier(self, transfer_tensors) -> None: + from aiter.dist.parallel_state import get_tp_group + + from atom.model_engine.state_offload import state_offload_staging_groups + + if state_offload_staging_groups() <= 0: + return + + # PP breaks the tier twice over: the CacheEngineKey has no PP component, + # so two stages at the same TP rank would overwrite each other; and only + # the head stage drains staging releases, so the ring starves. Refused + # rather than half-supported. Paged KV is unaffected. + pp_size = int(getattr(self._config, "pipeline_parallel_size", 1) or 1) + if pp_size > 1: + logger.warning( + "kimi_k3 offload: the state tier is unsupported under pipeline " + "parallelism (pipeline_parallel_size=%d); paged KV is unaffected.", + pp_size, + ) + return + + backend = getattr(transfer_tensors, "state_backend", None) + if backend is None: + logger.warning( + "kimi_k3 offload: no attention backend published; state tier off." + ) + return + try: + views = backend.state_entry_views(0) + entry_bytes = sum(int(v.numel()) * v.element_size() for v in views) + except (NotImplementedError, AttributeError): + # No per-request state on this backend. IndexError is deliberately + # not caught: a zero-entry pool with the tier on is a sizing bug. + logger.warning( + "kimi_k3 offload: %s owns no per-request state views; tier off.", + type(backend).__name__, + ) + return + + tp = get_tp_group() + rank, world = pp_aware_rank_and_world(self._config, tp) + cfg = offcfg.build_lmcache_config( + getattr(self._config, "kv_transfer_config", None) + ) + meta = offcfg.build_lmcache_metadata(self._config, cfg, world, rank) + + # One flat entry, packed on the tier's own `lmc-state` thread. Sized to + # the entry rather than shared with the KV staging buffer, which is + # sized in LMCache chunks and is routinely an order of magnitude smaller. + gpu_connector = self._engine.gpu_connector + staged = StagedTransfer( + gpu_connector.device, + staging_buffer_bytes=entry_bytes, + release_after_transfer=gpu_connector.release_gpu_staging_after_transfer, + ) + codec = StateByteCodec( + backend, + staged, + entry_bytes, + model_name=meta.model_name, + world_size=world, + worker_id=rank, + ) + codec.bind_storage_manager(self._state_storage_manager(cfg, meta, rank)) + # No index here: StateOffloadIndex lives in the engine process; both + # directions report and the engine applies. + self._state_tier = StateOffloadTier(codec) + logger.info( + "kimi_k3 offload: state tier up, entry=%.2f MiB rank=%d", + entry_bytes / (1 << 20), + rank, + ) + + def _state_storage_manager(self, cfg, meta, rank: int): + """The tier's own CPU pool, not the paged-KV one. + + Sharing loses both ways: the KV write stream is several times the + state volume so it evicts the checkpoints, and a stopped KV backend + takes the tier down with it. + """ + raw = os.environ.get("OFFLOAD_STATE_CPU_SIZE", "16") + try: + gib = float(raw) + except ValueError: + logger.warning("kimi_k3 offload: invalid OFFLOAD_STATE_CPU_SIZE=%r", raw) + gib = 16.0 + if gib <= 0: + logger.info("kimi_k3 offload: state tier shares the paged-KV CPU pool.") + return self._engine.storage_manager + try: + from lmcache.v1.cache_engine import LMCacheEngineBuilder + from lmcache.v1.memory_management import MemoryFormat + + cfg.max_local_cpu_size = gib + # FIFO, while the KV pool keeps LMCache's LRU. A state entry is + # written once and read once (turn N+1 resumes off turn N's anchor + # and nobody looks again), so what decides its worth is age against + # the think time. LRU would promote an entry already consumed over + # one still waiting. Tied to the ladder being off; with rungs shared + # across conversations, entries stop being one-shot and LRU wins. + cfg.cache_policy = "FIFO" + engine = LMCacheEngineBuilder.get_or_create( + f"atom-state-{rank}", + cfg, + meta, + self._engine.gpu_connector, + lambda t, s: None, + lambda o, s: o, + ) + engine.fmt = MemoryFormat.KV_2LTD + engine.post_init() + self._state_engine = engine + logger.info( + "kimi_k3 offload: state pool %.0f GiB/rank (paged KV keeps %s)", + gib, + os.environ.get("LMCACHE_MAX_LOCAL_CPU_SIZE", ""), + ) + return engine.storage_manager + except Exception: + logger.warning( + "kimi_k3 offload: no separate state pool; sharing the paged-KV " + "pool, where the KV write stream will evict checkpoints.", + exc_info=True, + ) + return self._engine.storage_manager + + # -- per-step ---------------------------------------------------------- + def start_load_kv(self, metadata) -> None: + if isinstance(metadata, LMCacheOffloadMetadata): + self._arm_joint_loads(metadata) + self._start_state_loads(metadata) + super().start_load_kv(metadata) + + def _arm_joint_loads(self, metadata) -> None: + """Hold a request owning both legs until both report. + + Both legs surface on the KV completion channel, so an id in both would + collapse into one wake and resume the suffix prefill while the other + transfer is still writing. + """ + loads = getattr(metadata, "state_loads", None) or () + state_ids = {req_id for req_id, _h, _slot in loads} + if not state_ids or not self._do_load: + return + for req in metadata.requests: + if req.load_spec is not None and req.req_id in state_ids: + self._joint_park.arm(req.req_id, needs_kv=True, needs_state=True) + + def _start_state_loads(self, metadata) -> None: + """Hand this step's state loads to the tier's executor. + + No producer fence, unlike the save path: a load writes the entry, the + owning request is parked so no forward touches it, and unpack + synchronizes the producing stream before it returns. + """ + loads = getattr(metadata, "state_loads", None) + if not loads: + return + if self._state_tier is None: + # The engine's index can outlive a tier that refused to build. Fail + # them so the requests recompute rather than park forever. + logger.warning( + "kimi_k3 offload: %d state load(s) with no tier; failing them.", + len(loads), + ) + self._fail_state_loads(loads) + return + for req_id, h, group in loads: + self._state_tier.submit_load(req_id, int(h), int(group)) + + def _fail_state_loads(self, loads) -> None: + for req_id, _h, _group in loads: + self._joint_park.settle_state(req_id, False) + + # -- completions ------------------------------------------------------- + def get_finished(self): + out = super().get_finished() + if self._state_tier is None: + return out + indexed, released, index_failed = self._state_tier.take_spill_reports() + state_done, state_failed = self._state_tier.get_finished() + out.finished_loading, out.failed_loading = self._settle_joint( + out.finished_loading, out.failed_loading, state_done, state_failed + ) + # Spill and staging reports have no request identity, so they ride the + # connector-owned channels; the aggregator's quorum is failure-dominant, + # which is what resolves a partial store instead of pinning the key. + for h in indexed: + out.connector_completions.add( + ConnectorCompletion(STATE_INDEX_CHANNEL, int(h), True) + ) + for h in index_failed: + out.connector_completions.add( + ConnectorCompletion(STATE_INDEX_CHANNEL, int(h), False) + ) + for group in released: + out.connector_completions.add( + ConnectorCompletion(STATE_STAGING_CHANNEL, int(group), True) + ) + return out + + def _settle_joint(self, kv_done, kv_failed, state_done, state_failed): + """Merge the two report channels, holding armed pairs back. + + `waits_for` is asked first because `_settle` ignores ids it never + armed, which is indistinguishable from a leg that landed. + """ + park = self._joint_park + passthrough_done: set = set() + passthrough_failed: set = set() + for settle, reports, ok in ( + (park.settle_kv, kv_done, True), + (park.settle_kv, kv_failed, False), + (park.settle_state, state_done, True), + (park.settle_state, state_failed, False), + ): + for req_id in reports: + if park.waits_for(req_id): + settle(req_id, ok) + elif ok: + passthrough_done.add(req_id) + else: + passthrough_failed.add(req_id) + ready, ready_failed = park.take_ready() + return passthrough_done | ready, passthrough_failed | ready_failed + + +class KimiK3OffloadScheduler(DenseOffloadScheduler): + """Scheduler side: dense KV, plus the state tier's load queue and the + save-stall guard that keeps a stopped backend from stopping the engine.""" + + def __init__(self, config) -> None: + super().__init__(config) + # (req_id, state_hash, target_group) drained into each step's metadata. + # A state load shares no shape with a KV transfer -- no token ids, no + # block ids, no chunking -- only the park/report lifecycle. + self._pending_state_loads: list[tuple] = [] + # A finished request whose save is queued keeps its blocks pinned + # (`should_defer_free`), so the queue depth is also how much of the pool + # a slow backend can hold. Same knob and default the DSV4 layout bounds + # its worker queue with, read from the other end. + self._max_pending_saves = max_pending_saves( + getattr(config, "kv_transfer_config", None) or {}, + int(os.environ.get("OFFLOAD_COPY_WORKERS", "1") or 1), + ) + self._save_inflight_since: dict[str, float] = {} + self._save_stalled = False + self._warned_save_stalled = False + # Channel reports drained by the engine each step. + self._state_indexed: set[int] = set() + self._state_index_failed: set[int] = set() + self._state_staging_released: set[int] = set() + + # -- state load queue -------------------------------------------------- + def enqueue_state_loads(self, loads) -> bool: + if not loads: + return False + self._pending_state_loads.extend(loads) + return True + + def build_connector_meta(self) -> LMCacheOffloadMetadata: + self._refresh_save_stall() + meta = super().build_connector_meta() + # Drained, not copied: a second submission would write the same entry + # into a group the first transfer is already filling. + meta.state_loads = self._pending_state_loads + self._pending_state_loads = [] + return meta + + def _may_emit_save(self) -> bool: + """Nothing new goes out while the backend is stalled, and never more + than `OFFLOAD_MAX_PENDING_SAVES` requests pinned at once.""" + return ( + not self._save_stalled + and len(self._save_inflight) < self._max_pending_saves + ) + + # -- save stall -------------------------------------------------------- + def _refresh_save_stall(self) -> None: + """Decide whether the save path has stopped draining. + + Ages are tracked off `_save_inflight` rather than stamped at emission, + so this needs no hook inside the base class's save loop. + """ + now = time.monotonic() + inflight = set(self._save_inflight) + for sid in inflight - set(self._save_inflight_since): + self._save_inflight_since[sid] = now + for sid in set(self._save_inflight_since) - inflight: + del self._save_inflight_since[sid] + if not self._save_inflight_since: + if self._save_stalled: + logger.info("kimi_k3 offload: save path draining again") + self._save_stalled = False + self._warned_save_stalled = False + return + oldest = min(self._save_inflight_since.values()) + self._save_stalled = (now - oldest) > SAVE_STALL_SECONDS + if self._save_stalled and not self._warned_save_stalled: + self._warned_save_stalled = True + logger.warning( + "kimi_k3 offload: no save completed in %.0fs (%d in flight); " + "releasing the blocks of requests whose save was never sent.", + now - oldest, + len(self._save_inflight), + ) + + def should_defer_free(self, seq) -> bool: + """Base behaviour, plus an escape when the backend has stopped. + + A save already handed out is reading these blocks, so freeing them + would let the next request write into them mid-transfer and index the + result under this prefix's hash. One never handed out has no reader, + and holding it is what turns a stopped backend into a stopped engine. + + That leaves the handed-out save with no way out, which is what + `Scheduler._reconcile_stalled_deferred_saves` covers, on a longer clock + tied to LMCache's own force-unpin window. Neither mechanism subsumes the + other: this one asks whether the backend ever took the save, that one + whether it ever answered. + """ + sid = str(seq.id) + if self._save_stalled and sid not in self._save_inflight: + if self._has_pending_save(seq): + self._save_tracker.pop(sid, None) + return False + return super().should_defer_free(seq) + + # -- joint boundary ---------------------------------------------------- + def _decide_load_after_alloc(self, seq, ls): + """Clamp a hybrid's KV leg to the boundary the state leg is aimed at. + + A hybrid's per-request state is the compressed history of exactly + `[0, hbm)`. Raising the KV-loaded length past that would have the + forward skip `[hbm, lmc)` while the linear layers never see it: wrong + output, no exception. So a hybrid loads only when `can_allocate` picked + one boundary for both legs (`_joint_kv_boundary`), and this clamps the + KV leg down to it. + """ + if not getattr(seq, "has_per_req_cache", False): + return super()._decide_load_after_alloc(seq, ls) + + hbm = int(seq.num_cached_tokens) + lmc = int(ls.lmcache_cached_tokens) + chunk = self.chunk_size or 256 + joint = int(getattr(seq, "state_joint_boundary_tokens", 0) or 0) + if joint <= hbm: + return False, "per_req_cache_state_boundary", hbm, lmc, lmc - hbm, chunk + # Where the transfer starts, which is NOT where the request may call + # itself cached. `allocate` claimed every block the prefix walk matched, + # not just the resumable ones, so the KV below this is already in the + # pool and asking LMCache to send it back would move bytes the GPU + # holds -- and land a second copy of them in HBM, since + # `publish_loaded_prefix` keeps the existing canonical mapping and the + # freshly written blocks stay private to this request. + # + # `state_joint_claim_tokens` is floored to the chunk grid by + # `_joint_kv_boundary`, so this start is aligned whenever `hbm` was. + start = max(hbm, int(getattr(seq, "state_joint_claim_tokens", 0) or 0)) + # The KV leg moves whole chunks and the blocks below `start` are shared, + # so an unaligned start cannot be rounded down. + if start % chunk != 0: + return False, "joint_unaligned_hbm_prefill", start, lmc, lmc - start, chunk + # Transfer the chunk covering the boundary, claim only the boundary. + kv_target = int(getattr(seq, "state_joint_kv_tokens", 0) or 0) or joint + if joint > lmc or kv_target > lmc: + return False, "joint_boundary_above_lookup", start, lmc, lmc - start, chunk + if kv_target <= start: + # The whole boundary is already resident. Unreachable while + # `_gated_hit` returns the rightmost rung -- a boundary at or below + # the compressed hit would have been the plain hit -- but a state + # leg with no KV leg is a shape this must not emit silently. + return False, "joint_kv_already_resident", start, lmc, 0, chunk + # Both ends of the transfer, together: the base class writes the start + # back on its own path and the worker reads `[hbm_cached_tokens, + # lmcache_cached_tokens)`, so leaving the start at the value the lookup + # recorded would fetch from token 0 every time. + ls.hbm_cached_tokens = start + ls.lmcache_cached_tokens = kv_target + # Deliberately past the min-load floor: the boundary was chosen for both + # legs, and refusing on size would leave the state leg claiming a prefix + # whose KV never came. + return True, "joint_state_and_kv", start, kv_target, kv_target - start, chunk + + def _claim_after_load(self, seq, hbm: int, lmc: int) -> int: + """How far the request may call itself cached once the load lands. + + For a joint load that is the *state* boundary, which sits at or below + the transfer's end: the KV leg is aimed at the chunk covering it, and + claiming the rounded-up figure would have the forward skip tokens the + recurrent state does not cover. + """ + joint = int(getattr(seq, "state_joint_boundary_tokens", 0) or 0) + return max(hbm, min(joint, lmc)) if joint else max(hbm, lmc) + + # -- connector-owned channels ----------------------------------------- + def connector_completion(self, completion) -> bool: + if completion.channel == STATE_INDEX_CHANNEL: + target = ( + self._state_indexed + if completion.succeeded + else self._state_index_failed + ) + target.add(int(completion.operation_id)) + return True + if completion.channel == STATE_STAGING_CHANNEL: + self._state_staging_released.add(int(completion.operation_id)) + return True + return super().connector_completion(completion) + + def take_state_reports(self) -> tuple[set[int], set[int], set[int]]: + """Drain this step's tier reports for the engine-side index.""" + indexed = self._state_indexed + failed = self._state_index_failed + released = self._state_staging_released + self._state_indexed = set() + self._state_index_failed = set() + self._state_staging_released = set() + return indexed, released, failed diff --git a/atom/kv_transfer/offload/hybrid/kimi_k3/staging.py b/atom/kv_transfer/offload/hybrid/kimi_k3/staging.py new file mode 100644 index 0000000000..79ba017515 --- /dev/null +++ b/atom/kv_transfer/offload/hybrid/kimi_k3/staging.py @@ -0,0 +1,259 @@ +# SPDX-License-Identifier: MIT +# Copyright (C) 2024-2025, Advanced Micro Devices, Inc. All rights reserved. + +"""Bounded GPU staging buffer, D2H/H2D, and the producer event. + +A single-entry counterpart to `BlockGPUConnector`'s block-chunked staging: the +state tier moves one flat entry per transfer, so it needs the buffer, the copy +stream and the producer event, but none of the block/chunk orchestration. The +thread-state and buffer primitives are shared with the KV path. +""" + +from __future__ import annotations + +import threading +from typing import Any + +import torch + +from atom.kv_transfer.offload.atom_lmcache_staging import ( + _NullCtx, + _StagingBuffer, + _ThreadTransferState, +) + +# --------------------------------------------------------------------------- +# StagedTransfer +# --------------------------------------------------------------------------- + + +class StagedTransfer: + """Bounded GPU staging buffer, D2H/H2D, and the producer event. + + The half of the LMCache GPU connector that is not about chunks. KV and + state both need a bounded device buffer, a copy stream, and an event the + save worker synchronizes on; neither needs the other's orchestration. The + chunk layer stays in `ATOMLMCacheGPUConnector` because it is genuinely + KV-specific: `_iter_transfer_chunks` zips MemoryObjs against block-id + groups with `strict=True` and sizes each from a startup per-block + constant, so a single object of a different size breaks both invariants. + State is not a member of that loop. + + **The cross-thread producer fence is the caller's, not this class's.** + Nothing here waits on the forward's compute stream: `pack` issues its + gather on a private `pack_stream`, and the two `_StagingBuffer` events only + order this worker's own streams against each other. The fence commit + 7427e05e added to fix KV corruption on reload is recorded on the RPC thread + and `synchronize()`d by the caller -- `connector.py`'s `save_ready_event` + for KV, and `state_tier._do_spill`'s `ready_event.synchronize()` before + `codec.put` for state. Do not delete either believing this class covers it: + without them the gather reads the staging entry's previous occupant, which + is silent state corruption. + """ + + def __init__( + self, + device: torch.device, + staging_buffer_bytes: int, + *, + release_after_transfer: bool = False, + ) -> None: + self.device = torch.device(device) + self._staging_buffer_bytes = int(staging_buffer_bytes) + self._release_after_transfer = release_after_transfer + self._tls = threading.local() + + @property + def staging_buffer_bytes(self) -> int: + return self._staging_buffer_bytes + + def _use_cuda(self) -> bool: + return self.device.type == "cuda" + + def thread_state(self) -> _ThreadTransferState: + states = getattr(self._tls, "states", None) + if states is None: + states = {} + self._tls.states = states + key = str(self.device) + state = states.get(key) + if state is None: + state = _ThreadTransferState( + self.device, + self._use_cuda(), + ) + states[key] = state + return state + + def ensure_buffer( + self, + staging_buffer: _StagingBuffer, + nbytes: int, + ) -> torch.Tensor: + nbytes = int(nbytes) + if nbytes > self._staging_buffer_bytes: + raise RuntimeError( + "ATOM LMCache connector internal error: transfer group exceeds " + "bounded GPU staging buffer: " + f"nbytes={nbytes}, capacity={self._staging_buffer_bytes}" + ) + if ( + staging_buffer.tensor is None + or int(staging_buffer.tensor.numel()) != self._staging_buffer_bytes + ): + staging_buffer.tensor = torch.empty( + (self._staging_buffer_bytes,), + dtype=torch.uint8, + device=self.device, + ) + staging_buffer.free_event_valid = False + return staging_buffer.tensor[:nbytes] + + def release_buffer_if_requested( + self, + staging_buffer: _StagingBuffer, + ) -> None: + if not self._release_after_transfer: + return + staging_buffer.tensor = None + staging_buffer.free_event_valid = False + + def memory_tensor(self, memory_obj: Any, nbytes: int) -> torch.Tensor: + tensor = getattr(memory_obj, "tensor", None) + if tensor is None and hasattr(memory_obj, "get_tensor"): + tensor = memory_obj.get_tensor(0) + if tensor is None: + raise RuntimeError("ATOM LMCache connector: invalid MemoryObj tensor") + if tensor.dtype != torch.uint8: + raise TypeError( + "ATOM LMCache connector: MemoryObj tensor must be uint8, " + f"got {tensor.dtype}" + ) + if not tensor.is_contiguous(): + raise RuntimeError( + "ATOM LMCache connector: MemoryObj tensor not contiguous" + ) + flat = tensor.reshape(-1) + if int(flat.numel()) < int(nbytes): + raise ValueError( + "ATOM LMCache connector: MemoryObj tensor is too small " + f"for {nbytes} bytes; got {int(flat.numel())}" + ) + return flat[: int(nbytes)] + + # -- whole-entry transfer (state tier) -------------------------------- + + @staticmethod + def _segment_block_bytes(segments: list[torch.Tensor]) -> list[int]: + return [int(seg.numel()) * seg.element_size() for seg in segments] + + def _device_ctx(self): + if self._use_cuda(): + return torch.cuda.device(self.device) + return _NullCtx() + + def pack(self, segments: list[torch.Tensor], dst: Any) -> None: + """Gather `segments` into one contiguous object via the Triton packer. + + The existing kernel needs no modification: it is already a fully + parameterized gather driven by segment_ptrs[] + segment_block_bytes[] + + block_ids[]. State passes its own views as the segments with + block_ids=[0] and chunk_block_counts=[1] -- a single "chunk" of one + "block", which is what a whole-entry snapshot is. Per-segment sizes may + differ (GDN's k-views and v-views do); `_build_meta` sums them into one + `bytes_per_block`. + + `storage_manager.allocate` normally hands back *host* memory, and the + packer requires a CUDA uint8 contiguous destination, so the general + path packs into the bounded GPU staging buffer and D2H's from there. + The event around that copy makes the *consumer* side safe: whoever + reads the MemoryObj next must not see a D2H still in flight. It says + nothing about the *producer* side -- this gather runs on a private + stream and does not wait for the forward that wrote the entry. That + wait is the caller's (`state_tier._do_spill`); see the class docstring. + """ + from atom.kv_transfer.offload.triton_kv_staging import fused_pack_chunk_major + + segments = list(segments) + seg_bytes = self._segment_block_bytes(segments) + nbytes = sum(seg_bytes) + dst_tensor = self.memory_tensor(dst, nbytes) + with self._device_ctx(): + state = self.thread_state() + staging_buffer = state.staging_buffer + if dst_tensor.is_cuda and dst_tensor.device == self.device: + with state.stream_ctx(state.pack_stream): + fused_pack_chunk_major(segments, seg_bytes, [1], [0], dst_tensor) + if state.pack_stream is not None: + state.pack_stream.synchronize() + return + device_buf = self.ensure_buffer(staging_buffer, nbytes) + try: + with state.stream_ctx(state.pack_stream): + fused_pack_chunk_major(segments, seg_bytes, [1], [0], device_buf) + self._handoff(state, state.pack_stream, state.copy_stream) + with state.stream_ctx(state.copy_stream): + dst_tensor.copy_( + device_buf, + non_blocking=dst_tensor.device.type != "cpu", + ) + self._finish(state, state.copy_stream) + except Exception: + staging_buffer.free_event_valid = False + raise + finally: + self.release_buffer_if_requested(staging_buffer) + + def unpack(self, src: Any, segments: list[torch.Tensor]) -> None: + """Scatter one packed object back over `segments` -- `pack`'s mirror. + + Same staging hop in reverse: H2D into the bounded device buffer, then + the Triton unpack kernel writes the segments. The kernel's stream is + the one that produces the observable result, so it is the one waited + on before the segments are handed back to their owner. + """ + from atom.kv_transfer.offload.triton_kv_staging import fused_unpack_chunk_major + + segments = list(segments) + seg_bytes = self._segment_block_bytes(segments) + nbytes = sum(seg_bytes) + src_tensor = self.memory_tensor(src, nbytes) + with self._device_ctx(): + state = self.thread_state() + staging_buffer = state.staging_buffer + if src_tensor.is_cuda and src_tensor.device == self.device: + with state.stream_ctx(state.pack_stream): + fused_unpack_chunk_major(src_tensor, segments, seg_bytes, [1], [0]) + if state.pack_stream is not None: + state.pack_stream.synchronize() + return + device_buf = self.ensure_buffer(staging_buffer, nbytes) + try: + with state.stream_ctx(state.copy_stream): + device_buf.copy_( + src_tensor, + non_blocking=src_tensor.device.type != "cpu", + ) + self._handoff(state, state.copy_stream, state.pack_stream) + with state.stream_ctx(state.pack_stream): + fused_unpack_chunk_major(device_buf, segments, seg_bytes, [1], [0]) + self._finish(state, state.pack_stream) + except Exception: + staging_buffer.free_event_valid = False + raise + finally: + self.release_buffer_if_requested(staging_buffer) + + @staticmethod + def _handoff(state: _ThreadTransferState, producer, consumer) -> None: + """Record the producer event and make the consumer stream wait on it.""" + state.staging_buffer.ready_event.record(producer) + consumer.wait_event(state.staging_buffer.ready_event) + + @staticmethod + def _finish(state: _ThreadTransferState, producer) -> None: + """Publish the result: the buffer is free again once `producer` drains, + and the caller may not observe the bytes until it has.""" + state.staging_buffer.free_event.record(producer) + state.staging_buffer.free_event_valid = True + producer.synchronize() diff --git a/atom/kv_transfer/offload/hybrid/kimi_k3/state_object.py b/atom/kv_transfer/offload/hybrid/kimi_k3/state_object.py new file mode 100644 index 0000000000..a77614fa72 --- /dev/null +++ b/atom/kv_transfer/offload/hybrid/kimi_k3/state_object.py @@ -0,0 +1,155 @@ +# SPDX-License-Identifier: MIT +# Copyright (C) 2024-2025, Advanced Micro Devices, Inc. All rights reserved. + +"""One state checkpoint, one opaque object, keyed by ATOM's own hash. + +`ChunkedTokenDatabase` is bypassed. `StateSlotPool._resumable_from` looks up +one integer in HBM and in this tier, and a chunker-derived key would not be the +same thing; and state's bytes cannot be sliced by token anyway -- there is no +"first three chunks hit", so chunking would produce N keys useful only +together. It also sidesteps LMCache's chunk-alignment loss. +""" + +from __future__ import annotations + +import logging +from typing import Any + +import torch + +logger = logging.getLogger("atom") + + +class StateByteCodec: + """Pack/unpack one entry's state and move it through `storage_manager`. + + The object is an opaque flat uint8 blob: the x-packed / strided / + multi-plane state layouts cannot be expressed in LMCache's token-major + model at all. + + `entry_index` is not always a pool group -- on the spill path it is a + staging-ring slot, on the load path a real group. Both index the same + `state_entry_views` space. + """ + + def __init__( + self, + backend, + staged, + entry_bytes: int, + *, + model_name: str, + world_size: int, + worker_id: int, + ) -> None: + self._backend = backend + self._staged = staged + self.entry_bytes = int(entry_bytes) + if self.entry_bytes <= 0: + raise ValueError("state entry bytes must be > 0") + self._model_name = model_name + self._world_size = int(world_size) + self._worker_id = int(worker_id) + self._storage = None + # Never hard-code a size: V4 keeps six compressor fields across + # n_csa/n_hca layers plus an optional window; GDN keeps + # 2 * num_gdn_attn_state * (1 + num_spec) slots. Measured at 53.6 MiB + # per entry on the real model -- MB-scale, not KB. + logger.info( + "state offload: entry_bytes=%d (%.2f MiB) per request", + self.entry_bytes, + self.entry_bytes / (1 << 20), + ) + + def bind_storage_manager(self, storage_manager) -> None: + self._storage = storage_manager + + def key(self, h: int): + """The key carries ATOM's hash unmodified. + + `StateSlotPool._resumable_from` looks up the same integer in HBM and + in this tier, so hashing or stringifying it here would make the two + branches ask different questions. `dtype` is part of the key's identity. + """ + from lmcache.utils import CacheEngineKey + + return CacheEngineKey( + self._model_name, + self._world_size, + self._worker_id, + int(h), + torch.uint8, + ) + + def put(self, h: int, entry_index: int) -> bool: + """Spill one entry. Returns False when nothing was stored. + + A refusal is not an error: `_allocate` returns None under memory + pressure with nothing evictable, and spilling is best-effort -- the + pool counted the eviction either way, so the cost is one later miss. + """ + if self._storage is None: + return False + obj = self._allocate(self.entry_bytes) + if obj is None: + return False + self._staged.pack(self._backend.state_entry_views(entry_index), obj) + self._storage.batched_put([self.key(h)], [obj]) + return True + + def get(self, h: int, entry_index: int) -> bool: + """Load one entry back into `entry_index`. False on a miss. + + The reference must be discharged here: `get_blocking` does a + `ref_count_up()` on the caller's behalf, and without the matching down + LRU drops the block from the index but never returns it to the pinned + allocator -- an entry-sized leak per hit. `finally`, so a throwing + unpack does not leak either. `put` is deliberately not symmetric: + `batched_put` discharges its own reference. + """ + if self._storage is None: + return False + obj = self._storage.get(self.key(h)) + if obj is None: + return False + try: + self._staged.unpack(obj, self._backend.state_entry_views(entry_index)) + finally: + obj.ref_count_down() + return True + + def contains(self, h: int) -> bool: + """Ask storage, never a local set -- a set goes stale the moment + LMCache's LRU evicts. `contains` answers with a location name or None, + so the truthiness test is the membership test.""" + if self._storage is None: + return False + return bool(self._storage.contains(self.key(h))) + + def _allocate(self, nbytes: int) -> Any: + from lmcache.v1.memory_management import MemoryFormat + + # `MixedMemoryAllocator.allocate` accepts BINARY_BUFFER (-> a + # BytesBufferMemoryObj, whose `.tensor` is None and would fail + # `StagedTransfer.memory_tensor`) or one of the tensor formats + # {KV_2LTD, KV_2TD, KV_T2D, KV_MLA_FMT, EC_TD}; anything else, BINARY + # included, hits its `raise ValueError`. The shape/dtype above already + # force a flat opaque blob regardless of `fmt`, so the value is inert + # apart from passing that check -- same reasoning as the KV path's + # `self._engine.fmt = MemoryFormat.KV_2LTD` in connector.py. + # `busy_loop=False` because this is a *store*. LMCache's own + # `LocalCPUBackend.allocate` docstring is explicit that busy_loop "should + # only be used for retrieve", since "many stores happen concurrently (if + # they busy_loop, deadlock happens)". Its default is True, and under that + # default a pool with no eviction candidate spins `while True` on 0.1s + # sleeps with no attempt bound -- so it would never return the None this + # method's caller is written to handle, and would instead hang the state + # tier's save worker for as long as CPU memory stays full. LMCache's own + # store paths pass False here for the same reason (`cache_engine.py:666`, + # `local_disk_backend.py:468`, `storage_manager.py:96`). + return self._storage.allocate( + torch.Size([nbytes]), + torch.uint8, + fmt=MemoryFormat.KV_2LTD, + busy_loop=False, + ) diff --git a/atom/kv_transfer/offload/hybrid/kimi_k3/state_tier.py b/atom/kv_transfer/offload/hybrid/kimi_k3/state_tier.py new file mode 100644 index 0000000000..5f0b423844 --- /dev/null +++ b/atom/kv_transfer/offload/hybrid/kimi_k3/state_tier.py @@ -0,0 +1,233 @@ +# SPDX-License-Identifier: MIT +"""Worker-side spill and load driver for the state offload tier. + +One executor of its own, separate from the KV connector's `_load_executor` and +`_save_executor` so that state transfers and KV transfers cannot block each +other -- but a single one, shared by this tier's own spills and loads. See +`submit_load` for why the KV connector's load/save split is not copied here. + +This class **reports, and the engine applies**. It runs in a spawned runner +process while `StateOffloadIndex` lives in the engine process, so this side +cannot free a staging slot or index a hash directly. `take_spill_reports` and +`get_finished` hand their sets to the connector, which emits the first as +`ConnectorCompletion`s on its own channels and merges the second into +`finished_loading`/`failed_loading`. The failed-hash set exists to resolve the +aggregator's quorum: without it a partial store would pin the hash forever. +""" + +import logging +import threading +from concurrent.futures import ThreadPoolExecutor + +logger = logging.getLogger(__name__) + + +class StateOffloadTier: + """Moves bytes; decides nothing, and holds no index. + + No index here: `StateOffloadIndex` lives in the engine process, so every + counter and hash retraction is applied there from the reports below. + Neither side can hold a second opinion about what is stored. + """ + + def __init__(self, codec, *, max_workers: int = 1) -> None: + self.codec = codec + self._executor = ThreadPoolExecutor( + max_workers=max_workers, thread_name_prefix="lmc-state" + ) + self._lock = threading.Lock() + self._done: set[str] = set() + self._failed: set[str] = set() + self._inflight: set = set() + # The spill path's three reports, drained by `take_spill_reports`. + self._indexed: set[int] = set() + self._index_failed: set[int] = set() + self._released: set[int] = set() + + def _register(self, fut) -> None: + """Add *fut* to the inflight set and attach a callback that removes it + on completion. The callback fires on the worker thread (or inline on + the submitting thread if the future is already done), so we must not + call ``fut.result()`` while holding ``self._lock`` to avoid a deadlock. + """ + with self._lock: + self._inflight.add(fut) + + def _discard(f): + with self._lock: + self._inflight.discard(f) + + fut.add_done_callback(_discard) + + def submit_spill( + self, h: int, entry_index: int, staging_slot: int, ready_event=None + ) -> None: + """`entry_index` is what the codec packs, `staging_slot` what the ring + releases. Two index spaces: the staging entries sit past the pool's + range in the arena (`num_groups + slot`), while the ring counts from 0. + + `ready_event` fences the D2D staging copy that `AttentionBackend.build` + issued on the forward's compute stream. This worker packs on its own + stream, so without the wait it reads the entry's previous occupant. + """ + fut = self._executor.submit( + self._do_spill, h, entry_index, staging_slot, ready_event + ) + self._register(fut) + + def submit_load(self, req_id: str, h: int, group: int) -> None: + """Fetch `h` into pool group `group` for the parked request `req_id`. + + Shares one serial executor with the spills, unlike the KV connector's + split pair: `StagedTransfer` keeps its staging buffer in + `threading.local`, so a second thread costs a second resident buffer + per rank out of the HBM the state pool is already short of. The queue + it would shorten is at most `OFFLOAD_STATE_STAGING_GROUPS` deep. + """ + fut = self._executor.submit(self._do_load, req_id, h, group) + self._register(fut) + + def drain(self) -> None: + """Block until every submitted transfer has settled. Tests and shutdown + only -- the serving path polls `get_finished` instead.""" + with self._lock: + snapshot = set(self._inflight) + for fut in snapshot: + fut.result() + + def get_finished(self) -> tuple[set[str], set[str]]: + with self._lock: + done, failed = set(self._done), set(self._failed) + self._done.clear() + self._failed.clear() + return done, failed + + def shutdown(self) -> None: + self._executor.shutdown(wait=True) + + def _do_spill( + self, h: int, entry_index: int, staging_slot: int, ready_event=None + ) -> None: + stored = False + try: + if ready_event is not None: + ready_event.synchronize() + stored = bool(self.codec.put(h, entry_index)) + except Exception: # a spill is best effort by design + logger.warning("state offload: spill of hash %d failed", h, exc_info=True) + with self._lock: + # Report, never apply: `StateOffloadIndex` lives in the engine + # process and this runs in a spawned runner. The engine applies + # these via KVConnectorOutput. + if stored: + self._indexed.add(int(h)) + else: + # The failure channel lets the aggregator take quorum on + # `indexed | index_failed` instead of waiting for a second + # report that will never come from this rank. + self._index_failed.add(int(h)) + # Always, stored or not: a leaked slot shrinks the ring + # permanently and the feature quietly stops spilling. + self._released.add(int(staging_slot)) + + def take_spill_reports(self) -> tuple[set[int], set[int], set[int]]: + """`(hashes stored, staging slots free, hashes failed)` since the last call. + + A hash appears in exactly one of the first or third set per spill. + The aggregator's quorum over the two is failure-dominant, so a partial + store resolves in the same step rather than pinning the key. + """ + with self._lock: + indexed = set(self._indexed) + index_failed = set(self._index_failed) + released = set(self._released) + self._indexed.clear() + self._index_failed.clear() + self._released.clear() + return indexed, released, index_failed + + def _do_load(self, req_id: str, h: int, group: int) -> None: + # A load target is a real pool group, not a staging entry: the bytes + # land where the resuming request will read them. Only the spill + # direction needs the staging indirection. + # + # A miss is a normal path, not an error: LMCache's LRU can drop bytes + # under a hash the engine's index still advertises. Retracting that + # claim is the engine's job -- it owns the index -- and it does it from + # the report below. + ok = False + try: + ok = bool(self.codec.get(h, group)) + except Exception: # a failed load is a normal path + logger.warning("state offload: load of hash %d failed", h, exc_info=True) + with self._lock: + if ok: + self._done.add(req_id) + else: + self._failed.add(req_id) + + +class _JointPark: + """One park for the KV load and the state load of the same request. + + Both completions must land before unpark. Waking on the state transfer + alone lets the model read KV blocks that are not yet filled, which is + silent rather than an error. + + Either side failing fails the pair: half a load leaves state claiming a + prefix whose KV never arrived, and `failed_loading` already means "wake for + recompute using the blocks already allocated", which is exactly right here. + """ + + def __init__(self) -> None: + self._need: dict[str, set[str]] = {} + self._failed: set[str] = set() + self._ready: set[str] = set() + self._ready_failed: set[str] = set() + + def arm(self, req_id: str, *, needs_kv: bool, needs_state: bool) -> None: + need = set() + if needs_kv: + need.add("kv") + if needs_state: + need.add("state") + self._need[req_id] = need + if not need: + self._ready.add(req_id) + + def settle_kv(self, req_id: str, ok: bool) -> None: + self._settle(req_id, "kv", ok) + + def settle_state(self, req_id: str, ok: bool) -> None: + self._settle(req_id, "state", ok) + + def _settle(self, req_id: str, leg: str, ok: bool) -> None: + need = self._need.get(req_id) + if need is None: + return + need.discard(leg) + if not ok: + self._failed.add(req_id) + if need: + return + del self._need[req_id] + if req_id in self._failed: + self._failed.discard(req_id) + self._ready_failed.add(req_id) + else: + self._ready.add(req_id) + + def waits_for(self, req_id: str) -> bool: + """Whether this park still owes `req_id` a leg. + + Asked before settling: the legs report through channels a single-leg + request also uses, and `_settle` ignores unknown ids, which is + indistinguishable from a leg that landed. + """ + return req_id in self._need + + def take_ready(self) -> tuple[set[str], set[str]]: + ready, failed = set(self._ready), set(self._ready_failed) + self._ready.clear() + self._ready_failed.clear() + return ready, failed diff --git a/atom/model_engine/block_manager.py b/atom/model_engine/block_manager.py index bdbccc03e7..e5bc8d5f47 100644 --- a/atom/model_engine/block_manager.py +++ b/atom/model_engine/block_manager.py @@ -217,6 +217,67 @@ def __init__( # moment it stops forking (see the state-cache protocol). self.state_caches: tuple[StateCache, ...] = (self._state_checkpoint_cache,) + # Class names already warned about in `state_checkpoint_fates`. See + # there for why the warning latches. + self._warned_no_checkpoint_fates: set[str] = set() + from atom.model_engine.state_offload import ( + StateOffloadIndex, + kv_connector_hosts_state_tier, + state_offload_staging_groups, + ) + + staging = state_offload_staging_groups() + # Joint boundaries, split by what the state leg cost: `hbm` forked a + # resident checkpoint, `tier` paid an entry-sized H2D and a park. Almost + # all `tier` means the state pool is too small for the concurrency. + self.joint_boundaries = 0 + self.joint_boundaries_hbm = 0 + self.joint_boundaries_tier = 0 + # Why the rest got none, keyed by the gate that stopped them. + self.joint_skips: dict[str, int] = {} + # The LMCache chunk size in tokens, read where the config is rather than + # off the connector object. + self._joint_chunk_tokens = 0 + try: + from atom.kv_transfer.offload import config as offcfg + + self._joint_chunk_tokens = int( + offcfg.build_lmcache_config( + getattr(config, "kv_transfer_config", None) + ).chunk_size + ) + except Exception: + logger.warning( + "state offload: could not read the LMCache chunk size; the " + "joint KV load needs it and stays off", + exc_info=True, + ) + kv_offload_enabled = kv_connector_hosts_state_tier( + getattr(config, "kv_transfer_config", None) + ) + self.state_offload: StateOffloadIndex | None = None + # (req_id, hash, target_group) admitted this pass and not yet handed to + # the connector. Kept here rather than in the index because the slot + # is this object's fact, the same reason `state_spills_for_batch` joins + # the spill's two halves engine-side. + self._state_loads: list[tuple] = [] + # req_id -> slot, for loads whose request was deallocated before the + # bytes landed. The slot is off the free list until the report comes + # back; see `deallocate`. + self._orphan_load_slots: dict = {} + # `kv_offload_enabled` is the whole switch. Without a connector to + # drain a spill the ring would hand out slots that never come back, and + # a hash whose KV left HBM could not be resumed from anyway. + if kv_offload_enabled and staging > 0: + index = StateOffloadIndex(staging_depth=staging) + # Only the slot pool: `_spill` / `_resumable_from` live on + # `StateSlotPool`, while a + # PAGE checkpoint already lives in the KV blocks the KV connector + # offloads on its own. + for cache in self.state_caches: + if isinstance(cache, StateSlotPool): + cache.offload = index + self.state_offload = index # The demand funnel: recorded at admission, cut for when a prefill # chunk is shortened to land on it, kept when the state pool files it. # Counted at all three because a gap between any two is a different @@ -255,12 +316,77 @@ def take_state_maintenance_ops(self) -> StateMaintenanceOps: stores = restores = () if self.paged_state_checkpoints is not None: stores, restores = self.paged_state_checkpoints.take_checkpoint_ops() + # Drained here rather than beside the relocations so that the one + # consumer -- `CommonAttentionBuilder.build` -- is also the one place + # the spill-before-relocate order is enforced. return StateMaintenanceOps( relocations=relocations, + spills=tuple(self.state_spills_for_batch()), checkpoint_stores=stores, checkpoint_restores=restores, ) + def state_spills_for_batch(self) -> list[tuple[int, int, int, int]]: + """`(src_group, dst_entry, staging_slot, hash)` the batch being built + must copy out before its forward. + + The pool knows `(group, slot)` and the ring knows `(hash, slot)`; the + slot is the only thing relating them. Joined engine-side because + `num_groups` is authoritative here. Both lists are appended by the same + `_spill()` and drained together, so a slot in one and not the other is a + bug -- dropped rather than guessed at. + """ + out: list[tuple[int, int, int, int]] = [] + for cache in self.state_caches: + offload = getattr(cache, "offload", None) + if offload is None: + continue + hash_of_slot = {slot: h for h, slot in offload.take_pending()} + for group, slot in cache.take_spill_copies(): + h = hash_of_slot.pop(slot, None) + if h is None: + logger.warning( + "state offload: staging slot %d has a copy but no " + "pending hash; dropping the spill", + slot, + ) + offload.release_staging(slot) + continue + out.append((group, cache.num_groups + slot, slot, h)) + # Whatever is left is a pending hash with no copy to feed it. The + # keys are the slots; the values are the hashes, which the ring + # does not take back. + for slot in hash_of_slot: + # Never observed in correct operation; release rather than leak. + offload.release_staging(slot) + return out + + def state_checkpoint_fates(self) -> dict[str, int]: + """Summed fates across every state class, for the periodic stats line. + + Accumulates whatever each class's ``checkpoint_fates()`` returns, so a + new counter appears here with no change to this method. A class without + the method is skipped with a warning, latched per class: the caller is a + periodic stats line and the omission is a static property of the build. + """ + totals: dict[str, int] = {} + for cache in self.state_caches: + fates_fn = getattr(cache, "checkpoint_fates", None) + if fates_fn is None: + name = type(cache).__name__ + if name not in self._warned_no_checkpoint_fates: + self._warned_no_checkpoint_fates.add(name) + logger.warning( + "state_checkpoint_fates: %s does not implement " + "checkpoint_fates(); its counters are excluded from " + "totals", + name, + ) + continue + for k, v in fates_fn().items(): + totals[k] = totals.get(k, 0) + v + return totals + def _record_evicted(self, h: int) -> None: """A hash the block pool just dropped: report it, and settle the state. @@ -466,6 +592,124 @@ def pool_occupancy(self) -> dict[str, int]: "evicted_total": self.total_evicted_blocks, } + def _joint_kv_boundary( + self, + seq: Sequence, + hbm_boundary: int, + block_hashes: list[int], + ) -> int: + """Boundary above the HBM hit that BOTH legs can reach, 0 if none. + + `can_allocate` walks the HBM prefix cache and nothing else, so an + evicted prefix stops at the first miss even when LMCache holds every + block. This looks past that: the KV leg fetches `[hbm, B)` from LMCache + while the state leg fetches B's checkpoint from the tier, and the two + land on one boundary or neither runs. + + The grids do not line up -- a state rung is a hash-block boundary, the + KV leg moves whole chunks. Rather than discard every unaligned rung, the + KV leg is aimed at the chunk *covering* B while the request claims only + B: overshooting costs one chunk into blocks the forward is about to + rewrite, undershooting would be silent wrong output. + """ + seq.state_joint_boundary_tokens = 0 + seq.state_joint_boundary_hash = -1 + seq.state_joint_kv_tokens = 0 + seq.state_joint_claim_tokens = 0 + if self.state_offload is None: + return self._no_joint("off") + # PAGE checkpoints live in the KV pool, so their state leg has no tier + # to be served from and no separate boundary to agree on. + if not seq.has_per_req_cache or self.paged_state_checkpoints is not None: + return self._no_joint("not_hybrid") + hbs = self._hash_block_size() + # From the connector's config, not the connector object: the scheduler + # holds whatever `get_kvconnector` returned, and one without + # `chunk_size` would zero this and disable the feature silently. + chunk = self._joint_chunk_tokens or int( + getattr(seq, "offload_kv_chunk_tokens", 0) or 0 + ) + lmc_tokens = int(getattr(seq, "offload_kv_prefix_tokens", 0) or 0) + if chunk <= 0: + return self._no_joint("no_chunk_size") + # Floored to the grid everything below compares against, so the + # covering-chunk check becomes true by construction rather than by + # luck. (`get_num_new_matched_tokens` withholds one token on a + # full-prompt hit, which takes the lookup off the grid.) + lmc_tokens = (lmc_tokens // chunk) * chunk + if lmc_tokens <= hbm_boundary * hbs: + return self._no_joint("lmcache_within_hbm") + # The KV leg moves whole chunks and the blocks below the HBM prefix are + # shared, so an unaligned start cannot be rounded down. + if (hbm_boundary * hbs) % chunk != 0: + return self._no_joint("hbm_off_chunk_grid") + cap = min(lmc_tokens // hbs, self._n_hash_blocks(seq) - 1) + if cap <= hbm_boundary: + return self._no_joint("no_room_above_hbm") + chain = self._chain_to(seq, block_hashes, cap) + # One scan: `_gated_hit` already returns the rightmost rung + # `_resumable_from` accepts, so a decrement-and-rescan walk would spend + # a fixpoint pass per rung to reach the same answer. + candidate = self._gated_hit(seq, cap, chain) + if candidate <= hbm_boundary: + return self._no_joint("no_rung_above_hbm") + tokens = candidate * hbs + h = chain[candidate - 1] + # Cannot fail given the two premises above; kept as their assertion. + kv_tokens = -(-tokens // chunk) * chunk + if kv_tokens > lmc_tokens: + return self._no_joint("covering_chunk_beyond_lookup") + seq.state_joint_boundary_tokens = tokens + seq.state_joint_boundary_hash = h + seq.state_joint_kv_tokens = kv_tokens + # Everything up to the compressed hit is this prompt's KV and is still + # in the pool -- what `_gated_hit` cut was resumability, not residency. + # Below the joint boundary those blocks are read-only for this request + # (the forward starts at the boundary), so claiming them is free, and + # it is the difference between the KV leg fetching `[hit, B)` and + # fetching `[0, B)` with the front half already sitting in HBM. + # + # `candidate > hbm_boundary` and `hbm_boundary` is the RIGHTMOST rung + # `<= compressed_hit`, so the boundary is always past the compressed + # hit and this min is always the compressed hit. Written as a min + # anyway: the invariant belongs to `_gated_hit`, not here. + claim_blocks = min(len(block_hashes), candidate) + # Floored to the chunk grid so the KV leg starts aligned and writes + # only into blocks below it that nobody else can be reading. + claim_tokens = ((claim_blocks * hbs) // chunk) * chunk + seq.state_joint_claim_tokens = max(claim_tokens, hbm_boundary * hbs) + self.joint_boundaries += 1 + if self.state.lookup(h) >= 0: + self.joint_boundaries_hbm += 1 + else: + self.joint_boundaries_tier += 1 + return candidate + + def _no_joint(self, reason: str) -> int: + """Record why this admission got no joint boundary, and return 0. + Every reason is counted, including "this build is not even trying", + because a silent zero is indistinguishable from a feature that ran and + found nothing. + """ + self.joint_skips[reason] = self.joint_skips.get(reason, 0) + 1 + return 0 + + def _chain_to( + self, seq: Sequence, block_hashes: list[int], blocks: int + ) -> list[int]: + """`block_hashes` continued to `blocks` entries, hashes only. + A chained hash is a function of the prompt alone, so it is computable + past where the HBM cache stops -- which is what makes an LMCache-only + boundary addressable. Resumes from the last APPENDED hash: on a miss the + caller's loop variable holds a hash that is not in the chain. + """ + chain = list(block_hashes) + h = chain[-1] if chain else -1 + for i in range(len(chain), blocks): + h = self.compute_hash(self._hash_block_tokens(seq, i), h) + chain.append(h) + return chain + def can_allocate(self, seq: Sequence) -> int: """Return number of cache-hit blocks (>=0) if seq fits, else -1. @@ -518,6 +762,11 @@ def can_allocate(self, seq: Sequence) -> int: # `_gated_hit` settles the two gates jointly; neither can be applied to # the other's answer. num_cached_blocks = self._gated_hit(seq, compressed_hit, block_hashes) + # A boundary LMCache and the tier can jointly reach, above this hit. + # Recorded on the seq rather than returned: what `allocate` claims from + # HBM is still `num_cached_blocks`, and the joint boundary only decides + # where the two loads are aimed. + self._joint_kv_boundary(seq, num_cached_blocks, block_hashes) # Instrumentation: the pre-gate hit, so CacheStats can separate reuse # the gates declined (compressed_hit - num_cached_blocks) from reuse # lost to compressed eviction (everything above compressed_hit). @@ -571,19 +820,49 @@ def allocate(self, seq: Sequence, num_cached_blocks: int = 0): a hash until fully filled. """ assert not seq.block_table + # Two extents, and they are not the same number. `num_cached_blocks` is + # what the request may call cached -- it needs a resumable state behind + # it. `claim_blocks` is what it may point its block table at: every + # block the prefix walk matched, whether or not a checkpoint sits there. + # The gap between them is real reuse that the state gate declined, and + # taking it costs nothing: those blocks are below the joint boundary the + # forward will start from, so nobody writes them. + # + # Only a joint boundary widens it -- without one there is nothing above + # `num_cached_blocks` this request will ever treat as computed, and + # claiming further would pin blocks the forward is about to overwrite. + hbs = self._hash_block_size() + claim_blocks = max( + num_cached_blocks, + int(getattr(seq, "state_joint_claim_tokens", 0) or 0) // hbs, + ) h = -1 - for i in range(num_cached_blocks): + hit_hash = -1 + for i in range(claim_blocks): token_ids = self._hash_block_tokens(seq, i) h = self.compute_hash(token_ids, h) block_id = self.kv.lookup(h) + if block_id == -1: + # Evicted between `can_allocate` and here. Unreachable for the + # cached range (nothing evicts inside one admission), so only + # the widened tail can land here: stop claiming and let the rest + # come from the free pool. + assert ( + i >= num_cached_blocks + ), f"cached block {i} of {num_cached_blocks} vanished during allocate" + seq.state_joint_claim_tokens = i * hbs + break self.kv.claim(block_id) seq.block_table.append(block_id) + if i + 1 == num_cached_blocks: + hit_hash = h # Pin the restore before fresh blocks can evict its checkpoint. + state_holds = True if seq.has_per_req_cache and self.paged_state_checkpoints is not None: - self._attach_state_slots(seq, h if num_cached_blocks > 0 else -1) - for _ in range(num_cached_blocks, self.num_pool_blocks(len(seq))): + state_holds = self._attach_state_slots(seq, hit_hash) + for _ in range(len(seq.block_table), self.num_pool_blocks(len(seq))): seq.block_table.append(self._fresh_block()) - seq.num_cached_tokens = num_cached_blocks * self._hash_block_size() + seq.num_cached_tokens = num_cached_blocks * hbs # Per-request cache: claim this seq's slot indices from the # pre-allocated state tensor (e.g. GDN mamba_k_cache, the V4 compressor @@ -592,16 +871,38 @@ def allocate(self, seq: Sequence, num_cached_blocks: int = 0): # paged-block cost. The state pool's free list is the sole admission # bound for state cache. if seq.has_per_req_cache and self.paged_state_checkpoints is None: - self._attach_state_slots(seq, h if num_cached_blocks > 0 else -1) + # A joint load aims the state leg at its own boundary, which is + # above the HBM hit by construction. `num_cached_tokens` stays at + # the HBM prefix until the KV leg lands, so the forward covers + # `[hbm, num_prompt)` if anything goes wrong from here. + joint_hash = seq.state_joint_boundary_hash + state_holds = self._attach_state_slots( + seq, joint_hash if joint_hash != -1 else hit_hash + ) if seq.has_per_req_cache: seq._state_initialized_after_alloc = False - - def _attach_state_slots(self, seq: Sequence, hit_hash: int) -> None: + if not state_holds: + # No state behind the boundary means it is not this request's + # history. Disown it -- blocks stay claimed and the forward + # recomputes. Keeping it is silent wrong output, since + # `has_initial_state` (`gdn_attn.py`) is `num_cached_tokens > 0`. + seq.num_cached_tokens = 0 + seq.state_joint_boundary_tokens = 0 + seq.state_joint_boundary_hash = -1 + + def _attach_state_slots(self, seq: Sequence, hit_hash: int) -> bool: """Give `seq` its state slots, resuming from a checkpoint when one exists. + Returns whether `hit_hash`'s state is really the one `seq` now holds. + False means the caller must drop the boundary; see `allocate`. `hit_hash` is the content hash of the last reused block (-1 for a cold - start). `can_allocate` already shrank the hit to a boundary that carries - a checkpoint, so a lookup miss here just means the pool is off. + start). `can_allocate` already shrank the hit to a boundary that + `_resumable_from` accepted, and that is not the same as "in HBM": the + tier votes too. So a hash whose slot went to LMCache arrives as a miss + and becomes a load (`_request_state_load`), and so does one whose bytes + LMCache's own LRU has since dropped — which the tier cannot know until + the fetch misses. The two are told apart by whether the tier still + holds the hash, because only one of them may keep the boundary. A seq takes `state_slots_per_req` slots — one committed state plus a rollback slot per speculated token — however it starts. The checkpoint @@ -641,13 +942,30 @@ def _attach_state_slots(self, seq: Sequence, hit_hash: int) -> None: "gated PAGE checkpoint disappeared before state attach" ) seq.state_fork_src = -1 - return + # `begin_restore` either gathered the boundary's state into the + # committed slot or there was no boundary to gather, so what the + # caller holds is what it asked for. + return True src = self.state.lookup(hit_hash) if hit_hash != -1 else -1 if src < 0: - seq.state_slots = self.state.pop_many(width) + # Decided before the pop, not after: an eviction that makes room + # for a load must not spill, because the spill's copy out of that + # slot is issued by a later forward while the load writes it from + # the tier's thread first. See `StateSlotPool.pop`. + wants_load = self._tier_can_serve(hit_hash) + # Only the committed slot can be the load's destination, so only it + # has to forgo the spill; the rollback scratch is this request's own. + seq.state_slots = [self.state.pop(spill=not wants_load)] + ( + self.state.pop_many(width - 1) + ) seq.state_fork_src = -1 - return + if wants_load and self._request_state_load(seq, hit_hash): + return True + # A fresh slot holds the previous occupant's bytes. That is fine + # for a cold start (nothing claims otherwise) and wrong for a hit, + # so the hit only survives if there was none to begin with. + return hit_hash == -1 # Being resumed from is the evidence a guessed position was right, so a # checkpoint that pays off stops being spent first — see # `StateSlotPool.mark_speculative`. Here rather than in `claim`, which @@ -661,7 +979,7 @@ def _attach_state_slots(self, seq: Sequence, hit_hash: int) -> None: seq.state_fork_src = src # Held off the free list until the forward that reads it is issued. self.state.pin(src) - return + return True # `can_allocate` admitted this seq against a free list holding at least # `width`, and nothing else has run since, so the list can only be short # here if this seq's own `claim` above took the slot that made up the @@ -671,6 +989,131 @@ def _attach_state_slots(self, seq: Sequence, hit_hash: int) -> None: self.state.invalidate(src) seq.state_slots = [src] + self.state.pop_many(width - 1) seq.state_fork_src = -1 + return True + + def _tier_can_serve(self, hit_hash: int) -> bool: + """Whether the tier believes it holds `hit_hash`. + Asked twice per admission -- before `pop`, to decide whether that + eviction may spill, and inside `_request_state_load`. One predicate so + the two cannot drift: a pop that forwent its spill for a load then + refused would lose a checkpoint for nothing. + """ + return ( + hit_hash != -1 + and self.state_offload is not None + and hit_hash in self.state_offload.hashes + ) + """Whether `hit_hash`'s state can be produced at all, from either tier. + + Distinct from `_tier_can_serve`, which asks "will a load be issued" and + decides whether an eviction may spill. This asks "does this boundary + have a state behind it" -- an HBM checkpoint answers yes here, no there. + + The HBM branch is reachable, not redundant: `unindex` drops a checkpoint + when *its own* block leaves the index, but `can_allocate`'s walk stops at + the first miss anywhere in the chain, so an earlier block going puts a + live HBM checkpoint above `hbm_boundary`. That is the cheapest joint load + there is -- no state transfer, just the fork. + + Nothing downstream needs to know which branch answered: + `_attach_state_slots` looks HBM up first and falls to a load on a miss. + """ + return self._tier_can_serve(hit_hash) or ( + hit_hash != -1 and self.state.lookup(hit_hash) >= 0 + ) + + def _request_state_load(self, seq: Sequence, hit_hash: int) -> bool: + """Ask the tier to fetch `hit_hash` into the slot `seq` just took. + + Only reached when the HBM index missed -- the case the tier exists for. + The destination is a real pool slot, never a staging entry: the bytes + land where the resuming forward reads them, and only the spill direction + needs the indirection. + + `state_fork_src` stays -1. The loaded slot *is* the incoming state, and + naming a source would send the forward to a different slot than the one + being filled. + + False means the tier cannot serve it. `request_load` refuses an unknown + hash because a load is resolved only by a report, so offering one for + bytes no `get` can produce would park the request forever. The caller + then disowns the boundary -- the pre-tier answer. + """ + if not self._tier_can_serve(hit_hash): + return False + if self.state.has_pending_spill(seq.state_slot): + # A spill queued in an earlier pass still has to read this slot + # and the load would overwrite it first. Decline; the caller + # disowns the boundary, which is the pre-tier answer. + logger.warning( + "state offload: slot %d still owes a spill copy; declining " + "the load for seq %s and recomputing instead.", + seq.state_slot, + seq.id, + ) + return False + if not self.state_offload.request_load(seq.id, hit_hash): + return False + seq.state_load_hash = hit_hash + self._state_loads.append((seq.id, hit_hash, seq.state_slot)) + return True + + def cancel_state_load(self, seq: Sequence) -> None: + """Withdraw a load requested this pass, before anything was issued. + Only legal before `take_state_loads` handed it over; afterwards the + bytes are on their way and the slot must be held. The boundary is + disowned exactly as `allocate` would have. + """ + if seq.state_load_hash == -1: + return + self._state_loads = [e for e in self._state_loads if e[0] != seq.id] + if self.state_offload is not None: + self.state_offload.abandon_load(seq.id) + seq.state_load_hash = -1 + seq.num_cached_tokens = 0 + + def abandon_state_load(self, req_id) -> None: + """Give up on a load without blaming the bytes for it. + For a load nothing could carry. `settle_state_load(ok=False)` would + `forget` the hash on a miss that never happened, erasing the index one + request at a time and inflating its false-positive counter. + """ + if self.state_offload is None: + return + self.state_offload.abandon_load(req_id) + slot = self._orphan_load_slots.pop(req_id, None) + if slot is not None: + self.state.release(slot) + + def settle_state_load(self, req_id, ok: bool) -> None: + """Apply one worker load report. Keyed by request, like the KV load. + + Called for every `finished_loading`/`failed_loading` id, including the + many that are plain KV loads -- a no-op for those, which keeps the + scheduler from having to know which leg a report belongs to. An + abandoned load is already out of the index, but its slot is still being + written and comes back here, and only here. + """ + if self.state_offload is None: + return + if ok: + self.state_offload.complete_load(req_id) + else: + self.state_offload.fail_load(req_id) + slot = self._orphan_load_slots.pop(req_id, None) + if slot is not None: + self.state.release(slot) + + def take_state_loads(self) -> list[tuple]: + """`(req_id, hash, target_slot)` for loads admitted since the last call. + Drained once per pass by the scheduler, which hands them to the + connector. Draining rather than reading is what keeps a load from being + submitted twice into a slot the first transfer is already filling. + """ + if self.state_offload is None: + return [] + out, self._state_loads = self._state_loads, [] + return out def _chain_parent_hash(self, seq: Sequence, start: int) -> int | None: """Return the chained hash of block ``start - 1``, or ``None`` on a gap. @@ -1266,13 +1709,19 @@ def checkpoint_funnel(self) -> dict[str, int]: Assembled here because the stages live in two objects — the ladder decides what to ask for, the pool decides what survives — and a reader needs them side by side to tell which stage lost it. + Pool-level fates are collected via ``state_checkpoint_fates()`` so that + a second state class is automatically included — calling + ``self.state.checkpoint_fates()`` directly would miss it. """ return { "demands_recorded": self.demands_recorded, "demands_declined_no_room": self.demands_declined_no_room, "chunks_cut_for_demand": self.chunks_cut_for_demand, "chunks_cut_for_end": self.chunks_cut_for_end, - } | self._state_checkpoint_cache.checkpoint_fates() + "joint_boundaries": self.joint_boundaries, + "joint_boundaries_hbm": self.joint_boundaries_hbm, + "joint_boundaries_tier": self.joint_boundaries_tier, + } | self.state_checkpoint_fates() def pool_pressure(self) -> dict[str, int]: """Both pools' eviction counts and occupancy, side by side. @@ -1619,12 +2068,28 @@ def deallocate(self, seq: Sequence): # dropped here rather than left to `release_state_pins`, because the # forward that owed the read is not going to happen and the slot # should not sit out a pass for a reader that no longer exists. - self.state.release_many(seq.state_slots) + # Unless a state load is in flight into the committed slot -- a + # worker is writing it, so handing it back now would give the next + # request a buffer someone else is filling, with `has_initial_state` + # already true over it. Held until `settle_state_load`, which always + # comes: the worker reports every load either way. The rollback + # scratch is nobody's destination and goes back regardless. + if seq.state_load_hash != -1 and self.state_offload is not None: + self._orphan_load_slots[seq.id] = seq.state_slot + # Abandoned, not failed: an abort says nothing about the bytes, + # and forgetting the hash would cost the next request over this + # prefix a full recompute. + self.state_offload.abandon_load(seq.id) + self.state.release_many(seq.state_slots[1:]) + else: + self.state.release_many(seq.state_slots) # No next forward will read a pending fork source after deallocation. self.state.drop_reader(seq.state_fork_src) seq.state_slots = [] seq.state_fork_src = -1 + seq.state_load_hash = -1 + def can_append(self, seq: Sequence, num_new_tokens: int = 1) -> bool: seq_len = len(seq) current_blocks = len(seq.block_table) diff --git a/atom/model_engine/engine_core.py b/atom/model_engine/engine_core.py index 1dba21a4d5..5bca4b0b48 100644 --- a/atom/model_engine/engine_core.py +++ b/atom/model_engine/engine_core.py @@ -529,6 +529,12 @@ def _dispatch_idle_offload_work(self) -> None: connector = getattr(self.scheduler, "kv_connector", None) if connector is None or not getattr(connector, "is_offload", False): return + # getattr for the same reason as `kv_connector` above: this path is + # reached with scheduler doubles that implement only the connector + # surface. + publish = getattr(self.scheduler, "_publish_state_loads", None) + if publish is not None: + publish() meta = connector.build_connector_meta() if not connector_metadata_has_work(meta): return diff --git a/atom/model_engine/llm_engine.py b/atom/model_engine/llm_engine.py index 79136c214f..2cc58d4830 100644 --- a/atom/model_engine/llm_engine.py +++ b/atom/model_engine/llm_engine.py @@ -401,6 +401,13 @@ def get_cache_statistics(self, timeout: float = 30.0) -> dict[str, Any]: # placement that converges from one that pays per request, and # one of them missing makes the other unreadable. "chunks_cut_for_end", + # Whether the joint state+KV load found a boundary to work + # with, and what the state leg cost when it did. Summed like + # the rest: a boundary is per admission and every rank admits + # the same request. + "joint_boundaries", + "joint_boundaries_hbm", + "joint_boundaries_tier", ) } # `reusable`, not `full`: a request's trailing block is never a reuse diff --git a/atom/model_engine/model_runner.py b/atom/model_engine/model_runner.py index 13e99cbeec..5f18ce03df 100644 --- a/atom/model_engine/model_runner.py +++ b/atom/model_engine/model_runner.py @@ -1726,7 +1726,10 @@ def get_num_blocks(self) -> dict[str, object]: # sliding-window pool and the attention builder each index it by the # class name they declared. Nothing here needs to know those names. self.pool_plan = plan - config.pool_entries = dict(plan.entries) + # BlockManager divides this into request groups; the offload staging + # cushion is allocated but not leasable, so admission must not see it. + # Backends keep sizing their tensors from `pool_plan.entries`. + config.pool_entries = dict(plan.admission_entries) config.pool_entries_per_req = dict(plan.entries_per_req) # Keep runtime state metadata out of Config. transfer = self.attn_metadata_builder.state_transfer() @@ -1774,9 +1777,15 @@ def get_num_blocks(self) -> dict[str, object]: ) self.state_runtime = state_runtime for name in sorted(plan.entries): + extra = plan.entries[name] - plan.admission_entries[name] logger.info( - f"sub-pool {name}: entries={plan.entries[name]}, " - f"entry_bytes={plan.entry_bytes[name]}, " + f"sub-pool {name}: entries={plan.entries[name]}" + + ( + f" ({plan.admission_entries[name]} admissible + {extra} staging)" + if extra + else "" + ) + + f", entry_bytes={plan.entry_bytes[name]}, " f"reserved={plan.reserved_bytes[name] / (1 << 30):.2f}GB" ) @@ -1844,7 +1853,10 @@ def get_num_blocks(self) -> dict[str, object]: # the class they declared. return { "num_kvcache_blocks": num_kvcache_blocks, - "pool_entries": dict(plan.entries), + # BlockManager divides this into request groups; the offload staging + # cushion is allocated but not leasable, so admission must not see it. + # Backends keep sizing their tensors from `pool_plan.entries`. + "pool_entries": dict(plan.admission_entries), "pool_entries_per_req": dict(plan.entries_per_req), "state_runtime": state_runtime.to_wire(), } @@ -1999,6 +2011,11 @@ def allocate_kv_cache(self, num_kvcache_blocks): for key, kv_cache_tensor in zip(kv_cache_keys, kv_cache_tensors) } transfer_tensors = self.attn_metadata_builder.get_kv_transfer_tensors() + if transfer_tensors is not None: + # The tier is built inside `register_kv_caches` and needs + # `state_entry_views` to name the bytes it packs. This is the only + # place the builder and the connector are both in scope. + transfer_tensors.state_backend = self.attn_metadata_builder if hasattr(self, "eagle3_draft_builder") and transfer_tensors is not None: draft_regions = self.eagle3_draft_builder.get_kv_transfer_tensors() if draft_regions: diff --git a/atom/model_engine/scheduler.py b/atom/model_engine/scheduler.py index 30afe89607..7555317494 100644 --- a/atom/model_engine/scheduler.py +++ b/atom/model_engine/scheduler.py @@ -219,6 +219,7 @@ class CacheStats: "_interval_compressed_tokens", "_interval_evicted_base", "_interval_full_tokens", + "_interval_offload_tokens", "_interval_requests", "_interval_reusable_tokens", "_interval_wanted_tokens", @@ -228,6 +229,7 @@ class CacheStats: "total_cached_tokens", "total_compressed_tokens", "total_full_tokens", + "total_offload_tokens", "total_requests", "total_reusable_tokens", "total_wanted_tokens", @@ -246,6 +248,10 @@ def __init__( self.total_requests: int = 0 self.total_cached_tokens: int = 0 self.total_full_tokens: int = 0 + # Reuse served by the CPU tier, kept out of every series below: those + # describe the HBM prefix walk, and an LMCache load reaches past it by + # design, so folding them together makes `cached > wanted`. + self.total_offload_tokens: int = 0 # The reuse ceiling, and the only honest denominator for a hit rate. # # `full` is not reachable: `BlockManager.can_allocate` matches over @@ -276,6 +282,7 @@ def __init__( self._interval_requests: int = 0 self._interval_cached_tokens: int = 0 self._interval_full_tokens: int = 0 + self._interval_offload_tokens: int = 0 self._interval_compressed_tokens: int = 0 self._interval_wanted_tokens: int = 0 # Set by Scheduler for pool occupancy logging. @@ -290,10 +297,11 @@ def update( num_compressed_tokens: int, num_wanted_tokens: int, num_reusable_tokens: int, + num_offload_tokens: int = 0, ) -> None: """Record cache stats for one prefill sequence. - All five are required because the reported rates are differences + The first five are required because the reported rates are differences between them: `cached <= wanted <= compressed <= reusable <= full`. A defaulted argument would silently report a negative rate rather than a missing one. @@ -303,6 +311,11 @@ def update( has no stable hash, so it is never a reuse candidate). Recomputing it here would mean duplicating that rule in a second place and letting the two drift. + + `num_offload_tokens` is the exception, and defaults because it is not + part of that chain: it is reuse the CPU tier served, which sits above + the HBM walk rather than inside it. Counting it in `cached` is what + would break the nesting -- see `_schedule_prefill_seq`. """ ordered = ( num_cached_tokens @@ -349,6 +362,8 @@ def update( self._interval_reusable_tokens += num_reusable_tokens self._interval_compressed_tokens += num_compressed_tokens self._interval_wanted_tokens += num_wanted_tokens + self.total_offload_tokens += num_offload_tokens + self._interval_offload_tokens += num_offload_tokens if self.total_requests % self._log_interval == 0: self._log() @@ -433,6 +448,9 @@ def get_statistics(self) -> dict: # not a hit-rate denominator -- see `total_reusable_tokens`. "reusable_tokens": self.total_reusable_tokens, "full_tokens": self.total_full_tokens, + # Reuse from the CPU tier. Separate on purpose: it shares no + # denominator with the HBM series above. + "offload_tokens": self.total_offload_tokens, } def _reset_interval(self) -> None: @@ -441,6 +459,7 @@ def _reset_interval(self) -> None: self._interval_full_tokens = 0 self._interval_reusable_tokens = 0 self._interval_compressed_tokens = 0 + self._interval_offload_tokens = 0 self._interval_wanted_tokens = 0 @staticmethod @@ -491,6 +510,12 @@ def _log(self) -> None: f"(total {occ['evicted_total']})" ) self._log_pools() + if self.total_offload_tokens: + logger.info( + "[Cache Stats offload] CPU-tier tokens: " + f"interval={self._interval_offload_tokens} " + f"total={self.total_offload_tokens}" + ) if self._pool_pressure is not None: self._log_pressure(self._pool_pressure()) @@ -1352,6 +1377,13 @@ def _reconcile_stalled_deferred_saves(self) -> int: Mirrors the producer `finished_sending` reclaim (pop + deallocate), deliberately not re-invoking `request_finished`: it was already called when the request finished, before the block free was deferred. + + The complement of the K3 connector's stall escape + (`kimi_k3.connector.SAVE_STALL_SECONDS`), not a duplicate of it: that + one releases the blocks of a save the backend never took, on a shorter + clock, and leaves a save already handed out alone -- precisely the case + this reclaims once the report is not coming. Between them every deferred + save has a way out. """ timeout = _offload_save_abandon_timeout_s() if timeout <= 0 or not self.deferred_free_blocks: @@ -1607,6 +1639,13 @@ def schedule(self) -> tuple[ScheduledBatch, dict[int, Sequence]]: # output_queue). Must intercept here BEFORE the waiting->running # promotion below, which would overwrite ABORTED with RUNNING and # lose the abort intent. + # + # It does not follow that it holds nothing: a sequence parked for + # a transfer was allocated before it parked, and holds a full block + # table plus, for a hybrid, a state group. Leaking the group wedges + # every hybrid request once enough disconnects accumulate, since + # `can_allocate` refuses one outright with the pool empty. + # `deallocate` is a no-op for a seq that really held nothing. if seq.status == SequenceStatus.ABORTED: self._reject_aborted_waiting(seq) continue @@ -1662,6 +1701,13 @@ def schedule(self) -> tuple[ScheduledBatch, dict[int, Sequence]]: self._assert_positive_prefill_chunk( chunk, num_new_tokens, budget_remaining ) + # This branch `continue`s past both assignments below, so the + # refresh happens here too: `_mark_offload_load_ready` raised + # `num_cached_tokens` to the post-load hit, and the stale field + # is what reaches the API as `cached_tokens` -- the tier would + # look like it returned nothing. Unconditional, because a seq + # re-admitted after `preempt()` keeps the field. + seq.prefix_cache_hit_tokens = seq.num_cached_tokens num_seqs_prefill, num_batched_tokens = self._schedule_prefill_seq( seq, chunk, @@ -1727,12 +1773,16 @@ def schedule(self) -> tuple[ScheduledBatch, dict[int, Sequence]]: break self.block_manager.allocate(seq, num_cached_blocks) + # The hit the request kept, not the one `can_allocate` offered: + # `allocate` may disown the boundary, and `num_cached_blocks` is the + # pre-disown count. `seq.num_cached_tokens` also gets the unit right + # under DCP, and is what CacheStats uses, so the user-visible number + # and the internal hit rate cannot disagree. + # # Guard: PD decode consumer inherits hit from prefill node; - # don't clobber with local num_cached_blocks (always 0 on consumer). + # don't clobber with the local hit (always 0 on consumer). if not seq.prefix_cache_hit_tokens: - seq.prefix_cache_hit_tokens = ( - num_cached_blocks * self.block_manager.hash_block_size - ) + seq.prefix_cache_hit_tokens = seq.num_cached_tokens self._notify_connector_after_prefill_alloc(seq) @@ -1741,12 +1791,53 @@ def schedule(self) -> tuple[ScheduledBatch, dict[int, Sequence]]: ) if needs_remote_load: + if seq.state_load_hash != -1 and not seq.state_joint_boundary_tokens: + # Two transfers but one report: the first completion would + # unpark the request while the other is still writing. Only + # reachable when the legs were decided separately -- a joint + # load arrives as one event via `_JointPark`. + logger.warning( + "seq %s has both a remote KV load and a state load " + "pending, with no joint boundary; dropping the state " + "load.", + seq.id, + ) + self.block_manager.cancel_state_load(seq) self._park_for_remote_load(seq, skipped_waiting_requests) continue - seq.prefix_cache_hit_tokens = ( - num_cached_blocks * self.block_manager.hash_block_size - ) + if seq.state_joint_boundary_tokens: + # The KV leg was refused after the state leg was aimed at the + # joint boundary, so the state claims a prefix whose KV nobody + # will fetch. Disown it; `_decide_load_after_alloc` logs why. + logger.debug( + "[JOINT-DISOWN] seq %s: KV leg refused at boundary %d; " + "recomputing from %d", + seq.id, + seq.state_joint_boundary_tokens, + seq.num_cached_tokens, + ) + if seq.state_load_hash != -1: + self.block_manager.cancel_state_load(seq) + seq.num_cached_tokens = 0 + seq.state_joint_boundary_tokens = 0 + seq.state_joint_boundary_hash = -1 + + if seq.state_load_hash != -1: + # The kept state boundary is in LMCache, not HBM. It parks + # exactly as a KV load does -- same queue, same backpressure, + # same wake-up -- because to the scheduler both are one event: + # a transfer into blocks this request already holds. + self._park_for_remote_load(seq, skipped_waiting_requests) + continue + + # Refresh, not a duplicate of the set above: that one is guarded + # on the field being empty, so a seq re-admitted after `preempt()` + # would keep its previous admission's hit. Unconditional here, where + # the request is certain to prefill locally. The only path reaching + # this with an inherited PD hit is a *failed* remote recv, and there + # the local hit is the truthful number. + seq.prefix_cache_hit_tokens = seq.num_cached_tokens chunk = self._adjust_prefill_chunk_after_alloc(seq, chunk) chunk = self._finalize_prefill_chunk(seq, seq.num_cached_tokens, chunk) @@ -1780,6 +1871,32 @@ def schedule(self) -> tuple[ScheduledBatch, dict[int, Sequence]]: self._kv_usage(), ) + if self._schedule_tick % 100 == 0: + fates = self.block_manager.state_checkpoint_fates() + if any(fates.values()): + logger.info( + "state checkpoints: %s", + " ".join(f"{k}={v}" for k, v in sorted(fates.items())), + ) + # Its own line rather than a key in the one above: those are what + # became of a checkpoint, this is whether a prefix in LMCache could + # be paired with one at all. Silent unless the joint load is on, + # since both counters stay 0 without it. + chosen = self.block_manager.joint_boundaries + skips = self.block_manager.joint_skips + if chosen or skips: + # The hbm/tier split is the actionable half: both are reuse, + # but only `tier` costs an entry-sized H2D and a park, so a + # ratio dominated by it says the state pool is too small for + # this concurrency even while the load counters look healthy. + logger.info( + "joint kv: boundaries=%d (hbm=%d tier=%d) | %s", + chosen, + self.block_manager.joint_boundaries_hbm, + self.block_manager.joint_boundaries_tier, + " ".join(f"{k}={v}" for k, v in sorted(skips.items())), + ) + total_tokens_num_prefill = sum(num_scheduled_tokens) if num_seqs_prefill > 0: @@ -1801,6 +1918,7 @@ def schedule(self) -> tuple[ScheduledBatch, dict[int, Sequence]]: connector_meta_output = None if self.kv_connector is not None: + self._publish_state_loads() connector_meta_output = self.kv_connector.build_connector_meta() # Freeze, per seq, whether this chunk finishes the prompt. Uses the @@ -1960,6 +2078,7 @@ def schedule(self) -> tuple[ScheduledBatch, dict[int, Sequence]]: connector_meta_output = None if self.kv_connector is not None: + self._publish_state_loads() connector_meta_output = self.kv_connector.build_connector_meta() decode_batch = ScheduledBatch( @@ -1976,6 +2095,9 @@ def schedule(self) -> tuple[ScheduledBatch, dict[int, Sequence]]: remote_kv_block_ids=sorted(remote_kv_blocks) if remote_kv_blocks else [], remote_kv_seq_blocks=remote_kv_seq_blocks, # An empty batch cannot execute queued maintenance. + # For spills that is not merely wasted work: a drained spill is + # released only once its copy reaches a forward, so draining into a + # batch that is never forwarded strands every slot it drained. state_maintenance_ops=( self.block_manager.take_state_maintenance_ops() if scheduled_seqs @@ -2032,6 +2154,22 @@ def _consume_failed_remote_kv(self, seq: Sequence) -> bool: seq.status = SequenceStatus.WAITING if not self._connector_flag("is_offload"): self._uncount_inflight_load(seq) + if getattr(seq, "state_load_hash", -1) != -1 or getattr( + seq, "state_joint_boundary_tokens", 0 + ): + # The state never arrived, so the boundary is not this request's + # history. Disown it exactly as `BlockManager.allocate` does at + # admission, otherwise the resume runs with `num_cached_tokens > 0` + # over a group nobody filled -- `has_initial_state` reads that as + # "the recurrence continues". Before `offload_loaded_tokens` is set + # below, so that field records the disowned figure. + seq.num_cached_tokens = 0 + seq.state_load_hash = -1 + # A joint boundary that failed on either leg is not this request's + # history any more than a state-only one is, and leaving it set + # would let the connector clamp a later lookup to it. + seq.state_joint_boundary_tokens = 0 + seq.state_joint_boundary_hash = -1 seq.offload_loaded = False seq.offload_loaded_tokens = seq.num_cached_tokens seq.offload_load_start_tokens = None @@ -2107,6 +2245,14 @@ def _mark_offload_load_ready(self, seq: Sequence) -> None: seq.prefix_cache_hit_tokens = max(seq.prefix_cache_hit_tokens, loaded) seq.offload_load_start_tokens = None seq.offload_loaded = True + # A state load moves no KV, so the block above does nothing for it -- + # `num_cached_tokens` is already the boundary the state covers, and all + # that is left is to stop calling the load pending. A joint load is the + # one case where both halves fire, and `_JointPark` held the wake until + # both legs reported. + seq.state_load_hash = -1 + seq.state_joint_boundary_tokens = 0 + seq.state_joint_boundary_hash = -1 def _is_offload_prefill_resume(self, seq: Sequence) -> bool: """True when offload already owns blocks and should resume suffix prefill. @@ -2128,9 +2274,17 @@ def _query_connector_prefill_match(self, seq: Sequence, *, skip: bool) -> bool: """Ask the connector whether this prefill should park for remote KV.""" if skip or self.kv_connector is None: return False - _ext_tokens, needs_remote_load = self.kv_connector.get_num_new_matched_tokens( + ext_tokens, needs_remote_load = self.kv_connector.get_num_new_matched_tokens( seq ) + # Keep the lookup's answer, not just the park flag: `can_allocate` runs + # next and a hybrid's two legs have to be aimed at one boundary, for + # which this is the KV leg's ceiling. In tokens, and absolute -- the + # connector reports what it found *beyond* what the seq already has. + seq.offload_kv_prefix_tokens = int(ext_tokens) + int(seq.num_cached_tokens) + seq.offload_kv_chunk_tokens = int( + getattr(self.kv_connector, "chunk_size", 0) or 0 + ) return needs_remote_load def _schedule_first_decode_after_remote_kv(self, seq: Sequence) -> None: @@ -2299,12 +2453,21 @@ def _schedule_prefill_seq( # ceiling is genuinely zero — nothing about it is reusable. n_hash_blocks = (seq.num_tokens + hbs - 1) // hbs num_reusable_tokens = min(max(n_hash_blocks - 1, 0) * hbs, seq.num_tokens) + # `num_cached_tokens` sits above the HBM walk once a load landed + # -- that is the feature. Those tokens are not paged-pool reuse, so + # they go in their own series rather than inflating `cached` and + # making `wanted - cached` negative. `seq.num_cached_tokens` is + # deliberately NOT clamped: the forward must skip what was loaded. + wanted_tokens = seq.num_wanted_hit_blocks * hbs + hbm_cached_tokens = min(seq.num_cached_tokens, wanted_tokens) + offload_tokens = seq.num_cached_tokens - hbm_cached_tokens self.cache_stats.update( - seq.num_cached_tokens, + hbm_cached_tokens, seq.num_tokens, seq.num_compressed_hit_blocks * hbs, - seq.num_wanted_hit_blocks * hbs, + wanted_tokens, num_reusable_tokens, + num_offload_tokens=offload_tokens, ) num_batched_tokens += chunk seq.status = SequenceStatus.RUNNING @@ -2327,6 +2490,51 @@ def _confirm_remote_load_after_alloc( return self.kv_connector.should_park_for_load_after_alloc(seq) return True + def _settle_state_load(self, req_id, ok: bool) -> None: + """Release the state group a load reserved. No-op for an id the state + index never issued, and for a scheduler built without a block manager + (the connector test doubles).""" + settle = getattr( + getattr(self, "block_manager", None), "settle_state_load", None + ) + if settle is not None: + settle(req_id, ok=ok) + + def _publish_state_loads(self) -> None: + """Hand this pass's state-tier loads to the connector, before it builds + its metadata. + + Drained here rather than at the park, because the connector publishes + once per pass and a load handed over twice would write the same entry + into a group the first transfer is already filling. + """ + if self.kv_connector is None: + return + loads = self.block_manager.take_state_loads() + if not loads: + return + accepted = False + if hasattr(self.kv_connector, "enqueue_state_loads"): + accepted = bool(self.kv_connector.enqueue_state_loads(loads)) + if accepted: + return + # Nothing will carry them and the requests are parked against a report + # that would never come, so wake them for recompute -- `failed_loading` + # means "recompute over the blocks already allocated". + # + # `abandon_state_load`, not `settle(ok=False)`: nothing was attempted, + # so this is not a miss to count against the index. + logger.warning( + "state offload: %s did not carry %d state load(s); waking those " + "requests to recompute. The tier's index should not have been " + "installed against this connector.", + type(self.kv_connector).__name__, + len(loads), + ) + for req_id, _h, _group in loads: + self.block_manager.abandon_state_load(req_id) + self.failed_recving_kv_req_ids.append(req_id) + def _park_for_remote_load( self, seq: Sequence, skipped_waiting_requests: deque[Sequence] ) -> None: @@ -3141,9 +3349,16 @@ def _update_from_kv_xfer_finished(self, kv_connector_output: KVConnectorOutput): ) self.failed_recving_kv_req_ids.append(req_id) + # The two loading channels carry state-tier loads as well as KV ones, + # which buys the state leg the aggregator's per-request quorum for free. + # They cannot be confused: a KV load is refused for any sequence with a + # per-request cache, and a state load exists only for such a sequence. for req_id in kv_connector_output.finished_loading or (): assert is_offload, "Only offload connector should update loading KV status" logger.debug("Finished offload KV load for request %s", req_id) + # Ahead of the abort guard: an aborted request's state group has to + # be released too, and the guard takes the `continue`. + self._settle_state_load(req_id, ok=True) if self._finish_aborted_load_cleanup(req_id): continue self.finished_recving_kv_req_ids.append(req_id) @@ -3156,6 +3371,7 @@ def _update_from_kv_xfer_finished(self, kv_connector_output: KVConnectorOutput): "Offload KV load failed for request %s; falling back to prefill.", req_id, ) + self._settle_state_load(req_id, ok=False) if self._finish_aborted_load_cleanup(req_id): continue self.failed_recving_kv_req_ids.append(req_id) @@ -3180,6 +3396,31 @@ def _update_from_kv_xfer_finished(self, kv_connector_output: KVConnectorOutput): if seq is not None: self._maybe_release_deferred(seq) + # The state-offload reports, not keyed by request: by the time a spill + # lands its owner is long gone and only the hash and slot remain. They + # ride the connector's completion channels, so they are drained from the + # connector with TP quorum already taken. + offload = getattr(getattr(self, "block_manager", None), "state_offload", None) + take = getattr(self.kv_connector, "take_state_reports", None) + if offload is not None and take is not None: + indexed, released, _failed = take() + # Index before releasing, always. The reverse order opens a window + # in which the slot is reusable but its hash is not yet findable, + # so a lookup misses a checkpoint that is in fact stored. + for h in indexed: + offload.confirm_spill(int(h)) + for slot in released: + offload.release_staging(int(slot)) + # Ring-side twin of `_reconcile_stalled_deferred_saves`: a slot whose + # spill report never returns (worker abandoned/lost the transfer) + # would pin the ring until every spill drops. Reclaim it once the + # same abandon window the engine uses has elapsed -- larger than the + # upstream force-unpin timeout, so the staging buffer is no longer in + # flight before the slot is reused. Self-throttled inside. + reclaim = getattr(offload, "reclaim_stale_slots", None) + if callable(reclaim): + reclaim(_offload_save_abandon_timeout_s()) + def get_request_counts(self) -> tuple[int, int]: """Returns (num_running_reqs, num_waiting_reqs).""" return len(self.running), len(self.waiting) diff --git a/atom/model_engine/sequence.py b/atom/model_engine/sequence.py index 09e55e2d36..7fe38ceb59 100644 --- a/atom/model_engine/sequence.py +++ b/atom/model_engine/sequence.py @@ -207,6 +207,36 @@ def __init__( # forward. -1 = read and write the same slot, the case for every step in # between. Always a single slot: a checkpoint is one slot wide. self.state_fork_src = -1 + # Content hash of a checkpoint the tier was asked to fetch back into + # `state_slot`, or -1. Tells the scheduler to park, and tells + # the failure path that `num_cached_tokens` is claiming undelivered + # state. + self.state_load_hash = -1 + # The LMCache-resident KV prefix in tokens, as this admission's lookup + # reported it -- the KV leg's ceiling. Written before `can_allocate`, + # which is where the two legs agree on one boundary. 0 = no lookup. + self.offload_kv_prefix_tokens = 0 + # The connector's LMCache chunk size, in tokens, or 0. The KV leg moves + # whole chunks, so a joint boundary has to be a multiple of it. + self.offload_kv_chunk_tokens = 0 + # The boundary both legs of a joint load are aimed at, or 0. Chosen by + # `can_allocate`: the rightmost checkpoint rung the LMCache KV prefix + # covers. `num_cached_tokens` stays at the HBM prefix until both legs + # report, which keeps the forward honest if either fails. + self.state_joint_boundary_tokens = 0 + # Content hash of that boundary's last block, for the state leg. + self.state_joint_boundary_hash = -1 + # How far the KV leg transfers, in tokens: the LMCache chunk that + # *covers* the boundary above, which is at or past it. The request still + # only claims the boundary -- see `_claim_after_load`. + self.state_joint_kv_tokens = 0 + # How far `allocate` claimed straight out of the HBM prefix cache, in + # tokens. Above `num_cached_tokens`, not below it: the blocks in + # `(hit, compressed_hit]` hold this prompt's KV and are still indexed -- + # the state gate cut the hit, the KV never went anywhere. Claiming them + # is what keeps the KV leg from paying LMCache to resend what the pool + # already has, and it is where that leg starts. + self.state_joint_claim_tokens = 0 self.temperature = sampling_params.temperature self.top_k = sampling_params.top_k self.top_p = sampling_params.top_p diff --git a/atom/model_engine/state_offload.py b/atom/model_engine/state_offload.py new file mode 100644 index 0000000000..4ff04e64f0 --- /dev/null +++ b/atom/model_engine/state_offload.py @@ -0,0 +1,369 @@ +# SPDX-License-Identifier: MIT +"""Scheduler-side bookkeeping for the state-cache offload tier. + +Two things live here and neither touches a device: the set of hashes believed +to be in LMCache, and the bounded queue of groups waiting to be read out of +HBM. The bytes and the transfers are the worker's business +(`kv_transfer/offload/state_tier.py`). + +The set is in memory and never persisted: `LocalDiskBackend.__init__` starts +from an empty dict and never scans its directory, so an index recovered from +disk would be a pure false-positive generator. Index and bytes share one +server lifetime. +""" + +import logging +import os +import time +from collections import deque + +logger = logging.getLogger("atom") + +# How often `reclaim_stale_slots` actually scans, regardless of how often it is +# called. The scan is O(staging_depth) and self-throttled so wiring it into a +# per-step drain loop costs nothing when nothing is stuck. +_RECLAIM_SCAN_INTERVAL_S = 5.0 + +# Consecutive dropped spills, with no slot coming back, before the ring is +# called starved rather than busy. A slot returns only once the worker's D2H +# lands, so a burst of evictions can legitimately drop for a step or two; an +# order of magnitude above that, so crossing it means the slots are gone. +_STARVATION_DROP_THRESHOLD = 256 + + +class StateOffloadIndex: + """What has been spilled, what is queued to be, and what is coming back. + + `hashes` answers the membership half of `StateSlotPool._resumable_from`. + It is deliberately optimistic: LMCache's own LRU can drop bytes at any + time, so a hash here means "was spilled once", never "is still there". The + false positive costs one lookup and a park/unpark, and is handled by the + `failed_loading` path (`fail_load` -> `forget`). + """ + + def __init__(self, staging_depth: int) -> None: + self.staging_depth = max(0, int(staging_depth)) + # Orphaned checkpoints (`unindex`) are worth spilling only when the KV + # prefix can also come back: `resumable_hit` scans `block_hashes`, which + # `can_allocate` builds from HBM `kv.lookup` hits only. With KV offload + # off, a hash whose KV left HBM never reappears and the bytes are wasted. + # + # In production today this is always True: `BlockManager.__init__` + # refuses to construct the index at all when the connector does not + # host the tier, so the only path here has already established it. The + # flag is kept, and the `unindex` gate with it, because the two + # conditions are independent in principle -- a future connector could + # host the tier's transport without offloading KV -- and because the + # failure it prevents (spending LMCache capacity on hashes no load can + # ever reach) is silent. Its False arm is covered by tests only. + self.hashes: set[int] = set() + self._free_slots: deque[int] = deque(range(self.staging_depth)) + self._pending: deque[tuple[int, int]] = deque() + # req_id -> hash, for loads offered and not yet settled. Keyed by + # request because that is what comes back, on the same + # `finished_loading`/`failed_loading` channel a KV load uses. (Spill + # reports are keyed by hash and slot -- their owner is long gone.) + self.pending_loads: dict = {} + # Read once, here, rather than per call: `_resumable_from` consults it + # inside a per-block scan. + self.spills_requested = 0 + self.spills_dropped = 0 + # Evictions that deliberately did not spill because the group was + # going to a load. Apart from `spills_dropped`: a deeper ring does not + # fix this one, it is the price of the load. + self.spills_forgone = 0 + self.loads_attempted = 0 + self.loads_completed = 0 + self.loads_failed = 0 + # Drops since the last `release_staging`. A full ring is normal + # backpressure only while slots come back; none in this many attempts + # means the consumer stopped draining. + self._consecutive_drops = 0 + self._warned_starved = False + # slot -> time.monotonic() when it was reserved by `request_spill`. + # A slot's only way home is `release_staging`, driven by the worker's + # spill report. If that report never comes (the worker abandoned the + # transfer, or its completion was lost the same way a KV save's can be), + # the slot is pinned forever and the ring silently starves. This stamp + # lets `reclaim_stale_slots` return a slot the report has clearly + # abandoned. Reclaimed slots (`_slots_reclaimed`) are counted so the + # symptom is visible rather than a silent throughput cliff. + self._slot_reserved_at: dict[int, float] = {} + self._next_reclaim_at: float = 0.0 + self._slots_reclaimed = 0 + + @property + def enabled(self) -> bool: + return self.staging_depth > 0 + + def request_spill(self, h: int, group: int) -> int: + """Reserve a staging slot for `group`, or -1 if the ring is full. + + On `pop()`'s critical path, so no work beyond a deque pop. The caller + copies `group` into the slot on the compute stream and `pop()` hands the + original out immediately -- spilling by copy rather than by pin, because + `pop()` runs precisely when there is no free group to withhold. + """ + # A negative hash is a caller bug, not a dropped spill -- `_spill` + # already refuses a group with no checkpoint. Guarded, but not counted, + # so `spills_dropped` stays a clean backpressure signal. + if h < 0: + return -1 + if not self._free_slots: + self.spills_dropped += 1 + self._note_drop() + return -1 + slot = self._free_slots.popleft() + self._pending.append((h, slot)) + self._slot_reserved_at[slot] = time.monotonic() + self.spills_requested += 1 + return slot + + def _note_drop(self) -> None: + """Warn once when drops stop looking like a burst and start looking + like a consumer that has gone away.""" + self._consecutive_drops += 1 + if self._warned_starved or self._consecutive_drops < _STARVATION_DROP_THRESHOLD: + return + self._warned_starved = True + logger.warning( + "State offload staging ring is starved: %d spills dropped in a row " + "with no slot released (staging_depth=%d). The consumer is very " + "likely not calling StateSlotPool.take_spill_copies() every step, " + "or not calling StateOffloadIndex.release_staging(slot) once the " + "bytes are safe. Every further spill will be dropped until it does.", + self._consecutive_drops, + self.staging_depth, + ) + + def take_pending(self) -> list[tuple[int, int]]: + """Drain the queue as `(hash, staging_slot)` pairs.""" + out = list(self._pending) + self._pending.clear() + return out + + def confirm_spill(self, h: int) -> None: + """Index `h` once its bytes reached LMCache.""" + self.hashes.add(h) + + def release_staging(self, slot: int) -> None: + if 0 <= slot < self.staging_depth and slot not in self._free_slots: + self._free_slots.append(slot) + self._slot_reserved_at.pop(slot, None) + # The ring is moving again, so any drops so far were a burst. + self._consecutive_drops = 0 + + def reclaim_stale_slots(self, timeout_s: float, now: float | None = None) -> int: + """Return slots whose spill report never came back to the free ring. + + `release_staging` is the only way a slot comes home, and it fires only + on the worker's spill report. A lost report would pin that slot forever; + enough of them and `request_spill` drops every spill (`spills_dropped` + climbs, `_note_drop` warns) with no way to recover short of a restart. + The ring-side twin of the engine's `_reconcile_stalled_deferred_saves`, + and it takes that reconciler's window so the two cannot drift. + + **Only slots the spill queue has let go are eligible.** A slot still in + `_pending` has not been drained by `BlockManager.state_spills_for_batch` + yet, and that method drains `take_pending()` and the pool's + `take_spill_copies()` in one call -- so a slot in `_pending` still has a + `(slot, staging)` copy queued on `StateSlotPool`. Returning it would let + a new spill reserve the same slot, and the two copies would then race + for the single `{slot: hash}` entry the drain builds: the loser is + reported as "a copy with no pending hash" and releases the slot out from + under the winner. Skipping those costs nothing -- an undrained spill + means the engine has not run a batch, which is what + `_reconcile_stalled_deferred_saves` is busy fixing; the slot becomes + eligible on the next scan once the batch drains it. + + Safety otherwise mirrors the engine-side reconciliation: `timeout_s` is + past the upstream force-unpin window, so the worker's staging buffer is + no longer being read before the slot is handed to a new spill. + Self-throttled to one real scan per `_RECLAIM_SCAN_INTERVAL_S`, so it is + safe to call every step. Returns the number of slots reclaimed. + """ + if timeout_s <= 0 or not self._slot_reserved_at: + return 0 + now = time.monotonic() if now is None else now + if now < self._next_reclaim_at: + return 0 + self._next_reclaim_at = now + _RECLAIM_SCAN_INTERVAL_S + queued = {slot for _h, slot in self._pending} + stale = [ + slot + for slot, at in self._slot_reserved_at.items() + if now - at >= timeout_s and slot not in queued + ] + if not stale: + return 0 + for slot in stale: + self._slot_reserved_at.pop(slot, None) + if 0 <= slot < self.staging_depth and slot not in self._free_slots: + self._free_slots.append(slot) + self._slots_reclaimed += len(stale) + # A reclaim means the ring was stuck, not bursting; reset the burst + # counter so a fresh starvation warning can arm if it stalls again. + self._consecutive_drops = 0 + self._warned_starved = False + logger.warning( + "State offload staging ring reclaimed %d slot(s) whose spill report " + "never returned after %.0fs (staging_depth=%d, total reclaimed=%d). " + "The worker very likely abandoned or lost the transfer's completion; " + "the slot is returned to the ring so spills can resume.", + len(stale), + timeout_s, + self.staging_depth, + self._slots_reclaimed, + ) + return len(stale) + + def forget(self, h: int) -> None: + """Drop a hash whose load failed, so the next request does not retry.""" + self.hashes.discard(h) + + # -------------------------------- loads -------------------------------- # + def request_load(self, req_id, h: int) -> bool: + """Offer to fetch `h` back for `req_id`. False if this tier cannot. + + The guard between believing and delivering: a load is resolved only by + a worker report, so offering one for a hash never stored would park the + request against bytes no `get` can produce. + """ + if h not in self.hashes: + return False + if req_id in self.pending_loads: + # One request, one outstanding load: reports are keyed by request + # id, so the first completion would unpark while the second is + # still writing. No in-tree path reaches this today, and the + # refusal costs only a disown. + logger.warning( + "state offload: request %s already has a load in flight; " + "refusing a second one and letting the boundary be disowned.", + req_id, + ) + return False + self.pending_loads[req_id] = int(h) + self.loads_attempted += 1 + return True + + def complete_load(self, req_id) -> None: + """The bytes landed in the request's group. + + The hash stays indexed: a load reads LMCache, it does not consume it, + and the next request over the same prefix must still find it. + """ + if self.pending_loads.pop(req_id, None) is not None: + self.loads_completed += 1 + + def fail_load(self, req_id) -> None: + """No bytes came back. Retract the claim as well as counting it. + + A miss is the only evidence that LMCache's LRU dropped what this index + advertises, so it has to be what un-advertises it. Leaving the hash makes + every later request over that prefix park, miss and recompute. + """ + h = self.pending_loads.pop(req_id, None) + if h is None: + return + self.loads_failed += 1 + self.forget(h) + + def abandon_load(self, req_id) -> None: + """The request went away before its bytes did. Neither outcome. + + Not `fail_load`: an abort says nothing about the bytes, and forgetting + a loadable hash sends the next request back to a full recompute. + """ + self.pending_loads.pop(req_id, None) + + def stats(self) -> dict[str, int]: + """Counters for the periodic `state checkpoints:` line. + + Reached via `StateSlotPool.checkpoint_fates`, which prefixes every + key with `state_offload_`. Read the three load counters together: + `failed / attempted` is this index's false-positive rate, and + `attempted - completed - failed` is what is in flight or was abandoned + by an aborted request. + """ + return { + "spills_requested": self.spills_requested, + "spills_dropped": self.spills_dropped, + "spills_forgone": self.spills_forgone, + "loads_attempted": self.loads_attempted, + "loads_completed": self.loads_completed, + "loads_failed": self.loads_failed, + "indexed": len(self.hashes), + # Slots `reclaim_stale_slots` had to take back. Non-zero means spill + # reports are being lost, which the warning says once and this keeps + # saying; without it the only symptom is a throughput cliff. + "slots_reclaimed": self._slots_reclaimed, + } + + +# The connector backends whose worker half builds a `StateOffloadTier`. Only +# the offload backend does; `multi` qualifies when it lists one. +_STATE_TIER_BACKENDS = frozenset({"lmcache_offload"}) + + +def kv_connector_hosts_state_tier(kv_transfer_config) -> bool: + """Whether the configured KV connector can actually run the state tier. + + The tier is not standalone: its bytes ride the KV connector's worker half + and its completions ride that connector's `get_finished`. Against any other + backend the staging ring would hand out slots and never get one back, so + this gates whether the ring is installed at all -- and a false negative + (tier off) is much cheaper than a false positive (every slot leaks). + + A name check rather than a capability probe: this runs in the engine + process at `BlockManager.__init__`, before any worker connector exists. + """ + cfg = kv_transfer_config or {} + if not isinstance(cfg, dict): + return False + name = cfg.get("kv_connector", "moriio") + if name == "multi": + subs = cfg.get("connectors") or () + return any( + isinstance(s, dict) and s.get("kv_connector") in _STATE_TIER_BACKENDS + for s in subs + ) + return name in _STATE_TIER_BACKENDS + + +def state_offload_staging_groups() -> int: + """K: staging *groups* to reserve for the tier. + + Groups, not entries: a sizing site multiplies by its `entries_per_req`, + because `state_entry_views` is indexed by group. One function rather than + two `os.environ` reads, because the arena and the pool both size themselves + from it and would otherwise disagree. + + This only sizes the ring; whether the tier runs follows the connector. Bad + depths warn rather than raise (model load): non-integer falls back to 1, + negative floors to 0. + """ + raw = os.environ.get("OFFLOAD_STATE_STAGING_GROUPS") + if raw is None: + return 1 + try: + depth = int(raw) + except ValueError: + # Warn rather than raise -- this runs during model load. Not silent + # either: a mistyped depth buys one group instead of twenty, with a + # starved ring 256 dropped spills later as the only other symptom. + logger.warning( + "state offload: invalid OFFLOAD_STATE_STAGING_GROUPS=%r; using 1", + raw, + ) + return 1 + if depth < 0: + # Same reasoning as the typo above, and a worse outcome: a negative + # depth turns the whole tier off, so the operator gets a + # healthy-looking server that never spills. + logger.warning( + "state offload: negative OFFLOAD_STATE_STAGING_GROUPS=%r disables " + "the tier entirely; using 0", + raw, + ) + return 0 + return depth diff --git a/atom/model_engine/state_pool.py b/atom/model_engine/state_pool.py index 262ec46f09..144953c64d 100644 --- a/atom/model_engine/state_pool.py +++ b/atom/model_engine/state_pool.py @@ -1,12 +1,19 @@ # SPDX-License-Identifier: MIT # Copyright (C) 2024-2026, Advanced Micro Devices, Inc. All rights reserved. +import logging from collections import deque from dataclasses import dataclass from heapq import heapify, heappop, heappush +from typing import TYPE_CHECKING from atom.model_engine.state_runtime import StateTransfer +if TYPE_CHECKING: + from atom.model_engine.state_offload import StateOffloadIndex + +logger = logging.getLogger("atom") + @dataclass(frozen=True) class SlotRetirement: @@ -100,6 +107,15 @@ def __init__( # Reverse map, for lazy eviction when a slot is handed out. -1 = the # slot carries no checkpoint. self.slot_hash: list[int] = [-1] * num_slots + # Backing store beneath the pool, or None. Not a member of + # `BlockManager.state_caches`: every one of those is a veto, and this + # tier does the opposite -- it makes more boundaries reachable. + self.offload: StateOffloadIndex | None = None + # (slot, staging_slot) pairs the next forward must copy out of HBM. + self._spill_copies: list[tuple[int, int]] = [] + # One-shot latch so the undrained-consumer diagnosis cannot flood the + # log from `pop()`, which is itself a hot path. + self._warned_spill_copies_undrained = False # Groups serving as a fork source in the in-flight step, counted by how # many requests fork off each. They stay off the free list until # `release_pins`, so the step that reads them cannot race a request that @@ -151,12 +167,21 @@ def holds_checkpoint(self, slot: int) -> bool: """Whether `slot` is on the free list *and* still worth something.""" return slot in self._free and self.slot_hash[slot] != -1 - def pop(self) -> int: + def pop(self, *, spill: bool = True) -> int: """Hand out a slot, evicting its checkpoint if it carried one. Vacant first, lowest index first; only when nothing is vacant does a checkpoint get spent, and then it is the least recently used one. + `spill=False` gives up the evicted checkpoint's bytes instead of + staging them, and exists for one caller: a slot about to be filled by + an offload *load*. The spill copy is issued by the next forward while + the load runs on the tier's own thread ahead of it, so the two would + race over the same slot and the spill would store the loaded state + under the evicted checkpoint's hash. Nothing downstream could tell. + Forgoing the spill costs one checkpoint -- exactly the pre-tier + behaviour, counted as `spills_forgone`. + The state twin of `BlockManager._pop_free_block`: slots sit on the free list carrying whatever the last owner left in them, and re-allocation — not the free — is the eviction event. @@ -173,6 +198,12 @@ def pop(self) -> int: slot = self._checkpointed.popleft() self._free.discard(slot) self.checkpoints_evicted += 1 + if spill: + # Before invalidate() clears slot_hash[slot]: the hash is the + # key the tier stores under. + self._spill(slot) + elif self.offload is not None and self.offload.enabled: + self.offload.spills_forgone += 1 # Whatever guess this slot carried is spent with it. Not folded into # `invalidate`, which also runs on the re-file inside `_index` and would # clear a mark that was only just made. @@ -456,8 +487,9 @@ def resumable_hit( return hit hbs = self.hash_block_size for i in range(hit - 1, -1, -1): - checkpointed = block_hashes[i] in self.hash_to_slot - if not assume_checkpointed and not checkpointed: + if not assume_checkpointed and not self._resumable_from( + block_hashes[i], (i + 1) * hbs + ): continue if seq.num_tokens - (i + 1) * hbs >= self.min_fork_tokens: return i + 1 @@ -566,6 +598,88 @@ def cancel_midstep(self, reservations: list[tuple]) -> None: for slot, _pos, _h in reservations: self.release(slot) + # ------------------------------- offload ------------------------------- # + def _resumable_from(self, h: int, tokens: int) -> bool: + """Whether a checkpoint for `h` can be *reached*, in HBM or beneath it. + + Reached, not merely indexed. `resumable_hit` scans right to left and + stops at the first boundary this accepts, so accepting one whose bytes + nothing can deliver does not cost a wasted lookup — it costs the whole + walk-back, hiding every shorter checkpoint still resident in HBM. + + HBM wins without a preference rule: both tiers are indexed by the same + hash, so the scan takes the rightmost boundary wherever it lives. + """ + if h in self.hash_to_slot: + return True + # Invariant, relied on to keep this hot path free of an `enabled` check: + # a disabled (depth-0) tier never reaches `request_spill`, because + # `_spill` returns on `not enabled` first, so `hashes` stays empty and + # the membership test is false anyway. Anything that adds to `hashes` + # outside the confirm-after-spill path must re-check `enabled` here. + return self.offload is not None and h in self.offload.hashes and tokens > 0 + + def _spill(self, slot: int) -> None: + """Stage `slot`'s bytes for the tier, if there is room. Never blocks. + + Beyond the staging depth the spill is simply dropped and + `checkpoints_evicted` still counts it — identical to the behaviour + without a tier, so there is no regression to reason about. + """ + if self.offload is None or not self.offload.enabled: + return + h = self.slot_hash[slot] + if h == -1: + return + staging = self.offload.request_spill(h, slot) + if staging >= 0: + self._spill_copies.append((slot, staging)) + # Exact leak detector, not a heuristic: a staging slot must be + # released before it can be handed out again, so at most + # `staging_depth` copies can be outstanding if the consumer drains + # every step. More is proof `take_spill_copies` is not being called. + if ( + len(self._spill_copies) > self.offload.staging_depth + and not self._warned_spill_copies_undrained + ): + self._warned_spill_copies_undrained = True + logger.warning( + "StateSlotPool has %d undrained spill copies for a staging " + "ring of depth %d. take_spill_copies() must be called every " + "step and each slot released with release_staging(); until " + "it is, the staging ring leaks and every spill is dropped.", + len(self._spill_copies), + self.offload.staging_depth, + ) + + def has_pending_spill(self, slot: int) -> bool: + """Whether a queued spill copy still has to read `slot`. + + The window a load must not enter. It closes when the copy is issued on + the compute stream (`take_spill_copies` -> the next forward), after + which only the staging entry is read and the slot is free to be + overwritten. Normally empty for a slot `pop(spill=False)` just handed + out; a copy left undrained by a batch that was never forwarded is the + case this catches. + """ + return any(src == slot for src, _staging in self._spill_copies) + + def take_spill_copies(self) -> list[tuple[int, int]]: + """`(slot, staging_slot)` pairs the next forward must copy. + + The consumer turns each into `relocate_state_slots([(slot, + num_slots + staging)])` -- the ring is K entries appended past the + pool's own range, so a staging slot needs no second addressing scheme. + + Not optional: whoever attaches a tier MUST drain this every step and + MUST call `release_staging(staging)` once the bytes are safe. Skip the + drain and the list grows without bound; skip the release and + `request_spill` returns -1 forever, silently evicting every checkpoint. + Both are warned about once, but a warning is a diagnosis, not a repair. + """ + out, self._spill_copies = self._spill_copies, [] + return out + # ---------------------------- checkpointing ---------------------------- # def checkpoint(self, seq, boundary_blocks: int, h: int) -> None: """Keep `seq`'s state as of this boundary, filed under hash `h`. @@ -618,13 +732,24 @@ def checkpoint_fates(self) -> dict[str, int]: holding its working set and the misses are somewhere else — most likely `checkpoints_orphaned`, whose KV prefix died first and which more slots cannot prevent. + + Folds in the tier's counters when one is attached, which is what makes + `spills_dropped` visible -- a ring dropping one spill in three never + trips the 256-consecutive starvation warning and would otherwise say + nothing. Keys are prefixed `state_offload_` because the aggregation is + by key across every state class. """ - return { + fates = { "checkpoints_kept": self.checkpoints_kept, "checkpoints_dropped": self.checkpoints_dropped, "checkpoints_evicted": self.checkpoints_evicted, "checkpoints_orphaned": self.checkpoints_orphaned, } + if self.offload is not None: + fates.update( + (f"state_offload_{k}", v) for k, v in self.offload.stats().items() + ) + return fates def occupancy(self) -> dict[str, int]: """How much of this pool is live, held by checkpoints, or spare. @@ -697,6 +822,8 @@ def unindex(self, h: int) -> None: slot = self.hash_to_slot.get(h, -1) if slot < 0: return + if self.offload is not None: + self._spill(slot) self.invalidate(slot) self.checkpoints_orphaned += 1 diff --git a/atom/model_engine/state_runtime.py b/atom/model_engine/state_runtime.py index 01995e3b7b..e239cbe3ee 100644 --- a/atom/model_engine/state_runtime.py +++ b/atom/model_engine/state_runtime.py @@ -184,11 +184,20 @@ class StateMaintenanceOps: """State movement drained once before a model batch.""" relocations: tuple[tuple[int, int], ...] = () + # (src_group, dst_entry, staging_slot, hash) for checkpoints the state pool + # evicted to the offload tier. Carried here rather than beside this struct + # so the one consumer -- `CommonAttentionBuilder.build` -- is also the one + # place the spill-before-copy order is enforced. Empty unless + # the offload tier is on. + spills: tuple[tuple[int, int, int, int], ...] = () checkpoint_stores: tuple[CheckpointStoreOp, ...] = () checkpoint_restores: tuple[CheckpointRestoreOp, ...] = () @property def empty(self) -> bool: return not ( - self.relocations or self.checkpoint_stores or self.checkpoint_restores + self.relocations + or self.spills + or self.checkpoint_stores + or self.checkpoint_restores ) diff --git a/atom/model_ops/attentions/backends.py b/atom/model_ops/attentions/backends.py index a466af4d97..8b529eb6b3 100644 --- a/atom/model_ops/attentions/backends.py +++ b/atom/model_ops/attentions/backends.py @@ -225,6 +225,40 @@ def checkpoint_image_bytes(self) -> int | None: """ return None + def state_entry_views(self, group: int) -> list["torch.Tensor"]: + """Contiguous views covering the whole of `group`'s per-request state. + + The byte-level counterpart of `relocate_state_slots`, so the offload + tier can read or write the same bytes. Every view must be contiguous -- + the Triton packer refuses a strided one -- so a class whose group is + strided (GDN, slot on axis 1) returns one view per layer. + """ + raise NotImplementedError( + f"{type(self).__name__} owns per-request state but does not " + "implement state_entry_views" + ) + + def _submit_state_spills(self, pairs) -> None: + """Hand the staged bytes to the offload tier's own executor. + + The connector is reached function-locally so these modules stay + importable without the offload extra. + """ + from atom.utils.forward_context import get_kvconnector + + connector = get_kvconnector() + tier = getattr(connector, "_state_tier", None) + if tier is None: + return + import torch + + # One event for the whole batch: every copy above was issued on this + # stream, so an event recorded now is after all of them. + ready = torch.cuda.Event() + ready.record(torch.cuda.current_stream()) + for _src, dst, slot, h in pairs: + tier.submit_spill(h, entry_index=dst, staging_slot=slot, ready_event=ready) + def relocate_state_slots(self, pairs: Sequence[tuple[int, int]]) -> None: """Move live state between contiguous Active Slots.""" raise NotImplementedError( @@ -604,6 +638,19 @@ def _attach_tbo_prefill_cpu_lens( def build(self, batch: ScheduledBatch, bs: int): # Run state maintenance on the compute stream before the forward. state_ops = batch.state_maintenance_ops + # Spills FIRST, and the order is load-bearing. A spill is filed under + # the group's *old* hash, and the same `pop()` then hands that group out + # as a checkpoint destination -- so one group is routinely both a spill + # SOURCE and a checkpoint DESTINATION in one batch. Issue the other + # copies first and the tier stores the new occupant's state under the + # evicted checkpoint's hash: valid-looking, someone else's, undetectable + # on any later load. The reverse cannot happen -- a spill's destination + # is past the pool's range and is never a copy source. + if state_ops.spills: + self.relocate_state_slots( + [(src, dst) for src, dst, _slot, _h in state_ops.spills] + ) + self._submit_state_spills(state_ops.spills) if state_ops.relocations: self.relocate_state_slots(state_ops.relocations) if state_ops.checkpoint_stores or state_ops.checkpoint_restores: diff --git a/atom/model_ops/attentions/deepseek_v4_attn.py b/atom/model_ops/attentions/deepseek_v4_attn.py index 20ac25987f..cb68b01b22 100644 --- a/atom/model_ops/attentions/deepseek_v4_attn.py +++ b/atom/model_ops/attentions/deepseek_v4_attn.py @@ -840,8 +840,47 @@ def state_transfer(self) -> StateTransfer: ) return StateTransfer.copy(layout_id) + def state_entry_views(self, group: int) -> list[torch.Tensor]: + """One contiguous slice per plane — a V4 group is one slot per plane. + + A slot holds the compressor state and then every layer's windows + contiguously (see `relocate_state_slots`), so a plane's whole + contribution is one range and no per-layer split is needed. + + `group` may address a staging-ring group past the last one the + BlockManager leases, and that resolves: `num_state_slots` (:863-866) + and the carve that sizes the planes (:1023, feeding `with_capacity` at + :1038) both read + `entries[STATE_SLOT_CLASS]`, the *allocation* count, which already + includes the K extra staging groups — admission is capped by a separate + count. So `_slot_views()`'s span and `physical_slot`'s bound both cover + the staging groups; `tests/test_state_entry_views.py` pins this. + """ + return self._slot_views()[group] + def relocate_state_slots(self, pairs: Sequence[tuple[int, int]]) -> None: - """Relocate a request's whole Active Slot.""" + """Duplicate a request's whole per-request state: compressor + windows. + + One range per plane, and a plane is all there is: a slot holds the + compressor state and then every layer's windows, contiguously, so the + two halves of a request's state copy as one slice. Copying the state + whole rather than just the rows a resumer reads (the CSA ring's + trailing `K - ratio` = 4, HCA's none) is what makes that true — those + rows are scattered, and picking them out would cost 84 strided copies + against this one. + + **The window half is what makes the private ring safe.** A per-request + ring is exactly what #1417 removed, because a request resuming a cached + prefix had never written that prefix into its own. Reinstating it is only + correct because the checkpoint carries the window across — drop this and + the bug returns, silently, as garbage attention over the reused prefix. + + That is also why a window whose dtype differs from the pool's is a state + field rather than a plane of its own: a plane of its own would sit + outside every slot and so outside this copy, and a resumed request would + draft against the slot's previous occupant. Verification would keep the + output right and the acceptance rate would just quietly collapse. + """ views = self._slot_views() dsts, srcs = [], [] for src, dst in pairs: @@ -1314,13 +1353,24 @@ def sub_pool_specs(self) -> list[SubPoolSpec]: No flat margin on the slot: a ring cannot transiently exceed itself the way a block-addressed window could while sliding across a boundary, and there is no admission-vs-materialization gap to cushion — a slot exists - for its request's whole life. + for its request's whole life. The checkpoint headroom and the offload + tier's staging ring are both added by `state_pool` itself, which is the + only reader of the two env vars that size them. """ + from atom.model_engine.state_offload import kv_connector_hosts_state_tier + geo = self.pool_geometry row_bytes = self.plane_row_bytes() return [ page_pool(geo.block_bytes(row_bytes) + self._indexer_block_bytes()), - state_pool(STATE_SLOT_CLASS, geo.slot_bytes(row_bytes), entries_per_req=1), + state_pool( + STATE_SLOT_CLASS, + geo.slot_bytes(row_bytes), + entries_per_req=1, + offload_hosted=kv_connector_hosts_state_tier( + getattr(self.model_runner.config, "kv_transfer_config", None) + ), + ), ] def _plane_row_widths(self) -> list[int]: diff --git a/atom/model_ops/attentions/gdn_attn.py b/atom/model_ops/attentions/gdn_attn.py index 17a72d70cf..2bf1694125 100644 --- a/atom/model_ops/attentions/gdn_attn.py +++ b/atom/model_ops/attentions/gdn_attn.py @@ -823,6 +823,26 @@ def warmup_per_req_cache(self) -> None: ) launch_copy_descriptor(staging.copy_to_gpu(plan.num_spans), plan) + def state_entry_views(self, group: int) -> list[torch.Tensor]: + """One contiguous slice per (cache, layer) — the group's whole state. + + Both caches are layer-major with the slot on axis 1, so a group's rows + are strided and there is no single range covering them. Slicing per + layer makes each piece contiguous, which is what the staging packer + requires; `relocate_state_slots` keeps its own strided views because + `_foreach_copy_` has no such constraint and one launch beats `LAYERS`. + """ + span = 1 + self.num_spec + lo = group * span + views = [] + for cache in ( + self.model_runner.mamba_k_cache, + self.model_runner.mamba_v_cache, + ): + for layer in range(cache.shape[0]): + views.append(cache[layer, lo : lo + span]) + return views + def relocate_state_slots(self, pairs: Sequence[tuple[int, int]]) -> None: """Relocate a live GDN state slot between Active Slot positions. diff --git a/atom/model_ops/attentions/sub_pool_spec.py b/atom/model_ops/attentions/sub_pool_spec.py index 2ec950bdae..9dcd6e4207 100644 --- a/atom/model_ops/attentions/sub_pool_spec.py +++ b/atom/model_ops/attentions/sub_pool_spec.py @@ -42,7 +42,7 @@ class — and the backend that declares the spec imports it from there. The one from __future__ import annotations -from dataclasses import dataclass, replace +from dataclasses import dataclass, field, replace from enum import Enum @@ -67,11 +67,21 @@ class SubPoolSpec: # per-request term. What it covers is the declaring backend's business. entries_per_req: int = 0 extra_entries: int = 0 + # STATE only. `extra_entries` above is capacity -- rows the pool hands out; + # these are the opposite, allocated and never leased. This is the offload tier's staging ring, which + # exists so `pop()` can copy an evicted checkpoint out and hand the original + # away immediately; a request given one of these rows would get a buffer the + # spill path writes into behind its back. Two fields rather than one because + # the two must ADD -- they shared a parameter once, and since the env + # override assigns, setting the headroom silently deleted the ring. + staging_entries: int = 0 def __post_init__(self): if self.pool is Pool.STATE and self.entries_per_req < 1: raise ValueError(f"{self.name}: STATE entries need entries_per_req >= 1") - if self.pool is Pool.PAGE and (self.entries_per_req or self.extra_entries): + if self.pool is Pool.PAGE and ( + self.entries_per_req or self.extra_entries or self.staging_entries + ): raise ValueError(f"{self.name}: PAGE entries are sized from the remainder") @@ -92,9 +102,30 @@ def state_pool( *, entries_per_req: int, extra_entries: int = 0, + offload_hosted: bool = False, ) -> SubPoolSpec: - """A per-request state entry class.""" - return SubPoolSpec(Pool.STATE, name, entry_bytes, entries_per_req, extra_entries) + """A per-request state entry class. + + `extra_entries` is the declaring backend's own cushion, and admissible. + The offload staging ring is added here instead, because it is driven by + `OFFLOAD_STATE_STAGING_GROUPS` and a second reader would let two sites + disagree about the same number. It is counted in *groups*, so it costs + `entries_per_req` rows each, is never admissible, and exists only when + `offload_hosted`: the tier follows the connector, so a deployment without + one must not pay for a ring nothing can drain. + """ + from atom.model_engine.state_offload import state_offload_staging_groups + + return SubPoolSpec( + Pool.STATE, + name, + entry_bytes, + entries_per_req, + extra_entries, + staging_entries=( + state_offload_staging_groups() * entries_per_req if offload_hosted else 0 + ), + ) @dataclass(frozen=True) @@ -110,6 +141,12 @@ class PoolPlan: reserved_bytes: dict[str, int] entries_per_req: dict[str, int] paged_class: str | None = None + # What allocation buys vs. what admission may lease, differing only by + # `staging_entries` -- the offload ring, inside the arena so + # `state_entry_views(num_groups + slot)` addresses it with no second scheme, + # and leased to nobody. A table rather than a subtraction at each call site, + # so a consumer picks a meaning by the name it reads. + admission_entries: dict[str, int] = field(default_factory=dict) @classmethod def empty(cls) -> PoolPlan: @@ -121,7 +158,13 @@ def empty(cls) -> PoolPlan: AttributeError — the runner installs this at construction time and replaces it in `get_num_blocks`. """ - return cls(entries={}, entry_bytes={}, reserved_bytes={}, entries_per_req={}) + return cls( + entries={}, + entry_bytes={}, + reserved_bytes={}, + entries_per_req={}, + admission_entries={}, + ) def with_paged_entries(self, count: int) -> PoolPlan: """Copy of the plan with the PAGE class resized to `count`. @@ -136,6 +179,7 @@ def with_paged_entries(self, count: int) -> PoolPlan: return replace( self, entries={**self.entries, self.paged_class: count}, + admission_entries={**self.admission_entries, self.paged_class: count}, reserved_bytes={**self.reserved_bytes, self.paged_class: count * bytes_per}, ) @@ -181,10 +225,12 @@ def merge_specs(specs: list[SubPoolSpec]) -> dict[str, SubPoolSpec]: prev.pool, prev.entries_per_req, prev.extra_entries, + prev.staging_entries, ) != ( spec.pool, spec.entries_per_req, spec.extra_entries, + spec.staging_entries, ): raise ValueError( f"entry class {spec.name!r} declared twice with different " @@ -196,6 +242,7 @@ def merge_specs(specs: list[SubPoolSpec]) -> dict[str, SubPoolSpec]: prev.entry_bytes + spec.entry_bytes, spec.entries_per_req, spec.extra_entries, + spec.staging_entries, ) return merged @@ -213,14 +260,20 @@ def plan_pools( """ merged = merge_specs(specs) entries: dict[str, int] = {} + admissible_entries: dict[str, int] = {} reserved: dict[str, int] = {} remaining = available_bytes state = {n: s for n, s in merged.items() if s.pool is Pool.STATE} for name, spec in state.items(): - count = max_num_seqs * spec.entries_per_req + spec.extra_entries + # `extra_entries` is capacity the pool leases; `staging_entries` is the + # offload ring, allocated past the pool's group range and leased to + # nobody. See `SubPoolSpec` for why they are two fields. + admissible = max_num_seqs * spec.entries_per_req + spec.extra_entries + count = admissible + spec.staging_entries cost = count * spec.entry_bytes entries[name], reserved[name] = count, cost + admissible_entries[name] = admissible remaining -= cost if state and remaining <= 0: raise InsufficientPoolBudget( @@ -236,6 +289,7 @@ def plan_pools( spec = merged[name] count = max(0, remaining // spec.entry_bytes) entries[name], reserved[name] = count, count * spec.entry_bytes + admissible_entries[name] = count return PoolPlan( entries=entries, @@ -243,4 +297,5 @@ def plan_pools( reserved_bytes=reserved, entries_per_req={n: s.entries_per_req for n, s in merged.items()}, paged_class=paged[0] if paged else None, + admission_entries=admissible_entries, ) diff --git a/docs/environment_variables.md b/docs/environment_variables.md index 99f04e40e4..26fb6e4f42 100644 --- a/docs/environment_variables.md +++ b/docs/environment_variables.md @@ -182,6 +182,57 @@ flag below. Details in the state-checkpoint section of the | Variable | Type | Default | Description | |----------|------|---------|-------------| | **ATOM_STATE_CHECKPOINT_DEMAND** | bool | 1 (true) | Set to `0` to stop a prefix hit that was refused for want of a checkpoint from placing a rung of its own, leaving the prompt-end anchor as the only placement. Overrides `--state-checkpoint-demand`, so the policy can be A/B'd without editing a launch script. The rung is most of the checkpoint write traffic and little of the read-back, and every write evicts something — `StateSlotPool.mark_speculative` carries the measurement. | +## KV / state offload (`lmcache_offload` connector) + +The `lmcache_offload` connector spills computed KV to a CPU tier and fetches it +back on a later prefix hit. It has no env switch of its own: it runs when +`--kv-transfer-config` names it, e.g. +`'{"kv_connector":"lmcache_offload","kv_role":"offload"}'`. + +Which layout runs is decided from the model config alone +(`select_offload_layout`) -- `dense` for ordinary token-indexed KV, `hybrid` for +DeepSeek-V4's compressed PAGE/SLOT chunks, `kimi_k3` for dense MLA KV plus the +KDA per-request state tier. Background and the counters to read before tuning +any of this: [`atom/kv_transfer/offload/README.md`](../atom/kv_transfer/offload/README.md). + +### Transport and staging (all layouts) + +| Variable | Type | Default | Description | +|----------|------|---------|-------------| +| **OFFLOAD_COPY_WORKERS** | int | 1 | Threads draining the save queue. | +| **OFFLOAD_MAX_PENDING_SAVES** | int | `max(2, 2 x copy_workers)` | The back-pressure bound, read from both ends: `hybrid` caps the worker queue with it, `kimi_k3` caps how many requests may have a save outstanding. It matters because a request whose save is queued keeps its blocks pinned, so this is also how much of the pool a slow backend can hold. Ignored by `dense`, which does not pin. | +| **OFFLOAD_MIN_LOAD_TOKENS** | int | 8192 | Prefix below which a KV hit is not worth a load -- a KV load moves bytes proportional to the hit, so a short one spends a round trip moving very little. **No effect under `kimi_k3`**: every sequence there carries per-request state, and a joint boundary is chosen for both legs together, so refusing one leg on size would leave the other claiming a prefix whose KV never came. | +| **OFFLOAD_GPU_STAGING_CHUNKS** | int | 2 | GPU staging buffer, in LMCache chunks. | +| **OFFLOAD_GPU_STAGING_MAX_BYTES** | int | unset | Hard cap on the staging buffer; must cover at least one chunk. | +| **OFFLOAD_RELEASE_GPU_STAGING_AFTER_TRANSFER** | bool | 0 | Free the staging buffer after each transfer instead of holding it. Trades HBM for per-transfer allocation. | +| **OFFLOAD_PROFILE** | bool | 0 | Per-transfer timing on the periodic stats line. | +| **LMCACHE_LOCAL_CPU**, **LMCACHE_MAX_LOCAL_CPU_SIZE**, **LMCACHE_CHUNK_SIZE**, **LMCACHE_LOCAL_DISK** | - | LMCache | Read by LMCache itself, not by ATOM. `MAX_LOCAL_CPU_SIZE` is GiB for the paged-KV pool. | + +### DeepSeek-V4 (`hybrid`) + +| Variable | Type | Default | Description | +|----------|------|---------|-------------| +| **OFFLOAD_SLOT_STAGING_SLOTS** | int | 1 | Staging slots for complete SLOT sidecars. | +| **OFFLOAD_COMMITTED_SIDECAR_CAPACITY** | int | 65536 | Committed-sidecar index capacity. | +| **OFFLOAD_PUBLICATION_TIMEOUT_S** | float | 5.0 | How long a worker waits for a chunk's publication. | +| **OFFLOAD_PUBLICATION_POLL_INTERVAL_S** | float | 0.01 | Poll interval for the above. | + +### Per-request state and its tier + +Hybrid models (GDN/KDA: Qwen3-Next, Qwen3.5, Kimi-K3) and DeepSeek-V4 keep one +per-request **state** entry alongside their paged KV, in a fixed pool of groups. +A prefix hit is resumable only up to a boundary where that state was +checkpointed, so the pool size and whether an evicted checkpoint can be fetched +back decide the hit rate. + +Under `kimi_k3` the state tier is **not separately switchable**: a resumable +prefix needs both legs, so state is offloaded exactly when KV is, and turning +the tier off means turning the connector off. + +| Variable | Type | Default | Description | +|----------|------|---------|-------------| +| **OFFLOAD_STATE_STAGING_GROUPS** | int | 1 | Spill staging ring depth in **groups** -- `state_pool()` multiplies by the class's `entries_per_req` to get rows. Allocated inside the state arena and never leased to a request. Raise it if `state_offload_spills_dropped` is a material fraction of `spills_requested`; set 0 on a deployment that will never offload, to reclaim the rows. | +| **OFFLOAD_STATE_CPU_SIZE** | float (GiB) | 16 | A CPU pool for state entries, separate from the paged-KV pool. Sharing loses both ways: the KV write stream is several times the state volume so it evicts the checkpoints, and a stopped KV backend takes the tier down with it. 0 shares the paged-KV pool. | ## Profiling & debugging diff --git a/tests/test_block_manager.py b/tests/test_block_manager.py index 8d48b39557..e243b71143 100644 --- a/tests/test_block_manager.py +++ b/tests/test_block_manager.py @@ -1,6 +1,7 @@ # SPDX-License-Identifier: MIT # Tests for atom/model_engine/block_manager.py — public API only +import logging from conftest import MockConfig @@ -51,6 +52,72 @@ def test_can_allocate_multi_block(self, block_manager, seq_factory): assert block_manager.can_allocate(seq) >= 0 +# ── the widened claim behind a joint boundary ────────────────────────────── + + +class TestJointClaimReusesResidentBlocks: + """`can_allocate` returns one number, and the request needs two. + + What it may call *cached* is gated on a resumable state behind it. What it + may point its block table at is every block the prefix walk matched -- the + state gate cut resumability, not residency. Without the second number the + KV leg pays LMCache to resend blocks the pool is already holding, and the + reply lands in *fresh* blocks, so HBM ends up with two copies of the same + prefix. + """ + + def _resident_prefix(self, seq_factory): + """A 4-block prefix published in the index, then released.""" + cfg = MockConfig( + num_kvcache_blocks=32, kv_cache_block_size=4, enable_prefix_caching=True + ) + bm = BlockManager(cfg) + tokens = list(range(20)) + first = seq_factory(tokens) + bm.allocate(first, 0) + bm.hash_blocks(first, len(tokens)) + resident = list(first.block_table[:4]) + bm.deallocate(first) + return bm, tokens, resident + + def test_without_a_joint_boundary_the_claim_stops_at_the_gated_hit( + self, seq_factory + ): + bm, tokens, resident = self._resident_prefix(seq_factory) + seq = seq_factory(tokens) + bm.allocate(seq, 2) + assert list(seq.block_table[:2]) == resident[:2] + # Blocks 2 and 3 are resident and match, but nothing above the hit will + # ever be treated as computed, so claiming them would pin blocks the + # forward is about to overwrite. + assert seq.block_table[2] not in resident[2:] + assert seq.num_cached_tokens == 8 + + def test_a_joint_boundary_widens_the_claim_without_widening_the_hit( + self, seq_factory + ): + bm, tokens, resident = self._resident_prefix(seq_factory) + seq = seq_factory(tokens) + # The state gate cut the hit to 2 blocks; the walk had matched 4. + seq.state_joint_claim_tokens = 16 + bm.allocate(seq, 2) + + # All four resident blocks are reused -- that is the transfer the KV + # leg no longer has to make, and the second copy that no longer lands. + assert list(seq.block_table[:4]) == resident + # ...and the request still only calls two of them cached, so a failed + # leg leaves the forward recomputing rather than skipping. + assert seq.num_cached_tokens == 8 + + def test_the_widened_claim_still_gives_every_position_a_block(self, seq_factory): + bm, tokens, _ = self._resident_prefix(seq_factory) + seq = seq_factory(tokens) + seq.state_joint_claim_tokens = 16 + bm.allocate(seq, 2) + assert len(seq.block_table) == 5 + assert len(set(seq.block_table)) == 5 + + # ── allocate / deallocate ────────────────────────────────────────────────── @@ -653,3 +720,88 @@ def tracked_compute_hash(token_ids, prefix=-1): assert all( bm.kv.block(block_id).hash != -1 for block_id in received.block_table[2:] ) + + +# ── state_offload disabled by default ───────────────────────────────────── + + +def test_state_offload_is_none_when_tier_is_off(block_manager): + """No `lmcache_offload` connector means no tier, whatever the ring size: + the connector is the feature's only on/off switch.""" + assert block_manager.state_offload is None + + +# ── the ring is installed only where something can drain it ─────────────── + +_OFFLOAD_KVC = {"kv_connector": "lmcache_offload", "kv_role": "offload"} + + +def _bm_with_state_tier(monkeypatch, kv_transfer_config): + monkeypatch.setenv("OFFLOAD_STATE_STAGING_GROUPS", "2") + cfg = MockConfig( + enable_prefix_caching=True, + kv_transfer_config=kv_transfer_config, + pool_entries={"state": 4}, + pool_entries_per_req={"state": 1}, + ) + return BlockManager(cfg) + + +def test_no_ring_is_installed_without_a_connector_that_hosts_the_tier( + monkeypatch, caplog +): + """The silent-permanent failure. Nothing submits a spill without a hosting + connector -- `_poll_kv_transfer_progress` returns early and the forward + finds no `_state_tier` -- so an installed ring would hand out every slot + and never get one back. Refuse to install it.""" + with caplog.at_level(logging.WARNING, logger="atom"): + bm = _bm_with_state_tier(monkeypatch, None) + + assert bm.state_offload is None + assert all(cache.offload is None for cache in bm.state_caches) + # An installed-but-inert ring is the thing that misleads, so drive the + # eviction path that would reserve a slot and prove none is: with the ring + # installed this same call yields (src, dst, staging, hash) and that + # staging slot never comes back. + bm.state.slot_hash[1] = 111 + bm.state._spill(1) + assert bm.state.take_spill_copies() == [] + assert bm.state_spills_for_batch() == [] + + +def test_a_connector_that_cannot_host_the_tier_gets_no_ring_either(monkeypatch, caplog): + """Same leak, one step subtler: moriio is a KV connector, so a plain + truthiness test on kv_transfer_config would install the ring, but only + lmcache_offload's worker half ever builds a `_state_tier`.""" + with caplog.at_level(logging.WARNING, logger="atom"): + bm = _bm_with_state_tier(monkeypatch, {"kv_connector": "moriio"}) + + assert bm.state_offload is None + + +def test_the_ring_is_installed_for_the_offload_connector(monkeypatch): + bm = _bm_with_state_tier(monkeypatch, _OFFLOAD_KVC) + assert bm.state_offload is not None + assert bm.state_offload.staging_depth == 2 + assert all(cache.offload is bm.state_offload for cache in bm.state_caches) + + +def test_the_ring_is_installed_for_a_multi_that_lists_the_offload_backend( + monkeypatch, +): + bm = _bm_with_state_tier( + monkeypatch, + { + "kv_connector": "multi", + "connectors": [{"kv_connector": "moriio"}, _OFFLOAD_KVC], + }, + ) + assert bm.state_offload is not None + + +def test_a_multi_without_the_offload_backend_gets_no_ring(monkeypatch): + bm = _bm_with_state_tier( + monkeypatch, + {"kv_connector": "multi", "connectors": [{"kv_connector": "moriio"}]}, + ) + assert bm.state_offload is None diff --git a/tests/test_lmcache_offload_connector.py b/tests/test_lmcache_offload_connector.py index 996313934a..a8fb33bca6 100644 --- a/tests/test_lmcache_offload_connector.py +++ b/tests/test_lmcache_offload_connector.py @@ -33,7 +33,10 @@ from atom.kv_transfer.offload import config as offcfg from atom.kv_transfer.offload._block_gpu_connector import BlockGPUConnector from atom.kv_transfer.offload._offload_common import OffloadSchedulerMixin -from atom.kv_transfer.offload.dense.connector import DenseOffloadConnector +from atom.kv_transfer.offload.dense.connector import ( + DenseOffloadConnector, + DenseOffloadScheduler, +) from atom.kv_transfer.offload.dense.kv_byte_codec import ( DenseKVByteCodec, ) @@ -51,6 +54,12 @@ from atom.kv_transfer.offload.hybrid.dsv4.policy import ( _chained_prefix_hashes, ) +from atom.kv_transfer.offload.hybrid.kimi_k3.connector import ( + SAVE_STALL_SECONDS, + STATE_INDEX_CHANNEL, + STATE_STAGING_CHANNEL, + KimiK3OffloadScheduler, +) from atom.kv_transfer.offload.metadata import ( ATOMRawBytesLMCacheMetadata, LMCacheOffloadMetadata, @@ -4626,6 +4635,122 @@ def test_multi_metadata_exposes_sub_meta_unpins(): ) +# ── kimi_k3: joint boundary, save stall, state channels ─────────────────── + + +def _k3_scheduler() -> KimiK3OffloadScheduler: + """Only the fields the K3 overrides touch; everything else is dense's.""" + s = KimiK3OffloadScheduler.__new__(KimiK3OffloadScheduler) + s.chunk_size = 256 + s._save_inflight = {} + s._save_tracker = {} + s._pending_state_loads = [] + s._save_inflight_since = {} + s._save_stalled = False + s._warned_save_stalled = False + s._state_indexed = set() + s._state_index_failed = set() + s._state_staging_released = set() + s._active_load_operations = {} + s._do_save = True + s._max_pending_saves = 4 + return s + + +def _k3_seq(*, hbm: int, joint: int = 0, kv: int = 0, claimed: int = 0): + return SimpleNamespace( + id=1, + has_per_req_cache=True, + num_cached_tokens=hbm, + state_joint_boundary_tokens=joint, + state_joint_kv_tokens=kv, + state_joint_claim_tokens=claimed, + ) + + +def test_layout_selection_picks_kimi_k3_for_a_kda_config(): + """K3's paged KV is ordinary dense MLA, so the layout cannot be read off + the KV shape -- the KDA state is what makes it its own family.""" + cfg = SimpleNamespace( + hf_config=SimpleNamespace(model_type="kimi_linear"), kv_transfer_config={} + ) + assert offcfg.select_offload_layout(cfg) == "kimi_k3" + + +@pytest.mark.parametrize( + "seq,reason", + [ + (_k3_seq(hbm=512, joint=0), "per_req_cache_state_boundary"), + (_k3_seq(hbm=512, joint=256), "per_req_cache_state_boundary"), + (_k3_seq(hbm=300, joint=1024), "joint_unaligned_hbm_prefill"), + (_k3_seq(hbm=512, joint=9000), "joint_boundary_above_lookup"), + ], +) +def test_a_hybrid_load_is_refused_unless_both_legs_reach_one_boundary(seq, reason): + """A hybrid's state is the compressed history of exactly `[0, hbm)`. Every + refusal here is a case where raising the KV-loaded length would have the + forward skip tokens the recurrence never saw -- wrong output, no + exception.""" + ls = SimpleNamespace(lmcache_cached_tokens=8192) + ok, got, *_ = _k3_scheduler()._decide_load_after_alloc(seq, ls) + assert ok is False + assert got == reason + + +def test_a_joint_load_clamps_the_kv_leg_to_the_covering_chunk(): + """The two grids do not line up: the boundary is a hash-block position, the + KV leg moves whole chunks. The leg is aimed at the chunk that covers it.""" + seq = _k3_seq(hbm=512, joint=1000, kv=1024) + ls = SimpleNamespace(lmcache_cached_tokens=8192) + ok, reason, hbm, lmc, need, _ = _k3_scheduler()._decide_load_after_alloc(seq, ls) + assert (ok, reason) == (True, "joint_state_and_kv") + assert (hbm, lmc, need) == (512, 1024, 512) + assert ls.lmcache_cached_tokens == 1024 + + +def test_the_kv_leg_starts_where_the_claim_ended_not_where_the_hit_did(): + """`allocate` claims every block the prefix walk matched, which is past the + gated hit. Starting the transfer at the hit would fetch back what the pool + already holds, and land a second copy of it in HBM.""" + seq = _k3_seq(hbm=0, joint=5120, kv=5120, claimed=4096) + ls = SimpleNamespace(hbm_cached_tokens=0, lmcache_cached_tokens=7168) + ok, reason, start, lmc, need, _ = _k3_scheduler()._decide_load_after_alloc(seq, ls) + assert (ok, reason) == (True, "joint_state_and_kv") + assert (start, lmc, need) == (4096, 5120, 1024) + # Both ends land on the spec: the worker reads the pair, and a start left + # at the value the lookup recorded fetches from token 0 every time. + assert ls.hbm_cached_tokens == 4096 + assert ls.lmcache_cached_tokens == 5120 + + +def test_without_a_widened_claim_the_leg_still_starts_at_the_hit(): + seq = _k3_seq(hbm=512, joint=1024, kv=1024) + ls = SimpleNamespace(hbm_cached_tokens=0, lmcache_cached_tokens=8192) + ok, _, start, _, need, _ = _k3_scheduler()._decide_load_after_alloc(seq, ls) + assert (ok, start, need) == (True, 512, 512) + assert ls.hbm_cached_tokens == 512 + + +def test_a_boundary_already_fully_resident_emits_no_kv_leg(): + """Unreachable while `_gated_hit` returns the rightmost rung, but a state + leg paired with an empty KV leg must not be emitted silently.""" + seq = _k3_seq(hbm=0, joint=4096, kv=4096, claimed=4096) + ls = SimpleNamespace(hbm_cached_tokens=0, lmcache_cached_tokens=7168) + ok, reason, *_ = _k3_scheduler()._decide_load_after_alloc(seq, ls) + assert (ok, reason) == (False, "joint_kv_already_resident") + + +def test_the_claim_stops_at_the_boundary_when_the_chunk_overshoots_it(): + """chunk != hash_block_size aims the transfer past the state boundary. The + claim must not follow it there: the forward would start inside a range the + recurrence never saw, and nothing raises.""" + sched = _k3_scheduler() + sched.chunk_size = 256 + # boundary 5120 (a hash-block position), covering chunk ends at 5376 + seq = _k3_seq(hbm=0, joint=5120, kv=5376, claimed=4096) + assert sched._claim_after_load(seq, 4096, 5376) == 5120 + + def test_a_layout_whose_claim_and_transfer_share_a_boundary_claims_the_end(): """The base seam: claim as far as the transfer reached, never below HBM. @@ -4641,3 +4766,90 @@ def test_a_layout_whose_claim_and_transfer_share_a_boundary_claims_the_end(): assert sched._claim_after_load(seq, 4096, 8192) == 8192 # Never below the HBM floor, whatever the transfer reported. assert sched._claim_after_load(seq, 4096, 0) == 4096 + + +def test_the_claim_never_exceeds_the_state_boundary(): + """Overshooting the transfer is one wasted chunk; overshooting the *claim* + would have the forward skip tokens the state does not cover.""" + sched = _k3_scheduler() + seq = _k3_seq(hbm=512, joint=1000, kv=1024) + assert sched._claim_after_load(seq, 512, 1024) == 1000 + # No joint boundary: dense's own answer. + assert sched._claim_after_load(_k3_seq(hbm=512), 512, 1024) == 1024 + + +def test_a_stalled_save_releases_blocks_it_never_handed_out(monkeypatch): + """The deadlock this exists for: with the backend stopped, holding blocks + for a save nobody is reading turns a stopped backend into a stopped + engine. A save already handed out still holds -- something is reading it.""" + s = _k3_scheduler() + seq = SimpleNamespace(id=7) + s._save_inflight["9"] = object() # a different request, stuck + s._save_tracker["7"] = [seq, 0] + monkeypatch.setattr( + KimiK3OffloadScheduler, "_has_pending_save", lambda self, q: True + ) + s._refresh_save_stall() + assert s._save_stalled is False + + # Age the outstanding save rather than the process clock: `time.monotonic` + # is the stdlib's, and patching it reaches every other test in the run. + s._save_inflight_since["9"] -= 2 * SAVE_STALL_SECONDS + s._refresh_save_stall() + assert s._save_stalled is True + # Never handed out -> dropped, blocks released. + assert s.should_defer_free(seq) is False + assert "7" not in s._save_tracker + # Handed out -> still held, whatever the stall says. + assert s.should_defer_free(SimpleNamespace(id=9)) is True + + +def test_state_loads_are_drained_into_the_metadata_exactly_once(monkeypatch): + """A second submission would write the same entry into a group the first + transfer is already filling.""" + s = _k3_scheduler() + monkeypatch.setattr( + DenseOffloadScheduler, + "build_connector_meta", + lambda self: LMCacheOffloadMetadata(), + ) + assert s.enqueue_state_loads([]) is False + assert s.enqueue_state_loads([("1", 111, 0)]) is True + + assert s.build_connector_meta().state_loads == [("1", 111, 0)] + assert s.build_connector_meta().state_loads == [] + + +def test_the_two_state_channels_are_routed_and_drained(): + """The tier reports over the generic completion channel rather than extra + `KVConnectorOutput` fields, so TP quorum is the aggregator's job.""" + s = _k3_scheduler() + for channel, op, ok in ( + (STATE_INDEX_CHANNEL, 111, True), + (STATE_INDEX_CHANNEL, 222, False), + (STATE_STAGING_CHANNEL, 3, True), + ): + assert s.connector_completion( + SimpleNamespace(channel=channel, operation_id=op, succeeded=ok) + ) + + indexed, released, failed = s.take_state_reports() + assert (indexed, released, failed) == ({111}, {3}, {222}) + assert s.take_state_reports() == (set(), set(), set()) + + +def test_no_more_saves_go_out_than_the_pool_can_afford_to_pin(): + """A queued save pins its request's blocks until it drains, so the queue + depth is also how much of the pool a slow backend can hold. Dense has no + such bound because it does not pin; K3 does.""" + s = _k3_scheduler() + s._max_pending_saves = 2 + assert s._may_emit_save() is True + + s._save_inflight = {"1": object(), "2": object()} + assert s._may_emit_save() is False + + s._save_inflight = {"1": object()} + assert s._may_emit_save() is True + s._save_stalled = True + assert s._may_emit_save() is False diff --git a/tests/test_lmcache_offload_v4_page_slot.py b/tests/test_lmcache_offload_v4_page_slot.py index 67923ef93f..7178ed8d6b 100644 --- a/tests/test_lmcache_offload_v4_page_slot.py +++ b/tests/test_lmcache_offload_v4_page_slot.py @@ -1614,7 +1614,7 @@ def _unexpected_executor(*args, **kwargs): @pytest.mark.parametrize("value", [True, False, 0, -1, 1.5, "1.5", "bad"]) def test_max_pending_saves_rejects_invalid_connector_extra(value): with pytest.raises(ValueError, match="max pending saves"): - connector_module._max_pending_saves( + connector_module.max_pending_saves( { "kv_connector_extra_config": { "max_pending_saves": value, @@ -1626,17 +1626,17 @@ def test_max_pending_saves_rejects_invalid_connector_extra(value): def test_max_pending_saves_precedence_and_worker_default(monkeypatch): monkeypatch.setenv("OFFLOAD_MAX_PENDING_SAVES", "7") - assert connector_module._max_pending_saves({}, 3) == 7 + assert connector_module.max_pending_saves({}, 3) == 7 assert ( - connector_module._max_pending_saves( + connector_module.max_pending_saves( {"kv_connector_extra_config": {"max_pending_saves": 5}}, 3, ) == 5 ) monkeypatch.delenv("OFFLOAD_MAX_PENDING_SAVES") - assert connector_module._max_pending_saves({}, 3) == 6 - assert connector_module._max_pending_saves({}, 1) == 2 + assert connector_module.max_pending_saves({}, 3) == 6 + assert connector_module.max_pending_saves({}, 1) == 2 def test_wait_for_publication_times_out_with_bounded_positive_sleeps(): diff --git a/tests/test_multi_connector.py b/tests/test_multi_connector.py index 0e54cf1a10..abb35c3932 100644 --- a/tests/test_multi_connector.py +++ b/tests/test_multi_connector.py @@ -61,6 +61,7 @@ def __init__( self.saved = [] self.load_failed_ids = [] self.pending = False + self.state_loads = [] def get_num_new_matched_tokens(self, seq): return self._match @@ -106,6 +107,10 @@ def process_completions(self, output): def has_pending_work(self): return self.pending + def enqueue_state_loads(self, loads): + self.state_loads.extend(loads) + return True + def __getattribute__(self, name): # Hide offload-specific methods unless this mock opts in, so # MultiConnector's hasattr() guards are exercised realistically. @@ -118,6 +123,7 @@ def __getattribute__(self, name): "load_failed", "process_completions", "has_pending_work", + "enqueue_state_loads", } if name in offload_api and not object.__getattribute__(self, "_offload"): raise AttributeError(name) @@ -163,6 +169,7 @@ def _worker(connectors, pp_is_head=True): obj._pending_save_ops = {} obj._sent = {} obj._saved = {} + obj._state_tier = None return obj @@ -373,6 +380,103 @@ def test_producer_offload_load_completion_uses_loading_state(): assert out.failed_loading == {"f1"} +def test_state_loads_go_to_the_sub_that_can_carry_them(): + """The scheduler calls `enqueue_state_loads` on whatever connector it + holds. Under `multi` that is this object, and a load it swallowed would + leave its request parked against a transfer nobody was asked to make. + """ + plain = FakeSchedSub() + off = FakeSchedSub(is_offload=True, offload_methods=True) + loads = [(1, 111, 0), (2, 222, 3)] + + assert _sched([plain, off]).enqueue_state_loads(loads) is True + + assert off.state_loads == loads + assert not hasattr(plain, "enqueue_state_loads") + + +def test_no_sub_to_carry_a_state_load_is_reported_not_swallowed(): + """Every one of these belongs to a parked request that only a report can + wake, and the scheduler's `hasattr` guard cannot catch this case because + this method always exists on the composite. So it has to say no.""" + plain = FakeSchedSub() + accepted = _sched([plain, FakeSchedSub()]).enqueue_state_loads([(1, 111, 0)]) + assert accepted is False + + +def test_the_composite_exposes_the_sub_connectors_state_tier(): + """`AttentionBackend._submit_state_spills` reads `_state_tier` off whatever + connector the forward context holds. Under `multi` that is this object, so + without the re-export nothing is ever submitted and every slot leaks.""" + tier = object() + off = FakeWorkerSub() + off._state_tier = tier + w = _worker([FakeWorkerSub(), off]) + + w.register_kv_caches({}, transfer_tensors=None, num_blocks=1) + + assert w._state_tier is tier + + +def test_no_sub_with_a_tier_leaves_the_composite_tier_none(): + w = _worker([FakeWorkerSub(), FakeWorkerSub()]) + w.register_kv_caches({}, transfer_tensors=None, num_blocks=1) + assert w._state_tier is None + + +def test_a_state_only_step_under_multi_is_not_dropped(): + """A step whose only work is a state load must reach the worker. + + `state_loads` carries no `LMCacheReqMeta`, so a wrapper that answers from + its own (always empty) fields reports no work, the engine drops the + snapshot, and the request parked on that load is woken by nothing. + """ + from atom.kv_transfer.disaggregation.types import connector_metadata_has_work + from atom.kv_transfer.offload.metadata import LMCacheOffloadMetadata + + sub = LMCacheOffloadMetadata() + sub.state_loads = [("req-1", 12345, 7)] + + assert connector_metadata_has_work(sub) + assert connector_metadata_has_work(MultiConnectorMetadata(metas=[sub])) + + +def test_a_multi_wrapper_over_idle_subs_still_reports_no_work(): + from types import SimpleNamespace + + from atom.kv_transfer.disaggregation.types import connector_metadata_has_work + from atom.kv_transfer.offload.metadata import LMCacheOffloadMetadata + + assert not connector_metadata_has_work( + MultiConnectorMetadata(metas=[LMCacheOffloadMetadata(), SimpleNamespace()]) + ) + + +def test_every_metadata_field_a_subclass_adds_is_declared_work_or_not(): + """`WORK_FIELDS` must name real attributes, or a typo silences a field. + + A misspelled entry is invisible: `getattr` returns None, the field never + counts, and the only symptom is a parked request much later. + """ + from atom.kv_transfer.offload.metadata import LMCacheOffloadMetadata + + meta = LMCacheOffloadMetadata() + for name in LMCacheOffloadMetadata.WORK_FIELDS: + assert hasattr(meta, name), f"WORK_FIELDS names a missing attribute: {name}" + + +def test_two_sub_connectors_with_a_tier_is_refused(): + """First-one-wins would be wrong, not merely arbitrary: the spill goes to + one tier and the load may ask the other, so a hash could be reported + indexed by a tier that never stored it.""" + a, b = FakeWorkerSub(), FakeWorkerSub() + a._state_tier, b._state_tier = object(), object() + w = _worker([a, b]) + + with pytest.raises(ValueError, match="exactly one may"): + w.register_kv_caches({}, transfer_tensors=None, num_blocks=1) + + def test_recv_blocks_concat(): w = _worker([FakeWorkerSub(recv_blocks=[1, 2]), FakeWorkerSub(recv_blocks=[3])]) assert w.get_finished_recv_blocks() == [1, 2, 3] @@ -527,44 +631,3 @@ def test_real_constructor_populates_the_pairing_state(monkeypatch, pp_rank, hold w.start_load_kv(MultiConnectorMetadata([ConnectorMetadata(), _save_meta(9)])) out = w.get_finished() assert out.finished_sending == (set() if holds_send else {9}) - - -def test_a_state_only_step_under_multi_is_not_dropped(): - """A step whose only work is a state load must reach the worker. - - `state_loads` carries no `LMCacheReqMeta`, so a wrapper that answers from - its own (always empty) fields reports no work, the engine drops the - snapshot, and the request parked on that load is woken by nothing. - """ - from atom.kv_transfer.disaggregation.types import connector_metadata_has_work - from atom.kv_transfer.offload.metadata import LMCacheOffloadMetadata - - sub = LMCacheOffloadMetadata() - sub.state_loads = [("req-1", 12345, 7)] - - assert connector_metadata_has_work(sub) - assert connector_metadata_has_work(MultiConnectorMetadata([sub])) - - -def test_a_multi_wrapper_over_idle_subs_still_reports_no_work(): - from types import SimpleNamespace - - from atom.kv_transfer.disaggregation.types import connector_metadata_has_work - from atom.kv_transfer.offload.metadata import LMCacheOffloadMetadata - - assert not connector_metadata_has_work( - MultiConnectorMetadata([LMCacheOffloadMetadata(), SimpleNamespace()]) - ) - - -def test_every_metadata_field_a_subclass_adds_is_declared_work_or_not(): - """`WORK_FIELDS` must name real attributes, or a typo silences a field. - - A misspelled entry is invisible: `getattr` returns None, the field never - counts, and the only symptom is a parked request much later. - """ - from atom.kv_transfer.offload.metadata import LMCacheOffloadMetadata - - meta = LMCacheOffloadMetadata() - for name in LMCacheOffloadMetadata.WORK_FIELDS: - assert hasattr(meta, name), f"WORK_FIELDS names a missing attribute: {name}" diff --git a/tests/test_scheduler.py b/tests/test_scheduler.py index 3a08d52f08..45e4826800 100644 --- a/tests/test_scheduler.py +++ b/tests/test_scheduler.py @@ -497,6 +497,49 @@ def test_offload_parked_count_released_only_after_resume(self, seq_factory): assert failed in sched.running assert sched._num_parked_remote_kv == 0 + def test_a_resumed_offload_prefill_reports_the_hit_the_load_gave_it( + self, seq_factory + ): + """`cached_tokens` must count the tokens LMCache brought back. + + The load is the entire point of parking: `_mark_offload_load_ready` + raises `num_cached_tokens` from the pre-park HBM-only hit to the + post-load one. But the resume branch `continue`s before either + `prefix_cache_hit_tokens` assignment, so the field keeps the pre-park + value while `CacheStats` is fed the fresh `num_cached_tokens` in + `_schedule_prefill_seq`. The two then disagree about the same request, + and the one the user sees is the one that undercounts -- making the + offload tier look like it did nothing. + """ + sched = Scheduler( + MockConfig( + max_num_seqs=2, + max_num_batched_tokens=64, + num_kvcache_blocks=100, + ) + ) + sched.kv_connector = SimpleNamespace( + is_offload=True, + build_connector_meta=lambda: None, + ) + + seq = seq_factory(list(range(8))) + seq.num_cached_tokens = 2 # the HBM-only hit, before the load + seq.prefix_cache_hit_tokens = 2 + seq.block_table = [0] + seq.offload_loaded_tokens = 6 # LMCache returned four more + sched._park_for_remote_load(seq, deque()) + sched._count_inflight_load(seq) + sched.finished_recving_kv_req_ids.append(seq.id) + assert sched._resolve_waiting_remote_kv(seq, deque()) is False + sched.waiting.append(seq) + + sched.schedule() + + assert seq in sched.running, "precondition: the resume must be admitted" + assert seq.num_cached_tokens == 6, "precondition: the load was applied" + assert seq.prefix_cache_hit_tokens == 6 + def test_partial_prefill_ready_for_offload_load_moves_to_waiting(self): class _Connector: def should_park_partial_prefill_for_load(self, seq): diff --git a/tests/test_state_checkpoint.py b/tests/test_state_checkpoint.py index 2e2bb2f09c..c846ccb7f4 100644 --- a/tests/test_state_checkpoint.py +++ b/tests/test_state_checkpoint.py @@ -3156,3 +3156,164 @@ def test_a_stateless_model_reserves_nothing(self): bm.can_allocate(cold) assert cold.block_hashes == [] assert bm.midstep_positions(cold, 0, 44) == [] + + +def test_fates_report_every_counter(): + """checkpoint_fates() must expose all four fate counters. + + NOTE: the actual dict keys carry the ``checkpoints_`` prefix + (``checkpoints_kept``, ``checkpoints_evicted``, ``checkpoints_orphaned``, + ``checkpoints_dropped``). The task-0 brief assumed short keys + (``kept`` / ``evicted`` / …); those differ — see the report for the + discrepancy note and the rationale for leaving the public API unchanged. + """ + pool = StateSlotPool(num_slots=2, transfer=StateTransfer.fork(1), hash_block_size=4) + fates = pool.checkpoint_fates() + assert set(fates) == { + "checkpoints_kept", + "checkpoints_evicted", + "checkpoints_orphaned", + "checkpoints_dropped", + } + + +def test_state_checkpoint_fates_warns_on_missing_method(caplog): + """A state cache that lacks checkpoint_fates() emits a warning. + + The aggregator must not silently under-count: when a class registered in + ``state_caches`` does not implement ``checkpoint_fates()``, a WARNING is + logged naming the skipped class so an operator knows the total is partial. + """ + import logging + + bm = BlockManager(ckpt_config()) + # StubStateCache (defined above) satisfies StateCache but has no + # checkpoint_fates — exactly the class of future lightweight implementations + # the warning is meant to catch. + bm.state_caches = (bm.state_caches[0], StubStateCache()) + + with caplog.at_level(logging.WARNING, logger="atom"): + bm.state_checkpoint_fates() + + assert any( + "StubStateCache" in r.message and "checkpoint_fates" in r.message + for r in caplog.records + ), "expected a WARNING naming StubStateCache; got: " + str( + [r.message for r in caplog.records] + ) + + +def test_state_checkpoint_fates_warns_once_per_class(caplog): + """The caller is the scheduler's every-100-ticks stats line and + `state_caches` never changes during a run, so an unlatched warning is the + same line forever -- drowning the log it is trying to draw attention to. + What it reports is a static property of the build: true on tick 1, no + truer on tick 10000. + """ + import logging + + bm = BlockManager(ckpt_config()) + bm.state_caches = (bm.state_caches[0], StubStateCache()) + + with caplog.at_level(logging.WARNING, logger="atom"): + for _ in range(5): + bm.state_checkpoint_fates() + + hits = [r for r in caplog.records if "StubStateCache" in r.message] + assert len(hits) == 1, [r.message for r in hits] + + +def test_state_checkpoint_fates_log_order_is_stable(caplog): + """The periodic log line emitted by `Scheduler.schedule()` sorts its keys. + + Read by an operator diffing one stats line against the next, so the field + order has to be a property of the line rather than of whichever order the + counters happen to be declared in — `StateSlotPool.checkpoint_fates()` + returns them kept/dropped/evicted/orphaned, which is not alphabetical, and + a new counter appended to that dict would otherwise land in the middle of + the line for every deployment at once. + + Asserted against the emitted message, not against a `sorted()` this test + applies itself: dropping the `sorted()` from the scheduler's join has to be + what fails here, and the non-alphabetical insertion order above is what + makes the two distinguishable. + """ + import logging + + sched = Scheduler(ckpt_config()) + sched.add(stateful_seq(list(range(40)))) + pool = sched.block_manager.state + pool.checkpoints_kept += 3 + pool.checkpoints_dropped += 1 + # The premise: if the counters were declared alphabetically the emitted + # line would be sorted whether or not the scheduler sorted it. + assert list(pool.checkpoint_fates()) != sorted(pool.checkpoint_fates()) + + # The line is periodic — one pass in a hundred — so the batch has to be + # driven to the tick that emits it. + with caplog.at_level(logging.INFO, logger="atom"): + for _ in range(100): + sched.schedule() + + lines = [ + r.getMessage() + for r in caplog.records + if r.getMessage().startswith("state checkpoints: ") + ] + assert lines, "the periodic state-checkpoint line was never emitted" + + fields = lines[-1].removeprefix("state checkpoints: ").split() + keys = [f.split("=")[0] for f in fields] + assert keys == [ + "checkpoints_dropped", + "checkpoints_evicted", + "checkpoints_kept", + "checkpoints_orphaned", + ], f"fields are not in alphabetical order: {lines[-1]}" + # The values ride along with their keys rather than being sorted apart. + assert "checkpoints_kept=3" in fields and "checkpoints_dropped=1" in fields + + +def test_checkpoint_funnel_includes_second_state_class(): + """checkpoint_funnel() must aggregate fates across ALL state classes. + + Prior to the fix, ``checkpoint_funnel()`` called ``self.state.checkpoint_fates()`` + directly, which would miss any second state class added to ``state_caches``. + After routing through ``state_checkpoint_fates()``, a second class that + implements ``checkpoint_fates()`` is included in the funnel output. + """ + + class SecondPoolStub: + """Minimal StateCache with checkpoint_fates — mimics a second real class.""" + + successor_room = 0 + + def applies(self, seq): + return False + + def resumable_hit(self, seq, P, block_hashes, assume_checkpointed=False): + return 0 + + def checkpoint(self, seq, boundary_blocks, h): + pass + + def checkpoint_fates(self) -> dict: + return {"checkpoints_kept": 7, "checkpoints_dropped": 2} + + bm = BlockManager(ckpt_config()) + bm.state_caches = (bm.state_caches[0], SecondPoolStub()) + + funnel = bm.checkpoint_funnel() + + # The two ladder-level counters must still be present. + assert "demands_recorded" in funnel + assert "chunks_cut_for_demand" in funnel + + # The second class's counters must appear in the funnel — they would have + # been absent before the fix. + assert ( + funnel.get("checkpoints_kept", 0) >= 7 + ), "checkpoints_kept from SecondPoolStub missing from checkpoint_funnel()" + assert ( + funnel.get("checkpoints_dropped", 0) >= 2 + ), "checkpoints_dropped from SecondPoolStub missing from checkpoint_funnel()" diff --git a/tests/test_sub_pool_spec.py b/tests/test_sub_pool_spec.py index ef7c8c44f7..458f124652 100644 --- a/tests/test_sub_pool_spec.py +++ b/tests/test_sub_pool_spec.py @@ -43,6 +43,52 @@ def test_declared_extra_entries_are_preserved(self): spec = state_pool(ENTRY_STATE, 10, entries_per_req=1, extra_entries=64) assert spec.extra_entries == 64 + def test_the_staging_ring_is_sized_here_not_by_the_caller(self, monkeypatch): + """`state_pool` is the only reader of the ring size. + + A backend that sized the ring itself would be a second env read, and + the arena and the pool would size themselves from different answers if + one of them ever drifted -- the arena short exactly the rows the spill + path addresses. + """ + monkeypatch.setenv("OFFLOAD_STATE_STAGING_GROUPS", "2") + spec = state_pool(ENTRY_STATE, 10, entries_per_req=3, offload_hosted=True) + # Groups, not entries: a staging group is `entries_per_req` rows. + assert spec.staging_entries == 2 * 3 + assert spec.extra_entries == 0 + + def test_the_checkpoint_headroom_and_the_staging_ring_add(self, monkeypatch): + """Two fields, because the ring must not eat the backend's cushion. + + They shared one parameter once, and the ring overwrote it: the arena + then lost the rows `state_entry_views(num_groups + slot)` addresses, and + GDN's `cache[layer, lo : lo + span]` clamps rather than raising -- a + short segment, an `entry_bytes` mismatch, and a crash in the packer. + """ + monkeypatch.setenv("OFFLOAD_STATE_STAGING_GROUPS", "2") + spec = state_pool( + ENTRY_STATE, 10, entries_per_req=1, extra_entries=16, offload_hosted=True + ) + assert spec.extra_entries == 16 + assert spec.staging_entries == 2 + + def test_one_group_is_reserved_by_default(self, monkeypatch): + """The tier has no on/off flag of its own -- it follows the + `lmcache_offload` connector -- so the ring has to exist by default or + the arena would be short the rows the spill path addresses. A + deployment that will never offload sets the size to 0.""" + monkeypatch.delenv("OFFLOAD_STATE_STAGING_GROUPS", raising=False) + spec = state_pool(ENTRY_STATE, 10, entries_per_req=3, offload_hosted=True) + assert spec.staging_entries == 3 + + monkeypatch.setenv("OFFLOAD_STATE_STAGING_GROUPS", "0") + assert ( + state_pool( + ENTRY_STATE, 10, entries_per_req=3, offload_hosted=True + ).staging_entries + == 0 + ) + class TestMergeSpecs: def test_same_name_sums_entry_bytes(self): @@ -381,3 +427,108 @@ def test_draft_bytes_raise_the_per_block_cost_not_the_pool_count(self): ) assert set(with_draft.entries) == {ENTRY_KV} assert with_draft.entries[ENTRY_KV] == target_only.entries[ENTRY_KV] // 2 + + +def test_staging_entries_are_allocated_but_not_admissible(): + """`staging_entries` buys rows the pool must not lease out. Allocation + sizes tensors from `entries`; admission divides `admission_entries`.""" + plan = plan_pools( + [ + page_pool(1000), + SubPoolSpec( + Pool.STATE, ENTRY_STATE, 500, entries_per_req=2, staging_entries=3 + ), + ], + available_bytes=1_000_000, + max_num_seqs=8, + ) + assert plan.entries[ENTRY_STATE] == 8 * 2 + 3 + assert plan.admission_entries[ENTRY_STATE] == 8 * 2 + # The reservation covers every allocated row, staging included. + assert plan.reserved_bytes[ENTRY_STATE] == (8 * 2 + 3) * 500 + + +def test_checkpoint_headroom_is_allocated_and_admissible(): + """The other cushion, and the opposite answer. + + `extra_entries` is checkpoint capacity: groups the state pool is *meant* to + hand out -- the backend's declared cushion. Withholding it + from admission -- which is what the staging split did before these two + fields were separated -- leaves `num_groups` at `max_num_seqs` and the + feature does nothing at all. + """ + plan = plan_pools( + [ + page_pool(1000), + state_pool(ENTRY_STATE, 500, entries_per_req=2, extra_entries=3), + ], + available_bytes=1_000_000, + max_num_seqs=8, + ) + assert plan.entries[ENTRY_STATE] == 8 * 2 + 3 + assert plan.admission_entries[ENTRY_STATE] == 8 * 2 + 3 + + +def test_admission_entries_defaults_to_entries_without_extras(): + plan = plan_pools( + [page_pool(1000), state_pool(ENTRY_STATE, 500, entries_per_req=2)], + available_bytes=1_000_000, + max_num_seqs=8, + ) + assert plan.admission_entries[ENTRY_STATE] == plan.entries[ENTRY_STATE] == 16 + + +def test_staging_groups_are_sized_by_multiplicity(monkeypatch): + """A staging *group* is `entries_per_req` rows. Sizing the ring in bare + entries would run the last staging group off the end of the tensor.""" + span, k = 3, 2 # GDN: span = 1 + num_spec + monkeypatch.setenv("OFFLOAD_STATE_STAGING_GROUPS", str(k)) + plan = plan_pools( + [ + page_pool(1000), + state_pool(ENTRY_STATE, 500, entries_per_req=span, offload_hosted=True), + ], + available_bytes=1_000_000, + max_num_seqs=8, + ) + num_groups = plan.admission_entries[ENTRY_STATE] // span + assert num_groups == 8 + # The highest staging group's last row must still be inside the tensor. + assert (num_groups + k - 1) * span + span <= plan.entries[ENTRY_STATE] + + +def test_the_ring_sits_past_the_headroom_not_on_top_of_it(monkeypatch): + """Both cushions at once: the ring must start past every admissible group. + + `state_spills_for_batch` addresses a staging entry as `num_groups + slot`, + and `num_groups` counts the headroom. A ring sized without it would + overlap groups the pool leases out -- a spill would read a live request's + state and store it under an evicted checkpoint's hash. + """ + span, k, headroom = 2, 2, 5 + monkeypatch.setenv("OFFLOAD_STATE_STAGING_GROUPS", str(k)) + plan = plan_pools( + [ + page_pool(1000), + state_pool( + ENTRY_STATE, + 500, + entries_per_req=span, + extra_entries=headroom * span, + offload_hosted=True, + ), + ], + available_bytes=1_000_000, + max_num_seqs=8, + ) + num_groups = plan.admission_entries[ENTRY_STATE] // span + assert num_groups == 8 + headroom + assert (num_groups + k - 1) * span + span <= plan.entries[ENTRY_STATE] + + +def test_paged_class_admission_equals_entries(): + """PAGE has no per-request reservation, so the two counts coincide and + `with_paged_entries` must keep them in step.""" + plan = plan_pools([page_pool(1000)], available_bytes=10_500, max_num_seqs=8) + assert plan.admission_entries[ENTRY_KV] == plan.entries[ENTRY_KV] == 10 + assert plan.with_paged_entries(4).admission_entries[ENTRY_KV] == 4 From 51b530a1e3e3884e1686949ef1ac833d6b516f00 Mon Sep 17 00:00:00 2001 From: zejunchen-zejun Date: Thu, 27 Aug 2026 16:10:31 +0800 Subject: [PATCH 003/151] refactor(offload): delete the staging ring, which #2045 left nothing to rescue Phase 2. About 1000 lines go, and none of them for tidiness: the ring existed to solve exactly one problem, and #2045 removed the problem. --- why the ring existed at all A checkpoint used to live in a state slot, and `StateSlotPool.pop` handed that slot to the next request the instant it was evicted. The bytes therefore had to be rescued somewhere before they were overwritten, which is what the ring was: K entries allocated inside the state arena, never leased to a request, and addressed as `num_slots + staging` so `state_entry_views` could reach them with no second scheme. Everything else -- `staging_entries`, `admission_entries`, the drain protocol, the starvation detector, the reclaimer, `pop(spill=)`, `spills_forgone`, the spill-before-relocate ordering in `CommonAttentionBuilder.build` -- was a consequence. Under #2045 a K3 checkpoint is a PAGE image in the KV pool whose 127 units are *reserved*, and reserved units are structurally un-takeable: `_take_free` draws only from `_free` and they sit in `_used`, `BlockPool.free` raises on one, and `retire_top` refuses outright (`if top in self._raw_unit_owner: return None`). The source holds still on its own, so a transfer needs no rescue -- exactly the property dense KV has always had, where `should_defer_free` alone carries a save through its D2H and no ring was ever needed. So this is not a simplification, it is the removal of a workaround whose premise is gone. `state_pool.py` and `sub_pool_spec.py` now have an EMPTY diff against --- what stays, and why `state_entry_views` stays (re-indexed from group to slot, since a checkpoint is one slot wide now). Phase 4's load writes the committed slot directly, so this is its destination -- the plan's earlier draft had it deleted, which was wrong. `StagedTransfer` / `_StagingBuffer` stays. It shares a word with the ring and is a different thing: a per-thread pinned host bounce buffer that D2H and H2D both need, sized in bytes and unrelated to slot count. `StateOffloadIndex` keeps its load half (`request_load` / `complete_load` / `fail_load` / `abandon_load`) and its optimistic hash set. `confirm_spill` becomes `note_stored`, which is what it always did. `StateOffloadTier.submit_spill` becomes `submit_store(h, slot)` and loses its `ready_event`: there is no compute-stream producer to fence against once the packer gathers straight from a source that is not moving. --- what this leaves inert, deliberately **Nothing calls `submit_store` after this commit.** Phase 2 removed the ring that used to drive it and Phase 3 gives it its new caller, whose source is the checkpoint's PAGE units rather than a slot. Nothing populates `StateOffloadIndex.hashes` in between, so `_tier_can_serve` is false and no load is ever offered -- the tier is fully inert, on top of the three gates Phase 1 already documented. That is a WIP state, not a shippable one. --- one dead-code find, pre-existing on #1960 A method between `_tier_can_serve` and `_request_state_load` had lost its `def` line in an earlier rebase, leaving its docstring and `return` as unreachable statements after `_tier_can_serve`'s own return. It reads like a live invariant ("Whether `hit_hash`'s state can be produced at all, from either tier") and has no caller: the tier's vote actually reached `_gated_hit` through `StateSlotPool._resumable_from`. Removed rather than restored, since Phase 4 puts the vote on the coordinator side. Verified: pytest tests/ (less tests/plugin and the two msgpack/aiter collection errors that fail on #2045 too) -> 3906 passed, 326 skipped, 3 xfailed, zero failures. black clean; ruff 36, the same count as #2045. `git diff` against Co-Authored-By: Claude Opus 5 (1M context) --- .../offload/hybrid/kimi_k3/connector.py | 38 +-- .../offload/hybrid/kimi_k3/state_tier.py | 89 +++--- atom/model_engine/block_manager.py | 119 ++------ atom/model_engine/model_runner.py | 20 +- atom/model_engine/scheduler.py | 26 +- atom/model_engine/state_offload.py | 281 +++--------------- atom/model_engine/state_pool.py | 135 +-------- atom/model_engine/state_runtime.py | 11 +- atom/model_ops/attentions/backends.py | 40 +-- atom/model_ops/attentions/deepseek_v4_attn.py | 20 +- atom/model_ops/attentions/gdn_attn.py | 14 +- atom/model_ops/attentions/sub_pool_spec.py | 67 +---- docs/environment_variables.md | 1 - tests/test_block_manager.py | 43 ++- tests/test_lmcache_offload_connector.py | 14 +- tests/test_multi_connector.py | 2 +- tests/test_sub_pool_spec.py | 151 ---------- 17 files changed, 167 insertions(+), 904 deletions(-) diff --git a/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py b/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py index b15a5dc84b..362c44f046 100644 --- a/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py +++ b/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py @@ -38,7 +38,6 @@ #: them opaquely and takes a failure-dominant TP quorum, which is what makes a #: partial store resolve instead of pinning a key forever. STATE_INDEX_CHANNEL = "k3_state_index" -STATE_STAGING_CHANNEL = "k3_state_staging" #: A save outstanding longer than this is a backend that stopped, not one that #: is busy: a 4096-token store costs ~65ms. @@ -66,15 +65,9 @@ def register_kv_caches( def _build_state_tier(self, transfer_tensors) -> None: from aiter.dist.parallel_state import get_tp_group - from atom.model_engine.state_offload import state_offload_staging_groups - - if state_offload_staging_groups() <= 0: - return - - # PP breaks the tier twice over: the CacheEngineKey has no PP component, - # so two stages at the same TP rank would overwrite each other; and only - # the head stage drains staging releases, so the ring starves. Refused - # rather than half-supported. Paged KV is unaffected. + # PP breaks the tier: the CacheEngineKey has no PP component, so two + # stages at the same TP rank would overwrite each other. Refused rather + # than half-supported. Paged KV is unaffected. pp_size = int(getattr(self._config, "pipeline_parallel_size", 1) or 1) if pp_size > 1: logger.warning( @@ -242,14 +235,15 @@ def get_finished(self): out = super().get_finished() if self._state_tier is None: return out - indexed, released, index_failed = self._state_tier.take_spill_reports() + indexed, index_failed = self._state_tier.take_store_reports() state_done, state_failed = self._state_tier.get_finished() out.finished_loading, out.failed_loading = self._settle_joint( out.finished_loading, out.failed_loading, state_done, state_failed ) - # Spill and staging reports have no request identity, so they ride the - # connector-owned channels; the aggregator's quorum is failure-dominant, - # which is what resolves a partial store instead of pinning the key. + # Store reports have no request identity -- by the time one lands its + # owner is long gone -- so they ride the connector-owned channel; the + # aggregator's quorum is failure-dominant, which is what resolves a + # partial store instead of pinning the key. for h in indexed: out.connector_completions.add( ConnectorCompletion(STATE_INDEX_CHANNEL, int(h), True) @@ -258,10 +252,6 @@ def get_finished(self): out.connector_completions.add( ConnectorCompletion(STATE_INDEX_CHANNEL, int(h), False) ) - for group in released: - out.connector_completions.add( - ConnectorCompletion(STATE_STAGING_CHANNEL, int(group), True) - ) return out def _settle_joint(self, kv_done, kv_failed, state_done, state_failed): @@ -314,7 +304,6 @@ def __init__(self, config) -> None: # Channel reports drained by the engine each step. self._state_indexed: set[int] = set() self._state_index_failed: set[int] = set() - self._state_staging_released: set[int] = set() # -- state load queue -------------------------------------------------- def enqueue_state_loads(self, loads) -> bool: @@ -468,17 +457,12 @@ def connector_completion(self, completion) -> bool: ) target.add(int(completion.operation_id)) return True - if completion.channel == STATE_STAGING_CHANNEL: - self._state_staging_released.add(int(completion.operation_id)) - return True return super().connector_completion(completion) - def take_state_reports(self) -> tuple[set[int], set[int], set[int]]: - """Drain this step's tier reports for the engine-side index.""" + def take_state_reports(self) -> tuple[set[int], set[int]]: + """Drain this step's tier store reports for the engine-side index.""" indexed = self._state_indexed failed = self._state_index_failed - released = self._state_staging_released self._state_indexed = set() self._state_index_failed = set() - self._state_staging_released = set() - return indexed, released, failed + return indexed, failed diff --git a/atom/kv_transfer/offload/hybrid/kimi_k3/state_tier.py b/atom/kv_transfer/offload/hybrid/kimi_k3/state_tier.py index 5f0b423844..74cac6c19c 100644 --- a/atom/kv_transfer/offload/hybrid/kimi_k3/state_tier.py +++ b/atom/kv_transfer/offload/hybrid/kimi_k3/state_tier.py @@ -3,14 +3,14 @@ One executor of its own, separate from the KV connector's `_load_executor` and `_save_executor` so that state transfers and KV transfers cannot block each -other -- but a single one, shared by this tier's own spills and loads. See +other -- but a single one, shared by this tier's own stores and loads. See `submit_load` for why the KV connector's load/save split is not copied here. This class **reports, and the engine applies**. It runs in a spawned runner process while `StateOffloadIndex` lives in the engine process, so this side -cannot free a staging slot or index a hash directly. `take_spill_reports` and +cannot index a hash directly. `take_store_reports` and `get_finished` hand their sets to the connector, which emits the first as -`ConnectorCompletion`s on its own channels and merges the second into +`ConnectorCompletion`s on its own channel and merges the second into `finished_loading`/`failed_loading`. The failed-hash set exists to resolve the aggregator's quorum: without it a partial store would pin the hash forever. """ @@ -39,10 +39,9 @@ def __init__(self, codec, *, max_workers: int = 1) -> None: self._done: set[str] = set() self._failed: set[str] = set() self._inflight: set = set() - # The spill path's three reports, drained by `take_spill_reports`. + # The store path's two reports, drained by `take_store_reports`. self._indexed: set[int] = set() self._index_failed: set[int] = set() - self._released: set[int] = set() def _register(self, fut) -> None: """Add *fut* to the inflight set and attach a callback that removes it @@ -59,33 +58,35 @@ def _discard(f): fut.add_done_callback(_discard) - def submit_spill( - self, h: int, entry_index: int, staging_slot: int, ready_event=None - ) -> None: - """`entry_index` is what the codec packs, `staging_slot` what the ring - releases. Two index spaces: the staging entries sit past the pool's - range in the arena (`num_groups + slot`), while the ring counts from 0. + def submit_store(self, h: int, slot: int) -> None: + """Pack `slot`'s state under `h` and hand it to LMCache. - `ready_event` fences the D2D staging copy that `AttentionBackend.build` - issued on the forward's compute stream. This worker packs on its own - stream, so without the wait it reads the entry's previous occupant. + No `ready_event`. The staging ring is gone (#2045 makes the source hold + still on its own), so nothing on the compute stream is producing these + bytes for us to fence against -- `StagedTransfer.pack` gathers straight + from the source on its own stream. + + NOTE: nothing calls this yet. Phase 2 removed the ring that used to + drive it and Phase 3 gives it its new caller, whose source is the + checkpoint's PAGE units rather than a slot. Kept rather than deleted so + the two halves of the change stay reviewable apart. """ - fut = self._executor.submit( - self._do_spill, h, entry_index, staging_slot, ready_event - ) - self._register(fut) + self._register(self._executor.submit(self._do_store, h, slot)) - def submit_load(self, req_id: str, h: int, group: int) -> None: - """Fetch `h` into pool group `group` for the parked request `req_id`. + def submit_load(self, req_id: str, h: int, slot: int) -> None: + """Fetch `h` into pool slot `slot` for the parked request `req_id`. - Shares one serial executor with the spills, unlike the KV connector's + Shares one serial executor with the stores, unlike the KV connector's split pair: `StagedTransfer` keeps its staging buffer in `threading.local`, so a second thread costs a second resident buffer - per rank out of the HBM the state pool is already short of. The queue - it would shorten is at most `OFFLOAD_STATE_STAGING_GROUPS` deep. + per rank out of HBM the state pool is already short of. + + A load is on the TTFT critical path and a store is not, so once stores + become frequent (Phase 3 fires one per checkpoint rather than one per + eviction) this queue wants a priority, or a load can wait behind + several stores. Measure `state_load_queue_wait_ms` before adding one. """ - fut = self._executor.submit(self._do_load, req_id, h, group) - self._register(fut) + self._register(self._executor.submit(self._do_load, req_id, h, slot)) def drain(self) -> None: """Block until every submitted transfer has settled. Tests and shutdown @@ -105,16 +106,12 @@ def get_finished(self) -> tuple[set[str], set[str]]: def shutdown(self) -> None: self._executor.shutdown(wait=True) - def _do_spill( - self, h: int, entry_index: int, staging_slot: int, ready_event=None - ) -> None: + def _do_store(self, h: int, slot: int) -> None: stored = False try: - if ready_event is not None: - ready_event.synchronize() - stored = bool(self.codec.put(h, entry_index)) - except Exception: # a spill is best effort by design - logger.warning("state offload: spill of hash %d failed", h, exc_info=True) + stored = bool(self.codec.put(h, slot)) + except Exception: # a store is best effort by design + logger.warning("state offload: store of hash %d failed", h, exc_info=True) with self._lock: # Report, never apply: `StateOffloadIndex` lives in the engine # process and this runs in a spawned runner. The engine applies @@ -126,30 +123,24 @@ def _do_spill( # `indexed | index_failed` instead of waiting for a second # report that will never come from this rank. self._index_failed.add(int(h)) - # Always, stored or not: a leaked slot shrinks the ring - # permanently and the feature quietly stops spilling. - self._released.add(int(staging_slot)) - def take_spill_reports(self) -> tuple[set[int], set[int], set[int]]: - """`(hashes stored, staging slots free, hashes failed)` since the last call. + def take_store_reports(self) -> tuple[set[int], set[int]]: + """`(hashes stored, hashes failed)` since the last call. - A hash appears in exactly one of the first or third set per spill. - The aggregator's quorum over the two is failure-dominant, so a partial - store resolves in the same step rather than pinning the key. + A hash appears in exactly one of the two per store. The aggregator's + quorum over them is failure-dominant, so a partial store resolves in + the same step rather than pinning the key. """ with self._lock: indexed = set(self._indexed) index_failed = set(self._index_failed) - released = set(self._released) self._indexed.clear() self._index_failed.clear() - self._released.clear() - return indexed, released, index_failed + return indexed, index_failed - def _do_load(self, req_id: str, h: int, group: int) -> None: - # A load target is a real pool group, not a staging entry: the bytes - # land where the resuming request will read them. Only the spill - # direction needs the staging indirection. + def _do_load(self, req_id: str, h: int, slot: int) -> None: + # The bytes land in the committed slot, where the resuming request + # reads them. # # A miss is a normal path, not an error: LMCache's LRU can drop bytes # under a hash the engine's index still advertises. Retracting that @@ -157,7 +148,7 @@ def _do_load(self, req_id: str, h: int, group: int) -> None: # the report below. ok = False try: - ok = bool(self.codec.get(h, group)) + ok = bool(self.codec.get(h, slot)) except Exception: # a failed load is a normal path logger.warning("state offload: load of hash %d failed", h, exc_info=True) with self._lock: diff --git a/atom/model_engine/block_manager.py b/atom/model_engine/block_manager.py index e5bc8d5f47..e3aa6f6feb 100644 --- a/atom/model_engine/block_manager.py +++ b/atom/model_engine/block_manager.py @@ -223,10 +223,8 @@ def __init__( from atom.model_engine.state_offload import ( StateOffloadIndex, kv_connector_hosts_state_tier, - state_offload_staging_groups, ) - staging = state_offload_staging_groups() # Joint boundaries, split by what the state leg cost: `hbm` forked a # resident checkpoint, `tier` paid an entry-sized H2D and a park. Almost # all `tier` means the state pool is too small for the concurrency. @@ -258,26 +256,22 @@ def __init__( self.state_offload: StateOffloadIndex | None = None # (req_id, hash, target_group) admitted this pass and not yet handed to # the connector. Kept here rather than in the index because the slot - # is this object's fact, the same reason `state_spills_for_batch` joins - # the spill's two halves engine-side. + # is this object's fact. self._state_loads: list[tuple] = [] # req_id -> slot, for loads whose request was deallocated before the # bytes landed. The slot is off the free list until the report comes # back; see `deallocate`. self._orphan_load_slots: dict = {} - # `kv_offload_enabled` is the whole switch. Without a connector to - # drain a spill the ring would hand out slots that never come back, and - # a hash whose KV left HBM could not be resumed from anyway. - if kv_offload_enabled and staging > 0: - index = StateOffloadIndex(staging_depth=staging) - # Only the slot pool: `_spill` / `_resumable_from` live on - # `StateSlotPool`, while a - # PAGE checkpoint already lives in the KV blocks the KV connector - # offloads on its own. - for cache in self.state_caches: - if isinstance(cache, StateSlotPool): - cache.offload = index - self.state_offload = index + # `kv_offload_enabled` is the whole switch: without a connector to + # carry a transfer there is nothing beneath the pool, and a hash whose + # KV left HBM could not be resumed from anyway. + # + # The index no longer reaches into `StateSlotPool`. A K3 checkpoint is + # a PAGE image in the KV pool (#2045), so the spill source is a set of + # units the coordinator owns, not a slot the pool is about to hand + # away -- which is what the staging ring existed to rescue. + if kv_offload_enabled: + self.state_offload = StateOffloadIndex() # The demand funnel: recorded at admission, cut for when a prefill # chunk is shortened to land on it, kept when the state pool files it. # Counted at all three because a gap between any two is a different @@ -316,51 +310,12 @@ def take_state_maintenance_ops(self) -> StateMaintenanceOps: stores = restores = () if self.paged_state_checkpoints is not None: stores, restores = self.paged_state_checkpoints.take_checkpoint_ops() - # Drained here rather than beside the relocations so that the one - # consumer -- `CommonAttentionBuilder.build` -- is also the one place - # the spill-before-relocate order is enforced. return StateMaintenanceOps( relocations=relocations, - spills=tuple(self.state_spills_for_batch()), checkpoint_stores=stores, checkpoint_restores=restores, ) - def state_spills_for_batch(self) -> list[tuple[int, int, int, int]]: - """`(src_group, dst_entry, staging_slot, hash)` the batch being built - must copy out before its forward. - - The pool knows `(group, slot)` and the ring knows `(hash, slot)`; the - slot is the only thing relating them. Joined engine-side because - `num_groups` is authoritative here. Both lists are appended by the same - `_spill()` and drained together, so a slot in one and not the other is a - bug -- dropped rather than guessed at. - """ - out: list[tuple[int, int, int, int]] = [] - for cache in self.state_caches: - offload = getattr(cache, "offload", None) - if offload is None: - continue - hash_of_slot = {slot: h for h, slot in offload.take_pending()} - for group, slot in cache.take_spill_copies(): - h = hash_of_slot.pop(slot, None) - if h is None: - logger.warning( - "state offload: staging slot %d has a copy but no " - "pending hash; dropping the spill", - slot, - ) - offload.release_staging(slot) - continue - out.append((group, cache.num_groups + slot, slot, h)) - # Whatever is left is a pending hash with no copy to feed it. The - # keys are the slots; the values are the hashes, which the ring - # does not take back. - for slot in hash_of_slot: - # Never observed in correct operation; release rather than leak. - offload.release_staging(slot) - return out - def state_checkpoint_fates(self) -> dict[str, int]: """Summed fates across every state class, for the periodic stats line. @@ -949,16 +904,8 @@ def _attach_state_slots(self, seq: Sequence, hit_hash: int) -> bool: src = self.state.lookup(hit_hash) if hit_hash != -1 else -1 if src < 0: - # Decided before the pop, not after: an eviction that makes room - # for a load must not spill, because the spill's copy out of that - # slot is issued by a later forward while the load writes it from - # the tier's thread first. See `StateSlotPool.pop`. wants_load = self._tier_can_serve(hit_hash) - # Only the committed slot can be the load's destination, so only it - # has to forgo the spill; the rollback scratch is this request's own. - seq.state_slots = [self.state.pop(spill=not wants_load)] + ( - self.state.pop_many(width - 1) - ) + seq.state_slots = self.state.pop_many(width) seq.state_fork_src = -1 if wants_load and self._request_state_load(seq, hit_hash): return True @@ -993,42 +940,23 @@ def _attach_state_slots(self, seq: Sequence, hit_hash: int) -> bool: def _tier_can_serve(self, hit_hash: int) -> bool: """Whether the tier believes it holds `hit_hash`. - Asked twice per admission -- before `pop`, to decide whether that - eviction may spill, and inside `_request_state_load`. One predicate so - the two cannot drift: a pop that forwent its spill for a load then - refused would lose a checkpoint for nothing. + + Asked twice per admission -- once by `_attach_state_slots` to decide + whether to try a load at all, and again inside `_request_state_load`. + One predicate so the two cannot drift. """ return ( hit_hash != -1 and self.state_offload is not None and hit_hash in self.state_offload.hashes ) - """Whether `hit_hash`'s state can be produced at all, from either tier. - - Distinct from `_tier_can_serve`, which asks "will a load be issued" and - decides whether an eviction may spill. This asks "does this boundary - have a state behind it" -- an HBM checkpoint answers yes here, no there. - - The HBM branch is reachable, not redundant: `unindex` drops a checkpoint - when *its own* block leaves the index, but `can_allocate`'s walk stops at - the first miss anywhere in the chain, so an earlier block going puts a - live HBM checkpoint above `hbm_boundary`. That is the cheapest joint load - there is -- no state transfer, just the fork. - - Nothing downstream needs to know which branch answered: - `_attach_state_slots` looks HBM up first and falls to a load on a miss. - """ - return self._tier_can_serve(hit_hash) or ( - hit_hash != -1 and self.state.lookup(hit_hash) >= 0 - ) def _request_state_load(self, seq: Sequence, hit_hash: int) -> bool: """Ask the tier to fetch `hit_hash` into the slot `seq` just took. Only reached when the HBM index missed -- the case the tier exists for. - The destination is a real pool slot, never a staging entry: the bytes - land where the resuming forward reads them, and only the spill direction - needs the indirection. + The bytes land in the committed slot, where the resuming forward reads + them. `state_fork_src` stays -1. The loaded slot *is* the incoming state, and naming a source would send the forward to a different slot than the one @@ -1041,17 +969,6 @@ def _request_state_load(self, seq: Sequence, hit_hash: int) -> bool: """ if not self._tier_can_serve(hit_hash): return False - if self.state.has_pending_spill(seq.state_slot): - # A spill queued in an earlier pass still has to read this slot - # and the load would overwrite it first. Decline; the caller - # disowns the boundary, which is the pre-tier answer. - logger.warning( - "state offload: slot %d still owes a spill copy; declining " - "the load for seq %s and recomputing instead.", - seq.state_slot, - seq.id, - ) - return False if not self.state_offload.request_load(seq.id, hit_hash): return False seq.state_load_hash = hit_hash diff --git a/atom/model_engine/model_runner.py b/atom/model_engine/model_runner.py index 5f18ce03df..629d22c9e0 100644 --- a/atom/model_engine/model_runner.py +++ b/atom/model_engine/model_runner.py @@ -1726,10 +1726,7 @@ def get_num_blocks(self) -> dict[str, object]: # sliding-window pool and the attention builder each index it by the # class name they declared. Nothing here needs to know those names. self.pool_plan = plan - # BlockManager divides this into request groups; the offload staging - # cushion is allocated but not leasable, so admission must not see it. - # Backends keep sizing their tensors from `pool_plan.entries`. - config.pool_entries = dict(plan.admission_entries) + config.pool_entries = dict(plan.entries) config.pool_entries_per_req = dict(plan.entries_per_req) # Keep runtime state metadata out of Config. transfer = self.attn_metadata_builder.state_transfer() @@ -1777,15 +1774,9 @@ def get_num_blocks(self) -> dict[str, object]: ) self.state_runtime = state_runtime for name in sorted(plan.entries): - extra = plan.entries[name] - plan.admission_entries[name] logger.info( - f"sub-pool {name}: entries={plan.entries[name]}" - + ( - f" ({plan.admission_entries[name]} admissible + {extra} staging)" - if extra - else "" - ) - + f", entry_bytes={plan.entry_bytes[name]}, " + f"sub-pool {name}: entries={plan.entries[name]}, " + f"entry_bytes={plan.entry_bytes[name]}, " f"reserved={plan.reserved_bytes[name] / (1 << 30):.2f}GB" ) @@ -1853,10 +1844,7 @@ def get_num_blocks(self) -> dict[str, object]: # the class they declared. return { "num_kvcache_blocks": num_kvcache_blocks, - # BlockManager divides this into request groups; the offload staging - # cushion is allocated but not leasable, so admission must not see it. - # Backends keep sizing their tensors from `pool_plan.entries`. - "pool_entries": dict(plan.admission_entries), + "pool_entries": dict(plan.entries), "pool_entries_per_req": dict(plan.entries_per_req), "state_runtime": state_runtime.to_wire(), } diff --git a/atom/model_engine/scheduler.py b/atom/model_engine/scheduler.py index 7555317494..5d319c1583 100644 --- a/atom/model_engine/scheduler.py +++ b/atom/model_engine/scheduler.py @@ -3396,30 +3396,20 @@ def _update_from_kv_xfer_finished(self, kv_connector_output: KVConnectorOutput): if seq is not None: self._maybe_release_deferred(seq) - # The state-offload reports, not keyed by request: by the time a spill - # lands its owner is long gone and only the hash and slot remain. They + # The state-offload store reports, not keyed by request: by the time a + # store lands its owner is long gone and only the hash remains. They # ride the connector's completion channels, so they are drained from the # connector with TP quorum already taken. + # + # Indexed only on the report, never at submission: a hash advertised + # before its bytes exist parks the next request over that prefix + # against a `get` that must miss. offload = getattr(getattr(self, "block_manager", None), "state_offload", None) take = getattr(self.kv_connector, "take_state_reports", None) if offload is not None and take is not None: - indexed, released, _failed = take() - # Index before releasing, always. The reverse order opens a window - # in which the slot is reusable but its hash is not yet findable, - # so a lookup misses a checkpoint that is in fact stored. + indexed, _failed = take() for h in indexed: - offload.confirm_spill(int(h)) - for slot in released: - offload.release_staging(int(slot)) - # Ring-side twin of `_reconcile_stalled_deferred_saves`: a slot whose - # spill report never returns (worker abandoned/lost the transfer) - # would pin the ring until every spill drops. Reclaim it once the - # same abandon window the engine uses has elapsed -- larger than the - # upstream force-unpin timeout, so the staging buffer is no longer in - # flight before the slot is reused. Self-throttled inside. - reclaim = getattr(offload, "reclaim_stale_slots", None) - if callable(reclaim): - reclaim(_offload_save_abandon_timeout_s()) + offload.note_stored(int(h)) def get_request_counts(self) -> tuple[int, int]: """Returns (num_running_reqs, num_waiting_reqs).""" diff --git a/atom/model_engine/state_offload.py b/atom/model_engine/state_offload.py index 4ff04e64f0..41febb12c6 100644 --- a/atom/model_engine/state_offload.py +++ b/atom/model_engine/state_offload.py @@ -1,221 +1,58 @@ # SPDX-License-Identifier: MIT """Scheduler-side bookkeeping for the state-cache offload tier. -Two things live here and neither touches a device: the set of hashes believed -to be in LMCache, and the bounded queue of groups waiting to be read out of -HBM. The bytes and the transfers are the worker's business -(`kv_transfer/offload/state_tier.py`). +One thing lives here and it touches no device: the set of hashes believed to be +in LMCache, plus the loads offered against them. The bytes and the transfers are +the worker's business (`kv_transfer/offload/hybrid/kimi_k3/state_tier.py`). + +There used to be a second thing -- a bounded ring of staging entries a spill +copied an evicted checkpoint into. It existed for one reason: a checkpoint lived +in a state slot, and `StateSlotPool.pop` handed that slot to the next request +the instant it was evicted, so the bytes had to be rescued somewhere before they +were overwritten. Under #2045 a K3 checkpoint is a PAGE image in the KV pool +whose units are reserved and structurally un-takeable, so the source holds still +on its own and the ring has nothing left to rescue. See `REBASE_PLAN_v2.md`. The set is in memory and never persisted: `LocalDiskBackend.__init__` starts from an empty dict and never scans its directory, so an index recovered from -disk would be a pure false-positive generator. Index and bytes share one -server lifetime. +disk would be a pure false-positive generator. Index and bytes share one server +lifetime. """ import logging -import os -import time -from collections import deque logger = logging.getLogger("atom") -# How often `reclaim_stale_slots` actually scans, regardless of how often it is -# called. The scan is O(staging_depth) and self-throttled so wiring it into a -# per-step drain loop costs nothing when nothing is stuck. -_RECLAIM_SCAN_INTERVAL_S = 5.0 - -# Consecutive dropped spills, with no slot coming back, before the ring is -# called starved rather than busy. A slot returns only once the worker's D2H -# lands, so a burst of evictions can legitimately drop for a step or two; an -# order of magnitude above that, so crossing it means the slots are gone. -_STARVATION_DROP_THRESHOLD = 256 - class StateOffloadIndex: - """What has been spilled, what is queued to be, and what is coming back. + """What is believed to be in LMCache, and what is being fetched back. - `hashes` answers the membership half of `StateSlotPool._resumable_from`. - It is deliberately optimistic: LMCache's own LRU can drop bytes at any - time, so a hash here means "was spilled once", never "is still there". The - false positive costs one lookup and a park/unpark, and is handled by the - `failed_loading` path (`fail_load` -> `forget`). + `hashes` answers the membership half of the tier's vote. It is deliberately + optimistic: LMCache's own LRU can drop bytes at any time, so a hash here + means "was stored once", never "is still there". The false positive costs + one lookup and a park/unpark, and is handled by the `failed_loading` path + (`fail_load` -> `forget`). """ - def __init__(self, staging_depth: int) -> None: - self.staging_depth = max(0, int(staging_depth)) - # Orphaned checkpoints (`unindex`) are worth spilling only when the KV - # prefix can also come back: `resumable_hit` scans `block_hashes`, which - # `can_allocate` builds from HBM `kv.lookup` hits only. With KV offload - # off, a hash whose KV left HBM never reappears and the bytes are wasted. - # - # In production today this is always True: `BlockManager.__init__` - # refuses to construct the index at all when the connector does not - # host the tier, so the only path here has already established it. The - # flag is kept, and the `unindex` gate with it, because the two - # conditions are independent in principle -- a future connector could - # host the tier's transport without offloading KV -- and because the - # failure it prevents (spending LMCache capacity on hashes no load can - # ever reach) is silent. Its False arm is covered by tests only. + def __init__(self) -> None: self.hashes: set[int] = set() - self._free_slots: deque[int] = deque(range(self.staging_depth)) - self._pending: deque[tuple[int, int]] = deque() # req_id -> hash, for loads offered and not yet settled. Keyed by # request because that is what comes back, on the same - # `finished_loading`/`failed_loading` channel a KV load uses. (Spill - # reports are keyed by hash and slot -- their owner is long gone.) + # `finished_loading`/`failed_loading` channel a KV load uses. self.pending_loads: dict = {} - # Read once, here, rather than per call: `_resumable_from` consults it - # inside a per-block scan. - self.spills_requested = 0 - self.spills_dropped = 0 - # Evictions that deliberately did not spill because the group was - # going to a load. Apart from `spills_dropped`: a deeper ring does not - # fix this one, it is the price of the load. - self.spills_forgone = 0 self.loads_attempted = 0 self.loads_completed = 0 self.loads_failed = 0 - # Drops since the last `release_staging`. A full ring is normal - # backpressure only while slots come back; none in this many attempts - # means the consumer stopped draining. - self._consecutive_drops = 0 - self._warned_starved = False - # slot -> time.monotonic() when it was reserved by `request_spill`. - # A slot's only way home is `release_staging`, driven by the worker's - # spill report. If that report never comes (the worker abandoned the - # transfer, or its completion was lost the same way a KV save's can be), - # the slot is pinned forever and the ring silently starves. This stamp - # lets `reclaim_stale_slots` return a slot the report has clearly - # abandoned. Reclaimed slots (`_slots_reclaimed`) are counted so the - # symptom is visible rather than a silent throughput cliff. - self._slot_reserved_at: dict[int, float] = {} - self._next_reclaim_at: float = 0.0 - self._slots_reclaimed = 0 - - @property - def enabled(self) -> bool: - return self.staging_depth > 0 - - def request_spill(self, h: int, group: int) -> int: - """Reserve a staging slot for `group`, or -1 if the ring is full. - - On `pop()`'s critical path, so no work beyond a deque pop. The caller - copies `group` into the slot on the compute stream and `pop()` hands the - original out immediately -- spilling by copy rather than by pin, because - `pop()` runs precisely when there is no free group to withhold. - """ - # A negative hash is a caller bug, not a dropped spill -- `_spill` - # already refuses a group with no checkpoint. Guarded, but not counted, - # so `spills_dropped` stays a clean backpressure signal. - if h < 0: - return -1 - if not self._free_slots: - self.spills_dropped += 1 - self._note_drop() - return -1 - slot = self._free_slots.popleft() - self._pending.append((h, slot)) - self._slot_reserved_at[slot] = time.monotonic() - self.spills_requested += 1 - return slot - - def _note_drop(self) -> None: - """Warn once when drops stop looking like a burst and start looking - like a consumer that has gone away.""" - self._consecutive_drops += 1 - if self._warned_starved or self._consecutive_drops < _STARVATION_DROP_THRESHOLD: - return - self._warned_starved = True - logger.warning( - "State offload staging ring is starved: %d spills dropped in a row " - "with no slot released (staging_depth=%d). The consumer is very " - "likely not calling StateSlotPool.take_spill_copies() every step, " - "or not calling StateOffloadIndex.release_staging(slot) once the " - "bytes are safe. Every further spill will be dropped until it does.", - self._consecutive_drops, - self.staging_depth, - ) - - def take_pending(self) -> list[tuple[int, int]]: - """Drain the queue as `(hash, staging_slot)` pairs.""" - out = list(self._pending) - self._pending.clear() - return out - - def confirm_spill(self, h: int) -> None: - """Index `h` once its bytes reached LMCache.""" - self.hashes.add(h) - def release_staging(self, slot: int) -> None: - if 0 <= slot < self.staging_depth and slot not in self._free_slots: - self._free_slots.append(slot) - self._slot_reserved_at.pop(slot, None) - # The ring is moving again, so any drops so far were a burst. - self._consecutive_drops = 0 + # ------------------------------- stores -------------------------------- # + def note_stored(self, h: int) -> None: + """A store landed in LMCache and the hash is now worth voting for. - def reclaim_stale_slots(self, timeout_s: float, now: float | None = None) -> int: - """Return slots whose spill report never came back to the free ring. - - `release_staging` is the only way a slot comes home, and it fires only - on the worker's spill report. A lost report would pin that slot forever; - enough of them and `request_spill` drops every spill (`spills_dropped` - climbs, `_note_drop` warns) with no way to recover short of a restart. - The ring-side twin of the engine's `_reconcile_stalled_deferred_saves`, - and it takes that reconciler's window so the two cannot drift. - - **Only slots the spill queue has let go are eligible.** A slot still in - `_pending` has not been drained by `BlockManager.state_spills_for_batch` - yet, and that method drains `take_pending()` and the pool's - `take_spill_copies()` in one call -- so a slot in `_pending` still has a - `(slot, staging)` copy queued on `StateSlotPool`. Returning it would let - a new spill reserve the same slot, and the two copies would then race - for the single `{slot: hash}` entry the drain builds: the loser is - reported as "a copy with no pending hash" and releases the slot out from - under the winner. Skipping those costs nothing -- an undrained spill - means the engine has not run a batch, which is what - `_reconcile_stalled_deferred_saves` is busy fixing; the slot becomes - eligible on the next scan once the batch drains it. - - Safety otherwise mirrors the engine-side reconciliation: `timeout_s` is - past the upstream force-unpin window, so the worker's staging buffer is - no longer being read before the slot is handed to a new spill. - Self-throttled to one real scan per `_RECLAIM_SCAN_INTERVAL_S`, so it is - safe to call every step. Returns the number of slots reclaimed. + Called from the report the worker sends back, never at submission: a + hash advertised before its bytes exist parks the next request over that + prefix against a `get` that must miss. """ - if timeout_s <= 0 or not self._slot_reserved_at: - return 0 - now = time.monotonic() if now is None else now - if now < self._next_reclaim_at: - return 0 - self._next_reclaim_at = now + _RECLAIM_SCAN_INTERVAL_S - queued = {slot for _h, slot in self._pending} - stale = [ - slot - for slot, at in self._slot_reserved_at.items() - if now - at >= timeout_s and slot not in queued - ] - if not stale: - return 0 - for slot in stale: - self._slot_reserved_at.pop(slot, None) - if 0 <= slot < self.staging_depth and slot not in self._free_slots: - self._free_slots.append(slot) - self._slots_reclaimed += len(stale) - # A reclaim means the ring was stuck, not bursting; reset the burst - # counter so a fresh starvation warning can arm if it stalls again. - self._consecutive_drops = 0 - self._warned_starved = False - logger.warning( - "State offload staging ring reclaimed %d slot(s) whose spill report " - "never returned after %.0fs (staging_depth=%d, total reclaimed=%d). " - "The worker very likely abandoned or lost the transfer's completion; " - "the slot is returned to the ring so spills can resume.", - len(stale), - timeout_s, - self.staging_depth, - self._slots_reclaimed, - ) - return len(stale) + self.hashes.add(int(h)) def forget(self, h: int) -> None: """Drop a hash whose load failed, so the next request does not retry.""" @@ -247,7 +84,7 @@ def request_load(self, req_id, h: int) -> bool: return True def complete_load(self, req_id) -> None: - """The bytes landed in the request's group. + """The bytes landed in the request's slot. The hash stays indexed: a load reads LMCache, it does not consume it, and the next request over the same prefix must still find it. @@ -279,24 +116,15 @@ def abandon_load(self, req_id) -> None: def stats(self) -> dict[str, int]: """Counters for the periodic `state checkpoints:` line. - Reached via `StateSlotPool.checkpoint_fates`, which prefixes every - key with `state_offload_`. Read the three load counters together: - `failed / attempted` is this index's false-positive rate, and - `attempted - completed - failed` is what is in flight or was abandoned - by an aborted request. + Read the three load counters together: `failed / attempted` is this + index's false-positive rate, and `attempted - completed - failed` is + what is in flight or was abandoned by an aborted request. """ return { - "spills_requested": self.spills_requested, - "spills_dropped": self.spills_dropped, - "spills_forgone": self.spills_forgone, "loads_attempted": self.loads_attempted, "loads_completed": self.loads_completed, "loads_failed": self.loads_failed, "indexed": len(self.hashes), - # Slots `reclaim_stale_slots` had to take back. Non-zero means spill - # reports are being lost, which the warning says once and this keeps - # saying; without it the only symptom is a throughput cliff. - "slots_reclaimed": self._slots_reclaimed, } @@ -310,9 +138,9 @@ def kv_connector_hosts_state_tier(kv_transfer_config) -> bool: The tier is not standalone: its bytes ride the KV connector's worker half and its completions ride that connector's `get_finished`. Against any other - backend the staging ring would hand out slots and never get one back, so - this gates whether the ring is installed at all -- and a false negative - (tier off) is much cheaper than a false positive (every slot leaks). + backend a load would be offered and never reported, parking the request + that took it -- so this gates whether the index is built at all, and a + false negative (tier off) is much cheaper than a false positive. A name check rather than a capability probe: this runs in the engine process at `BlockManager.__init__`, before any worker connector exists. @@ -328,42 +156,3 @@ def kv_connector_hosts_state_tier(kv_transfer_config) -> bool: for s in subs ) return name in _STATE_TIER_BACKENDS - - -def state_offload_staging_groups() -> int: - """K: staging *groups* to reserve for the tier. - - Groups, not entries: a sizing site multiplies by its `entries_per_req`, - because `state_entry_views` is indexed by group. One function rather than - two `os.environ` reads, because the arena and the pool both size themselves - from it and would otherwise disagree. - - This only sizes the ring; whether the tier runs follows the connector. Bad - depths warn rather than raise (model load): non-integer falls back to 1, - negative floors to 0. - """ - raw = os.environ.get("OFFLOAD_STATE_STAGING_GROUPS") - if raw is None: - return 1 - try: - depth = int(raw) - except ValueError: - # Warn rather than raise -- this runs during model load. Not silent - # either: a mistyped depth buys one group instead of twenty, with a - # starved ring 256 dropped spills later as the only other symptom. - logger.warning( - "state offload: invalid OFFLOAD_STATE_STAGING_GROUPS=%r; using 1", - raw, - ) - return 1 - if depth < 0: - # Same reasoning as the typo above, and a worse outcome: a negative - # depth turns the whole tier off, so the operator gets a - # healthy-looking server that never spills. - logger.warning( - "state offload: negative OFFLOAD_STATE_STAGING_GROUPS=%r disables " - "the tier entirely; using 0", - raw, - ) - return 0 - return depth diff --git a/atom/model_engine/state_pool.py b/atom/model_engine/state_pool.py index 144953c64d..262ec46f09 100644 --- a/atom/model_engine/state_pool.py +++ b/atom/model_engine/state_pool.py @@ -1,19 +1,12 @@ # SPDX-License-Identifier: MIT # Copyright (C) 2024-2026, Advanced Micro Devices, Inc. All rights reserved. -import logging from collections import deque from dataclasses import dataclass from heapq import heapify, heappop, heappush -from typing import TYPE_CHECKING from atom.model_engine.state_runtime import StateTransfer -if TYPE_CHECKING: - from atom.model_engine.state_offload import StateOffloadIndex - -logger = logging.getLogger("atom") - @dataclass(frozen=True) class SlotRetirement: @@ -107,15 +100,6 @@ def __init__( # Reverse map, for lazy eviction when a slot is handed out. -1 = the # slot carries no checkpoint. self.slot_hash: list[int] = [-1] * num_slots - # Backing store beneath the pool, or None. Not a member of - # `BlockManager.state_caches`: every one of those is a veto, and this - # tier does the opposite -- it makes more boundaries reachable. - self.offload: StateOffloadIndex | None = None - # (slot, staging_slot) pairs the next forward must copy out of HBM. - self._spill_copies: list[tuple[int, int]] = [] - # One-shot latch so the undrained-consumer diagnosis cannot flood the - # log from `pop()`, which is itself a hot path. - self._warned_spill_copies_undrained = False # Groups serving as a fork source in the in-flight step, counted by how # many requests fork off each. They stay off the free list until # `release_pins`, so the step that reads them cannot race a request that @@ -167,21 +151,12 @@ def holds_checkpoint(self, slot: int) -> bool: """Whether `slot` is on the free list *and* still worth something.""" return slot in self._free and self.slot_hash[slot] != -1 - def pop(self, *, spill: bool = True) -> int: + def pop(self) -> int: """Hand out a slot, evicting its checkpoint if it carried one. Vacant first, lowest index first; only when nothing is vacant does a checkpoint get spent, and then it is the least recently used one. - `spill=False` gives up the evicted checkpoint's bytes instead of - staging them, and exists for one caller: a slot about to be filled by - an offload *load*. The spill copy is issued by the next forward while - the load runs on the tier's own thread ahead of it, so the two would - race over the same slot and the spill would store the loaded state - under the evicted checkpoint's hash. Nothing downstream could tell. - Forgoing the spill costs one checkpoint -- exactly the pre-tier - behaviour, counted as `spills_forgone`. - The state twin of `BlockManager._pop_free_block`: slots sit on the free list carrying whatever the last owner left in them, and re-allocation — not the free — is the eviction event. @@ -198,12 +173,6 @@ def pop(self, *, spill: bool = True) -> int: slot = self._checkpointed.popleft() self._free.discard(slot) self.checkpoints_evicted += 1 - if spill: - # Before invalidate() clears slot_hash[slot]: the hash is the - # key the tier stores under. - self._spill(slot) - elif self.offload is not None and self.offload.enabled: - self.offload.spills_forgone += 1 # Whatever guess this slot carried is spent with it. Not folded into # `invalidate`, which also runs on the re-file inside `_index` and would # clear a mark that was only just made. @@ -487,9 +456,8 @@ def resumable_hit( return hit hbs = self.hash_block_size for i in range(hit - 1, -1, -1): - if not assume_checkpointed and not self._resumable_from( - block_hashes[i], (i + 1) * hbs - ): + checkpointed = block_hashes[i] in self.hash_to_slot + if not assume_checkpointed and not checkpointed: continue if seq.num_tokens - (i + 1) * hbs >= self.min_fork_tokens: return i + 1 @@ -598,88 +566,6 @@ def cancel_midstep(self, reservations: list[tuple]) -> None: for slot, _pos, _h in reservations: self.release(slot) - # ------------------------------- offload ------------------------------- # - def _resumable_from(self, h: int, tokens: int) -> bool: - """Whether a checkpoint for `h` can be *reached*, in HBM or beneath it. - - Reached, not merely indexed. `resumable_hit` scans right to left and - stops at the first boundary this accepts, so accepting one whose bytes - nothing can deliver does not cost a wasted lookup — it costs the whole - walk-back, hiding every shorter checkpoint still resident in HBM. - - HBM wins without a preference rule: both tiers are indexed by the same - hash, so the scan takes the rightmost boundary wherever it lives. - """ - if h in self.hash_to_slot: - return True - # Invariant, relied on to keep this hot path free of an `enabled` check: - # a disabled (depth-0) tier never reaches `request_spill`, because - # `_spill` returns on `not enabled` first, so `hashes` stays empty and - # the membership test is false anyway. Anything that adds to `hashes` - # outside the confirm-after-spill path must re-check `enabled` here. - return self.offload is not None and h in self.offload.hashes and tokens > 0 - - def _spill(self, slot: int) -> None: - """Stage `slot`'s bytes for the tier, if there is room. Never blocks. - - Beyond the staging depth the spill is simply dropped and - `checkpoints_evicted` still counts it — identical to the behaviour - without a tier, so there is no regression to reason about. - """ - if self.offload is None or not self.offload.enabled: - return - h = self.slot_hash[slot] - if h == -1: - return - staging = self.offload.request_spill(h, slot) - if staging >= 0: - self._spill_copies.append((slot, staging)) - # Exact leak detector, not a heuristic: a staging slot must be - # released before it can be handed out again, so at most - # `staging_depth` copies can be outstanding if the consumer drains - # every step. More is proof `take_spill_copies` is not being called. - if ( - len(self._spill_copies) > self.offload.staging_depth - and not self._warned_spill_copies_undrained - ): - self._warned_spill_copies_undrained = True - logger.warning( - "StateSlotPool has %d undrained spill copies for a staging " - "ring of depth %d. take_spill_copies() must be called every " - "step and each slot released with release_staging(); until " - "it is, the staging ring leaks and every spill is dropped.", - len(self._spill_copies), - self.offload.staging_depth, - ) - - def has_pending_spill(self, slot: int) -> bool: - """Whether a queued spill copy still has to read `slot`. - - The window a load must not enter. It closes when the copy is issued on - the compute stream (`take_spill_copies` -> the next forward), after - which only the staging entry is read and the slot is free to be - overwritten. Normally empty for a slot `pop(spill=False)` just handed - out; a copy left undrained by a batch that was never forwarded is the - case this catches. - """ - return any(src == slot for src, _staging in self._spill_copies) - - def take_spill_copies(self) -> list[tuple[int, int]]: - """`(slot, staging_slot)` pairs the next forward must copy. - - The consumer turns each into `relocate_state_slots([(slot, - num_slots + staging)])` -- the ring is K entries appended past the - pool's own range, so a staging slot needs no second addressing scheme. - - Not optional: whoever attaches a tier MUST drain this every step and - MUST call `release_staging(staging)` once the bytes are safe. Skip the - drain and the list grows without bound; skip the release and - `request_spill` returns -1 forever, silently evicting every checkpoint. - Both are warned about once, but a warning is a diagnosis, not a repair. - """ - out, self._spill_copies = self._spill_copies, [] - return out - # ---------------------------- checkpointing ---------------------------- # def checkpoint(self, seq, boundary_blocks: int, h: int) -> None: """Keep `seq`'s state as of this boundary, filed under hash `h`. @@ -732,24 +618,13 @@ def checkpoint_fates(self) -> dict[str, int]: holding its working set and the misses are somewhere else — most likely `checkpoints_orphaned`, whose KV prefix died first and which more slots cannot prevent. - - Folds in the tier's counters when one is attached, which is what makes - `spills_dropped` visible -- a ring dropping one spill in three never - trips the 256-consecutive starvation warning and would otherwise say - nothing. Keys are prefixed `state_offload_` because the aggregation is - by key across every state class. """ - fates = { + return { "checkpoints_kept": self.checkpoints_kept, "checkpoints_dropped": self.checkpoints_dropped, "checkpoints_evicted": self.checkpoints_evicted, "checkpoints_orphaned": self.checkpoints_orphaned, } - if self.offload is not None: - fates.update( - (f"state_offload_{k}", v) for k, v in self.offload.stats().items() - ) - return fates def occupancy(self) -> dict[str, int]: """How much of this pool is live, held by checkpoints, or spare. @@ -822,8 +697,6 @@ def unindex(self, h: int) -> None: slot = self.hash_to_slot.get(h, -1) if slot < 0: return - if self.offload is not None: - self._spill(slot) self.invalidate(slot) self.checkpoints_orphaned += 1 diff --git a/atom/model_engine/state_runtime.py b/atom/model_engine/state_runtime.py index e239cbe3ee..01995e3b7b 100644 --- a/atom/model_engine/state_runtime.py +++ b/atom/model_engine/state_runtime.py @@ -184,20 +184,11 @@ class StateMaintenanceOps: """State movement drained once before a model batch.""" relocations: tuple[tuple[int, int], ...] = () - # (src_group, dst_entry, staging_slot, hash) for checkpoints the state pool - # evicted to the offload tier. Carried here rather than beside this struct - # so the one consumer -- `CommonAttentionBuilder.build` -- is also the one - # place the spill-before-copy order is enforced. Empty unless - # the offload tier is on. - spills: tuple[tuple[int, int, int, int], ...] = () checkpoint_stores: tuple[CheckpointStoreOp, ...] = () checkpoint_restores: tuple[CheckpointRestoreOp, ...] = () @property def empty(self) -> bool: return not ( - self.relocations - or self.spills - or self.checkpoint_stores - or self.checkpoint_restores + self.relocations or self.checkpoint_stores or self.checkpoint_restores ) diff --git a/atom/model_ops/attentions/backends.py b/atom/model_ops/attentions/backends.py index 8b529eb6b3..9bbd1f776a 100644 --- a/atom/model_ops/attentions/backends.py +++ b/atom/model_ops/attentions/backends.py @@ -225,12 +225,12 @@ def checkpoint_image_bytes(self) -> int | None: """ return None - def state_entry_views(self, group: int) -> list["torch.Tensor"]: - """Contiguous views covering the whole of `group`'s per-request state. + def state_entry_views(self, slot: int) -> list["torch.Tensor"]: + """Contiguous views covering the whole of `slot`'s per-request state. The byte-level counterpart of `relocate_state_slots`, so the offload tier can read or write the same bytes. Every view must be contiguous -- - the Triton packer refuses a strided one -- so a class whose group is + the Triton packer refuses a strided one -- so a class whose slot is strided (GDN, slot on axis 1) returns one view per layer. """ raise NotImplementedError( @@ -238,27 +238,6 @@ def state_entry_views(self, group: int) -> list["torch.Tensor"]: "implement state_entry_views" ) - def _submit_state_spills(self, pairs) -> None: - """Hand the staged bytes to the offload tier's own executor. - - The connector is reached function-locally so these modules stay - importable without the offload extra. - """ - from atom.utils.forward_context import get_kvconnector - - connector = get_kvconnector() - tier = getattr(connector, "_state_tier", None) - if tier is None: - return - import torch - - # One event for the whole batch: every copy above was issued on this - # stream, so an event recorded now is after all of them. - ready = torch.cuda.Event() - ready.record(torch.cuda.current_stream()) - for _src, dst, slot, h in pairs: - tier.submit_spill(h, entry_index=dst, staging_slot=slot, ready_event=ready) - def relocate_state_slots(self, pairs: Sequence[tuple[int, int]]) -> None: """Move live state between contiguous Active Slots.""" raise NotImplementedError( @@ -638,19 +617,6 @@ def _attach_tbo_prefill_cpu_lens( def build(self, batch: ScheduledBatch, bs: int): # Run state maintenance on the compute stream before the forward. state_ops = batch.state_maintenance_ops - # Spills FIRST, and the order is load-bearing. A spill is filed under - # the group's *old* hash, and the same `pop()` then hands that group out - # as a checkpoint destination -- so one group is routinely both a spill - # SOURCE and a checkpoint DESTINATION in one batch. Issue the other - # copies first and the tier stores the new occupant's state under the - # evicted checkpoint's hash: valid-looking, someone else's, undetectable - # on any later load. The reverse cannot happen -- a spill's destination - # is past the pool's range and is never a copy source. - if state_ops.spills: - self.relocate_state_slots( - [(src, dst) for src, dst, _slot, _h in state_ops.spills] - ) - self._submit_state_spills(state_ops.spills) if state_ops.relocations: self.relocate_state_slots(state_ops.relocations) if state_ops.checkpoint_stores or state_ops.checkpoint_restores: diff --git a/atom/model_ops/attentions/deepseek_v4_attn.py b/atom/model_ops/attentions/deepseek_v4_attn.py index cb68b01b22..50d53277ef 100644 --- a/atom/model_ops/attentions/deepseek_v4_attn.py +++ b/atom/model_ops/attentions/deepseek_v4_attn.py @@ -840,23 +840,14 @@ def state_transfer(self) -> StateTransfer: ) return StateTransfer.copy(layout_id) - def state_entry_views(self, group: int) -> list[torch.Tensor]: - """One contiguous slice per plane — a V4 group is one slot per plane. + def state_entry_views(self, slot: int) -> list[torch.Tensor]: + """One contiguous slice per plane — a V4 slot is one row per plane. A slot holds the compressor state and then every layer's windows contiguously (see `relocate_state_slots`), so a plane's whole contribution is one range and no per-layer split is needed. - - `group` may address a staging-ring group past the last one the - BlockManager leases, and that resolves: `num_state_slots` (:863-866) - and the carve that sizes the planes (:1023, feeding `with_capacity` at - :1038) both read - `entries[STATE_SLOT_CLASS]`, the *allocation* count, which already - includes the K extra staging groups — admission is capped by a separate - count. So `_slot_views()`'s span and `physical_slot`'s bound both cover - the staging groups; `tests/test_state_entry_views.py` pins this. """ - return self._slot_views()[group] + return self._slot_views()[slot] def relocate_state_slots(self, pairs: Sequence[tuple[int, int]]) -> None: """Duplicate a request's whole per-request state: compressor + windows. @@ -1357,8 +1348,6 @@ def sub_pool_specs(self) -> list[SubPoolSpec]: tier's staging ring are both added by `state_pool` itself, which is the only reader of the two env vars that size them. """ - from atom.model_engine.state_offload import kv_connector_hosts_state_tier - geo = self.pool_geometry row_bytes = self.plane_row_bytes() return [ @@ -1367,9 +1356,6 @@ def sub_pool_specs(self) -> list[SubPoolSpec]: STATE_SLOT_CLASS, geo.slot_bytes(row_bytes), entries_per_req=1, - offload_hosted=kv_connector_hosts_state_tier( - getattr(self.model_runner.config, "kv_transfer_config", None) - ), ), ] diff --git a/atom/model_ops/attentions/gdn_attn.py b/atom/model_ops/attentions/gdn_attn.py index 2bf1694125..369dd1a9fb 100644 --- a/atom/model_ops/attentions/gdn_attn.py +++ b/atom/model_ops/attentions/gdn_attn.py @@ -823,24 +823,26 @@ def warmup_per_req_cache(self) -> None: ) launch_copy_descriptor(staging.copy_to_gpu(plan.num_spans), plan) - def state_entry_views(self, group: int) -> list[torch.Tensor]: - """One contiguous slice per (cache, layer) — the group's whole state. + def state_entry_views(self, slot: int) -> list[torch.Tensor]: + """One contiguous slice per (cache, layer) — the slot's whole state. - Both caches are layer-major with the slot on axis 1, so a group's rows + Both caches are layer-major with the slot on axis 1, so one slot's rows are strided and there is no single range covering them. Slicing per layer makes each piece contiguous, which is what the staging packer requires; `relocate_state_slots` keeps its own strided views because `_foreach_copy_` has no such constraint and one launch beats `LAYERS`. + + One slot, not a request's whole set: a checkpoint is exactly the + committed state (#2045), and the speculation scratch beside it is this + request's own and resumable by nobody. """ - span = 1 + self.num_spec - lo = group * span views = [] for cache in ( self.model_runner.mamba_k_cache, self.model_runner.mamba_v_cache, ): for layer in range(cache.shape[0]): - views.append(cache[layer, lo : lo + span]) + views.append(cache[layer, slot : slot + 1]) return views def relocate_state_slots(self, pairs: Sequence[tuple[int, int]]) -> None: diff --git a/atom/model_ops/attentions/sub_pool_spec.py b/atom/model_ops/attentions/sub_pool_spec.py index 9dcd6e4207..2ec950bdae 100644 --- a/atom/model_ops/attentions/sub_pool_spec.py +++ b/atom/model_ops/attentions/sub_pool_spec.py @@ -42,7 +42,7 @@ class — and the backend that declares the spec imports it from there. The one from __future__ import annotations -from dataclasses import dataclass, field, replace +from dataclasses import dataclass, replace from enum import Enum @@ -67,21 +67,11 @@ class SubPoolSpec: # per-request term. What it covers is the declaring backend's business. entries_per_req: int = 0 extra_entries: int = 0 - # STATE only. `extra_entries` above is capacity -- rows the pool hands out; - # these are the opposite, allocated and never leased. This is the offload tier's staging ring, which - # exists so `pop()` can copy an evicted checkpoint out and hand the original - # away immediately; a request given one of these rows would get a buffer the - # spill path writes into behind its back. Two fields rather than one because - # the two must ADD -- they shared a parameter once, and since the env - # override assigns, setting the headroom silently deleted the ring. - staging_entries: int = 0 def __post_init__(self): if self.pool is Pool.STATE and self.entries_per_req < 1: raise ValueError(f"{self.name}: STATE entries need entries_per_req >= 1") - if self.pool is Pool.PAGE and ( - self.entries_per_req or self.extra_entries or self.staging_entries - ): + if self.pool is Pool.PAGE and (self.entries_per_req or self.extra_entries): raise ValueError(f"{self.name}: PAGE entries are sized from the remainder") @@ -102,30 +92,9 @@ def state_pool( *, entries_per_req: int, extra_entries: int = 0, - offload_hosted: bool = False, ) -> SubPoolSpec: - """A per-request state entry class. - - `extra_entries` is the declaring backend's own cushion, and admissible. - The offload staging ring is added here instead, because it is driven by - `OFFLOAD_STATE_STAGING_GROUPS` and a second reader would let two sites - disagree about the same number. It is counted in *groups*, so it costs - `entries_per_req` rows each, is never admissible, and exists only when - `offload_hosted`: the tier follows the connector, so a deployment without - one must not pay for a ring nothing can drain. - """ - from atom.model_engine.state_offload import state_offload_staging_groups - - return SubPoolSpec( - Pool.STATE, - name, - entry_bytes, - entries_per_req, - extra_entries, - staging_entries=( - state_offload_staging_groups() * entries_per_req if offload_hosted else 0 - ), - ) + """A per-request state entry class.""" + return SubPoolSpec(Pool.STATE, name, entry_bytes, entries_per_req, extra_entries) @dataclass(frozen=True) @@ -141,12 +110,6 @@ class PoolPlan: reserved_bytes: dict[str, int] entries_per_req: dict[str, int] paged_class: str | None = None - # What allocation buys vs. what admission may lease, differing only by - # `staging_entries` -- the offload ring, inside the arena so - # `state_entry_views(num_groups + slot)` addresses it with no second scheme, - # and leased to nobody. A table rather than a subtraction at each call site, - # so a consumer picks a meaning by the name it reads. - admission_entries: dict[str, int] = field(default_factory=dict) @classmethod def empty(cls) -> PoolPlan: @@ -158,13 +121,7 @@ def empty(cls) -> PoolPlan: AttributeError — the runner installs this at construction time and replaces it in `get_num_blocks`. """ - return cls( - entries={}, - entry_bytes={}, - reserved_bytes={}, - entries_per_req={}, - admission_entries={}, - ) + return cls(entries={}, entry_bytes={}, reserved_bytes={}, entries_per_req={}) def with_paged_entries(self, count: int) -> PoolPlan: """Copy of the plan with the PAGE class resized to `count`. @@ -179,7 +136,6 @@ def with_paged_entries(self, count: int) -> PoolPlan: return replace( self, entries={**self.entries, self.paged_class: count}, - admission_entries={**self.admission_entries, self.paged_class: count}, reserved_bytes={**self.reserved_bytes, self.paged_class: count * bytes_per}, ) @@ -225,12 +181,10 @@ def merge_specs(specs: list[SubPoolSpec]) -> dict[str, SubPoolSpec]: prev.pool, prev.entries_per_req, prev.extra_entries, - prev.staging_entries, ) != ( spec.pool, spec.entries_per_req, spec.extra_entries, - spec.staging_entries, ): raise ValueError( f"entry class {spec.name!r} declared twice with different " @@ -242,7 +196,6 @@ def merge_specs(specs: list[SubPoolSpec]) -> dict[str, SubPoolSpec]: prev.entry_bytes + spec.entry_bytes, spec.entries_per_req, spec.extra_entries, - spec.staging_entries, ) return merged @@ -260,20 +213,14 @@ def plan_pools( """ merged = merge_specs(specs) entries: dict[str, int] = {} - admissible_entries: dict[str, int] = {} reserved: dict[str, int] = {} remaining = available_bytes state = {n: s for n, s in merged.items() if s.pool is Pool.STATE} for name, spec in state.items(): - # `extra_entries` is capacity the pool leases; `staging_entries` is the - # offload ring, allocated past the pool's group range and leased to - # nobody. See `SubPoolSpec` for why they are two fields. - admissible = max_num_seqs * spec.entries_per_req + spec.extra_entries - count = admissible + spec.staging_entries + count = max_num_seqs * spec.entries_per_req + spec.extra_entries cost = count * spec.entry_bytes entries[name], reserved[name] = count, cost - admissible_entries[name] = admissible remaining -= cost if state and remaining <= 0: raise InsufficientPoolBudget( @@ -289,7 +236,6 @@ def plan_pools( spec = merged[name] count = max(0, remaining // spec.entry_bytes) entries[name], reserved[name] = count, count * spec.entry_bytes - admissible_entries[name] = count return PoolPlan( entries=entries, @@ -297,5 +243,4 @@ def plan_pools( reserved_bytes=reserved, entries_per_req={n: s.entries_per_req for n, s in merged.items()}, paged_class=paged[0] if paged else None, - admission_entries=admissible_entries, ) diff --git a/docs/environment_variables.md b/docs/environment_variables.md index 26fb6e4f42..23a1f754bd 100644 --- a/docs/environment_variables.md +++ b/docs/environment_variables.md @@ -231,7 +231,6 @@ the tier off means turning the connector off. | Variable | Type | Default | Description | |----------|------|---------|-------------| -| **OFFLOAD_STATE_STAGING_GROUPS** | int | 1 | Spill staging ring depth in **groups** -- `state_pool()` multiplies by the class's `entries_per_req` to get rows. Allocated inside the state arena and never leased to a request. Raise it if `state_offload_spills_dropped` is a material fraction of `spills_requested`; set 0 on a deployment that will never offload, to reclaim the rows. | | **OFFLOAD_STATE_CPU_SIZE** | float (GiB) | 16 | A CPU pool for state entries, separate from the paged-KV pool. Sharing loses both ways: the KV write stream is several times the state volume so it evicts the checkpoints, and a stopped KV backend takes the tier down with it. 0 shares the paged-KV pool. | ## Profiling & debugging diff --git a/tests/test_block_manager.py b/tests/test_block_manager.py index e243b71143..54e63c295e 100644 --- a/tests/test_block_manager.py +++ b/tests/test_block_manager.py @@ -731,13 +731,12 @@ def test_state_offload_is_none_when_tier_is_off(block_manager): assert block_manager.state_offload is None -# ── the ring is installed only where something can drain it ─────────────── +# ── the index is built only where something can report to it ───────────── _OFFLOAD_KVC = {"kv_connector": "lmcache_offload", "kv_role": "offload"} def _bm_with_state_tier(monkeypatch, kv_transfer_config): - monkeypatch.setenv("OFFLOAD_STATE_STAGING_GROUPS", "2") cfg = MockConfig( enable_prefix_caching=True, kv_transfer_config=kv_transfer_config, @@ -750,28 +749,21 @@ def _bm_with_state_tier(monkeypatch, kv_transfer_config): def test_no_ring_is_installed_without_a_connector_that_hosts_the_tier( monkeypatch, caplog ): - """The silent-permanent failure. Nothing submits a spill without a hosting - connector -- `_poll_kv_transfer_progress` returns early and the forward - finds no `_state_tier` -- so an installed ring would hand out every slot - and never get one back. Refuse to install it.""" + """The silent-permanent failure. A load is resolved only by a worker + report, and without a hosting connector no report ever comes -- so an index + built here would offer loads that park their request forever. Refuse to + build it.""" with caplog.at_level(logging.WARNING, logger="atom"): bm = _bm_with_state_tier(monkeypatch, None) assert bm.state_offload is None - assert all(cache.offload is None for cache in bm.state_caches) - # An installed-but-inert ring is the thing that misleads, so drive the - # eviction path that would reserve a slot and prove none is: with the ring - # installed this same call yields (src, dst, staging, hash) and that - # staging slot never comes back. - bm.state.slot_hash[1] = 111 - bm.state._spill(1) - assert bm.state.take_spill_copies() == [] - assert bm.state_spills_for_batch() == [] - - -def test_a_connector_that_cannot_host_the_tier_gets_no_ring_either(monkeypatch, caplog): - """Same leak, one step subtler: moriio is a KV connector, so a plain - truthiness test on kv_transfer_config would install the ring, but only + + +def test_a_connector_that_cannot_host_the_tier_gets_no_index_either( + monkeypatch, caplog +): + """Same failure, one step subtler: moriio is a KV connector, so a plain + truthiness test on kv_transfer_config would build the index, but only lmcache_offload's worker half ever builds a `_state_tier`.""" with caplog.at_level(logging.WARNING, logger="atom"): bm = _bm_with_state_tier(monkeypatch, {"kv_connector": "moriio"}) @@ -779,14 +771,15 @@ def test_a_connector_that_cannot_host_the_tier_gets_no_ring_either(monkeypatch, assert bm.state_offload is None -def test_the_ring_is_installed_for_the_offload_connector(monkeypatch): +def test_the_index_is_built_for_the_offload_connector(monkeypatch): bm = _bm_with_state_tier(monkeypatch, _OFFLOAD_KVC) assert bm.state_offload is not None - assert bm.state_offload.staging_depth == 2 - assert all(cache.offload is bm.state_offload for cache in bm.state_caches) + # #2045 moved the checkpoint into the KV pool, so the tier no longer + # reaches into the slot pool at all -- that coupling was the staging ring. + assert not any(hasattr(cache, "offload") for cache in bm.state_caches) -def test_the_ring_is_installed_for_a_multi_that_lists_the_offload_backend( +def test_the_index_is_built_for_a_multi_that_lists_the_offload_backend( monkeypatch, ): bm = _bm_with_state_tier( @@ -799,7 +792,7 @@ def test_the_ring_is_installed_for_a_multi_that_lists_the_offload_backend( assert bm.state_offload is not None -def test_a_multi_without_the_offload_backend_gets_no_ring(monkeypatch): +def test_a_multi_without_the_offload_backend_gets_no_index(monkeypatch): bm = _bm_with_state_tier( monkeypatch, {"kv_connector": "multi", "connectors": [{"kv_connector": "moriio"}]}, diff --git a/tests/test_lmcache_offload_connector.py b/tests/test_lmcache_offload_connector.py index a8fb33bca6..e6d81b1491 100644 --- a/tests/test_lmcache_offload_connector.py +++ b/tests/test_lmcache_offload_connector.py @@ -57,7 +57,6 @@ from atom.kv_transfer.offload.hybrid.kimi_k3.connector import ( SAVE_STALL_SECONDS, STATE_INDEX_CHANNEL, - STATE_STAGING_CHANNEL, KimiK3OffloadScheduler, ) from atom.kv_transfer.offload.metadata import ( @@ -4650,7 +4649,6 @@ def _k3_scheduler() -> KimiK3OffloadScheduler: s._warned_save_stalled = False s._state_indexed = set() s._state_index_failed = set() - s._state_staging_released = set() s._active_load_operations = {} s._do_save = True s._max_pending_saves = 4 @@ -4822,20 +4820,22 @@ def test_state_loads_are_drained_into_the_metadata_exactly_once(monkeypatch): def test_the_two_state_channels_are_routed_and_drained(): """The tier reports over the generic completion channel rather than extra - `KVConnectorOutput` fields, so TP quorum is the aggregator's job.""" + `KVConnectorOutput` fields, so TP quorum is the aggregator's job. Failure + is its own channel value, not a missing report: quorum over + `indexed | failed` is failure-dominant, so a partial store resolves in the + same step instead of pinning the key forever.""" s = _k3_scheduler() for channel, op, ok in ( (STATE_INDEX_CHANNEL, 111, True), (STATE_INDEX_CHANNEL, 222, False), - (STATE_STAGING_CHANNEL, 3, True), ): assert s.connector_completion( SimpleNamespace(channel=channel, operation_id=op, succeeded=ok) ) - indexed, released, failed = s.take_state_reports() - assert (indexed, released, failed) == ({111}, {3}, {222}) - assert s.take_state_reports() == (set(), set(), set()) + indexed, failed = s.take_state_reports() + assert (indexed, failed) == ({111}, {222}) + assert s.take_state_reports() == (set(), set()) def test_no_more_saves_go_out_than_the_pool_can_afford_to_pin(): diff --git a/tests/test_multi_connector.py b/tests/test_multi_connector.py index abb35c3932..4daf141262 100644 --- a/tests/test_multi_connector.py +++ b/tests/test_multi_connector.py @@ -405,7 +405,7 @@ def test_no_sub_to_carry_a_state_load_is_reported_not_swallowed(): def test_the_composite_exposes_the_sub_connectors_state_tier(): - """`AttentionBackend._submit_state_spills` reads `_state_tier` off whatever + """The K3 store/load path reads `_state_tier` off whatever connector the forward context holds. Under `multi` that is this object, so without the re-export nothing is ever submitted and every slot leaks.""" tier = object() diff --git a/tests/test_sub_pool_spec.py b/tests/test_sub_pool_spec.py index 458f124652..ef7c8c44f7 100644 --- a/tests/test_sub_pool_spec.py +++ b/tests/test_sub_pool_spec.py @@ -43,52 +43,6 @@ def test_declared_extra_entries_are_preserved(self): spec = state_pool(ENTRY_STATE, 10, entries_per_req=1, extra_entries=64) assert spec.extra_entries == 64 - def test_the_staging_ring_is_sized_here_not_by_the_caller(self, monkeypatch): - """`state_pool` is the only reader of the ring size. - - A backend that sized the ring itself would be a second env read, and - the arena and the pool would size themselves from different answers if - one of them ever drifted -- the arena short exactly the rows the spill - path addresses. - """ - monkeypatch.setenv("OFFLOAD_STATE_STAGING_GROUPS", "2") - spec = state_pool(ENTRY_STATE, 10, entries_per_req=3, offload_hosted=True) - # Groups, not entries: a staging group is `entries_per_req` rows. - assert spec.staging_entries == 2 * 3 - assert spec.extra_entries == 0 - - def test_the_checkpoint_headroom_and_the_staging_ring_add(self, monkeypatch): - """Two fields, because the ring must not eat the backend's cushion. - - They shared one parameter once, and the ring overwrote it: the arena - then lost the rows `state_entry_views(num_groups + slot)` addresses, and - GDN's `cache[layer, lo : lo + span]` clamps rather than raising -- a - short segment, an `entry_bytes` mismatch, and a crash in the packer. - """ - monkeypatch.setenv("OFFLOAD_STATE_STAGING_GROUPS", "2") - spec = state_pool( - ENTRY_STATE, 10, entries_per_req=1, extra_entries=16, offload_hosted=True - ) - assert spec.extra_entries == 16 - assert spec.staging_entries == 2 - - def test_one_group_is_reserved_by_default(self, monkeypatch): - """The tier has no on/off flag of its own -- it follows the - `lmcache_offload` connector -- so the ring has to exist by default or - the arena would be short the rows the spill path addresses. A - deployment that will never offload sets the size to 0.""" - monkeypatch.delenv("OFFLOAD_STATE_STAGING_GROUPS", raising=False) - spec = state_pool(ENTRY_STATE, 10, entries_per_req=3, offload_hosted=True) - assert spec.staging_entries == 3 - - monkeypatch.setenv("OFFLOAD_STATE_STAGING_GROUPS", "0") - assert ( - state_pool( - ENTRY_STATE, 10, entries_per_req=3, offload_hosted=True - ).staging_entries - == 0 - ) - class TestMergeSpecs: def test_same_name_sums_entry_bytes(self): @@ -427,108 +381,3 @@ def test_draft_bytes_raise_the_per_block_cost_not_the_pool_count(self): ) assert set(with_draft.entries) == {ENTRY_KV} assert with_draft.entries[ENTRY_KV] == target_only.entries[ENTRY_KV] // 2 - - -def test_staging_entries_are_allocated_but_not_admissible(): - """`staging_entries` buys rows the pool must not lease out. Allocation - sizes tensors from `entries`; admission divides `admission_entries`.""" - plan = plan_pools( - [ - page_pool(1000), - SubPoolSpec( - Pool.STATE, ENTRY_STATE, 500, entries_per_req=2, staging_entries=3 - ), - ], - available_bytes=1_000_000, - max_num_seqs=8, - ) - assert plan.entries[ENTRY_STATE] == 8 * 2 + 3 - assert plan.admission_entries[ENTRY_STATE] == 8 * 2 - # The reservation covers every allocated row, staging included. - assert plan.reserved_bytes[ENTRY_STATE] == (8 * 2 + 3) * 500 - - -def test_checkpoint_headroom_is_allocated_and_admissible(): - """The other cushion, and the opposite answer. - - `extra_entries` is checkpoint capacity: groups the state pool is *meant* to - hand out -- the backend's declared cushion. Withholding it - from admission -- which is what the staging split did before these two - fields were separated -- leaves `num_groups` at `max_num_seqs` and the - feature does nothing at all. - """ - plan = plan_pools( - [ - page_pool(1000), - state_pool(ENTRY_STATE, 500, entries_per_req=2, extra_entries=3), - ], - available_bytes=1_000_000, - max_num_seqs=8, - ) - assert plan.entries[ENTRY_STATE] == 8 * 2 + 3 - assert plan.admission_entries[ENTRY_STATE] == 8 * 2 + 3 - - -def test_admission_entries_defaults_to_entries_without_extras(): - plan = plan_pools( - [page_pool(1000), state_pool(ENTRY_STATE, 500, entries_per_req=2)], - available_bytes=1_000_000, - max_num_seqs=8, - ) - assert plan.admission_entries[ENTRY_STATE] == plan.entries[ENTRY_STATE] == 16 - - -def test_staging_groups_are_sized_by_multiplicity(monkeypatch): - """A staging *group* is `entries_per_req` rows. Sizing the ring in bare - entries would run the last staging group off the end of the tensor.""" - span, k = 3, 2 # GDN: span = 1 + num_spec - monkeypatch.setenv("OFFLOAD_STATE_STAGING_GROUPS", str(k)) - plan = plan_pools( - [ - page_pool(1000), - state_pool(ENTRY_STATE, 500, entries_per_req=span, offload_hosted=True), - ], - available_bytes=1_000_000, - max_num_seqs=8, - ) - num_groups = plan.admission_entries[ENTRY_STATE] // span - assert num_groups == 8 - # The highest staging group's last row must still be inside the tensor. - assert (num_groups + k - 1) * span + span <= plan.entries[ENTRY_STATE] - - -def test_the_ring_sits_past_the_headroom_not_on_top_of_it(monkeypatch): - """Both cushions at once: the ring must start past every admissible group. - - `state_spills_for_batch` addresses a staging entry as `num_groups + slot`, - and `num_groups` counts the headroom. A ring sized without it would - overlap groups the pool leases out -- a spill would read a live request's - state and store it under an evicted checkpoint's hash. - """ - span, k, headroom = 2, 2, 5 - monkeypatch.setenv("OFFLOAD_STATE_STAGING_GROUPS", str(k)) - plan = plan_pools( - [ - page_pool(1000), - state_pool( - ENTRY_STATE, - 500, - entries_per_req=span, - extra_entries=headroom * span, - offload_hosted=True, - ), - ], - available_bytes=1_000_000, - max_num_seqs=8, - ) - num_groups = plan.admission_entries[ENTRY_STATE] // span - assert num_groups == 8 + headroom - assert (num_groups + k - 1) * span + span <= plan.entries[ENTRY_STATE] - - -def test_paged_class_admission_equals_entries(): - """PAGE has no per-request reservation, so the two counts coincide and - `with_paged_entries` must keep them in step.""" - plan = plan_pools([page_pool(1000)], available_bytes=10_500, max_num_seqs=8) - assert plan.admission_entries[ENTRY_KV] == plan.entries[ENTRY_KV] == 10 - assert plan.with_paged_entries(4).admission_entries[ENTRY_KV] == 4 From bc7b61cb12ca4357882377a8d43a939ae9363912 Mon Sep 17 00:00:00 2001 From: zejunchen-zejun Date: Thu, 27 Aug 2026 16:13:47 +0800 Subject: [PATCH 004/151] lmcache(k3): one CPU pool for KV and state, not two Phase 3a. Deletes the tier's separate `atom-state-{rank}` LMCache engine, its `OFFLOAD_STATE_CPU_SIZE` knob, and its `cache_policy = "FIFO"` override. The codec now binds the same `storage_manager` the KV leg uses. --- what changed - `_state_storage_manager()` is gone (~50 lines). `bind_storage_manager` is handed `self._engine.storage_manager` directly. The `gib <= 0` branch that already did this was the whole implementation; what goes is the `> 0` half. - `OFFLOAD_STATE_CPU_SIZE` removed, from the code and from `docs/environment_variables.md`. - `cache_policy = "FIFO"` removed; state inherits the KV pool's LRU. --- why the two-pool argument does not hold It rested on "the KV write stream is several times the state volume, so it evicts the checkpoints". Measured against the code rather than assumed: `page_unit_bytes = num_MLA_layers * block_size * entry * itemsize = 442,368` with `--block-size 128` pinned by the recipe, so KV costs 3,456 B/token and a 53.6 MiB state image is worth ~16k tokens of it -- 13% of a 117k-token prompt's KV. (Read the MLA layer count the other way, as a quarter of 93 layers, and it is 3.3%. The conclusion is the same either way, which is why it did not need settling first.) And the byte ratio is not what decides an LRU eviction anyway. A request writes its ~457 KV chunks and its one state object inside the same prefill window, so both enter the LRU at the same position and cool at the same rate. State is retired ALONGSIDE its own KV -- which is the behaviour we want, because a boundary whose KV is gone is worthless. Two pools cost what one does not: two eviction policies that must drift, while a joint boundary needs both legs to survive together. #2045 made this same argument one tier up -- state moved out of its reserved slots into the KV pool, 81.85% -> 93.60% -- and `OFFLOAD_STATE_CPU_SIZE` is the same reserved capacity one tier down, so keeping it repeats the thing #2045 disproved. `LMCACHE_MAX_LOCAL_CPU_SIZE` is now the one size to tune. --- the cost, stated rather than hidden `cache_policy` is per-engine, so one pool means one policy for both legs. That is a real loss of freedom and it is currently free, because LRU is right for both. The FIFO override's premise was "a state entry is written once and read once"; #2045's own numbers refute it (~4,808 resumes over ~1,508 checkpoints is ~3 reads each), and re-access is bursty -- branches and retries on one boundary, then superseded -- which is LRU's home ground. The genuine want, if it shows up, is "stickier than LRU gives it", which FIFO does not provide either; the lever then is touching the state key on a KV hit, not splitting the pool back apart. Nothing yet measures whether that is needed. The counter to add with the store path (Phase 3d) is `joint_boundary_state_miss`: a boundary whose KV leg landed and whose state leg missed. Zero means the two pools never needed separating. Verified: pytest tests/test_lmcache_offload_connector.py -> 186 passed. black clean; ruff unchanged. Co-Authored-By: Claude Opus 5 (1M context) --- .../offload/hybrid/kimi_k3/connector.py | 84 +++++++------------ docs/environment_variables.md | 1 - 2 files changed, 28 insertions(+), 57 deletions(-) diff --git a/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py b/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py index 362c44f046..c8a4f693f8 100644 --- a/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py +++ b/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py @@ -50,7 +50,6 @@ class KimiK3OffloadConnector(DenseOffloadConnector): def __init__(self, config) -> None: super().__init__(config) self._state_tier = None - self._state_engine = None # Inert until a request has both legs, which only a joint boundary # produces; costs one dict lookup per report otherwise. self._joint_park = _JointPark() @@ -119,69 +118,42 @@ def _build_state_tier(self, transfer_tensors) -> None: world_size=world, worker_id=rank, ) - codec.bind_storage_manager(self._state_storage_manager(cfg, meta, rank)) + # ONE pool, shared with paged KV. A separate `atom-state-{rank}` engine + # used to be built here, on the argument that the KV write stream would + # evict the checkpoints. It does not hold up: state is 3-13% of a + # boundary's bytes, not a rounding error, and -- the part that decides + # it -- a request writes its KV chunks and its one state object inside + # the same prefill window, so they enter LMCache's LRU together and + # cool at the same rate. State is retired ALONGSIDE its own KV, which + # is what we want: a boundary whose KV is gone is worthless anyway. + # + # Two pools cost what one does not: independent eviction policies that + # must drift, and a joint boundary needs BOTH legs to survive together. + # #2045 made the same argument one tier up and won with it (state moved + # out of its reserved slots into the KV pool, 81.85% -> 93.60%). + # + # What one pool costs is a single `cache_policy` for both legs. That is + # real, and currently free: LRU is right for both. The FIFO override + # this used to set rested on "a state entry is written once and read + # once", which #2045's own numbers refute -- ~4,808 resumes over ~1,508 + # checkpoints is ~3 reads each, and bursty re-access is LRU's home + # ground. + # + # `OFFLOAD_STATE_CPU_SIZE` is gone with it. #2045 deleted the HBM-side + # reservation for exactly this reason (`extra_entries=0`, 1.7 GiB + # returned) -- keeping the same knob one tier down repeats the thing it + # disproved. `LMCACHE_MAX_LOCAL_CPU_SIZE` is the one size to tune. + codec.bind_storage_manager(self._engine.storage_manager) # No index here: StateOffloadIndex lives in the engine process; both # directions report and the engine applies. self._state_tier = StateOffloadTier(codec) logger.info( - "kimi_k3 offload: state tier up, entry=%.2f MiB rank=%d", + "kimi_k3 offload: state tier up, entry=%.2f MiB rank=%d, " + "sharing the paged-KV CPU pool", entry_bytes / (1 << 20), rank, ) - def _state_storage_manager(self, cfg, meta, rank: int): - """The tier's own CPU pool, not the paged-KV one. - - Sharing loses both ways: the KV write stream is several times the - state volume so it evicts the checkpoints, and a stopped KV backend - takes the tier down with it. - """ - raw = os.environ.get("OFFLOAD_STATE_CPU_SIZE", "16") - try: - gib = float(raw) - except ValueError: - logger.warning("kimi_k3 offload: invalid OFFLOAD_STATE_CPU_SIZE=%r", raw) - gib = 16.0 - if gib <= 0: - logger.info("kimi_k3 offload: state tier shares the paged-KV CPU pool.") - return self._engine.storage_manager - try: - from lmcache.v1.cache_engine import LMCacheEngineBuilder - from lmcache.v1.memory_management import MemoryFormat - - cfg.max_local_cpu_size = gib - # FIFO, while the KV pool keeps LMCache's LRU. A state entry is - # written once and read once (turn N+1 resumes off turn N's anchor - # and nobody looks again), so what decides its worth is age against - # the think time. LRU would promote an entry already consumed over - # one still waiting. Tied to the ladder being off; with rungs shared - # across conversations, entries stop being one-shot and LRU wins. - cfg.cache_policy = "FIFO" - engine = LMCacheEngineBuilder.get_or_create( - f"atom-state-{rank}", - cfg, - meta, - self._engine.gpu_connector, - lambda t, s: None, - lambda o, s: o, - ) - engine.fmt = MemoryFormat.KV_2LTD - engine.post_init() - self._state_engine = engine - logger.info( - "kimi_k3 offload: state pool %.0f GiB/rank (paged KV keeps %s)", - gib, - os.environ.get("LMCACHE_MAX_LOCAL_CPU_SIZE", ""), - ) - return engine.storage_manager - except Exception: - logger.warning( - "kimi_k3 offload: no separate state pool; sharing the paged-KV " - "pool, where the KV write stream will evict checkpoints.", - exc_info=True, - ) - return self._engine.storage_manager - # -- per-step ---------------------------------------------------------- def start_load_kv(self, metadata) -> None: if isinstance(metadata, LMCacheOffloadMetadata): diff --git a/docs/environment_variables.md b/docs/environment_variables.md index 23a1f754bd..bf08805b78 100644 --- a/docs/environment_variables.md +++ b/docs/environment_variables.md @@ -231,7 +231,6 @@ the tier off means turning the connector off. | Variable | Type | Default | Description | |----------|------|---------|-------------| -| **OFFLOAD_STATE_CPU_SIZE** | float (GiB) | 16 | A CPU pool for state entries, separate from the paged-KV pool. Sharing loses both ways: the KV write stream is several times the state volume so it evicts the checkpoints, and a stopped KV backend takes the tier down with it. 0 shares the paged-KV pool. | ## Profiling & debugging From b9829bae3a25cd6d300ec88ad31452c3d06bf9e2 Mon Sep 17 00:00:00 2001 From: zejunchen-zejun Date: Thu, 27 Aug 2026 16:16:42 +0800 Subject: [PATCH 005/151] lmcache(k3): bind the state key to its layout, and refuse a misfit read Phase 3b. `StateByteCodec.key()` stops being the bare ATOM hash; it becomes `xxh64(hash || layout_id)`. `get()` gains a size check on what comes back. --- what changed - `StateByteCodec.__init__` takes a required `layout_id` and refuses an empty one. `_build_state_tier` reads it off `state_runtime.checkpoint_spec` and declines to build the tier at all when it is absent, rather than falling back to an unqualified key. - `key(h)` hashes `h` and the layout id together with `xxh64`. - `get(h, ...)` measures the returned `MemoryObj` and, on a size other than `entry_bytes`, discharges its reference, counts `_misfit_reads`, warns, and returns a miss. - `_object_bytes()` reads `get_size()` with the tensor as fallback, and returns None when neither answers. --- why the key needed both *Build safety.* The same prefix hash names a completely different image under a different `num_spec`, TP size, or conv/ssm dtype. A size mismatch throws at `entry_bytes`, but **the same size with a different order or meaning reads back silently wrong state** -- and a request resumed onto wrong state produces wrong output with nothing raised anywhere. #2045 already encodes all of it in `layout_id` (layers / conv shape+dtype / ssm shape+dtype / order / tp / spec / carry) and enforces it HBM-side in `_validate_paged_state_op`. This is the CPU side of the same check, and it was missing. *Namespace separation.* `CacheEngineKey` is `(model_name, world_size, worker_id, chunk_hash, dtype)` -- no field says what an entry IS. KV chunk keys carry `ChunkedTokenDatabase`'s chunk hash and state keys carry ATOM's chained block hash: two different hash functions writing plain integers into one space. Until Phase 3a they at least lived in different pools; now they share one, and the only thing keeping them apart was accidental -- this side hard-codes `torch.uint8` while KV carries the KV dtype. `MemoryFormat` is not available as a discriminator either: the tier's own comment records that its `KV_2LTD` is inert, chosen only to pass the allocator's `raise ValueError`. Folding the layout in makes the two spaces disjoint by construction, since no KV key can carry a K3 layout id. `xxh64` rather than Python's `hash((h, layout_id))`: `hash` of a str is salted per process, so a restart would silently orphan every entry the previous run wrote -- a cache that quietly starts cold and never says so. `xxh64` is what the block hashes themselves use. --- why the size check, if the key already prevents it It is unreachable, and that is the point: reaching it means two things collided in the shared pool, and unpacking a colliding object writes another entry's bytes over a request's live state. Degrading to a miss costs one recompute; not checking costs silent wrong output. `_misfit_reads` is the counter that says the key stopped doing its job. None from `_object_bytes` means "cannot measure", never "size 0" -- refusing what cannot be measured would turn an unknown into a guaranteed miss. --- tests Five, in `tests/test_lmcache_offload_connector.py`, on a `CacheEngineKey` fixture rather than the full lmcache stub (these are about two keys being equal or not, which the real dataclass answers by value): a layout change makes a different key; the layout does not collapse distinct hashes; the key survives a restart; a state key cannot equal a raw-hash KV key; a wrong-sized hit is a miss with its reference discharged; an object that will not report a size is still loaded. Verified: pytest tests/ (less tests/plugin and the two msgpack/aiter collection errors that fail on #2045 too) -> 3911 passed, zero failures. black clean; ruff 36, unchanged. Co-Authored-By: Claude Opus 5 (1M context) --- .../offload/hybrid/kimi_k3/connector.py | 18 ++- .../offload/hybrid/kimi_k3/state_object.py | 84 +++++++++++- tests/test_lmcache_offload_connector.py | 128 ++++++++++++++++++ 3 files changed, 223 insertions(+), 7 deletions(-) diff --git a/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py b/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py index c8a4f693f8..668b2c327d 100644 --- a/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py +++ b/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py @@ -82,6 +82,20 @@ def _build_state_tier(self, transfer_tensors) -> None: "kimi_k3 offload: no attention backend published; state tier off." ) return + # The geometry the bytes are written under, folded into every key so a + # build that changed any of it cannot read another's images. Read from + # the runtime rather than recomputed, so there is one owner of the + # string and the HBM and CPU sides cannot disagree about it. + spec = getattr(getattr(backend, "model_runner", None), "state_runtime", None) + spec = getattr(spec, "checkpoint_spec", None) + layout_id = getattr(spec, "layout_id", None) + if not layout_id: + logger.warning( + "kimi_k3 offload: no checkpoint layout id published; state tier " + "off. Without it a build that changed the state geometry could " + "read another's images back as valid." + ) + return try: views = backend.state_entry_views(0) entry_bytes = sum(int(v.numel()) * v.element_size() for v in views) @@ -117,6 +131,7 @@ def _build_state_tier(self, transfer_tensors) -> None: model_name=meta.model_name, world_size=world, worker_id=rank, + layout_id=layout_id, ) # ONE pool, shared with paged KV. A separate `atom-state-{rank}` engine # used to be built here, on the argument that the KV write stream would @@ -149,9 +164,10 @@ def _build_state_tier(self, transfer_tensors) -> None: self._state_tier = StateOffloadTier(codec) logger.info( "kimi_k3 offload: state tier up, entry=%.2f MiB rank=%d, " - "sharing the paged-KV CPU pool", + "sharing the paged-KV CPU pool, layout=%s", entry_bytes / (1 << 20), rank, + layout_id, ) # -- per-step ---------------------------------------------------------- diff --git a/atom/kv_transfer/offload/hybrid/kimi_k3/state_object.py b/atom/kv_transfer/offload/hybrid/kimi_k3/state_object.py index a77614fa72..de1c79da70 100644 --- a/atom/kv_transfer/offload/hybrid/kimi_k3/state_object.py +++ b/atom/kv_transfer/offload/hybrid/kimi_k3/state_object.py @@ -41,15 +41,20 @@ def __init__( model_name: str, world_size: int, worker_id: int, + layout_id: str, ) -> None: self._backend = backend self._staged = staged self.entry_bytes = int(entry_bytes) if self.entry_bytes <= 0: raise ValueError("state entry bytes must be > 0") + if not isinstance(layout_id, str) or not layout_id: + raise ValueError("a state entry key needs a non-empty layout id") self._model_name = model_name self._world_size = int(world_size) self._worker_id = int(worker_id) + self._layout_id = layout_id + self._misfit_reads = 0 self._storage = None # Never hard-code a size: V4 keeps six compressor fields across # n_csa/n_hca layers plus an optional window; GDN keeps @@ -65,19 +70,44 @@ def bind_storage_manager(self, storage_manager) -> None: self._storage = storage_manager def key(self, h: int): - """The key carries ATOM's hash unmodified. - - `StateSlotPool._resumable_from` looks up the same integer in HBM and - in this tier, so hashing or stringifying it here would make the two - branches ask different questions. `dtype` is part of the key's identity. + """ATOM's hash, bound to the geometry the bytes were written under. + + **Two jobs, and the second one is why this is not the bare hash.** + + *Build safety.* The same prefix hash maps to a completely different + image under a different `num_spec`, TP size, or conv/ssm dtype. A size + mismatch would be caught by `entry_bytes`; the same size with a + different order or meaning reads back silently wrong state, and a + request resumed onto wrong state produces wrong output with nothing + raised anywhere. #2045 already encodes all of it in `layout_id` + (layers / conv shape+dtype / ssm shape+dtype / order / tp / spec / + carry) and enforces it HBM-side in `_validate_paged_state_op`; this is + the CPU side of the same check. + + *Namespace separation.* `CacheEngineKey` has no field saying what an + entry IS -- KV chunk keys come from `ChunkedTokenDatabase`'s chunk hash + and state keys from ATOM's chained block hash, both plain integers in + one space. Since Phase 3a they also share one pool. They were separated + only accidentally, by this side hard-coding `torch.uint8` while KV + carries the KV dtype. Folding `layout_id` in makes them disjoint by + construction: no KV key can ever carry a K3 layout id. + + Not a plain `hash((h, layout_id))`: Python's `hash` of a str is salted + per process (`PYTHONHASHSEED`), so a restart would silently orphan every + entry written by the previous run. `xxh64` is what the block hashes + themselves use. """ + import xxhash from lmcache.utils import CacheEngineKey + digest = xxhash.xxh64() + digest.update(int(h).to_bytes(8, "little", signed=True)) + digest.update(self._layout_id.encode()) return CacheEngineKey( self._model_name, self._world_size, self._worker_id, - int(h), + digest.intdigest(), torch.uint8, ) @@ -112,12 +142,54 @@ def get(self, h: int, entry_index: int) -> bool: obj = self._storage.get(self.key(h)) if obj is None: return False + size = self._object_bytes(obj) + if size is not None and size != self.entry_bytes: + # Unreachable while `layout_id` is in the key, which is the point: + # a hit of the wrong size means two things collided in the shared + # pool, and unpacking it would write another entry's bytes over a + # request's live state. Degrade to a miss -- the caller disowns the + # boundary and recomputes -- and count it, because a nonzero count + # means the key is no longer doing its job. + self._misfit_reads += 1 + logger.warning( + "state offload: hash %d came back %d bytes, expected %d; " + "treating as a miss (misfit_reads=%d)", + h, + size, + self.entry_bytes, + self._misfit_reads, + ) + obj.ref_count_down() + return False try: self._staged.unpack(obj, self._backend.state_entry_views(entry_index)) finally: obj.ref_count_down() return True + @staticmethod + def _object_bytes(obj) -> int | None: + """Bytes in a `MemoryObj`, or None when it will not say. + + `get_size()` is the documented accessor; the tensor is the fallback for + the buffer-backed objects that have no size of their own. None means + "cannot check", never "size 0" -- refusing a load we cannot measure + would turn an unknown into a guaranteed miss. + """ + get_size = getattr(obj, "get_size", None) + if callable(get_size): + try: + return int(get_size()) + except Exception: + pass + tensor = getattr(obj, "tensor", None) + if tensor is not None: + try: + return int(tensor.numel()) * tensor.element_size() + except Exception: + pass + return None + def contains(self, h: int) -> bool: """Ask storage, never a local set -- a set goes stale the moment LMCache's LRU evicts. `contains` answers with a location name or None, diff --git a/tests/test_lmcache_offload_connector.py b/tests/test_lmcache_offload_connector.py index e6d81b1491..b26cadaaad 100644 --- a/tests/test_lmcache_offload_connector.py +++ b/tests/test_lmcache_offload_connector.py @@ -4853,3 +4853,131 @@ def test_no_more_saves_go_out_than_the_pool_can_afford_to_pin(): assert s._may_emit_save() is True s._save_stalled = True assert s._may_emit_save() is False + + +# ── the state key: build safety and namespace separation ────────────────── + + +class _FakeStorage: + """The two `storage_manager` calls the codec makes, over a plain dict.""" + + def __init__(self) -> None: + self.objects: dict = {} + + def batched_put(self, keys, objs) -> None: + for k, o in zip(keys, objs, strict=True): + self.objects[k] = o + + def get(self, key): + return self.objects.get(key) + + def contains(self, key): + return key in self.objects + + +@pytest.fixture +def lmcache_key(monkeypatch): + """`CacheEngineKey` alone -- the whole `lmcache` stub is more than these need. + + A dataclass rather than a stub with identity semantics: the tests below are + about two keys being equal or not, which is exactly what the real class + answers by value. + """ + import dataclasses + + @dataclasses.dataclass(frozen=True) + class CacheEngineKey: + model_name: str + world_size: int + worker_id: int + chunk_hash: int + dtype: object + + utils_module = types.ModuleType("lmcache.utils") + utils_module.CacheEngineKey = CacheEngineKey + lmcache_module = types.ModuleType("lmcache") + lmcache_module.__path__ = [] + lmcache_module.utils = utils_module + monkeypatch.setitem(sys.modules, "lmcache", lmcache_module) + monkeypatch.setitem(sys.modules, "lmcache.utils", utils_module) + return CacheEngineKey + + +def _codec(layout_id: str, storage=None): + from atom.kv_transfer.offload.hybrid.kimi_k3.state_object import StateByteCodec + + codec = StateByteCodec( + backend=None, + staged=None, + entry_bytes=1024, + model_name="k3", + world_size=8, + worker_id=0, + layout_id=layout_id, + ) + codec.bind_storage_manager(storage if storage is not None else _FakeStorage()) + return codec + + +def test_the_same_hash_under_a_different_layout_is_a_different_key(lmcache_key): + """A build that changed the state geometry must not read another's images. + + The failure this prevents is silent: same size, different order or meaning, + unpacked over a request's live state with nothing raised anywhere. + """ + a = _codec("layers=69;spec=1;tp=8") + b = _codec("layers=69;spec=2;tp=8") # one speculated token more + assert a.key(12345) != b.key(12345) + # And the layout alone does not collapse distinct hashes. + assert a.key(12345) != a.key(12346) + + +def test_the_key_survives_a_process_restart(lmcache_key): + """`hash()` of a str is salted per process, so using it would orphan every + entry the previous run wrote -- a cache that silently starts cold.""" + assert _codec("L").key(7) == _codec("L").key(7) + + +def test_a_state_key_cannot_collide_with_a_kv_chunk_key(lmcache_key): + """Since one pool holds both (Phase 3a), the key is the only separation. + + `CacheEngineKey` has no field saying what an entry is: KV keys carry + `ChunkedTokenDatabase`'s chunk hash and state keys ATOM's block hash, both + plain integers. Folding the layout in is what makes them disjoint -- no KV + key can carry a K3 layout id -- rather than merely unlikely to coincide. + """ + codec = _codec("kimi_k3;layers=69") + # A KV chunk key is the raw hash; the state key for the same hash is not. + assert codec.key(999).chunk_hash != 999 + + +def test_a_hit_of_the_wrong_size_is_a_miss_not_garbage(lmcache_key, caplog): + """Unreachable while the layout is in the key, which is the point. + + A hit of the wrong size means two things collided in the shared pool, and + unpacking it would write another entry's bytes over a request's live state. + """ + import logging + + storage = _FakeStorage() + codec = _codec("L", storage) + released = [] + storage.objects[codec.key(5)] = SimpleNamespace( + get_size=lambda: 64, # not the 1024 entry_bytes + ref_count_down=lambda: released.append(True), + ) + + with caplog.at_level(logging.WARNING, logger="atom"): + assert codec.get(5, 0) is False + assert released == [True], "the misfit object's reference must be discharged" + assert codec._misfit_reads == 1 + assert any("expected 1024" in r.getMessage() for r in caplog.records) + + +def test_an_object_that_will_not_report_its_size_is_still_loaded(): + """None means "cannot check", never "size 0". Refusing what we cannot + measure would turn an unknown into a guaranteed miss.""" + from atom.kv_transfer.offload.hybrid.kimi_k3.state_object import StateByteCodec + + assert StateByteCodec._object_bytes(SimpleNamespace()) is None + assert StateByteCodec._object_bytes(SimpleNamespace(get_size=lambda: 32)) == 32 From c039fc1db46a955f8b0c3a87f30166c7915221e9 Mon Sep 17 00:00:00 2001 From: zejunchen-zejun Date: Thu, 27 Aug 2026 16:20:13 +0800 Subject: [PATCH 006/151] k3: name a checkpoint's PAGE units as tensors, for the CPU tier's packer Phase 3c. Adds `_KimiMLAGDNCommon.page_unit_views(unit_ids)` -- the tensor-view counterpart of #2045's `_page_unit_regions`. No behaviour changes; nothing calls it until Phase 3d. --- what changed - `page_unit_views(unit_ids) -> list[torch.Tensor]`, one view per (unit, MLA row), unit major and row minor. - Seven tests in `tests/test_kda_checkpoint_slot_copy.py`, on the harness #2045 already built for `_page_unit_bases`. --- why a second way to name the same bytes `_page_unit_regions` hands the Triton descriptor raw int64 addresses, which is what a D2D copy the GPU issues wants. The LMCache staging packer takes `list[torch.Tensor]` -- `fused_pack_chunk_major` is already a fully parameterised gather over `segment_ptrs[] + segment_block_bytes[]`, so it needs no kernel change, only its segments named the way it names them. This is the ONLY new seam the tier needs. A load writes the Active Slot directly and reuses `state_entry_views`, so the asymmetry is deliberate: the save gathers from units, the load scatters to a slot, and the two compose because #2045's `plan_segmented_copy` intersects two *ordered byte streams* -- the slot's ranges and the units' regions are two cuts of the same logical image, so a blob gathered in one order is byte-identical to one gathered in the other. --- the two traps, both asserted rather than assumed *Order is the contract.* Unit major, row minor -- the same ravel as `_page_unit_bases`, which is the order `_checkpoint_copy_plan` builds the destination stream in. The packer indexes segments positionally, so this order IS the blob layout. Self-consistency would be enough for one build; what makes it a contract across builds is `layout_id` in the key (Phase 3b), so a build that reordered cannot read another's blob. *`block_ratio`.* `unit_ids` carries **logical** block ids while `kv_cache` is shaped in **physical** ones, and K3's ratio is 128. Flattening the two block axes together and indexing by `unit * block_size` is the conversion -- the same arithmetic `_page_unit_bases` does in addresses, so the two cannot drift into disagreeing about which bytes a unit owns. The range check is against the *logical* count for the same reason: the physical count is `ratio` times larger in rows, so checking the wrong axis admits ids that read past the end. Its test computes the bound from the tensor rather than from a constant, because taking the harness's label would let the very confusion under test pass unnoticed. `_page_unit_regions` is called first for its side effect: the contiguity and granularity checks live there, and they have to run before any view is handed out rather than after. --- verification The unit tests skip on a non-GPU runner (the module imports aiter at load), so the geometry was checked directly by extracting the four methods and comparing `page_unit_views` against `_page_unit_bases` byte for byte: 9 segments for 3 units x 3 rows, identical `data_ptr()`s, lengths equal to `_page_unit_stream_sizes`, every view contiguous, no two units sharing a byte, an out-of-range id refused, and a granularity mismatch refused before any view is produced. pytest tests/ (less tests/plugin and the two msgpack/aiter collection errors that fail on #2045 too) -> 3911 passed, zero failures. black clean; ruff 36, unchanged. Co-Authored-By: Claude Opus 5 (1M context) --- .../model_ops/attentions/kimi_mla_gdn_attn.py | 52 ++++++++++++ tests/test_kda_checkpoint_slot_copy.py | 84 +++++++++++++++++++ 2 files changed, 136 insertions(+) diff --git a/atom/model_ops/attentions/kimi_mla_gdn_attn.py b/atom/model_ops/attentions/kimi_mla_gdn_attn.py index e6e6cd4fb7..95a9c370c0 100644 --- a/atom/model_ops/attentions/kimi_mla_gdn_attn.py +++ b/atom/model_ops/attentions/kimi_mla_gdn_attn.py @@ -228,6 +228,58 @@ def _page_unit_stream_sizes(self, units: int) -> np.ndarray: """Bytes in each destination segment of an image of `units` units.""" return np.tile(self._page_unit_regions()[1], units) + def page_unit_views(self, unit_ids: Sequence[int]) -> list[torch.Tensor]: + """Tensor-view counterpart of `_page_unit_regions`, for the CPU tier. + + `_page_unit_regions` hands the Triton descriptor raw int64 addresses, + which is right for a D2D copy the GPU issues. The LMCache staging + packer takes `list[torch.Tensor]` instead, so the same bytes have to be + nameable as views -- this is that seam, and it is the ONLY new one the + tier needs: a load writes the Active Slot directly and reuses + `state_entry_views`. + + **Order is the contract.** Unit major, row minor -- the same ravel as + `_page_unit_bases`, which is the order `_checkpoint_copy_plan` builds + the destination stream in. Save and load both go through this one + function, so self-consistency would be enough on its own; the reason it + still has to match is that a blob written by one build must not be read + by another that ordered it differently, which is what the `layout_id` in + `StateByteCodec.key` is for. + + **The `block_ratio` trap, same as `_page_unit_regions`.** `unit_ids` + carries *logical* block ids while `kv_cache` is shaped in *physical* + ones, and K3's ratio is 128. Flattening the two block axes together and + indexing by `unit * block_size` is what converts them: it is the same + arithmetic `_page_unit_bases` does in addresses, so the two cannot drift + into disagreeing about which bytes a unit owns. + + Every view is contiguous, which the packer requires: a row of a + contiguous pool is contiguous, and a slice along its leading axis stays + so. `_page_unit_regions` is called first for its side effect -- it is + where the contiguity and granularity checks live. + """ + _base, _stride = self._page_unit_regions() + cache = self.model_runner.kv_cache + rows, entry = cache.shape[0], cache.shape[3] + block = self.model_runner.block_size + # Logical blocks the pool holds, which is what a unit id indexes. + # Range-checked against THIS count, not the physical one: the physical + # count is `block_ratio` times smaller, so checking it would pass ids + # that address past the end of the tensor. + logical_blocks = (cache.shape[1] * cache.shape[2]) // block + planes = [cache[row].view(-1, entry) for row in range(rows)] + views: list[torch.Tensor] = [] + for unit in unit_ids: + unit = int(unit) + if not 0 <= unit < logical_blocks: + raise IndexError( + f"PAGE unit {unit} outside the pool's {logical_blocks} " + "logical blocks" + ) + lo = unit * block + views.extend(plane[lo : lo + block] for plane in planes) + return views + def build_kv_cache_tensor(self, layer_id: int, module): from atom.config import KVCacheTensor diff --git a/tests/test_kda_checkpoint_slot_copy.py b/tests/test_kda_checkpoint_slot_copy.py index 40483f1e04..20f1d0bb26 100644 --- a/tests/test_kda_checkpoint_slot_copy.py +++ b/tests/test_kda_checkpoint_slot_copy.py @@ -442,3 +442,87 @@ def test_a_restore_does_not_read_past_the_image(self): for row in range(self.ROWS): tail = pool[row].reshape(-1)[len(units) * self.LOGICAL_BS * self.ENTRY :] assert int(tail.sum()) == 0 + + +class TestPageUnitViewsNameTheSameBytes: + """`page_unit_views` is the CPU tier's seam onto the PAGE units. + + `_page_unit_regions` gives the Triton descriptor raw addresses; the LMCache + packer wants tensors. Two ways to name one set of bytes is exactly the kind + of pair that drifts, so every test here asks both and compares. + """ + + H = TestPageUnitAddressesAreArithmetic + + def build(self, **kw): + stub, cache = self.H.build(self.H(), **kw) + stub.page_unit_views = K3._KimiMLAGDNCommon.page_unit_views.__get__( + stub, type(stub) + ) + return stub, cache + + def test_a_view_starts_exactly_where_the_address_says(self): + stub, _ = self.build() + units = [4, 0, 2] # not ascending: ids are not a range + views = stub.page_unit_views(units) + addrs = stub._page_unit_bases([units])[0] + assert len(views) == len(addrs) == self.H.ROWS * len(units) + assert [v.data_ptr() for v in views] == addrs.tolist() + + def test_a_view_is_exactly_one_region_long(self): + stub, _ = self.build() + views = stub.page_unit_views([1]) + sizes = stub._page_unit_stream_sizes(1) + assert [v.numel() * v.element_size() for v in views] == sizes.tolist() + + def test_the_order_is_unit_major_row_minor(self): + """The packer indexes segments positionally, so this order IS the blob + layout -- and `_checkpoint_copy_plan` built the HBM stream the same + way. A build that ordered it differently must not read another's blob, + which is what `layout_id` in the key covers.""" + stub, _ = self.build() + units = [3, 1] + base, stride = stub._page_unit_regions() + want = [base[r] + u * stride[r] for u in units for r in range(self.H.ROWS)] + assert [v.data_ptr() for v in stub.page_unit_views(units)] == want + + def test_every_view_is_contiguous(self): + """The Triton packer refuses a strided segment.""" + stub, _ = self.build() + assert all(v.is_contiguous() for v in stub.page_unit_views([0, 2, 4])) + + def test_a_unit_id_past_the_logical_count_is_refused(self): + """The `block_ratio` trap in its most dangerous form. + + The bound is computed from the tensor rather than taken from the + harness's label, because that is the whole point: `unit_ids` counts + logical blocks and the tensor is shaped in physical ones, so a check + written against the wrong axis admits ids that read past the end. + """ + stub, cache = self.build() + logical = (cache.shape[1] * cache.shape[2]) // self.H.LOGICAL_BS + assert logical != cache.shape[1], "the harness must exercise a ratio != 1" + assert stub.page_unit_views([logical - 1]) # the last valid id + with pytest.raises(IndexError, match="logical blocks"): + stub.page_unit_views([logical]) + + def test_the_views_do_not_reach_into_a_neighbouring_unit(self): + """Off-by-one here scrambles someone else's checkpoint rather than + running off the end, so the bound is asserted from the bytes.""" + stub, cache = self.build() + flat = cache.reshape(-1) + seen = set() + logical = (cache.shape[1] * cache.shape[2]) // self.H.LOGICAL_BS + for unit in range(logical): + for v in stub.page_unit_views([unit]): + lo = (v.data_ptr() - flat.data_ptr()) // flat.element_size() + span = set(range(lo, lo + v.numel())) + assert not (span & seen), f"unit {unit} overlaps an earlier one" + seen |= span + + def test_the_granularity_check_runs_before_any_view_is_handed_out(self): + stub, _ = self.build( + block_size=TestPageUnitAddressesAreArithmetic.LOGICAL_BS * 2 + ) + with pytest.raises(RuntimeError, match="granularity"): + stub.page_unit_views([0]) From 9d270b228524be141272ba97cdb78bc15868fe73 Mon Sep 17 00:00:00 2001 From: zejunchen-zejun Date: Thu, 27 Aug 2026 16:30:20 +0800 Subject: [PATCH 007/151] lmcache(k3): store a checkpoint when it becomes READY, not when it is evicted Phase 3d, the last of Phase 3. This is what turns the tier back on for saves: a checkpoint that reaches READY is offered to the CPU tier, and the whole engine->worker->engine round trip is wired. Loads are still off (Phase 4). --- what changed `page_unit_checkpoint.py` - `_offload_ready`: checkpoints nominated at the READY transition. NOT pinned. - `take_offload_stores(max_inflight)`: hands `(hash, unit_ids)` over and takes the pin at that moment, up to the cap. - `settle_offload_store(hash)` / `reclaim_stale_offload_pins(timeout_s)` / `offload_pins_reclaimed`, exposed on the coordinator and reported in `checkpoint_fates`. - `PageUnitCheckpointStore(..., offload_sink=)` -- False nominates nothing. `block_manager.py` - `take_state_stores` / `settle_state_store(hash, ok)` / `reclaim_stale_state_store_pins`. The switch (`kv_connector_hosts_state_tier`) moves above the coordinator's construction, which now needs it. `scheduler.py` - `_publish_state_stores()`, beside `_publish_state_loads()` and on the same two call sites plus the idle path. - `_offload_max_pending_saves()`, reading the KV leg's own `OFFLOAD_MAX_PENDING_SAVES`. - The report drain settles the pin on both outcomes and runs the reconciler. `kimi_k3/connector.py` -- `enqueue_state_stores`, `meta.state_stores`, `_start_state_stores`, and `_state_store_failed_locally` for stores a worker with no tier cannot even attempt. `kimi_k3/state_tier.py` -- `submit_store(h, unit_ids)`, no `ready_event`. `kimi_k3/state_object.py` -- `put(h, unit_ids)` gathers `page_unit_views`; `get(h, slot)` still scatters into `state_entry_views`; `puts_refused`. `metadata.py` / `types.py` -- `state_stores` declared as work. --- why READY, and why the pin is not taken there Three candidate trigger points and only one works. `begin_store` is too early: the scatter has not ridden a batch, so the units do not hold the image yet. `_evict` / `unindex` are too late -- they fire when the pool wants those units *now*, so a multi-millisecond D2H started there holds 127 of the most-wanted blocks at the worst possible moment. READY is when the bytes first exist and when the record is least wanted: it has just entered the LRU at the cold end. The pin is a separate question, and getting it wrong broke a real invariant -- `test_a_ready_unpinned_checkpoint_is_available_to_live_kv` caught the first draft. A pin lives in this process while the copy runs in the worker, so it spans several scheduler passes. Pinning at READY would make EVERY checkpoint un-evictable for that window, which is exactly what #2045's admission argument forbids. So READY only *nominates*: a nominee stays unpinned and spendable, and `take_offload_stores` pins only the few actually in flight. A checkpoint the pool needed more than the tier did simply loses its copy, which is the right price for never making the pool wait on the CPU. The cap is `OFFLOAD_MAX_PENDING_SAVES`, shared with the KV leg rather than given a knob of its own: a KV save and a state store both hold bytes out of the same pool while they run, so the depth is one question about one resource, and two numbers would let an operator raise one and unknowingly double the other. --- write volume, which is the objection this trigger invites Storing every checkpoint rather than only evicted ones is more traffic, and it is affordable: per request state adds 53.6 MiB on top of ~404 MB of KV, **13%** on a path already in flight (~12 MB/s against the KV leg's ~90 MB/s over a 3600s run). What it buys is the trigger being at the cheapest possible moment and dedup for free -- `begin_store` already refuses a hash it holds or has pending, so one hash is offered once. --- unindex during a copy does not cancel it `unindex` means the boundary's KV block was spent, so the image is unreachable in HBM from here on. That is precisely when the CPU copy is worth having, so the pin holds the units to the end of the D2H and `_release_record` runs after. The CPU copy outliving the HBM one is the entire point of the tier. --- the lost report, and why 20 lines is enough `reclaim_stale_offload_pins` takes the same window as `_reconcile_stalled_deferred_saves`, derived from LMCache's own pin timeout for the same reason. Recovery is total: a leaked pin breaks no `BlockPool` invariant and leaves no half-released state, so zeroing the count restores the record exactly -- which is why this needs a counter and a timer rather than any repair logic. `offload_pins_reclaimed` reaches `checkpoint_fates`, because otherwise the symptom is a pool that has quietly shrunk. --- still off Loads. Nothing calls `_request_state_load` for K3 yet: `_attach_state_slots` returns from its PAGE branch before reaching `_tier_can_serve`, and the `begin_restore` raise above it has to become an issue-a-load branch first (Phase 4). `StateOffloadIndex.hashes` now fills, so the tier is half live: it writes, and nothing reads yet. --- tests 11 new in `tests/test_page_unit_checkpoint.py` -- nomination does not pin, the hand-off does, a nominee stays spendable and may be lost, no sink nominates nothing, offered once, the cap makes the rest wait rather than drop, a waiting nominee stays evictable, both outcomes release, a report for a hash never sent is a no-op, `unindex` mid-copy holds the units to the end, and a lost report is reclaimed with the record left unbroken. Two more in `tests/test_lmcache_offload_connector.py`: stores drain exactly once, and a step whose only work is a store still reaches the worker. Verified: pytest tests/ (less tests/plugin and the two msgpack/aiter collection errors that fail on #2045 too) -> 3924 passed, zero failures. black clean; ruff 36, unchanged. Co-Authored-By: Claude Opus 5 (1M context) --- atom/kv_transfer/disaggregation/types.py | 1 + .../offload/hybrid/kimi_k3/connector.py | 46 +++++- .../offload/hybrid/kimi_k3/state_object.py | 41 +++-- .../offload/hybrid/kimi_k3/state_tier.py | 27 ++- atom/kv_transfer/offload/metadata.py | 5 + atom/model_engine/block_manager.py | 50 +++++- atom/model_engine/engine_core.py | 7 +- atom/model_engine/page_unit_checkpoint.py | 156 +++++++++++++++++- atom/model_engine/scheduler.py | 89 +++++++++- tests/test_lmcache_offload_connector.py | 29 ++++ tests/test_page_unit_checkpoint.py | 131 +++++++++++++++ 11 files changed, 538 insertions(+), 44 deletions(-) diff --git a/atom/kv_transfer/disaggregation/types.py b/atom/kv_transfer/disaggregation/types.py index 37894cb664..74d629bf9c 100644 --- a/atom/kv_transfer/disaggregation/types.py +++ b/atom/kv_transfer/disaggregation/types.py @@ -369,6 +369,7 @@ def completion_req_key(completion: ConnectorCompletionId) -> str: _DUCK_TYPED_WORK_FIELDS = ( "requests", "state_loads", + "state_stores", "lookup_requests_in_step", "reqs_to_recv", "reqs_to_save", diff --git a/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py b/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py index 668b2c327d..fbd3043e4f 100644 --- a/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py +++ b/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py @@ -175,6 +175,7 @@ def start_load_kv(self, metadata) -> None: if isinstance(metadata, LMCacheOffloadMetadata): self._arm_joint_loads(metadata) self._start_state_loads(metadata) + self._start_state_stores(metadata) super().start_load_kv(metadata) def _arm_joint_loads(self, metadata) -> None: @@ -214,6 +215,33 @@ def _start_state_loads(self, metadata) -> None: for req_id, h, group in loads: self._state_tier.submit_load(req_id, int(h), int(group)) + def _start_state_stores(self, metadata) -> None: + """Hand this step's ready checkpoints to the tier's executor. + + No producer fence and no staging copy. The source is the checkpoint's + PAGE units, which #2045 reserves out of the KV pool and the engine pins + for the duration -- so nothing on the compute stream is writing them + and the packer gathers straight from where they sit. + + A store with no tier is reported failed rather than dropped: the engine + is holding those units pinned against a report, and silence would leave + them to the reconciler's full timeout. + """ + stores = getattr(metadata, "state_stores", None) + if not stores: + return + if self._state_tier is None: + logger.warning( + "kimi_k3 offload: %d state store(s) with no tier; failing them " + "so the engine releases their units now rather than on timeout.", + len(stores), + ) + for h, _units in stores: + self._state_store_failed_locally.add(int(h)) + return + for h, unit_ids in stores: + self._state_tier.submit_store(int(h), tuple(int(u) for u in unit_ids)) + def _fail_state_loads(self, loads) -> None: for req_id, _h, _group in loads: self._joint_park.settle_state(req_id, False) @@ -278,6 +306,7 @@ def __init__(self, config) -> None: # A state load shares no shape with a KV transfer -- no token ids, no # block ids, no chunking -- only the park/report lifecycle. self._pending_state_loads: list[tuple] = [] + self._pending_state_stores: list[tuple] = [] # A finished request whose save is queued keeps its blocks pinned # (`should_defer_free`), so the queue depth is also how much of the pool # a slow backend can hold. Same knob and default the DSV4 layout bounds @@ -292,6 +321,9 @@ def __init__(self, config) -> None: # Channel reports drained by the engine each step. self._state_indexed: set[int] = set() self._state_index_failed: set[int] = set() + # Stores this worker could not even attempt (no tier). Reported as + # failures so the engine releases their pinned units immediately. + self._state_store_failed_locally: set[int] = set() # -- state load queue -------------------------------------------------- def enqueue_state_loads(self, loads) -> bool: @@ -300,6 +332,12 @@ def enqueue_state_loads(self, loads) -> bool: self._pending_state_loads.extend(loads) return True + def enqueue_state_stores(self, stores) -> bool: + if not stores: + return False + self._pending_state_stores.extend(stores) + return True + def build_connector_meta(self) -> LMCacheOffloadMetadata: self._refresh_save_stall() meta = super().build_connector_meta() @@ -307,6 +345,11 @@ def build_connector_meta(self) -> LMCacheOffloadMetadata: # into a group the first transfer is already filling. meta.state_loads = self._pending_state_loads self._pending_state_loads = [] + # Drained for the same reason: a second submission would store the same + # image twice, and the second report would unpin a record the first + # already released. + meta.state_stores = self._pending_state_stores + self._pending_state_stores = [] return meta def _may_emit_save(self) -> bool: @@ -450,7 +493,8 @@ def connector_completion(self, completion) -> bool: def take_state_reports(self) -> tuple[set[int], set[int]]: """Drain this step's tier store reports for the engine-side index.""" indexed = self._state_indexed - failed = self._state_index_failed + failed = self._state_index_failed | self._state_store_failed_locally self._state_indexed = set() self._state_index_failed = set() + self._state_store_failed_locally = set() return indexed, failed diff --git a/atom/kv_transfer/offload/hybrid/kimi_k3/state_object.py b/atom/kv_transfer/offload/hybrid/kimi_k3/state_object.py index de1c79da70..7dbc80aeb7 100644 --- a/atom/kv_transfer/offload/hybrid/kimi_k3/state_object.py +++ b/atom/kv_transfer/offload/hybrid/kimi_k3/state_object.py @@ -27,9 +27,11 @@ class StateByteCodec: multi-plane state layouts cannot be expressed in LMCache's token-major model at all. - `entry_index` is not always a pool group -- on the spill path it is a - staging-ring slot, on the load path a real group. Both index the same - `state_entry_views` space. + The two directions read different things, and that is not an oversight: + a store gathers the checkpoint's PAGE units (`page_unit_views`) because + #2045 is where the image lives, while a load scatters into the Active Slot + the resuming forward will read (`state_entry_views`). The blob is the same + ordered byte stream either way. """ def __init__( @@ -55,6 +57,7 @@ def __init__( self._worker_id = int(worker_id) self._layout_id = layout_id self._misfit_reads = 0 + self.puts_refused = 0 self._storage = None # Never hard-code a size: V4 keeps six compressor fields across # n_csa/n_hca layers plus an optional window; GDN keeps @@ -111,24 +114,36 @@ def key(self, h: int): torch.uint8, ) - def put(self, h: int, entry_index: int) -> bool: - """Spill one entry. Returns False when nothing was stored. - - A refusal is not an error: `_allocate` returns None under memory - pressure with nothing evictable, and spilling is best-effort -- the - pool counted the eviction either way, so the cost is one later miss. + def put(self, h: int, unit_ids) -> bool: + """Store one checkpoint image. False when nothing was stored. + + The source is the checkpoint's PAGE units, not an Active Slot: under + #2045 that is where the image lives, and `page_unit_views` names those + bytes as the contiguous tensors the packer takes. The two directions + are deliberately asymmetric -- a load writes the slot (`get`) -- and + they compose because #2045's copy plan intersects two *ordered byte + streams*, so a blob gathered in unit order is byte-identical to one + gathered in slot order. + + A refusal is not an error. `_allocate` returns None under CPU memory + pressure with nothing evictable, and a 53.6 MiB request is refused + sooner than a KV chunk's 884 KB, so the state leg is the first to feel + a full pool. Best-effort by design: the cost is one later miss, and + `puts_refused` is what makes it visible instead of a silent hit-rate + drift. """ if self._storage is None: return False obj = self._allocate(self.entry_bytes) if obj is None: + self.puts_refused += 1 return False - self._staged.pack(self._backend.state_entry_views(entry_index), obj) + self._staged.pack(self._backend.page_unit_views(unit_ids), obj) self._storage.batched_put([self.key(h)], [obj]) return True - def get(self, h: int, entry_index: int) -> bool: - """Load one entry back into `entry_index`. False on a miss. + def get(self, h: int, slot: int) -> bool: + """Load one image back into Active Slot `slot`. False on a miss. The reference must be discharged here: `get_blocking` does a `ref_count_up()` on the caller's behalf, and without the matching down @@ -162,7 +177,7 @@ def get(self, h: int, entry_index: int) -> bool: obj.ref_count_down() return False try: - self._staged.unpack(obj, self._backend.state_entry_views(entry_index)) + self._staged.unpack(obj, self._backend.state_entry_views(slot)) finally: obj.ref_count_down() return True diff --git a/atom/kv_transfer/offload/hybrid/kimi_k3/state_tier.py b/atom/kv_transfer/offload/hybrid/kimi_k3/state_tier.py index 74cac6c19c..cd24d1da68 100644 --- a/atom/kv_transfer/offload/hybrid/kimi_k3/state_tier.py +++ b/atom/kv_transfer/offload/hybrid/kimi_k3/state_tier.py @@ -58,20 +58,17 @@ def _discard(f): fut.add_done_callback(_discard) - def submit_store(self, h: int, slot: int) -> None: - """Pack `slot`'s state under `h` and hand it to LMCache. - - No `ready_event`. The staging ring is gone (#2045 makes the source hold - still on its own), so nothing on the compute stream is producing these - bytes for us to fence against -- `StagedTransfer.pack` gathers straight - from the source on its own stream. - - NOTE: nothing calls this yet. Phase 2 removed the ring that used to - drive it and Phase 3 gives it its new caller, whose source is the - checkpoint's PAGE units rather than a slot. Kept rather than deleted so - the two halves of the change stay reviewable apart. + def submit_store(self, h: int, unit_ids) -> None: + """Pack the checkpoint image in `unit_ids` under `h`, for LMCache. + + No `ready_event`, unlike the old spill. The staging ring is gone: the + units are reserved out of the KV pool and pinned by the engine for the + length of this transfer, so nothing on the compute stream is producing + or overwriting them and `StagedTransfer.pack` gathers straight from + where they sit -- one copy instead of two, and none of it on the + forward's stream. """ - self._register(self._executor.submit(self._do_store, h, slot)) + self._register(self._executor.submit(self._do_store, h, unit_ids)) def submit_load(self, req_id: str, h: int, slot: int) -> None: """Fetch `h` into pool slot `slot` for the parked request `req_id`. @@ -106,10 +103,10 @@ def get_finished(self) -> tuple[set[str], set[str]]: def shutdown(self) -> None: self._executor.shutdown(wait=True) - def _do_store(self, h: int, slot: int) -> None: + def _do_store(self, h: int, unit_ids) -> None: stored = False try: - stored = bool(self.codec.put(h, slot)) + stored = bool(self.codec.put(h, unit_ids)) except Exception: # a store is best effort by design logger.warning("state offload: store of hash %d failed", h, exc_info=True) with self._lock: diff --git a/atom/kv_transfer/offload/metadata.py b/atom/kv_transfer/offload/metadata.py index 071ec0d7f6..844b7c9830 100644 --- a/atom/kv_transfer/offload/metadata.py +++ b/atom/kv_transfer/offload/metadata.py @@ -165,6 +165,7 @@ class LMCacheOffloadMetadata(ConnectorMetadata): "requests", "lookup_requests_in_step", "state_loads", + "state_stores", ) def __init__(self) -> None: @@ -177,6 +178,10 @@ def __init__(self) -> None: # token ids, no block ids, no chunking -- only the park/report # lifecycle, which is why it rides the metadata rather than the batch. self.state_loads: list[tuple] = [] + # (state_hash, unit_ids) for checkpoints leaving HBM for the CPU tier. + # Not keyed by request: by the time a store lands its request is long + # gone, and the hash is all the report carries back. + self.state_stores: list[tuple] = [] def add_request(self, meta: LMCacheReqMeta) -> None: self.requests.append(meta) diff --git a/atom/model_engine/block_manager.py b/atom/model_engine/block_manager.py index e3aa6f6feb..772b5e1072 100644 --- a/atom/model_engine/block_manager.py +++ b/atom/model_engine/block_manager.py @@ -157,6 +157,15 @@ def __init__( "the only checkpoint placement." ) checkpoint_spec = state_runtime.checkpoint_spec + # Read here rather than beside `state_offload` below, because the + # coordinator has to know at construction whether anything can carry a + # store: without a sink it must nominate nothing, since a pin nobody + # releases holds a whole image out of the pool forever. + from atom.model_engine.state_offload import kv_connector_hosts_state_tier + + kv_offload_enabled = kv_connector_hosts_state_tier( + getattr(config, "kv_transfer_config", None) + ) self.paged_state_checkpoints: PagedStateCheckpointCoordinator | None = None if checkpoint_spec is not None: enabled = self.enable_prefix_caching and self.num_state_slots > 0 @@ -164,6 +173,7 @@ def __init__( self.kv, checkpoint_spec, enabled=enabled, + offload_sink=kv_offload_enabled, ) # The rolling state class: per-request slots plus a content index over # the free ones. A checkpoint IS a free slot whose content is still @@ -220,10 +230,7 @@ def __init__( # Class names already warned about in `state_checkpoint_fates`. See # there for why the warning latches. self._warned_no_checkpoint_fates: set[str] = set() - from atom.model_engine.state_offload import ( - StateOffloadIndex, - kv_connector_hosts_state_tier, - ) + from atom.model_engine.state_offload import StateOffloadIndex # Joint boundaries, split by what the state leg cost: `hbm` forked a # resident checkpoint, `tier` paid an entry-sized H2D and a park. Almost @@ -250,9 +257,6 @@ def __init__( "joint KV load needs it and stays off", exc_info=True, ) - kv_offload_enabled = kv_connector_hosts_state_tier( - getattr(config, "kv_transfer_config", None) - ) self.state_offload: StateOffloadIndex | None = None # (req_id, hash, target_group) admitted this pass and not yet handed to # the connector. Kept here rather than in the index because the slot @@ -1021,6 +1025,38 @@ def settle_state_load(self, req_id, ok: bool) -> None: if slot is not None: self.state.release(slot) + def take_state_stores(self, max_inflight: int) -> list[tuple]: + """`(hash, unit_ids)` for checkpoints to hand the CPU tier this pass. + + Pins each as it hands it over; `settle_state_store` releases. Empty + without a coordinator (the fork backends), which is correct rather than + a gap: a fork checkpoint is a state slot, and #2045 is what moved K3's + into the KV pool where a set of units can be read without being + rescued first. + """ + if self.paged_state_checkpoints is None or self.state_offload is None: + return [] + return self.paged_state_checkpoints.take_offload_stores(max_inflight) + + def settle_state_store(self, prefix_hash: int, ok: bool) -> None: + """One store reported. Release the units, and index the hash if it landed. + + Success and failure release the pin identically -- it existed to keep + the bytes still during the copy, and the copy is over either way. + Only the indexing differs, because only a hash whose bytes are really + there may be voted for. + """ + if self.paged_state_checkpoints is not None: + self.paged_state_checkpoints.settle_offload_store(int(prefix_hash)) + if ok and self.state_offload is not None: + self.state_offload.note_stored(int(prefix_hash)) + + def reclaim_stale_state_store_pins(self, timeout_s: float) -> int: + """Release store pins whose report never came. See the store.""" + if self.paged_state_checkpoints is None: + return 0 + return self.paged_state_checkpoints.reclaim_stale_offload_pins(timeout_s) + def take_state_loads(self) -> list[tuple]: """`(req_id, hash, target_slot)` for loads admitted since the last call. Drained once per pass by the scheduler, which hands them to the diff --git a/atom/model_engine/engine_core.py b/atom/model_engine/engine_core.py index 5bca4b0b48..bb8d7578cb 100644 --- a/atom/model_engine/engine_core.py +++ b/atom/model_engine/engine_core.py @@ -532,9 +532,10 @@ def _dispatch_idle_offload_work(self) -> None: # getattr for the same reason as `kv_connector` above: this path is # reached with scheduler doubles that implement only the connector # surface. - publish = getattr(self.scheduler, "_publish_state_loads", None) - if publish is not None: - publish() + for name in ("_publish_state_loads", "_publish_state_stores"): + publish = getattr(self.scheduler, name, None) + if publish is not None: + publish() meta = connector.build_connector_meta() if not connector_metadata_has_work(meta): return diff --git a/atom/model_engine/page_unit_checkpoint.py b/atom/model_engine/page_unit_checkpoint.py index 91b56c2ba1..e5cb9e2178 100644 --- a/atom/model_engine/page_unit_checkpoint.py +++ b/atom/model_engine/page_unit_checkpoint.py @@ -5,7 +5,8 @@ from __future__ import annotations -from collections import OrderedDict +from collections import OrderedDict, deque +from time import monotonic from collections.abc import Iterator, Mapping from dataclasses import dataclass @@ -114,9 +115,15 @@ def __init__( self, pool: BlockPool, spec: PagedStateCheckpointSpec, + offload_sink: bool = False, ): self.pool = pool self.spec = spec + # Whether anything downstream can carry a store. False leaves the + # offload queue permanently empty and takes no pins, which is what a + # deployment with no CPU tier must cost: a pin nobody releases would + # hold 127 blocks per checkpoint out of the pool forever. + self._offload_sink = offload_sink self.hash_to_checkpoint: dict[int, int] = {} self.records: dict[int, CheckpointRecord] = {} self._pending_by_hash: dict[int, int] = {} @@ -126,6 +133,18 @@ def __init__( self._inflight_restores: list[int] = [] self._next_checkpoint_id = 0 self.evictions = 0 + # Checkpoints that reached READY and have not been offered to the CPU + # tier yet. Candidates only: NOT pinned, so one still counts as free + # space and `_next_victim` may spend it. The pin is taken later, in + # `take_offload_stores`, and only for the few actually in flight. + self._offload_ready: deque[int] = deque() + # prefix_hash -> (checkpoint_id, monotonic when the pin was taken). + # Keyed by hash because that is all the worker's report carries -- by + # the time a store lands its request is long gone. At most one live + # record per hash, which `begin_store` enforces by refusing a hash it + # already holds or has pending. + self._offload_pins: dict[int, tuple[int, float]] = {} + self.offload_pins_reclaimed = 0 @property def units_per_checkpoint(self) -> int: @@ -326,11 +345,126 @@ def complete_inflight(self) -> None: record.state = READY self.hash_to_checkpoint[record.prefix_hash] = checkpoint_id self._lru[checkpoint_id] = None + self._queue_offload_store(checkpoint_id, record) restores, self._inflight_restores = self._inflight_restores, [] for checkpoint_id in restores: self._release_restore_pin(checkpoint_id) + # ------------------------------- offload ------------------------------- # + def _queue_offload_store(self, checkpoint_id: int, record) -> None: + """Nominate this checkpoint for the CPU tier. Takes no pin. + + **The READY transition, and not anywhere else.** Three candidate + moments and only this one works: + + - `begin_store` is too early: the scatter has not ridden a batch, so + the units do not hold the image yet. + - `_evict` / `unindex` are too late. They fire when the pool wants + those units *now*, so a multi-millisecond D2H started there holds 127 + of the most-wanted blocks at the worst possible moment. + - READY is the moment the bytes first exist and the moment the record + is least wanted -- it has just entered the LRU at the cold end, which + is the furthest any checkpoint ever is from being spent. + + Nomination, not reservation. A queued candidate is unpinned, so it + still counts as free space and `_next_victim` may spend it: the pool + never waits on the CPU tier, and a checkpoint the pool needed more than + the tier did simply loses its copy. Dedup is free -- `begin_store` + already refuses a hash it holds or has pending, so one hash is + nominated once. + """ + if not self._offload_sink: + return + self._offload_ready.append(checkpoint_id) + + def take_offload_stores( + self, max_inflight: int + ) -> list[tuple[int, tuple[int, ...]]]: + """`(prefix_hash, unit_ids)` to hand the tier now, pinning each. + + **This is where the pin is taken**, not at READY, and that is what + bounds it. A pin lives in this process while the D2H runs in the + worker, so it spans several scheduler passes; pinning at READY would + make every checkpoint un-evictable for that whole window and break the + one thing #2045's admission argument rests on -- that a READY unpinned + checkpoint counts as available to live KV. Pinning only what is + actually in flight caps the damage at `max_inflight` images. + + Over the cap, candidates wait rather than being dropped: they stay + nominated, unpinned and evictable, and are offered again next pass if + they are still around. + + Drained, not read: a second submission would store the same image + twice, and the second report would unpin a record the first released. + """ + out: list[tuple[int, tuple[int, ...]]] = [] + while self._offload_ready and len(self._offload_pins) < max_inflight: + checkpoint_id = self._offload_ready.popleft() + record = self.records.get(checkpoint_id) + # Spent while it waited -- which nomination deliberately allows -- + # or already in flight under this hash. Nothing to send either way. + if record is None or record.state != READY: + continue + if record.prefix_hash in self._offload_pins: + continue + record.pin_count += 1 + self._offload_pins[record.prefix_hash] = (checkpoint_id, monotonic()) + out.append((record.prefix_hash, record.unit_ids)) + return out + + def settle_offload_store(self, prefix_hash: int) -> None: + """One store reported, either way. Release the units it was holding. + + Success and failure release identically: the pin exists to keep the + bytes still during the copy, and the copy is over either way. Whether + the hash becomes reachable is `StateOffloadIndex`'s business, not this + one's. + """ + entry = self._offload_pins.pop(prefix_hash, None) + if entry is None: + return + self._release_offload_pin(entry[0]) + + def reclaim_stale_offload_pins(self, timeout_s: float) -> int: + """Release pins whose report never came, after `timeout_s`. + + The pin lives here in the engine process and the D2H runs in the + worker, so a lost report -- a crashed worker, a dropped completion -- + would hold 127 blocks out of the pool forever. This is the unit-side + twin of `Scheduler._reconcile_stalled_deferred_saves` and takes the + same window, derived from LMCache's own pin timeout, for the same + reason: by then upstream has force-unpinned and stopped reading. + + Recovery is total, which is what makes 20 lines enough: a leaked pin + breaks no `BlockPool` invariant and leaves no half-released state, so + zeroing the count restores the record exactly. + """ + if timeout_s <= 0 or not self._offload_pins: + return 0 + cutoff = monotonic() - timeout_s + stale = [h for h, (_cid, at) in self._offload_pins.items() if at <= cutoff] + for prefix_hash in stale: + checkpoint_id, _at = self._offload_pins.pop(prefix_hash) + self._release_offload_pin(checkpoint_id) + self.offload_pins_reclaimed += 1 + return len(stale) + + def _release_offload_pin(self, checkpoint_id: int) -> None: + record = self.records.get(checkpoint_id) + if record is None: + return + if record.pin_count <= 0: + raise AssertionError("checkpoint offload pin underflow") + record.pin_count -= 1 + if record.state == EVICTING and record.pin_count == 0: + # `unindex` fired during the copy: the boundary's KV block was + # spent, so this image is unreachable in HBM from now on. The copy + # was still worth finishing -- the CPU copy outliving the HBM one + # is the entire point of the tier -- and only now do the units go + # back. + self._release_record(checkpoint_id) + def _release_restore_pin(self, checkpoint_id: int) -> None: record = self.records.get(checkpoint_id) if record is None: @@ -418,9 +552,10 @@ def __init__( pool: BlockPool, spec: PagedStateCheckpointSpec, enabled: bool, + offload_sink: bool = False, ) -> None: self.enabled = enabled - self.store = PageUnitCheckpointStore(pool, spec) + self.store = PageUnitCheckpointStore(pool, spec, offload_sink=offload_sink) # Keyed by `(seq id, prefix hash)` rather than by seq: two boundaries of # one prompt are two checkpoints, and keying by seq alone let the later # one overwrite the earlier before either was stored. Re-reaching the @@ -543,6 +678,20 @@ def has_available_units( def ensure_free_units(self, count: int) -> bool: return self.store.ensure_free_units(count) + def take_offload_stores( + self, max_inflight: int + ) -> list[tuple[int, tuple[int, ...]]]: + """`(prefix_hash, unit_ids)` to hand the tier now. See the store.""" + return self.store.take_offload_stores(max_inflight) + + def settle_offload_store(self, prefix_hash: int) -> None: + """One store reported, either way; release the units it was holding.""" + self.store.settle_offload_store(prefix_hash) + + def reclaim_stale_offload_pins(self, timeout_s: float) -> int: + """Release offload pins whose report never came. See the store.""" + return self.store.reclaim_stale_offload_pins(timeout_s) + def unindex(self, h: int) -> None: # `_pending` is keyed by `(seq, hash)`, so one hash can be pending for # several sequences at once -- two turns of a conversation reaching the @@ -573,5 +722,8 @@ def checkpoint_fates(self) -> dict[str, int]: "checkpoints_kept": self.checkpoints_kept, "checkpoints_dropped": self.checkpoints_dropped, "checkpoints_evicted": self.store.evictions, + # Non-zero means store reports are being lost, which would + # otherwise show only as a pool that has quietly shrunk. + "offload_pins_reclaimed": self.store.offload_pins_reclaimed, "checkpoints_orphaned": self.checkpoints_orphaned, } diff --git a/atom/model_engine/scheduler.py b/atom/model_engine/scheduler.py index 5d319c1583..28f279d24b 100644 --- a/atom/model_engine/scheduler.py +++ b/atom/model_engine/scheduler.py @@ -64,6 +64,34 @@ # it on a 1ms poll and the answer is a process constant. _SAVE_ABANDON_TIMEOUT_S: float | None = None +# Memoised result of `_offload_max_pending_saves`, a process constant likewise. +_MAX_PENDING_OFFLOAD: int | None = None + + +def _offload_max_pending_saves() -> int: + """How many offload transfers may be outstanding at once, engine-side. + + Shared with the KV leg's `OFFLOAD_MAX_PENDING_SAVES` rather than given a + knob of its own: a KV save and a state store both hold the bytes they are + reading out of the same pool while they run, so the queue depth is one + question about one resource. Two independent numbers would let an operator + raise one and unknowingly double what a slow backend can pin. + """ + global _MAX_PENDING_OFFLOAD + if _MAX_PENDING_OFFLOAD is not None: + return _MAX_PENDING_OFFLOAD + raw = os.environ.get("OFFLOAD_MAX_PENDING_SAVES", "2") + try: + value = int(raw) + except ValueError: + logger.warning( + "invalid OFFLOAD_MAX_PENDING_SAVES=%r; using 2 for the state tier", + raw, + ) + value = 2 + _MAX_PENDING_OFFLOAD = max(1, value) + return _MAX_PENDING_OFFLOAD + def _offload_save_abandon_timeout_s() -> float: """Seconds a finished request's blocks may sit deferred on an offload save @@ -1919,6 +1947,7 @@ def schedule(self) -> tuple[ScheduledBatch, dict[int, Sequence]]: connector_meta_output = None if self.kv_connector is not None: self._publish_state_loads() + self._publish_state_stores() connector_meta_output = self.kv_connector.build_connector_meta() # Freeze, per seq, whether this chunk finishes the prompt. Uses the @@ -2079,6 +2108,7 @@ def schedule(self) -> tuple[ScheduledBatch, dict[int, Sequence]]: connector_meta_output = None if self.kv_connector is not None: self._publish_state_loads() + self._publish_state_stores() connector_meta_output = self.kv_connector.build_connector_meta() decode_batch = ScheduledBatch( @@ -2500,6 +2530,44 @@ def _settle_state_load(self, req_id, ok: bool) -> None: if settle is not None: settle(req_id, ok=ok) + def _publish_state_stores(self) -> None: + """Hand this pass's ready checkpoints to the connector for the CPU tier. + + Beside `_publish_state_loads` and on the same schedule, so one pass + makes one decision about the tier. The cap is the same + `OFFLOAD_MAX_PENDING_SAVES` the KV leg uses and for the same reason: + each outstanding transfer holds the bytes it is reading out of the + pool, so the queue depth is also how much of the pool a slow backend + can pin. + + `take_state_stores` pins as it hands over, so anything the connector + will not carry has to be settled here rather than left to the + reconciler's full timeout -- the units are already held. + """ + if self.kv_connector is None: + return + take = getattr(getattr(self, "block_manager", None), "take_state_stores", None) + if take is None: + return + stores = take(_offload_max_pending_saves()) + if not stores: + return + accepted = False + enqueue = getattr(self.kv_connector, "enqueue_state_stores", None) + if enqueue is not None: + accepted = bool(enqueue(stores)) + if accepted: + return + logger.warning( + "state offload: %s did not carry %d state store(s); releasing " + "their units now. The tier's index should not have been installed " + "against this connector.", + type(self.kv_connector).__name__, + len(stores), + ) + for prefix_hash, _units in stores: + self.block_manager.settle_state_store(prefix_hash, ok=False) + def _publish_state_loads(self) -> None: """Hand this pass's state-tier loads to the connector, before it builds its metadata. @@ -3404,12 +3472,27 @@ def _update_from_kv_xfer_finished(self, kv_connector_output: KVConnectorOutput): # Indexed only on the report, never at submission: a hash advertised # before its bytes exist parks the next request over that prefix # against a `get` that must miss. - offload = getattr(getattr(self, "block_manager", None), "state_offload", None) + bm = getattr(self, "block_manager", None) + offload = getattr(bm, "state_offload", None) take = getattr(self.kv_connector, "take_state_reports", None) if offload is not None and take is not None: - indexed, _failed = take() + indexed, failed = take() + # Both settle the pin; only success indexes the hash. A hash whose + # bytes are not really there must never be voted for, and a pin + # exists to keep the source still during the copy -- which is over + # either way. for h in indexed: - offload.note_stored(int(h)) + bm.settle_state_store(int(h), ok=True) + for h in failed: + bm.settle_state_store(int(h), ok=False) + # Unit-side twin of `_reconcile_stalled_deferred_saves`, on the same + # window and for the same reason: the pin lives in this process + # while the D2H runs in the worker, so a report lost to a crashed + # worker or a dropped completion would hold a whole image out of + # the pool forever. Recovery is total -- a leaked pin breaks no + # `BlockPool` invariant -- so zeroing the count restores the record + # exactly. + bm.reclaim_stale_state_store_pins(_offload_save_abandon_timeout_s()) def get_request_counts(self) -> tuple[int, int]: """Returns (num_running_reqs, num_waiting_reqs).""" diff --git a/tests/test_lmcache_offload_connector.py b/tests/test_lmcache_offload_connector.py index b26cadaaad..ba5c4ad45d 100644 --- a/tests/test_lmcache_offload_connector.py +++ b/tests/test_lmcache_offload_connector.py @@ -4644,11 +4644,13 @@ def _k3_scheduler() -> KimiK3OffloadScheduler: s._save_inflight = {} s._save_tracker = {} s._pending_state_loads = [] + s._pending_state_stores = [] s._save_inflight_since = {} s._save_stalled = False s._warned_save_stalled = False s._state_indexed = set() s._state_index_failed = set() + s._state_store_failed_locally = set() s._active_load_operations = {} s._do_save = True s._max_pending_saves = 4 @@ -4981,3 +4983,30 @@ def test_an_object_that_will_not_report_its_size_is_still_loaded(): assert StateByteCodec._object_bytes(SimpleNamespace()) is None assert StateByteCodec._object_bytes(SimpleNamespace(get_size=lambda: 32)) == 32 + + +def test_state_stores_are_drained_into_the_metadata_exactly_once(monkeypatch): + """A second submission would store the same image twice, and the second + report would unpin a record the first already released.""" + s = _k3_scheduler() + monkeypatch.setattr( + DenseOffloadScheduler, + "build_connector_meta", + lambda self: LMCacheOffloadMetadata(), + ) + assert s.enqueue_state_stores([]) is False + assert s.enqueue_state_stores([(111, (1, 2, 3))]) is True + + assert s.build_connector_meta().state_stores == [(111, (1, 2, 3))] + assert s.build_connector_meta().state_stores == [] + + +def test_a_step_whose_only_work_is_a_state_store_reaches_the_worker(): + """Same shape as the `state_loads` bug: a store carries no `LMCacheReqMeta`, + so a work test that only counts requests drops the step -- and here the cost + is 127 blocks pinned against a report nobody was asked to produce.""" + from atom.kv_transfer.disaggregation.types import connector_metadata_has_work + + meta = LMCacheOffloadMetadata() + meta.state_stores = [(111, (1, 2))] + assert connector_metadata_has_work(meta) diff --git a/tests/test_page_unit_checkpoint.py b/tests/test_page_unit_checkpoint.py index 076a06972d..a9249ac454 100644 --- a/tests/test_page_unit_checkpoint.py +++ b/tests/test_page_unit_checkpoint.py @@ -380,3 +380,134 @@ def test_a_store_refuses_when_the_policy_leaves_the_loop_short(): assert store._next_checkpoint_id == before, "a refused store took an identity" assert len(store.records) == 100 + + +# ── offloading a checkpoint to the CPU tier ─────────────────────────────── + + +def offload_store(**kw): + pool, store = make_store(**kw) + store._offload_sink = True + return pool, store + + +class TestOffloadNomination: + """READY nominates; `take_offload_stores` is what pins. + + Splitting the two is the whole design. A pin lives in the engine process + while the D2H runs in the worker, so it spans several scheduler passes; + pinning at READY would make EVERY checkpoint un-evictable for that window + and break the one thing #2045's admission argument rests on -- that a READY + unpinned checkpoint counts as available to live KV. + """ + + def test_reaching_ready_nominates_without_pinning(self): + _pool, store = offload_store() + cid, _op = ready(store, 11) + assert store.records[cid].pin_count == 0 + assert store._is_evictable(cid), "a nominee must still be spendable" + + def test_handing_it_over_is_what_pins(self): + _pool, store = offload_store() + cid, _op = ready(store, 11) + stores = store.take_offload_stores(max_inflight=4) + assert stores == [(11, store.records[cid].unit_ids)] + assert store.records[cid].pin_count == 1 + assert not store._is_evictable(cid) + + def test_a_nominee_the_pool_needed_more_is_simply_gone(self): + """Nomination deliberately leaves it spendable, so `_next_victim` may + take it. Losing a CPU copy is the right price for never making the + pool wait on the tier.""" + _pool, store = offload_store() + cid, _op = ready(store, 11) + store._evict(cid) + assert store.take_offload_stores(max_inflight=4) == [] + + def test_no_sink_nominates_nothing(self): + """A pin nobody can release holds a whole image out of the pool + forever, so a deployment with no tier must take none.""" + _pool, store = make_store() # _offload_sink defaults False + ready(store, 11) + assert store.take_offload_stores(max_inflight=4) == [] + + def test_it_is_offered_once(self): + _pool, store = offload_store() + ready(store, 11) + assert len(store.take_offload_stores(max_inflight=4)) == 1 + assert store.take_offload_stores(max_inflight=4) == [] + + +class TestOffloadInflightCap: + """The cap is what bounds how much of the pool a slow backend can hold.""" + + def test_over_the_cap_the_rest_wait_rather_than_drop(self): + _pool, store = offload_store(num_units=40) + for h in (11, 12, 13): + ready(store, h) + first = store.take_offload_stores(max_inflight=2) + assert len(first) == 2 + # The third is still nominated, unpinned, and offered again next pass. + assert store.take_offload_stores(max_inflight=2) == [] + store.settle_offload_store(first[0][0]) + assert len(store.take_offload_stores(max_inflight=2)) == 1 + + def test_a_waiting_nominee_stays_evictable(self): + _pool, store = offload_store(num_units=40) + for h in (11, 12): + ready(store, h) + store.take_offload_stores(max_inflight=1) + waiting = store.lookup(12) + assert store._is_evictable(waiting) + + +class TestOffloadPinRelease: + def test_success_and_failure_release_identically(self): + """The pin keeps the bytes still during the copy, and the copy is over + either way. Whether the hash becomes reachable is the index's business.""" + for _ in range(2): + _pool, store = offload_store() + cid, _op = ready(store, 11) + store.take_offload_stores(max_inflight=4) + store.settle_offload_store(11) + assert store.records[cid].pin_count == 0 + + def test_a_report_for_a_hash_never_sent_is_a_no_op(self): + _pool, store = offload_store() + cid, _op = ready(store, 11) + store.settle_offload_store(11) + assert store.records[cid].pin_count == 0 + + def test_unindex_during_the_copy_holds_the_units_to_the_end(self): + """The tier's whole point: the CPU copy outliving the HBM one. So + `unindex` must not pull the source out from under a running D2H.""" + pool, store = offload_store() + cid, _op = ready(store, 11) + units = store.records[cid].unit_ids + store.take_offload_stores(max_inflight=4) + + store.unindex(11) + assert cid in store.records, "the units are still being read" + assert all(pool.is_used(u) for u in units) + + store.settle_offload_store(11) + assert cid not in store.records + assert not any(pool.is_used(u) for u in units) + + def test_a_lost_report_is_reclaimed_and_counted(self): + """The pin is in this process and the copy is in the worker, so a + crashed worker would otherwise hold an image out of the pool forever.""" + _pool, store = offload_store() + cid, _op = ready(store, 11) + store.take_offload_stores(max_inflight=4) + + assert store.reclaim_stale_offload_pins(timeout_s=3600) == 0 + assert store.reclaim_stale_offload_pins(timeout_s=-1) == 0, "disabled" + assert store.reclaim_stale_offload_pins(timeout_s=0.0) == 0, "disabled" + + assert store.reclaim_stale_offload_pins(timeout_s=1e-9) == 1 + assert store.records[cid].pin_count == 0 + assert store.offload_pins_reclaimed == 1 + # And recovery is total: the record is spendable again, unbroken. + assert store._is_evictable(cid) + assert store.lookup(11) == cid From 24afbcecaa096be7abf28171156dfd06f6af8007 Mon Sep 17 00:00:00 2001 From: zejunchen-zejun Date: Thu, 27 Aug 2026 18:40:25 +0800 Subject: [PATCH 008/151] fix(state-cache): a gated boundary neither tier has is disowned, not raised Phase 4a, and it has to be first: with nothing else changed this commit is behaviour-preserving, but without it the first request that actually uses the CPU tier takes the engine down. --- what changed `BlockManager._attach_state_slots`, PAGE branch. Was: if hit_hash != -1 and not begin_restore(hit_hash, seq.state_slot): release_many(seq.state_slots); seq.state_slots = [] raise RuntimeError("gated PAGE checkpoint disappeared before state attach") Now: cold start returns early; `begin_restore` success returns as before; a miss falls through to `_request_state_load`, and only if that also declines is the boundary disowned (`return False`, which `allocate` turns into `num_cached_tokens = 0`). New counter `state_gate_lost_boundary`, reported in `checkpoint_funnel`. --- why the raise could not stay It asserted that the gate and the HBM store agree. That held while `can_allocate` only ever accepted boundaries the HBM index carried. Phase 4b gives the CPU tier a vote, so a hash the gate accepted may live only there -- and every such request would hit this raise. Turning it into a load branch has to land before the vote does, not after, or the two commits are only safe as a pair. --- why disowning is reachable rather than defensive Two ways to arrive with neither tier able to produce it: * the tier's index is optimistic by construction -- `hashes` means "was stored once", never "is still there", because LMCache's own LRU can drop bytes at any time; * an HBM checkpoint can be unindexed between `can_allocate` and `allocate` by another seq's `_fresh_block` in the same pass. So this is a normal path with a counter, not an assertion. A large fraction of `joint_boundaries` landing here would mean the gate is accepting boundaries that do not survive to attach. --- the slots are kept, unlike the old abort The old code released the slots because it was aborting. Disowning is not an abort: the request is about to recompute its whole prefix, and it writes that state into these very slots. Releasing them would hand the next request a buffer this one is still filling. `test_missing_gated_checkpoint_releases_the_new_slot_and_raises` becomes `test_a_gated_boundary_neither_tier_has_is_disowned_not_raised`, and now pins all three: returns False, keeps its slots, counts one. Verified: pytest tests/ (less tests/plugin and the two msgpack/aiter collection errors that fail on #2045 too) -> zero failures. black clean; ruff unchanged. Co-Authored-By: Claude Opus 5 (1M context) --- atom/model_engine/block_manager.py | 44 ++++++++++++++++++++++-------- tests/test_state_checkpoint.py | 23 ++++++++++++---- 2 files changed, 50 insertions(+), 17 deletions(-) diff --git a/atom/model_engine/block_manager.py b/atom/model_engine/block_manager.py index 772b5e1072..4724ad173c 100644 --- a/atom/model_engine/block_manager.py +++ b/atom/model_engine/block_manager.py @@ -238,6 +238,12 @@ def __init__( self.joint_boundaries = 0 self.joint_boundaries_hbm = 0 self.joint_boundaries_tier = 0 + # Admissions whose gated boundary neither tier could produce by the time + # `allocate` ran. Non-zero is expected under pressure (the CPU index is + # optimistic, and an HBM checkpoint can be unindexed inside the same + # pass); a large fraction of `joint_boundaries` means the gate is + # accepting boundaries that do not survive to attach. + self.state_gate_lost_boundary = 0 # Why the rest got none, keyed by the gate that stopped them. self.joint_skips: dict[str, int] = {} # The LMCache chunk size in tokens, read where the config is rather than @@ -892,19 +898,32 @@ def _attach_state_slots(self, seq: Sequence, hit_hash: int) -> bool: width = self.state_slots_per_req if self.paged_state_checkpoints is not None: seq.state_slots = self.state.pop_many(width) - if hit_hash != -1 and not self.paged_state_checkpoints.begin_restore( - hit_hash, seq.state_slot - ): - self.state.release_many(seq.state_slots) - seq.state_slots = [] - raise RuntimeError( - "gated PAGE checkpoint disappeared before state attach" - ) seq.state_fork_src = -1 - # `begin_restore` either gathered the boundary's state into the - # committed slot or there was no boundary to gather, so what the - # caller holds is what it asked for. - return True + if hit_hash == -1: + return True # cold start: nothing claimed, nothing to restore + if self.paged_state_checkpoints.begin_restore(hit_hash, seq.state_slot): + # Queued for the next batch, which gathers the image out of its + # PAGE units into the committed slot. What the caller holds is + # what it asked for. + return True + # HBM does not have it. That used to be an invariant violation and + # this used to raise -- `can_allocate` had shrunk the hit to a + # boundary the HBM index carried, so a miss here meant the gate and + # the store disagreed. + # + # It is a normal path now, and it has to be: the gate consults the + # CPU tier too, so a hash it accepted may live only there. Leaving + # the raise in would take the engine down on the first request that + # actually used the tier. + if self._request_state_load(seq, hit_hash): + return True + # Neither tier can produce it. Reachable rather than defensive: + # the tier's index is optimistic (`hashes` means "was stored once"), + # and an HBM checkpoint can be unindexed between `can_allocate` and + # here by another seq's `_fresh_block` in the same pass. Disown the + # boundary -- blocks stay claimed and the forward recomputes. + self.state_gate_lost_boundary += 1 + return False src = self.state.lookup(hit_hash) if hit_hash != -1 else -1 if src < 0: @@ -1674,6 +1693,7 @@ def checkpoint_funnel(self) -> dict[str, int]: "joint_boundaries": self.joint_boundaries, "joint_boundaries_hbm": self.joint_boundaries_hbm, "joint_boundaries_tier": self.joint_boundaries_tier, + "state_gate_lost_boundary": self.state_gate_lost_boundary, } | self.state_checkpoint_fates() def pool_pressure(self) -> dict[str, int]: diff --git a/tests/test_state_checkpoint.py b/tests/test_state_checkpoint.py index c846ccb7f4..afe1509a9d 100644 --- a/tests/test_state_checkpoint.py +++ b/tests/test_state_checkpoint.py @@ -1389,7 +1389,20 @@ def test_deallocate_cancels_a_queued_restore_before_reusing_its_slot(self): assert third.state_slot == dst assert bm.take_state_maintenance_ops().checkpoint_restores == () - def test_missing_gated_checkpoint_releases_the_new_slot_and_raises(self): + def test_a_gated_boundary_neither_tier_has_is_disowned_not_raised(self): + """This used to raise, and could not stay that way. + + The raise asserted that the gate and the HBM store agree, which held + while `can_allocate` only ever accepted boundaries the HBM index + carried. The gate now consults the CPU tier as well, so a hash it + accepted may live only there -- and with no tier able to produce it, + the answer is to disown the boundary, not to take the engine down. + + The slots are KEPT, unlike the old abort: the request is about to + recompute its whole prefix and it writes that state into these very + slots. Releasing them here would hand the next request a buffer this + one is still filling. + """ bm = make_block_manager( paged_copy_config(), state_runtime=PAGED_COPY_RUNTIME, @@ -1403,11 +1416,11 @@ def test_missing_gated_checkpoint_releases_the_new_slot_and_raises(self): bm.paged_state_checkpoints.unindex(h) second = stateful_seq(list(range(48))) - with pytest.raises(RuntimeError, match="disappeared"): - bm._attach_state_slots(second, h) + assert bm._attach_state_slots(second, h) is False - assert second.state_slot == -1 - assert bm.state.num_free() == free_slots + assert second.state_slot >= 0, "the seq keeps slots to recompute into" + assert bm.state.num_free() == free_slots - bm.state_slots_per_req + assert bm.checkpoint_funnel()["state_gate_lost_boundary"] == 1 def test_copy_transfer_can_checkpoint_a_speculative_decode_boundary(self): spec = SimpleNamespace(num_speculative_tokens=3, use_dspark=lambda: False) From ca04f3d42118c5d8eb1c2bdd3042372f97743013 Mon Sep 17 00:00:00 2001 From: zejunchen-zejun Date: Thu, 27 Aug 2026 18:44:03 +0800 Subject: [PATCH 009/151] lmcache(k3): give the CPU tier a vote, and let a PAGE resume come from it Phase 4b. This is the commit that makes the tier reachable: until now K3 wrote 53.6 MiB per checkpoint to LMCache and nothing ever read it. --- what changed `page_unit_checkpoint.py` - `PagedStateCheckpointCoordinator.offload` + `attach_offload(index)`. - `resumable_hit` now asks `_reachable(h)` instead of `store.contains(h)`: HBM first, then `offload.hashes`. - The constructor's `offload_sink: bool` is gone -- `attach_offload` sets the vote and the sink together (see below). `block_manager.py` - Attaches the index to the coordinator once both exist. The coordinator is built from `state_runtime.checkpoint_spec` and the index from the connector config, so they are constructed in the wrong order for a ctor argument. `kimi_k3/connector.py` - Refuses to build the tier when `image_bytes != entry_bytes`. --- why the vote is the whole feature `can_allocate` walks the HBM prefix and asks every `Pool.STATE` class "the rightmost boundary <= X that you accept". With the coordinator answering only from `store.contains`, a hash whose image went to LMCache is invisible: the walk stops short, nothing ever asks for it back, and the store path is pure cost. `_reachable` is where the tier gets to say "I have that one". It stops at the first boundary it accepts, scanning right to left, so a hash accepted here that nothing can deliver does not cost a wasted lookup -- it costs the whole walk-back, hiding every shorter checkpoint still resident in HBM. That is why the disown path (Phase 4a) had to land first. No preference rule between the tiers is needed: both are keyed by the same content hash, so the scan takes the rightmost boundary wherever it lives, and `_attach_state_slots` tries `begin_restore` before `_request_state_load` regardless. A resident image never pays a park. --- why the vote is optimistic, deliberately `hashes` means "was stored once", never "is still there" -- LMCache's own LRU can drop bytes under it at any moment. A false positive costs one park plus a recompute, and retracts itself: `fail_load` calls `forget(h)`, so the next request over that prefix is not sent down the same hole. Being certain instead would mean a synchronous cross-process lookup on the admission path. --- why attaching is one call, not two fields Half-attached is worse than off in both directions. A vote with no sink accepts hashes nothing will ever store, so every accepting request parks and recomputes. A sink with no vote pins 127 units per checkpoint for a transfer nobody will read. `attach_offload` sets both, so they cannot diverge. --- the load's destination is the Active Slot Not 127 freshly reserved units. The two are interchangeable because #2045's copy plan intersects two *ordered byte streams* -- the slot's ranges and the units' regions are two cuts of one image -- so a blob gathered in unit order unpacks correctly in slot order. Writing the slot directly means no `ensure_free_units` on the admission path, no refusal branch, no units pinned for the length of an H2D, and no cap needed on concurrent loads. It does mean the store reads `image_bytes` and the load writes a whole slot, so the connector now refuses to build a tier where those differ, at the one point both numbers are in scope. They are equal for K3; a model where they are not would silently truncate the store or over-read the load. --- tests Six in `tests/test_page_unit_checkpoint.py`: a tier-only hash is accepted, no tier means no acceptance, the scan still takes the rightmost, attaching turns on both halves together, and the optimistic half retracts on `forget`. Three in `tests/test_state_checkpoint.py`, end to end through `_attach_state_slots`: a tier-only boundary becomes a queued load with no restore op and the slot as its destination; a boundary in both is served from HBM with no park; a tier that declines disowns and counts. Verified: pytest tests/ (less tests/plugin and the two msgpack/aiter collection errors that fail on #2045 too) -> zero failures. black clean; ruff unchanged. Co-Authored-By: Claude Opus 5 (1M context) --- .../offload/hybrid/kimi_k3/connector.py | 17 ++++ atom/model_engine/block_manager.py | 7 +- atom/model_engine/page_unit_checkpoint.py | 50 ++++++++++- tests/test_page_unit_checkpoint.py | 68 +++++++++++++++ tests/test_state_checkpoint.py | 84 +++++++++++++++++++ 5 files changed, 222 insertions(+), 4 deletions(-) diff --git a/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py b/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py index fbd3043e4f..daf68b3036 100644 --- a/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py +++ b/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py @@ -108,6 +108,23 @@ def _build_state_tier(self, transfer_tensors) -> None: ) return + # The store reads the checkpoint's PAGE units and the load writes an + # Active Slot, so the blob has to be the same length both ways. They are + # equal for K3 (a checkpoint covers the whole slot), and a model where + # they differ would silently truncate the store or over-read the load -- + # so it is asserted here rather than assumed, at the one point both + # numbers are in scope. + image_bytes = int(getattr(spec, "image_bytes", 0) or 0) + if image_bytes and image_bytes != entry_bytes: + logger.warning( + "kimi_k3 offload: a checkpoint image is %d B but an Active Slot " + "is %d B; the store reads units and the load writes a slot, so " + "they must match. State tier off.", + image_bytes, + entry_bytes, + ) + return + tp = get_tp_group() rank, world = pp_aware_rank_and_world(self._config, tp) cfg = offcfg.build_lmcache_config( diff --git a/atom/model_engine/block_manager.py b/atom/model_engine/block_manager.py index 4724ad173c..50a4dc2920 100644 --- a/atom/model_engine/block_manager.py +++ b/atom/model_engine/block_manager.py @@ -173,7 +173,6 @@ def __init__( self.kv, checkpoint_spec, enabled=enabled, - offload_sink=kv_offload_enabled, ) # The rolling state class: per-request slots plus a content index over # the free ones. A checkpoint IS a free slot whose content is still @@ -282,6 +281,12 @@ def __init__( # away -- which is what the staging ring existed to rescue. if kv_offload_enabled: self.state_offload = StateOffloadIndex() + # Attached rather than passed at construction: the coordinator is + # built before the switch is read (it needs `checkpoint_spec`, + # which comes off `state_runtime`). One object, two uses -- the + # coordinator votes off `hashes` and drains stores into it. + if self.paged_state_checkpoints is not None: + self.paged_state_checkpoints.attach_offload(self.state_offload) # The demand funnel: recorded at admission, cut for when a prefill # chunk is shortened to land on it, kept when the state pool files it. # Counted at all three because a gap between any two is a different diff --git a/atom/model_engine/page_unit_checkpoint.py b/atom/model_engine/page_unit_checkpoint.py index e5cb9e2178..9bc2a5ca22 100644 --- a/atom/model_engine/page_unit_checkpoint.py +++ b/atom/model_engine/page_unit_checkpoint.py @@ -552,10 +552,18 @@ def __init__( pool: BlockPool, spec: PagedStateCheckpointSpec, enabled: bool, - offload_sink: bool = False, + offload=None, ) -> None: self.enabled = enabled - self.store = PageUnitCheckpointStore(pool, spec, offload_sink=offload_sink) + # The CPU tier beneath this one, or None. Not a member of + # `BlockManager.state_caches`: every one of those is a veto over where a + # prefix may resume, and this does the opposite -- it makes MORE + # boundaries reachable. It is consulted in exactly two places, + # `resumable_hit` (the vote) and `take_offload_stores` (the sink). + self.offload = offload + self.store = PageUnitCheckpointStore( + pool, spec, offload_sink=offload is not None + ) # Keyed by `(seq id, prefix hash)` rather than by seq: two boundaries of # one prompt are two checkpoints, and keying by seq alone let the later # one overwrite the earlier before either was stored. Re-reaching the @@ -580,10 +588,32 @@ def resumable_hit( if not self.applies(seq): return hit for i in range(hit - 1, -1, -1): - if assume_checkpointed or self.store.contains(block_hashes[i]): + if assume_checkpointed or self._reachable(block_hashes[i]): return i + 1 return 0 + def _reachable(self, h: int) -> bool: + """Whether `h`'s image can be *produced*, in HBM or from the CPU tier. + + Produced, not merely indexed. This scan runs right to left and stops at + the first boundary it accepts, so accepting one whose bytes nothing can + deliver does not cost a wasted lookup -- it costs the whole walk-back, + hiding every shorter checkpoint still resident in HBM. + + HBM needs no preference rule: both tiers are keyed by the same content + hash, so the scan takes the rightmost boundary wherever it lives, and + `_attach_state_slots` tries HBM first regardless. + + The tier's half is deliberately optimistic -- `hashes` means "was stored + once", never "is still there", since LMCache's own LRU can drop bytes at + any time. A false positive costs one park plus a recompute and retracts + itself (`fail_load` -> `forget`); refusing to vote until certain would + cost a synchronous cross-process lookup on the admission path. + """ + if self.store.contains(h): + return True + return self.offload is not None and h in self.offload.hashes + def checkpoint(self, seq: Sequence, boundary_blocks: int, h: int) -> None: """File a boundary to be stored, keyed by hash rather than by seq. @@ -678,6 +708,20 @@ def has_available_units( def ensure_free_units(self, count: int) -> bool: return self.store.ensure_free_units(count) + def attach_offload(self, index) -> None: + """Wire the CPU tier in, after both objects exist. + + Not a constructor argument because the two are built in the wrong + order: this class needs `checkpoint_spec`, which comes off + `state_runtime`, while the index needs the connector config. One call + sets both halves -- the vote (`_reachable`) and the sink (whether a + READY checkpoint is nominated at all) -- so they cannot be turned on + separately, which would either vote for hashes nothing stores or pin + units nothing releases. + """ + self.offload = index + self.store._offload_sink = index is not None + def take_offload_stores( self, max_inflight: int ) -> list[tuple[int, tuple[int, ...]]]: diff --git a/tests/test_page_unit_checkpoint.py b/tests/test_page_unit_checkpoint.py index a9249ac454..fcda08550e 100644 --- a/tests/test_page_unit_checkpoint.py +++ b/tests/test_page_unit_checkpoint.py @@ -511,3 +511,71 @@ def test_a_lost_report_is_reclaimed_and_counted(self): # And recovery is total: the record is spendable again, unbroken. assert store._is_evictable(cid) assert store.lookup(11) == cid + + +class TestTheTierVotes: + """`resumable_hit` accepts a boundary the CPU tier holds, not just HBM. + + This is what makes the whole tier reachable: without it a hash whose image + went to LMCache is invisible to `can_allocate`, and the bytes are written + and never read. + """ + + class _Index: + def __init__(self, *hashes): + self.hashes = set(hashes) + + @staticmethod + def coordinator(num_units=40, offload=None): + from atom.model_engine.page_unit_checkpoint import ( + PagedStateCheckpointCoordinator, + PagedStateCheckpointSpec, + ) + + c = PagedStateCheckpointCoordinator( + BlockPool(num_units), + PagedStateCheckpointSpec(10, 25, "layout-v1", image_bytes=25), + enabled=True, + ) + if offload is not None: + c.attach_offload(offload) + return c + + @staticmethod + def seq(): + from types import SimpleNamespace + + return SimpleNamespace(has_per_req_cache=True) + + def test_a_hash_only_the_tier_holds_is_accepted(self): + c = self.coordinator(offload=self._Index(77)) + assert c.resumable_hit(self.seq(), 3, [11, 77, 99]) == 2 + + def test_without_a_tier_it_is_not(self): + c = self.coordinator() + assert c.resumable_hit(self.seq(), 3, [11, 77, 99]) == 0 + + def test_the_scan_still_takes_the_rightmost_boundary(self): + """Both tiers are keyed by the same content hash, so no preference rule + is needed -- and `_attach_state_slots` tries HBM first regardless.""" + c = self.coordinator(offload=self._Index(11, 99)) + assert c.resumable_hit(self.seq(), 3, [11, 77, 99]) == 3 + + def test_attaching_turns_on_the_vote_and_the_sink_together(self): + """Half-attached is worse than off: a vote with no sink accepts hashes + nothing ever stores; a sink with no vote pins units nothing reads.""" + c = self.coordinator() + assert c.offload is None and c.store._offload_sink is False + c.attach_offload(self._Index()) + assert c.offload is not None and c.store._offload_sink is True + + def test_the_tier_half_is_optimistic_on_purpose(self): + """`hashes` means "was stored once", never "is still there" -- LMCache's + own LRU can drop bytes under it. A false positive costs one park plus a + recompute and retracts itself; being certain would cost a synchronous + cross-process lookup on the admission path.""" + index = self._Index(77) + c = self.coordinator(offload=index) + assert c.resumable_hit(self.seq(), 3, [11, 77, 99]) == 2 + index.hashes.discard(77) # what `fail_load` -> `forget` does + assert c.resumable_hit(self.seq(), 3, [11, 77, 99]) == 0 diff --git a/tests/test_state_checkpoint.py b/tests/test_state_checkpoint.py index afe1509a9d..a1941ee3cf 100644 --- a/tests/test_state_checkpoint.py +++ b/tests/test_state_checkpoint.py @@ -1389,6 +1389,90 @@ def test_deallocate_cancels_a_queued_restore_before_reusing_its_slot(self): assert third.state_slot == dst assert bm.take_state_maintenance_ops().checkpoint_restores == () + # ── the CPU tier's leg of a PAGE resume ──────────────────────────── + + class _TierIndex: + """The engine-side index, reduced to what `_attach_state_slots` reads.""" + + def __init__(self, *hashes): + self.hashes = set(hashes) + self.pending_loads = {} + self.requested = [] + + def request_load(self, req_id, h): + if h not in self.hashes: + return False + self.pending_loads[req_id] = h + self.requested.append((req_id, h)) + return True + + def _with_tier(self, bm, *hashes): + index = self._TierIndex(*hashes) + bm.state_offload = index + bm.paged_state_checkpoints.attach_offload(index) + return index + + def test_a_boundary_only_the_tier_has_becomes_a_load(self): + """The path the whole tier exists for. HBM misses, the tier votes, and + the request parks on a load instead of disowning the boundary.""" + bm = make_block_manager(paged_copy_config(), state_runtime=PAGED_COPY_RUNTIME) + first = self._admitted(bm) + bm.hash_blocks(first, bm.checkpoint_limit(first) - first.num_cached_tokens) + h = boundary_hash(bm, first) + bm.take_state_maintenance_ops() + bm.complete_previous_state_batch() + # The image left HBM but the tier still advertises it. + bm.paged_state_checkpoints.unindex(h) + index = self._with_tier(bm, h) + + second = stateful_seq(list(range(48))) + assert bm._attach_state_slots(second, h) is True + + assert second.state_load_hash == h + assert second.state_slot >= 0, "the H2D writes this slot directly" + assert index.requested == [(second.id, h)] + # No restore queued: there is nothing in HBM to gather from, and the + # bytes land in the slot rather than in PAGE units. + assert bm.take_state_maintenance_ops().checkpoint_restores == () + assert bm.take_state_loads() == [(second.id, h, second.state_slot)] + + def test_hbm_is_preferred_over_the_tier(self): + """Both tiers are keyed by the same hash, so the gate does not say which + one answered -- `_attach_state_slots` tries HBM first and only falls to + a load on a miss. A resident image must not pay a park.""" + bm = make_block_manager(paged_copy_config(), state_runtime=PAGED_COPY_RUNTIME) + first = self._admitted(bm) + bm.hash_blocks(first, bm.checkpoint_limit(first) - first.num_cached_tokens) + h = boundary_hash(bm, first) + bm.take_state_maintenance_ops() + bm.complete_previous_state_batch() + index = self._with_tier(bm, h) # advertised in BOTH + + second = stateful_seq(list(range(48))) + assert bm._attach_state_slots(second, h) is True + + assert second.state_load_hash == -1 + assert index.requested == [], "a resident image must not pay a park" + assert len(bm.take_state_maintenance_ops().checkpoint_restores) == 1 + + def test_a_tier_that_declines_disowns_rather_than_parking(self): + """`request_load` refuses a hash it never stored, because a load is + resolved only by a report -- offering one would park the request + against bytes no `get` can produce.""" + bm = make_block_manager(paged_copy_config(), state_runtime=PAGED_COPY_RUNTIME) + first = self._admitted(bm) + bm.hash_blocks(first, bm.checkpoint_limit(first) - first.num_cached_tokens) + h = boundary_hash(bm, first) + bm.take_state_maintenance_ops() + bm.complete_previous_state_batch() + bm.paged_state_checkpoints.unindex(h) + self._with_tier(bm) # votes for nothing + + second = stateful_seq(list(range(48))) + assert bm._attach_state_slots(second, h) is False + assert second.state_load_hash == -1 + assert bm.checkpoint_funnel()["state_gate_lost_boundary"] == 1 + def test_a_gated_boundary_neither_tier_has_is_disowned_not_raised(self): """This used to raise, and could not stay that way. From 7e9a87be60750289c62457032b7ea6cb2cd6f8e2 Mon Sep 17 00:00:00 2001 From: zejunchen-zejun Date: Thu, 27 Aug 2026 18:51:30 +0800 Subject: [PATCH 010/151] lmcache(k3): the joint boundary now serves PAGE, which is its only reason to exist MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Phase 5, the last one. `_joint_kv_boundary` refused every PAGE sequence; it now serves only PAGE. With this the tier is reachable end to end. --- what changed `block_manager.py` - The gate inverts: - if not seq.has_per_req_cache or self.paged_state_checkpoints is not None: - return self._no_joint("not_hybrid") + if not seq.has_per_req_cache or self.paged_state_checkpoints is None: + return self._no_joint("no_paged_checkpoints") - The hbm/tier split reads the coordinator instead of the slot pool. - `joint_boundaries_hbm` / `_tier` become `state_hbm` / `state_tier`. `page_unit_checkpoint.py` -- `contains(h)` delegation, so the split does not reach into `.store`. `scheduler.py` / `llm_engine.py` -- the log line and the Prometheus tuple follow the rename. --- why the old refusal was right, and why it stopped being right It read `is not None` and refused, because a K3 checkpoint was an Active Slot that the tier spilled out of `StateSlotPool` -- a PAGE image lived in the KV pool, which the KV connector already offloads on its own, so there was no second boundary to agree on. #2045 moved K3's image into the KV pool and with it the entire reason the joint path exists. HBM now enforces `state ⊆ KV`: `_record_evicted` unindexes a checkpoint the instant its boundary block is spent, so a checkpoint can no longer outlive its KV there. **When LMCache hands the KV back, nothing hands the state back unless the two are fetched together.** Leaving this refusal in place makes the tier dead weight -- it writes 53.6 MiB per checkpoint and nothing ever reads it -- and no other counter in the system would say so. --- why the counters were renamed rather than left alone `joint_boundaries_hbm/_tier` described the boundary as a whole, which was true when the KV leg had no tier of its own to be served from. It has one now (`_decide_load_after_alloc` decides it separately), so the old names read as if they covered both legs. `state_hbm` / `state_tier` say what they measure. The split also means something new. It used to say "the state pool is too small for this concurrency". Under #2045 checkpoints share the paged pool with the KV write stream, so a ratio dominated by `state_tier` says **the paged pool** is too small -- checkpoints are being squeezed out by the KV they now live beside. That diagnosis did not exist before. `state_tier` is also the only counter here that cannot be non-zero with the CPU tier switched off, which makes it the honest test of "did this feature run". No passing unit test can produce it in production. --- tests Three in `tests/test_state_checkpoint.py`, on a harness that had to reproduce the shape the joint path is for: publish a prompt, place ONE rung at an interior boundary (a checkpoint under the last block is one no scan can look up, since `can_allocate` matches over `range(n_hash_blocks - 1)`), then evict a KV block below it so the walk stops short. A PAGE seq gets a joint boundary at the rung with real work for the KV leg; the split reports `hbm` while the checkpoint is resident and `tier` once it is not; no tier means no boundary. Verified: pytest tests/ (less tests/plugin and the two msgpack/aiter collection errors that fail on #2045 too) -> 3935 passed, zero failures. black clean; ruff unchanged. Co-Authored-By: Claude Opus 5 (1M context) --- atom/model_engine/block_manager.py | 34 ++++-- atom/model_engine/llm_engine.py | 4 +- atom/model_engine/page_unit_checkpoint.py | 9 ++ atom/model_engine/scheduler.py | 14 ++- tests/test_state_checkpoint.py | 130 ++++++++++++++++++++++ 5 files changed, 172 insertions(+), 19 deletions(-) diff --git a/atom/model_engine/block_manager.py b/atom/model_engine/block_manager.py index 50a4dc2920..183248a0cc 100644 --- a/atom/model_engine/block_manager.py +++ b/atom/model_engine/block_manager.py @@ -235,8 +235,11 @@ def __init__( # resident checkpoint, `tier` paid an entry-sized H2D and a park. Almost # all `tier` means the state pool is too small for the concurrency. self.joint_boundaries = 0 - self.joint_boundaries_hbm = 0 - self.joint_boundaries_tier = 0 + self.state_hbm = 0 + # The number that says the tier is doing anything at all. Every other + # counter here can be non-zero with the CPU tier switched off; this one + # cannot, which makes it the only honest test of "did this feature run". + self.state_tier = 0 # Admissions whose gated boundary neither tier could produce by the time # `allocate` ran. Non-zero is expected under pressure (the CPU index is # optimistic, and an HBM checkpoint can be unindexed inside the same @@ -588,10 +591,16 @@ def _joint_kv_boundary( seq.state_joint_claim_tokens = 0 if self.state_offload is None: return self._no_joint("off") - # PAGE checkpoints live in the KV pool, so their state leg has no tier - # to be served from and no separate boundary to agree on. - if not seq.has_per_req_cache or self.paged_state_checkpoints is not None: - return self._no_joint("not_hybrid") + # PAGE is now the served class, not the excluded one. This used to + # read `is not None` and refuse -- correct while a K3 checkpoint was an + # Active Slot the tier spilled out of `StateSlotPool`. #2045 moved the + # image into the KV pool, and with it the whole reason the joint path + # exists: HBM's `state ⊆ KV` (`_record_evicted` unindexes a checkpoint + # the instant its boundary block is spent) means a checkpoint can no + # longer outlive its KV in HBM -- so when LMCache hands the KV back, + # nothing hands the state back unless the two are fetched together. + if not seq.has_per_req_cache or self.paged_state_checkpoints is None: + return self._no_joint("no_paged_checkpoints") hbs = self._hash_block_size() # From the connector's config, not the connector object: the scheduler # holds whatever `get_kvconnector` returned, and one without @@ -649,10 +658,13 @@ def _joint_kv_boundary( claim_tokens = ((claim_blocks * hbs) // chunk) * chunk seq.state_joint_claim_tokens = max(claim_tokens, hbm_boundary * hbs) self.joint_boundaries += 1 - if self.state.lookup(h) >= 0: - self.joint_boundaries_hbm += 1 + # Which tier the STATE leg came from. The KV leg's own tier is decided + # separately by `_decide_load_after_alloc`, which is why these are not + # named for the boundary as a whole. + if self.paged_state_checkpoints.contains(h): + self.state_hbm += 1 else: - self.joint_boundaries_tier += 1 + self.state_tier += 1 return candidate def _no_joint(self, reason: str) -> int: @@ -1696,8 +1708,8 @@ def checkpoint_funnel(self) -> dict[str, int]: "chunks_cut_for_demand": self.chunks_cut_for_demand, "chunks_cut_for_end": self.chunks_cut_for_end, "joint_boundaries": self.joint_boundaries, - "joint_boundaries_hbm": self.joint_boundaries_hbm, - "joint_boundaries_tier": self.joint_boundaries_tier, + "state_hbm": self.state_hbm, + "state_tier": self.state_tier, "state_gate_lost_boundary": self.state_gate_lost_boundary, } | self.state_checkpoint_fates() diff --git a/atom/model_engine/llm_engine.py b/atom/model_engine/llm_engine.py index 2cc58d4830..e14fa0a8f8 100644 --- a/atom/model_engine/llm_engine.py +++ b/atom/model_engine/llm_engine.py @@ -406,8 +406,8 @@ def get_cache_statistics(self, timeout: float = 30.0) -> dict[str, Any]: # the rest: a boundary is per admission and every rank admits # the same request. "joint_boundaries", - "joint_boundaries_hbm", - "joint_boundaries_tier", + "state_hbm", + "state_tier", ) } # `reusable`, not `full`: a request's trailing block is never a reuse diff --git a/atom/model_engine/page_unit_checkpoint.py b/atom/model_engine/page_unit_checkpoint.py index 9bc2a5ca22..a14194e80c 100644 --- a/atom/model_engine/page_unit_checkpoint.py +++ b/atom/model_engine/page_unit_checkpoint.py @@ -708,6 +708,15 @@ def has_available_units( def ensure_free_units(self, count: int) -> bool: return self.store.ensure_free_units(count) + def contains(self, h: int) -> bool: + """Whether HBM holds a READY image for `h` right now. + + Distinct from `_reachable`, which also counts the CPU tier: this one + answers "would a resume be free", and is what the `state_hbm` / + `state_tier` split reports. + """ + return self.store.contains(h) + def attach_offload(self, index) -> None: """Wire the CPU tier in, after both objects exist. diff --git a/atom/model_engine/scheduler.py b/atom/model_engine/scheduler.py index 28f279d24b..b6f8feb288 100644 --- a/atom/model_engine/scheduler.py +++ b/atom/model_engine/scheduler.py @@ -1914,14 +1914,16 @@ def schedule(self) -> tuple[ScheduledBatch, dict[int, Sequence]]: skips = self.block_manager.joint_skips if chosen or skips: # The hbm/tier split is the actionable half: both are reuse, - # but only `tier` costs an entry-sized H2D and a park, so a - # ratio dominated by it says the state pool is too small for - # this concurrency even while the load counters look healthy. + # but only `tier` costs an image-sized H2D and a park. Under + # #2045 a ratio dominated by `tier` says the PAGED pool is too + # small for this concurrency -- checkpoints are being squeezed + # out by the KV write stream they now share a pool with -- and + # that is a diagnosis the fork-era split could not make. logger.info( - "joint kv: boundaries=%d (hbm=%d tier=%d) | %s", + "joint kv: boundaries=%d (state_hbm=%d state_tier=%d) | %s", chosen, - self.block_manager.joint_boundaries_hbm, - self.block_manager.joint_boundaries_tier, + self.block_manager.state_hbm, + self.block_manager.state_tier, " ".join(f"{k}={v}" for k, v in sorted(skips.items())), ) diff --git a/tests/test_state_checkpoint.py b/tests/test_state_checkpoint.py index a1941ee3cf..c5ce823bea 100644 --- a/tests/test_state_checkpoint.py +++ b/tests/test_state_checkpoint.py @@ -1473,6 +1473,136 @@ def test_a_tier_that_declines_disowns_rather_than_parking(self): assert second.state_load_hash == -1 assert bm.checkpoint_funnel()["state_gate_lost_boundary"] == 1 + # ── the joint boundary, which #2045 makes the only source of value ── + + def _joint_bm(self, chunk=BLOCK, **overrides): + bm = make_block_manager( + paged_copy_config(**overrides), state_runtime=PAGED_COPY_RUNTIME + ) + # Normally read off the LMCache config at construction; that import is + # unavailable here, and the value is the KV leg's transfer grid. + bm._joint_chunk_tokens = chunk + return bm + + def _prompt_with_a_rung_at(self, bm, blocks: int): + """Run PROMPT once and leave one READY checkpoint `blocks` blocks in. + + Placed explicitly rather than by the ladder because the position + matters: `can_allocate` matches over `range(n_hash_blocks - 1)`, so a + checkpoint filed under the LAST block is one no scan can ever look up. + An interior rung is what a resume actually lands on. + """ + seq = self._admitted(bm, PROMPT) + # Publish the prefix, which is what makes its blocks lookup-able and so + # what a second request's walk can match against. + bm.hash_blocks(seq, seq.num_prompt_tokens) + bm.take_state_maintenance_ops() + bm.complete_previous_state_batch() + # Whatever the ladder placed is not the subject here; keep exactly one + # rung, at a position chosen for being interior. + bm.paged_state_checkpoints.clear_index() + + h = bm._chain_to(seq, [], blocks)[blocks - 1] + bm.paged_state_checkpoints.checkpoint(seq, blocks, h) + bm.take_state_maintenance_ops() + bm.complete_previous_state_batch() + assert bm.paged_state_checkpoints.contains(h), "precondition: it is READY" + assert bm.kv.lookup(h) >= 0, "precondition: its KV block is published" + return seq, h + + @staticmethod + def _break_the_kv_chain_at(bm, block_id: int) -> None: + """Evict one published KV block so the prefix walk stops before the rung. + + Without this there is nothing for a joint boundary to do: a fully + resident prefix already gates to its rung, `_attach_state_slots` issues + the state load on its own, and the KV leg has nothing to fetch. The + joint path exists for exactly the case this creates -- HBM lost the KV, + LMCache still has it. + + Call after `deallocate`: `allocate` is the eviction event and it asserts + the block is unheld. + """ + assert bm.kv.block(block_id).hash != -1, "precondition: it was published" + bm.kv.allocate(block_id) # takes it for fresh content, dropping the hash + + def _reset_joint_counters(self, bm): + bm.joint_boundaries = bm.state_hbm = bm.state_tier = 0 + bm.joint_skips.clear() + + def test_a_page_class_now_gets_a_joint_boundary(self): + """The inversion, and the single most important assertion in Phase 5. + + This gate used to refuse every PAGE seq (`not_hybrid`), which was right + while a K3 checkpoint was an Active Slot the tier spilled out of the + slot pool. #2045 moved the image into the KV pool, and HBM's + `state ⊆ KV` means a checkpoint can no longer outlive its KV there -- + so when LMCache hands the KV back, nothing hands the state back unless + the two are fetched together. Refusing here makes the whole tier dead + weight, and no other counter in the system would say so. + """ + bm = self._joint_bm() + first, h = self._prompt_with_a_rung_at(bm, 8) # 32 tokens in + + victim = first.block_table[1] + bm.deallocate(first) + self._break_the_kv_chain_at(bm, victim) + bm.paged_state_checkpoints.unindex(h) # the image left HBM with it + self._with_tier(bm, h) # ...but LMCache still has it + self._reset_joint_counters(bm) + + second = stateful_seq(PROMPT) + second.offload_kv_prefix_tokens = len(PROMPT) + hbm_hit = bm.can_allocate(second) + + assert bm.joint_boundaries == 1, bm.joint_skips + assert second.state_joint_boundary_hash == h + assert second.state_joint_boundary_tokens == 8 * BLOCK + # The KV leg has real work: the walk stopped well below the boundary. + assert hbm_hit * BLOCK < second.state_joint_kv_tokens + + def test_the_split_says_which_tier_the_state_leg_came_from(self): + """`state_tier` is the only counter here that cannot be non-zero with + the CPU tier switched off, which makes it the honest test of "did this + feature run" -- no passing unit test can produce it in production.""" + bm = self._joint_bm() + first, h = self._prompt_with_a_rung_at(bm, 8) + victim = first.block_table[1] + bm.deallocate(first) + self._break_the_kv_chain_at(bm, victim) + self._with_tier(bm, h) + self._reset_joint_counters(bm) + + # The checkpoint outlived the block that broke the chain, so the state + # leg is free -- a gather out of resident units, no transfer. + resident = stateful_seq(PROMPT) + resident.offload_kv_prefix_tokens = len(PROMPT) + bm.can_allocate(resident) + assert (bm.state_hbm, bm.state_tier) == (1, 0) + + # Drop it from HBM too: now the state leg costs an image-sized H2D. + bm.paged_state_checkpoints.unindex(h) + from_cpu = stateful_seq(PROMPT) + from_cpu.offload_kv_prefix_tokens = len(PROMPT) + bm.can_allocate(from_cpu) + assert (bm.state_hbm, bm.state_tier) == (1, 1) + + def test_no_tier_means_no_joint_boundary(self): + """`state_offload is None` short-circuits before anything else: a + boundary both legs must reach is meaningless with one leg missing.""" + bm = self._joint_bm() + first, h = self._prompt_with_a_rung_at(bm, 8) + victim = first.block_table[1] + bm.deallocate(first) + self._break_the_kv_chain_at(bm, victim) + self._reset_joint_counters(bm) + + seq = stateful_seq(PROMPT) + seq.offload_kv_prefix_tokens = len(PROMPT) + bm.can_allocate(seq) + assert bm.joint_boundaries == 0 + assert bm.joint_skips.get("off") == 1 + def test_a_gated_boundary_neither_tier_has_is_disowned_not_raised(self): """This used to raise, and could not stay that way. From 5c7e516fa694596d420464d95224ae96dba7ad49 Mon Sep 17 00:00:00 2001 From: zejchen Date: Thu, 27 Aug 2026 11:36:53 +0000 Subject: [PATCH 011/151] fix(offload): the delegating shell never forwarded `enqueue_state_stores` Found by the first end-to-end run of the K3 CPU state tier on this branch: the tier reported healthy on both sides and stored exactly nothing. 94 checkpoints were nominated, handed to `_publish_state_stores`, refused, and had their PAGE units released again. --- the symptom Server log, 94 times over a 6-minute agentic replay at concurrency 8: state offload: LMCacheOffloadConnectorScheduler did not carry 1 state store(s); releasing their units now. with everything else looking correct: lmcache_offload: worker family=kimi_k3 kimi_k3 offload: state tier up, entry=55.39 MiB rank=0, sharing the paged-KV CPU pool, layout=kda-paged-state-v1:layers=69:... state checkpoints: checkpoints_kept=112 ... offload_pins_reclaimed=0 So the worker half built the tier, the coordinator nominated checkpoints at READY, and `take_offload_stores` pinned and handed them over -- and the scheduler half dropped every one. --- the cause `LMCacheOffloadConnectorScheduler` is a shell that owns no behaviour: it holds `self._impl = _build_scheduler(config)` and forwards each method by hand. `7b6b58e0` added `enqueue_state_stores` to `KimiK3OffloadScheduler` (kimi_k3/connector.py:352) but did not add the matching forwarder to the shell, so the method existed on the implementation and was invisible from outside: $ git grep -c enqueue_state_stores atom/kv_transfer/offload/connector.py 0 `Scheduler._publish_state_stores` probes with `getattr(self.kv_connector, "enqueue_state_stores", None)`, and `self.kv_connector` is the shell, so the probe returned None unconditionally and the store path took its "nothing will carry these" branch on every pass. This is not a race or a config error -- it could never have worked. The load leg was unaffected only by luck: `enqueue_state_loads` and `take_state_reports` were both given forwarders when they were added. Loads still moved no bytes, because with nothing ever stored `offload.hashes` stayed empty and the tier had nothing to vote for. --- the fix Add the missing forwarder, in the same shape as its `enqueue_state_loads` neighbour so the two cannot drift again. --- the second change, and why it is in this commit The warning that reported the failure named a cause that was not the one: The tier's index should not have been installed against this connector. That reads as a configuration mistake, and it sent the first pass of this diagnosis at `kv_connector_hosts_state_tier` and the layout selector -- both of which were behaving correctly. The class name printed is the *shell*, which tells you nothing about whether `_impl` supports the call. Both warnings now name the two possible causes and say which class is being reported. --- verification Static: the probe is `getattr` on a method the shell does not define, so the refusal is unconditional; adding the forwarder is the whole of the repair, and it is additive -- no existing path changes. Runtime: re-running the same workload with this commit applied, the `did not carry` warning no longer appears. Lines conform to black's 88-column limit; black and ruff are not installed in the benchmark image, so formatting was checked by inspection against the surrounding style rather than by running them. Co-Authored-By: Claude Opus 5 (1M context) --- atom/kv_transfer/offload/connector.py | 4 ++++ atom/model_engine/scheduler.py | 13 +++++++++---- 2 files changed, 13 insertions(+), 4 deletions(-) diff --git a/atom/kv_transfer/offload/connector.py b/atom/kv_transfer/offload/connector.py index c35ffaab1a..ebc241391b 100644 --- a/atom/kv_transfer/offload/connector.py +++ b/atom/kv_transfer/offload/connector.py @@ -160,6 +160,10 @@ def enqueue_state_loads(self, loads) -> bool: callback = getattr(self._impl, "enqueue_state_loads", None) return bool(callback(loads)) if callback is not None else False + def enqueue_state_stores(self, stores) -> bool: + callback = getattr(self._impl, "enqueue_state_stores", None) + return bool(callback(stores)) if callback is not None else False + def take_state_reports(self): callback = getattr(self._impl, "take_state_reports", None) return callback() if callback is not None else (set(), set(), set()) diff --git a/atom/model_engine/scheduler.py b/atom/model_engine/scheduler.py index b6f8feb288..3714682c8e 100644 --- a/atom/model_engine/scheduler.py +++ b/atom/model_engine/scheduler.py @@ -2562,8 +2562,11 @@ def _publish_state_stores(self) -> None: return logger.warning( "state offload: %s did not carry %d state store(s); releasing " - "their units now. The tier's index should not have been installed " - "against this connector.", + "their units now. Either this connector has no state tier (the " + "index should not have been installed against it), or the " + "delegating shell is missing an `enqueue_state_stores` forwarder " + "to the implementation that does -- check both before assuming " + "the configuration is wrong.", type(self.kv_connector).__name__, len(stores), ) @@ -2596,8 +2599,10 @@ def _publish_state_loads(self) -> None: # so this is not a miss to count against the index. logger.warning( "state offload: %s did not carry %d state load(s); waking those " - "requests to recompute. The tier's index should not have been " - "installed against this connector.", + "requests to recompute. Either this connector has no state tier, " + "or the delegating shell is missing an `enqueue_state_loads` " + "forwarder -- the class named here is the shell, not the " + "implementation behind it.", type(self.kv_connector).__name__, len(loads), ) From 9fabaa2a55c3346636c41722168101d2f24bc042 Mon Sep 17 00:00:00 2001 From: zejchen Date: Thu, 27 Aug 2026 11:47:44 +0000 Subject: [PATCH 012/151] feat(offload): counters for the store leg, and stop reporting a pool that holds none Three observability defects, found while trying to judge whether the K3 CPU state tier was working. Together they meant the tier could store nothing at all and no number anywhere would say so -- which is exactly what happened in 8a5364ab, where 94 refused stores produced one warning line and four counters that all read zero for unrelated reasons. --- 1. the store leg had no counters at all `StateOffloadIndex.stats()` reported `loads_attempted / completed / failed` and `indexed`. On a K3 run the load counters stay zero until something has been stored, so the only store-side signal was `indexed` growing -- and `indexed` is a set size that also grows for reasons unrelated to this pass. Added `stores_attempted` (handed to the connector, counted in `take_state_stores`), `stores_completed` / `stores_failed` (split in `settle_state_store` by the worker's report), and `stores_refused`. `stores_refused` is deliberately apart from `stores_failed`, and it is the one that matters most: failed means the worker tried and could not, refused means nobody tried. The shell-forwarding bug fixed in 8a5364ab was a pure `refused` condition, and with this counter it would have been one glance at the periodic line instead of a hunt through the connector factory. The worker's own `puts_refused` needs no separate channel: a refused `put` already returns False and settles as `ok=False`, so it lands in `stores_failed`. --- 2. `[Pool Pressure]` read the pool that holds no checkpoints `BlockManager.pool_pressure()` folded in `self.state.checkpoint_fates()`. Under #2045 a K3 checkpoint is a PAGE image the coordinator owns and the slot pool holds none, so the line printed [Checkpoint Fates] kept: 0, dropped: 0, evicted: 0, orphaned: 0 on the same run whose other line said `checkpoints_kept=112`. Two counters for one fact, disagreeing in one log, with the wrong one printed in the section a reader goes to for pool pressure. Now reads `self._state_checkpoint_cache`, which is already the coordinator when there is one and the slot pool otherwise -- the same object every other checkpoint question is asked of. `occupancy()` stays on the slot pool, which is the thing that has slots. --- 3. the tier's counters vanished when K3's checkpoints moved to PAGE `StateGroupPool.checkpoint_fates` used to fold the tier's stats in under a `state_offload_` prefix. `PagedStateCheckpointCoordinator.checkpoint_fates` was written without that, so once #2045 moved K3's checkpoints into the coordinator the whole `state_offload_*` family disappeared from the logs -- a silent regression, since nothing errors when a key stops being emitted. Restored on the coordinator. The fold is `getattr`-guarded because `attach_offload` accepts anything answering `hashes`, and the tests attach a double with no counters; a missing `stats` means "nothing to fold", not an error. (`test_a_tier_that_declines_disowns_rather_than_parking` caught this on the first draft.) --- also added: `state_offload_store_backlog` A gauge rather than a counter: `len(_offload_ready)`, the nominations waiting for a slot under `OFFLOAD_MAX_PENDING_SAVES`. That queue is unbounded and drained oldest-first, so checkpoints reaching READY faster than the tier drains them shows up as a value that climbs and stays high, with fresh nominations queueing behind stale ones. There is no other way to see it -- the stale entries are skipped cheaply on the way out, so nothing else registers the backlog. --- how to read the new line state_offload_stores_refused > 0 -- wiring fault, the connector never took it stores_attempted - completed - failed -- in flight stores_completed vs checkpoints_kept -- how much of what HBM keeps the tier got loads_failed / loads_attempted -- the index's false-positive rate store_backlog climbing -- READY outruns the drain Verified: pytest tests/test_page_unit_checkpoint.py tests/test_state_checkpoint.py tests/test_lmcache_offload_connector.py tests/test_block_manager.py -> 466 passed. Lines conform to black's 88-column limit; black and ruff are absent from the benchmark image, so formatting was checked by inspection. Co-Authored-By: Claude Opus 5 (1M context) --- atom/model_engine/block_manager.py | 17 ++++++++++-- atom/model_engine/page_unit_checkpoint.py | 33 ++++++++++++++++++++++- atom/model_engine/scheduler.py | 3 +++ atom/model_engine/state_offload.py | 28 ++++++++++++++++--- 4 files changed, 75 insertions(+), 6 deletions(-) diff --git a/atom/model_engine/block_manager.py b/atom/model_engine/block_manager.py index 183248a0cc..319406d1c3 100644 --- a/atom/model_engine/block_manager.py +++ b/atom/model_engine/block_manager.py @@ -1072,7 +1072,9 @@ def take_state_stores(self, max_inflight: int) -> list[tuple]: """ if self.paged_state_checkpoints is None or self.state_offload is None: return [] - return self.paged_state_checkpoints.take_offload_stores(max_inflight) + out = self.paged_state_checkpoints.take_offload_stores(max_inflight) + self.state_offload.stores_attempted += len(out) + return out def settle_state_store(self, prefix_hash: int, ok: bool) -> None: """One store reported. Release the units, and index the hash if it landed. @@ -1084,8 +1086,13 @@ def settle_state_store(self, prefix_hash: int, ok: bool) -> None: """ if self.paged_state_checkpoints is not None: self.paged_state_checkpoints.settle_offload_store(int(prefix_hash)) - if ok and self.state_offload is not None: + if self.state_offload is None: + return + if ok: + self.state_offload.stores_completed += 1 self.state_offload.note_stored(int(prefix_hash)) + else: + self.state_offload.stores_failed += 1 def reclaim_stale_state_store_pins(self, timeout_s: float) -> int: """Release store pins whose report never came. See the store.""" @@ -1731,6 +1738,12 @@ def pool_pressure(self) -> dict[str, int]: concludes checkpointing never fires. Occupancy stays on `self.state`: that is the slot pool either way, and the coordinator has none. """ + # `_state_checkpoint_cache`, not `self.state`: under #2045 a K3 + # checkpoint is a PAGE image the coordinator owns, and the slot pool + # holds none -- so reading the pool here printed `kept: 0` beside a + # `state checkpoints:` line saying 112, two counters for one fact + # disagreeing in the same log. `occupancy()` stays on the pool, which + # is the thing that has slots. return ( self.kv.eviction_stats() | self.state.occupancy() diff --git a/atom/model_engine/page_unit_checkpoint.py b/atom/model_engine/page_unit_checkpoint.py index a14194e80c..e23687be60 100644 --- a/atom/model_engine/page_unit_checkpoint.py +++ b/atom/model_engine/page_unit_checkpoint.py @@ -378,6 +378,11 @@ def _queue_offload_store(self, checkpoint_id: int, record) -> None: return self._offload_ready.append(checkpoint_id) + @property + def store_backlog(self) -> deque: + """Nominations waiting for an in-flight slot. A gauge for the log.""" + return self._offload_ready + def take_offload_stores( self, max_inflight: int ) -> list[tuple[int, tuple[int, ...]]]: @@ -771,7 +776,20 @@ def clear_index(self) -> None: self.store.clear() def checkpoint_fates(self) -> dict[str, int]: - return { + """What became of this class's checkpoints, plus the CPU tier's own. + + The tier's counters are folded in here rather than reported separately + because the two only mean anything against each other: `stores_completed` + is worth reading only as a fraction of `checkpoints_kept`, and a tier + that stored nothing is indistinguishable from a tier that was never + asked unless both numbers sit on one line. `StateGroupPool` did this + before #2045 moved K3's checkpoints here; restoring it makes the load + and store legs visible again under PAGE. + + Keys are prefixed `state_offload_` because the aggregation upstream is + by key across every state class. + """ + fates = { "checkpoints_kept": self.checkpoints_kept, "checkpoints_dropped": self.checkpoints_dropped, "checkpoints_evicted": self.store.evictions, @@ -780,3 +798,16 @@ def checkpoint_fates(self) -> dict[str, int]: "offload_pins_reclaimed": self.store.offload_pins_reclaimed, "checkpoints_orphaned": self.checkpoints_orphaned, } + # `getattr`, not a direct call: `attach_offload` accepts anything that + # answers `hashes`, and the tests attach a double that does not carry + # counters. A missing `stats` means "no numbers to fold", not an error. + stats = getattr(self.offload, "stats", None) if self.offload else None + if callable(stats): + fates.update((f"state_offload_{k}", v) for k, v in stats().items()) + # A gauge, not a counter: how many nominations are waiting for a + # slot under `OFFLOAD_MAX_PENDING_SAVES`. The queue is unbounded and + # drained oldest-first, so a value that climbs and stays high means + # checkpoints are reaching READY faster than the tier drains them + # and fresh nominations are queueing behind stale ones. + fates["state_offload_store_backlog"] = len(self.store.store_backlog) + return fates diff --git a/atom/model_engine/scheduler.py b/atom/model_engine/scheduler.py index 3714682c8e..de04d0bf08 100644 --- a/atom/model_engine/scheduler.py +++ b/atom/model_engine/scheduler.py @@ -2560,6 +2560,9 @@ def _publish_state_stores(self) -> None: accepted = bool(enqueue(stores)) if accepted: return + offload = getattr(getattr(self, "block_manager", None), "state_offload", None) + if offload is not None: + offload.stores_refused += len(stores) logger.warning( "state offload: %s did not carry %d state store(s); releasing " "their units now. Either this connector has no state tier (the " diff --git a/atom/model_engine/state_offload.py b/atom/model_engine/state_offload.py index 41febb12c6..e0b36407cd 100644 --- a/atom/model_engine/state_offload.py +++ b/atom/model_engine/state_offload.py @@ -43,6 +43,18 @@ def __init__(self) -> None: self.loads_attempted = 0 self.loads_completed = 0 self.loads_failed = 0 + # Store-side counters. Until this commit the only store-side signal was + # `indexed` growing, which is why a shell that refused every store + # (`enqueue_state_stores` never forwarded) looked identical to a tier + # with nothing to do -- 94 refusals produced one warning line and no + # number anywhere. `stores_refused` is the probe that would have said + # so on the first pass, and it is deliberately apart from + # `stores_failed`: refused means nobody tried, failed means the worker + # tried and could not. + self.stores_attempted = 0 + self.stores_completed = 0 + self.stores_failed = 0 + self.stores_refused = 0 # ------------------------------- stores -------------------------------- # def note_stored(self, h: int) -> None: @@ -116,11 +128,21 @@ def abandon_load(self, req_id) -> None: def stats(self) -> dict[str, int]: """Counters for the periodic `state checkpoints:` line. - Read the three load counters together: `failed / attempted` is this - index's false-positive rate, and `attempted - completed - failed` is - what is in flight or was abandoned by an aborted request. + Read the load counters together: `failed / attempted` is this index's + false-positive rate, and `attempted - completed - failed` is what is in + flight or was abandoned by an aborted request. Read the store counters + the same way, and read `stores_completed` against `checkpoints_kept`: + the gap is how much of what HBM keeps the CPU tier never received. """ return { + # Store leg. `attempted - completed - failed` is in flight; + # `refused` is a wiring fault, not backpressure -- any non-zero + # value means the connector never took the store at all. + "stores_attempted": self.stores_attempted, + "stores_completed": self.stores_completed, + "stores_failed": self.stores_failed, + "stores_refused": self.stores_refused, + # Load leg. "loads_attempted": self.loads_attempted, "loads_completed": self.loads_completed, "loads_failed": self.loads_failed, From 445693b2ebb09c42255388b28fcf7ba2e37ea3a7 Mon Sep 17 00:00:00 2001 From: zejchen Date: Thu, 27 Aug 2026 12:48:03 +0000 Subject: [PATCH 013/151] fix(offload): the K3 state packer imported a module that does not exist Every state store on this branch has failed since the store path was wired. Found by the first run where stores were actually carried (8a5364ab): 3,904 tracebacks, all identical, and not one checkpoint reached LMCache. --- the symptom Traceback (most recent call last): File "kimi_k3/state_tier.py", line 109, in _do_store stored = bool(self.codec.put(h, unit_ids)) File "kimi_k3/state_object.py", line 141, in put self._staged.pack(self._backend.page_unit_views(unit_ids), obj) File "kimi_k3/staging.py", line 175, in pack from atom.kv_transfer.offload.triton_kv_staging import fused_pack_chunk_major ModuleNotFoundError: No module named 'atom.kv_transfer.offload.triton_kv_staging' state offload: store of hash 10785477046942485201 failed 3,904 of them over a 50-minute concurrency-16 agentic replay -- one per store attempt, on every rank. `stores_failed` would have counted all of them, and `indexed` never moved. --- cause 1: the module is one package deeper `fused_pack_chunk_major` lives in `atom.kv_transfer.offload.dense.triton_kv_staging`. The dense codec, which has used it all along, imports it from there (`dense/kv_byte_codec.py:127`). The K3 stager's import dropped the `.dense`. Nothing caught it because the import is function-local, inside `pack`, so it is only attempted when a store actually runs -- and until 8a5364ab no store ever reached the worker. The unit tests do not reach it either: they exercise the codec against doubles that never call the real packer. --- cause 2: a failed `put` leaked its allocation, once per failure obj = self._allocate(self.entry_bytes) # ref_count = 1 self._staged.pack(...) # raises self._storage.batched_put([key], [obj]) # never reached `batched_put` discharges the reference it is handed (`StorageManager.batched_put` ends in `memory_obj.ref_count_down()`), so the success path correctly owes nothing -- but only if it is reached. With `pack` throwing, the allocation was stranded at ref_count=1 and LMCache reported it much later: MemoryObj at ... is being garbage collected with ref_count=1, pin_count=0. This indicates ref_count_down()/unpin() was not called 3,768 such warnings against 3,904 failed stores: one leak per failure. Left alone this shrinks the shared CPU pool by one 53.6 MiB entry per failed store until nothing can be allocated at all -- so the second-order effect of the import bug was a slow starvation of the pool the KV leg shares. `get` already guarded its own reference with `finally` for exactly this reason; `put` did not. It does now, on the exception path only, matching what the DSV4 codec does (`hybrid/dsv4/codec.py:1293`). --- why one commit The leak is only reachable through a throwing `pack`, and the import was the only thing throwing. Fixing the import alone would hide the leak rather than remove it, and the next exception from that line -- an out-of-memory in the staging buffer, say -- would strand allocations again with no traceback to explain the pool shrinking. --- verification python -c "from atom.kv_transfer.offload.dense.triton_kv_staging import fused_pack_chunk_major" -> OK pytest tests/test_lmcache_offload_connector.py tests/test_state_checkpoint.py tests/test_page_unit_checkpoint.py -> 412 passed The import is still function-local, so this remains untested by anything that does not run a real store. A unit test that calls `StagedTransfer.pack` against real CUDA tensors would have caught it; that is worth adding and is not in this commit. Co-Authored-By: Claude Opus 5 (1M context) --- atom/kv_transfer/offload/hybrid/kimi_k3/staging.py | 4 +++- .../offload/hybrid/kimi_k3/state_object.py | 13 ++++++++++++- 2 files changed, 15 insertions(+), 2 deletions(-) diff --git a/atom/kv_transfer/offload/hybrid/kimi_k3/staging.py b/atom/kv_transfer/offload/hybrid/kimi_k3/staging.py index 79ba017515..ed3e4ade53 100644 --- a/atom/kv_transfer/offload/hybrid/kimi_k3/staging.py +++ b/atom/kv_transfer/offload/hybrid/kimi_k3/staging.py @@ -172,7 +172,9 @@ def pack(self, segments: list[torch.Tensor], dst: Any) -> None: stream and does not wait for the forward that wrote the entry. That wait is the caller's (`state_tier._do_spill`); see the class docstring. """ - from atom.kv_transfer.offload.triton_kv_staging import fused_pack_chunk_major + from atom.kv_transfer.offload.dense.triton_kv_staging import ( + fused_pack_chunk_major, + ) segments = list(segments) seg_bytes = self._segment_block_bytes(segments) diff --git a/atom/kv_transfer/offload/hybrid/kimi_k3/state_object.py b/atom/kv_transfer/offload/hybrid/kimi_k3/state_object.py index 7dbc80aeb7..7542350fd1 100644 --- a/atom/kv_transfer/offload/hybrid/kimi_k3/state_object.py +++ b/atom/kv_transfer/offload/hybrid/kimi_k3/state_object.py @@ -138,7 +138,18 @@ def put(self, h: int, unit_ids) -> bool: if obj is None: self.puts_refused += 1 return False - self._staged.pack(self._backend.page_unit_views(unit_ids), obj) + # `batched_put` discharges the reference it is handed, so the success + # path owes nothing -- but only if it is reached. A throwing `pack` + # skips it and strands the allocation at ref_count=1, which LMCache + # reports much later as "garbage collected with ref_count=1, + # pin_count=0" and which shrinks the CPU pool by one entry per failure + # until the tier can no longer allocate at all. `get` already guards + # its own reference with `finally` for the same reason. + try: + self._staged.pack(self._backend.page_unit_views(unit_ids), obj) + except Exception: + obj.ref_count_down() + raise self._storage.batched_put([self.key(h)], [obj]) return True From 29a7cac8e4e6f7e118a85e845dd02e345877513c Mon Sep 17 00:00:00 2001 From: zejunchen-zejun Date: Thu, 27 Aug 2026 21:25:38 +0800 Subject: [PATCH 014/151] fix(offload): the same two wiring gaps, in the places the fixes did not reach Fix-then-sweep over `dfab0fa0` and `8a5364ab`. Both of those repaired one instance of a bug that had two, and neither instance the fix reached is covered by any test. --- 1. `unpack` imports the same module that does not exist `dfab0fa0` corrected `StagedTransfer.pack`'s import of `fused_pack_chunk_major` (the module is one package deeper, under `.dense`). `unpack` imports `fused_unpack_chunk_major` from the same wrong path, eleven lines further down, and was left alone. It has never fired because no state load has ever run -- with nothing stored, `offload.hashes` stayed empty and the tier had nothing to vote for. The first successful store makes the first load reachable, and it would have raised `ModuleNotFoundError` on that load, on every rank. Both kernels live in `atom.kv_transfer.offload.dense.triton_kv_staging`; verified by importing both. --- 2. `chunk_size` is not on the delegating shell either `8a5364ab` added the missing `enqueue_state_stores` forwarder. Sweeping every member `Scheduler` reads off `self.kv_connector` against what `LMCacheOffloadConnectorScheduler` defines turns up one more: `chunk_size`. scheduler.py:2284 seq.offload_kv_chunk_tokens = int( getattr(self.kv_connector, "chunk_size", 0) or 0) `self.kv_connector` is the shell, the shell has no `chunk_size`, so this has always stamped 0. It is latent rather than live: `_joint_kv_boundary` reads `self._joint_chunk_tokens` first and only falls back to this, and that primary comes from `BlockManager.__init__` reading the LMCache config directly. But the fallback exists precisely for the case where that read failed -- and it was dead, so a build that could not read the config at startup would refuse every joint boundary with `no_chunk_size` and never say why. Forwarded as a property, with the reason in its docstring. --- 3. the class of bug, closed rather than fixed twice Both are the same shape: the shell forwards by hand, so a member added to an implementation is invisible from outside, and every reader is a `getattr(..., default)` that does not raise -- it takes the default forever. Two instances in two days says the next one is a matter of time. `test_every_member_the_scheduler_reads_is_reachable_through_the_shell` scans `scheduler.py`'s source for `getattr(self.kv_connector, "x")`, `self.kv_connector.x`, and `_connector_flag("x")`, and asserts every name is reachable on the shell class. It reads the source rather than a hand-kept list, because a hand-kept list is the same failure one level up. The scan asserts it found something, so a regex that stops matching fails loudly instead of passing vacuously. Falsified: renaming the new `chunk_size` property makes it fail and name the missing member. Verified: pytest tests/ (less tests/plugin and the two msgpack/aiter collection errors) -> 3936 passed, zero failures. black clean; ruff 36, unchanged. Co-Authored-By: Claude Opus 5 (1M context) --- atom/kv_transfer/offload/connector.py | 24 ++++++++++- .../offload/hybrid/kimi_k3/staging.py | 4 +- tests/test_lmcache_offload_connector.py | 41 +++++++++++++++++++ 3 files changed, 67 insertions(+), 2 deletions(-) diff --git a/atom/kv_transfer/offload/connector.py b/atom/kv_transfer/offload/connector.py index ebc241391b..581b4b123b 100644 --- a/atom/kv_transfer/offload/connector.py +++ b/atom/kv_transfer/offload/connector.py @@ -101,7 +101,17 @@ def get_finished_recv_blocks(self): class LMCacheOffloadConnectorScheduler(KVConnectorSchedulerBase): - """Scheduler-side shell delegating to the selected implementation.""" + """Scheduler-side shell delegating to the selected implementation. + + **Every member `Scheduler` reads off `self.kv_connector` has to appear here**, + because `self.kv_connector` IS this object -- the implementation behind + `_impl` is invisible from outside. A member the scheduler probes with + `getattr(..., default)` and this class does not define does not raise; it + silently takes the default forever, which is how `enqueue_state_stores` + refused every state store from the day it was written. `tests/ + test_lmcache_offload_connector.py` sweeps for the gap rather than trusting + this comment. + """ is_producer = False is_offload = True @@ -109,6 +119,18 @@ class LMCacheOffloadConnectorScheduler(KVConnectorSchedulerBase): def __init__(self, config) -> None: self._impl = _build_scheduler(config) + @property + def chunk_size(self) -> int | None: + """The KV leg's transfer grid, in tokens. + + Read by `Scheduler._resolve_waiting_remote_kv` to stamp + `seq.offload_kv_chunk_tokens`, which is `_joint_kv_boundary`'s fallback + when `BlockManager` could not read the LMCache config at startup. + Without this forwarder that fallback was dead -- `getattr` returned its + 0 default and the joint path refused with `no_chunk_size`. + """ + return getattr(self._impl, "chunk_size", None) + def get_num_new_matched_tokens(self, seq): return self._impl.get_num_new_matched_tokens(seq) diff --git a/atom/kv_transfer/offload/hybrid/kimi_k3/staging.py b/atom/kv_transfer/offload/hybrid/kimi_k3/staging.py index ed3e4ade53..0967c40fe2 100644 --- a/atom/kv_transfer/offload/hybrid/kimi_k3/staging.py +++ b/atom/kv_transfer/offload/hybrid/kimi_k3/staging.py @@ -214,7 +214,9 @@ def unpack(self, src: Any, segments: list[torch.Tensor]) -> None: the one that produces the observable result, so it is the one waited on before the segments are handed back to their owner. """ - from atom.kv_transfer.offload.triton_kv_staging import fused_unpack_chunk_major + from atom.kv_transfer.offload.dense.triton_kv_staging import ( + fused_unpack_chunk_major, + ) segments = list(segments) seg_bytes = self._segment_block_bytes(segments) diff --git a/tests/test_lmcache_offload_connector.py b/tests/test_lmcache_offload_connector.py index ba5c4ad45d..d271ea98b9 100644 --- a/tests/test_lmcache_offload_connector.py +++ b/tests/test_lmcache_offload_connector.py @@ -5010,3 +5010,44 @@ def test_a_step_whose_only_work_is_a_state_store_reaches_the_worker(): meta = LMCacheOffloadMetadata() meta.state_stores = [(111, (1, 2))] assert connector_metadata_has_work(meta) + + +# ── the delegating shell must expose everything the scheduler reads ──────── + + +def test_every_member_the_scheduler_reads_is_reachable_through_the_shell(): + """The shell forwards by hand, so a new method is a silent no-op until + someone remembers to add it here too. + + `Scheduler` holds the SHELL in `self.kv_connector`, never the + implementation. Most of those reads are `getattr(..., default)`, which does + not raise on a missing member -- it takes the default, forever. That is how + `enqueue_state_stores` refused every state store from the day it was + written: the method existed on `KimiK3OffloadScheduler`, the probe ran + against the shell, and the store path took its "nothing will carry these" + branch on every pass with one warning line to show for it. + + So sweep for it. This reads the scheduler's source rather than a + hand-kept list, because a hand-kept list is the same failure one level up. + """ + import inspect + import re + + import atom.model_engine.scheduler as sched_mod + from atom.kv_transfer.offload.connector import LMCacheOffloadConnectorScheduler + + src = inspect.getsource(sched_mod) + probed = set(re.findall(r'getattr\(\s*self\.kv_connector,\s*"([a-z_]+)"', src)) + probed |= set(re.findall(r"self\.kv_connector\.([a-z_]+)\(", src)) + probed |= set(re.findall(r"self\.kv_connector\.([a-z_]+)\b", src)) + # `_connector_flag(name)` is the same read one indirection along. + probed |= set(re.findall(r'_connector_flag\(\s*"([a-z_]+)"', src)) + probed.discard("_impl") + assert probed, "the scan found nothing -- it has stopped matching the source" + + shell = LMCacheOffloadConnectorScheduler + missing = sorted(n for n in probed if not hasattr(shell, n)) + assert not missing, ( + "Scheduler reads these off `kv_connector`, but the delegating shell " + f"does not expose them, so each silently takes its default: {missing}" + ) From 8b849e207aca74b71559617cb2cfc120e50528f5 Mon Sep 17 00:00:00 2001 From: zejchen Date: Thu, 27 Aug 2026 14:07:35 +0000 Subject: [PATCH 015/151] fix(state-cache): the CPU tier gathered a checkpoint's padding as if it were image Fourth and last blocker in the chain that kept the K3 CPU tier from storing anything. With the previous three fixed, a 25-minute concurrency-16 replay still ended at state_offload_stores_attempted=462 state_offload_stores_completed=0 state_offload_stores_failed=450 state_offload_indexed=0 and 3,696 identical tracebacks: File "kimi_k3/state_object.py", line 149, in put self._staged.pack(self._backend.page_unit_views(unit_ids), obj) File "kimi_k3/staging.py", line 182, in pack dst_tensor = self.memory_tensor(dst, nbytes) ValueError: ATOM LMCache connector: MemoryObj tensor is too small for 59867136 bytes; got 58079232 --- the geometry From the run's own startup line: unit_bytes=2138112 image_bytes=58079232 units_per_checkpoint=28 ceil(58079232 / 2138112) = ceil(27.163) = 28 28 x 2138112 = 59867136 <- what page_unit_views gathered 58079232 <- what put() allocated difference = 1787904 <- padding in the 28th unit An image occupies whole units, and the last one is 83.6% padding here. Nothing owns those bytes. --- why only the CPU side hit it The HBM path never had the problem because it already passes the image size down: `_checkpoint_copy_plan` calls plan_segmented_copy(src_sizes, self._page_unit_stream_sizes(units), spec.image_bytes) so the D2D copy writes exactly `image_bytes` into the unit stream and leaves the tail untouched. The image is, by construction, the LEADING `image_bytes` of that stream. `page_unit_views` was written as the tensor-view counterpart of the same addressing but did not carry the third argument across, so it named whole units to a consumer that sizes its destination at `entry_bytes` -- which for K3 is exactly `image_bytes`. The connector's `image_bytes != entry_bytes` guard cannot catch this: both are 58079232 and agree. The quantity that disagrees is a third one, `sum(page_unit_views)`, which no check compared against either. --- the fix Trim the gathered stream to `image_bytes`, dropping whole views past the budget and slicing the one that straddles it. The slice is byte-exact rather than view-aligned because the image does not end on a view boundary: 58079232 leaves 350208 B of the last unit in use, which is 20.96 of its rows. That view is reinterpreted as `uint8` and sliced; the packer sizes segments as `numel * element_size`, so a uint8 segment costs it nothing. The load side needs no counterpart: it scatters into `state_entry_views`, which sums to `entry_bytes` exactly, so both ends of the round trip are now the same 58079232 bytes. A budget left unspent raises rather than storing a short blob -- that would be a real geometry disagreement, and a truncated image read back as valid is the one outcome worth crashing over. --- two notes on shape Inline rather than a helper method: the existing tests drive `page_unit_views` with a `types.SimpleNamespace` stand-in for `self`, and a second attribute would have to be stubbed in every one of them. For the same reason the spec is read with `getattr(spec, "image_bytes", 0)` -- a fork build carries no spec at all, and there the whole-unit stream is the right answer. --- verification pytest tests/test_kda_checkpoint_slot_copy.py tests/test_lmcache_offload_connector.py tests/test_state_checkpoint.py tests/test_page_unit_checkpoint.py -> 441 passed Also confirmed from the same run that the previous commit's leak guard works: `MemoryObj ... garbage collected with ref_count=1` went from 3,768 occurrences to 0, even though every store still failed. None of the four bugs in this chain was reachable by the unit tests, because all of them live past the first line that only a real store executes. A test that packs real CUDA tensors through `StagedTransfer` would have caught three of the four; it is still worth adding. Co-Authored-By: Claude Opus 5 (1M context) --- .../model_ops/attentions/kimi_mla_gdn_attn.py | 55 ++++++++++++++++++- 1 file changed, 54 insertions(+), 1 deletion(-) diff --git a/atom/model_ops/attentions/kimi_mla_gdn_attn.py b/atom/model_ops/attentions/kimi_mla_gdn_attn.py index 95a9c370c0..bfad3094bb 100644 --- a/atom/model_ops/attentions/kimi_mla_gdn_attn.py +++ b/atom/model_ops/attentions/kimi_mla_gdn_attn.py @@ -278,7 +278,60 @@ def page_unit_views(self, unit_ids: Sequence[int]) -> list[torch.Tensor]: ) lo = unit * block views.extend(plane[lo : lo + block] for plane in planes) - return views + + # ---- cut the padding tail off the final unit ---- + # + # An image is `image_bytes`, but it occupies `ceil(image_bytes / + # page_unit_bytes)` WHOLE units, so the last one is mostly padding: at + # K3's 2,138,112 B unit and 58,079,232 B image that is 28 units = + # 59,867,136 B, of which 1,787,904 belong to no image. + # + # `_checkpoint_copy_plan` already knows this -- it hands + # `plan_segmented_copy` the unit stream *and* `spec.image_bytes`, so the + # HBM copy writes only the leading `image_bytes` and leaves the tail + # alone. Gathering whole units for the CPU tier skipped that argument + # and asked the packer for the padding too, against a MemoryObj + # `StateByteCodec.put` sizes at `entry_bytes` -- which IS `image_bytes`. + # Every store died with "MemoryObj tensor is too small for 59867136 + # bytes; got 58079232". + # + # The trim is byte-exact rather than view-aligned because the image does + # not end on one: 58,079,232 leaves 350,208 B of the last unit in use, + # which is 20.96 rows of it. That view is reinterpreted as `uint8` and + # sliced; the packer sizes segments as `numel * element_size`, so it + # costs nothing to accept. The load side needs no counterpart -- it + # scatters into `state_entry_views`, which sums to `entry_bytes`. + # + # Inline rather than a helper method: the unit tests drive this function + # with a `SimpleNamespace` stand-in for `self`, so a second attribute + # would have to be stubbed by every one of them. + runtime = getattr(self.model_runner, "state_runtime", None) + spec = None if runtime is None else runtime.checkpoint_spec + # `getattr`: a fork build carries no spec, and the unit tests hand this + # a stand-in that carries no `image_bytes`. Either way there is no image + # size to trim against, and the whole-unit stream is the right answer. + budget = int(getattr(spec, "image_bytes", 0) or 0) if spec else 0 + if not budget: + return views + out: list[torch.Tensor] = [] + for view in views: + nbytes = view.numel() * view.element_size() + if budget >= nbytes: + out.append(view) + budget -= nbytes + continue + if budget > 0: + out.append(view.reshape(-1).view(torch.uint8)[:budget]) + budget = 0 + break + if budget: + raise RuntimeError( + f"a checkpoint image is {spec.image_bytes} B but its " + f"{len(views)} unit views hold " + f"{int(spec.image_bytes) - budget} B; " + "the unit geometry and the image size disagree" + ) + return out def build_kv_cache_tensor(self, layer_id: int, module): from atom.config import KVCacheTensor From 06cbdd4085543b062515b32672baeea5f1e6f69b Mon Sep 17 00:00:00 2001 From: zejchen Date: Thu, 27 Aug 2026 14:38:39 +0000 Subject: [PATCH 016/151] fix(offload): the state key treated an unsigned block hash as signed About half of every K3 state store failed. Measured over a concurrency-16 agentic replay with the four earlier blockers fixed: state_offload_stores_attempted=46 state_offload_stores_completed=19 state_offload_stores_failed=27 <- 59% with 296 identical tracebacks: File "kimi_k3/state_object.py", line 153, in put self._storage.batched_put([self.key(h)], [obj]) ^^^^^^^^^^^ File "kimi_k3/state_object.py", line 107, in key digest.update(int(h).to_bytes(8, "little", signed=True)) OverflowError: int too big to convert state offload: store of hash 12983297547834104159 failed --- cause 1: the hash is unsigned and the key said otherwise 12983297547834104159 > 2**63-1, so `to_bytes(8, signed=True)` cannot hold it. An ATOM block hash is `xxhash.xxh64().intdigest()`, which spans the whole 0..2**64-1, and `BlockManager.compute_hash` chains one into the next with `prefix.to_bytes(8, "little")` -- unsigned, no keyword. The state key was the only place in the chain that read the same value as signed, so it rejected every hash with the top bit set. That is half the key space, and the observed 59% is that with sampling noise. Nothing downstream noticed because the failure is inside a `try` in `_do_store`, which counts a failed store and moves on. Before the counters added in a8045614 it produced no number at all. --- cause 2: the raise stranded a MemoryObj, in a window the last guard missed `dfab0fa0` wrapped `pack` so a throwing gather would discharge the allocation. `key(h)` is evaluated in `batched_put`'s argument list, which is after the allocation and before `batched_put` can take ownership -- outside that guard. So each of the 296 overflows also leaked a 53.6 MiB entry, and the LMCache warning reappeared next to the traceback: MemoryObj at 136839168 is being garbage collected with ref_count=1 Fixed by computing the key *before* allocating. `key` is pure, so hoisting it removes the window rather than guarding it, and the `except` now covers `batched_put` itself as well -- matching what the DSV4 codec does. --- what this unblocks The same `key` is used by `get`, so roughly half of all loads would have missed in a way indistinguishable from an LRU eviction -- `fail_load` would then `forget` a hash whose bytes were never stored under a name it could find, and the index would decay for a reason nothing reported. Verified: pytest over the state and offload suites -> 442 passed. The overflow itself is confirmed directly: >>> (12983297547834104159).to_bytes(8, "little", signed=True) OverflowError: int too big to convert >>> (12983297547834104159).to_bytes(8, "little", signed=False).hex() '5f19dacf54f02db4' Co-Authored-By: Claude Opus 5 (1M context) --- .../offload/hybrid/kimi_k3/state_object.py | 17 +++++++++++++++-- 1 file changed, 15 insertions(+), 2 deletions(-) diff --git a/atom/kv_transfer/offload/hybrid/kimi_k3/state_object.py b/atom/kv_transfer/offload/hybrid/kimi_k3/state_object.py index 7542350fd1..da63945c36 100644 --- a/atom/kv_transfer/offload/hybrid/kimi_k3/state_object.py +++ b/atom/kv_transfer/offload/hybrid/kimi_k3/state_object.py @@ -104,7 +104,13 @@ def key(self, h: int): from lmcache.utils import CacheEngineKey digest = xxhash.xxh64() - digest.update(int(h).to_bytes(8, "little", signed=True)) + # Unsigned, and it must be: an ATOM block hash is `xxh64().intdigest()`, + # which spans the full 0..2**64-1, and `BlockManager.compute_hash` + # chains it with `prefix.to_bytes(8, "little")` -- unsigned too. With + # `signed=True` every hash above 2**63-1 raised `OverflowError: int too + # big to convert` from inside the `batched_put` argument list, which is + # about half of them: a measured 27 of 46 stores. + digest.update(int(h).to_bytes(8, "little", signed=False)) digest.update(self._layout_id.encode()) return CacheEngineKey( self._model_name, @@ -134,6 +140,13 @@ def put(self, h: int, unit_ids) -> bool: """ if self._storage is None: return False + # Before the allocation, not inside the `batched_put` argument list. + # `key` is pure and can raise (it did: see the `signed` note there), and + # an argument-position raise happens *after* the allocation and *before* + # `batched_put` can take ownership -- stranding the MemoryObj exactly + # the way a throwing `pack` used to. Computing it first removes the + # window rather than guarding it. + key = self.key(h) obj = self._allocate(self.entry_bytes) if obj is None: self.puts_refused += 1 @@ -147,10 +160,10 @@ def put(self, h: int, unit_ids) -> bool: # its own reference with `finally` for the same reason. try: self._staged.pack(self._backend.page_unit_views(unit_ids), obj) + self._storage.batched_put([key], [obj]) except Exception: obj.ref_count_down() raise - self._storage.batched_put([self.key(h)], [obj]) return True def get(self, h: int, slot: int) -> bool: From 1d201dcac6c67fe99ad24056577b13b30186cfbf Mon Sep 17 00:00:00 2001 From: zejunchen-zejun Date: Fri, 28 Aug 2026 09:39:19 +0800 Subject: [PATCH 017/151] test(state-cache): pin the image trim, which nothing reached MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `961f80d6` fixed `page_unit_views` gathering a checkpoint's padding, and the suite it ran stayed green before and after — because no test reaches the trim at all. The addressing tests build their stub spec as checkpoint_spec=SimpleNamespace(page_unit_bytes=...) with no `image_bytes`, so `budget` is 0 and every one of them takes the `return views` early path. The trim shipped untested, which is how a bug that failed 3,696 stores in a row got there in the first place. --- what is pinned Seven cases in `TestPageUnitViewsStopAtTheImage`, on the harness the addressing tests already use (`build` gains optional `image_bytes` and `dtype`; both stay off by default so the existing tests keep exercising whole units): * the gathered stream is exactly `image_bytes` — the invariant `put`'s allocation depends on, and the one that was broken; * a whole view past the budget is dropped, not truncated to zero length; * an image that lands on a view boundary is not sliced at all; * **the kept bytes are the leading bytes and nothing else** — trimming truncates, it does not reorder. The blob is read back by scattering into the slot in the same order, so a byte that moved here would land in the wrong layer there with nothing raised; * a multi-byte dtype is sliced by BYTES, not elements — the straddling view is reinterpreted as `uint8` first, and on bf16 an element-wise slice would keep twice what it should. The production cache is 1 byte per element, so only a test with a wider dtype can hold this; * a spec with no image size keeps whole units (the fork build, which has no spec to trim against); * units that cannot cover the image raise rather than storing a short blob. --- verification `tests/test_kda_checkpoint_slot_copy.py` skips wholesale on a non-GPU runner (the module imports aiter at load), so the seven cases were additionally run by extracting `page_unit_views` and its three neighbours out of the source and driving them directly. All seven hold, including the bf16 one. pytest tests/ (less tests/plugin and the two msgpack/aiter collection errors) -> 3936 passed, zero failures. black clean; ruff 36, unchanged. --- one note on `put`'s exception shape, checked not changed `2c5c3457` moved `batched_put` inside the `try` whose `except` calls `ref_count_down`. That is safe only if `batched_put` takes ownership just on success — and it is the pattern the DSV4 codec already uses (`hybrid/dsv4/codec.py:1284`, "StorageManager owns the MemoryObj after a successful batched_put"). Consistent with the repo rather than a new assumption, so left alone. The window between `_allocate` and `batched_put` is now fully covered: `page_unit_views` can raise (the geometry check above) and is inside the try; `key(h)` is computed before the allocation. Co-Authored-By: Claude Opus 5 (1M context) --- tests/test_kda_checkpoint_slot_copy.py | 117 +++++++++++++++++++++++-- 1 file changed, 108 insertions(+), 9 deletions(-) diff --git a/tests/test_kda_checkpoint_slot_copy.py b/tests/test_kda_checkpoint_slot_copy.py index 20f1d0bb26..eb83610b9e 100644 --- a/tests/test_kda_checkpoint_slot_copy.py +++ b/tests/test_kda_checkpoint_slot_copy.py @@ -221,18 +221,27 @@ class TestPageUnitAddressesAreArithmetic: RATIO = 4 # block_ratio: physical blocks per logical block N_LOGICAL = 5 - def build(self, block_size=None, spec_bytes=None): + def build(self, block_size=None, spec_bytes=None, image_bytes=None, dtype=None): phys_bs = self.LOGICAL_BS // self.RATIO or 1 - cache = torch.arange( - self.ROWS * self.N_LOGICAL * self.RATIO * phys_bs * self.ENTRY, - dtype=torch.uint8, - ).reshape(self.ROWS, self.N_LOGICAL * self.RATIO, phys_bs, self.ENTRY) - region = self.LOGICAL_BS * self.ENTRY - runtime = SimpleNamespace( - checkpoint_spec=SimpleNamespace( - page_unit_bytes=self.ROWS * region if spec_bytes is None else spec_bytes + dtype = dtype or torch.uint8 + cache = ( + torch.arange( + self.ROWS * self.N_LOGICAL * self.RATIO * phys_bs * self.ENTRY, + dtype=torch.int64, ) + .to(dtype) + .reshape(self.ROWS, self.N_LOGICAL * self.RATIO, phys_bs, self.ENTRY) + ) + region = self.LOGICAL_BS * self.ENTRY * cache.element_size() + spec = SimpleNamespace( + page_unit_bytes=self.ROWS * region if spec_bytes is None else spec_bytes ) + # Left OFF unless a test asks for it: `page_unit_views` trims to + # `image_bytes` only when the spec carries one, and the addressing tests + # above are about whole units. + if image_bytes is not None: + spec.image_bytes = image_bytes + runtime = SimpleNamespace(checkpoint_spec=spec) runner = SimpleNamespace( kv_cache=cache, block_size=self.LOGICAL_BS if block_size is None else block_size, @@ -526,3 +535,93 @@ def test_the_granularity_check_runs_before_any_view_is_handed_out(self): ) with pytest.raises(RuntimeError, match="granularity"): stub.page_unit_views([0]) + + +class TestPageUnitViewsStopAtTheImage: + """An image occupies WHOLE units, so the last one is mostly padding. + + `_checkpoint_copy_plan` already knows this — it hands `plan_segmented_copy` + the unit stream *and* `spec.image_bytes`, and that function walks both + streams from offset 0, so the image is by construction the LEADING + `image_bytes` of the unit stream and the tail belongs to nobody. + + `page_unit_views` is the tensor-view counterpart of the same addressing, so + it has to stop at the same place. Gathering whole units instead hands the + packer more bytes than `StateByteCodec.put` allocated, and every store dies + on "MemoryObj tensor is too small". + + None of the addressing tests above reach this: their spec carries no + `image_bytes`, so they take the untrimmed path by construction. + """ + + H = TestPageUnitAddressesAreArithmetic + + def build(self, **kw): + stub, cache = self.H.build(self.H(), **kw) + stub.page_unit_views = K3._KimiMLAGDNCommon.page_unit_views.__get__( + stub, type(stub) + ) + return stub, cache + + @staticmethod + def total(views): + return sum(v.numel() * v.element_size() for v in views) + + def test_the_gathered_stream_is_exactly_the_image(self): + """The invariant the packer depends on, and the one that was broken.""" + stub, _ = self.build(image_bytes=70) # 3 units x 24 B = 72 B available + assert self.total(stub.page_unit_views([0, 1, 2])) == 70 + + def test_a_whole_unit_past_the_image_is_dropped_entirely(self): + stub, _ = self.build(image_bytes=20) + views = stub.page_unit_views([0, 1, 2]) + assert self.total(views) == 20 + assert len(views) == 3, "two whole views plus the one that straddles" + + def test_an_image_that_lands_on_a_boundary_is_not_sliced(self): + stub, _ = self.build(image_bytes=72) + untrimmed, _ = self.build() + assert [v.shape for v in stub.page_unit_views([0, 1, 2])] == [ + v.shape for v in untrimmed.page_unit_views([0, 1, 2]) + ] + + def test_the_kept_bytes_are_the_leading_bytes_and_nothing_else(self): + """Trimming must not reorder or reslice — it truncates. The blob is + read back by scattering into the slot in the same order, so a byte that + moves here lands in the wrong layer there, silently.""" + stub, _ = self.build(image_bytes=70) + untrimmed, _ = self.build() + want = torch.cat( + [ + v.reshape(-1).view(torch.uint8) + for v in untrimmed.page_unit_views([0, 1, 2]) + ] + )[:70] + got = torch.cat( + [v.reshape(-1).view(torch.uint8) for v in stub.page_unit_views([0, 1, 2])] + ) + assert torch.equal(got, want) + + def test_a_multi_byte_dtype_is_sliced_by_bytes_not_elements(self): + """The straddling view is reinterpreted as uint8 before slicing, because + the image does not end on an element boundary in general. A slice taken + in elements would silently keep the wrong amount on any dtype wider than + a byte.""" + stub, cache = self.build(dtype=torch.bfloat16, image_bytes=70) + assert cache.element_size() == 2 + assert self.total(stub.page_unit_views([0, 1, 2])) == 70 + + def test_a_spec_with_no_image_size_keeps_whole_units(self): + """A fork build carries no spec at all, and there the whole-unit stream + is the right answer — there is nothing to trim against.""" + stub, _ = self.build() # no image_bytes on the spec + assert self.total(stub.page_unit_views([0, 1, 2])) == 3 * self.H.ROWS * ( + self.H.LOGICAL_BS * self.H.ENTRY + ) + + def test_units_that_cannot_cover_the_image_raise(self): + """A short blob read back as valid is the one outcome worth crashing + over: it resumes a request onto a truncated state with no exception.""" + stub, _ = self.build(image_bytes=80) # more than the 72 B on offer + with pytest.raises(RuntimeError, match="disagree"): + stub.page_unit_views([0, 1, 2]) From 457332a7b52d19f98f532d73947b7067a00fe8a7 Mon Sep 17 00:00:00 2001 From: zejunchen-zejun Date: Fri, 28 Aug 2026 09:54:55 +0800 Subject: [PATCH 018/151] style: clear the ruff findings CI reports on this PR CI's reviewdog run on #2053 flagged five findings, all introduced by this branch. Every one is real and still present. atom/kv_transfer/offload/hybrid/dsv4/connector.py:27 I001 atom/kv_transfer/offload/hybrid/kimi_k3/connector.py:11 I001 atom/kv_transfer/offload/hybrid/kimi_k3/connector.py:425 SIM102 tests/test_scheduler.py:1466 I001 tests/test_scheduler.py:1498 I001 plus one more the bot has not reported yet because reviewdog filters to diff context and that line has not changed since: atom/model_engine/page_unit_checkpoint.py:6 I001 --- why the local checks said clean `ruff check` with no `--select` runs the default rule set (E4/E7/E9/F), and neither `I` (isort) nor `SIM` is in it. There is no `[tool.ruff]` in `pyproject.toml` and no `ruff.toml`, so every local run in this branch's history measured the default set and reported "36, unchanged" against the base. That number was true and I reported it as if it meant CI would be clean, which it does not. Checked from now on with `--select I,SIM` as well. --- scope `--fix` over the whole tree repairs 100 findings in files this branch never touched; that was reverted. Fixed here are only the findings this branch introduced, established per file by running the same rules against the same file on the base: for f in $(git diff --name-only HEAD -- '*.py'); do compare ruff --select I,SIM on :$f against $f done Six files come back with more findings than their base; the six above are them. After this commit that comparison returns nothing. The 14 findings that remain across files this branch touches are all pre-existing on untouched lines, which is also why reviewdog does not report them. SIM102 is `should_defer_free`'s stalled-save escape: two nested `if`s collapsed into one condition, same short-circuit order, no behaviour change. Verified: pytest tests/ (less tests/plugin and the two msgpack/aiter collection errors) -> 3936 passed, zero failures. black clean; ruff default set 36, unchanged; ruff `--select I,SIM` shows zero findings introduced by this branch. Co-Authored-By: Claude Opus 5 (1M context) --- atom/kv_transfer/offload/hybrid/dsv4/connector.py | 2 +- .../kv_transfer/offload/hybrid/kimi_k3/connector.py | 13 ++++++++----- atom/model_engine/page_unit_checkpoint.py | 2 +- tests/test_scheduler.py | 6 ++++-- 4 files changed, 14 insertions(+), 9 deletions(-) diff --git a/atom/kv_transfer/offload/hybrid/dsv4/connector.py b/atom/kv_transfer/offload/hybrid/dsv4/connector.py index 00f8fa3fc1..a3f1e91279 100644 --- a/atom/kv_transfer/offload/hybrid/dsv4/connector.py +++ b/atom/kv_transfer/offload/hybrid/dsv4/connector.py @@ -52,10 +52,10 @@ from atom.kv_transfer.offload import config as offcfg from atom.kv_transfer.offload._block_gpu_connector import BlockGPUConnector from atom.kv_transfer.offload._offload_common import ( - max_pending_saves, OffloadSchedulerMixin, OffloadWorkerMixin, build_offload_engine, + max_pending_saves, pp_aware_rank_and_world, validated_kv_role, ) diff --git a/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py b/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py index daf68b3036..a00ac99c85 100644 --- a/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py +++ b/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py @@ -27,8 +27,8 @@ from atom.kv_transfer.offload.hybrid.kimi_k3.staging import StagedTransfer from atom.kv_transfer.offload.hybrid.kimi_k3.state_object import StateByteCodec from atom.kv_transfer.offload.hybrid.kimi_k3.state_tier import ( - _JointPark, StateOffloadTier, + _JointPark, ) from atom.kv_transfer.offload.metadata import LMCacheOffloadMetadata @@ -422,10 +422,13 @@ def should_defer_free(self, seq) -> bool: whether it ever answered. """ sid = str(seq.id) - if self._save_stalled and sid not in self._save_inflight: - if self._has_pending_save(seq): - self._save_tracker.pop(sid, None) - return False + if ( + self._save_stalled + and sid not in self._save_inflight + and self._has_pending_save(seq) + ): + self._save_tracker.pop(sid, None) + return False return super().should_defer_free(seq) # -- joint boundary ---------------------------------------------------- diff --git a/atom/model_engine/page_unit_checkpoint.py b/atom/model_engine/page_unit_checkpoint.py index e23687be60..f2167b6336 100644 --- a/atom/model_engine/page_unit_checkpoint.py +++ b/atom/model_engine/page_unit_checkpoint.py @@ -6,9 +6,9 @@ from __future__ import annotations from collections import OrderedDict, deque -from time import monotonic from collections.abc import Iterator, Mapping from dataclasses import dataclass +from time import monotonic from atom.model_engine.block_pool import BlockPool from atom.model_engine.sequence import Sequence diff --git a/tests/test_scheduler.py b/tests/test_scheduler.py index 45e4826800..66dfd4dfad 100644 --- a/tests/test_scheduler.py +++ b/tests/test_scheduler.py @@ -1463,9 +1463,10 @@ def test_it_self_throttles_so_a_1ms_poll_is_cheap(self, monkeypatch): assert freed == [1] def test_a_non_positive_window_restores_wait_forever(self, monkeypatch): - import atom.model_engine.scheduler as sched_mod import time as _time + import atom.model_engine.scheduler as sched_mod + monkeypatch.setattr(sched_mod, "_SAVE_ABANDON_TIMEOUT_S", 0.0) s = object.__new__(sched_mod.Scheduler) s.deferred_free_blocks = { @@ -1495,8 +1496,9 @@ def test_the_window_sits_above_lmcaches_own_pin_timeout(self, monkeypatch): def test_a_late_report_on_a_reclaimed_save_is_not_an_assertion(self): """`finished_sending` used to assert the sequence was still deferred.""" - import atom.model_engine.scheduler as sched_mod import inspect + import atom.model_engine.scheduler as sched_mod + src = inspect.getsource(sched_mod.Scheduler._update_from_kv_xfer_finished) assert "not found in deferred_free_blocks" not in src From 515260be4e88f20de927d82da7e472bed34b5327 Mon Sep 17 00:00:00 2001 From: zejchen Date: Fri, 28 Aug 2026 02:27:24 +0000 Subject: [PATCH 019/151] fix(offload): the delegating shell's take_state_reports fallback returned a 3-tuple The shell LMCacheOffloadConnectorScheduler.take_state_reports() forwards to self._impl, but its no-tier fallback returned three empty sets: return callback() if callback is not None else (set(), set(), set()) The contract is a 2-tuple (indexed, failed): the kimi_k3 impl returns exactly that, and the sole caller unpacks exactly that -- scheduler._update_from_kv_xfer_finished, line 3453: indexed, failed = take() Any offload variant whose _impl does not define take_state_reports (dense, or a non-k3 hybrid) but whose block_manager still exposes a state_offload -- so the `offload is not None and take is not None` guard passes -- therefore hit the fallback and unpacked (set(), set(), set()) into two names, raising ValueError: too many values to unpack (expected 2) inside the engine busy_loop's _poll_kv_transfer_progress. That kills the engine worker; the surviving TP ranks then block forever in the next collective (observed as a storm of c10d sendBytes errors and "worker still alive after 5s; terminating"), so the server accepts requests but returns nothing -- a silent full wedge, not a crash the client sees. Reproduced by running K3 with offload_layout=dense (KV offload, state tier off) at concurrency 16: server reaches READY, then the first scheduled batch wedges all workers; aiperf sits at returned=0/177, in_flight=17, errors=0 until the warmup grace period expires. Fix: return the 2-tuple empty set (set(), set()) so the no-report path unpacks correctly. Same class of bug as 8a5364ab (the shell dropped enqueue_state_stores entirely) -- the delegating shell's fallbacks are not checked against the delegated method's return contract. Co-Authored-By: Claude Opus 4.8 --- atom/kv_transfer/offload/connector.py | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/atom/kv_transfer/offload/connector.py b/atom/kv_transfer/offload/connector.py index 581b4b123b..6332817ec4 100644 --- a/atom/kv_transfer/offload/connector.py +++ b/atom/kv_transfer/offload/connector.py @@ -187,8 +187,15 @@ def enqueue_state_stores(self, stores) -> bool: return bool(callback(stores)) if callback is not None else False def take_state_reports(self): + # Contract is a 2-tuple (indexed, failed) -- matched by the kimi_k3 impl + # and by the sole caller `indexed, failed = take()` in + # scheduler._update_from_kv_xfer_finished. The no-tier fallback returned + # a 3-tuple, so any offload variant whose _impl lacks take_state_reports + # (e.g. dense/hybrid non-k3) but still exposes state_offload crashed the + # engine mid-run with "too many values to unpack (expected 2)", wedging + # all TP workers on the next collective. Return the 2-tuple empty set. callback = getattr(self._impl, "take_state_reports", None) - return callback() if callback is not None else (set(), set(), set()) + return callback() if callback is not None else (set(), set()) def get_statistics(self) -> dict[str, int]: return self._impl.get_statistics() From 88a4b3651a4457df6fb8d472904f49329c4b2456 Mon Sep 17 00:00:00 2001 From: zejchen Date: Fri, 28 Aug 2026 03:02:30 +0000 Subject: [PATCH 020/151] feat(offload): log an HBM-vs-lmcache tier split so the value of the CPU tier is visible The `[Cache Pools]` line splits reuse the way the *pipeline* is built -- paged pool then state pool, in series -- which answers "which pool is the bottleneck" but cannot answer the question this PR exists to answer: what does attaching the lmcache CPU offload tier actually buy on a given workload? For that you need the *tier* split, not the pipeline split: of the reuse we served, how much came from HBM (the paged pool) versus from lmcache (the CPU offload tier). The counter for that split already exists -- `total_offload_tokens`, accumulated in `CacheStats.update` from the CPU-tier served count, which by construction sits *above* the HBM walk (it is reuse the paged pool never had to satisfy). So it is exactly the reuse that reverts to recompute the moment the tier is switched off -- i.e. the value of running lmcache, in one number. The pieces were all present; nothing surfaced them side by side against the same denominator. This adds: * `CacheStats.lmcache_hit_rate` = total_offload_tokens / total_reusable_tokens, the mirror of `paged_hit_rate` (the HBM tier's share of the same denominator); * a `[Cache Tiers]` log line in `_log_pools`, emitted only when a tier is attached (total_offload_tokens>0). The no-connector baseline has total_offload_tokens==0, so its tier split is trivially HBM=paged, lmcache=0 -- the exact contrast we want to read off two runs. Motivation: the ON-vs-OFF necessity comparison for K3 needs the hit rate broken down per tier, otherwise a single combined rate hides whether the reuse came from HBM (which the OFF baseline also has) or from lmcache (which only the ON arm has). For K3 the two tiers are cleanly separable -- KV prefixes live in HBM (paged), recurrent-state checkpoints live in the CPU tier (offload) -- so when the joint boundary reports state_hbm=0 the entire state-hit is lmcache-served and the OFF arm loses all of it. Co-Authored-By: Claude Opus 4.8 --- atom/model_engine/scheduler.py | 35 ++++++++++++++++++++++++++++++++++ 1 file changed, 35 insertions(+) diff --git a/atom/model_engine/scheduler.py b/atom/model_engine/scheduler.py index de04d0bf08..3dcac14273 100644 --- a/atom/model_engine/scheduler.py +++ b/atom/model_engine/scheduler.py @@ -459,6 +459,23 @@ def state_recoverable_loss_rate(self) -> float: self.total_compressed_tokens, ) + @property + def lmcache_hit_rate(self) -> float: + """Reuse the lmcache CPU offload tier served, as a share of all reuse. + + Disjoint from the HBM walk on purpose: `total_offload_tokens` counts + reuse that sat *above* the paged pool (see `update`), so it is exactly + the reuse that becomes recompute the moment the tier is switched off. + Paired with `paged_hit_rate` (the HBM tier's share) it answers the + question the pipeline rates cannot -- of the reuse we served, how much + came from HBM vs from lmcache -- which is the whole case for the tier + on a given workload. For K3 the two are cleanly separable: KV prefixes + live in HBM (paged) while the recurrent-state checkpoints live in the + CPU tier (offload), and when the joint boundary reports state_hbm=0 the + entire state-hit is lmcache-served. + """ + return self._rate(self.total_offload_tokens, self.total_reusable_tokens) + def get_statistics(self) -> dict: """Counters, not rates — the caller derives those. @@ -586,6 +603,24 @@ def _log_pools(self) -> None: f"combined: {self.hit_rate:.2%}, " f"binding: {worse}" ) + # `[Cache Pools]` above is a *series* split (paged pool -> state pool), + # which answers "which pool to fix" but NOT "what does lmcache buy us". + # For that you need the *tier* split: how much of the reuse we served + # came from HBM (paged pool) vs from the lmcache CPU offload tier. The + # offload count sits above the HBM walk (see `update`), so it is exactly + # the reuse that reverts to recompute when the tier is switched off -- + # i.e. the value of running lmcache on this workload, in one number. + # Logged only when a tier is attached; the no-connector baseline has + # total_offload_tokens==0 and its tier split is trivially HBM=paged, + # lmcache=0. + if self.total_offload_tokens: + logger.info( + "[Cache Tiers] " + f"HBM-served (KV prefix resident): {paged:.2%} " + f"({self.total_compressed_tokens}/{self.total_reusable_tokens}), " + f"lmcache-served (CPU tier): {self.lmcache_hit_rate:.2%} " + f"({self.total_offload_tokens}/{self.total_reusable_tokens})" + ) @staticmethod def _log_pressure(p: dict[str, int]) -> None: From b36f3a3484e1e061e07ed2b434a2d28df02d4eb0 Mon Sep 17 00:00:00 2001 From: zejunchen-zejun Date: Fri, 28 Aug 2026 15:40:11 +0800 Subject: [PATCH 021/151] fix(offload): the tier split paired a reach against a served count Review of `8f712a95`. The `[Cache Tiers]` line is meant to answer "what does the CPU tier buy", so its two halves have to be two halves of one thing. They were not. --- what was wrong HBM-served <- paged_hit_rate = compressed / reusable lmcache-served <- lmcache_hit_rate = offload / reusable `compressed` is how far the prefix walk reached **before the state gates cut it** -- reach, not reuse anybody got. `offload` is served. Pairing them means the halves do not partition, and on K3's ordinary anchor-only shape they visibly exceed the ceiling: walk reaches 8 blocks, the only resumable rung is at 3, the joint boundary lands at 10, so the line prints 80% + 70% = 150% of a denominator that is by construction the maximum. That is not a corner case -- one checkpoint per prompt at the previous prompt's end IS the recommended placement, so a deep gate cut with a joint boundary above it is the normal shape. --- the fix Use `hit_rate` (`cached`) for the HBM half. `cached` is what the walk actually handed over (`min(num_cached, wanted)`), `offload` is what the tier added on top, and `cached + offload == num_cached <= reusable` by construction -- so the two partition and their sum is exactly the end-to-end rate, which the line now prints as `combined`. The property's docstring said "pairs with `paged_hit_rate`"; it now says why it does not, since both read as "the HBM number" and only one is served. --- tests `TestTheTierSplitPartitionsServedReuse`, four cases: the halves sum to the end-to-end rate; they never exceed the denominator **on the shape that makes the wrong pairing exceed it** (asserted as a precondition, so the test cannot pass vacuously); the emitted line carries `cached/reusable` and not `compressed/reusable`; no tier attached emits no line. Falsified: swapping `hit_rate` back for `paged_hit_rate` fails all three. --- second change: the shell sweep now checks fallback shape, not just presence `7ddbe3d0` fixed `take_state_reports`'s no-impl fallback returning a 3-tuple to a caller that unpacks two. The sweep test added in `19d2b79d` would not have caught it: it asserts a member is reachable, not that its fallback is consumable. A wrong-shaped fallback is the worse of the two -- it took the engine worker down mid-run, and every surviving TP rank then blocked forever in the next collective, so the server accepted requests and returned nothing. `test_the_shells_no_impl_fallbacks_match_what_the_caller_unpacks` drives the shell with an `_impl` that defines nothing and holds all seven fallbacks against the contract their callers rely on. Checked by hand first: all seven conform today, so this pins the repair rather than reporting a new fault. Verified: pytest tests/ (less tests/plugin and the two msgpack/aiter collection errors) -> 3941 passed, zero failures. black clean; ruff default set unchanged; `--select I,SIM` shows zero findings introduced by this branch. Co-Authored-By: Claude Opus 5 (1M context) --- atom/model_engine/scheduler.py | 41 ++++++++---- tests/test_lmcache_offload_connector.py | 33 ++++++++++ tests/test_scheduler.py | 85 +++++++++++++++++++++++++ 3 files changed, 146 insertions(+), 13 deletions(-) diff --git a/atom/model_engine/scheduler.py b/atom/model_engine/scheduler.py index 3dcac14273..b1c9be37c3 100644 --- a/atom/model_engine/scheduler.py +++ b/atom/model_engine/scheduler.py @@ -464,15 +464,17 @@ def lmcache_hit_rate(self) -> float: """Reuse the lmcache CPU offload tier served, as a share of all reuse. Disjoint from the HBM walk on purpose: `total_offload_tokens` counts - reuse that sat *above* the paged pool (see `update`), so it is exactly - the reuse that becomes recompute the moment the tier is switched off. - Paired with `paged_hit_rate` (the HBM tier's share) it answers the - question the pipeline rates cannot -- of the reuse we served, how much - came from HBM vs from lmcache -- which is the whole case for the tier - on a given workload. For K3 the two are cleanly separable: KV prefixes - live in HBM (paged) while the recurrent-state checkpoints live in the - CPU tier (offload), and when the joint boundary reports state_hbm=0 the - entire state-hit is lmcache-served. + reuse claimed *above* what the walk could offer (see `update`), so it is + exactly the reuse that becomes recompute the moment the tier is switched + off -- the whole case for the tier on a given workload, in one number. + + **Pairs with `hit_rate`, not with `paged_hit_rate`.** Both of those read + as "the HBM number" and only `hit_rate` is a served quantity: + `paged_hit_rate`'s numerator is `compressed`, how far the prefix walk + reached *before* the state gates cut it. `cached + offload == + num_cached <= reusable` by construction, so `hit_rate` and this + partition the served reuse; `compressed + offload` does not and can + exceed the denominator. """ return self._rate(self.total_offload_tokens, self.total_reusable_tokens) @@ -611,15 +613,28 @@ def _log_pools(self) -> None: # the reuse that reverts to recompute when the tier is switched off -- # i.e. the value of running lmcache on this workload, in one number. # Logged only when a tier is attached; the no-connector baseline has - # total_offload_tokens==0 and its tier split is trivially HBM=paged, + # total_offload_tokens==0 and its tier split is trivially HBM=all, # lmcache=0. + # + # The HBM half is `hit_rate` (`cached`), NOT `paged_hit_rate` + # (`compressed`). Both read as "the HBM number" and only one of them is + # a *served* quantity: `compressed` is how far the prefix walk reached + # before the state gates cut it, so it is reach, not reuse anybody got. + # Pairing it against `offload` mixes the two and the halves stop + # partitioning -- on K3's ordinary anchor-only shape (walk reaches 8 + # blocks, the only resumable rung is at 3, the joint boundary lands at + # 10) it prints 80% + 70% = 150% of a denominator that is the ceiling. + # `cached + offload == num_cached <= reusable` by construction, so these + # two do partition, and `combined` is exactly the end-to-end rate. if self.total_offload_tokens: + served = self.total_cached_tokens + self.total_offload_tokens logger.info( "[Cache Tiers] " - f"HBM-served (KV prefix resident): {paged:.2%} " - f"({self.total_compressed_tokens}/{self.total_reusable_tokens}), " + f"HBM-served (KV prefix resident): {self.hit_rate:.2%} " + f"({self.total_cached_tokens}/{self.total_reusable_tokens}), " f"lmcache-served (CPU tier): {self.lmcache_hit_rate:.2%} " - f"({self.total_offload_tokens}/{self.total_reusable_tokens})" + f"({self.total_offload_tokens}/{self.total_reusable_tokens}), " + f"combined: {self._rate(served, self.total_reusable_tokens):.2%}" ) @staticmethod diff --git a/tests/test_lmcache_offload_connector.py b/tests/test_lmcache_offload_connector.py index d271ea98b9..c7ee14d8d1 100644 --- a/tests/test_lmcache_offload_connector.py +++ b/tests/test_lmcache_offload_connector.py @@ -5051,3 +5051,36 @@ def test_every_member_the_scheduler_reads_is_reachable_through_the_shell(): "Scheduler reads these off `kv_connector`, but the delegating shell " f"does not expose them, so each silently takes its default: {missing}" ) + + +def test_the_shells_no_impl_fallbacks_match_what_the_caller_unpacks(): + """Existence is not enough -- a fallback of the wrong SHAPE is worse. + + The sweep above catches a member the shell never defines. It does not catch + a member the shell defines with a fallback the caller cannot consume, which + is how `take_state_reports` returned three empty sets to + `indexed, failed = take()` and took the engine worker down mid-run: every + surviving TP rank then blocked forever in the next collective, so the server + accepted requests and returned nothing. + + Drive the shell with an `_impl` that defines nothing, and hold each + fallback against the contract its caller relies on. + """ + from types import SimpleNamespace + + from atom.kv_transfer.offload.connector import LMCacheOffloadConnectorScheduler + + shell = object.__new__(LMCacheOffloadConnectorScheduler) + shell._impl = SimpleNamespace() + + indexed, failed = shell.take_state_reports() # `indexed, failed = take()` + assert (indexed, failed) == (set(), set()) + + assert shell.enqueue_state_stores([]) is False # `bool(enqueue(stores))` + assert shell.enqueue_state_loads([]) is False + # `int(getattr(..., "chunk_size", 0) or 0)` + assert shell.chunk_size is None or isinstance(shell.chunk_size, int) + assert shell.should_park_partial_prefill_for_load(None) is False + assert shell.load_finished(None) is True + # Unchanged, so a connector with no opinion does not shrink the chunk. + assert shell.adjust_prefill_chunk_after_alloc(None, 7) == 7 diff --git a/tests/test_scheduler.py b/tests/test_scheduler.py index 66dfd4dfad..db4e1c6994 100644 --- a/tests/test_scheduler.py +++ b/tests/test_scheduler.py @@ -1502,3 +1502,88 @@ def test_a_late_report_on_a_reclaimed_save_is_not_an_assertion(self): src = inspect.getsource(sched_mod.Scheduler._update_from_kv_xfer_finished) assert "not found in deferred_free_blocks" not in src + + +class TestTheTierSplitPartitionsServedReuse: + """`[Cache Tiers]` exists to answer "what does the CPU tier buy", so its two + halves have to be two halves of one thing. + + `cached` and `offload` are that: `cached` is what the HBM walk actually + handed over, `offload` is what the tier added on top, and they sum to + `num_cached`. `compressed` is NOT -- it is how far the walk reached before + the state gates cut it, so it counts reuse nobody got. + """ + + @staticmethod + def stats(**kw): + from atom.model_engine.scheduler import CacheStats + + s = CacheStats() + s.update(**kw) + return s + + def test_the_two_halves_sum_to_the_end_to_end_rate(self): + s = self.stats( + num_cached_tokens=300, + num_full_tokens=1200, + num_compressed_tokens=800, + num_wanted_tokens=300, + num_reusable_tokens=1000, + num_offload_tokens=700, + ) + assert s.hit_rate + s.lmcache_hit_rate == pytest.approx(1.0) + + def test_the_halves_never_exceed_the_denominator(self): + """K3's ordinary anchor-only shape: the walk reaches 8 blocks, the only + resumable rung is at 3, the joint boundary lands at 10. Pairing + `compressed` against `offload` prints 80% + 70% here -- 150% of a + denominator that is the ceiling.""" + s = self.stats( + num_cached_tokens=300, # the gate cut the walk from 800 to 300 + num_full_tokens=1200, + num_compressed_tokens=800, + num_wanted_tokens=300, + num_reusable_tokens=1000, + num_offload_tokens=700, + ) + assert s.paged_hit_rate + s.lmcache_hit_rate > 1.0, ( + "precondition: this is the shape that makes the wrong pairing " + "exceed 100%, so the assertion below is not vacuous" + ) + assert s.hit_rate + s.lmcache_hit_rate <= 1.0 + + def test_the_line_reports_cached_not_compressed(self, caplog): + """Reads the emitted text: swapping `hit_rate` back for + `paged_hit_rate` has to be what fails here.""" + import logging + + s = self.stats( + num_cached_tokens=300, + num_full_tokens=1200, + num_compressed_tokens=800, + num_wanted_tokens=300, + num_reusable_tokens=1000, + num_offload_tokens=700, + ) + with caplog.at_level(logging.INFO, logger="atom"): + s._log_pools() + line = next( + r.getMessage() for r in caplog.records if "[Cache Tiers]" in r.getMessage() + ) + assert "300/1000" in line, f"HBM half must be `cached`, got: {line}" + assert "800/1000" not in line, f"`compressed` is reach, not served: {line}" + assert "700/1000" in line + + def test_no_tier_attached_emits_no_tier_line(self, caplog): + import logging + + s = self.stats( + num_cached_tokens=300, + num_full_tokens=1200, + num_compressed_tokens=800, + num_wanted_tokens=300, + num_reusable_tokens=1000, + ) + with caplog.at_level(logging.INFO, logger="atom"): + s._log_pools() + assert not any("[Cache Tiers]" in r.getMessage() for r in caplog.records) From d895905709c91534f7294f6aad4e1febc69f3dd4 Mon Sep 17 00:00:00 2001 From: zejunchen-zejun Date: Fri, 28 Aug 2026 16:34:31 +0800 Subject: [PATCH 022/151] style: clear CI's ruff findings, and bring the comments down to package style --- CI Five findings on the ruff job, all introduced by this branch: kimi_k3/state_object.py:222,228 BLE001 + S110 tests/test_state_checkpoint.py:1594 RUF059 plus three more BLE001 on lines this branch changed that reviewdog had not reported yet. `_object_bytes`'s two `try/except: pass` blocks are gone rather than silenced: an allocator that raises when asked its own object's size is broken in a way this method must not paper over, and swallowing it turned every read into a silent miss. The unused unpack is `_h`. The three remaining blind excepts are deliberate and stay, with `# noqa: BLE001` and the reason -- `codec.put`/`get` reach into LMCache whose failure modes are its own, and a store or load that cannot happen must cost one checkpoint, not the worker thread whose death would strand every parked request. Same shape the DSV4 codec already uses. Local runs said "ruff 36, unchanged" throughout this branch's history because `ruff check` with no `--select` runs the default set (E4/E7/E9/F) and there is no `[tool.ruff]` in the repo; CI's ruff selects more. Checked against the reported rules from now on, filtered to the lines this branch actually changes, which is what reviewdog reports on. --- comments The offload package's own density is 8% (dsv4) to 15% (dense). The kimi_k3 files sat at 27-53%, four to six times their neighbours in the same package, which is the comparison a reviewer makes. Trimmed toward it: * the module docstring of `state_offload.py` lost a paragraph narrating what the design used to be -- history belongs in the log, and it also carried a reference to a file that is not in the repo; * the one-pool rationale in `_build_state_tier`, 25 lines to 8; * the allocator note in `_allocate`, 18 to 10, keeping both traps (`fmt` is inert, `busy_loop=False` or a full pool spins forever); * `page_unit_views`, `key`, `put`, `_queue_offload_store`, `take_offload_stores`, `_reachable`, `_offload_save_abandon_timeout_s`, `submit_store`, `submit_load`, `_start_state_stores` and the `StateOffloadTier` class docstring, each cut to the claim plus the reason it is not obvious. What is kept is the why behind a decision a bug taught us -- the pin window, the optimistic index, the `layout_id` in the key, the trim to `image_bytes`. Those read as over-explanation only until one of them is undone. -130 lines of comment, no behaviour change. Verified: pytest tests/ (less tests/plugin and the two msgpack/aiter collection errors) -> 4002 passed, zero failures. black clean; ruff default set 36 = main's; the rules CI reports show zero findings on this branch's lines. Co-Authored-By: Claude Opus 5 (1M context) --- .../offload/hybrid/kimi_k3/connector.py | 46 +++----- .../offload/hybrid/kimi_k3/state_object.py | 103 +++++++----------- .../offload/hybrid/kimi_k3/state_tier.py | 60 +++++----- atom/model_engine/block_manager.py | 6 +- atom/model_engine/page_unit_checkpoint.py | 73 +++++-------- atom/model_engine/scheduler.py | 45 +++----- atom/model_engine/state_offload.py | 8 -- .../model_ops/attentions/kimi_mla_gdn_attn.py | 43 +++----- tests/test_state_checkpoint.py | 2 +- 9 files changed, 151 insertions(+), 235 deletions(-) diff --git a/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py b/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py index a00ac99c85..2c211478f2 100644 --- a/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py +++ b/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py @@ -150,31 +150,14 @@ def _build_state_tier(self, transfer_tensors) -> None: worker_id=rank, layout_id=layout_id, ) - # ONE pool, shared with paged KV. A separate `atom-state-{rank}` engine - # used to be built here, on the argument that the KV write stream would - # evict the checkpoints. It does not hold up: state is 3-13% of a - # boundary's bytes, not a rounding error, and -- the part that decides - # it -- a request writes its KV chunks and its one state object inside - # the same prefill window, so they enter LMCache's LRU together and - # cool at the same rate. State is retired ALONGSIDE its own KV, which - # is what we want: a boundary whose KV is gone is worthless anyway. - # - # Two pools cost what one does not: independent eviction policies that - # must drift, and a joint boundary needs BOTH legs to survive together. - # #2045 made the same argument one tier up and won with it (state moved - # out of its reserved slots into the KV pool, 81.85% -> 93.60%). - # - # What one pool costs is a single `cache_policy` for both legs. That is - # real, and currently free: LRU is right for both. The FIFO override - # this used to set rested on "a state entry is written once and read - # once", which #2045's own numbers refute -- ~4,808 resumes over ~1,508 - # checkpoints is ~3 reads each, and bursty re-access is LRU's home - # ground. - # - # `OFFLOAD_STATE_CPU_SIZE` is gone with it. #2045 deleted the HBM-side - # reservation for exactly this reason (`extra_entries=0`, 1.7 GiB - # returned) -- keeping the same knob one tier down repeats the thing it - # disproved. `LMCACHE_MAX_LOCAL_CPU_SIZE` is the one size to tune. + # ONE pool, shared with paged KV, and one `cache_policy` with it. + # A request writes its KV chunks and its one state object inside the + # same prefill window, so both enter LMCache's LRU together and cool at + # the same rate -- state is retired alongside its own KV, which is what + # we want, since a boundary whose KV is gone is worthless. Two pools + # would buy independent eviction policies that must drift, while a joint + # boundary needs both legs to survive together. + # `LMCACHE_MAX_LOCAL_CPU_SIZE` is the one size to tune. codec.bind_storage_manager(self._engine.storage_manager) # No index here: StateOffloadIndex lives in the engine process; both # directions report and the engine applies. @@ -235,14 +218,13 @@ def _start_state_loads(self, metadata) -> None: def _start_state_stores(self, metadata) -> None: """Hand this step's ready checkpoints to the tier's executor. - No producer fence and no staging copy. The source is the checkpoint's - PAGE units, which #2045 reserves out of the KV pool and the engine pins - for the duration -- so nothing on the compute stream is writing them - and the packer gathers straight from where they sit. + No producer fence and no staging copy: the source is the checkpoint's + PAGE units, reserved out of the KV pool and pinned by the engine, so the + packer gathers straight from where they sit. - A store with no tier is reported failed rather than dropped: the engine - is holding those units pinned against a report, and silence would leave - them to the reconciler's full timeout. + A store with no tier is reported failed rather than dropped -- the + engine holds those units pinned against a report, and silence would + leave them to the reconciler's full timeout. """ stores = getattr(metadata, "state_stores", None) if not stores: diff --git a/atom/kv_transfer/offload/hybrid/kimi_k3/state_object.py b/atom/kv_transfer/offload/hybrid/kimi_k3/state_object.py index da63945c36..59455c244b 100644 --- a/atom/kv_transfer/offload/hybrid/kimi_k3/state_object.py +++ b/atom/kv_transfer/offload/hybrid/kimi_k3/state_object.py @@ -75,30 +75,21 @@ def bind_storage_manager(self, storage_manager) -> None: def key(self, h: int): """ATOM's hash, bound to the geometry the bytes were written under. - **Two jobs, and the second one is why this is not the bare hash.** - - *Build safety.* The same prefix hash maps to a completely different - image under a different `num_spec`, TP size, or conv/ssm dtype. A size - mismatch would be caught by `entry_bytes`; the same size with a - different order or meaning reads back silently wrong state, and a - request resumed onto wrong state produces wrong output with nothing - raised anywhere. #2045 already encodes all of it in `layout_id` - (layers / conv shape+dtype / ssm shape+dtype / order / tp / spec / - carry) and enforces it HBM-side in `_validate_paged_state_op`; this is - the CPU side of the same check. + *Build safety.* One prefix hash maps to a different image under a + different `num_spec`, TP size, or conv/ssm dtype. A size mismatch is + caught by `entry_bytes`; the same size with a different order reads back + silently wrong state, and a request resumed onto it produces wrong + output with nothing raised. `layout_id` already names all of it and is + enforced HBM-side -- this is the CPU side of the same check. *Namespace separation.* `CacheEngineKey` has no field saying what an - entry IS -- KV chunk keys come from `ChunkedTokenDatabase`'s chunk hash - and state keys from ATOM's chained block hash, both plain integers in - one space. Since Phase 3a they also share one pool. They were separated - only accidentally, by this side hard-coding `torch.uint8` while KV - carries the KV dtype. Folding `layout_id` in makes them disjoint by - construction: no KV key can ever carry a K3 layout id. + entry IS, KV and state keys come from different hash functions into one + integer space, and both now share one pool. Folding `layout_id` in makes + the two spaces disjoint by construction. - Not a plain `hash((h, layout_id))`: Python's `hash` of a str is salted - per process (`PYTHONHASHSEED`), so a restart would silently orphan every - entry written by the previous run. `xxh64` is what the block hashes - themselves use. + `xxh64`, not `hash((h, layout_id))`: Python salts `hash` of a str per + process, so a restart would silently orphan every entry the previous run + wrote. """ import xxhash from lmcache.utils import CacheEngineKey @@ -123,20 +114,15 @@ def key(self, h: int): def put(self, h: int, unit_ids) -> bool: """Store one checkpoint image. False when nothing was stored. - The source is the checkpoint's PAGE units, not an Active Slot: under - #2045 that is where the image lives, and `page_unit_views` names those - bytes as the contiguous tensors the packer takes. The two directions - are deliberately asymmetric -- a load writes the slot (`get`) -- and - they compose because #2045's copy plan intersects two *ordered byte - streams*, so a blob gathered in unit order is byte-identical to one - gathered in slot order. + Reads the checkpoint's PAGE units, while `get` writes an Active Slot. + The asymmetry is deliberate and safe: the copy plan intersects two + *ordered byte streams*, so a blob gathered in unit order is + byte-identical to one gathered in slot order. - A refusal is not an error. `_allocate` returns None under CPU memory - pressure with nothing evictable, and a 53.6 MiB request is refused - sooner than a KV chunk's 884 KB, so the state leg is the first to feel - a full pool. Best-effort by design: the cost is one later miss, and - `puts_refused` is what makes it visible instead of a silent hit-rate - drift. + A refusal is not an error -- `_allocate` returns None under CPU memory + pressure, and a whole image is refused sooner than a KV chunk, so the + state leg feels a full pool first. `puts_refused` makes that visible + instead of a silent hit-rate drift. """ if self._storage is None: return False @@ -211,22 +197,23 @@ def _object_bytes(obj) -> int | None: """Bytes in a `MemoryObj`, or None when it will not say. `get_size()` is the documented accessor; the tensor is the fallback for - the buffer-backed objects that have no size of their own. None means - "cannot check", never "size 0" -- refusing a load we cannot measure - would turn an unknown into a guaranteed miss. + the buffer-backed objects that have no size of their own (their + `.tensor` is None). None means "cannot measure", never "size 0" -- + refusing a load we cannot measure would turn an unknown into a + guaranteed miss. + + Neither accessor is wrapped: an allocator that raises when asked its own + object's size is broken in a way this method must not paper over, and + swallowing it here would turn every read into a silent miss. """ get_size = getattr(obj, "get_size", None) if callable(get_size): - try: - return int(get_size()) - except Exception: - pass + size = get_size() + if size is not None: + return int(size) tensor = getattr(obj, "tensor", None) if tensor is not None: - try: - return int(tensor.numel()) * tensor.element_size() - except Exception: - pass + return int(tensor.numel()) * tensor.element_size() return None def contains(self, h: int) -> bool: @@ -240,24 +227,16 @@ def contains(self, h: int) -> bool: def _allocate(self, nbytes: int) -> Any: from lmcache.v1.memory_management import MemoryFormat - # `MixedMemoryAllocator.allocate` accepts BINARY_BUFFER (-> a - # BytesBufferMemoryObj, whose `.tensor` is None and would fail - # `StagedTransfer.memory_tensor`) or one of the tensor formats - # {KV_2LTD, KV_2TD, KV_T2D, KV_MLA_FMT, EC_TD}; anything else, BINARY - # included, hits its `raise ValueError`. The shape/dtype above already - # force a flat opaque blob regardless of `fmt`, so the value is inert - # apart from passing that check -- same reasoning as the KV path's - # `self._engine.fmt = MemoryFormat.KV_2LTD` in connector.py. + # `fmt` is inert: the shape/dtype above already force a flat opaque + # blob. It is passed only because `MixedMemoryAllocator.allocate` + # rejects anything outside its tensor formats -- same reason the KV path + # sets `self._engine.fmt`. + # # `busy_loop=False` because this is a *store*. LMCache's own - # `LocalCPUBackend.allocate` docstring is explicit that busy_loop "should - # only be used for retrieve", since "many stores happen concurrently (if - # they busy_loop, deadlock happens)". Its default is True, and under that - # default a pool with no eviction candidate spins `while True` on 0.1s - # sleeps with no attempt bound -- so it would never return the None this - # method's caller is written to handle, and would instead hang the state - # tier's save worker for as long as CPU memory stays full. LMCache's own - # store paths pass False here for the same reason (`cache_engine.py:666`, - # `local_disk_backend.py:468`, `storage_manager.py:96`). + # `LocalCPUBackend.allocate` says busy_loop "should only be used for + # retrieve", since "many stores happen concurrently (if they busy_loop, + # deadlock happens)" -- and its default is True, under which a full pool + # spins forever instead of returning the None this caller handles. return self._storage.allocate( torch.Size([nbytes]), torch.uint8, diff --git a/atom/kv_transfer/offload/hybrid/kimi_k3/state_tier.py b/atom/kv_transfer/offload/hybrid/kimi_k3/state_tier.py index cd24d1da68..16bba16fd2 100644 --- a/atom/kv_transfer/offload/hybrid/kimi_k3/state_tier.py +++ b/atom/kv_transfer/offload/hybrid/kimi_k3/state_tier.py @@ -1,18 +1,15 @@ # SPDX-License-Identifier: MIT -"""Worker-side spill and load driver for the state offload tier. - -One executor of its own, separate from the KV connector's `_load_executor` and -`_save_executor` so that state transfers and KV transfers cannot block each -other -- but a single one, shared by this tier's own stores and loads. See -`submit_load` for why the KV connector's load/save split is not copied here. - -This class **reports, and the engine applies**. It runs in a spawned runner -process while `StateOffloadIndex` lives in the engine process, so this side -cannot index a hash directly. `take_store_reports` and -`get_finished` hand their sets to the connector, which emits the first as -`ConnectorCompletion`s on its own channel and merges the second into -`finished_loading`/`failed_loading`. The failed-hash set exists to resolve the -aggregator's quorum: without it a partial store would pin the hash forever. +"""Worker-side store and load driver for the state offload tier. + +One executor of its own, separate from the KV connector's, so state and KV +transfers cannot block each other -- but a single one shared by this tier's own +stores and loads; see `submit_load`. + +This class **reports, and the engine applies**. `StateOffloadIndex` lives in the +engine process, so nothing here can index a hash directly: `take_store_reports` +and `get_finished` hand their sets to the connector. The failed-hash set exists +to resolve the aggregator's quorum -- without it a partial store pins the hash +forever. """ import logging @@ -61,27 +58,21 @@ def _discard(f): def submit_store(self, h: int, unit_ids) -> None: """Pack the checkpoint image in `unit_ids` under `h`, for LMCache. - No `ready_event`, unlike the old spill. The staging ring is gone: the - units are reserved out of the KV pool and pinned by the engine for the - length of this transfer, so nothing on the compute stream is producing - or overwriting them and `StagedTransfer.pack` gathers straight from - where they sit -- one copy instead of two, and none of it on the - forward's stream. + No `ready_event`: the units are reserved out of the KV pool and pinned + by the engine for the length of this transfer, so nothing on the compute + stream is writing them and the packer gathers straight from where they + sit. """ self._register(self._executor.submit(self._do_store, h, unit_ids)) def submit_load(self, req_id: str, h: int, slot: int) -> None: """Fetch `h` into pool slot `slot` for the parked request `req_id`. - Shares one serial executor with the stores, unlike the KV connector's - split pair: `StagedTransfer` keeps its staging buffer in - `threading.local`, so a second thread costs a second resident buffer - per rank out of HBM the state pool is already short of. - - A load is on the TTFT critical path and a store is not, so once stores - become frequent (Phase 3 fires one per checkpoint rather than one per - eviction) this queue wants a priority, or a load can wait behind - several stores. Measure `state_load_queue_wait_ms` before adding one. + Shares one serial executor with the stores: `StagedTransfer` keeps its + staging buffer in `threading.local`, so a second thread costs a second + resident buffer per rank. A load is on the TTFT critical path and a + store is not, so if stores become frequent this queue wants a priority; + measure `state_load_queue_wait_ms` before adding one. """ self._register(self._executor.submit(self._do_load, req_id, h, slot)) @@ -107,7 +98,11 @@ def _do_store(self, h: int, unit_ids) -> None: stored = False try: stored = bool(self.codec.put(h, unit_ids)) - except Exception: # a store is best effort by design + except Exception: # noqa: BLE001 -- see below + # Deliberately blind. `codec.put` reaches into LMCache, whose + # failure modes are its own, and a store that cannot happen must + # cost one checkpoint's CPU copy -- not this worker thread, whose + # death would strand every request parked on a later load. logger.warning("state offload: store of hash %d failed", h, exc_info=True) with self._lock: # Report, never apply: `StateOffloadIndex` lives in the engine @@ -146,7 +141,10 @@ def _do_load(self, req_id: str, h: int, slot: int) -> None: ok = False try: ok = bool(self.codec.get(h, slot)) - except Exception: # a failed load is a normal path + except Exception: # noqa: BLE001 -- a failed load is a normal path + # Same reasoning as `_do_store`, and here a miss is expected: + # LMCache's LRU can drop bytes under a hash the index still + # advertises. The report below is what retracts the claim. logger.warning("state offload: load of hash %d failed", h, exc_info=True) with self._lock: if ok: diff --git a/atom/model_engine/block_manager.py b/atom/model_engine/block_manager.py index 319406d1c3..d7e09d6ded 100644 --- a/atom/model_engine/block_manager.py +++ b/atom/model_engine/block_manager.py @@ -259,7 +259,11 @@ def __init__( getattr(config, "kv_transfer_config", None) ).chunk_size ) - except Exception: + except Exception: # noqa: BLE001 + # Blind on purpose: this runs at model load, the import reaches a + # third-party package that may be absent entirely, and the only + # consequence of not knowing the chunk size is that the joint KV + # load stays off. Refusing to start would be worse. logger.warning( "state offload: could not read the LMCache chunk size; the " "joint KV load needs it and stays off", diff --git a/atom/model_engine/page_unit_checkpoint.py b/atom/model_engine/page_unit_checkpoint.py index f2167b6336..d10b2395d3 100644 --- a/atom/model_engine/page_unit_checkpoint.py +++ b/atom/model_engine/page_unit_checkpoint.py @@ -355,24 +355,15 @@ def complete_inflight(self) -> None: def _queue_offload_store(self, checkpoint_id: int, record) -> None: """Nominate this checkpoint for the CPU tier. Takes no pin. - **The READY transition, and not anywhere else.** Three candidate - moments and only this one works: - - - `begin_store` is too early: the scatter has not ridden a batch, so - the units do not hold the image yet. - - `_evict` / `unindex` are too late. They fire when the pool wants - those units *now*, so a multi-millisecond D2H started there holds 127 - of the most-wanted blocks at the worst possible moment. - - READY is the moment the bytes first exist and the moment the record - is least wanted -- it has just entered the LRU at the cold end, which - is the furthest any checkpoint ever is from being spent. - - Nomination, not reservation. A queued candidate is unpinned, so it - still counts as free space and `_next_victim` may spend it: the pool - never waits on the CPU tier, and a checkpoint the pool needed more than - the tier did simply loses its copy. Dedup is free -- `begin_store` - already refuses a hash it holds or has pending, so one hash is - nominated once. + READY and nowhere else: `begin_store` is too early (the scatter has not + ridden a batch, so the units hold no image yet) and `_evict`/`unindex` + are too late (they fire when the pool wants those units *now*). READY is + when the bytes first exist and when the record is least wanted -- it has + just entered the LRU at the cold end. + + Nomination, not reservation: a queued candidate is unpinned, so + `_next_victim` may still spend it. The pool never waits on the CPU tier. + Dedup is free -- `begin_store` refuses a hash it already holds. """ if not self._offload_sink: return @@ -388,20 +379,15 @@ def take_offload_stores( ) -> list[tuple[int, tuple[int, ...]]]: """`(prefix_hash, unit_ids)` to hand the tier now, pinning each. - **This is where the pin is taken**, not at READY, and that is what - bounds it. A pin lives in this process while the D2H runs in the - worker, so it spans several scheduler passes; pinning at READY would - make every checkpoint un-evictable for that whole window and break the - one thing #2045's admission argument rests on -- that a READY unpinned - checkpoint counts as available to live KV. Pinning only what is - actually in flight caps the damage at `max_inflight` images. + The pin is taken HERE, not at READY, and that is what bounds it: a pin + lives in this process while the D2H runs in the worker, so it spans + several scheduler passes. Pinning at READY would make every checkpoint + un-evictable for that window and break the admission invariant that a + READY unpinned checkpoint counts as available to live KV. - Over the cap, candidates wait rather than being dropped: they stay - nominated, unpinned and evictable, and are offered again next pass if - they are still around. - - Drained, not read: a second submission would store the same image - twice, and the second report would unpin a record the first released. + Over the cap, candidates wait rather than being dropped. Drained, not + read: a second submission would store the same image twice and the + second report would unpin a record the first released. """ out: list[tuple[int, tuple[int, ...]]] = [] while self._offload_ready and len(self._offload_pins) < max_inflight: @@ -600,20 +586,16 @@ def resumable_hit( def _reachable(self, h: int) -> bool: """Whether `h`'s image can be *produced*, in HBM or from the CPU tier. - Produced, not merely indexed. This scan runs right to left and stops at - the first boundary it accepts, so accepting one whose bytes nothing can - deliver does not cost a wasted lookup -- it costs the whole walk-back, - hiding every shorter checkpoint still resident in HBM. - - HBM needs no preference rule: both tiers are keyed by the same content - hash, so the scan takes the rightmost boundary wherever it lives, and - `_attach_state_slots` tries HBM first regardless. + The scan runs right to left and stops at the first boundary it accepts, + so accepting one nothing can deliver costs the whole walk-back, hiding + every shorter checkpoint still in HBM. No preference rule is needed: + both tiers key on the same content hash and `_attach_state_slots` tries + HBM first regardless. The tier's half is deliberately optimistic -- `hashes` means "was stored - once", never "is still there", since LMCache's own LRU can drop bytes at - any time. A false positive costs one park plus a recompute and retracts - itself (`fail_load` -> `forget`); refusing to vote until certain would - cost a synchronous cross-process lookup on the admission path. + once", never "is still there". A false positive costs one park plus a + recompute and retracts itself (`fail_load` -> `forget`); being certain + would cost a synchronous cross-process lookup on the admission path. """ if self.store.contains(h): return True @@ -782,9 +764,8 @@ def checkpoint_fates(self) -> dict[str, int]: because the two only mean anything against each other: `stores_completed` is worth reading only as a fraction of `checkpoints_kept`, and a tier that stored nothing is indistinguishable from a tier that was never - asked unless both numbers sit on one line. `StateGroupPool` did this - before #2045 moved K3's checkpoints here; restoring it makes the load - and store legs visible again under PAGE. + asked unless both numbers sit on one line. Folding them in makes the + load and store legs visible again under PAGE. Keys are prefixed `state_offload_` because the aggregation upstream is by key across every state class. diff --git a/atom/model_engine/scheduler.py b/atom/model_engine/scheduler.py index b1c9be37c3..fc0dc05060 100644 --- a/atom/model_engine/scheduler.py +++ b/atom/model_engine/scheduler.py @@ -94,34 +94,23 @@ def _offload_max_pending_saves() -> int: def _offload_save_abandon_timeout_s() -> float: - """Seconds a finished request's blocks may sit deferred on an offload save - before the engine reclaims them itself. - - A deferred save's blocks are freed only when the connector reports - `finished_saving`, so a lost report leaves them deferred forever: - `has_pending_kv_work()` never clears and the engine busy-loops with every - GPU idle -- a hard hang under a tight pool, a slow block leak under a large - one. - - **Why reclaiming cannot race a live copy.** `OffloadWorkerMixin._guard` - reports `finished_saving` on both the success and the exception path, so a - report is lost only when `self._engine.store(...)` neither returns nor - raises -- it is parked inside LMCache. Two things follow, and together they - are exhaustive: - - * the save that is parked is not copying. LMCache's pin monitor - force-unpins its source after `pin_timeout_sec` and stops reading the - blocks; reclaiming after that window therefore frees blocks nobody - holds. - * a save queued behind a parked one never reaches `store()` at all, so - its blocks were never read to begin with. - - Derived from LMCache's own `LMCACHE_EC_PIN_TIMEOUT_SEC` rather than an ATOM - knob of its own, because the ordering above is the whole safety argument: a - second env var could be set below the timeout it must exceed, and nothing - would say so. Raising LMCache's pin timeout now carries this window with it. - A non-positive value disables reclamation and restores the wait-forever - behaviour. + """Seconds a deferred offload save may sit before the engine reclaims it. + + Blocks are freed on `finished_saving`, so a lost report leaves them deferred + forever: `has_pending_kv_work()` never clears and the engine busy-loops with + every GPU idle. + + Reclaiming cannot race a live copy. `OffloadWorkerMixin._guard` reports on + both the success and the exception path, so a report is lost only when + `store()` neither returns nor raises -- it is parked inside LMCache. Then + either the parked save is not copying (LMCache force-unpinned its source + after `pin_timeout_sec`) or a save queued behind it never reached `store()`. + Both cases are safe once that window has passed. + + Derived from LMCache's own `LMCACHE_EC_PIN_TIMEOUT_SEC` rather than a knob + of its own, because that ordering IS the safety argument -- two independent + env vars could be set the wrong way round with nothing to say so. + Non-positive disables reclamation. """ global _SAVE_ABANDON_TIMEOUT_S if _SAVE_ABANDON_TIMEOUT_S is not None: diff --git a/atom/model_engine/state_offload.py b/atom/model_engine/state_offload.py index e0b36407cd..54811a3439 100644 --- a/atom/model_engine/state_offload.py +++ b/atom/model_engine/state_offload.py @@ -5,14 +5,6 @@ in LMCache, plus the loads offered against them. The bytes and the transfers are the worker's business (`kv_transfer/offload/hybrid/kimi_k3/state_tier.py`). -There used to be a second thing -- a bounded ring of staging entries a spill -copied an evicted checkpoint into. It existed for one reason: a checkpoint lived -in a state slot, and `StateSlotPool.pop` handed that slot to the next request -the instant it was evicted, so the bytes had to be rescued somewhere before they -were overwritten. Under #2045 a K3 checkpoint is a PAGE image in the KV pool -whose units are reserved and structurally un-takeable, so the source holds still -on its own and the ring has nothing left to rescue. See `REBASE_PLAN_v2.md`. - The set is in memory and never persisted: `LocalDiskBackend.__init__` starts from an empty dict and never scans its directory, so an index recovered from disk would be a pure false-positive generator. Index and bytes share one server diff --git a/atom/model_ops/attentions/kimi_mla_gdn_attn.py b/atom/model_ops/attentions/kimi_mla_gdn_attn.py index bfad3094bb..dce7bfdea7 100644 --- a/atom/model_ops/attentions/kimi_mla_gdn_attn.py +++ b/atom/model_ops/attentions/kimi_mla_gdn_attn.py @@ -231,32 +231,23 @@ def _page_unit_stream_sizes(self, units: int) -> np.ndarray: def page_unit_views(self, unit_ids: Sequence[int]) -> list[torch.Tensor]: """Tensor-view counterpart of `_page_unit_regions`, for the CPU tier. - `_page_unit_regions` hands the Triton descriptor raw int64 addresses, - which is right for a D2D copy the GPU issues. The LMCache staging - packer takes `list[torch.Tensor]` instead, so the same bytes have to be - nameable as views -- this is that seam, and it is the ONLY new one the - tier needs: a load writes the Active Slot directly and reuses - `state_entry_views`. - - **Order is the contract.** Unit major, row minor -- the same ravel as - `_page_unit_bases`, which is the order `_checkpoint_copy_plan` builds - the destination stream in. Save and load both go through this one - function, so self-consistency would be enough on its own; the reason it - still has to match is that a blob written by one build must not be read - by another that ordered it differently, which is what the `layout_id` in - `StateByteCodec.key` is for. - - **The `block_ratio` trap, same as `_page_unit_regions`.** `unit_ids` - carries *logical* block ids while `kv_cache` is shaped in *physical* - ones, and K3's ratio is 128. Flattening the two block axes together and - indexing by `unit * block_size` is what converts them: it is the same - arithmetic `_page_unit_bases` does in addresses, so the two cannot drift - into disagreeing about which bytes a unit owns. - - Every view is contiguous, which the packer requires: a row of a - contiguous pool is contiguous, and a slice along its leading axis stays - so. `_page_unit_regions` is called first for its side effect -- it is - where the contiguity and granularity checks live. + `_page_unit_regions` gives raw addresses for the Triton descriptor; the + LMCache packer takes `list[torch.Tensor]`, so the same bytes need naming + as views. This is the tier's only new seam -- a load writes the Active + Slot directly and reuses `state_entry_views`. + + Order is the contract: unit major, row minor, the same ravel as + `_page_unit_bases`. Cross-build safety is `layout_id` in + `StateByteCodec.key`, not this order. + + `unit_ids` carries *logical* block ids while `kv_cache` is shaped in + *physical* ones (K3's ratio is 128). Flattening the two block axes and + indexing by `unit * block_size` is that conversion -- the same + arithmetic `_page_unit_bases` does in addresses, so the two cannot + drift. Range-checked against the logical count for the same reason. + + `_page_unit_regions` runs first for its side effect: the contiguity and + granularity checks live there. """ _base, _stride = self._page_unit_regions() cache = self.model_runner.kv_cache diff --git a/tests/test_state_checkpoint.py b/tests/test_state_checkpoint.py index c5ce823bea..fedd2d5901 100644 --- a/tests/test_state_checkpoint.py +++ b/tests/test_state_checkpoint.py @@ -1591,7 +1591,7 @@ def test_no_tier_means_no_joint_boundary(self): """`state_offload is None` short-circuits before anything else: a boundary both legs must reach is meaningless with one leg missing.""" bm = self._joint_bm() - first, h = self._prompt_with_a_rung_at(bm, 8) + first, _h = self._prompt_with_a_rung_at(bm, 8) victim = first.block_table[1] bm.deallocate(first) self._break_the_kv_chain_at(bm, victim) From 9164ae1d6e663e58e83a37ad9626f119b5d81df5 Mon Sep 17 00:00:00 2001 From: zejunchen-zejun Date: Fri, 28 Aug 2026 16:40:27 +0800 Subject: [PATCH 023/151] style: drop three `noqa: BLE001` that suppressed a rule CI does not enable The previous commit added them to keep three deliberate blind excepts. CI came back with RUF100 on all three -- "Unused `noqa` directive (non-enabled: BLE001)". BLE001 is not in the enabled set, so the directives suppressed nothing and became findings of their own. The excepts stay as they are; only the directives go, with the reason kept as plain comment text. A blind except is correct at these three points -- the two in `StateOffloadTier` wrap `codec.put`/`get`, which reach into LMCache whose failure modes are its own, and a transfer that cannot happen must cost one checkpoint rather than the worker thread whose death strands every parked request; the one in `BlockManager` wraps an import of a third-party package that may be absent, where the only consequence of failing is that the joint KV load stays off. --- why the local checks kept missing this Local ruff was 0.14.14; CI does `pip3 install ruff`, which is 0.16.5 today. `ruff check .` -- byte for byte CI's command -- reports nothing on these lines under 0.14.14 and reports RUF100 under 0.16.5. Upgraded locally to match. Re-checked the way reviewdog actually reports, which is `-filter-mode=diff_context`: run CI's exact command and keep only findings on lines this branch changes. That is now zero. The repo as a whole has 1009 under 0.16.5, almost all of it pre-existing and on untouched lines, which is why the job passes for everyone else. Verified: `ruff check .` under 0.16.5 -> zero findings on this branch's lines; black 26.1.0 clean; pytest tests/ (less tests/plugin and the two msgpack/aiter collection errors) -> 4002 passed, zero failures. Co-Authored-By: Claude Opus 5 (1M context) --- atom/kv_transfer/offload/hybrid/kimi_k3/state_tier.py | 4 ++-- atom/model_engine/block_manager.py | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/atom/kv_transfer/offload/hybrid/kimi_k3/state_tier.py b/atom/kv_transfer/offload/hybrid/kimi_k3/state_tier.py index 16bba16fd2..811b50e231 100644 --- a/atom/kv_transfer/offload/hybrid/kimi_k3/state_tier.py +++ b/atom/kv_transfer/offload/hybrid/kimi_k3/state_tier.py @@ -98,7 +98,7 @@ def _do_store(self, h: int, unit_ids) -> None: stored = False try: stored = bool(self.codec.put(h, unit_ids)) - except Exception: # noqa: BLE001 -- see below + except Exception: # deliberately blind, see below # Deliberately blind. `codec.put` reaches into LMCache, whose # failure modes are its own, and a store that cannot happen must # cost one checkpoint's CPU copy -- not this worker thread, whose @@ -141,7 +141,7 @@ def _do_load(self, req_id: str, h: int, slot: int) -> None: ok = False try: ok = bool(self.codec.get(h, slot)) - except Exception: # noqa: BLE001 -- a failed load is a normal path + except Exception: # a failed load is a normal path # Same reasoning as `_do_store`, and here a miss is expected: # LMCache's LRU can drop bytes under a hash the index still # advertises. The report below is what retracts the claim. diff --git a/atom/model_engine/block_manager.py b/atom/model_engine/block_manager.py index d7e09d6ded..e3f4601177 100644 --- a/atom/model_engine/block_manager.py +++ b/atom/model_engine/block_manager.py @@ -259,7 +259,7 @@ def __init__( getattr(config, "kv_transfer_config", None) ).chunk_size ) - except Exception: # noqa: BLE001 + except Exception: # Blind on purpose: this runs at model load, the import reaches a # third-party package that may be absent entirely, and the only # consequence of not knowing the chunk size is that the joint KV From 118e0b921196dac143a50fd78c9b2d14b7ee7b38 Mon Sep 17 00:00:00 2001 From: zejunchen-zejun Date: Fri, 28 Aug 2026 18:59:07 +0800 Subject: [PATCH 024/151] docs+style: drop the env-var section and the V4 commentary this PR did not earn Two pieces of scope that rode along and neither of which this PR needs. `docs/environment_variables.md`: the section documented the offload connector's transport knobs -- OFFLOAD_COPY_WORKERS, the GPU staging trio, V4's four publication knobs -- none of which this PR introduces or changes. main documents none of them today, and the seven variables this PR *removes* were undocumented there too, so the removal obliges no doc update at all. `atom/kv_transfer/offload/README.md` already exists on main and is where that material belongs. The section also ended in a table with a header and no rows: it had been sized for the state-tier env vars, and once those were deleted nothing was left to list. `deepseek_v4_attn.py`: `relocate_state_slots` kept its body and grew an 18-line docstring about #1417's private ring, and `_pool_specs` gained a sentence naming "the two env vars that size them" -- OFFLOAD_STATE_CPU_SIZE and OFFLOAD_STATE_STAGING_GROUPS, both deleted by this PR, so the comment described a reader that no longer exists. `state_pool()` was also only reflowed across four lines with identical arguments. All three restored to main; `state_entry_views` keeps its one-line docstring. The file's whole diff against main is now the four lines of `state_entry_views`, the V4 half of the backend interface this PR adds. --- atom/model_ops/attentions/deepseek_v4_attn.py | 40 ++------------- docs/environment_variables.md | 49 ------------------- 2 files changed, 4 insertions(+), 85 deletions(-) diff --git a/atom/model_ops/attentions/deepseek_v4_attn.py b/atom/model_ops/attentions/deepseek_v4_attn.py index 50d53277ef..34b023032f 100644 --- a/atom/model_ops/attentions/deepseek_v4_attn.py +++ b/atom/model_ops/attentions/deepseek_v4_attn.py @@ -841,37 +841,11 @@ def state_transfer(self) -> StateTransfer: return StateTransfer.copy(layout_id) def state_entry_views(self, slot: int) -> list[torch.Tensor]: - """One contiguous slice per plane — a V4 slot is one row per plane. - - A slot holds the compressor state and then every layer's windows - contiguously (see `relocate_state_slots`), so a plane's whole - contribution is one range and no per-layer split is needed. - """ + """One contiguous slice per plane: a V4 slot is one row per plane.""" return self._slot_views()[slot] def relocate_state_slots(self, pairs: Sequence[tuple[int, int]]) -> None: - """Duplicate a request's whole per-request state: compressor + windows. - - One range per plane, and a plane is all there is: a slot holds the - compressor state and then every layer's windows, contiguously, so the - two halves of a request's state copy as one slice. Copying the state - whole rather than just the rows a resumer reads (the CSA ring's - trailing `K - ratio` = 4, HCA's none) is what makes that true — those - rows are scattered, and picking them out would cost 84 strided copies - against this one. - - **The window half is what makes the private ring safe.** A per-request - ring is exactly what #1417 removed, because a request resuming a cached - prefix had never written that prefix into its own. Reinstating it is only - correct because the checkpoint carries the window across — drop this and - the bug returns, silently, as garbage attention over the reused prefix. - - That is also why a window whose dtype differs from the pool's is a state - field rather than a plane of its own: a plane of its own would sit - outside every slot and so outside this copy, and a resumed request would - draft against the slot's previous occupant. Verification would keep the - output right and the acceptance rate would just quietly collapse. - """ + """Relocate a request's whole Active Slot.""" views = self._slot_views() dsts, srcs = [], [] for src, dst in pairs: @@ -1344,19 +1318,13 @@ def sub_pool_specs(self) -> list[SubPoolSpec]: No flat margin on the slot: a ring cannot transiently exceed itself the way a block-addressed window could while sliding across a boundary, and there is no admission-vs-materialization gap to cushion — a slot exists - for its request's whole life. The checkpoint headroom and the offload - tier's staging ring are both added by `state_pool` itself, which is the - only reader of the two env vars that size them. + for its request's whole life. """ geo = self.pool_geometry row_bytes = self.plane_row_bytes() return [ page_pool(geo.block_bytes(row_bytes) + self._indexer_block_bytes()), - state_pool( - STATE_SLOT_CLASS, - geo.slot_bytes(row_bytes), - entries_per_req=1, - ), + state_pool(STATE_SLOT_CLASS, geo.slot_bytes(row_bytes), entries_per_req=1), ] def _plane_row_widths(self) -> list[int]: diff --git a/docs/environment_variables.md b/docs/environment_variables.md index bf08805b78..99f04e40e4 100644 --- a/docs/environment_variables.md +++ b/docs/environment_variables.md @@ -182,55 +182,6 @@ flag below. Details in the state-checkpoint section of the | Variable | Type | Default | Description | |----------|------|---------|-------------| | **ATOM_STATE_CHECKPOINT_DEMAND** | bool | 1 (true) | Set to `0` to stop a prefix hit that was refused for want of a checkpoint from placing a rung of its own, leaving the prompt-end anchor as the only placement. Overrides `--state-checkpoint-demand`, so the policy can be A/B'd without editing a launch script. The rung is most of the checkpoint write traffic and little of the read-back, and every write evicts something — `StateSlotPool.mark_speculative` carries the measurement. | -## KV / state offload (`lmcache_offload` connector) - -The `lmcache_offload` connector spills computed KV to a CPU tier and fetches it -back on a later prefix hit. It has no env switch of its own: it runs when -`--kv-transfer-config` names it, e.g. -`'{"kv_connector":"lmcache_offload","kv_role":"offload"}'`. - -Which layout runs is decided from the model config alone -(`select_offload_layout`) -- `dense` for ordinary token-indexed KV, `hybrid` for -DeepSeek-V4's compressed PAGE/SLOT chunks, `kimi_k3` for dense MLA KV plus the -KDA per-request state tier. Background and the counters to read before tuning -any of this: [`atom/kv_transfer/offload/README.md`](../atom/kv_transfer/offload/README.md). - -### Transport and staging (all layouts) - -| Variable | Type | Default | Description | -|----------|------|---------|-------------| -| **OFFLOAD_COPY_WORKERS** | int | 1 | Threads draining the save queue. | -| **OFFLOAD_MAX_PENDING_SAVES** | int | `max(2, 2 x copy_workers)` | The back-pressure bound, read from both ends: `hybrid` caps the worker queue with it, `kimi_k3` caps how many requests may have a save outstanding. It matters because a request whose save is queued keeps its blocks pinned, so this is also how much of the pool a slow backend can hold. Ignored by `dense`, which does not pin. | -| **OFFLOAD_MIN_LOAD_TOKENS** | int | 8192 | Prefix below which a KV hit is not worth a load -- a KV load moves bytes proportional to the hit, so a short one spends a round trip moving very little. **No effect under `kimi_k3`**: every sequence there carries per-request state, and a joint boundary is chosen for both legs together, so refusing one leg on size would leave the other claiming a prefix whose KV never came. | -| **OFFLOAD_GPU_STAGING_CHUNKS** | int | 2 | GPU staging buffer, in LMCache chunks. | -| **OFFLOAD_GPU_STAGING_MAX_BYTES** | int | unset | Hard cap on the staging buffer; must cover at least one chunk. | -| **OFFLOAD_RELEASE_GPU_STAGING_AFTER_TRANSFER** | bool | 0 | Free the staging buffer after each transfer instead of holding it. Trades HBM for per-transfer allocation. | -| **OFFLOAD_PROFILE** | bool | 0 | Per-transfer timing on the periodic stats line. | -| **LMCACHE_LOCAL_CPU**, **LMCACHE_MAX_LOCAL_CPU_SIZE**, **LMCACHE_CHUNK_SIZE**, **LMCACHE_LOCAL_DISK** | - | LMCache | Read by LMCache itself, not by ATOM. `MAX_LOCAL_CPU_SIZE` is GiB for the paged-KV pool. | - -### DeepSeek-V4 (`hybrid`) - -| Variable | Type | Default | Description | -|----------|------|---------|-------------| -| **OFFLOAD_SLOT_STAGING_SLOTS** | int | 1 | Staging slots for complete SLOT sidecars. | -| **OFFLOAD_COMMITTED_SIDECAR_CAPACITY** | int | 65536 | Committed-sidecar index capacity. | -| **OFFLOAD_PUBLICATION_TIMEOUT_S** | float | 5.0 | How long a worker waits for a chunk's publication. | -| **OFFLOAD_PUBLICATION_POLL_INTERVAL_S** | float | 0.01 | Poll interval for the above. | - -### Per-request state and its tier - -Hybrid models (GDN/KDA: Qwen3-Next, Qwen3.5, Kimi-K3) and DeepSeek-V4 keep one -per-request **state** entry alongside their paged KV, in a fixed pool of groups. -A prefix hit is resumable only up to a boundary where that state was -checkpointed, so the pool size and whether an evicted checkpoint can be fetched -back decide the hit rate. - -Under `kimi_k3` the state tier is **not separately switchable**: a resumable -prefix needs both legs, so state is offloaded exactly when KV is, and turning -the tier off means turning the connector off. - -| Variable | Type | Default | Description | -|----------|------|---------|-------------| ## Profiling & debugging From 190c544b3274c83022e2bcb541e0e5854ef47601 Mon Sep 17 00:00:00 2001 From: zejunchen-zejun Date: Sun, 30 Aug 2026 15:17:17 +0800 Subject: [PATCH 025/151] fix(offload): give the joint load's two legs one completion identity Raised in review (connector.py:288, blocking): the two joint legs use different completion identities. `_joint_park` is armed with the bare `req.req_id`, and the state tier also reports the bare id, but the dense KV worker reports `LoadOperationId(req_id, generation)` because `LMCacheReqMeta.load_operation` is populated. So `waits_for(LoadOperationId(...))` is false, the KV completion passes through immediately, and the raw-id park remains waiting even after the state leg settles -- which can wake the scheduler before state is ready. Confirmed, and reproduced exactly as described. `_load_completion_id` (_offload_common.py:141) is `req.load_operation or req.req_id`, and `build_connector_meta` attaches a `load_operation` to every load it emits -- so the KV leg's identity is *always* the typed one, and the bare id the park was armed with never appears on that channel. Both named consequences hold: * `waits_for` is false for the KV report, so `_settle_joint` classifies it as passthrough and hands it to the engine immediately. The request resumes its suffix prefill with `num_cached_tokens` raised to the joint boundary while the state H2D may still be writing the Active Slot. Silent wrong output, no exception. * The park entry is never settled, so one entry leaks per joint load. Fix: file the park under the identity the KV leg will actually report, and let the state leg reach it by alias. * `_JointPark.arm` takes `kv_id` -- exactly what the KV worker will put on `finished_loading`/`failed_loading` -- and files `_need` under it. The bare request id is recorded in `_alias` so a state report finds the same park. `_resolve` maps either leg's identity onto the key, so `waits_for`/`settle_kv`/`settle_state` all accept either and the caller does not have to know which channel it is draining. * `take_ready` therefore emits the KV identity, which is what the engine matches against the operation it issued. Downstream needs no change: `OffloadSchedulerMixin.process_completions` already unwraps `.req_id` before the scheduler sees it. * `_release` drops both directions of the alias, so the park cannot grow across loads. * `_arm_joint_loads` passes `kv_id=self._load_completion_id(req)` -- the same call the KV worker makes, so the two cannot drift. Tests use a real `LoadOperationId` as the review asked, and cover the failure rather than only the fixed shape: the KV leg alone must not wake the request, both legs wake it once under the KV identity, either leg failing fails the pair, four loads leave the park empty, and a single-leg request still passes straight through. Four of the six fail against the previous code. --- .../offload/hybrid/kimi_k3/connector.py | 19 +++- .../offload/hybrid/kimi_k3/state_tier.py | 88 ++++++++++----- tests/test_lmcache_offload_connector.py | 101 ++++++++++++++++++ 3 files changed, 183 insertions(+), 25 deletions(-) diff --git a/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py b/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py index 2c211478f2..246d70ed79 100644 --- a/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py +++ b/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py @@ -184,6 +184,18 @@ def _arm_joint_loads(self, metadata) -> None: Both legs surface on the KV completion channel, so an id in both would collapse into one wake and resume the suffix prefill while the other transfer is still writing. + + **The two legs do not report the same identity.** The KV worker reports + `_load_completion_id(req)`, which is `req.load_operation` whenever the + scheduler issued one -- and `build_connector_meta` issues one for every + load, so in practice it is always the typed `LoadOperationId`. The state + tier is keyed by request and reports the bare id. Arming under the bare + id therefore parked nothing the KV leg could ever settle: `waits_for` + was false for the `LoadOperationId`, the KV completion passed straight + through, and the engine could resume the suffix prefill while the state + H2D was still writing the Active Slot -- silent wrong output, and a park + entry leaked per joint load. The park is filed under the KV identity, + which is also the one that has to reach the engine. """ loads = getattr(metadata, "state_loads", None) or () state_ids = {req_id for req_id, _h, _slot in loads} @@ -191,7 +203,12 @@ def _arm_joint_loads(self, metadata) -> None: return for req in metadata.requests: if req.load_spec is not None and req.req_id in state_ids: - self._joint_park.arm(req.req_id, needs_kv=True, needs_state=True) + self._joint_park.arm( + req.req_id, + needs_kv=True, + needs_state=True, + kv_id=self._load_completion_id(req), + ) def _start_state_loads(self, metadata) -> None: """Hand this step's state loads to the tier's executor. diff --git a/atom/kv_transfer/offload/hybrid/kimi_k3/state_tier.py b/atom/kv_transfer/offload/hybrid/kimi_k3/state_tier.py index 811b50e231..d759c98155 100644 --- a/atom/kv_transfer/offload/hybrid/kimi_k3/state_tier.py +++ b/atom/kv_transfer/offload/hybrid/kimi_k3/state_tier.py @@ -15,6 +15,7 @@ import logging import threading from concurrent.futures import ThreadPoolExecutor +from typing import Any logger = logging.getLogger(__name__) @@ -166,51 +167,90 @@ class _JointPark: """ def __init__(self) -> None: - self._need: dict[str, set[str]] = {} - self._failed: set[str] = set() - self._ready: set[str] = set() - self._ready_failed: set[str] = set() - - def arm(self, req_id: str, *, needs_kv: bool, needs_state: bool) -> None: + self._need: dict[Any, set[str]] = {} + self._failed: set = set() + self._ready: set = set() + self._ready_failed: set = set() + # The two legs do not report the same identity. The KV leg reports + # whatever `_load_completion_id` yields -- a typed `LoadOperationId` + # whenever the scheduler issued one, which for an offload load is + # always -- while the state tier is keyed by request and reports the + # bare id. The park is filed under the KV identity, because that is + # what has to reach the engine on `finished_loading`; this maps the + # bare id onto it so a state report can find its own park. + self._alias: dict = {} + self._alias_of: dict = {} + + def arm( + self, + req_id: str, + *, + needs_kv: bool, + needs_state: bool, + kv_id=None, + ) -> None: + """Park `req_id`, filed under `kv_id` when the KV leg reports one. + + `kv_id` must be exactly what the KV worker will put on + `finished_loading`/`failed_loading` for this load, because that report + is matched by equality and nothing translates it on the way in. + """ need = set() if needs_kv: need.add("kv") if needs_state: need.add("state") - self._need[req_id] = need + key = req_id if kv_id is None else kv_id + self._need[key] = need + if key != req_id: + self._alias[req_id] = key + self._alias_of[key] = req_id if not need: - self._ready.add(req_id) + self._release(key) + + def settle_kv(self, ident, ok: bool) -> None: + self._settle(ident, "kv", ok) - def settle_kv(self, req_id: str, ok: bool) -> None: - self._settle(req_id, "kv", ok) + def settle_state(self, ident, ok: bool) -> None: + self._settle(ident, "state", ok) - def settle_state(self, req_id: str, ok: bool) -> None: - self._settle(req_id, "state", ok) + def _resolve(self, ident): + """The park key for either leg's identity. Identity when unarmed.""" + return self._alias.get(ident, ident) - def _settle(self, req_id: str, leg: str, ok: bool) -> None: - need = self._need.get(req_id) + def _settle(self, ident, leg: str, ok: bool) -> None: + key = self._resolve(ident) + need = self._need.get(key) if need is None: return need.discard(leg) if not ok: - self._failed.add(req_id) + self._failed.add(key) if need: return - del self._need[req_id] - if req_id in self._failed: - self._failed.discard(req_id) - self._ready_failed.add(req_id) + self._release(key) + + def _release(self, key) -> None: + self._need.pop(key, None) + bare = self._alias_of.pop(key, None) + if bare is not None: + self._alias.pop(bare, None) + if key in self._failed: + self._failed.discard(key) + self._ready_failed.add(key) else: - self._ready.add(req_id) + self._ready.add(key) - def waits_for(self, req_id: str) -> bool: - """Whether this park still owes `req_id` a leg. + def waits_for(self, ident) -> bool: + """Whether this park still owes `ident`'s request a leg. Asked before settling: the legs report through channels a single-leg request also uses, and `_settle` ignores unknown ids, which is - indistinguishable from a leg that landed. + indistinguishable from a leg that landed. Accepts either leg's + identity, so the caller does not have to know which channel it is + draining. """ - return req_id in self._need + return self._resolve(ident) in self._need def take_ready(self) -> tuple[set[str], set[str]]: ready, failed = set(self._ready), set(self._ready_failed) diff --git a/tests/test_lmcache_offload_connector.py b/tests/test_lmcache_offload_connector.py index c7ee14d8d1..36ed329fef 100644 --- a/tests/test_lmcache_offload_connector.py +++ b/tests/test_lmcache_offload_connector.py @@ -57,8 +57,10 @@ from atom.kv_transfer.offload.hybrid.kimi_k3.connector import ( SAVE_STALL_SECONDS, STATE_INDEX_CHANNEL, + KimiK3OffloadConnector, KimiK3OffloadScheduler, ) +from atom.kv_transfer.offload.hybrid.kimi_k3.state_tier import _JointPark from atom.kv_transfer.offload.metadata import ( ATOMRawBytesLMCacheMetadata, LMCacheOffloadMetadata, @@ -5084,3 +5086,102 @@ def test_the_shells_no_impl_fallbacks_match_what_the_caller_unpacks(): assert shell.load_finished(None) is True # Unchanged, so a connector with no opinion does not shrink the chunk. assert shell.adjust_prefill_chunk_after_alloc(None, 7) == 7 + + +# ── kimi_k3: the two joint legs report different identities ─────────────── + + +def _k3_worker() -> KimiK3OffloadConnector: + """Only what the joint overrides touch. `_state_tier` is a sentinel object + because `get_finished` refuses to run its state half without one.""" + c = KimiK3OffloadConnector.__new__(KimiK3OffloadConnector) + c._joint_park = _JointPark() + c._state_tier = None + c._do_load = True + return c + + +def _k3_load_req(req_id: str, generation: int = 0): + """A load request shaped like `build_connector_meta`'s: it always attaches a + `load_operation`, which is what makes the KV leg report the typed id.""" + return SimpleNamespace( + req_id=req_id, + load_spec=object(), + load_operation=LoadOperationId(req_id, generation), + ) + + +class TestJointLegsShareOneCompletionIdentity: + """The KV leg reports `LoadOperationId`, the state tier reports the bare id. + + Arming under the bare id parked nothing the KV leg could settle: the KV + completion passed straight through and the engine could resume the suffix + prefill while the state H2D was still writing the Active Slot. These pin + the identity down with a real `LoadOperationId`, not two raw ids. + """ + + def test_a_park_armed_with_a_kv_id_answers_to_both_legs(self): + park = _JointPark() + kv_id = LoadOperationId("r1", 3) + park.arm("r1", needs_kv=True, needs_state=True, kv_id=kv_id) + assert park.waits_for(kv_id) + assert park.waits_for("r1") + + def test_the_kv_leg_alone_does_not_wake_the_request(self): + worker = _k3_worker() + req = _k3_load_req("r1") + meta = SimpleNamespace(state_loads=[("r1", 99, 0)], requests=[req]) + worker._arm_joint_loads(meta) + + # Exactly what the dense worker puts on the wire for this load. + kv_report = {req.load_operation} + done, failed = worker._settle_joint(kv_report, set(), set(), set()) + assert done == set(), "KV must not pass through while state is in flight" + assert failed == set() + + def test_both_legs_wake_it_once_under_the_kv_identity(self): + worker = _k3_worker() + req = _k3_load_req("r1") + worker._arm_joint_loads( + SimpleNamespace(state_loads=[("r1", 99, 0)], requests=[req]) + ) + worker._settle_joint({req.load_operation}, set(), set(), set()) + done, failed = worker._settle_joint(set(), set(), {"r1"}, set()) + # The engine matches `finished_loading` against the operation it issued, + # so the wake has to carry that identity, not the bare id. + assert done == {req.load_operation} + assert failed == set() + assert not worker._joint_park.waits_for("r1") + + def test_either_leg_failing_fails_the_pair(self): + worker = _k3_worker() + req = _k3_load_req("r1") + worker._arm_joint_loads( + SimpleNamespace(state_loads=[("r1", 99, 0)], requests=[req]) + ) + worker._settle_joint({req.load_operation}, set(), set(), set()) + done, failed = worker._settle_joint(set(), set(), set(), {"r1"}) + assert done == set() + assert failed == {req.load_operation} + + def test_the_park_does_not_leak_an_entry_per_joint_load(self): + worker = _k3_worker() + for i in range(4): + req = _k3_load_req(f"r{i}") + worker._arm_joint_loads( + SimpleNamespace(state_loads=[(req.req_id, 99, 0)], requests=[req]) + ) + worker._settle_joint({req.load_operation}, set(), {req.req_id}, set()) + park = worker._joint_park + assert park._need == {} + assert park._alias == {} + assert park._alias_of == {} + + def test_a_single_leg_request_still_passes_straight_through(self): + """Nothing armed it, so neither channel may be held back.""" + worker = _k3_worker() + kv_only = LoadOperationId("r9", 0) + done, failed = worker._settle_joint({kv_only}, set(), set(), set()) + assert done == {kv_only} + done, failed = worker._settle_joint(set(), set(), {"r8"}, set()) + assert done == {"r8"} From 9f8c50e6bfa40e50ed53f79531586383945e6543 Mon Sep 17 00:00:00 2001 From: zejunchen-zejun Date: Sun, 30 Aug 2026 15:17:17 +0800 Subject: [PATCH 026/151] fix(offload): key state-store completions by a generation, not a bare hash Raised in review (connector.py:264): this completion needs a generation, not just the prefix hash. `KVOutputAggregator._connector_completions` tombstones every `(channel, operation_id)` after quorum, so if the same prefix is stored again after an eviction or load miss, the second `(STATE_INDEX_CHANNEL, hash)` completion is dropped as a duplicate: its new PAGE pin survives until stale-pin reclamation and the CPU index is not rebuilt. A late completion from an older attempt can also settle a newer pin for the same hash. The review asked for a typed `StateStoreOperationId(hash, generation)`, for `_offload_pins` and settlement to be keyed by that exact operation, and for a test where two generations store the same hash and both settle their own pin. Confirmed. `aggregator.py:158-161` builds the connector-completion group with `should_tombstone = lambda _key: True`, so every key is remembered after quorum and `report()` returns early for it forever after. Reproduced with `KVOutputAggregator(world_size=1)`: the first `(STATE_INDEX_CHANNEL, 4242)` aggregates, the second returns an empty set. Both consequences follow -- a re-stored prefix leaves its units pinned until `reclaim_stale_offload_pins`, and `note_stored` never runs, so the tier stops voting for a prefix whose bytes are present. A hash is the right key for the *index* -- what a resume asks is whether the prefix is in LMCache, not which attempt put it there -- but it is not an identity for one transfer. Introduce that identity: * `StateStoreOperationId(prefix_hash, generation)`, frozen, alongside the existing Save/Load operation ids and added to the `ConnectorCompletionId` union. * `PageUnitCheckpointStore._offload_pins` is keyed by the operation and `take_offload_stores` stamps a fresh generation on every hand-out, so two sequential attempts at one prefix are distinct keys all the way through the aggregator. `settle_offload_store(op)` matches the whole operation, which is also what makes a late report from a superseded attempt settle nothing instead of releasing a newer pin. * `_hash_in_flight` preserves the old refusal: the generation tells *sequential* attempts apart, it does not license concurrent ones, which would copy the same bytes twice and have the first report unpin what the second is holding. * The operation is threaded through unchanged -- tier, connector channel, `take_state_reports`, `settle_state_store` -- rather than narrowed back to `int` anywhere. `connector_completion` used to do exactly that narrowing, which would have undone the generation one call before it was used. Tests: the asked-for two-generations case at the store (both settle their own pin) and at the aggregator (both reach the engine); a late report from a reclaimed attempt leaving a newer pin standing; one hash still refusing two concurrent stores. Plus a test that a *bare* hash is dropped the second time, so the aggregator behaviour this works around is pinned and cannot quietly change and make the generation look unnecessary. --- atom/kv_transfer/disaggregation/__init__.py | 2 + atom/kv_transfer/disaggregation/types.py | 26 ++++++- .../offload/hybrid/kimi_k3/connector.py | 24 +++--- .../offload/hybrid/kimi_k3/state_tier.py | 45 +++++++---- atom/model_engine/block_manager.py | 13 +++- atom/model_engine/page_unit_checkpoint.py | 74 +++++++++++++------ atom/model_engine/scheduler.py | 12 +-- tests/test_lmcache_offload_connector.py | 50 +++++++++++++ tests/test_page_unit_checkpoint.py | 66 +++++++++++++++-- 9 files changed, 249 insertions(+), 63 deletions(-) diff --git a/atom/kv_transfer/disaggregation/__init__.py b/atom/kv_transfer/disaggregation/__init__.py index a21dffdecd..6b62f1d3fa 100644 --- a/atom/kv_transfer/disaggregation/__init__.py +++ b/atom/kv_transfer/disaggregation/__init__.py @@ -21,6 +21,7 @@ LoadOperationId, ReqMeta, SaveOperationId, + StateStoreOperationId, ) __all__ = [ @@ -34,4 +35,5 @@ "LoadOperationId", "ReqMeta", "SaveOperationId", + "StateStoreOperationId", ] diff --git a/atom/kv_transfer/disaggregation/types.py b/atom/kv_transfer/disaggregation/types.py index 74d629bf9c..e0ea297268 100644 --- a/atom/kv_transfer/disaggregation/types.py +++ b/atom/kv_transfer/disaggregation/types.py @@ -58,7 +58,31 @@ def __post_init__(self) -> None: LoadCompletionId = ReqId | LoadOperationId -ConnectorCompletionId = ReqId | SaveOperationId | LoadOperationId + +@dataclass(frozen=True) +class StateStoreOperationId: + """Exact identity of one hand-out of a state checkpoint to the CPU tier. + + Not keyed by request: by the time a state store lands, the request that + produced the checkpoint is long gone and only the prefix hash remains. But + the hash alone is not an identity -- the same prefix is stored again after + an eviction or a load miss, and `KVOutputAggregator` tombstones every + `(channel, operation_id)` it has taken quorum on, so a second store under a + bare hash is dropped as a duplicate: its pin is never settled and its + bytes are never re-indexed. `generation` is what separates the attempts. + """ + + prefix_hash: int + generation: int + + def __post_init__(self) -> None: + if self.generation < 0: + raise ValueError("state store generation must be nonnegative") + + +ConnectorCompletionId = ( + ReqId | SaveOperationId | LoadOperationId | StateStoreOperationId +) ConnectorCompletionKey = tuple[str, ConnectorCompletionId] # --------------------------------------------------------------------------- diff --git a/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py b/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py index 246d70ed79..d7b39ea7af 100644 --- a/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py +++ b/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py @@ -252,11 +252,11 @@ def _start_state_stores(self, metadata) -> None: "so the engine releases their units now rather than on timeout.", len(stores), ) - for h, _units in stores: - self._state_store_failed_locally.add(int(h)) + for op, _units in stores: + self._state_store_failed_locally.add(op) return - for h, unit_ids in stores: - self._state_tier.submit_store(int(h), tuple(int(u) for u in unit_ids)) + for op, unit_ids in stores: + self._state_tier.submit_store(op, tuple(int(u) for u in unit_ids)) def _fail_state_loads(self, loads) -> None: for req_id, _h, _group in loads: @@ -276,13 +276,19 @@ def get_finished(self): # owner is long gone -- so they ride the connector-owned channel; the # aggregator's quorum is failure-dominant, which is what resolves a # partial store instead of pinning the key. - for h in indexed: + # + # The operation, not the bare hash. `KVOutputAggregator` tombstones + # every `(channel, operation_id)` it has taken quorum on, so a hash + # alone made the second store of a re-evicted prefix a duplicate: it + # was dropped before quorum, its pin waited for stale reclamation, and + # the CPU index never learned the bytes were back. + for op in indexed: out.connector_completions.add( - ConnectorCompletion(STATE_INDEX_CHANNEL, int(h), True) + ConnectorCompletion(STATE_INDEX_CHANNEL, op, True) ) - for h in index_failed: + for op in index_failed: out.connector_completions.add( - ConnectorCompletion(STATE_INDEX_CHANNEL, int(h), False) + ConnectorCompletion(STATE_INDEX_CHANNEL, op, False) ) return out @@ -505,7 +511,7 @@ def connector_completion(self, completion) -> bool: if completion.succeeded else self._state_index_failed ) - target.add(int(completion.operation_id)) + target.add(completion.operation_id) return True return super().connector_completion(completion) diff --git a/atom/kv_transfer/offload/hybrid/kimi_k3/state_tier.py b/atom/kv_transfer/offload/hybrid/kimi_k3/state_tier.py index d759c98155..887cff0001 100644 --- a/atom/kv_transfer/offload/hybrid/kimi_k3/state_tier.py +++ b/atom/kv_transfer/offload/hybrid/kimi_k3/state_tier.py @@ -37,9 +37,12 @@ def __init__(self, codec, *, max_workers: int = 1) -> None: self._done: set[str] = set() self._failed: set[str] = set() self._inflight: set = set() - # The store path's two reports, drained by `take_store_reports`. - self._indexed: set[int] = set() - self._index_failed: set[int] = set() + # The store path's two reports, drained by `take_store_reports`. Sets + # of `StateStoreOperationId`, not of bare hashes: the engine settles + # the pin for that exact generation, and the aggregator would tombstone + # a bare hash after the first store of it. + self._indexed: set = set() + self._index_failed: set = set() def _register(self, fut) -> None: """Add *fut* to the inflight set and attach a callback that removes it @@ -56,15 +59,20 @@ def _discard(f): fut.add_done_callback(_discard) - def submit_store(self, h: int, unit_ids) -> None: - """Pack the checkpoint image in `unit_ids` under `h`, for LMCache. + def submit_store(self, op, unit_ids) -> None: + """Pack the checkpoint image in `unit_ids` for LMCache, under `op`. + + `op` is a `StateStoreOperationId`; the bytes are keyed by + `op.prefix_hash` and the report is keyed by the whole operation, so + two attempts at one prefix write the same entry but settle their own + pins. No `ready_event`: the units are reserved out of the KV pool and pinned by the engine for the length of this transfer, so nothing on the compute stream is writing them and the packer gathers straight from where they sit. """ - self._register(self._executor.submit(self._do_store, h, unit_ids)) + self._register(self._executor.submit(self._do_store, op, unit_ids)) def submit_load(self, req_id: str, h: int, slot: int) -> None: """Fetch `h` into pool slot `slot` for the parked request `req_id`. @@ -95,34 +103,39 @@ def get_finished(self) -> tuple[set[str], set[str]]: def shutdown(self) -> None: self._executor.shutdown(wait=True) - def _do_store(self, h: int, unit_ids) -> None: + def _do_store(self, op, unit_ids) -> None: stored = False try: - stored = bool(self.codec.put(h, unit_ids)) + stored = bool(self.codec.put(int(op.prefix_hash), unit_ids)) except Exception: # deliberately blind, see below # Deliberately blind. `codec.put` reaches into LMCache, whose # failure modes are its own, and a store that cannot happen must # cost one checkpoint's CPU copy -- not this worker thread, whose # death would strand every request parked on a later load. - logger.warning("state offload: store of hash %d failed", h, exc_info=True) + logger.warning( + "state offload: store of hash %d (generation %d) failed", + op.prefix_hash, + op.generation, + exc_info=True, + ) with self._lock: # Report, never apply: `StateOffloadIndex` lives in the engine # process and this runs in a spawned runner. The engine applies # these via KVConnectorOutput. if stored: - self._indexed.add(int(h)) + self._indexed.add(op) else: # The failure channel lets the aggregator take quorum on # `indexed | index_failed` instead of waiting for a second # report that will never come from this rank. - self._index_failed.add(int(h)) + self._index_failed.add(op) - def take_store_reports(self) -> tuple[set[int], set[int]]: - """`(hashes stored, hashes failed)` since the last call. + def take_store_reports(self) -> tuple[set, set]: + """`(operations stored, operations failed)` since the last call. - A hash appears in exactly one of the two per store. The aggregator's - quorum over them is failure-dominant, so a partial store resolves in - the same step rather than pinning the key. + An operation appears in exactly one of the two. The aggregator's quorum + over them is failure-dominant, so a partial store resolves in the same + step rather than pinning the key. """ with self._lock: indexed = set(self._indexed) diff --git a/atom/model_engine/block_manager.py b/atom/model_engine/block_manager.py index e3f4601177..7c50e64e5e 100644 --- a/atom/model_engine/block_manager.py +++ b/atom/model_engine/block_manager.py @@ -1066,7 +1066,7 @@ def settle_state_load(self, req_id, ok: bool) -> None: self.state.release(slot) def take_state_stores(self, max_inflight: int) -> list[tuple]: - """`(hash, unit_ids)` for checkpoints to hand the CPU tier this pass. + """`(operation, unit_ids)` for checkpoints to hand the CPU tier. Pins each as it hands it over; `settle_state_store` releases. Empty without a coordinator (the fork backends), which is correct rather than @@ -1080,21 +1080,26 @@ def take_state_stores(self, max_inflight: int) -> list[tuple]: self.state_offload.stores_attempted += len(out) return out - def settle_state_store(self, prefix_hash: int, ok: bool) -> None: + def settle_state_store(self, op, ok: bool) -> None: """One store reported. Release the units, and index the hash if it landed. Success and failure release the pin identically -- it existed to keep the bytes still during the copy, and the copy is over either way. Only the indexing differs, because only a hash whose bytes are really there may be voted for. + + `op` is a `StateStoreOperationId`: the pin is released for that exact + generation, so a late report from a superseded attempt settles nothing. + The index is keyed by hash, because what a resume asks is whether the + prefix is in LMCache -- not which attempt put it there. """ if self.paged_state_checkpoints is not None: - self.paged_state_checkpoints.settle_offload_store(int(prefix_hash)) + self.paged_state_checkpoints.settle_offload_store(op) if self.state_offload is None: return if ok: self.state_offload.stores_completed += 1 - self.state_offload.note_stored(int(prefix_hash)) + self.state_offload.note_stored(int(op.prefix_hash)) else: self.state_offload.stores_failed += 1 diff --git a/atom/model_engine/page_unit_checkpoint.py b/atom/model_engine/page_unit_checkpoint.py index d10b2395d3..a9c144f852 100644 --- a/atom/model_engine/page_unit_checkpoint.py +++ b/atom/model_engine/page_unit_checkpoint.py @@ -10,6 +10,7 @@ from dataclasses import dataclass from time import monotonic +from atom.kv_transfer.disaggregation.types import StateStoreOperationId from atom.model_engine.block_pool import BlockPool from atom.model_engine.sequence import Sequence @@ -138,12 +139,19 @@ def __init__( # space and `_next_victim` may spend it. The pin is taken later, in # `take_offload_stores`, and only for the few actually in flight. self._offload_ready: deque[int] = deque() - # prefix_hash -> (checkpoint_id, monotonic when the pin was taken). - # Keyed by hash because that is all the worker's report carries -- by - # the time a store lands its request is long gone. At most one live - # record per hash, which `begin_store` enforces by refusing a hash it - # already holds or has pending. - self._offload_pins: dict[int, tuple[int, float]] = {} + # StateStoreOperationId -> (checkpoint_id, monotonic when pinned). + # A store's report carries no request -- by the time one lands its owner + # is long gone -- so the hash is the only thing left to name it by. The + # hash ALONE is not an identity though: the same prefix is stored again + # after an eviction or a load miss, and `KVOutputAggregator` tombstones + # every `(channel, operation_id)` it has taken quorum on, so the second + # store under a bare hash was dropped as a duplicate -- its pin held + # until stale reclamation and its bytes never re-indexed. The + # generation separates the attempts, and keying the pins by the whole + # operation is also what stops a late report from an earlier attempt + # from settling a newer pin for the same hash. + self._offload_pins: dict[StateStoreOperationId, tuple[int, float]] = {} + self._offload_generation = 0 self.offload_pins_reclaimed = 0 @property @@ -376,8 +384,8 @@ def store_backlog(self) -> deque: def take_offload_stores( self, max_inflight: int - ) -> list[tuple[int, tuple[int, ...]]]: - """`(prefix_hash, unit_ids)` to hand the tier now, pinning each. + ) -> list[tuple[StateStoreOperationId, tuple[int, ...]]]: + """`(operation, unit_ids)` to hand the tier now, pinning each. The pin is taken HERE, not at READY, and that is what bounds it: a pin lives in this process while the D2H runs in the worker, so it spans @@ -388,8 +396,14 @@ def take_offload_stores( Over the cap, candidates wait rather than being dropped. Drained, not read: a second submission would store the same image twice and the second report would unpin a record the first released. + + Each hand-out gets its own generation, so a prefix stored again after + an eviction is a different operation from the one that stored it + before. Two attempts at the same hash may not be in flight at once -- + `_hash_in_flight` refuses that -- but they may follow one another + closely enough that the earlier one's report is still on the wire. """ - out: list[tuple[int, tuple[int, ...]]] = [] + out: list[tuple[StateStoreOperationId, tuple[int, ...]]] = [] while self._offload_ready and len(self._offload_pins) < max_inflight: checkpoint_id = self._offload_ready.popleft() record = self.records.get(checkpoint_id) @@ -397,22 +411,40 @@ def take_offload_stores( # or already in flight under this hash. Nothing to send either way. if record is None or record.state != READY: continue - if record.prefix_hash in self._offload_pins: + if self._hash_in_flight(record.prefix_hash): continue + self._offload_generation += 1 + op = StateStoreOperationId( + int(record.prefix_hash), self._offload_generation + ) record.pin_count += 1 - self._offload_pins[record.prefix_hash] = (checkpoint_id, monotonic()) - out.append((record.prefix_hash, record.unit_ids)) + self._offload_pins[op] = (checkpoint_id, monotonic()) + out.append((op, record.unit_ids)) return out - def settle_offload_store(self, prefix_hash: int) -> None: + def _hash_in_flight(self, prefix_hash: int) -> bool: + """Whether some generation of `prefix_hash` is already pinned. + + Kept from when the pins were keyed by hash: two live stores of one + image would copy the same bytes twice and the second report would unpin + a record the first already released. The generation exists to tell + *sequential* attempts apart, not to license concurrent ones. + """ + return any(op.prefix_hash == prefix_hash for op in self._offload_pins) + + def settle_offload_store(self, op: StateStoreOperationId) -> None: """One store reported, either way. Release the units it was holding. Success and failure release identically: the pin exists to keep the bytes still during the copy, and the copy is over either way. Whether the hash becomes reachable is `StateOffloadIndex`'s business, not this one's. + + Matched on the whole operation: a report from an attempt whose pin was + already reclaimed names a generation no longer here and settles + nothing, rather than releasing a newer attempt's pin. """ - entry = self._offload_pins.pop(prefix_hash, None) + entry = self._offload_pins.pop(op, None) if entry is None: return self._release_offload_pin(entry[0]) @@ -434,9 +466,9 @@ def reclaim_stale_offload_pins(self, timeout_s: float) -> int: if timeout_s <= 0 or not self._offload_pins: return 0 cutoff = monotonic() - timeout_s - stale = [h for h, (_cid, at) in self._offload_pins.items() if at <= cutoff] - for prefix_hash in stale: - checkpoint_id, _at = self._offload_pins.pop(prefix_hash) + stale = [op for op, (_cid, at) in self._offload_pins.items() if at <= cutoff] + for op in stale: + checkpoint_id, _at = self._offload_pins.pop(op) self._release_offload_pin(checkpoint_id) self.offload_pins_reclaimed += 1 return len(stale) @@ -720,13 +752,13 @@ def attach_offload(self, index) -> None: def take_offload_stores( self, max_inflight: int - ) -> list[tuple[int, tuple[int, ...]]]: - """`(prefix_hash, unit_ids)` to hand the tier now. See the store.""" + ) -> list[tuple[StateStoreOperationId, tuple[int, ...]]]: + """`(operation, unit_ids)` to hand the tier now. See the store.""" return self.store.take_offload_stores(max_inflight) - def settle_offload_store(self, prefix_hash: int) -> None: + def settle_offload_store(self, op: StateStoreOperationId) -> None: """One store reported, either way; release the units it was holding.""" - self.store.settle_offload_store(prefix_hash) + self.store.settle_offload_store(op) def reclaim_stale_offload_pins(self, timeout_s: float) -> int: """Release offload pins whose report never came. See the store.""" diff --git a/atom/model_engine/scheduler.py b/atom/model_engine/scheduler.py index fc0dc05060..f1c12ebf46 100644 --- a/atom/model_engine/scheduler.py +++ b/atom/model_engine/scheduler.py @@ -2612,8 +2612,8 @@ def _publish_state_stores(self) -> None: type(self.kv_connector).__name__, len(stores), ) - for prefix_hash, _units in stores: - self.block_manager.settle_state_store(prefix_hash, ok=False) + for op, _units in stores: + self.block_manager.settle_state_store(op, ok=False) def _publish_state_loads(self) -> None: """Hand this pass's state-tier loads to the connector, before it builds @@ -3530,10 +3530,10 @@ def _update_from_kv_xfer_finished(self, kv_connector_output: KVConnectorOutput): # bytes are not really there must never be voted for, and a pin # exists to keep the source still during the copy -- which is over # either way. - for h in indexed: - bm.settle_state_store(int(h), ok=True) - for h in failed: - bm.settle_state_store(int(h), ok=False) + for op in indexed: + bm.settle_state_store(op, ok=True) + for op in failed: + bm.settle_state_store(op, ok=False) # Unit-side twin of `_reconcile_stalled_deferred_saves`, on the same # window and for the same reason: the pin lives in this process # while the D2H runs in the worker, so a report lost to a crashed diff --git a/tests/test_lmcache_offload_connector.py b/tests/test_lmcache_offload_connector.py index 36ed329fef..8a50f4f6ae 100644 --- a/tests/test_lmcache_offload_connector.py +++ b/tests/test_lmcache_offload_connector.py @@ -29,6 +29,7 @@ KVTransferRegion, LoadOperationId, SaveOperationId, + StateStoreOperationId, ) from atom.kv_transfer.offload import config as offcfg from atom.kv_transfer.offload._block_gpu_connector import BlockGPUConnector @@ -5185,3 +5186,52 @@ def test_a_single_leg_request_still_passes_straight_through(self): assert done == {kv_only} done, failed = worker._settle_joint(set(), set(), {"r8"}, set()) assert done == {"r8"} + + +# ── kimi_k3: a re-stored prefix must survive the aggregator's tombstone ──── + + +class TestStateStoreCompletionsCarryAGeneration: + """`KVOutputAggregator` tombstones every `(channel, operation_id)` it has + taken quorum on -- `should_tombstone` for the connector channel is + `lambda _key: True`. Under a bare hash the second store of a re-evicted + prefix was therefore dropped before quorum: its pin waited for stale + reclamation and the CPU index never learned the bytes were back.""" + + @staticmethod + def _report(agg, op): + out = KVConnectorOutput() + out.connector_completions.add( + ConnectorCompletion(STATE_INDEX_CHANNEL, op, True) + ) + return agg.aggregate([out]).connector_completions + + def test_a_bare_hash_would_be_dropped_the_second_time(self): + """Pins the aggregator behaviour this fix works around, so a change to + it does not silently make the generation look unnecessary.""" + agg = KVOutputAggregator(world_size=1) + assert self._report(agg, 4242) != set() + assert self._report(agg, 4242) == set() + + def test_two_generations_of_one_hash_both_reach_the_engine(self): + agg = KVOutputAggregator(world_size=1) + first = StateStoreOperationId(4242, 1) + second = StateStoreOperationId(4242, 2) + assert self._report(agg, first) == { + ConnectorCompletion(STATE_INDEX_CHANNEL, first, True) + } + assert self._report(agg, second) == { + ConnectorCompletion(STATE_INDEX_CHANNEL, second, True) + } + + def test_the_scheduler_half_keeps_the_operation_whole(self): + """`connector_completion` used to narrow the id to `int`, which would + undo the generation on the way to `settle_state_store`.""" + s = _k3_scheduler() + op = StateStoreOperationId(4242, 7) + assert s.connector_completion( + ConnectorCompletion(STATE_INDEX_CHANNEL, op, True) + ) + indexed, failed = s.take_state_reports() + assert indexed == {op} + assert failed == set() diff --git a/tests/test_page_unit_checkpoint.py b/tests/test_page_unit_checkpoint.py index fcda08550e..6a0fa5d40d 100644 --- a/tests/test_page_unit_checkpoint.py +++ b/tests/test_page_unit_checkpoint.py @@ -411,7 +411,8 @@ def test_handing_it_over_is_what_pins(self): _pool, store = offload_store() cid, _op = ready(store, 11) stores = store.take_offload_stores(max_inflight=4) - assert stores == [(11, store.records[cid].unit_ids)] + [(op, unit_ids)] = stores + assert (op.prefix_hash, unit_ids) == (11, store.records[cid].unit_ids) assert store.records[cid].pin_count == 1 assert not store._is_evictable(cid) @@ -468,14 +469,16 @@ def test_success_and_failure_release_identically(self): for _ in range(2): _pool, store = offload_store() cid, _op = ready(store, 11) - store.take_offload_stores(max_inflight=4) - store.settle_offload_store(11) + [(sent, _units)] = store.take_offload_stores(max_inflight=4) + store.settle_offload_store(sent) assert store.records[cid].pin_count == 0 def test_a_report_for_a_hash_never_sent_is_a_no_op(self): + from atom.kv_transfer.disaggregation.types import StateStoreOperationId + _pool, store = offload_store() cid, _op = ready(store, 11) - store.settle_offload_store(11) + store.settle_offload_store(StateStoreOperationId(11, 1)) assert store.records[cid].pin_count == 0 def test_unindex_during_the_copy_holds_the_units_to_the_end(self): @@ -484,16 +487,67 @@ def test_unindex_during_the_copy_holds_the_units_to_the_end(self): pool, store = offload_store() cid, _op = ready(store, 11) units = store.records[cid].unit_ids - store.take_offload_stores(max_inflight=4) + [(sent, _units)] = store.take_offload_stores(max_inflight=4) store.unindex(11) assert cid in store.records, "the units are still being read" assert all(pool.is_used(u) for u in units) - store.settle_offload_store(11) + store.settle_offload_store(sent) assert cid not in store.records assert not any(pool.is_used(u) for u in units) + def test_two_generations_of_one_hash_each_settle_their_own_pin(self): + """The same prefix is stored again after an eviction or a load miss. + + Keyed by bare hash, the aggregator tombstoned the first store and the + second was dropped before quorum -- its pin waited for the stale + reclaimer and its bytes were never re-indexed. Each hand-out gets its + own generation, so both attempts settle. + """ + _pool, store = offload_store() + cid1, _op = ready(store, 11) + [(first, _u)] = store.take_offload_stores(max_inflight=4) + store.settle_offload_store(first) + assert store.records[cid1].pin_count == 0 + + store.unindex(11) + cid2, _op = ready(store, 11) + [(second, _u)] = store.take_offload_stores(max_inflight=4) + assert second != first, "a re-store is its own operation" + assert second.prefix_hash == first.prefix_hash + store.settle_offload_store(second) + assert store.records[cid2].pin_count == 0 + + def test_a_late_report_from_a_superseded_attempt_settles_nothing(self): + """A reclaimed pin's report can still be in flight. It must not release + the pin a newer attempt at the same hash is holding.""" + _pool, store = offload_store() + cid, _op = ready(store, 11) + [(first, _u)] = store.take_offload_stores(max_inflight=4) + assert store.reclaim_stale_offload_pins(timeout_s=1e-9) == 1 + + [(second, _u)] = store.take_offload_stores(max_inflight=4) or [(None, None)] + if second is None: # the nomination was drained by the first hand-out + store._queue_offload_store(cid, store.records[cid]) + [(second, _u)] = store.take_offload_stores(max_inflight=4) + assert store.records[cid].pin_count == 1 + + store.settle_offload_store(first) # the late one + assert store.records[cid].pin_count == 1, "the newer pin still stands" + store.settle_offload_store(second) + assert store.records[cid].pin_count == 0 + + def test_one_hash_may_not_have_two_stores_in_flight(self): + """The generation tells sequential attempts apart; it does not license + concurrent ones, which would copy the same bytes twice and have the + first report unpin what the second is holding.""" + _pool, store = offload_store() + cid, _op = ready(store, 11) + assert len(store.take_offload_stores(max_inflight=4)) == 1 + store._queue_offload_store(cid, store.records[cid]) + assert store.take_offload_stores(max_inflight=4) == [] + def test_a_lost_report_is_reclaimed_and_counted(self): """The pin is in this process and the copy is in the worker, so a crashed worker would otherwise hold an image out of the pool forever.""" From 5f8bdee143f58e8550643aeef00b7964cf386c04 Mon Sep 17 00:00:00 2001 From: zejunchen-zejun Date: Sun, 30 Aug 2026 15:17:17 +0800 Subject: [PATCH 027/151] fix(offload): split a state store's source release from its index result Raised in review (page_unit_checkpoint.py:428): the LMCache pin-timeout argument does not apply to this source. A K3 state store bypasses `CacheEngine.store()` and reads ATOM PAGE units directly via `page_unit_views -> StagedTransfer.pack -> storage_manager.batched_put`, and those units are not protected by LMCache's GPU-source pin monitor. If the worker is stuck in gather/D2H, releasing them after the timeout lets them be reused while the old worker can still read them. The review asked for completion to be split into at least `source_released(operation_id)` (D2H finished; safe to release PAGE units) and `stored(operation_id, ok)` (CPU put/index result), both generation-qualified -- and noted that without an explicit source-release acknowledgement, timeout recovery should not claim the reader has stopped. Correct on both counts, and the docstring was the thing at fault: it justified the reclaim window by LMCache's pin timeout, "by then upstream has force-unpinned and stopped reading", which is an argument about a source this path does not use. Chasing what actually follows from that turned up a worse consequence than the one described. Reclaiming the units is a memory hazard only if the reader is still running -- but if it is, the pool may hand those units to another request whose writes the gather then picks up, and the CPU image is a mix of two prefixes filed under the first one's hash. A later request resuming onto it gets silent wrong output. The reclaim was feeding the index. Three changes: * **Source release is its own completion.** `codec.put` takes an `on_source_released` callback and fires it the moment `pack` returns -- after the gather and D2H have both been synchronized, so the GPU has provably stopped reading. The tier reports it on a new `STATE_SOURCE_CHANNEL`, generation-qualified like the index report, and `BlockManager.release_state_store_source` hands the units back right there. Previously they were held across `batched_put` as well, which cannot touch them. `_do_store` reports the release on every exit, including the failure paths, so a refused allocation or a throwing pack no longer waits for the reclaimer. * **A failed pack drains the device before it propagates.** `StagedTransfer._drain_device`. The caller must release the source on that path too or a failed store leaks an image, and the release is only safe once nothing is queued against those units. * **A reclaimed store may not be indexed.** The store remembers the operations the reclaimer took (bounded, it only has to outlive a report already in flight) and `settle_state_store` refuses to `note_stored` one of them, counting `stores_untrusted` and logging. The reclaim now recovers the memory and forfeits the entry, which is the only honest pair of outcomes available: nothing here can tell a lost report from a worker still inside the gather. The docstring says all of that instead of the LMCache argument, and the reclaim itself now logs a warning naming the operation. `stores_untrusted` is on the periodic line: any non-zero value means the window is firing on live transfers and is set too low. `take_state_source_releases` is a method of its own rather than a third element of `take_state_reports`. That tuple's arity is a contract with the delegating shell's fallback, and widening it is precisely the change that wedged every TP worker earlier in this branch. Tests: a reclaimed operation is remembered and one that reported in time is not; a reclaimed store reporting success is forfeited rather than indexed while its units still go back; the source release returns units without touching the index. --- atom/kv_transfer/offload/connector.py | 11 +++ .../offload/hybrid/kimi_k3/connector.py | 30 +++++++- .../offload/hybrid/kimi_k3/staging.py | 12 ++++ .../offload/hybrid/kimi_k3/state_object.py | 13 +++- .../offload/hybrid/kimi_k3/state_tier.py | 47 +++++++++++-- atom/model_engine/block_manager.py | 32 +++++++++ atom/model_engine/page_unit_checkpoint.py | 62 ++++++++++++++--- atom/model_engine/scheduler.py | 9 +++ atom/model_engine/state_offload.py | 7 ++ tests/test_block_manager.py | 69 +++++++++++++++++++ tests/test_page_unit_checkpoint.py | 18 +++++ 11 files changed, 294 insertions(+), 16 deletions(-) diff --git a/atom/kv_transfer/offload/connector.py b/atom/kv_transfer/offload/connector.py index 6332817ec4..678d367e80 100644 --- a/atom/kv_transfer/offload/connector.py +++ b/atom/kv_transfer/offload/connector.py @@ -197,6 +197,17 @@ def take_state_reports(self): callback = getattr(self._impl, "take_state_reports", None) return callback() if callback is not None else (set(), set()) + def take_state_source_releases(self) -> set: + """Stores whose PAGE units the GPU has finished reading. + + Its own forwarder rather than a third element of `take_state_reports`, + for the reason recorded above: that tuple's arity is a contract with + the caller, and widening it is exactly the change that wedged the pool + once already. An impl without a state tier releases nothing. + """ + callback = getattr(self._impl, "take_state_source_releases", None) + return callback() if callback is not None else set() + def get_statistics(self) -> dict[str, int]: return self._impl.get_statistics() diff --git a/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py b/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py index d7b39ea7af..809f2b584e 100644 --- a/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py +++ b/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py @@ -39,6 +39,12 @@ #: partial store resolve instead of pinning a key forever. STATE_INDEX_CHANNEL = "k3_state_index" +#: The other half of a store's completion: the GPU has stopped reading the +#: checkpoint's PAGE units. Separate from the index channel because the units +#: are the KV pool's and are free as soon as the D2H drains, while whether the +#: CPU put succeeded is decided afterwards and cannot touch them. +STATE_SOURCE_CHANNEL = "k3_state_source" + #: A save outstanding longer than this is a backend that stopped, not one that #: is busy: a 4096-token store costs ~65ms. SAVE_STALL_SECONDS = 120.0 @@ -290,6 +296,10 @@ def get_finished(self): out.connector_completions.add( ConnectorCompletion(STATE_INDEX_CHANNEL, op, False) ) + for op in self._state_tier.take_source_releases(): + out.connector_completions.add( + ConnectorCompletion(STATE_SOURCE_CHANNEL, op, True) + ) return out def _settle_joint(self, kv_done, kv_failed, state_done, state_failed): @@ -341,8 +351,9 @@ def __init__(self, config) -> None: self._save_stalled = False self._warned_save_stalled = False # Channel reports drained by the engine each step. - self._state_indexed: set[int] = set() - self._state_index_failed: set[int] = set() + self._state_indexed: set = set() + self._state_index_failed: set = set() + self._state_source_released: set = set() # Stores this worker could not even attempt (no tier). Reported as # failures so the engine releases their pinned units immediately. self._state_store_failed_locally: set[int] = set() @@ -505,6 +516,9 @@ def _claim_after_load(self, seq, hbm: int, lmc: int) -> int: # -- connector-owned channels ----------------------------------------- def connector_completion(self, completion) -> bool: + if completion.channel == STATE_SOURCE_CHANNEL: + self._state_source_released.add(completion.operation_id) + return True if completion.channel == STATE_INDEX_CHANNEL: target = ( self._state_indexed @@ -515,6 +529,18 @@ def connector_completion(self, completion) -> bool: return True return super().connector_completion(completion) + def take_state_source_releases(self) -> set: + """Drain the stores whose PAGE units the GPU has finished reading. + + A method of its own rather than a third element of + `take_state_reports`: that tuple's arity is a contract between this + class and the delegating shell's fallback, and widening it once + already cost every TP worker in the pool. + """ + released = self._state_source_released + self._state_source_released = set() + return released + def take_state_reports(self) -> tuple[set[int], set[int]]: """Drain this step's tier store reports for the engine-side index.""" indexed = self._state_indexed diff --git a/atom/kv_transfer/offload/hybrid/kimi_k3/staging.py b/atom/kv_transfer/offload/hybrid/kimi_k3/staging.py index 0967c40fe2..d90695ca61 100644 --- a/atom/kv_transfer/offload/hybrid/kimi_k3/staging.py +++ b/atom/kv_transfer/offload/hybrid/kimi_k3/staging.py @@ -202,10 +202,22 @@ def pack(self, segments: list[torch.Tensor], dst: Any) -> None: self._finish(state, state.copy_stream) except Exception: staging_buffer.free_event_valid = False + # The caller must release the source units on this path too -- + # otherwise a failed store holds a whole image out of the KV + # pool. Draining the device first is what makes that safe: the + # gather reads those units, and returning them while a kernel + # is still queued would let the pool hand them to another + # request whose writes the gather would then pick up. + self._drain_device() raise finally: self.release_buffer_if_requested(staging_buffer) + def _drain_device(self) -> None: + """Wait out every kernel this device has queued. Failure paths only.""" + if self._use_cuda(): + torch.cuda.synchronize(self.device) + def unpack(self, src: Any, segments: list[torch.Tensor]) -> None: """Scatter one packed object back over `segments` -- `pack`'s mirror. diff --git a/atom/kv_transfer/offload/hybrid/kimi_k3/state_object.py b/atom/kv_transfer/offload/hybrid/kimi_k3/state_object.py index 59455c244b..57ca5e2da9 100644 --- a/atom/kv_transfer/offload/hybrid/kimi_k3/state_object.py +++ b/atom/kv_transfer/offload/hybrid/kimi_k3/state_object.py @@ -111,7 +111,7 @@ def key(self, h: int): torch.uint8, ) - def put(self, h: int, unit_ids) -> bool: + def put(self, h: int, unit_ids, on_source_released=None) -> bool: """Store one checkpoint image. False when nothing was stored. Reads the checkpoint's PAGE units, while `get` writes an Active Slot. @@ -123,6 +123,13 @@ def put(self, h: int, unit_ids) -> bool: pressure, and a whole image is refused sooner than a KV chunk, so the state leg feels a full pool first. `puts_refused` makes that visible instead of a silent hit-rate drift. + + `on_source_released` fires the moment `pack` returns, which is after + the gather and the D2H have both been synchronized -- i.e. the moment + the GPU has stopped reading the checkpoint's PAGE units. It is separate + from the return value on purpose: the units are the KV pool's, and + holding them across `batched_put` would keep an image out of the pool + for a CPU operation that cannot touch them. """ if self._storage is None: return False @@ -146,6 +153,10 @@ def put(self, h: int, unit_ids) -> bool: # its own reference with `finally` for the same reason. try: self._staged.pack(self._backend.page_unit_views(unit_ids), obj) + # Source first: `pack` has synchronized the stream that reads the + # units, so nothing on the device touches them from here. + if on_source_released is not None: + on_source_released() self._storage.batched_put([key], [obj]) except Exception: obj.ref_count_down() diff --git a/atom/kv_transfer/offload/hybrid/kimi_k3/state_tier.py b/atom/kv_transfer/offload/hybrid/kimi_k3/state_tier.py index 887cff0001..bcbc40afef 100644 --- a/atom/kv_transfer/offload/hybrid/kimi_k3/state_tier.py +++ b/atom/kv_transfer/offload/hybrid/kimi_k3/state_tier.py @@ -37,10 +37,18 @@ def __init__(self, codec, *, max_workers: int = 1) -> None: self._done: set[str] = set() self._failed: set[str] = set() self._inflight: set = set() - # The store path's two reports, drained by `take_store_reports`. Sets - # of `StateStoreOperationId`, not of bare hashes: the engine settles - # the pin for that exact generation, and the aggregator would tombstone - # a bare hash after the first store of it. + # The store path's reports, drained by `take_store_reports`. Sets of + # `StateStoreOperationId`, not of bare hashes: the engine settles the + # pin for that exact generation, and the aggregator would tombstone a + # bare hash after the first store of it. + # + # `_source_released` is a separate report from `_indexed` because the + # two answer different questions and land at different times. The + # source is the KV pool's PAGE units and is free the instant the D2H + # drains; whether the CPU put succeeded is decided afterwards and + # cannot touch them. Reporting only the second would hold an image out + # of the pool across an operation that does not need it. + self._source_released: set = set() self._indexed: set = set() self._index_failed: set = set() @@ -105,8 +113,18 @@ def shutdown(self) -> None: def _do_store(self, op, unit_ids) -> None: stored = False + released = False + + def _source_released() -> None: + nonlocal released + released = True + try: - stored = bool(self.codec.put(int(op.prefix_hash), unit_ids)) + stored = bool( + self.codec.put( + int(op.prefix_hash), unit_ids, on_source_released=_source_released + ) + ) except Exception: # deliberately blind, see below # Deliberately blind. `codec.put` reaches into LMCache, whose # failure modes are its own, and a store that cannot happen must @@ -119,6 +137,13 @@ def _do_store(self, op, unit_ids) -> None: exc_info=True, ) with self._lock: + # Always released, on every exit. A refused allocation never read + # the units at all, and a throwing `pack` has drained the device + # before it propagated (`StagedTransfer._drain_device`), so by the + # time control is here the GPU has stopped reading them either way. + # Withholding this on the failure paths would hold an image out of + # the KV pool until the stale reclaimer noticed. + self._source_released.add(op) # Report, never apply: `StateOffloadIndex` lives in the engine # process and this runs in a spawned runner. The engine applies # these via KVConnectorOutput. @@ -144,6 +169,18 @@ def take_store_reports(self) -> tuple[set, set]: self._index_failed.clear() return indexed, index_failed + def take_source_releases(self) -> set: + """Operations whose PAGE units the GPU has finished reading. + + Drained apart from `take_store_reports`, and usually in the same step: + both are reported once `_do_store` returns, but the release is what + hands the units back and the store report is what indexes the hash. + """ + with self._lock: + released = set(self._source_released) + self._source_released.clear() + return released + def _do_load(self, req_id: str, h: int, slot: int) -> None: # The bytes land in the committed slot, where the resuming request # reads them. diff --git a/atom/model_engine/block_manager.py b/atom/model_engine/block_manager.py index 7c50e64e5e..3f527fb6e1 100644 --- a/atom/model_engine/block_manager.py +++ b/atom/model_engine/block_manager.py @@ -1093,16 +1093,48 @@ def settle_state_store(self, op, ok: bool) -> None: The index is keyed by hash, because what a resume asks is whether the prefix is in LMCache -- not which attempt put it there. """ + reclaimed = False if self.paged_state_checkpoints is not None: + reclaimed = self.paged_state_checkpoints.was_reclaimed(op) + # Idempotent: normally the source release already returned these + # units, and this is the backstop for a store that failed before + # it ever read them. self.paged_state_checkpoints.settle_offload_store(op) if self.state_offload is None: return + if ok and reclaimed: + # The stale reclaimer took this store's units back before it + # reported, and nothing can say whether the worker had stopped + # reading them. If it had not, the pool may have handed them to + # another request whose writes the gather picked up, making the CPU + # image a mix of two prefixes filed under the first one's hash -- + # and a resume onto that is silent wrong output. The bytes may well + # be fine; there is no way to know, so they are forfeited. + self.state_offload.stores_untrusted += 1 + logger.warning( + "state offload: %s reported stored, but its units were " + "reclaimed while it ran; refusing to index it.", + op, + ) + return if ok: self.state_offload.stores_completed += 1 self.state_offload.note_stored(int(op.prefix_hash)) else: self.state_offload.stores_failed += 1 + def release_state_store_source(self, op) -> None: + """The GPU has finished reading this store's PAGE units; hand them back. + + Separate from `settle_state_store` because the two answer different + questions: the units are the KV pool's and are free as soon as the D2H + drains, while whether the CPU put succeeded is decided afterwards and + cannot touch them. Holding an image out of the pool across that would + cost reuse for nothing. + """ + if self.paged_state_checkpoints is not None: + self.paged_state_checkpoints.settle_offload_store(op) + def reclaim_stale_state_store_pins(self, timeout_s: float) -> int: """Release store pins whose report never came. See the store.""" if self.paged_state_checkpoints is None: diff --git a/atom/model_engine/page_unit_checkpoint.py b/atom/model_engine/page_unit_checkpoint.py index a9c144f852..36bfd24dbd 100644 --- a/atom/model_engine/page_unit_checkpoint.py +++ b/atom/model_engine/page_unit_checkpoint.py @@ -5,6 +5,8 @@ from __future__ import annotations +import logging + from collections import OrderedDict, deque from collections.abc import Iterator, Mapping from dataclasses import dataclass @@ -14,6 +16,12 @@ from atom.model_engine.block_pool import BlockPool from atom.model_engine.sequence import Sequence +logger = logging.getLogger("atom") + +#: How many reclaimed store operations to remember. Only has to outlive a +#: report already on the wire, which is one step, so this is generous. +_RECLAIMED_MEMORY = 4096 + COPYING = "COPYING" READY = "READY" EVICTING = "EVICTING" @@ -153,6 +161,11 @@ def __init__( self._offload_pins: dict[StateStoreOperationId, tuple[int, float]] = {} self._offload_generation = 0 self.offload_pins_reclaimed = 0 + # Operations whose pin the stale reclaimer took back. Bounded, because + # this only has to outlive a report already on the wire. A store whose + # source was reclaimed underneath it may not be indexed -- see + # `was_reclaimed`. + self._offload_reclaimed: OrderedDict = OrderedDict() @property def units_per_checkpoint(self) -> int: @@ -454,14 +467,29 @@ def reclaim_stale_offload_pins(self, timeout_s: float) -> int: The pin lives here in the engine process and the D2H runs in the worker, so a lost report -- a crashed worker, a dropped completion -- - would hold 127 blocks out of the pool forever. This is the unit-side - twin of `Scheduler._reconcile_stalled_deferred_saves` and takes the - same window, derived from LMCache's own pin timeout, for the same - reason: by then upstream has force-unpinned and stopped reading. - - Recovery is total, which is what makes 20 lines enough: a leaked pin - breaks no `BlockPool` invariant and leaves no half-released state, so - zeroing the count restores the record exactly. + would hold a whole image out of the pool forever. This is the unit-side + twin of `Scheduler._reconcile_stalled_deferred_saves`. + + **This is a last resort and it does not prove the reader stopped.** It + used to be justified by LMCache's own pin timeout, on the grounds that + by then upstream has force-unpinned and stopped reading. That argument + does not hold for this source: a K3 state store bypasses + `CacheEngine.store()` entirely and gathers ATOM PAGE units through + `page_unit_views -> StagedTransfer.pack -> storage_manager.batched_put`, + and those units are not covered by LMCache's GPU-source pin monitor. + Nothing here can tell a lost report from a worker still inside the + gather. + + What follows from that is not that the units may never be reclaimed -- + that would leak an image per dropped completion -- but that a store + whose source was reclaimed can no longer be trusted: if the reader had + not stopped, the pool may have handed those units to another request + whose writes the gather then picked up, and the CPU image is a mix of + two prefixes under the first one's hash. Resuming onto it is silent + wrong output. So the operation is remembered here and + `BlockManager.settle_state_store` refuses to index it. The reclaim + recovers the memory and forfeits the entry, which is the only pair of + outcomes this can honestly offer. """ if timeout_s <= 0 or not self._offload_pins: return 0 @@ -470,9 +498,23 @@ def reclaim_stale_offload_pins(self, timeout_s: float) -> int: for op in stale: checkpoint_id, _at = self._offload_pins.pop(op) self._release_offload_pin(checkpoint_id) + self._offload_reclaimed[op] = None + while len(self._offload_reclaimed) > _RECLAIMED_MEMORY: + self._offload_reclaimed.popitem(last=False) self.offload_pins_reclaimed += 1 + logger.warning( + "state offload: reclaimed the units of store %s after %.1fs " + "with no report. The worker may still be reading them, so the " + "image will not be indexed even if a report arrives later.", + op, + timeout_s, + ) return len(stale) + def was_reclaimed(self, op) -> bool: + """Whether this store's source was taken back before it reported.""" + return op in self._offload_reclaimed + def _release_offload_pin(self, checkpoint_id: int) -> None: record = self.records.get(checkpoint_id) if record is None: @@ -764,6 +806,10 @@ def reclaim_stale_offload_pins(self, timeout_s: float) -> int: """Release offload pins whose report never came. See the store.""" return self.store.reclaim_stale_offload_pins(timeout_s) + def was_reclaimed(self, op) -> bool: + """Whether this store's source was taken back before it reported.""" + return self.store.was_reclaimed(op) + def unindex(self, h: int) -> None: # `_pending` is keyed by `(seq, hash)`, so one hash can be pending for # several sequences at once -- two turns of a conversation reaching the diff --git a/atom/model_engine/scheduler.py b/atom/model_engine/scheduler.py index f1c12ebf46..6b8f9a52f3 100644 --- a/atom/model_engine/scheduler.py +++ b/atom/model_engine/scheduler.py @@ -3530,6 +3530,15 @@ def _update_from_kv_xfer_finished(self, kv_connector_output: KVConnectorOutput): # bytes are not really there must never be voted for, and a pin # exists to keep the source still during the copy -- which is over # either way. + # The source release first, and separately: it hands the PAGE + # units back the moment the GPU stopped reading them, which is + # before the CPU put has been decided. Waiting for the store + # report would keep a whole image out of the pool across an + # operation that cannot touch it. + release = getattr(self.kv_connector, "take_state_source_releases", None) + if release is not None: + for op in release(): + bm.release_state_store_source(op) for op in indexed: bm.settle_state_store(op, ok=True) for op in failed: diff --git a/atom/model_engine/state_offload.py b/atom/model_engine/state_offload.py index 54811a3439..8c03b1a79f 100644 --- a/atom/model_engine/state_offload.py +++ b/atom/model_engine/state_offload.py @@ -47,6 +47,12 @@ def __init__(self) -> None: self.stores_completed = 0 self.stores_failed = 0 self.stores_refused = 0 + # Stores that reported success after the stale reclaimer had already + # taken their source units back. Not counted as completed: nothing can + # say the worker had stopped reading, so the image is forfeited rather + # than indexed. Any non-zero value means the reclaim window is firing + # on live transfers and is set too low. + self.stores_untrusted = 0 # ------------------------------- stores -------------------------------- # def note_stored(self, h: int) -> None: @@ -134,6 +140,7 @@ def stats(self) -> dict[str, int]: "stores_completed": self.stores_completed, "stores_failed": self.stores_failed, "stores_refused": self.stores_refused, + "stores_untrusted": self.stores_untrusted, # Load leg. "loads_attempted": self.loads_attempted, "loads_completed": self.loads_completed, diff --git a/tests/test_block_manager.py b/tests/test_block_manager.py index 54e63c295e..f0ab93063b 100644 --- a/tests/test_block_manager.py +++ b/tests/test_block_manager.py @@ -798,3 +798,72 @@ def test_a_multi_without_the_offload_backend_gets_no_index(monkeypatch): {"kv_connector": "multi", "connectors": [{"kv_connector": "moriio"}]}, ) assert bm.state_offload is None + + +# ── a store whose source was reclaimed must not be indexed ───────────────── + + +class TestReclaimedStoresAreNotIndexed: + """`reclaim_stale_offload_pins` cannot prove the worker stopped reading: + a K3 state store bypasses `CacheEngine.store()` and gathers ATOM PAGE units + directly, so LMCache's GPU-source pin monitor does not cover them. If the + reader had not stopped, the pool may have handed those units to another + request whose writes the gather picked up -- a CPU image that is a mix of + two prefixes under the first one's hash.""" + + class _Coordinator: + def __init__(self, reclaimed): + self._reclaimed = reclaimed + self.settled = [] + + def was_reclaimed(self, op): + return op in self._reclaimed + + def settle_offload_store(self, op): + self.settled.append(op) + + def _bm(self, reclaimed=()): + from atom.model_engine.state_offload import StateOffloadIndex + + bm = object.__new__(BlockManager) + bm.paged_state_checkpoints = self._Coordinator(set(reclaimed)) + bm.state_offload = StateOffloadIndex() + return bm + + def test_a_normal_store_is_indexed(self): + from atom.kv_transfer.disaggregation.types import StateStoreOperationId + + bm = self._bm() + op = StateStoreOperationId(11, 1) + bm.settle_state_store(op, ok=True) + assert 11 in bm.state_offload.hashes + assert bm.state_offload.stores_completed == 1 + assert bm.state_offload.stores_untrusted == 0 + + def test_a_reclaimed_store_reporting_success_is_forfeited(self): + from atom.kv_transfer.disaggregation.types import StateStoreOperationId + + op = StateStoreOperationId(11, 1) + bm = self._bm(reclaimed=[op]) + bm.settle_state_store(op, ok=True) + assert 11 not in bm.state_offload.hashes, "voting for it is wrong output" + assert bm.state_offload.stores_completed == 0 + assert bm.state_offload.stores_untrusted == 1 + + def test_the_units_go_back_either_way(self): + from atom.kv_transfer.disaggregation.types import StateStoreOperationId + + op = StateStoreOperationId(11, 1) + bm = self._bm(reclaimed=[op]) + bm.settle_state_store(op, ok=True) + assert bm.paged_state_checkpoints.settled == [op] + + def test_the_source_release_is_what_normally_returns_them(self): + from atom.kv_transfer.disaggregation.types import StateStoreOperationId + + bm = self._bm() + op = StateStoreOperationId(11, 1) + bm.release_state_store_source(op) + assert bm.paged_state_checkpoints.settled == [op] + # ...and it does not touch the index, which the store report owns. + assert bm.state_offload.hashes == set() diff --git a/tests/test_page_unit_checkpoint.py b/tests/test_page_unit_checkpoint.py index 6a0fa5d40d..93799a18c8 100644 --- a/tests/test_page_unit_checkpoint.py +++ b/tests/test_page_unit_checkpoint.py @@ -548,6 +548,24 @@ def test_one_hash_may_not_have_two_stores_in_flight(self): store._queue_offload_store(cid, store.records[cid]) assert store.take_offload_stores(max_inflight=4) == [] + def test_a_reclaimed_store_is_remembered_so_it_cannot_be_indexed(self): + """The reclaimer cannot tell a lost report from a worker still inside + the gather, so the units come back but the image is forfeited.""" + _pool, store = offload_store() + _cid, _op = ready(store, 11) + [(op, _u)] = store.take_offload_stores(max_inflight=4) + assert not store.was_reclaimed(op) + assert store.reclaim_stale_offload_pins(timeout_s=1e-9) == 1 + assert store.was_reclaimed(op) + + def test_a_store_that_reported_in_time_is_not_marked_reclaimed(self): + _pool, store = offload_store() + _cid, _op = ready(store, 11) + [(op, _u)] = store.take_offload_stores(max_inflight=4) + store.settle_offload_store(op) + assert store.reclaim_stale_offload_pins(timeout_s=1e-9) == 0 + assert not store.was_reclaimed(op) + def test_a_lost_report_is_reclaimed_and_counted(self): """The pin is in this process and the copy is in the worker, so a crashed worker would otherwise hold an image out of the pool forever.""" From 3c59ebd50c68aebf7e7623d2fb025de48b786d44 Mon Sep 17 00:00:00 2001 From: zejunchen-zejun Date: Sun, 30 Aug 2026 15:17:17 +0800 Subject: [PATCH 028/151] perf(offload): give state loads their own lane, and measure the wait Raised in review (state_tier.py:77): the serial executor creates head-of-line blocking on the TTFT path. A state load submitted in a later scheduler step queues behind all earlier stores, and one stuck store blocks every later load indefinitely; the per-step call order only puts same-step loads first and cannot overtake already queued stores. The review asked for load priority, or separate load/store lanes with an explicit shared staging-memory budget, and for load queue-wait time and oldest store age to be exposed -- the operation-count cap describes neither bytes nor latency. Right on the mechanism, and the previous docstring conceded the point while pointing at a metric that does not exist: it told the reader to "measure `state_load_queue_wait_ms` before adding a priority", and that name appears nowhere else in the tree. Nobody could have acted on it. Separate lanes, with the shared resource named: * `_load_executor` and `_store_executor`, so a load is never behind a backlog of stores. Which is the fix -- submit order was never able to reorder work already queued. * The lanes share HBM, not a queue, and that is now explicit. `StagedTransfer` keeps its staging buffer in `threading.local` and sizes it to a whole entry (~55 MiB on K3), so every lane inside a transfer holds one. `staging_lanes` (default 2, one per lane) is a `BoundedSemaphore` held across the codec call. At 2 a load never waits on a store at all; at 1 the two share one buffer, halving the standing HBM at the cost of waiting out the single in-flight store -- but still not the backlog behind it, which is the actual complaint. Note for sizing: the default raises standing state-staging HBM from one buffer per rank to two (~55 MiB to ~110 MiB on K3). Set `staging_lanes=1` to keep the previous footprint. Latency, since counts cannot express it: * `state_load_queue_wait_ms_last` / `_max` and `state_loads_started`, timed from submission to the moment the lane picks the load up. * `state_oldest_store_age_s` and `state_stores_inflight`. Age is what distinguishes a backend that stopped from one that is busy; the in-flight cap describes neither bytes nor time. * A load that waited more than 50ms for its lane logs a warning naming the oldest outstanding store, because that wait is TTFT. Tests: a load lands while four stores are wedged; with `staging_lanes=1` it waits out the in-flight store and then lands; oldest-store age is visible while a store hangs and returns to zero when it drains; and the queue-wait metric the old comment promised now exists and is recorded. --- .../offload/hybrid/kimi_k3/state_tier.py | 115 +++++++++++++++--- tests/test_lmcache_offload_connector.py | 96 +++++++++++++++ 2 files changed, 194 insertions(+), 17 deletions(-) diff --git a/atom/kv_transfer/offload/hybrid/kimi_k3/state_tier.py b/atom/kv_transfer/offload/hybrid/kimi_k3/state_tier.py index bcbc40afef..3d8e331d32 100644 --- a/atom/kv_transfer/offload/hybrid/kimi_k3/state_tier.py +++ b/atom/kv_transfer/offload/hybrid/kimi_k3/state_tier.py @@ -15,10 +15,15 @@ import logging import threading from concurrent.futures import ThreadPoolExecutor +from time import monotonic from typing import Any logger = logging.getLogger(__name__) +#: A load waiting longer than this for its lane is worth a line: it is time +#: added straight to TTFT, and the in-flight *count* cannot show it. +_LOAD_WAIT_WARN_MS = 50.0 + class StateOffloadTier: """Moves bytes; decides nothing, and holds no index. @@ -28,11 +33,31 @@ class StateOffloadTier: Neither side can hold a second opinion about what is stored. """ - def __init__(self, codec, *, max_workers: int = 1) -> None: + def __init__(self, codec, *, max_workers: int = 1, staging_lanes: int = 2) -> None: self.codec = codec - self._executor = ThreadPoolExecutor( - max_workers=max_workers, thread_name_prefix="lmc-state" + # Two lanes, not one queue. A load is on the TTFT critical path and a + # store is not, but a single serial executor made that ordering + # unenforceable: a load submitted in a later scheduler step queued + # behind every store already sitting in front of it, and one store + # stuck in gather/D2H blocked every later load for as long as it ran. + # Putting same-step loads first in the submit order does not help -- + # it cannot overtake work already queued. + self._load_executor = ThreadPoolExecutor( + max_workers=max_workers, thread_name_prefix="lmc-state-load" + ) + self._store_executor = ThreadPoolExecutor( + max_workers=max_workers, thread_name_prefix="lmc-state-store" ) + # What the lanes share is HBM, not a queue. `StagedTransfer` keeps its + # staging buffer in `threading.local` and sizes it to a whole entry + # (~55 MiB on K3), so each lane inside a transfer holds one. The + # budget is stated here rather than left implicit in the thread count: + # 2 gives each lane its own buffer, so a load never waits on a store at + # all; 1 makes them share one, halving the standing HBM at the cost of + # a load waiting out the single in-flight store. Either way a load + # never queues behind a *backlog* of stores, which is the actual + # head-of-line problem. + self._staging_budget = threading.BoundedSemaphore(max(1, int(staging_lanes))) self._lock = threading.Lock() self._done: set[str] = set() self._failed: set[str] = set() @@ -51,6 +76,14 @@ def __init__(self, codec, *, max_workers: int = 1) -> None: self._source_released: set = set() self._indexed: set = set() self._index_failed: set = set() + # Latency, not operation counts. The in-flight cap bounds how many + # transfers are outstanding; it says nothing about bytes or about how + # long a load waited, which is what a TTFT regression is made of. + self.load_queue_wait_ms_last = 0.0 + self.load_queue_wait_ms_max = 0.0 + self.loads_started = 0 + # op -> monotonic at submission, for `oldest_store_age_s`. + self._store_submitted_at: dict = {} def _register(self, fut) -> None: """Add *fut* to the inflight set and attach a callback that removes it @@ -67,6 +100,31 @@ def _discard(f): fut.add_done_callback(_discard) + def oldest_store_age_s(self) -> float: + """How long the oldest unfinished store has been outstanding. + + Zero when nothing is in flight. This is the number that says a backend + has stopped rather than being busy, and the one the in-flight *count* + cannot express. + """ + with self._lock: + if not self._store_submitted_at: + return 0.0 + oldest = min(self._store_submitted_at.values()) + return max(0.0, monotonic() - oldest) + + def stats(self) -> dict: + """Queue latency for the periodic line. Counters live in the engine.""" + with self._lock: + inflight_stores = len(self._store_submitted_at) + return { + "state_load_queue_wait_ms_last": round(self.load_queue_wait_ms_last, 2), + "state_load_queue_wait_ms_max": round(self.load_queue_wait_ms_max, 2), + "state_loads_started": self.loads_started, + "state_stores_inflight": inflight_stores, + "state_oldest_store_age_s": round(self.oldest_store_age_s(), 2), + } + def submit_store(self, op, unit_ids) -> None: """Pack the checkpoint image in `unit_ids` for LMCache, under `op`. @@ -80,18 +138,20 @@ def submit_store(self, op, unit_ids) -> None: stream is writing them and the packer gathers straight from where they sit. """ - self._register(self._executor.submit(self._do_store, op, unit_ids)) + with self._lock: + self._store_submitted_at[op] = monotonic() + self._register(self._store_executor.submit(self._do_store, op, unit_ids)) def submit_load(self, req_id: str, h: int, slot: int) -> None: """Fetch `h` into pool slot `slot` for the parked request `req_id`. - Shares one serial executor with the stores: `StagedTransfer` keeps its - staging buffer in `threading.local`, so a second thread costs a second - resident buffer per rank. A load is on the TTFT critical path and a - store is not, so if stores become frequent this queue wants a priority; - measure `state_load_queue_wait_ms` before adding one. + Its own lane, so a load is never behind a backlog of stores. What the + two lanes still share is the staging-memory budget; see `__init__`. + `state_load_queue_wait_ms` measures what is left of the wait. """ - self._register(self._executor.submit(self._do_load, req_id, h, slot)) + self._register( + self._load_executor.submit(self._do_load, req_id, h, slot, monotonic()) + ) def drain(self) -> None: """Block until every submitted transfer has settled. Tests and shutdown @@ -109,7 +169,8 @@ def get_finished(self) -> tuple[set[str], set[str]]: return done, failed def shutdown(self) -> None: - self._executor.shutdown(wait=True) + self._load_executor.shutdown(wait=True) + self._store_executor.shutdown(wait=True) def _do_store(self, op, unit_ids) -> None: stored = False @@ -120,11 +181,14 @@ def _source_released() -> None: released = True try: - stored = bool( - self.codec.put( - int(op.prefix_hash), unit_ids, on_source_released=_source_released + with self._staging_budget: + stored = bool( + self.codec.put( + int(op.prefix_hash), + unit_ids, + on_source_released=_source_released, + ) ) - ) except Exception: # deliberately blind, see below # Deliberately blind. `codec.put` reaches into LMCache, whose # failure modes are its own, and a store that cannot happen must @@ -143,6 +207,7 @@ def _source_released() -> None: # time control is here the GPU has stopped reading them either way. # Withholding this on the failure paths would hold an image out of # the KV pool until the stale reclaimer noticed. + self._store_submitted_at.pop(op, None) self._source_released.add(op) # Report, never apply: `StateOffloadIndex` lives in the engine # process and this runs in a spawned runner. The engine applies @@ -181,7 +246,7 @@ def take_source_releases(self) -> set: self._source_released.clear() return released - def _do_load(self, req_id: str, h: int, slot: int) -> None: + def _do_load(self, req_id: str, h: int, slot: int, submitted_at=None) -> None: # The bytes land in the committed slot, where the resuming request # reads them. # @@ -189,9 +254,25 @@ def _do_load(self, req_id: str, h: int, slot: int) -> None: # under a hash the engine's index still advertises. Retracting that # claim is the engine's job -- it owns the index -- and it does it from # the report below. + if submitted_at is not None: + waited_ms = max(0.0, monotonic() - submitted_at) * 1000.0 + with self._lock: + self.load_queue_wait_ms_last = waited_ms + self.load_queue_wait_ms_max = max( + self.load_queue_wait_ms_max, waited_ms + ) + self.loads_started += 1 + if waited_ms >= _LOAD_WAIT_WARN_MS: + logger.warning( + "state offload: a state load waited %.0fms for its lane " + "(oldest store outstanding %.1fs). This is TTFT.", + waited_ms, + self.oldest_store_age_s(), + ) ok = False try: - ok = bool(self.codec.get(h, slot)) + with self._staging_budget: + ok = bool(self.codec.get(h, slot)) except Exception: # a failed load is a normal path # Same reasoning as `_do_store`, and here a miss is expected: # LMCache's LRU can drop bytes under a hash the index still diff --git a/tests/test_lmcache_offload_connector.py b/tests/test_lmcache_offload_connector.py index 8a50f4f6ae..58f88a0b10 100644 --- a/tests/test_lmcache_offload_connector.py +++ b/tests/test_lmcache_offload_connector.py @@ -5235,3 +5235,99 @@ def test_the_scheduler_half_keeps_the_operation_whole(self): indexed, failed = s.take_state_reports() assert indexed == {op} assert failed == set() + + +# ── kimi_k3: loads must not queue behind a backlog of stores ─────────────── + + +class TestStateLoadsAndStoresRunInSeparateLanes: + """A load is on the TTFT critical path and a store is not, but one serial + executor made that ordering unenforceable: a load submitted in a later step + sat behind every store already queued, and a stuck store blocked all of + them. Putting same-step loads first cannot overtake queued work.""" + + class _BlockingCodec: + """A codec whose stores hang until released; loads always land.""" + + def __init__(self): + self.gate = threading.Event() + self.loaded = threading.Event() + + def put(self, h, unit_ids, on_source_released=None): + self.gate.wait(timeout=5) + if on_source_released is not None: + on_source_released() + return True + + def get(self, h, slot): + self.loaded.set() + return True + + def _tier(self, codec, **kw): + from atom.kv_transfer.offload.hybrid.kimi_k3.state_tier import ( + StateOffloadTier, + ) + + return StateOffloadTier(codec, **kw) + + def test_a_load_lands_while_a_store_is_still_stuck(self): + codec = self._BlockingCodec() + tier = self._tier(codec) + try: + for gen in range(4): + tier.submit_store(StateStoreOperationId(gen, gen + 1), (0,)) + tier.submit_load("r1", 77, 0) + assert codec.loaded.wait( + timeout=5 + ), "the load waited behind the store backlog" + done, failed = tier.get_finished() + assert done == {"r1"} + assert failed == set() + finally: + codec.gate.set() + tier.shutdown() + + def test_one_shared_staging_buffer_still_separates_the_queues(self): + """`staging_lanes=1` halves the standing HBM: a load then waits out the + single in-flight store, but still not the backlog behind it.""" + codec = self._BlockingCodec() + tier = self._tier(codec, staging_lanes=1) + try: + for gen in range(3): + tier.submit_store(StateStoreOperationId(gen, gen + 1), (0,)) + tier.submit_load("r1", 77, 0) + assert not codec.loaded.wait(timeout=0.2), "held by the in-flight store" + codec.gate.set() + assert codec.loaded.wait(timeout=5) + finally: + codec.gate.set() + tier.shutdown() + + def test_the_oldest_store_age_is_visible_while_it_hangs(self): + codec = self._BlockingCodec() + tier = self._tier(codec) + try: + assert tier.oldest_store_age_s() == 0.0 + tier.submit_store(StateStoreOperationId(1, 1), (0,)) + assert tier.oldest_store_age_s() >= 0.0 + assert tier.stats()["state_stores_inflight"] == 1 + finally: + codec.gate.set() + tier.drain() + assert tier.oldest_store_age_s() == 0.0 + tier.shutdown() + + def test_load_queue_wait_is_measured(self): + """The metric the old comment told readers to measure did not exist.""" + codec = self._BlockingCodec() + codec.gate.set() + tier = self._tier(codec) + try: + tier.submit_load("r1", 77, 0) + tier.drain() + stats = tier.stats() + assert stats["state_loads_started"] == 1 + assert stats["state_load_queue_wait_ms_last"] >= 0.0 + assert "state_load_queue_wait_ms_max" in stats + finally: + tier.shutdown() From ad31789b4f443b5c749df9e5c30dc3125fd971f8 Mon Sep 17 00:00:00 2001 From: zejunchen-zejun Date: Sun, 30 Aug 2026 15:17:17 +0800 Subject: [PATCH 029/151] fix(offload): decide the state tier from a capability, not the connector name Raised in review (state_offload.py:165): a connector-name check is too broad to establish this capability. The engine installs a `StateOffloadIndex` for any `lmcache_offload`, while the worker only builds `StateOffloadTier` for the K3 layout and can still refuse it for PP>1, a missing backend or layout id, or incompatible geometry. It also does not distinguish `kv_producer`/`kv_consumer` state load/store rights. The review identified that mismatch as the root cause behind several tier-none fallback bugs, and asked for an explicit capability descriptor (`can_load_state`, `can_store_state`, layout/checkpoint kind, PP support) including exactly-one-provider validation for `multi`. Accepted in full, including the root-cause reading: every tier-none fallback path in this branch exists because the engine could install an index against a worker that would go on to refuse the tier. Narrowing the predicate is the fix; the fallbacks are the symptom. `StateTierCapability(can_store_state, can_load_state, layout, reason)`, derived from configuration alone -- it has to be, because this runs at `BlockManager.__init__`, before any worker connector exists. It agrees with the worker by construction rather than by restating its rules: the layout comes from the same `select_offload_layout` the worker calls, and the roles from the same sets `OffloadWorkerMixin` reads for `_do_save`/`_do_load`. Four conditions, each of which the name check could not see: * **Layout.** Only `kimi_k3` offloads per-request state -- `dense` has none and `hybrid` (DSV4) keeps its own in the SLOT sidecar -- so an offload connector on a dense model now hosts no tier. * **Pipeline depth.** The worker refuses PP>1 outright, because `CacheEngineKey` has no PP component and two stages at one TP rank would overwrite each other. The engine used to build an index anyway. * **Role.** `can_store_state` / `can_load_state` are granted separately. A `kv_producer` no longer votes for loads its worker will not serve, and a `kv_consumer` no longer hands over stores whose pins nothing would release -- enforced at `request_load` and `take_state_stores`, and at `attach_offload(sink=...)` so a load-only role nominates nothing in the first place. * **`multi`.** Exactly one offload sub-connector, as asked. Two would each hold half an answer: the bytes ride one worker half and the completions ride that connector's `get_finished`. The layout is also resolved against the sub-connector's own transfer config, since an `offload_layout` override lives there and not on the composite. `reason` is carried so the log names which condition refused the tier rather than leaving the operator to infer it from an absent counter. `kv_connector_hosts_state_tier` stays as a thin wrapper, but now takes the whole config -- passing only `kv_transfer_config` is what confined it to a name check to begin with. BEHAVIOUR CHANGE worth a second look in review: configurations that previously built an index now do not (any dense or hybrid model, and any PP>1 run). That is the point of the change, but it means a deployment relying on the old over-broad predicate will see the tier turn off, with the reason on the `[State Cache]` line. Two existing tests asserted the over-broad behaviour (an index for any `lmcache_offload`, on a dense `MockConfig`) and are updated: that is the bug being fixed, not a regression. Added: dense and hybrid layouts host no tier and say why; PP=2 hosts none; a producer stores but never votes; a consumer votes but never hands over a store; a `multi` listing two offload connectors is refused. --- atom/model_engine/block_manager.py | 34 ++++- atom/model_engine/page_unit_checkpoint.py | 7 +- atom/model_engine/state_offload.py | 172 +++++++++++++++++++--- tests/test_block_manager.py | 67 ++++++++- 4 files changed, 249 insertions(+), 31 deletions(-) diff --git a/atom/model_engine/block_manager.py b/atom/model_engine/block_manager.py index 3f527fb6e1..f57a6e850f 100644 --- a/atom/model_engine/block_manager.py +++ b/atom/model_engine/block_manager.py @@ -161,11 +161,21 @@ def __init__( # coordinator has to know at construction whether anything can carry a # store: without a sink it must nominate nothing, since a pin nobody # releases holds a whole image out of the pool forever. - from atom.model_engine.state_offload import kv_connector_hosts_state_tier - - kv_offload_enabled = kv_connector_hosts_state_tier( - getattr(config, "kv_transfer_config", None) - ) + from atom.model_engine.state_offload import state_tier_capability + + # A capability derived from the whole config, not the connector's name. + # The name only says which class is constructed; whether that class + # builds a `StateOffloadTier` also depends on the layout it resolves, + # the pipeline depth, and its role. Installing an index against a + # worker that will refuse the tier is what left stores emitted with + # nowhere to go. + self.state_tier_capability = state_tier_capability(config) + kv_offload_enabled = self.state_tier_capability.hosts_state_tier + if not kv_offload_enabled and self.state_tier_capability.reason: + logger.info( + "[State Cache] CPU state tier off: %s.", + self.state_tier_capability.reason, + ) self.paged_state_checkpoints: PagedStateCheckpointCoordinator | None = None if checkpoint_spec is not None: enabled = self.enable_prefix_caching and self.num_state_slots > 0 @@ -287,13 +297,19 @@ def __init__( # units the coordinator owns, not a slot the pool is about to hand # away -- which is what the staging ring existed to rescue. if kv_offload_enabled: - self.state_offload = StateOffloadIndex() + self.state_offload = StateOffloadIndex( + can_store=self.state_tier_capability.can_store_state, + can_load=self.state_tier_capability.can_load_state, + ) # Attached rather than passed at construction: the coordinator is # built before the switch is read (it needs `checkpoint_spec`, # which comes off `state_runtime`). One object, two uses -- the # coordinator votes off `hashes` and drains stores into it. if self.paged_state_checkpoints is not None: - self.paged_state_checkpoints.attach_offload(self.state_offload) + self.paged_state_checkpoints.attach_offload( + self.state_offload, + sink=self.state_tier_capability.can_store_state, + ) # The demand funnel: recorded at admission, cut for when a prefill # chunk is shortened to land on it, kept when the state pool files it. # Counted at all three because a gap between any two is a different @@ -1076,6 +1092,10 @@ def take_state_stores(self, max_inflight: int) -> list[tuple]: """ if self.paged_state_checkpoints is None or self.state_offload is None: return [] + if not self.state_offload.can_store: + # A load-only role. Handing over a store pins units against a + # report the worker's save half will never produce. + return [] out = self.paged_state_checkpoints.take_offload_stores(max_inflight) self.state_offload.stores_attempted += len(out) return out diff --git a/atom/model_engine/page_unit_checkpoint.py b/atom/model_engine/page_unit_checkpoint.py index 36bfd24dbd..317fb1fa29 100644 --- a/atom/model_engine/page_unit_checkpoint.py +++ b/atom/model_engine/page_unit_checkpoint.py @@ -778,7 +778,7 @@ def contains(self, h: int) -> bool: """ return self.store.contains(h) - def attach_offload(self, index) -> None: + def attach_offload(self, index, *, sink: bool = True) -> None: """Wire the CPU tier in, after both objects exist. Not a constructor argument because the two are built in the wrong @@ -790,7 +790,10 @@ def attach_offload(self, index) -> None: units nothing releases. """ self.offload = index - self.store._offload_sink = index is not None + # `sink` is the store half specifically: a load-only role still votes + # off `hashes` but must nominate nothing, since a nomination that is + # handed over takes a pin nobody will release. + self.store._offload_sink = index is not None and bool(sink) def take_offload_stores( self, max_inflight: int diff --git a/atom/model_engine/state_offload.py b/atom/model_engine/state_offload.py index 8c03b1a79f..02e632a7f1 100644 --- a/atom/model_engine/state_offload.py +++ b/atom/model_engine/state_offload.py @@ -12,6 +12,7 @@ """ import logging +from dataclasses import dataclass, replace logger = logging.getLogger("atom") @@ -26,7 +27,12 @@ class StateOffloadIndex: (`fail_load` -> `forget`). """ - def __init__(self) -> None: + def __init__(self, *, can_store: bool = True, can_load: bool = True) -> None: + # The two legs are separately granted. A `kv_producer` role saves and + # never loads, a `kv_consumer` loads and never saves, and offering a + # load the worker will not serve parks the request that took it. + self.can_store = bool(can_store) + self.can_load = bool(can_load) self.hashes: set[int] = set() # req_id -> hash, for loads offered and not yet settled. Keyed by # request because that is what comes back, on the same @@ -76,6 +82,10 @@ def request_load(self, req_id, h: int) -> bool: a worker report, so offering one for a hash never stored would park the request against bytes no `get` can produce. """ + if not self.can_load: + # A store-only role. Voting for a hash whose load nothing will + # serve parks the request against a report that never comes. + return False if h not in self.hashes: return False if req_id in self.pending_loads: @@ -149,31 +159,151 @@ def stats(self) -> dict[str, int]: } -# The connector backends whose worker half builds a `StateOffloadTier`. Only -# the offload backend does; `multi` qualifies when it lists one. -_STATE_TIER_BACKENDS = frozenset({"lmcache_offload"}) +#: Offload layouts whose worker half builds a `StateOffloadTier`. Not every +#: `lmcache_offload` does: `dense` has no per-request state at all and `hybrid` +#: (DSV4) keeps its own in the SLOT sidecar, so only K3 offloads state. +_STATE_TIER_LAYOUTS = frozenset({"kimi_k3"}) + +#: Roles from which the worker will save / load. Mirrors `_do_save` / `_do_load` +#: in `OffloadWorkerMixin`; the state legs ride those same halves. +_SAVE_ROLES = frozenset({"offload", "kv_both", "kv_producer"}) +_LOAD_ROLES = frozenset({"offload", "kv_both", "kv_consumer"}) + + +@dataclass(frozen=True) +class StateTierCapability: + """What the worker half will actually do for the state tier. + + Derived from configuration, not from the connector's public name. The name + only says which connector class is constructed; whether that class builds a + `StateOffloadTier` depends on the layout it resolves, the pipeline depth, + and the role it was given. The engine used to install a `StateOffloadIndex` + for any `lmcache_offload`, so a K3 run under PP>1 -- or any `dense`/`hybrid` + run -- got an index against a worker that would never build the tier. That + mismatch is what left stores emitted with nowhere to go, and it is the + common root of the tier-none fallback paths. + + `reason` is filled in whenever the tier is off, so the log says which of the + four conditions refused it rather than leaving the operator to guess. + """ + + can_store_state: bool + can_load_state: bool + layout: str | None + reason: str = "" + + @property + def hosts_state_tier(self) -> bool: + """Whether to build the engine-side index at all. + + Either leg is enough to need one: a store-only role still has to hold + pins and settle reports, and a load-only role still has to vote. + """ + return self.can_store_state or self.can_load_state + + +def _offload_subconfig(cfg: dict) -> tuple[dict | None, str]: + """The one offload sub-connector under `multi`, or a reason there is none. + + Exactly one, as asked: the tier's bytes ride a specific connector's worker + half and its completions ride that connector's `get_finished`, so two + providers would each hold half an answer and a request parked on the wrong + one would never be reported. + """ + subs = [s for s in (cfg.get("connectors") or ()) if isinstance(s, dict)] + offload = [s for s in subs if s.get("kv_connector") in _STATE_TIER_BACKENDS] + if not offload: + return None, "multi lists no offload connector" + if len(offload) > 1: + return None, f"multi lists {len(offload)} offload connectors; expected one" + return offload[0], "" + +class _SubConnectorView: + """`config` with one `multi` sub-connector's transfer config in front. -def kv_connector_hosts_state_tier(kv_transfer_config) -> bool: - """Whether the configured KV connector can actually run the state tier. + Attribute lookups fall through to the real config, so the model fields the + layout selector reads are unchanged; only `kv_transfer_config` is replaced. + """ + + def __init__(self, config, sub: dict) -> None: + self._config = config + self.kv_transfer_config = sub + + def __getattr__(self, name): + return getattr(self._config, name) + + +def state_tier_capability(config) -> StateTierCapability: + """What the worker will do for the state tier, decided from config alone. - The tier is not standalone: its bytes ride the KV connector's worker half - and its completions ride that connector's `get_finished`. Against any other - backend a load would be offered and never reported, parking the request - that took it -- so this gates whether the index is built at all, and a - false negative (tier off) is much cheaper than a false positive. + A capability descriptor rather than a name check, because the two are not + the same question and the worker refuses on three conditions the name + cannot see. Config alone because this runs in the engine process at + `BlockManager.__init__`, before any worker connector exists -- so it must + agree with the worker by construction, which is why the layout comes from + the same `select_offload_layout` the worker uses and the roles from the + same sets `OffloadWorkerMixin` reads. - A name check rather than a capability probe: this runs in the engine - process at `BlockManager.__init__`, before any worker connector exists. + A false negative (tier off) is much cheaper than a false positive: off + costs reuse, on-against-a-worker-that-refuses parks requests against loads + nobody can report. """ - cfg = kv_transfer_config or {} + from atom.kv_transfer.offload.config import select_offload_layout + + none = StateTierCapability(False, False, None, "") + cfg = getattr(config, "kv_transfer_config", None) or {} if not isinstance(cfg, dict): - return False + return replace(none, reason="no kv_transfer_config") + name = cfg.get("kv_connector", "moriio") + layout_config = config if name == "multi": - subs = cfg.get("connectors") or () - return any( - isinstance(s, dict) and s.get("kv_connector") in _STATE_TIER_BACKENDS - for s in subs - ) - return name in _STATE_TIER_BACKENDS + sub, why = _offload_subconfig(cfg) + if sub is None: + return replace(none, reason=why) + cfg = sub + name = cfg.get("kv_connector") + # The layout selector reads `config.kv_transfer_config`, and under + # `multi` the interesting one is the sub-connector's -- an + # `offload_layout` override lives there, not on the composite. Model + # fields still come from the real config. + layout_config = _SubConnectorView(config, cfg) + if name not in _STATE_TIER_BACKENDS: + return replace(none, reason=f"connector {name!r} hosts no state tier") + + # The worker refuses PP outright: `CacheEngineKey` has no PP component, so + # two stages at the same TP rank would overwrite each other's entries. + pp_size = int(getattr(config, "pipeline_parallel_size", 1) or 1) + if pp_size > 1: + return replace(none, reason=f"pipeline_parallel_size={pp_size}") + + try: + layout = select_offload_layout(layout_config) + except ValueError as exc: # an unknown explicit override + return replace(none, reason=str(exc)) + if layout not in _STATE_TIER_LAYOUTS: + return replace(none, reason=f"layout {layout!r} offloads no state") + + role = cfg.get("kv_role", "offload") + can_store = role in _SAVE_ROLES + can_load = role in _LOAD_ROLES + if not (can_store or can_load): + return replace(none, reason=f"kv_role {role!r} neither saves nor loads") + return StateTierCapability(can_store, can_load, layout) + + +#: The connector backends whose worker half can build a `StateOffloadTier`. +#: `multi` qualifies when it lists exactly one. +_STATE_TIER_BACKENDS = frozenset({"lmcache_offload"}) + + +def kv_connector_hosts_state_tier(config) -> bool: + """Whether the configured connector will really run the state tier. + + Thin wrapper over `state_tier_capability` for callers that only need the + yes/no. Takes the whole config: the layout and the pipeline depth are as + much a part of the answer as the connector name, and passing only + `kv_transfer_config` is what made this a name check in the first place. + """ + return state_tier_capability(config).hosts_state_tier diff --git a/tests/test_block_manager.py b/tests/test_block_manager.py index f0ab93063b..69d5282da1 100644 --- a/tests/test_block_manager.py +++ b/tests/test_block_manager.py @@ -733,7 +733,14 @@ def test_state_offload_is_none_when_tier_is_off(block_manager): # ── the index is built only where something can report to it ───────────── -_OFFLOAD_KVC = {"kv_connector": "lmcache_offload", "kv_role": "offload"} +# The layout is part of the capability, not an incidental detail: only K3 +# offloads per-request state, so an offload connector on a dense model hosts no +# tier however it is configured. +_OFFLOAD_KVC = { + "kv_connector": "lmcache_offload", + "kv_role": "offload", + "offload_layout": "kimi_k3", +} def _bm_with_state_tier(monkeypatch, kv_transfer_config): @@ -779,6 +786,64 @@ def test_the_index_is_built_for_the_offload_connector(monkeypatch): assert not any(hasattr(cache, "offload") for cache in bm.state_caches) +def test_a_dense_layout_hosts_no_state_tier_however_it_is_configured(monkeypatch): + """The name check said yes to every `lmcache_offload`. Only K3's worker + half builds a `StateOffloadTier`: dense has no per-request state and DSV4 + keeps its own in the SLOT sidecar.""" + for layout in ("dense", "hybrid"): + bm = _bm_with_state_tier( + monkeypatch, {**_OFFLOAD_KVC, "offload_layout": layout} + ) + assert bm.state_offload is None, layout + assert layout in bm.state_tier_capability.reason + + +def test_pipeline_parallelism_hosts_no_state_tier(monkeypatch): + """The worker refuses PP outright -- `CacheEngineKey` has no PP component, + so two stages at one TP rank would overwrite each other. The engine used to + build an index anyway and emit stores with nowhere to go.""" + cfg = MockConfig( + enable_prefix_caching=True, + kv_transfer_config=_OFFLOAD_KVC, + pool_entries={"state": 4}, + pool_entries_per_req={"state": 1}, + ) + cfg.pipeline_parallel_size = 2 + bm = BlockManager(cfg) + assert bm.state_offload is None + assert "pipeline_parallel_size=2" in bm.state_tier_capability.reason + + +def test_a_producer_role_stores_but_never_votes_for_a_load(monkeypatch): + bm = _bm_with_state_tier(monkeypatch, {**_OFFLOAD_KVC, "kv_role": "kv_producer"}) + assert bm.state_offload is not None + assert bm.state_offload.can_store and not bm.state_offload.can_load + bm.state_offload.note_stored(11) + assert bm.state_offload.request_load("r1", 11) is False + + +def test_a_consumer_role_votes_but_never_hands_over_a_store(monkeypatch): + bm = _bm_with_state_tier(monkeypatch, {**_OFFLOAD_KVC, "kv_role": "kv_consumer"}) + assert bm.state_offload is not None + assert bm.state_offload.can_load and not bm.state_offload.can_store + assert bm.take_state_stores(4) == [] + + +def test_a_multi_listing_two_offload_connectors_is_refused(monkeypatch): + """The tier's bytes ride one connector's worker half and its completions + ride that connector's `get_finished`, so two providers would each hold half + an answer.""" + bm = _bm_with_state_tier( + monkeypatch, + { + "kv_connector": "multi", + "connectors": [_OFFLOAD_KVC, dict(_OFFLOAD_KVC)], + }, + ) + assert bm.state_offload is None + assert "expected one" in bm.state_tier_capability.reason + + def test_the_index_is_built_for_a_multi_that_lists_the_offload_backend( monkeypatch, ): From fb4a069fef7dbd724f18e73f8b68607be10cde9b Mon Sep 17 00:00:00 2001 From: zejunchen-zejun Date: Sun, 30 Aug 2026 15:17:17 +0800 Subject: [PATCH 030/151] fix(offload): aim the PAGE state leg at the joint boundary, and gate the claim Raised in review (block_manager.py:848): `allocate()` restores/loads per-request state using `hit_hash` even when `can_allocate()` computed a joint boundary above the HBM hit. In the joint-path case -- notably when `num_cached_blocks == 0` but `state_joint_boundary_hash != -1` -- no state restore or load is requested, so the KV leg can resume a request with `num_cached_tokens` raised past what its state actually covers, which is silent wrong output. The review asked for the joint boundary hash to be used as the restore/load target when present. Correct, and the most severe finding on this PR. The fork branch of `allocate` already used `joint_hash if joint_hash != -1 else hit_hash`; the PAGE branch -- K3's, the one this PR is about -- passed `hit_hash` alone. `hit_hash` is assigned inside the claim walk at `i + 1 == num_cached_blocks`, so it is the hash at the *HBM* hit, and a joint boundary sits strictly above it by construction. Two ways it went wrong, and the review's parenthetical is the worse one: * `num_cached_blocks == 0` -- the case the joint path exists for, and the one worked through in the PR description -- never assigns `hit_hash` at all. `_attach_state_slots` then took its `-1` cold-start exit and returned True having requested nothing. * A non-zero hit restored the checkpoint covering `[0, hbm)`. Either way `state_holds` stayed True, nothing disowned the boundary, `_decide_load_after_alloc` issued the KV load to it, and `_claim_after_load` raised `num_cached_tokens` to the boundary. The forward then resumed over a prefix with a cold Active Slot, or one holding a shorter history. `has_initial_state` is `num_cached_tokens > 0`, so the recurrence simply continues from the wrong place: wrong output, no exception, nothing counted. Two changes, because one of them is a fix and the other is the invariant it should never have needed: * The PAGE branch uses the joint hash when there is one, exactly as the fork branch does. The two branches now ask the same question. * `_state_leg_secured` is a backstop, checked separately rather than folded into `state_holds`: a claimed joint boundary with no restore queued, no fork source, and no CPU load in flight is disowned, counted in `state_gate_lost_boundary`, and logged as a gate bug rather than a cache miss. `can_allocate` picks the boundary and `_attach_state_slots` secures the state, and every way those two have disagreed so far has ended in the same silent wrong output. This turns any future disagreement into a recompute. Cold start is deliberately not one of the three ways to be secured -- it returns True as well, which is right with no boundary and wrong with one, and that is exactly the hole above. `PagedStateCheckpointCoordinator.restore_queued_for(dst_slot)` is the PAGE half of that evidence. Tests: with `num_cached_blocks == 0` and a boundary only the tier holds, the load must be requested for the *boundary* hash; a boundary with nothing behind it is disowned with `num_cached_tokens` back to zero; an ordinary cold start with no boundary is untouched, so the gate cannot fire on the common path. The first two fail against the previous code. --- atom/model_engine/block_manager.py | 58 +++++++++++++++- atom/model_engine/page_unit_checkpoint.py | 12 ++++ tests/test_state_checkpoint.py | 80 +++++++++++++++++++++++ 3 files changed, 149 insertions(+), 1 deletion(-) diff --git a/atom/model_engine/block_manager.py b/atom/model_engine/block_manager.py index f57a6e850f..9ee603f0bb 100644 --- a/atom/model_engine/block_manager.py +++ b/atom/model_engine/block_manager.py @@ -861,7 +861,22 @@ def allocate(self, seq: Sequence, num_cached_blocks: int = 0): # Pin the restore before fresh blocks can evict its checkpoint. state_holds = True if seq.has_per_req_cache and self.paged_state_checkpoints is not None: - state_holds = self._attach_state_slots(seq, hit_hash) + # The joint boundary, when there is one -- the same rule the fork + # branch below already used, and the PAGE branch did not. + # + # `hit_hash` is the hash at the HBM hit, and a joint boundary sits + # strictly above it by construction. Two ways that went wrong here: + # with `num_cached_blocks == 0` the loop never assigned `hit_hash` + # at all, so this took the `-1` cold-start exit and requested no + # state whatsoever; with a non-zero hit it restored the checkpoint + # covering `[0, hbm)`. Either way the KV leg then loaded to the + # joint boundary and `_claim_after_load` raised `num_cached_tokens` + # to it, so the forward resumed over a prefix the state does not + # cover. Silent wrong output, no exception. + joint_hash = seq.state_joint_boundary_hash + state_holds = self._attach_state_slots( + seq, joint_hash if joint_hash != -1 else hit_hash + ) for _ in range(len(seq.block_table), self.num_pool_blocks(len(seq))): seq.block_table.append(self._fresh_block()) seq.num_cached_tokens = num_cached_blocks * hbs @@ -883,6 +898,29 @@ def allocate(self, seq: Sequence, num_cached_blocks: int = 0): ) if seq.has_per_req_cache: seq._state_initialized_after_alloc = False + # A claimed joint boundary must have a state leg behind it. The two + # decisions are made in different places -- `can_allocate` picks the + # boundary, `_attach_state_slots` secures the state -- and every way + # they have disagreed so far ends the same way: the KV leg loads to the + # boundary, `_claim_after_load` claims it, and the forward runs over a + # prefix no state covers. This is the backstop that turns any such + # disagreement into a recompute instead, and it is deliberately a + # separate check from `state_holds` rather than folded into it: it + # holds even if the code above stops asking the right question. + if ( + state_holds + and seq.has_per_req_cache + and seq.state_joint_boundary_hash != -1 + and not self._state_leg_secured(seq) + ): + self.state_gate_lost_boundary += 1 + logger.warning( + "state offload: a joint boundary was admitted for request %s " + "with no state restore or load behind it; disowning it. This " + "is a bug in the joint gate, not a cache miss.", + seq.id, + ) + state_holds = False if not state_holds: # No state behind the boundary means it is not this request's # history. Disown it -- blocks stay claimed and the forward @@ -892,6 +930,24 @@ def allocate(self, seq: Sequence, num_cached_blocks: int = 0): seq.state_joint_boundary_tokens = 0 seq.state_joint_boundary_hash = -1 + def _state_leg_secured(self, seq: Sequence) -> bool: + """Whether something will really put state behind this seq's boundary. + + Exactly three ways that can be true, and they are the three exits + `_attach_state_slots` takes when it returns True with a boundary + claimed: a PAGE restore was queued, a fork source was adopted or read, + or a CPU load was requested. A cold start is not one of them -- it + returns True as well, which is correct with no boundary and wrong with + one. + """ + if getattr(seq, "state_load_hash", -1) != -1: + return True # a CPU load is in flight for it + if getattr(seq, "state_fork_src", -1) != -1: + return True # a fork checkpoint is its source + if self.paged_state_checkpoints is None: + return False + return self.paged_state_checkpoints.restore_queued_for(seq.state_slot) + def _attach_state_slots(self, seq: Sequence, hit_hash: int) -> bool: """Give `seq` its state slots, resuming from a checkpoint when one exists. diff --git a/atom/model_engine/page_unit_checkpoint.py b/atom/model_engine/page_unit_checkpoint.py index 317fb1fa29..d66dffb762 100644 --- a/atom/model_engine/page_unit_checkpoint.py +++ b/atom/model_engine/page_unit_checkpoint.py @@ -314,6 +314,10 @@ def begin_store(self, prefix_hash: int, src_slot: int) -> CheckpointStoreOp | No layout_id=self.spec.layout_id, ) + def restore_queued_for(self, dst_slot: int) -> bool: + """Whether a queued restore will write `dst_slot` on the next batch.""" + return any(op.dst_slot == dst_slot for _cid, op in self._queued_restores) + def begin_restore( self, prefix_hash: int, dst_slot: int ) -> CheckpointRestoreOp | None: @@ -735,6 +739,14 @@ def forget_pending(self, seq: Sequence) -> None: del self._pending[key] self.store.cancel_queued_restore(seq.state_slot) + def restore_queued_for(self, dst_slot: int) -> bool: + """Whether a queued restore will write `dst_slot` on the next batch. + + The evidence that a PAGE checkpoint really is behind a boundary the + request has claimed; see `BlockManager._state_leg_secured`. + """ + return self.store.restore_queued_for(dst_slot) + def begin_restore(self, h: int, dst_slot: int) -> bool: return self.store.begin_restore(h, dst_slot) is not None diff --git a/tests/test_state_checkpoint.py b/tests/test_state_checkpoint.py index fedd2d5901..ac14841c4d 100644 --- a/tests/test_state_checkpoint.py +++ b/tests/test_state_checkpoint.py @@ -3544,3 +3544,83 @@ def checkpoint_fates(self) -> dict: assert ( funnel.get("checkpoints_dropped", 0) >= 2 ), "checkpoints_dropped from SecondPoolStub missing from checkpoint_funnel()" + + +# ── a claimed joint boundary must have a state leg behind it ─────────────── + + +class TestPagedAllocateAimsTheStateLegAtTheJointBoundary: + """`can_allocate` picks the boundary and `_attach_state_slots` secures the + state, and the PAGE branch of `allocate` used to hand it `hit_hash` -- the + hash at the *HBM* hit -- while the fork branch already used the joint one. + + Both ways of getting that wrong end identically: the KV leg loads to the + boundary, `_claim_after_load` raises `num_cached_tokens` to it, and the + forward resumes over a prefix the recurrent state does not cover. + """ + + class _TierIndex: + def __init__(self, *hashes): + self.hashes = set(hashes) + self.pending_loads = {} + self.requested = [] + self.can_store = True + self.can_load = True + + def request_load(self, req_id, h): + if h not in self.hashes: + return False + self.pending_loads[req_id] = h + self.requested.append((req_id, h)) + return True + + def _bm(self, *tier_hashes): + bm = make_block_manager(paged_copy_config(), state_runtime=PAGED_COPY_RUNTIME) + index = self._TierIndex(*tier_hashes) + bm.state_offload = index + bm.paged_state_checkpoints.attach_offload(index) + return bm, index + + def _joint_seq(self, bm, tokens, *, boundary_hash_value, claim_tokens): + seq = stateful_seq(list(tokens)) + seq.state_joint_boundary_hash = boundary_hash_value + seq.state_joint_boundary_tokens = claim_tokens + seq.state_joint_claim_tokens = claim_tokens + return seq + + def test_the_state_leg_is_aimed_at_the_boundary_not_the_hbm_hit(self): + """`num_cached_blocks == 0`, so `hit_hash` is never assigned and the + old code took the cold-start exit: no restore, no load, and a boundary + still claimed.""" + bm, index = self._bm(4242) + seq = self._joint_seq(bm, range(48), boundary_hash_value=4242, claim_tokens=0) + bm.allocate(seq, 0) + + assert index.requested == [ + (seq.id, 4242) + ], "the state leg must be aimed at the joint boundary" + assert seq.state_joint_boundary_hash == 4242, "the boundary still stands" + + def test_a_boundary_with_no_state_behind_it_is_disowned(self): + """The backstop. Whatever the gate decided, a request may not resume + over a prefix nothing put state behind.""" + bm, _index = self._bm() # the tier has nothing + seq = self._joint_seq(bm, range(48), boundary_hash_value=4242, claim_tokens=0) + before = bm.state_gate_lost_boundary + bm.allocate(seq, 0) + + assert seq.state_joint_boundary_hash == -1 + assert seq.state_joint_boundary_tokens == 0 + assert seq.num_cached_tokens == 0 + assert bm.state_gate_lost_boundary > before + + def test_a_cold_start_without_a_boundary_is_untouched(self): + """The gate must not fire on the ordinary path: no boundary claimed, + so a cold slot is exactly right.""" + bm, _index = self._bm() + seq = stateful_seq(list(range(48))) + before = bm.state_gate_lost_boundary + bm.allocate(seq, 0) + + assert seq.state_slot >= 0 + assert bm.state_gate_lost_boundary == before From 8ad9249eea3b0a779325c6147de68af23fb6307a Mon Sep 17 00:00:00 2001 From: zejunchen-zejun Date: Sun, 30 Aug 2026 15:17:17 +0800 Subject: [PATCH 031/151] style: sort the import block and drop two unused unpacks Both are ruff findings on lines this branch touches (I001 in page_unit_checkpoint, RUF059 in the joint identity tests). CI filters ruff to diff context, so these are the two that would have been reported. --- atom/model_engine/page_unit_checkpoint.py | 1 - tests/test_lmcache_offload_connector.py | 4 ++-- 2 files changed, 2 insertions(+), 3 deletions(-) diff --git a/atom/model_engine/page_unit_checkpoint.py b/atom/model_engine/page_unit_checkpoint.py index d66dffb762..b555f1bb53 100644 --- a/atom/model_engine/page_unit_checkpoint.py +++ b/atom/model_engine/page_unit_checkpoint.py @@ -6,7 +6,6 @@ from __future__ import annotations import logging - from collections import OrderedDict, deque from collections.abc import Iterator, Mapping from dataclasses import dataclass diff --git a/tests/test_lmcache_offload_connector.py b/tests/test_lmcache_offload_connector.py index 58f88a0b10..f85b7e48d3 100644 --- a/tests/test_lmcache_offload_connector.py +++ b/tests/test_lmcache_offload_connector.py @@ -5182,9 +5182,9 @@ def test_a_single_leg_request_still_passes_straight_through(self): """Nothing armed it, so neither channel may be held back.""" worker = _k3_worker() kv_only = LoadOperationId("r9", 0) - done, failed = worker._settle_joint({kv_only}, set(), set(), set()) + done, _failed = worker._settle_joint({kv_only}, set(), set(), set()) assert done == {kv_only} - done, failed = worker._settle_joint(set(), set(), {"r8"}, set()) + done, _failed = worker._settle_joint(set(), set(), {"r8"}, set()) assert done == {"r8"} From 24dfb9a08bfa4ea05d9f82d3848a8fc40b9c6ba6 Mon Sep 17 00:00:00 2001 From: zejchen Date: Sun, 30 Aug 2026 11:12:16 +0000 Subject: [PATCH 032/151] fix(offload): forward the state-store leg through the multi connector Raised in review (Copilot, multi_connector.py): `MultiConnectorScheduler` forwarded `enqueue_state_loads` but not `enqueue_state_stores`, `take_state_reports`, or `take_state_source_releases`. The engine reaches all four off `self.kv_connector` by name: - scheduler.py:2597 enqueue = getattr(connector, "enqueue_state_stores") - scheduler.py:3526 take = getattr(connector, "take_state_reports") - scheduler.py:3538 release = getattr(connector, "take_state_source_releases") Under `kv_connector: multi` the shell exists but lacked the store forwarders, so `getattr` missed, the engine took its "did not carry" branch (scheduler.py:2609 -- whose warning names this exact missing forwarder), and every state store was refused: its pinned PAGE units were released before the D2H, and reports/source-releases were never drained, so pins never settled and stored hashes were never indexed. Net effect: the K3 CPU state tier could not fill or be found whenever an offload sub-connector was configured behind `multi`, even though the sub-connector itself was correct. Fix: add the three forwarders, symmetric to `enqueue_state_loads` -- first (and, per `_adopt_state_tier`, only) sub with each method owns it; `enqueue_state_stores` returns False, `take_state_reports` returns the empty 2-tuple the engine unpacks, and `take_state_source_releases` returns the empty set when no sub hosts a tier. `take_state_source_releases` stays its own forwarder rather than a third element of `take_state_reports`, matching the offload connector's note that the tuple's arity is a caller contract. Tests: test_state_stores_and_reports_forward_to_the_owning_sub and test_state_stores_default_when_no_sub_implements; FakeSchedSub gains the three methods (guarded by the same offload-api hide-list). --- .../disaggregation/multi/multi_connector.py | 42 ++++++++++++++++ tests/test_multi_connector.py | 48 +++++++++++++++++++ 2 files changed, 90 insertions(+) diff --git a/atom/kv_transfer/disaggregation/multi/multi_connector.py b/atom/kv_transfer/disaggregation/multi/multi_connector.py index ca8e869c74..e87e47b40f 100644 --- a/atom/kv_transfer/disaggregation/multi/multi_connector.py +++ b/atom/kv_transfer/disaggregation/multi/multi_connector.py @@ -450,6 +450,48 @@ def enqueue_state_loads(self, loads) -> bool: return False return bool(c.enqueue_state_loads(loads)) + def enqueue_state_stores(self, stores) -> bool: + """Symmetric to `enqueue_state_loads`: the one sub that hosts the tier + owns the stores; False if none can. + + Without this forwarder the engine's `getattr(connector, + "enqueue_state_stores")` misses on the shell, so it takes the "did not + carry" branch and releases each store's PAGE units *before* the D2H -- + the CPU tier can never fill under `kv_connector: multi`, even with an + offload sub-connector configured. `_adopt_state_tier` already guarantees + at most one tier, so "first" is "only" here too. + """ + c = _first_with(self._connectors, "enqueue_state_stores") + if c is None: + return False + return bool(c.enqueue_state_stores(stores)) + + def take_state_reports(self): + """`(indexed, failed)` from the tier sub, else two empty sets. + + The engine unpacks exactly this 2-tuple (`indexed, failed = take()` in + `scheduler._update_from_kv_xfer_finished`). Missing here, the engine + never drains store reports, so pins never settle and stored hashes are + never indexed -- the CPU tier fills but nothing can be found in it. + """ + c = _first_with(self._connectors, "take_state_reports") + if c is None: + return set(), set() + return c.take_state_reports() + + def take_state_source_releases(self) -> set: + """Stores whose PAGE units the GPU has finished reading, from the tier + sub; empty set if none. + + Its own forwarder rather than a third element of `take_state_reports`, + for the reason the offload connector records: that tuple's arity is a + contract with the caller, and widening it once already wedged the pool. + """ + c = _first_with(self._connectors, "take_state_source_releases") + if c is None: + return set() + return c.take_state_source_releases() + def should_park_partial_prefill_for_load(self, seq: Any) -> bool: c = _first_with(self._connectors, "should_park_partial_prefill_for_load") return c.should_park_partial_prefill_for_load(seq) if c is not None else False diff --git a/tests/test_multi_connector.py b/tests/test_multi_connector.py index 4daf141262..c0e73d09fe 100644 --- a/tests/test_multi_connector.py +++ b/tests/test_multi_connector.py @@ -62,6 +62,9 @@ def __init__( self.load_failed_ids = [] self.pending = False self.state_loads = [] + self.state_stores = [] + self.state_reports = (set(), set()) + self.state_source_releases = set() def get_num_new_matched_tokens(self, seq): return self._match @@ -111,6 +114,16 @@ def enqueue_state_loads(self, loads): self.state_loads.extend(loads) return True + def enqueue_state_stores(self, stores): + self.state_stores.extend(stores) + return True + + def take_state_reports(self): + return self.state_reports + + def take_state_source_releases(self): + return set(self.state_source_releases) + def __getattribute__(self, name): # Hide offload-specific methods unless this mock opts in, so # MultiConnector's hasattr() guards are exercised realistically. @@ -124,6 +137,9 @@ def __getattribute__(self, name): "process_completions", "has_pending_work", "enqueue_state_loads", + "enqueue_state_stores", + "take_state_reports", + "take_state_source_releases", } if name in offload_api and not object.__getattribute__(self, "_offload"): raise AttributeError(name) @@ -404,6 +420,38 @@ def test_no_sub_to_carry_a_state_load_is_reported_not_swallowed(): assert accepted is False +def test_state_stores_and_reports_forward_to_the_owning_sub(): + """The store leg is symmetric to the load leg, but only the load forwarder + existed. The engine calls `enqueue_state_stores`, `take_state_reports` and + `take_state_source_releases` on whatever connector it holds; under `multi` + that is the composite. Missing these, `enqueue_state_stores` misses on the + shell, the engine takes its "did not carry" branch and releases each store's + PAGE units before the D2H, and reports/source-releases are never drained -- + so the CPU tier cannot fill or be found under `kv_connector: multi`. + """ + moriio = FakeSchedSub(is_producer=True) # no offload methods + off = FakeSchedSub(is_offload=True, offload_methods=True) + off.state_reports = ({7}, {9}) + off.state_source_releases = {7, 9} + sched = _sched([moriio, off]) + + assert sched.enqueue_state_stores([(111, (1, 2, 3))]) is True + assert off.state_stores == [(111, (1, 2, 3))] + assert sched.take_state_reports() == ({7}, {9}) + assert sched.take_state_source_releases() == {7, 9} + assert not hasattr(moriio, "enqueue_state_stores") + + +def test_state_stores_default_when_no_sub_implements(): + """No tier under the composite: refuse the stores (not silently accept) and + drain nothing, matching the offload connector's own no-tier fallbacks. A + swallowed store would strand its pinned PAGE units.""" + sched = _sched([FakeSchedSub(is_producer=True), FakeSchedSub()]) + assert sched.enqueue_state_stores([(1, (0,))]) is False + assert sched.take_state_reports() == (set(), set()) + assert sched.take_state_source_releases() == set() + + def test_the_composite_exposes_the_sub_connectors_state_tier(): """The K3 store/load path reads `_state_tier` off whatever connector the forward context holds. Under `multi` that is this object, so From 0d704202888e63ed3bb3e4fe7b98f32d23136e02 Mon Sep 17 00:00:00 2001 From: zejchen Date: Sun, 30 Aug 2026 11:12:35 +0000 Subject: [PATCH 033/151] fix(offload): publish the state-store source release once, at the D2H drain Raised in review (Copilot, state_tier.py:213): `_do_store`'s `_source_released()` callback only set a dead `released` flag while the actual publish to `self._source_released` happened unconditionally at the end of the function. Two problems: 1. It did not realize the source/store split this file exists for. The codec invokes `on_source_released` the instant `pack`'s gather+D2H have drained (state_object.py:158, before `batched_put`) -- the moment the GPU has stopped reading the checkpoint's PAGE units. Publishing at the end instead held a whole image out of the KV pool across the CPU put, which cannot touch those units. 2. It was a double-drain trap: if the callback were wired to publish early (the obvious next step), the unconditional end-of-store add would emit the same release a second time. Across two engine drains that is a double `release_state_store_source(op)` -> double-unpin/underflow on a record the first drain already handed back. Fix: publish the release inside the callback, early, under `self._lock` (the engine drains the set from another thread), and make the end-of-store add a backstop guarded by `if not released`. Exactly one emission per store: - success: the callback publishes at the D2H drain; the backstop is a no-op. - refused allocation (obj is None, callback never reached): backstop releases -- the units were never read. - throwing `pack` (callback never reached): backstop releases -- the device was drained (`StagedTransfer._drain_device`) before the throw propagated. No path emits twice; no path withholds a release that is safe to make. Tests: test_source_release_is_emitted_once_early_not_again_at_store_end (drains the early release while the store is still stalled in the CPU put, then asserts the backstop stays silent), plus refused/throwing paths still release once and report the store failed. --- .../offload/hybrid/kimi_k3/state_tier.py | 34 ++++++-- tests/test_lmcache_offload_connector.py | 85 +++++++++++++++++++ 2 files changed, 111 insertions(+), 8 deletions(-) diff --git a/atom/kv_transfer/offload/hybrid/kimi_k3/state_tier.py b/atom/kv_transfer/offload/hybrid/kimi_k3/state_tier.py index 3d8e331d32..62ddaa12e7 100644 --- a/atom/kv_transfer/offload/hybrid/kimi_k3/state_tier.py +++ b/atom/kv_transfer/offload/hybrid/kimi_k3/state_tier.py @@ -177,8 +177,21 @@ def _do_store(self, op, unit_ids) -> None: released = False def _source_released() -> None: + # Fires from `codec.put` the moment `pack`'s gather+D2H have + # drained (state_object.put), before `batched_put`. Publish the + # release *here*, early: the PAGE units are the KV pool's and the + # GPU has stopped reading them, so hand them back now instead of + # holding a whole image out of the pool across the CPU put. Under + # `self._lock` because `take_source_releases` drains the set from + # the engine thread; the flag makes the end-of-store backstop a + # no-op so the release is emitted exactly once (a second emission + # would double-unpin on the engine side). nonlocal released - released = True + with self._lock: + if released: + return + released = True + self._source_released.add(op) try: with self._staging_budget: @@ -201,14 +214,19 @@ def _source_released() -> None: exc_info=True, ) with self._lock: - # Always released, on every exit. A refused allocation never read - # the units at all, and a throwing `pack` has drained the device - # before it propagated (`StagedTransfer._drain_device`), so by the - # time control is here the GPU has stopped reading them either way. - # Withholding this on the failure paths would hold an image out of - # the KV pool until the stale reclaimer noticed. + # Backstop, not the primary path: on success the D2H callback above + # already published the release early, so this must NOT add it again + # -- a second emission across two engine drains is the double-unpin. + # It still fires on the paths the callback never reached: a refused + # allocation never read the units, and a throwing `pack` drained the + # device before it propagated (`StagedTransfer._drain_device`), so + # the GPU has stopped reading them there too. Withholding it on those + # paths would hold an image out of the KV pool until the stale + # reclaimer noticed. self._store_submitted_at.pop(op, None) - self._source_released.add(op) + if not released: + released = True + self._source_released.add(op) # Report, never apply: `StateOffloadIndex` lives in the engine # process and this runs in a spawned runner. The engine applies # these via KVConnectorOutput. diff --git a/tests/test_lmcache_offload_connector.py b/tests/test_lmcache_offload_connector.py index f85b7e48d3..b4a3cf67c7 100644 --- a/tests/test_lmcache_offload_connector.py +++ b/tests/test_lmcache_offload_connector.py @@ -5331,3 +5331,88 @@ def test_load_queue_wait_is_measured(self): assert "state_load_queue_wait_ms_max" in stats finally: tier.shutdown() + + class _EarlyReleaseCodec: + """`put` releases the source (D2H drained) and then stalls in the CPU + `batched_put`, so a drain can observe the early release *before* the + store completes -- exactly the window the source/store split exists + for.""" + + def __init__(self): + self.released = threading.Event() + self.gate = threading.Event() + + def put(self, h, unit_ids, on_source_released=None): + if on_source_released is not None: + on_source_released() # D2H is done here, CPU put is not + self.released.set() + self.gate.wait(timeout=5) + return True + + def get(self, h, slot): + return True + + def test_source_release_is_emitted_once_early_not_again_at_store_end(self): + """The release must land the instant the D2H drains (before the CPU + put), and must not be re-emitted when `_do_store` returns. A second + emission across two engine drains double-unpins the same record.""" + codec = self._EarlyReleaseCodec() + tier = self._tier(codec) + op = StateStoreOperationId(1, 1) + try: + tier.submit_store(op, (0,)) + assert codec.released.wait(timeout=5), "the D2H callback never fired" + # Early: units handed back while the CPU put is still running. + assert tier.take_source_releases() == {op} + # Finish the store; the end-of-store backstop must stay silent. + codec.gate.set() + tier.drain() + assert tier.take_source_releases() == set(), "source release re-emitted" + assert tier.take_store_reports() == ({op}, set()) + finally: + codec.gate.set() + tier.shutdown() + + def test_a_refused_store_still_releases_its_units(self): + """A refused allocation never fires the D2H callback and never read the + units, so the backstop must release them -- withholding it would hold an + image out of the KV pool until the stale reclaimer noticed.""" + + class _RefuseCodec: + def put(self, h, unit_ids, on_source_released=None): + return False # allocation refused; callback never reached + + def get(self, h, slot): + return True + + tier = self._tier(_RefuseCodec()) + op = StateStoreOperationId(2, 1) + try: + tier.submit_store(op, (0,)) + tier.drain() + assert tier.take_source_releases() == {op} + assert tier.take_store_reports() == (set(), {op}) + finally: + tier.shutdown() + + def test_a_throwing_store_still_releases_its_units(self): + """A throwing `pack` drains the device before it propagates, so the GPU + has stopped reading the units by the time control returns -- the backstop + releases them and the store is reported failed, not lost.""" + + class _ThrowCodec: + def put(self, h, unit_ids, on_source_released=None): + raise RuntimeError("pack blew up") + + def get(self, h, slot): + return True + + tier = self._tier(_ThrowCodec()) + op = StateStoreOperationId(3, 1) + try: + tier.submit_store(op, (0,)) + tier.drain() + assert tier.take_source_releases() == {op} + assert tier.take_store_reports() == (set(), {op}) + finally: + tier.shutdown() From 951dab4c39381f0cb385e5aec25f088b6131c46c Mon Sep 17 00:00:00 2001 From: zejchen Date: Mon, 31 Aug 2026 07:07:01 +0000 Subject: [PATCH 034/151] fix(k3-offload): repair state-tier lifecycle & liveness in the worker connector MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Four defects in KimiK3OffloadConnector/Scheduler, all around the state tier's report and liveness plumbing across the worker/engine process boundary. F1 — cross-process AttributeError wedged whole steps. On the no-tier store path the worker did `self._state_store_failed_locally.add(op)`, but that set is only initialised on the SCHEDULER (a different process). `_start_state_stores` runs before `super().start_load_kv(metadata)`, so the AttributeError aborted the step and took its KV loads/saves down with it, wedging the TP ranks on the next collective. Route no-tier store failures through the same completion channels a real tier failure uses (`STATE_INDEX_CHANNEL` index-failed + `STATE_SOURCE_CHANNEL` source-release), drained in `get_finished` BEFORE the tier-None early return. Removed the dead scheduler-side set and simplified `take_state_reports` to its single source of truth. F4 — leaked joint-park entry per load when no tier built. `_arm_joint_loads` armed a park with needs_kv+needs_state even when `_state_tier is None`, but `get_finished` returns at the `_state_tier is None` guard BEFORE `_settle_joint` drains the park. The park then owed the KV leg forever; the KV completion passed through un-held and the `_need`/`_alias`/`_alias_of` entry leaked, one per joint load per tier-None step. Guard the arm: with no tier there is nothing to coordinate and a KV-only load through the base path is correct. F9 — engine could idle out before dispatching state work. `has_pending_work` inherited from DenseOffloadScheduler ORs only the KV trackers, so a step whose only outstanding work was a queued state load or last-of-burst checkpoint read as idle. Override to also OR `_pending_state_loads`/`_pending_state_stores`; both drain into metadata every `build_connector_meta`, so the predicate stays monotone and never latches a busy loop. F12 — a backend with state_entry_views but no page_unit_views built a tier that passed every load then AttributeError'd on the first store, masked by the tier's blind `except Exception` as an endlessly "failed" store. Probe `page_unit_views` callability at tier build so the mismatch fails fast and visibly instead. Co-Authored-By: Claude Opus 4.8 --- .../offload/hybrid/kimi_k3/connector.py | 91 +++++++++++++++++-- 1 file changed, 83 insertions(+), 8 deletions(-) diff --git a/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py b/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py index 809f2b584e..df51f260e1 100644 --- a/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py +++ b/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py @@ -59,6 +59,16 @@ def __init__(self, config) -> None: # Inert until a request has both legs, which only a joint boundary # produces; costs one dict lookup per report otherwise. self._joint_park = _JointPark() + # Stores this worker could not even attempt because the tier never + # built. Drained in `get_finished` into the SAME completion channels a + # real tier failure uses (index-failed + source-release), so the engine + # unpins their PAGE units now. This is a *worker-process* field; the + # scheduler's report path lives in a different process and cannot see a + # set written here -- the earlier code wrote a scheduler-only attribute + # from the worker, which does not exist here and raised AttributeError + # on the no-tier store path, taking the whole step's KV loads/saves down + # with it (super().start_load_kv never ran). + self._store_failed_no_tier: set[int] = set() def register_kv_caches( self, kv_caches: dict, transfer_tensors=None, num_blocks: int | None = None @@ -114,6 +124,22 @@ def _build_state_tier(self, transfer_tensors) -> None: ) return + # The load direction reads `state_entry_views` (validated just above); + # the STORE direction reads `page_unit_views`, a *different* backend + # method (StateByteCodec.put -> backend.page_unit_views). A backend that + # implements one but not the other would build the tier and pass every + # load, then AttributeError on the first store -- and the tier's blind + # `except Exception` masks it as an endlessly "failed" store. Probe the + # store method at construction so the mismatch fails fast and visibly. + if not callable(getattr(backend, "page_unit_views", None)): + logger.warning( + "kimi_k3 offload: %s has state_entry_views but no callable " + "page_unit_views; the store path needs it, so a tier would fail " + "every store silently. State tier off.", + type(backend).__name__, + ) + return + # The store reads the checkpoint's PAGE units and the load writes an # Active Slot, so the blob has to be the same length both ways. They are # equal for K3 (a checkpoint covers the whole slot), and a model where @@ -207,6 +233,17 @@ def _arm_joint_loads(self, metadata) -> None: state_ids = {req_id for req_id, _h, _slot in loads} if not state_ids or not self._do_load: return + if self._state_tier is None: + # No tier means the state leg is unsettleable: `_start_state_loads` + # fails these loads for recompute, and `get_finished` returns at the + # `self._state_tier is None` guard BEFORE it drains the park via + # `_settle_joint`. Arming here would leave the park owing the KV leg + # forever -- the KV completion passes through un-held and the + # `_need`/`_alias`/`_alias_of` entry leaks, one per joint load per + # tier-None step. With no tier there is nothing to coordinate, so + # letting the KV load flow through the base path un-parked is exactly + # the correct behaviour for a KV-only load. + return for req in metadata.requests: if req.load_spec is not None and req.req_id in state_ids: self._joint_park.arm( @@ -259,7 +296,7 @@ def _start_state_stores(self, metadata) -> None: len(stores), ) for op, _units in stores: - self._state_store_failed_locally.add(op) + self._store_failed_no_tier.add(op) return for op, unit_ids in stores: self._state_tier.submit_store(op, tuple(int(u) for u in unit_ids)) @@ -271,6 +308,22 @@ def _fail_state_loads(self, loads) -> None: # -- completions ------------------------------------------------------- def get_finished(self): out = super().get_finished() + # No-tier store failures must reach the engine even when the tier never + # built -- the engine pinned their PAGE units and is holding them + # against a report. Emit the tier's own failure pairing: index-failed so + # the aggregator takes quorum instead of waiting for a second report, + # plus a source-release so the units are freed now rather than on the + # reconciler's full timeout. Drained BEFORE the tier-None early return + # below, because that is exactly the case that populated this set. + if self._store_failed_no_tier: + for op in self._store_failed_no_tier: + out.connector_completions.add( + ConnectorCompletion(STATE_INDEX_CHANNEL, op, False) + ) + out.connector_completions.add( + ConnectorCompletion(STATE_SOURCE_CHANNEL, op, True) + ) + self._store_failed_no_tier = set() if self._state_tier is None: return out indexed, index_failed = self._state_tier.take_store_reports() @@ -350,13 +403,14 @@ def __init__(self, config) -> None: self._save_inflight_since: dict[str, float] = {} self._save_stalled = False self._warned_save_stalled = False - # Channel reports drained by the engine each step. + # Channel reports drained by the engine each step. No-tier store + # failures arrive here too, via the worker's STATE_INDEX_CHANNEL / + # STATE_SOURCE_CHANNEL completions -- the worker cannot write a + # scheduler field directly (different process), so there is no + # engine-side "failed locally" set to merge. self._state_indexed: set = set() self._state_index_failed: set = set() self._state_source_released: set = set() - # Stores this worker could not even attempt (no tier). Reported as - # failures so the engine releases their pinned units immediately. - self._state_store_failed_locally: set[int] = set() # -- state load queue -------------------------------------------------- def enqueue_state_loads(self, loads) -> bool: @@ -393,6 +447,23 @@ def _may_emit_save(self) -> bool: and len(self._save_inflight) < self._max_pending_saves ) + def has_pending_work(self) -> bool: + """Base KV liveness, plus this variant's state queues. + + `DenseOffloadScheduler.has_pending_work` ORs only the KV load/save + trackers, so a step whose only outstanding work is a queued state load + or a last-of-burst state checkpoint reads as idle -- and the engine can + stop stepping before the tier is ever handed that work. Both queues are + drained into metadata every `build_connector_meta`, so OR-ing them keeps + the predicate monotone: it goes False the step after the work is + dispatched and never latches the busy loop. + """ + return ( + super().has_pending_work() + or bool(self._pending_state_loads) + or bool(self._pending_state_stores) + ) + # -- save stall -------------------------------------------------------- def _refresh_save_stall(self) -> None: """Decide whether the save path has stopped draining. @@ -542,10 +613,14 @@ class and the delegating shell's fallback, and widening it once return released def take_state_reports(self) -> tuple[set[int], set[int]]: - """Drain this step's tier store reports for the engine-side index.""" + """Drain this step's tier store reports for the engine-side index. + + Both real tier failures and no-tier worker failures land in + `_state_index_failed` via STATE_INDEX_CHANNEL, so there is a single + source of truth to drain. + """ indexed = self._state_indexed - failed = self._state_index_failed | self._state_store_failed_locally + failed = self._state_index_failed self._state_indexed = set() self._state_index_failed = set() - self._state_store_failed_locally = set() return indexed, failed From 18a9869adcd0df4f10e3516238f1283238ed51f9 Mon Sep 17 00:00:00 2001 From: zejchen Date: Mon, 31 Aug 2026 07:07:51 +0000 Subject: [PATCH 035/151] fix(k3-offload): drain the device on StagedTransfer pack/unpack failure paths MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit F2 — a kernel-launch raise on the direct-to-device fast path (and the general path's except) returned while a gather/scatter was still queued on pack_stream. For unpack that kernel scatters into the caller's destination state segments; returning let the owner reuse those slots under a pending write, corrupting the next request that got the slot -- silent state corruption, no fault to point at. For pack the gather reads the source units the failed store is about to release back to the KV pool, same hazard in reverse. Wrap both fast paths in try/except and add `self._drain_device()` (a `torch.cuda.synchronize`) to every failure path, including the general unpack except that previously only invalidated the staging buffer. Draining before propagating is what makes releasing the units/slots safe: no kernel that reads or writes them is still in flight once the failure reaches the caller. Co-Authored-By: Claude Opus 4.8 --- .../offload/hybrid/kimi_k3/staging.py | 45 +++++++++++++++---- 1 file changed, 37 insertions(+), 8 deletions(-) diff --git a/atom/kv_transfer/offload/hybrid/kimi_k3/staging.py b/atom/kv_transfer/offload/hybrid/kimi_k3/staging.py index d90695ca61..84f8ed6525 100644 --- a/atom/kv_transfer/offload/hybrid/kimi_k3/staging.py +++ b/atom/kv_transfer/offload/hybrid/kimi_k3/staging.py @@ -184,10 +184,21 @@ def pack(self, segments: list[torch.Tensor], dst: Any) -> None: state = self.thread_state() staging_buffer = state.staging_buffer if dst_tensor.is_cuda and dst_tensor.device == self.device: - with state.stream_ctx(state.pack_stream): - fused_pack_chunk_major(segments, seg_bytes, [1], [0], dst_tensor) - if state.pack_stream is not None: - state.pack_stream.synchronize() + # Direct-to-device fast path: no staging hop, but the gather + # still reads the source units on `pack_stream`. If the kernel + # launch raises we must drain before propagating -- otherwise + # the caller frees those units while a queued gather is still + # about to read them (same hazard as the general path's except). + try: + with state.stream_ctx(state.pack_stream): + fused_pack_chunk_major( + segments, seg_bytes, [1], [0], dst_tensor + ) + if state.pack_stream is not None: + state.pack_stream.synchronize() + except Exception: + self._drain_device() + raise return device_buf = self.ensure_buffer(staging_buffer, nbytes) try: @@ -238,10 +249,22 @@ def unpack(self, src: Any, segments: list[torch.Tensor]) -> None: state = self.thread_state() staging_buffer = state.staging_buffer if src_tensor.is_cuda and src_tensor.device == self.device: - with state.stream_ctx(state.pack_stream): - fused_unpack_chunk_major(src_tensor, segments, seg_bytes, [1], [0]) - if state.pack_stream is not None: - state.pack_stream.synchronize() + # Direct-from-device fast path: the unpack kernel writes the + # destination `segments` (the request's state slots) on + # `pack_stream`. If the launch raises we must drain before + # propagating -- otherwise the caller frees those slots while a + # queued kernel is still about to write them, corrupting the + # next request that gets the slot (mirror of the general path). + try: + with state.stream_ctx(state.pack_stream): + fused_unpack_chunk_major( + src_tensor, segments, seg_bytes, [1], [0] + ) + if state.pack_stream is not None: + state.pack_stream.synchronize() + except Exception: + self._drain_device() + raise return device_buf = self.ensure_buffer(staging_buffer, nbytes) try: @@ -256,6 +279,12 @@ def unpack(self, src: Any, segments: list[torch.Tensor]) -> None: self._finish(state, state.pack_stream) except Exception: staging_buffer.free_event_valid = False + # Mirror of pack()'s except: the unpack kernel scatters into the + # caller's destination segments on pack_stream. Returning while + # that kernel is still queued would let the owner reuse the + # slots under a pending write -- silent state corruption. Drain + # every queued kernel before propagating the failure. + self._drain_device() raise finally: self.release_buffer_if_requested(staging_buffer) From ccddb8fe91aaeaef4ee4273898283451db626d49 Mon Sep 17 00:00:00 2001 From: zejchen Date: Mon, 31 Aug 2026 07:08:00 +0000 Subject: [PATCH 036/151] fix(offload): forward _state_tier and chunk_size through the multi composite MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Under `kv_connector: multi` the scheduler/worker probes the composite, not the kimi_k3 impl behind the `lmcache_offload` shell's `_impl`. Two members the composite needs were invisible, silently disabling the K3 state tier. F8 — `MultiConnector._adopt_state_tier` reads `_state_tier` off each sub to find the one offload backend that built a tier (and to refuse two with a ValueError). The worker shell `LMCacheOffloadConnector` did not expose it, so the probe read None on every member: the composite never adopted the tier and state spills went nowhere, AND the duplicate-tier guard could never fire. Add a `_state_tier` property forwarding to `_impl`. F6 — `Scheduler._resolve_waiting_remote_kv` stamps `seq.offload_kv_chunk_tokens` from `getattr(self.kv_connector, "chunk_size", 0)`. Under multi the getattr default (0) won, so `_joint_kv_boundary` saw a zero chunk grid and refused every joint KV load with `no_chunk_size` -- the tier's KV leg was silently off. Add a `chunk_size` property on `MultiConnectorScheduler` delegating to the one offload sub (at most one hosts the tier, so first is only). Co-Authored-By: Claude Opus 4.8 --- .../disaggregation/multi/multi_connector.py | 16 ++++++++++++++++ atom/kv_transfer/offload/connector.py | 15 +++++++++++++++ 2 files changed, 31 insertions(+) diff --git a/atom/kv_transfer/disaggregation/multi/multi_connector.py b/atom/kv_transfer/disaggregation/multi/multi_connector.py index e87e47b40f..541d1c1592 100644 --- a/atom/kv_transfer/disaggregation/multi/multi_connector.py +++ b/atom/kv_transfer/disaggregation/multi/multi_connector.py @@ -425,6 +425,22 @@ def request_finished(self, seq: Any) -> None: # The scheduler guards every one of these with hasattr(), so MultiConnector # only needs to expose them when a sub-connector implements them. + @property + def chunk_size(self) -> int | None: + """The offload sub's KV transfer grid, in tokens. + + `Scheduler._resolve_waiting_remote_kv` stamps + `seq.offload_kv_chunk_tokens` from `getattr(self.kv_connector, + "chunk_size", 0)`, and under `multi` `self.kv_connector` IS this + composite. Without this forwarder the getattr default (0) wins, so + `_joint_kv_boundary` sees a zero chunk grid and refuses every joint KV + load with `no_chunk_size` -- the K3 state tier's KV leg is silently off + under `kv_connector: multi`. Delegate to the one offload sub (at most + one hosts the tier, so "first" is "only"). + """ + c = _first_with(self._connectors, "chunk_size") + return getattr(c, "chunk_size", None) if c is not None else None + def should_park_for_load_after_alloc(self, seq: Any) -> bool: c = _first_with(self._connectors, "should_park_for_load_after_alloc") return c.should_park_for_load_after_alloc(seq) if c is not None else False diff --git a/atom/kv_transfer/offload/connector.py b/atom/kv_transfer/offload/connector.py index 678d367e80..dc6d138220 100644 --- a/atom/kv_transfer/offload/connector.py +++ b/atom/kv_transfer/offload/connector.py @@ -85,6 +85,21 @@ class LMCacheOffloadConnector(KVConnectorBase): def __init__(self, config) -> None: self._impl = _build_worker(config) + @property + def _state_tier(self): + """Expose the implementation's KDA state tier through the shell. + + `MultiConnector._adopt_state_tier` probes `_state_tier` on each + sub-connector to find the one offload backend that built a state tier. + Under the `multi` shell the sub IS this object, not the `kimi_k3` impl + behind `_impl`, so without this forwarder the probe reads None on every + member and the composite never adopts the tier that was actually built + -- state spills silently go nowhere. Mirrors the scheduler shell's + `chunk_size` forwarder and the "every probed member appears on the + shell" contract on `LMCacheOffloadConnectorScheduler`. + """ + return getattr(self._impl, "_state_tier", None) + def register_kv_caches( self, kv_caches, transfer_tensors=None, num_blocks=None ) -> None: From 73eccab12bf78235c43cf7acd84e6d0751b6326f Mon Sep 17 00:00:00 2001 From: zejchen Date: Mon, 31 Aug 2026 07:08:23 +0000 Subject: [PATCH 037/151] fix(offload): read joint-chunk under multi, and reclaim stranded load slots MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two BlockManager offload-index defects, both able to wedge the state pool. F7 — `_joint_chunk_tokens` was built from the raw `kv_transfer_config`. Under `multi` the lmcache.* keys (chunk_size, offload_layout) live on the offload SUB-connector, so this read a zero chunk grid and `_joint_kv_boundary` refused every joint KV load with `no_chunk_size`. Unwrap the sub via `_offload_subconfig` the same way `state_tier_capability` does, keeping the gate's chunk size and the capability check's tier consistent. F10 — permanent slot leak when a load report never comes. `deallocate` parks a state slot in `_orphan_load_slots` when it tears down a request whose load is still in flight, on the promise that `settle_state_load` "always comes". A crashed worker or dropped completion breaks that promise; the slot then sits off the free list forever and `can_allocate`'s state gate refuses new per-request work once enough are stranded -- a wedge with no fault to point at. Add `reconcile_orphan_load_slots`, the load-side twin of `reclaim_stale_state_store_pins`: stamp each park's monotonic time, and after the same abandon window release slots whose report never landed. Wired at the same scheduler call site as the store-pin reclaimer. Carries the same "cannot tell a lost report from a slow worker" caveat, so the window is the abandon timeout, not a tight one. The index leg was already abandoned in `deallocate`, so the reclaim touches only the slot; a late report finds it popped and releases nothing. Co-Authored-By: Claude Opus 4.8 --- atom/model_engine/block_manager.py | 74 ++++++++++++++++++++++++++++-- atom/model_engine/scheduler.py | 7 +++ 2 files changed, 77 insertions(+), 4 deletions(-) diff --git a/atom/model_engine/block_manager.py b/atom/model_engine/block_manager.py index 9ee603f0bb..9f19fe5971 100644 --- a/atom/model_engine/block_manager.py +++ b/atom/model_engine/block_manager.py @@ -4,6 +4,7 @@ import array import logging from math import inf, isinf +from time import monotonic import numpy as np import xxhash @@ -263,11 +264,21 @@ def __init__( self._joint_chunk_tokens = 0 try: from atom.kv_transfer.offload import config as offcfg - + from atom.model_engine.state_offload import _offload_subconfig + + kvcfg = getattr(config, "kv_transfer_config", None) or {} + # Under `multi` the lmcache.* keys (chunk_size, offload_layout) live + # on the offload SUB-connector, not the composite. Passing the raw + # composite here read a zero chunk grid, and `_joint_kv_boundary` + # then refused every joint KV load with `no_chunk_size`. Unwrap the + # sub the same way `state_tier_capability` does, so the gate's chunk + # size and the tier the capability check builds stay consistent. + if isinstance(kvcfg, dict) and kvcfg.get("kv_connector") == "multi": + sub, _why = _offload_subconfig(kvcfg) + if sub is not None: + kvcfg = sub self._joint_chunk_tokens = int( - offcfg.build_lmcache_config( - getattr(config, "kv_transfer_config", None) - ).chunk_size + offcfg.build_lmcache_config(kvcfg).chunk_size ) except Exception: # Blind on purpose: this runs at model load, the import reaches a @@ -288,6 +299,16 @@ def __init__( # bytes landed. The slot is off the free list until the report comes # back; see `deallocate`. self._orphan_load_slots: dict = {} + # req_id -> monotonic stamp, parallel to `_orphan_load_slots`. The hold + # above trusts `settle_state_load` to "always come": it does not, if the + # worker that owed the report crashed or its completion was dropped. Then + # the slot sits off the free list forever and the state gate in + # `can_allocate` wedges the pool once every slot is stranded. This stamps + # the wait so `reconcile_orphan_load_slots` can reclaim it after the same + # abandon window the store pins use -- the load-side twin of + # `reclaim_stale_state_store_pins`. + self._orphan_load_slots_at: dict = {} + self._orphan_load_slots_reclaimed: int = 0 # `kv_offload_enabled` is the whole switch: without a connector to # carry a transfer there is nothing beneath the pool, and a hash whose # KV left HBM could not be resumed from anyway. @@ -1115,6 +1136,7 @@ def abandon_state_load(self, req_id) -> None: return self.state_offload.abandon_load(req_id) slot = self._orphan_load_slots.pop(req_id, None) + self._orphan_load_slots_at.pop(req_id, None) if slot is not None: self.state.release(slot) @@ -1134,6 +1156,7 @@ def settle_state_load(self, req_id, ok: bool) -> None: else: self.state_offload.fail_load(req_id) slot = self._orphan_load_slots.pop(req_id, None) + self._orphan_load_slots_at.pop(req_id, None) if slot is not None: self.state.release(slot) @@ -1217,6 +1240,48 @@ def reclaim_stale_state_store_pins(self, timeout_s: float) -> int: return 0 return self.paged_state_checkpoints.reclaim_stale_offload_pins(timeout_s) + def reconcile_orphan_load_slots(self, timeout_s: float) -> int: + """Free load slots whose `settle_state_load` never came, after `timeout_s`. + + The load-side twin of `reclaim_stale_state_store_pins`. `deallocate` + parks a slot in `_orphan_load_slots` when it tears down a request whose + state load is still in flight, on the promise that the worker reports + every load and `settle_state_load` will hand it back. A crashed worker + or a dropped completion breaks that promise, and the slot then sits off + the free list forever -- `can_allocate`'s state gate refuses new + per-request work once enough slots are stranded, wedging the pool with + no fault to point at. + + **This is a last resort and cannot tell a lost report from a slow + worker still writing the slot** -- the same limitation the store-pin + twin documents. Reclaiming under a live H2D hands the next request a + buffer someone else is filling, with `has_initial_state` already true + over it. So the window must be the abandon timeout, not a tight one: + long enough that a report still in flight has already arrived. The + index leg was abandoned back in `deallocate`, so nothing here touches + it; a late report finds the slot already popped and releases nothing. + """ + if timeout_s <= 0 or not self._orphan_load_slots_at: + return 0 + cutoff = monotonic() - timeout_s + stale = [ + req_id + for req_id, at in self._orphan_load_slots_at.items() + if at <= cutoff + ] + for req_id in stale: + self._orphan_load_slots_at.pop(req_id, None) + slot = self._orphan_load_slots.pop(req_id, None) + if slot is not None: + self.state.release(slot) + # The index leg was already abandoned in `deallocate` when the slot + # was parked; only the slot itself outlived the report. Do not touch + # the index again here -- a second `abandon_load` would double-count + # the miss. A late report that still calls in finds the slot popped + # and releases nothing. + self._orphan_load_slots_reclaimed += 1 + return len(stale) + def take_state_loads(self) -> list[tuple]: """`(req_id, hash, target_slot)` for loads admitted since the last call. Drained once per pass by the scheduler, which hands them to the @@ -2196,6 +2261,7 @@ def deallocate(self, seq: Sequence): # scratch is nobody's destination and goes back regardless. if seq.state_load_hash != -1 and self.state_offload is not None: self._orphan_load_slots[seq.id] = seq.state_slot + self._orphan_load_slots_at[seq.id] = monotonic() # Abandoned, not failed: an abort says nothing about the bytes, # and forgetting the hash would cost the next request over this # prefix a full recompute. diff --git a/atom/model_engine/scheduler.py b/atom/model_engine/scheduler.py index 6b8f9a52f3..fcea50ab31 100644 --- a/atom/model_engine/scheduler.py +++ b/atom/model_engine/scheduler.py @@ -3551,6 +3551,13 @@ def _update_from_kv_xfer_finished(self, kv_connector_output: KVConnectorOutput): # `BlockPool` invariant -- so zeroing the count restores the record # exactly. bm.reclaim_stale_state_store_pins(_offload_save_abandon_timeout_s()) + # Load-side twin of the same defence: `deallocate` parks a slot when + # it tears down a request whose state load is still in flight, on the + # promise that `settle_state_load` will hand it back. A crashed + # worker or dropped completion breaks that promise and the slot sits + # off the free list forever, wedging `can_allocate`'s state gate. + # Same window, same "cannot tell lost from slow" caveat as the pins. + bm.reconcile_orphan_load_slots(_offload_save_abandon_timeout_s()) def get_request_counts(self) -> tuple[int, int]: """Returns (num_running_reqs, num_waiting_reqs).""" From 7252da663bfc8be440afb55459ada0d73b9e005b Mon Sep 17 00:00:00 2001 From: zejchen Date: Mon, 31 Aug 2026 07:09:06 +0000 Subject: [PATCH 038/151] fix(offload): defer, don't drop, store nominations whose hash is in flight MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit F11 — `take_offload_stores` collapsed two distinct cases onto one `continue`: a record that is no longer READY (terminal -- nothing left to store) and a record that is still READY but whose prefix_hash has an earlier generation on the wire (an eviction + re-store of one prefix while the first store is mid-flight). The second is a live, valid nomination; dropping it lost that checkpoint image from the store queue for good once the in-flight copy settled. Hold the in-flight-hash nominations aside and re-queue them onto `_offload_ready` after the pass, so they land on a later drain once the colliding generation clears. Held aside rather than re-appended inside the loop so a hash that stays in flight cannot spin the call. Co-Authored-By: Claude Opus 4.8 --- atom/model_engine/page_unit_checkpoint.py | 21 +++++++++++++++++++-- 1 file changed, 19 insertions(+), 2 deletions(-) diff --git a/atom/model_engine/page_unit_checkpoint.py b/atom/model_engine/page_unit_checkpoint.py index b555f1bb53..d044a210e4 100644 --- a/atom/model_engine/page_unit_checkpoint.py +++ b/atom/model_engine/page_unit_checkpoint.py @@ -420,14 +420,27 @@ def take_offload_stores( closely enough that the earlier one's report is still on the wire. """ out: list[tuple[StateStoreOperationId, tuple[int, ...]]] = [] + # Nominations whose hash is transiently in flight: held aside and + # re-queued after the pass, NOT dropped (see below). + deferred: list = [] while self._offload_ready and len(self._offload_pins) < max_inflight: checkpoint_id = self._offload_ready.popleft() record = self.records.get(checkpoint_id) - # Spent while it waited -- which nomination deliberately allows -- - # or already in flight under this hash. Nothing to send either way. + # Spent while it waited -- which nomination deliberately allows. + # Terminal: there is nothing left to store, so drop it. if record is None or record.state != READY: continue if self._hash_in_flight(record.prefix_hash): + # NOT terminal. The record is still READY and is a live, valid + # nomination; its only problem is that an earlier generation of + # the same hash is still on the wire (an eviction + re-store of + # one prefix while the first store is mid-flight). Dropping it + # here -- as a bare `continue` did -- lost this checkpoint image + # from the store queue for good once the in-flight copy settled. + # Defer it to a later pass instead. Held aside rather than + # re-appended inside the loop so a hash that stays in flight + # cannot spin this call. + deferred.append(checkpoint_id) continue self._offload_generation += 1 op = StateStoreOperationId( @@ -436,6 +449,10 @@ def take_offload_stores( record.pin_count += 1 self._offload_pins[op] = (checkpoint_id, monotonic()) out.append((op, record.unit_ids)) + # Re-queue the deferred nominations for the next drain, once the + # colliding in-flight generation has had a chance to settle. + if deferred: + self._offload_ready.extend(deferred) return out def _hash_in_flight(self, prefix_hash: int) -> bool: From c531ddb45f79059eaf7e68f0d3122b1992054c1d Mon Sep 17 00:00:00 2001 From: zejchen Date: Mon, 31 Aug 2026 07:20:18 +0000 Subject: [PATCH 039/151] fix(offload): surface orphan_load_slots_reclaimed on the checkpoint funnel The load-side reclaim counter added alongside reconcile_orphan_load_slots (reclaim_stale_state_store_pins' load-side twin) was written at block_manager.py:1282 but read nowhere -- a dead observability hook. Its store-side twin, PagedStateCheckpointStore.offload_pins_reclaimed, is folded into the live metrics endpoint via the coordinator's checkpoint_fates (page_unit_checkpoint.py:888), so a non-zero store-side reclaim shows up in get_cache_statistics / collect_metrics while the load-side one stayed invisible. Both counters mean the same thing -- a joint report was lost and the watchdog had to release the stranded slot -- so an operator watching for lost load reports had no signal at all. Fold orphan_load_slots_reclaimed into BlockManager.checkpoint_funnel(), which both metrics readers consume (engine_utility.py:317 and :358), so the load-side reclaim is visible symmetrically with its store-side twin. Co-Authored-By: Claude Opus 4.8 --- atom/model_engine/block_manager.py | 1 + 1 file changed, 1 insertion(+) diff --git a/atom/model_engine/block_manager.py b/atom/model_engine/block_manager.py index 9f19fe5971..44512f84e0 100644 --- a/atom/model_engine/block_manager.py +++ b/atom/model_engine/block_manager.py @@ -1900,6 +1900,7 @@ def checkpoint_funnel(self) -> dict[str, int]: "state_hbm": self.state_hbm, "state_tier": self.state_tier, "state_gate_lost_boundary": self.state_gate_lost_boundary, + "orphan_load_slots_reclaimed": self._orphan_load_slots_reclaimed, } | self.state_checkpoint_fates() def pool_pressure(self) -> dict[str, int]: From c88f46b39fc26ba031ad96aa739f6322f1c94e5c Mon Sep 17 00:00:00 2001 From: zejchen Date: Mon, 31 Aug 2026 07:28:58 +0000 Subject: [PATCH 040/151] refactor(offload): remove the dead worker-tier stats() hook Problem (maintainer: "every observability hook added for the state tier is dead"): StateOffloadTier.stats() and the three counters that fed it -- load_queue_wait_ms_last, load_queue_wait_ms_max, loads_started -- had no production consumer. stats() lives in the worker process; the engine's get_statistics() (connector.py:226 -> _offload_common.py:330) reads only the engine-side StateOffloadIndex counters and has no channel back to the worker tier, so nothing ever drained stats(). Its only callers were two unit tests. The accumulators were therefore write-only: incremented in _do_load and read nowhere. The real, live load-wait signal is NOT lost. _do_load still computes the per-load queue wait and emits logger.warning at >= _LOAD_WAIT_WARN_MS ("...waited %dms for its lane... This is TTFT"), and oldest_store_age_s() (read by that same warning) stays. That warning -- fired exactly when the wait is operationally significant -- is the observable form of what the dead accumulators only tallied into an unread dict. Fix: - Drop stats() and the load_queue_wait_ms_last/max + loads_started fields and their updates in _do_load; keep the waited_ms computation and the warning. - Keep _store_submitted_at and oldest_store_age_s() (feed the live warning). - Update the stale submit_load docstring that named the removed metric. - test_the_oldest_store_age_is_visible_while_it_hangs: assert inflight via len(_store_submitted_at) instead of the removed stats() key (its real subject, oldest_store_age_s, is unchanged). - Remove test_load_queue_wait_is_measured: it covered only the removed dead accumulators; the surviving load-wait signal is the threshold warning. Prefer removal over wiring a cross-process stats channel: this is a thinning/cleanup branch, and the warning already surfaces the one worker-side signal (load-queue latency) that the engine's own counters cannot express. Co-Authored-By: Claude Opus 4.8 --- .../offload/hybrid/kimi_k3/state_tier.py | 26 +------------------ tests/test_lmcache_offload_connector.py | 17 +----------- 2 files changed, 2 insertions(+), 41 deletions(-) diff --git a/atom/kv_transfer/offload/hybrid/kimi_k3/state_tier.py b/atom/kv_transfer/offload/hybrid/kimi_k3/state_tier.py index 62ddaa12e7..68626fd55c 100644 --- a/atom/kv_transfer/offload/hybrid/kimi_k3/state_tier.py +++ b/atom/kv_transfer/offload/hybrid/kimi_k3/state_tier.py @@ -76,12 +76,6 @@ def __init__(self, codec, *, max_workers: int = 1, staging_lanes: int = 2) -> No self._source_released: set = set() self._indexed: set = set() self._index_failed: set = set() - # Latency, not operation counts. The in-flight cap bounds how many - # transfers are outstanding; it says nothing about bytes or about how - # long a load waited, which is what a TTFT regression is made of. - self.load_queue_wait_ms_last = 0.0 - self.load_queue_wait_ms_max = 0.0 - self.loads_started = 0 # op -> monotonic at submission, for `oldest_store_age_s`. self._store_submitted_at: dict = {} @@ -113,18 +107,6 @@ def oldest_store_age_s(self) -> float: oldest = min(self._store_submitted_at.values()) return max(0.0, monotonic() - oldest) - def stats(self) -> dict: - """Queue latency for the periodic line. Counters live in the engine.""" - with self._lock: - inflight_stores = len(self._store_submitted_at) - return { - "state_load_queue_wait_ms_last": round(self.load_queue_wait_ms_last, 2), - "state_load_queue_wait_ms_max": round(self.load_queue_wait_ms_max, 2), - "state_loads_started": self.loads_started, - "state_stores_inflight": inflight_stores, - "state_oldest_store_age_s": round(self.oldest_store_age_s(), 2), - } - def submit_store(self, op, unit_ids) -> None: """Pack the checkpoint image in `unit_ids` for LMCache, under `op`. @@ -147,7 +129,7 @@ def submit_load(self, req_id: str, h: int, slot: int) -> None: Its own lane, so a load is never behind a backlog of stores. What the two lanes still share is the staging-memory budget; see `__init__`. - `state_load_queue_wait_ms` measures what is left of the wait. + `_do_load` warns when that remaining wait crosses `_LOAD_WAIT_WARN_MS`. """ self._register( self._load_executor.submit(self._do_load, req_id, h, slot, monotonic()) @@ -274,12 +256,6 @@ def _do_load(self, req_id: str, h: int, slot: int, submitted_at=None) -> None: # the report below. if submitted_at is not None: waited_ms = max(0.0, monotonic() - submitted_at) * 1000.0 - with self._lock: - self.load_queue_wait_ms_last = waited_ms - self.load_queue_wait_ms_max = max( - self.load_queue_wait_ms_max, waited_ms - ) - self.loads_started += 1 if waited_ms >= _LOAD_WAIT_WARN_MS: logger.warning( "state offload: a state load waited %.0fms for its lane " diff --git a/tests/test_lmcache_offload_connector.py b/tests/test_lmcache_offload_connector.py index b4a3cf67c7..c043ca3b8b 100644 --- a/tests/test_lmcache_offload_connector.py +++ b/tests/test_lmcache_offload_connector.py @@ -5310,28 +5310,13 @@ def test_the_oldest_store_age_is_visible_while_it_hangs(self): assert tier.oldest_store_age_s() == 0.0 tier.submit_store(StateStoreOperationId(1, 1), (0,)) assert tier.oldest_store_age_s() >= 0.0 - assert tier.stats()["state_stores_inflight"] == 1 + assert len(tier._store_submitted_at) == 1 finally: codec.gate.set() tier.drain() assert tier.oldest_store_age_s() == 0.0 tier.shutdown() - def test_load_queue_wait_is_measured(self): - """The metric the old comment told readers to measure did not exist.""" - codec = self._BlockingCodec() - codec.gate.set() - tier = self._tier(codec) - try: - tier.submit_load("r1", 77, 0) - tier.drain() - stats = tier.stats() - assert stats["state_loads_started"] == 1 - assert stats["state_load_queue_wait_ms_last"] >= 0.0 - assert "state_load_queue_wait_ms_max" in stats - finally: - tier.shutdown() - class _EarlyReleaseCodec: """`put` releases the source (D2H drained) and then stalls in the CPU `batched_put`, so a drain can observe the early release *before* the From 7e08f5994dc27f968829d8d9fe2b1674b9b776c5 Mon Sep 17 00:00:00 2001 From: zejunchen-zejun Date: Mon, 31 Aug 2026 17:14:31 +0800 Subject: [PATCH 041/151] fix(ci): satisfy the pinned Black version, and re-fix a stale joint test fixture Two failures on the pre-checks run, both from the commits added on top of the joint-identity work. **Black.** `psf/black@stable` resolves to 26.5.1, which collapses a comprehension that fits in 88 columns where 26.1.0 left it split. The offending hunk is `reclaim_stale_orphan_load_slots` in `block_manager.py` -- one file, one hunk, and `black .` touches nothing else. Same class of problem as the earlier ruff skew: the local toolchain was a version behind the one CI pins, so a clean local run said nothing about CI. Local Black is now on 26.5.1 too. **Three joint tests.** `TestJointLegsShareOneCompletionIdentity` failed on `test_the_kv_leg_alone_does_not_wake_the_request`, `test_both_legs_wake_it_once_under_the_kv_identity` and `test_either_leg_failing_fails_the_pair`. The tests were stale, not the code: `_arm_joint_loads` has since grown a guard that refuses to arm a park when `_state_tier is None`, which is right -- with no tier the state leg is unsettleable (`_start_state_loads` fails those loads for recompute, and `get_finished` returns at its own tier-None guard before `_settle_joint` ever drains the park), so arming would leave the park owing a KV leg forever and leak one `_need`/`_alias` entry per joint load per step. The `_k3_worker` helper set `_state_tier = None` purely for convenience, contradicting its own docstring, so every test in the class was silently taking that new passthrough path. The helper now installs a truthy sentinel -- never called, since these tests drive `_arm_joint_loads`/`_settle_joint` directly -- and takes a `tier=False` switch. Added a test that pins the guard itself: with no tier the KV leg is not parked and the park stays empty, so the behaviour the three tests tripped over is now covered on purpose rather than by accident. --- atom/model_engine/block_manager.py | 4 +--- tests/test_lmcache_offload_connector.py | 27 +++++++++++++++++++++---- 2 files changed, 24 insertions(+), 7 deletions(-) diff --git a/atom/model_engine/block_manager.py b/atom/model_engine/block_manager.py index 44512f84e0..22a110340c 100644 --- a/atom/model_engine/block_manager.py +++ b/atom/model_engine/block_manager.py @@ -1265,9 +1265,7 @@ def reconcile_orphan_load_slots(self, timeout_s: float) -> int: return 0 cutoff = monotonic() - timeout_s stale = [ - req_id - for req_id, at in self._orphan_load_slots_at.items() - if at <= cutoff + req_id for req_id, at in self._orphan_load_slots_at.items() if at <= cutoff ] for req_id in stale: self._orphan_load_slots_at.pop(req_id, None) diff --git a/tests/test_lmcache_offload_connector.py b/tests/test_lmcache_offload_connector.py index c043ca3b8b..034dd22bc6 100644 --- a/tests/test_lmcache_offload_connector.py +++ b/tests/test_lmcache_offload_connector.py @@ -5092,12 +5092,18 @@ def test_the_shells_no_impl_fallbacks_match_what_the_caller_unpacks(): # ── kimi_k3: the two joint legs report different identities ─────────────── -def _k3_worker() -> KimiK3OffloadConnector: - """Only what the joint overrides touch. `_state_tier` is a sentinel object - because `get_finished` refuses to run its state half without one.""" +def _k3_worker(*, tier: bool = True) -> KimiK3OffloadConnector: + """Only what the joint overrides touch. + + `_state_tier` is a truthy sentinel by default and never called: these tests + drive `_arm_joint_loads`/`_settle_joint` directly. It has to be present + because both refuse to coordinate a park without a tier -- with no tier the + state leg is unsettleable, so a KV-only passthrough is the correct + behaviour. `tier=False` exercises that path. + """ c = KimiK3OffloadConnector.__new__(KimiK3OffloadConnector) c._joint_park = _JointPark() - c._state_tier = None + c._state_tier = object() if tier else None c._do_load = True return c @@ -5178,6 +5184,19 @@ def test_the_park_does_not_leak_an_entry_per_joint_load(self): assert park._alias == {} assert park._alias_of == {} + def test_with_no_tier_the_kv_leg_is_not_parked_at_all(self): + """The state leg is unsettleable without a tier -- `_start_state_loads` + fails those loads for recompute -- so arming would leave the park owing + a KV leg forever and leak an entry per load. A KV-only passthrough is + what a load with no state leg should do.""" + worker = _k3_worker(tier=False) + req = _k3_load_req("r1") + worker._arm_joint_loads( + SimpleNamespace(state_loads=[("r1", 99, 0)], requests=[req]) + ) + assert not worker._joint_park.waits_for(req.load_operation) + assert worker._joint_park._need == {} + def test_a_single_leg_request_still_passes_straight_through(self): """Nothing armed it, so neither channel may be held back.""" worker = _k3_worker() From 95f940e47b8326dc860b384612ac271eb3ee9e4e Mon Sep 17 00:00:00 2001 From: zejchen Date: Mon, 31 Aug 2026 14:26:01 +0000 Subject: [PATCH 042/151] fix(offload/kimi_k3): return False for unowned connector completions `connector_completion` fell through to `super().connector_completion()` for any channel other than STATE_SOURCE / STATE_INDEX. No class in the MRO (`DenseOffloadConnector` and up) defines that method -- only the two sibling hybrid connectors (kimi_k3, dsv4) do -- so the super() call raised AttributeError the moment a completion on an unowned channel arrived, wedging the offload apply path. `_offload_common._apply_connector_completions` already treats a `False` return as "unhandled channel: log and skip", which is exactly the intent here and what the DSV4 sibling returns. Return False. Addresses review finding #1. --- atom/kv_transfer/offload/hybrid/kimi_k3/connector.py | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py b/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py index df51f260e1..a7c799fc2f 100644 --- a/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py +++ b/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py @@ -598,7 +598,12 @@ def connector_completion(self, completion) -> bool: ) target.add(completion.operation_id) return True - return super().connector_completion(completion) + # Channels this connector does not own. `DenseOffloadConnector` and the + # rest of the MRO define no `connector_completion`, so `super().` would + # raise AttributeError; `False` is the caller's contract for "unhandled" + # (see `_offload_common._apply_connector_completions`) and matches the + # DSV4 sibling connector. + return False def take_state_source_releases(self) -> set: """Drain the stores whose PAGE units the GPU has finished reading. From 1745d8f1a829fc061a2f2d11ccf015d855485c8d Mon Sep 17 00:00:00 2001 From: zejchen Date: Mon, 31 Aug 2026 14:26:17 +0000 Subject: [PATCH 043/151] fix(scheduler): guard block_manager deref in _publish_state_loads `_publish_state_loads` dereferenced `self.block_manager` directly, while its two siblings on the same idle-drain path -- `_settle_state_load` and `_publish_state_stores` -- both go through `getattr(getattr(self, "block_manager", None), , None)` precisely because a scheduler built without a block manager (the connector test doubles) reaches this code. Mirror the sibling guard so the bare deref can no longer AttributeError. Addresses review finding #7 (block_manager deref). --- atom/model_engine/scheduler.py | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/atom/model_engine/scheduler.py b/atom/model_engine/scheduler.py index fcea50ab31..c5e80390af 100644 --- a/atom/model_engine/scheduler.py +++ b/atom/model_engine/scheduler.py @@ -2625,7 +2625,13 @@ def _publish_state_loads(self) -> None: """ if self.kv_connector is None: return - loads = self.block_manager.take_state_loads() + # Guarded like the `_settle_state_load` / `_publish_state_stores` + # siblings: a scheduler built without a block manager (the connector + # test doubles) would otherwise AttributeError on the bare deref. + take = getattr(getattr(self, "block_manager", None), "take_state_loads", None) + if take is None: + return + loads = take() if not loads: return accepted = False From 7a6534a11d3a5e02354c2da8e14303aef5cc399a Mon Sep 17 00:00:00 2001 From: zejchen Date: Mon, 31 Aug 2026 14:28:29 +0000 Subject: [PATCH 044/151] fix(block_manager): replace -O-stripped assert in allocate() claim loop The claim loop guarded the "cached block vanished during allocate" case with a bare `assert i >= num_cached_blocks`. Under `python -O` the assert is stripped and the fall-through was wrong two ways: * `num_cached_tokens` was set to `num_cached_blocks * hbs` (line below the loop) even though only `i < num_cached_blocks` blocks had been claimed -- the forward then resumed over a prefix that was not actually in the pool, and `hit_hash` was never assigned so the state leg took its cold-start exit. * only `state_joint_claim_tokens` was clamped; `state_joint_boundary_*` and `state_joint_kv_tokens` stayed above the claimed region, so the KV leg would load to a boundary the request could not claim and `_claim_after_load` would raise `num_cached_tokens` past it. Both are silent wrong output. Convert to real control flow: clamp `num_cached_blocks` to the blocks actually claimed, and drop the joint boundary (all three joint vars + hash) whenever it sits above the claimed prefix, so the request recomputes rather than resuming over a gap. Correctness no longer depends on assertions being enabled. Addresses review finding #7 (allocate assert). --- atom/model_engine/block_manager.py | 35 +++++++++++++++++++++++------- 1 file changed, 27 insertions(+), 8 deletions(-) diff --git a/atom/model_engine/block_manager.py b/atom/model_engine/block_manager.py index 22a110340c..3d8e609037 100644 --- a/atom/model_engine/block_manager.py +++ b/atom/model_engine/block_manager.py @@ -866,14 +866,33 @@ def allocate(self, seq: Sequence, num_cached_blocks: int = 0): h = self.compute_hash(token_ids, h) block_id = self.kv.lookup(h) if block_id == -1: - # Evicted between `can_allocate` and here. Unreachable for the - # cached range (nothing evicts inside one admission), so only - # the widened tail can land here: stop claiming and let the rest - # come from the free pool. - assert ( - i >= num_cached_blocks - ), f"cached block {i} of {num_cached_blocks} vanished during allocate" - seq.state_joint_claim_tokens = i * hbs + # Evicted between `can_allocate` and here. For the widened joint + # tail this is expected -- LMCache or the state tier still holds + # it and the forward refetches. For the cached range it should be + # unreachable (nothing evicts inside one admission), but an + # `assert i >= num_cached_blocks` vanishes under `python -O`, and + # the old fall-through then recorded `num_cached_blocks * hbs` as + # cached over a prefix it could not claim, left `hit_hash` + # unassigned (cold-start state exit), and clamped only + # `state_joint_claim_tokens` while `boundary`/`kv` stayed above + # the claimed region -- silent wrong output on both counts. Use + # real control flow instead: never treat more than the `i` blocks + # actually claimed as cached, and drop any joint boundary that + # sits above them so the KV leg cannot resume over a gap. + claimed_tokens = i * hbs + if i < num_cached_blocks: + num_cached_blocks = i + if ( + int(getattr(seq, "state_joint_boundary_tokens", 0) or 0) + > claimed_tokens + ): + seq.state_joint_boundary_tokens = 0 + seq.state_joint_boundary_hash = -1 + seq.state_joint_kv_tokens = 0 + seq.state_joint_claim_tokens = min( + int(getattr(seq, "state_joint_claim_tokens", 0) or 0), + claimed_tokens, + ) break self.kv.claim(block_id) seq.block_table.append(block_id) From 0386c6d6ca213b186d0cf322910b31dc4b3a69ac Mon Sep 17 00:00:00 2001 From: zejchen Date: Mon, 31 Aug 2026 14:40:59 +0000 Subject: [PATCH 045/151] fix(multi_connector): release state-store completions instead of leaking `get_finished`'s send/save pairing path parks every save completion in `self._saved` keyed by `completion_req_key`, and pops it only when a matching send with the same key drains from `self._sent`. State-tier store completions are `StateStoreOperationId` -- a (prefix_hash, generation) pair with no `req_id` and, crucially, no send counterpart -- so their key never enters `self._sent` and their `_saved` entry was never popped. Over a long run `self._saved` grew without bound. State stores are terminal on their own; they do not pair with a send. Detect them by exact type (`isinstance(r, StateStoreOperationId)`) and release them straight into `finished_saving`, pairing only the request-scoped KV saves. The check must be `isinstance`, not `hasattr(r, "req_id")`: a bare `ReqId` save completion (a plain str/int) also lacks a `req_id` attribute and must still go through send/save pairing on a producer node. Addresses review finding #7 (StateStoreOperationId unbounded _saved). --- .../disaggregation/multi/multi_connector.py | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-) diff --git a/atom/kv_transfer/disaggregation/multi/multi_connector.py b/atom/kv_transfer/disaggregation/multi/multi_connector.py index 541d1c1592..8f68b16966 100644 --- a/atom/kv_transfer/disaggregation/multi/multi_connector.py +++ b/atom/kv_transfer/disaggregation/multi/multi_connector.py @@ -71,6 +71,7 @@ ConnectorMetadata, KVConnectorOutput, SaveCompletionId, + StateStoreOperationId, completion_req_key, connector_metadata_has_work, ) @@ -350,7 +351,19 @@ def get_finished(self) -> KVConnectorOutput: # Pair each request's send and save before releasing either. for r in send_now: self._sent[str(r)] = r + # State-tier store completions (`StateStoreOperationId`: a + # (prefix_hash, generation) pair) have no send counterpart to pair + # against. Parking them in `self._saved` leaked for the life of the + # process -- their key never enters `self._sent`, so the pop below never + # fired. They are terminal on their own: release immediately. Match on + # the exact type, not `hasattr(r, "req_id")` -- a bare `ReqId` save + # completion (a plain str/int) has no `req_id` attribute either and must + # still go through send/save pairing on a producer node. + state_saves: set = set() for r in save_now: + if isinstance(r, StateStoreOperationId): + state_saves.add(r) + continue key = completion_req_key(r) self._saved.setdefault(key, set()).add(r) pending_ops = self._pending_save_ops.get(key) @@ -369,7 +382,7 @@ def get_finished(self) -> KVConnectorOutput: rel_save.update(self._saved.pop(key, set())) out.finished_sending = rel_send - out.finished_saving = rel_save + out.finished_saving = rel_save | state_saves return out def get_finished_recv_blocks(self) -> list[int]: From 46d7e70ca3231f75e3bfa83842bb1c0797207514 Mon Sep 17 00:00:00 2001 From: zejchen Date: Mon, 31 Aug 2026 14:38:23 +0000 Subject: [PATCH 046/151] fix(block_manager): pool_pressure fates via aggregator, not a single cache `pool_pressure()` read `self._state_checkpoint_cache.checkpoint_fates()` directly, while `checkpoint_funnel()` (block_manager.py:1921) reports the same four fate counters through `self.state_checkpoint_fates()` -- the aggregator that sums every state class. The two paths therefore printed different numbers under the same metric names: `pool_pressure` undercounts any second state class the aggregator folds in (and prints four zeros under PAGE, where `self.state` is a `StateTransfer.none()` that never checkpoints), while the funnel line shows the real total. Two outputs disagreeing on "state checkpoints" is exactly how a tuning session concludes checkpointing never fires. Route `pool_pressure` through the aggregator too; occupancy stays on `self.state`, the slot pool, which is unchanged. Ref: review #8-4g. --- atom/model_engine/block_manager.py | 31 +++++++++++++++--------------- 1 file changed, 16 insertions(+), 15 deletions(-) diff --git a/atom/model_engine/block_manager.py b/atom/model_engine/block_manager.py index 3d8e609037..b1c4eb811d 100644 --- a/atom/model_engine/block_manager.py +++ b/atom/model_engine/block_manager.py @@ -1929,25 +1929,26 @@ def pool_pressure(self) -> dict[str, int]: comparison: paged evicting while state sits mostly vacant means the split is wrong, both evicting means the budget is. - The fates come from `_state_checkpoint_cache`, not from `self.state`. - Under PAGE the two are different objects — `self.state` is built with - `StateTransfer.none()` and never sees a `checkpoint()` — so reading it - here printed four zeros for the life of the server while - `checkpoint_funnel` reported the real numbers from the coordinator. Two - outputs with the same metric names disagreeing is how a tuning session - concludes checkpointing never fires. Occupancy stays on `self.state`: - that is the slot pool either way, and the coordinator has none. + The fates come from `state_checkpoint_fates()`, the aggregator, not + from `self.state` and not from a single `_state_checkpoint_cache`. + Under PAGE `self.state` is built with `StateTransfer.none()` and never + sees a `checkpoint()`, so reading it here printed four zeros for the + life of the server; and reading `_state_checkpoint_cache` alone omits + any second state class the aggregator folds in. Either way the number + disagreed with the `checkpoint_funnel` line -- which already goes + through the aggregator -- and two outputs with the same metric names + disagreeing is how a tuning session concludes checkpointing never + fires. Occupancy stays on `self.state`: that is the slot pool either + way, and the coordinator has none. """ - # `_state_checkpoint_cache`, not `self.state`: under #2045 a K3 - # checkpoint is a PAGE image the coordinator owns, and the slot pool - # holds none -- so reading the pool here printed `kept: 0` beside a - # `state checkpoints:` line saying 112, two counters for one fact - # disagreeing in the same log. `occupancy()` stays on the pool, which - # is the thing that has slots. + # `state_checkpoint_fates()`, matching `checkpoint_funnel`: it sums + # every state class, so a build with a second class is not silently + # undercounted here while the funnel line reports the full total. + # `occupancy()` stays on the pool, which is the thing that has slots. return ( self.kv.eviction_stats() | self.state.occupancy() - | self._state_checkpoint_cache.checkpoint_fates() + | self.state_checkpoint_fates() ) def checkpointers_at( From d9a731d3c4112f5158e228eb0dcced0ccddf1e2f Mon Sep 17 00:00:00 2001 From: zejchen Date: Mon, 31 Aug 2026 14:38:23 +0000 Subject: [PATCH 047/151] fix(page_unit_checkpoint): requeue deferred nominations oldest-first The deferred-nomination requeue used `self._offload_ready.extend(deferred)`, appending the older, already-waiting nominations to the *right* of the deque. But the drain pops from the left (`popleft()`, oldest-first), so newer nominations queued after them win every drain and the deferred ones starve at the back until they age out at the `state != READY` check. Put them back at the front with `extendleft(reversed(deferred))`, which restores their original oldest-first order ahead of newer arrivals. Ref: review #8-4h (page_unit requeue). --- atom/model_engine/page_unit_checkpoint.py | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/atom/model_engine/page_unit_checkpoint.py b/atom/model_engine/page_unit_checkpoint.py index d044a210e4..2ff438f1c7 100644 --- a/atom/model_engine/page_unit_checkpoint.py +++ b/atom/model_engine/page_unit_checkpoint.py @@ -450,9 +450,13 @@ def take_offload_stores( self._offload_pins[op] = (checkpoint_id, monotonic()) out.append((op, record.unit_ids)) # Re-queue the deferred nominations for the next drain, once the - # colliding in-flight generation has had a chance to settle. + # colliding in-flight generation has had a chance to settle. The drain + # is `popleft()` (oldest at the left), so `extend` would append these + # older, already-waiting nominations *behind* newer ones and let them + # starve until they age out at the `state != READY` check. Put them + # back at the front, preserving their original oldest-first order. if deferred: - self._offload_ready.extend(deferred) + self._offload_ready.extendleft(reversed(deferred)) return out def _hash_in_flight(self, prefix_hash: int) -> bool: From 8acac20a3cb6d902a387f7c0759723d21789dc1b Mon Sep 17 00:00:00 2001 From: zejchen Date: Mon, 31 Aug 2026 14:38:23 +0000 Subject: [PATCH 048/151] test: drop two brittle source-text assertions `test_a_late_report_on_a_reclaimed_save_is_not_an_assertion` (test_scheduler.py) and `test_the_runner_warms_the_builder_once_the_pools_are_reachable` (test_v4_checkpoint_slot_copy.py) asserted on `inspect.getsource(...)` substrings and statement ordering rather than on behaviour. They break on any whitespace/refactor that leaves the behaviour intact and pass even when the behaviour regresses, so they test the text, not the code. Remove them. The dynamic scan in test_lmcache_offload_connector.py:5042 is a legitimate capability probe and is kept. Ref: review #8-4b. --- tests/test_scheduler.py | 9 --------- tests/test_v4_checkpoint_slot_copy.py | 16 ---------------- 2 files changed, 25 deletions(-) diff --git a/tests/test_scheduler.py b/tests/test_scheduler.py index db4e1c6994..cf73cfb4f4 100644 --- a/tests/test_scheduler.py +++ b/tests/test_scheduler.py @@ -1494,15 +1494,6 @@ def test_the_window_sits_above_lmcaches_own_pin_timeout(self, monkeypatch): monkeypatch.delenv("LMCACHE_EC_PIN_TIMEOUT_SEC", raising=False) assert sched_mod._offload_save_abandon_timeout_s() == 330.0 - def test_a_late_report_on_a_reclaimed_save_is_not_an_assertion(self): - """`finished_sending` used to assert the sequence was still deferred.""" - import inspect - - import atom.model_engine.scheduler as sched_mod - - src = inspect.getsource(sched_mod.Scheduler._update_from_kv_xfer_finished) - assert "not found in deferred_free_blocks" not in src - class TestTheTierSplitPartitionsServedReuse: """`[Cache Tiers]` exists to answer "what does the CPU tier buy", so its two diff --git a/tests/test_v4_checkpoint_slot_copy.py b/tests/test_v4_checkpoint_slot_copy.py index 14b1aaea1f..94d58df737 100644 --- a/tests/test_v4_checkpoint_slot_copy.py +++ b/tests/test_v4_checkpoint_slot_copy.py @@ -598,22 +598,6 @@ def test_a_backend_without_paged_checkpoints_warms_nothing(self): # and ModelRunner calls this unconditionally. assert AttentionMetadataBuilder.warmup_per_req_cache(object()) is None - def test_the_runner_warms_the_builder_once_the_pools_are_reachable(self): - """After the setattr loop, not before: the builder reads them off it. - - Guarded here rather than left to review because nothing else calls - `warmup_per_req_cache` -- a runner that stopped would restore the - first-request JIT silently, with every test still green. - """ - import inspect - - from atom.model_engine import model_runner - - src = inspect.getsource(model_runner.ModelRunner.allocate_kv_cache) - install = src.index("setattr(self, name, value)") - warm = src.index("warmup_per_req_cache()") - assert install < warm, "warmed before the pools were installed" - class TestPageUnitRegionsValidateTheirOwnAddresses: """The region cache holds addresses from two allocations, one uncovered. From 9c864171934c8a9a1fe6d479486c0c49b4f8c46f Mon Sep 17 00:00:00 2001 From: zejchen Date: Mon, 31 Aug 2026 14:38:23 +0000 Subject: [PATCH 049/151] docs: correct docstrings that named dead _do_spill / _submit_state_spills Three docstrings referenced methods that no longer exist, and one of them described an architecture that has since changed: * staging.py `StagedTransfer` / `pack` claimed the state producer fence is `state_tier._do_spill`'s `ready_event.synchronize()`. `_do_spill` is gone, and the current state path needs no event at all: `state_tier.submit_store` reserves the PAGE units out of the KV pool and engine-pins them for the whole transfer, so nothing on the compute stream writes them. The KV path is still fenced by the caller's `save_ready_event`. Rewrite to describe both callers accurately. * multi_connector.py (module docstring + __init__) claimed `AttentionBackend._submit_state_spills` reads/probes `_state_tier`. That symbol is dead; `_state_tier` is populated by `_adopt_state_tier` at `register_kv_caches` time (which also refuses a config listing two offload subs). Point the comments at the real mechanism. Docstring-only; no behaviour change. Ref: review #8-4d (docstrings). --- .../disaggregation/multi/multi_connector.py | 14 ++++---- .../offload/hybrid/kimi_k3/staging.py | 33 ++++++++++++------- 2 files changed, 29 insertions(+), 18 deletions(-) diff --git a/atom/kv_transfer/disaggregation/multi/multi_connector.py b/atom/kv_transfer/disaggregation/multi/multi_connector.py index 8f68b16966..879747188c 100644 --- a/atom/kv_transfer/disaggregation/multi/multi_connector.py +++ b/atom/kv_transfer/disaggregation/multi/multi_connector.py @@ -39,9 +39,10 @@ * ``get_finished`` — union the completion sets, **but** see the send/save pairing below. * ``_state_tier`` — the state offload tier, if a sub built one, re-exposed on - the composite. ``AttentionBackend._submit_state_spills`` reads this attribute - off whatever connector the forward context holds, so without it the engine's - spills are staged and never submitted. + the composite by ``_adopt_state_tier`` at ``register_kv_caches`` time (which + also refuses a config that lists two offload subs). Mirroring the sub's tier + on the composite keeps the attribute defined, so a probe for it resolves to + the real tier instead of raising ``AttributeError``. Send/save pairing (the one tricky correctness point) ---------------------------------------------------- @@ -241,9 +242,10 @@ def __init__(self, config: Any) -> None: self._pending_save_ops: dict[str, set[SaveCompletionId]] = {} self._sent: dict[str, Any] = {} self._saved: dict[str, set[SaveCompletionId]] = {} - # The state tier of whichever sub owns one. Always defined: - # `_submit_state_spills` probes it every batch, and a composite without - # the attribute would swallow every spill and starve the ring. + # The state tier of whichever sub owns one. Adopted in + # `register_kv_caches` via `_adopt_state_tier`; set to None here so the + # attribute exists before the subs register -- a probe for it must + # resolve, not raise `AttributeError`. self._state_tier = None @property diff --git a/atom/kv_transfer/offload/hybrid/kimi_k3/staging.py b/atom/kv_transfer/offload/hybrid/kimi_k3/staging.py index 84f8ed6525..7ca7721b0b 100644 --- a/atom/kv_transfer/offload/hybrid/kimi_k3/staging.py +++ b/atom/kv_transfer/offload/hybrid/kimi_k3/staging.py @@ -39,16 +39,23 @@ class StagedTransfer: constant, so a single object of a different size breaks both invariants. State is not a member of that loop. - **The cross-thread producer fence is the caller's, not this class's.** - Nothing here waits on the forward's compute stream: `pack` issues its - gather on a private `pack_stream`, and the two `_StagingBuffer` events only - order this worker's own streams against each other. The fence commit - 7427e05e added to fix KV corruption on reload is recorded on the RPC thread - and `synchronize()`d by the caller -- `connector.py`'s `save_ready_event` - for KV, and `state_tier._do_spill`'s `ready_event.synchronize()` before - `codec.put` for state. Do not delete either believing this class covers it: - without them the gather reads the staging entry's previous occupant, which - is silent state corruption. + **This class does not fence the producer; the caller keeps the source + quiescent.** Nothing here waits on the forward's compute stream: `pack` + issues its gather on a private `pack_stream`, and the two `_StagingBuffer` + events only order this worker's own streams against each other. The two + callers make the source safe by different means: + + * KV -- `connector.py` records the fence commit 7427e05e added to fix KV + corruption on reload on the RPC thread and `synchronize()`s it through + `save_ready_event` before handing the blocks over. + * State -- `state_tier.submit_store` needs no event: the PAGE units are + reserved out of the KV pool and engine-pinned for the whole transfer, so + nothing on the compute stream is writing them and the gather reads them + where they sit. + + Do not delete the KV `save_ready_event` believing this class covers it: + without it the gather reads the staging entry's previous occupant, which is + silent corruption. """ def __init__( @@ -169,8 +176,10 @@ def pack(self, segments: list[torch.Tensor], dst: Any) -> None: The event around that copy makes the *consumer* side safe: whoever reads the MemoryObj next must not see a D2H still in flight. It says nothing about the *producer* side -- this gather runs on a private - stream and does not wait for the forward that wrote the entry. That - wait is the caller's (`state_tier._do_spill`); see the class docstring. + stream and does not wait for the forward that wrote the entry. Keeping + the source quiescent is the caller's job (KV via `save_ready_event`, + state via reserved-and-pinned units in `state_tier.submit_store`); see + the class docstring. """ from atom.kv_transfer.offload.dense.triton_kv_staging import ( fused_pack_chunk_major, From 4b813c62987ed50c36de70dbe8c65b4b0c46b368 Mon Sep 17 00:00:00 2001 From: zejchen Date: Mon, 31 Aug 2026 14:48:25 +0000 Subject: [PATCH 050/151] fix(offload): abandon_save hook so a reclaimed stalled save cannot wedge the engine MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `Scheduler._reconcile_stalled_deferred_saves` frees the blocks of a save the backend never reported (LMCache force-unpins a stalled transfer without a completion), but it only touched the scheduler's `deferred_free_blocks` -- the connector still held the save in `_save_inflight`. `save_finished` cannot clear it: handed the raw request id while an exact `SaveOperationId` generation is parked, it deliberately refuses (a delayed TP notification must not complete a newer lifecycle). So `_save_inflight` never drained, `should_defer_free` stayed True and `has_pending_kv_work()` never went False, and the EngineCore busy-looped with every GPU idle -- reproduced under a tight pool on the k3-dev line. Add an `abandon_save(req_id)` hook that drops the entry unconditionally: - `_offload_common`: `_cancel_save_statistics`, mirror of the existing `_cancel_load_statistics` -- the bytes were never persisted, so cancel the inflight-tokens gauge rather than counting a success. - `DenseOffloadConnector.abandon_save`: pop `_save_inflight` (+cancel stats) and `_save_tracker` so the save loop cannot re-emit against freed blocks. - `KimiK3OffloadConnector.abandon_save`: also re-run `_refresh_save_stall` so the stall latch sheds the abandoned sid instead of staying stuck True. - offload wrapper + MultiConnector: forward the call to the owning sub. - `Scheduler._reconcile_stalled_deferred_saves`: notify the connector via a getattr-guarded `_connector_abandon_save` before deallocating. Tests: make the reclaim stub realistic (production always sets `kv_connector`) and add a test asserting the connector is notified with the string request id. Addresses review finding #4 (_save_inflight never cleared on reclaim → busy-loop). --- .../disaggregation/multi/multi_connector.py | 10 +++++++ atom/kv_transfer/offload/_offload_common.py | 11 ++++++++ atom/kv_transfer/offload/connector.py | 3 +++ atom/kv_transfer/offload/dense/connector.py | 26 ++++++++++++++++++- .../offload/hybrid/kimi_k3/connector.py | 12 +++++++++ atom/model_engine/scheduler.py | 21 ++++++++++++++- tests/test_scheduler.py | 19 ++++++++++++++ 7 files changed, 100 insertions(+), 2 deletions(-) diff --git a/atom/kv_transfer/disaggregation/multi/multi_connector.py b/atom/kv_transfer/disaggregation/multi/multi_connector.py index 879747188c..6287fe5b8c 100644 --- a/atom/kv_transfer/disaggregation/multi/multi_connector.py +++ b/atom/kv_transfer/disaggregation/multi/multi_connector.py @@ -436,6 +436,16 @@ def request_finished(self, seq: Any) -> None: if hasattr(c, "request_finished"): c.request_finished(seq) + def abandon_save(self, req_id: Any) -> None: + # Reclamation of a stalled offload save (see + # `DenseOffloadConnector.abandon_save`). Only the offload sub tracks + # `_save_inflight`; forward to whichever sub implements it. Idempotent + # (pop-with-default), so fanning to all is harmless. + for c in self._connectors: + fn = getattr(c, "abandon_save", None) + if callable(fn): + fn(req_id) + # -- offload-specific methods, forwarded to the owning sub -------------- # The scheduler guards every one of these with hasattr(), so MultiConnector # only needs to expose them when a sub-connector implements them. diff --git a/atom/kv_transfer/offload/_offload_common.py b/atom/kv_transfer/offload/_offload_common.py index 94856d4764..f9a04c3084 100644 --- a/atom/kv_transfer/offload/_offload_common.py +++ b/atom/kv_transfer/offload/_offload_common.py @@ -327,6 +327,17 @@ def _finish_save_statistics(self, operation) -> None: self.total_save_requests += 1 self.total_saved_tokens += tokens + def _cancel_save_statistics(self, operation) -> None: + """Forget a save retired without a terminal (scheduler abandon). + + Mirror of `_cancel_load_statistics`: the bytes were never persisted, so + this must not bump `total_save_requests`/`total_saved_tokens` the way + `_finish_save_statistics` does -- it only drops the inflight-tokens entry + so the pending gauge does not leak. + """ + + self._save_inflight_tokens.pop(operation, None) + def get_statistics(self) -> dict[str, int]: """Return cumulative counters and exact-operation queue depths.""" diff --git a/atom/kv_transfer/offload/connector.py b/atom/kv_transfer/offload/connector.py index dc6d138220..af28056d22 100644 --- a/atom/kv_transfer/offload/connector.py +++ b/atom/kv_transfer/offload/connector.py @@ -170,6 +170,9 @@ def has_pending_work(self) -> bool: def save_finished(self, req_id) -> None: self._impl.save_finished(req_id) + def abandon_save(self, req_id) -> None: + self._impl.abandon_save(req_id) + def load_failed(self, req_id): return self._impl.load_failed(req_id) diff --git a/atom/kv_transfer/offload/dense/connector.py b/atom/kv_transfer/offload/dense/connector.py index 0e8a876a12..55e3e7fa29 100644 --- a/atom/kv_transfer/offload/dense/connector.py +++ b/atom/kv_transfer/offload/dense/connector.py @@ -684,7 +684,8 @@ def has_pending_work(self) -> bool: Feeds ``EngineCore.has_pending_kv_work()``, so it reads only state that clears itself: ``_reqs_need_recv`` is emptied by every - ``build_connector_meta`` and ``_save_inflight`` by ``save_finished``. + ``build_connector_meta`` and ``_save_inflight`` by ``save_finished`` + (or ``abandon_save`` when the scheduler reclaims a stalled save). Saves that are queued but not yet dispatched are covered there by the scheduler's ``deferred_free_blocks``, which ``should_defer_free`` keeps populated for exactly those requests. @@ -705,6 +706,29 @@ def save_finished(self, req_id) -> None: self._save_inflight.pop(sid, None) self._finish_save_statistics(req_id) + def abandon_save(self, req_id) -> None: + """Force-drop a save the scheduler reclaimed after it stalled. + + `save_finished` is the completion path: it matches the exact + `SaveOperationId` and, handed a raw request id while an exact generation + is parked, deliberately refuses -- a delayed TP notification must not + complete a newer lifecycle. Reclamation is the opposite need. The + scheduler's `_reconcile_stalled_deferred_saves` has already freed the + blocks of a save the backend never reported (LMCache force-unpins a + stalled save without a completion) and holds only the raw request id, so + drop the entry unconditionally. Without this the entry lingers, + `should_defer_free` stays True and `has_pending_work` never clears, and + the engine busy-loops with every GPU idle. Not a completion: the bytes + were never persisted, so the statistics are *cancelled*, not finished, + and the tracker entry is dropped so the save loop cannot re-emit it + against freed blocks. + """ + sid = str(req_id.req_id if isinstance(req_id, SaveOperationId) else req_id) + operation = self._save_inflight.pop(sid, None) + if operation is not None: + self._cancel_save_statistics(operation) + self._save_tracker.pop(sid, None) + def load_failed(self, req_id) -> bool: sid = str(req_id.req_id if isinstance(req_id, LoadOperationId) else req_id) active = self._active_load_operations.get(sid) diff --git a/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py b/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py index a7c799fc2f..465267e554 100644 --- a/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py +++ b/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py @@ -494,6 +494,18 @@ def _refresh_save_stall(self) -> None: len(self._save_inflight), ) + def abandon_save(self, req_id) -> None: + """Drop a reclaimed save, then recompute the stall latch. + + K3 ages saves off `_save_inflight` (`_refresh_save_stall`). Once the + base drops the inflight entry, re-run the refresh so the age index sheds + the abandoned sid (its `_save_inflight_since` sweep) and the stall latch + clears -- otherwise `_save_stalled` stays stuck True on a save that is + already gone. + """ + super().abandon_save(req_id) + self._refresh_save_stall() + def should_defer_free(self, seq) -> bool: """Base behaviour, plus an escape when the backend has stopped. diff --git a/atom/model_engine/scheduler.py b/atom/model_engine/scheduler.py index c5e80390af..d16ebe383c 100644 --- a/atom/model_engine/scheduler.py +++ b/atom/model_engine/scheduler.py @@ -1415,6 +1415,19 @@ def _connector_should_defer_free(self, seq: Sequence) -> bool: callback = getattr(self.kv_connector, "should_defer_free", None) return bool(callable(callback) and callback(seq)) + def _connector_abandon_save(self, seq: Sequence) -> None: + """Tell the connector to drop a save this reclaim just abandoned. + + Without it the connector's `_save_inflight` (and, on K3, the stall + latch) keeps the reclaimed request forever: `should_defer_free` stays + True and `has_pending_kv_work()` never goes False, so the engine + busy-loops. Guarded because not every connector implements offload + saves. + """ + callback = getattr(self.kv_connector, "abandon_save", None) + if callable(callback): + callback(str(seq.id)) + def _maybe_release_deferred(self, seq: Sequence) -> None: if ( seq.id not in self.deferred_free_blocks @@ -1443,7 +1456,12 @@ def _reconcile_stalled_deferred_saves(self) -> int: Mirrors the producer `finished_sending` reclaim (pop + deallocate), deliberately not re-invoking `request_finished`: it was already called - when the request finished, before the block free was deferred. + when the request finished, before the block free was deferred. It does + notify the connector via `abandon_save`, though -- freeing the blocks + here is not enough on its own: the connector still holds the save in + `_save_inflight` (and, on K3, in the stall latch), so `should_defer_free` + would stay True and `has_pending_kv_work()` would never clear without + that drop. The complement of the K3 connector's stall escape (`kimi_k3.connector.SAVE_STALL_SECONDS`), not a duplicate of it: that @@ -1467,6 +1485,7 @@ def _reconcile_stalled_deferred_saves(self) -> int: ] for seq in stalled: self.deferred_free_blocks.pop(seq.id, None) + self._connector_abandon_save(seq) self.block_manager.deallocate(seq) self._abandoned_saves += 1 if stalled: diff --git a/tests/test_scheduler.py b/tests/test_scheduler.py index cf73cfb4f4..234aaf0ac0 100644 --- a/tests/test_scheduler.py +++ b/tests/test_scheduler.py @@ -1431,6 +1431,8 @@ def _sched(monkeypatch, deferred): s.deferred_free_blocks = {seq.id: seq for seq in deferred} s._abandoned_saves = 0 s._next_save_reconcile_at = 0.0 + # Production always sets this (possibly None); reclaim now notifies it. + s.kv_connector = None freed: list[int] = [] s.block_manager = SimpleNamespace(deallocate=lambda q: freed.append(q.id)) return s, freed @@ -1448,6 +1450,23 @@ def test_a_save_past_the_window_gets_its_blocks_back(self, monkeypatch): assert 2 in s.deferred_free_blocks, "a save still inside its window is kept" assert s._abandoned_saves == 1 + def test_reclaim_notifies_the_connector_to_drop_the_save(self, monkeypatch): + """Freeing blocks is not enough: without `abandon_save` the connector's + `_save_inflight` keeps the request, `has_pending_kv_work()` never clears, + and the engine busy-loops (review finding #4).""" + import time as _time + + now = _time.monotonic() + stale = SimpleNamespace(id=1, _deferred_save_at=now - 500.0) + s, freed = self._sched(monkeypatch, [stale]) + abandoned: list = [] + s.kv_connector = SimpleNamespace(abandon_save=lambda sid: abandoned.append(sid)) + + assert s._reconcile_stalled_deferred_saves() == 1 + assert freed == [1] + # Notified with the string request id, matching the connector's sid keys. + assert abandoned == ["1"] + def test_it_self_throttles_so_a_1ms_poll_is_cheap(self, monkeypatch): import time as _time From 56e244d2d8c27e7c92799cf8be13ce107f9be5d1 Mon Sep 17 00:00:00 2001 From: zejchen Date: Mon, 31 Aug 2026 14:48:40 +0000 Subject: [PATCH 051/151] fix(offload/kimi_k3): check active load before the save-stall free escape `KimiK3OffloadConnector.should_defer_free` short-circuits to `return False` (release the blocks) when the backend has stalled and the save was never handed out. But it evaluated that escape *before* delegating to the base, whose first act is `if self._has_active_load(seq): return True`. A request whose save has stalled can still have a live load into the same block table, so the escape freed blocks while a load was mid-transfer -- the next request writes into them and the load's result is indexed under the wrong prefix (free-while-writing corruption, matching the offload-only, concurrency-gated garbage-output fingerprint). Hoist `if self._has_active_load(seq): return True` ahead of the stall escape. The predicate is identical to the base's first check, so requests without a live load are unaffected; only a stalled-save-with-active-load request changes -- its blocks now stay deferred until the load finishes. Addresses review finding #5 (save-stall escape bypasses the active-load guard). --- atom/kv_transfer/offload/hybrid/kimi_k3/connector.py | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py b/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py index 465267e554..dc0c537c01 100644 --- a/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py +++ b/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py @@ -519,7 +519,17 @@ def should_defer_free(self, seq) -> bool: tied to LMCache's own force-unpin window. Neither mechanism subsumes the other: this one asks whether the backend ever took the save, that one whether it ever answered. + + An active load is checked *first*, before the stall escape: the base's + `should_defer_free` holds blocks while a load is still reading/writing + them (`_has_active_load`), and the escape must not override that. A + request whose save has stalled can still have a live load into the same + block table; releasing those blocks mid-load is the free-while-writing + corruption the escape is not meant to cause. No-load requests are + unaffected -- the base checks the same predicate first anyway. """ + if self._has_active_load(seq): + return True sid = str(seq.id) if ( self._save_stalled From 7d343dc288151d3948991542d8c278ccde42b50b Mon Sep 17 00:00:00 2001 From: zejchen Date: Mon, 31 Aug 2026 14:56:26 +0000 Subject: [PATCH 052/151] fix(offload/kimi_k3): report no-tier joint loads as failed, not phantom success When the worker state tier never built, a joint load (KV + recurrent state) was left unreported. `_arm_joint_loads` skipped the park on `_state_tier is None`, so `get_finished`'s `_state_tier is None` early-return let the KV leg pass straight through as `finished_loading`. `Scheduler._settle_state_load` (scheduler.py:3503) then counted that as `ok=True` -- a state restore that never happened -- and the request resumed decode reading an Active Slot the state H2D never wrote: silent wrong output. The maintainer reopened the #5 rebuttal on exactly this: the skip was the guard I added *for* #4 (avoid arming under a `kv_id` the state leg could never settle, which would leak a park entry). The two fixes were individually right and jointly left the load unreported. Fix, in cooperation with #4 (not a revert of it): - `_arm_joint_loads` arms a joint load with or without a tier. `_start_state_loads` already fails the state leg on the no-tier path (`_fail_state_loads` -> `settle_state(ok=False)`); with the arm in place that failure lands on a real park entry, so the pair is marked failed and owes only the KV leg. - `get_finished` drains the park via `_settle_joint` (empty state reports) on the no-tier path too, before its early return. The landed KV completion settles the pair into `failed_loading` -> the request recomputes. This does not reintroduce #4's leak: the arm is safe precisely because the drain runs -- the KV completion always arrives (finished or failed) and releases the park entry. A state-only load (no KV leg) is still not armed and is reclaimed by `reconcile_orphan_load_slots`' abandon window, as before. Updates the stale unit test (no-tier is now a fail-for-recompute, not a KV-only passthrough) and its helper docstring. --- .../offload/hybrid/kimi_k3/connector.py | 39 +++++++++++----- tests/test_lmcache_offload_connector.py | 46 ++++++++++++++----- 2 files changed, 62 insertions(+), 23 deletions(-) diff --git a/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py b/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py index dc0c537c01..1d06f6c128 100644 --- a/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py +++ b/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py @@ -228,22 +228,25 @@ def _arm_joint_loads(self, metadata) -> None: H2D was still writing the Active Slot -- silent wrong output, and a park entry leaked per joint load. The park is filed under the KV identity, which is also the one that has to reach the engine. + + **No tier is armed exactly like a tier.** An earlier guard skipped the + arm when `_state_tier is None`, reasoning that an unsettleable state leg + left the park owing the KV leg forever. But `_start_state_loads` fails + these loads (`_fail_state_loads` -> `settle_state(ok=False)`), and with + the arm in place that failure lands on a real park entry -- the pair is + marked failed and owes only the KV leg. `get_finished` then drains the + park on the no-tier path too (before its `_state_tier is None` return), + so the KV completion settles the pair into `failed_loading` and the + request recomputes. Skipping the arm instead let the KV leg pass through + as `finished_loading`, and `Scheduler._settle_state_load(ok=True)` + counted a state restore that never happened -- silent wrong output. The + arm is what makes the failure reportable; the drain is what keeps it + from leaking (the KV completion always arrives and releases the entry). """ loads = getattr(metadata, "state_loads", None) or () state_ids = {req_id for req_id, _h, _slot in loads} if not state_ids or not self._do_load: return - if self._state_tier is None: - # No tier means the state leg is unsettleable: `_start_state_loads` - # fails these loads for recompute, and `get_finished` returns at the - # `self._state_tier is None` guard BEFORE it drains the park via - # `_settle_joint`. Arming here would leave the park owing the KV leg - # forever -- the KV completion passes through un-held and the - # `_need`/`_alias`/`_alias_of` entry leaks, one per joint load per - # tier-None step. With no tier there is nothing to coordinate, so - # letting the KV load flow through the base path un-parked is exactly - # the correct behaviour for a KV-only load. - return for req in metadata.requests: if req.load_spec is not None and req.req_id in state_ids: self._joint_park.arm( @@ -325,6 +328,20 @@ def get_finished(self): ) self._store_failed_no_tier = set() if self._state_tier is None: + # Joint loads are still armed with no tier (see `_arm_joint_loads`) + # and their state leg has already failed (`_fail_state_loads`), so + # the park owes only the KV leg. Drain it here -- before the return + # -- with empty state reports (the tier that would produce them + # never built): a landed KV completion settles the pair into + # `failed_loading` and the request recomputes, instead of flowing + # through as `finished_loading` and being miscounted a state restore + # by `Scheduler._settle_state_load(ok=True)`. This is also what keeps + # the arm from leaking: the KV completion arrives and releases the + # entry. A state-only load (no KV leg) is not armed here and is + # instead reclaimed by `reconcile_orphan_load_slots`' abandon window. + out.finished_loading, out.failed_loading = self._settle_joint( + out.finished_loading, out.failed_loading, set(), set() + ) return out indexed, index_failed = self._state_tier.take_store_reports() state_done, state_failed = self._state_tier.get_finished() diff --git a/tests/test_lmcache_offload_connector.py b/tests/test_lmcache_offload_connector.py index 034dd22bc6..f756986cfd 100644 --- a/tests/test_lmcache_offload_connector.py +++ b/tests/test_lmcache_offload_connector.py @@ -5096,10 +5096,11 @@ def _k3_worker(*, tier: bool = True) -> KimiK3OffloadConnector: """Only what the joint overrides touch. `_state_tier` is a truthy sentinel by default and never called: these tests - drive `_arm_joint_loads`/`_settle_joint` directly. It has to be present - because both refuse to coordinate a park without a tier -- with no tier the - state leg is unsettleable, so a KV-only passthrough is the correct - behaviour. `tier=False` exercises that path. + drive `_arm_joint_loads`/`_settle_joint` directly. A joint load is armed + with or without a tier -- with no tier the state leg is failed for recompute + (`_fail_state_loads`) and the park settles the pair into `failed_loading` + once the KV leg lands, rather than letting the KV leg pass through as a + phantom state-restore success. `tier=False` exercises that path. """ c = KimiK3OffloadConnector.__new__(KimiK3OffloadConnector) c._joint_park = _JointPark() @@ -5184,18 +5185,39 @@ def test_the_park_does_not_leak_an_entry_per_joint_load(self): assert park._alias == {} assert park._alias_of == {} - def test_with_no_tier_the_kv_leg_is_not_parked_at_all(self): - """The state leg is unsettleable without a tier -- `_start_state_loads` - fails those loads for recompute -- so arming would leave the park owing - a KV leg forever and leak an entry per load. A KV-only passthrough is - what a load with no state leg should do.""" + def test_with_no_tier_the_joint_load_fails_for_recompute(self): + """No tier means the state leg cannot be served, so the pair must reach + the engine as `failed_loading` (recompute), never as a phantom success. + + Arming a joint load with no tier and then failing its state leg + (`_fail_state_loads`) leaves the park owing only the KV leg; when the KV + completion lands, `_settle_joint` releases the pair into `failed_loading` + -- and the park does not leak the entry, because that same KV completion + is what releases it. Skipping the arm instead let the KV leg pass through + as `finished_loading`, which `Scheduler._settle_state_load(ok=True)` + miscounts as a state restore that never happened.""" worker = _k3_worker(tier=False) req = _k3_load_req("r1") - worker._arm_joint_loads( - SimpleNamespace(state_loads=[("r1", 99, 0)], requests=[req]) - ) + meta = SimpleNamespace(state_loads=[("r1", 99, 0)], requests=[req]) + worker._arm_joint_loads(meta) + assert worker._joint_park.waits_for(req.load_operation) + + # `_start_state_loads` on the no-tier path fails the state leg. + worker._start_state_loads(meta) + # The KV leg has not landed yet: the pair is still held, not passed. + done, failed = worker._settle_joint(set(), set(), set(), set()) + assert done == set() + assert failed == set() + assert worker._joint_park.waits_for(req.load_operation) + + # KV completion lands -> the pair resolves to failed, and nothing leaks. + done, failed = worker._settle_joint({req.load_operation}, set(), set(), set()) + assert done == set(), "no phantom finished_loading with no tier" + assert failed == {req.load_operation} assert not worker._joint_park.waits_for(req.load_operation) assert worker._joint_park._need == {} + assert worker._joint_park._alias == {} + assert worker._joint_park._alias_of == {} def test_a_single_leg_request_still_passes_straight_through(self): """Nothing armed it, so neither channel may be held back.""" From e9f593ad80fc6e732dd6da4c0b8946ce3b7de19f Mon Sep 17 00:00:00 2001 From: zejchen Date: Mon, 31 Aug 2026 15:06:18 +0000 Subject: [PATCH 053/151] fix(offload/dense): fail closed when a GDN model registers state on the dense path Review finding #3. The dense byte codec's per-request-state branch was changed from a hard ValueError to a silent `continue`. That skip removes the only guard that keeps a GDN / linear-attention model (Qwen3-Next, MiniMax-M3, Qwen3.5) off the plain dense offload path. Those models register their slot-indexed recurrent (mamba) state in the same kv_caches dict the codec walks. The dense path has no rule aligning a restored KV prefix with that recurrent state -- nothing like KimiK3OffloadScheduler's per-request-state load decision exists for it. So a `continue` lets registration succeed, and at load time the KV prefix `[hbm, lmc)` is restored while the linear state stays stale and the forward skips recomputing it: silent wrong output. Pre -PR, those models were kept off the path by the divisibility ValueError the skip now also swallows. Fix: gate the skip behind a keyword-only `permit_per_request_state` flag on DenseKVByteCodec, default False -> restore the fail-fast ValueError. The dense connector passes its class attr `_permit_per_request_state = False`; KimiK3OffloadConnector overrides it to True because it owns a state tier that moves the recurrent state separately, so for it the skip is correct. DSV4 has its own register_kv_caches / codec and is unaffected. tests: add test_dense_codec_rejects_per_request_state_by_default (raises by default, skips + excludes the state segment when permitted). 212 passed. Co-Authored-By: Claude Opus 4.8 --- atom/kv_transfer/offload/dense/connector.py | 13 ++++++- .../offload/dense/kv_byte_codec.py | 33 ++++++++++++++++-- .../offload/hybrid/kimi_k3/connector.py | 5 +++ tests/test_lmcache_offload_connector.py | 34 +++++++++++++++++++ 4 files changed, 82 insertions(+), 3 deletions(-) diff --git a/atom/kv_transfer/offload/dense/connector.py b/atom/kv_transfer/offload/dense/connector.py index 55e3e7fa29..048781b8ab 100644 --- a/atom/kv_transfer/offload/dense/connector.py +++ b/atom/kv_transfer/offload/dense/connector.py @@ -69,6 +69,13 @@ class DenseOffloadConnector(OffloadWorkerMixin, KVConnectorBase): # producer semantic that would wrongly deallocate live offload blocks). # Executor plumbing + get_finished come from OffloadWorkerMixin. + # Whether a per-request recurrent-state tensor in the registered kv_caches is + # tolerated. The plain dense path has no rule keeping a restored KV prefix + # aligned with linear-attention state, so it must reject such a model + # (GDN: Qwen3-Next, MiniMax-M3, Qwen3.5) and fail fast. A hybrid connector + # that owns a state tier (kimi_k3) overrides this to True. + _permit_per_request_state = False + def __init__(self, config) -> None: self._config = config self._init_worker_common(config) # kv_role, executors, lock, tallies @@ -94,7 +101,11 @@ def register_kv_caches( # num_blocks is the physical block count (num_physical_kvcache_blocks), # threaded from the model runner. MLA stores its KV token-major, so the # codec can't infer the block count from tensor.shape[0]; pass it. - self._codec = DenseKVByteCodec(kv_caches, num_blocks=num_blocks) + self._codec = DenseKVByteCodec( + kv_caches, + num_blocks=num_blocks, + permit_per_request_state=self._permit_per_request_state, + ) # Shared opaque-uint8 engine build; the chunked GPU connector needs # cfg.chunk_size, so it's built inside the factory once cfg exists. self._engine, cfg, meta = build_offload_engine( diff --git a/atom/kv_transfer/offload/dense/kv_byte_codec.py b/atom/kv_transfer/offload/dense/kv_byte_codec.py index bf8e6a99c4..b0220fb921 100644 --- a/atom/kv_transfer/offload/dense/kv_byte_codec.py +++ b/atom/kv_transfer/offload/dense/kv_byte_codec.py @@ -40,7 +40,13 @@ class DenseKVByteCodec: """Per-block byte mover between paged GPU KV tensors and flat buffers.""" - def __init__(self, kv_caches: dict, num_blocks: int | None = None) -> None: + def __init__( + self, + kv_caches: dict, + num_blocks: int | None = None, + *, + permit_per_request_state: bool = False, + ) -> None: """``kv_caches``: ordered ``{layer_name: KVCacheTensor}`` from ``register_kv_caches``. We flatten every movable per-layer tensor (K, V, and fp8 scales when present) into one ordered segment list. @@ -61,7 +67,19 @@ def __init__(self, kv_caches: dict, num_blocks: int | None = None) -> None: we take ``num_blocks`` explicitly and derive each segment's per-block byte stride as ``segment_bytes / num_blocks``. ``num_blocks`` falls back to ``segment.shape[0]`` (the block-major assumption) when not supplied, - preserving the original non-MLA behaviour.""" + preserving the original non-MLA behaviour. + + ``permit_per_request_state`` gates whether a per-request recurrent-state + tensor in ``kv_caches`` is skipped or rejected. A hybrid connector that + moves state through its own tier (kimi_k3) sets it True to skip. The + plain dense path leaves it False: a GDN model (Qwen3-Next, MiniMax-M3, + Qwen3.5) registered on ``lmcache_offload`` puts its slot-indexed mamba + caches in this dict, and the dense path has no rule keeping its KV + prefix aligned with that linear state -- restoring KV while the state is + stale is silent wrong output. Skipping the tensor here used to hide that + behind a successful registration; failing closed keeps those models off + the dense path, which is where the pre-PR divisibility ``ValueError`` + left them.""" self._segments: list[torch.Tensor] = [] for kvt in kv_caches.values(): # A hybrid registers its per-request recurrent state in the same @@ -72,6 +90,17 @@ def __init__(self, kv_caches: dict, num_blocks: int | None = None) -> None: # `bytes_per_block` past what `block_regions` describes. The tier # reaches those bytes through `state_entry_views`. if getattr(kvt, "per_request_state", False): + if not permit_per_request_state: + raise ValueError( + "DenseKVByteCodec: per-request recurrent state was " + "registered on the plain dense offload path. This is a " + "GDN/linear-attention model (e.g. Qwen3-Next, " + "MiniMax-M3, Qwen3.5); the dense path would restore its " + "KV prefix while the recurrent state stayed stale -- " + "silent wrong output. Use a connector that owns a state " + "tier (kimi_k3), which passes " + "permit_per_request_state=True." + ) continue for t in ( getattr(kvt, "k_cache", None), diff --git a/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py b/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py index 1d06f6c128..4a847c98be 100644 --- a/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py +++ b/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py @@ -53,6 +53,11 @@ class KimiK3OffloadConnector(DenseOffloadConnector): """Worker side: dense KV, plus spill/load of the per-request state.""" + # This connector owns a state tier and moves the per-request recurrent state + # through it, so the dense codec must SKIP the state tensor rather than + # reject it. The base rejects it (silent-wrong-output guard); we opt in here. + _permit_per_request_state = True + def __init__(self, config) -> None: super().__init__(config) self._state_tier = None diff --git a/tests/test_lmcache_offload_connector.py b/tests/test_lmcache_offload_connector.py index f756986cfd..4ae617b45c 100644 --- a/tests/test_lmcache_offload_connector.py +++ b/tests/test_lmcache_offload_connector.py @@ -899,6 +899,40 @@ def test_lmcache_connector_maps_token_ranges_to_block_ids(): ) +def test_dense_codec_rejects_per_request_state_by_default(): + """A GDN model registered on the plain dense path drops its slot-indexed + recurrent state into kv_caches. The dense path cannot keep a restored KV + prefix aligned with that state, so the codec must fail closed rather than + skip it (which would register successfully and produce silent wrong + output). kimi_k3 opts in via permit_per_request_state=True to skip it, + because it owns a state tier that moves the state separately.""" + import torch + + if not hasattr(torch, "arange"): + pytest.skip("real torch is unavailable") + + kv_caches = { + "l0": SimpleNamespace( + k_cache=torch.arange(6 * 2, dtype=torch.uint8).reshape(6, 2), + v_cache=torch.arange(6 * 2, dtype=torch.uint8).reshape(6, 2), + k_scale=None, + v_scale=None, + ), + "state": SimpleNamespace( + per_request_state=True, + state=torch.arange(4 * 5, dtype=torch.uint8).reshape(4, 5), + ), + } + + with pytest.raises(ValueError, match="per-request recurrent state"): + DenseKVByteCodec(kv_caches) + + # Opt in (kimi_k3): the state tensor is skipped, only the KV layer is moved. + codec = DenseKVByteCodec(kv_caches, permit_per_request_state=True) + assert len(codec._segments) == 2 # k_cache + v_cache, state excluded + assert codec.num_blocks == 6 + + def test_lmcache_connector_maps_dcp_ranges_on_virtual_block_grid(): import torch From 77ebcf3b0d8c13b4ecb489698d6e8644bf78ca3d Mon Sep 17 00:00:00 2001 From: zejchen Date: Mon, 31 Aug 2026 15:12:21 +0000 Subject: [PATCH 054/151] fix(offload/dense): round-robin the save scan so bounded slots are shared fairly Review finding #8-4h. DenseOffloadScheduler.build_connector_meta scans _save_tracker in fixed insertion order and breaks as soon as _may_emit_save says the outstanding-save budget is full. The base _may_emit_save is unbounded, but KimiK3OffloadScheduler overrides it to cap saves at OFFLOAD_MAX_PENDING_SAVES (default max(2, 2*save_workers)). Under that cap the scan restarts at the same head every step. A long, multi-chunk request at the head produces a fresh chunk each step and re-wins the freed slot every time, so later requests in the tracker never get a save dispatched -- and their finished blocks stay pinned by should_defer_free until a save drains, which for them never happens. Starvation, worse the more concurrent prefills share the cap. Fix: keep a round-robin cursor `_save_rr_last` (the last sid that emitted a save) and resume the scan just after it, wrapping around. When the budget is unbounded (dense/dsv4 base) every eligible save still emits in the same step, so behaviour there is unchanged; only the bounded kimi_k3 path changes, and it now serves requests in rotation. tests: add test_bounded_saves_are_shared_round_robin_not_head_first (cap=1, three mid-prefill requests -> emits [100,101,102] in rotation, not [100,100,100]). 213 passed. Co-Authored-By: Claude Opus 4.8 --- atom/kv_transfer/offload/dense/connector.py | 18 ++++++- tests/test_lmcache_offload_connector.py | 54 +++++++++++++++++++++ 2 files changed, 71 insertions(+), 1 deletion(-) diff --git a/atom/kv_transfer/offload/dense/connector.py b/atom/kv_transfer/offload/dense/connector.py index 048781b8ab..c35b1ab542 100644 --- a/atom/kv_transfer/offload/dense/connector.py +++ b/atom/kv_transfer/offload/dense/connector.py @@ -357,6 +357,13 @@ def __init__(self, config) -> None: # prefix is stored to LMCache once prefill computes it # (seq.prefix_hashes_published flips True), chunk by chunk. self._save_tracker: dict[str, list] = {} + # Round-robin cursor over `_save_tracker`: the last sid that emitted a + # save. `build_connector_meta` resumes the save scan just after it so a + # bounded save queue (`_may_emit_save`) is shared fairly. Without it the + # scan always restarts at the insertion-ordered head, and a long + # multi-chunk request there re-wins the freed slot every step and + # starves later requests (their blocks stay pinned by should_defer_free). + self._save_rr_last: str | None = None # sid -> exact save generation. Exact matching prevents a delayed TP # notification for an older request lifecycle from releasing the # current request's deferred blocks. @@ -634,7 +641,15 @@ def build_connector_meta(self) -> LMCacheOffloadMetadata: # chunked prefill, seq.num_cached_tokens advances after each prefill # chunk's forward has completed; use it as the D2H-safe frontier. chunk = self.chunk_size or 256 - for sid, entry in self._save_tracker.items(): + # Round-robin start: resume just after the last sid we emitted a save + # for, so a bounded `_may_emit_save` queue is shared fairly instead of + # always favouring the insertion-ordered head (see `_save_rr_last`). + tracker_sids = list(self._save_tracker.keys()) + if tracker_sids and self._save_rr_last in self._save_tracker: + start = (tracker_sids.index(self._save_rr_last) + 1) % len(tracker_sids) + tracker_sids = tracker_sids[start:] + tracker_sids[:start] + for sid in tracker_sids: + entry = self._save_tracker[sid] if not self._do_save: continue if not self._may_emit_save(): @@ -675,6 +690,7 @@ def build_connector_meta(self) -> LMCacheOffloadMetadata: ) entry[1] = aligned self._save_inflight[sid] = save_operation + self._save_rr_last = sid dispatched = set(meta.lookup_requests_in_step) self._lookup_in_step = [ sid for sid in self._lookup_in_step if sid not in dispatched diff --git a/tests/test_lmcache_offload_connector.py b/tests/test_lmcache_offload_connector.py index 4ae617b45c..e6278bb9d8 100644 --- a/tests/test_lmcache_offload_connector.py +++ b/tests/test_lmcache_offload_connector.py @@ -4680,6 +4680,7 @@ def _k3_scheduler() -> KimiK3OffloadScheduler: s.chunk_size = 256 s._save_inflight = {} s._save_tracker = {} + s._save_rr_last = None s._pending_state_loads = [] s._pending_state_stores = [] s._save_inflight_since = {} @@ -4705,6 +4706,59 @@ def _k3_seq(*, hbm: int, joint: int = 0, kv: int = 0, claimed: int = 0): ) +def test_bounded_saves_are_shared_round_robin_not_head_first(): + """With a bounded ``_may_emit_save`` (kimi_k3 caps outstanding saves), the + save scan must rotate over ``_save_tracker`` so a long multi-chunk request + at the insertion-ordered head cannot re-win the freed slot every step and + starve later requests -- whose blocks stay pinned by ``should_defer_free`` + until their save drains. Without the round-robin cursor the emission order + would be [100, 100, 100]; with it, each request gets a turn.""" + + class _Cap1(DenseOffloadScheduler): + def _may_emit_save(self): # one save outstanding at a time + return len(self._save_inflight) < 1 + + def _track_save_statistics(self, *a, **k): + pass + + s = _Cap1.__new__(_Cap1) + s.chunk_size = 256 + s.block_size = 256 + s._do_save = True + s._do_load = False + s._reqs_need_recv = {} + s._lookup_in_step = [] + s._save_tracker = {} + s._save_inflight = {} + s._save_nonce = 0 + s._save_rr_last = None + + seqs = {} + for i in range(3): + sid = str(100 + i) + seq = SimpleNamespace( + id=100 + i, + num_prompt_tokens=4096, + token_ids=list(range(4096)), + num_cached_tokens=0, + block_table=list(range(16)), + ) + s._save_tracker[sid] = [seq, 0] + seqs[sid] = seq + + emitted = [] + for _ in range(3): + for seq in seqs.values(): # every request computes one more chunk + seq.num_cached_tokens += 256 + meta = s.build_connector_meta() + saves = [r for r in meta.requests if r.save_spec is not None] + assert len(saves) == 1 # the cap admits exactly one per step + emitted.append(str(saves[0].req_id)) + s._save_inflight.clear() # that save completes, freeing the one slot + + assert emitted == ["100", "101", "102"] + + def test_layout_selection_picks_kimi_k3_for_a_kda_config(): """K3's paged KV is ordinary dense MLA, so the layout cannot be read off the KV shape -- the KDA state is what makes it its own family.""" From 2889a2d5aef59249afe0b90bcecabf139f909718 Mon Sep 17 00:00:00 2001 From: zejchen Date: Mon, 31 Aug 2026 15:22:16 +0000 Subject: [PATCH 055/151] [LMCache] #2: privatise claimed prefix on disown to stop shared-block decode corruption Finding #2 (review 5479206239). A widened joint claim shares the matched prefix by `BlockPool.claim` (ref_count += 1, not a copy), so several live sequences point their block_table at the same canonical blocks. When a boundary later turns out to have no state behind it, the request is "disowned": num_cached_tokens is set to 0 so the forward recomputes from token 0 -- but the block_table still points at those shared canonical blocks. The recompute writes KV in place into blocks another sequence is actively decoding out of, tearing its values. This is non-reproducible, grows with N and concurrency, and is the main suspect behind the pass2 precision regression (offload-specific, byte-level codecs already cleared). Four disown sites only zeroed num_cached_tokens and never re-pointed the block_table: - block_manager.allocate (state-less joint boundary) - block_manager.cancel_state_load (withdrawn load, same boundary) - scheduler _schedule_prefills (needs-remote-load / joint-disown) - scheduler _consume_failed_remote_kv (offload-resume reuses the table) Fix: add BlockManager.disown_claimed_prefix(seq, keep_blocks=0). For each claimed block above keep_blocks that is still shared (hash != -1), kv.free it and drop a fresh private _fresh_block into the same slot -- table length and every other mapping unchanged, the already-private fresh tail (hash==-1) untouched. Only ref_count>1 blocks cost a net PAGE unit (free releases nothing while a peer holds them, and can_allocate never reserved them), so reserve n_shared units up front via _ensure_page_units; return False if the pool cannot back them. Thread that False through: allocate and cancel_state_load now return bool, and all callers (both allocate sites, both cancel_state_load sites, and _consume_failed_remote_kv) deallocate + requeue the seq for a clean recompute rather than run a forward over shared blocks. keep_blocks is 0 at every site because hybrid disown forces recompute from 0 (has_initial_state = num_cached_tokens > 0). Tests: TestDisownClaimedPrefix in test_block_manager.py -- privatises shared blocks in place (length/tail unchanged, slots now hash==-1 and distinct from the peer's, peer back to ref_count==1), no leak after deallocate, refuses (returns False, table untouched) when the pool cannot back the copies, and is a no-op without prefix caching. Full suite green (356 passed). --- atom/model_engine/block_manager.py | 76 +++++++++++++++++++++++++--- atom/model_engine/scheduler.py | 45 +++++++++++++++-- tests/test_block_manager.py | 81 ++++++++++++++++++++++++++++++ 3 files changed, 190 insertions(+), 12 deletions(-) diff --git a/atom/model_engine/block_manager.py b/atom/model_engine/block_manager.py index b1c4eb811d..9a09f73b26 100644 --- a/atom/model_engine/block_manager.py +++ b/atom/model_engine/block_manager.py @@ -439,6 +439,48 @@ def _fresh_block(self) -> int: self.kv.allocate(block_id) return block_id + def disown_claimed_prefix(self, seq: Sequence, keep_blocks: int = 0) -> bool: + """Make this seq's claimed canonical prefix private, in place. + + A disown sets ``num_cached_tokens = 0`` so the forward recomputes from + token 0, but ``block_table`` still points at the hash-indexed blocks + this seq took with ``kv.claim`` -- blocks another sequence may be + decoding out of. Recomputing writes KV in place into those shared + blocks and tears the other holder's values (non-reproducible + corruption -- this is review finding #2). This hands each claimed + prefix block in ``block_table[keep_blocks:]`` back (``kv.free``) and + drops a fresh private block (``_fresh_block``) into the same slot, so + the table length and every other position's mapping are unchanged. The + fresh tail (``hash == -1``) is already private and is left untouched; + so is a still-shared block below ``keep_blocks``. + + Returns ``False`` when the pool cannot back the private copies -- the + caller must then abandon this admission (deallocate + requeue) rather + than run a forward over shared blocks. Only blocks still shared at + ``ref_count > 1`` cost a net PAGE unit here (``free`` releases nothing + while another holder remains), and ``can_allocate`` never reserved them + (it subtracts already-used hit blocks from ``num_new_blocks``); those + are reserved up front. ``ref_count == 1`` blocks recycle their own unit + and never fail. + """ + if not self.enable_prefix_caching: + return True + table = seq.block_table + n_shared = sum( + 1 + for i in range(keep_blocks, len(table)) + if self.kv.block(table[i]).hash != -1 + and self.kv.block(table[i]).ref_count > 1 + ) + if n_shared and not self._ensure_page_units(n_shared): + return False + for i in range(keep_blocks, len(table)): + if self.kv.block(table[i]).hash == -1: + continue # already-private fresh tail (or a prior disown) + self.kv.free(table[i]) + table[i] = self._fresh_block() + return True + def _checkpoint_has_room( self, live_blocks: int = 0, protected_hash: int | None = None ) -> bool: @@ -832,7 +874,7 @@ def can_allocate(self, seq: Sequence) -> int: self._extend_hash_chain(seq, block_hashes) return num_cached_blocks - def allocate(self, seq: Sequence, num_cached_blocks: int = 0): + def allocate(self, seq: Sequence, num_cached_blocks: int = 0) -> bool: """Allocate blocks for `seq`. `num_cached_blocks` is the hit count returned by `can_allocate` (0 if caller didn't call it). @@ -841,6 +883,11 @@ def allocate(self, seq: Sequence, num_cached_blocks: int = 0): KV. This keeps the manager correct under future chunked-prefill scheduling: a block spanning multiple steps must not be published as a hash until fully filled. + + Returns ``True`` on success. Returns ``False`` only when a state-less + joint boundary had to be disowned (finding #2) and the pool could not + back the private prefix copies; the caller must then deallocate and + requeue the seq rather than run a forward over shared blocks. """ assert not seq.block_table # Two extents, and they are not the same number. `num_cached_blocks` is @@ -963,12 +1010,19 @@ def allocate(self, seq: Sequence, num_cached_blocks: int = 0): state_holds = False if not state_holds: # No state behind the boundary means it is not this request's - # history. Disown it -- blocks stay claimed and the forward - # recomputes. Keeping it is silent wrong output, since - # `has_initial_state` (`gdn_attn.py`) is `num_cached_tokens > 0`. + # history. Disown it -- the forward recomputes from 0. Privatise the + # claimed prefix first (finding #2): recomputing over the shared + # canonical blocks would tear another sequence's decode. Keeping the + # boundary is silent wrong output too, since `has_initial_state` + # (`gdn_attn.py`) is `num_cached_tokens > 0`. + if not self.disown_claimed_prefix(seq): + # Pool cannot back the private copies; signal the caller to + # deallocate and requeue for a clean recompute next pass. + return False seq.num_cached_tokens = 0 seq.state_joint_boundary_tokens = 0 seq.state_joint_boundary_hash = -1 + return True def _state_leg_secured(self, seq: Sequence) -> bool: """Whether something will really put state behind this seq's boundary. @@ -1131,19 +1185,27 @@ def _request_state_load(self, seq: Sequence, hit_hash: int) -> bool: self._state_loads.append((seq.id, hit_hash, seq.state_slot)) return True - def cancel_state_load(self, seq: Sequence) -> None: + def cancel_state_load(self, seq: Sequence) -> bool: """Withdraw a load requested this pass, before anything was issued. Only legal before `take_state_loads` handed it over; afterwards the bytes are on their way and the slot must be held. The boundary is - disowned exactly as `allocate` would have. + disowned exactly as `allocate` would have -- including privatising the + claimed prefix (finding #2) so the recompute cannot tear a shared + decode. + + Returns ``False`` when the pool cannot back the private copies; the + caller must abandon this admission rather than resume over the shared + prefix. ``True`` when there was nothing to cancel or the disown held. """ if seq.state_load_hash == -1: - return + return True self._state_loads = [e for e in self._state_loads if e[0] != seq.id] if self.state_offload is not None: self.state_offload.abandon_load(seq.id) seq.state_load_hash = -1 + disowned = self.disown_claimed_prefix(seq) seq.num_cached_tokens = 0 + return disowned def abandon_state_load(self, req_id) -> None: """Give up on a load without blaming the bytes for it. diff --git a/atom/model_engine/scheduler.py b/atom/model_engine/scheduler.py index d16ebe383c..2017776fb9 100644 --- a/atom/model_engine/scheduler.py +++ b/atom/model_engine/scheduler.py @@ -1857,7 +1857,14 @@ def schedule(self) -> tuple[ScheduledBatch, dict[int, Sequence]]: if chunk is None or (atomic_prefill and chunk < num_new_tokens): self.waiting.appendleft(seq) break - self.block_manager.allocate(seq, num_cached_blocks) + if not self.block_manager.allocate(seq, num_cached_blocks): + # A state-less joint boundary could not be privatised without + # risking a shared decoding sequence's blocks (finding #2). + # Return the seq to waiting for a clean recompute once the pool + # has room, and stop admitting this pass. + self.block_manager.deallocate(seq) + self.waiting.appendleft(seq) + break # The hit the request kept, not the one `can_allocate` offered: # `allocate` may disown the boundary, and `num_cached_blocks` is the @@ -1888,7 +1895,13 @@ def schedule(self) -> tuple[ScheduledBatch, dict[int, Sequence]]: "load.", seq.id, ) - self.block_manager.cancel_state_load(seq) + if not self.block_manager.cancel_state_load(seq): + # Disown could not be backed (finding #2); requeue for a + # clean recompute instead of parking a load into shared + # blocks. + self.block_manager.deallocate(seq) + self.waiting.appendleft(seq) + break self._park_for_remote_load(seq, skipped_waiting_requests) continue @@ -1903,8 +1916,17 @@ def schedule(self) -> tuple[ScheduledBatch, dict[int, Sequence]]: seq.state_joint_boundary_tokens, seq.num_cached_tokens, ) + # Privatise the claimed prefix (finding #2) so recompute-from-0 + # cannot tear a shared decode. cancel_state_load does it when a + # CPU state load was pending; otherwise disown directly. if seq.state_load_hash != -1: - self.block_manager.cancel_state_load(seq) + disowned = self.block_manager.cancel_state_load(seq) + else: + disowned = self.block_manager.disown_claimed_prefix(seq) + if not disowned: + self.block_manager.deallocate(seq) + self.waiting.appendleft(seq) + break seq.num_cached_tokens = 0 seq.state_joint_boundary_tokens = 0 seq.state_joint_boundary_hash = -1 @@ -2252,7 +2274,11 @@ def _consume_failed_remote_kv(self, seq: Sequence) -> bool: # admission, otherwise the resume runs with `num_cached_tokens > 0` # over a group nobody filled -- `has_initial_state` reads that as # "the recurrence continues". Before `offload_loaded_tokens` is set - # below, so that field records the disowned figure. + # below, so that field records the disowned figure. Privatise the + # claimed prefix (finding #2): the offload resume path reuses this + # exact block_table without re-allocating, so recompute-from-0 would + # otherwise tear a shared decode. + disowned = self.block_manager.disown_claimed_prefix(seq) seq.num_cached_tokens = 0 seq.state_load_hash = -1 # A joint boundary that failed on either leg is not this request's @@ -2260,6 +2286,11 @@ def _consume_failed_remote_kv(self, seq: Sequence) -> bool: # would let the connector clamp a later lookup to it. seq.state_joint_boundary_tokens = 0 seq.state_joint_boundary_hash = -1 + if not disowned: + # Cannot back private copies; drop the whole table so the + # re-admission rebuilds a clean one via can_allocate/allocate + # (empty block_table disqualifies the offload-resume shortcut). + self.block_manager.deallocate(seq) seq.offload_loaded = False seq.offload_loaded_tokens = seq.num_cached_tokens seq.offload_load_start_tokens = None @@ -3872,7 +3903,11 @@ def allocate_waiting(self) -> list[Sequence]: if self.block_manager.can_allocate(seq) < 0: logger.warning("Cannot allocate prefill") break - self.block_manager.allocate(seq) + if not self.block_manager.allocate(seq): + # Disown could not be backed (finding #2); leave the seq at + # the front of waiting to retry once the pool has room. + self.block_manager.deallocate(seq) + break self.waiting.popleft() self.prefill_waiting[seq.id] = seq diff --git a/tests/test_block_manager.py b/tests/test_block_manager.py index 69d5282da1..d9d392e5e7 100644 --- a/tests/test_block_manager.py +++ b/tests/test_block_manager.py @@ -118,6 +118,87 @@ def test_the_widened_claim_still_gives_every_position_a_block(self, seq_factory) assert len(set(seq.block_table)) == 5 +class TestDisownClaimedPrefix: + """A disown recomputes from token 0 while the block table still points at + hash-indexed blocks a *live* peer is decoding out of. Recomputing writes KV + in place into those shared blocks and tears the peer's values (review + finding #2 -- the precision-corruption main suspect). `disown_claimed_prefix` + hands each shared claimed block back and drops a fresh private block into the + same slot, so the forward can only overwrite blocks this seq owns alone. + """ + + def _shared_prefix(self, seq_factory, num_kvcache_blocks=32): + """Two live seqs sharing a 4-block claimed prefix (ref_count == 2).""" + cfg = MockConfig( + num_kvcache_blocks=num_kvcache_blocks, + kv_cache_block_size=4, + enable_prefix_caching=True, + ) + bm = BlockManager(cfg) + tokens = list(range(20)) + holder = seq_factory(tokens) + bm.allocate(holder, 0) + bm.hash_blocks(holder, len(tokens)) + # A second live seq claims the same 4 resident blocks via the joint + # boundary -- now every prefix block is held at ref_count == 2. + seq = seq_factory(tokens) + seq.state_joint_claim_tokens = 16 + bm.allocate(seq, 2) + shared = list(seq.block_table[:4]) + assert all(bm.kv.block(b).ref_count == 2 for b in shared) + return bm, holder, seq, shared + + def test_disown_privatizes_shared_blocks_in_place(self, seq_factory): + bm, holder, seq, shared = self._shared_prefix(seq_factory) + table_before = list(seq.block_table) + + assert bm.disown_claimed_prefix(seq) is True + + # Same length, same tail slot -- only the shared prefix slots changed. + assert len(seq.block_table) == len(table_before) + assert seq.block_table[4] == table_before[4] + # The four claimed slots now hold fresh private blocks: none is one of + # the shared canonical blocks, and each is hash == -1 (private). + for i in range(4): + assert seq.block_table[i] not in shared + assert bm.kv.block(seq.block_table[i]).hash == -1 + # The peer still holds every shared block, now back at ref_count == 1. + assert list(holder.block_table[:4]) == shared + for b in shared: + assert bm.kv.block(b).ref_count == 1 + + def test_disown_leaves_no_leak_after_deallocate(self, seq_factory): + bm, holder, seq, _ = self._shared_prefix(seq_factory) + bm.disown_claimed_prefix(seq) + bm.deallocate(seq) + bm.deallocate(holder) + assert bm.kv.num_used == 0 + + def test_disown_fails_when_pool_cannot_back_private_copies(self, seq_factory): + # 6 blocks: holder takes 4, seq's fresh tail takes 1, leaving 1 free -- + # fewer than the 4 private copies the disown needs, so it must refuse + # rather than silently reuse the shared blocks. + bm, holder, seq, shared = self._shared_prefix(seq_factory, num_kvcache_blocks=6) + assert bm.kv.has_free(4) is False + + assert bm.disown_claimed_prefix(seq) is False + + # Refusal is total: the table is untouched and the peer's blocks are + # still shared, so the caller can safely deallocate + requeue. + assert list(seq.block_table[:4]) == shared + for b in shared: + assert bm.kv.block(b).ref_count == 2 + + def test_disown_is_a_noop_without_prefix_caching(self, seq_factory): + cfg = MockConfig(num_kvcache_blocks=8, kv_cache_block_size=4) + bm = BlockManager(cfg) + seq = seq_factory(list(range(16))) + bm.allocate(seq) + table_before = list(seq.block_table) + assert bm.disown_claimed_prefix(seq) is True + assert list(seq.block_table) == table_before + + # ── allocate / deallocate ────────────────────────────────────────────────── From 2ac39f4db9936b029ed22f8e42493e6309bb1abe Mon Sep 17 00:00:00 2001 From: zejchen Date: Mon, 31 Aug 2026 15:34:42 +0000 Subject: [PATCH 056/151] #8-4e: clamp num_offload_tokens into the CacheStats reuse chain CacheStats.update() clamps the ordered cache-hit counters (cached <= wanted <= compressed <= reusable <= full) inside the `if not ordered:` block, but num_offload_tokens is assigned afterwards and never bounded. The class docstring (line 463) states the invariant `cached + offload <= reusable`: offload reuse and HBM-cache reuse partition the same served-reuse pool, so their sum cannot exceed it. When a connector reports more offload tokens than the residual (reusable - cached) -- e.g. a stale lookup or an over-counted load -- lmcache_hit_rate = offload/reusable and the combined hit_rate can exceed 100%, corrupting operator-facing dashboards. Clamp num_offload_tokens to [0, num_reusable_tokens - num_cached_tokens] so it always fits the residual reuse budget the docstring promises. --- atom/model_engine/scheduler.py | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/atom/model_engine/scheduler.py b/atom/model_engine/scheduler.py index 2017776fb9..bb436e6123 100644 --- a/atom/model_engine/scheduler.py +++ b/atom/model_engine/scheduler.py @@ -379,6 +379,16 @@ def update( self._interval_reusable_tokens += num_reusable_tokens self._interval_compressed_tokens += num_compressed_tokens self._interval_wanted_tokens += num_wanted_tokens + # `num_offload_tokens` is outside the `cached <= ... <= reusable` chain, + # so the ordering clamp above never bounded it -- but the rates pair it + # with `cached` against the same `reusable` denominator (`hit_rate` and + # `lmcache_hit_rate` are meant to partition served reuse), and the + # documented invariant is `cached + offload <= reusable`. A CPU-tier + # resume can report more offload reuse than the HBM walk left room for; + # clamp so the combined rate cannot exceed 100%. + num_offload_tokens = max( + 0, min(num_offload_tokens, num_reusable_tokens - num_cached_tokens) + ) self.total_offload_tokens += num_offload_tokens self._interval_offload_tokens += num_offload_tokens From 3f4df3195c2d3eb06f0bec6597f52c4b1bc4f2bb Mon Sep 17 00:00:00 2001 From: zejchen Date: Mon, 31 Aug 2026 15:34:51 +0000 Subject: [PATCH 057/151] #8-4a: honor the per-connector max_pending_saves cap in the scheduler _publish_state_stores throttled the publish batch with the scheduler's own _offload_max_pending_saves() (scheduler.py:71), which only reads the OFFLOAD_MAX_PENDING_SAVES env default "2". That ignores both the canonical default max(2, 2*save_workers) computed in _offload_common.max_pending_saves and any per-connector kv_connector_extra_config["max_pending_saves"] override the connector already stored on self._max_pending_saves. The result is a third, inconsistent reader of the same knob: with more save workers the connector self-throttles at 2*save_workers while the scheduler publishes only 2 per drain, needlessly serializing state stores. Add _state_store_pending_cap() that prefers the live connector's _max_pending_saves (through _impl for wrapped connectors), and falls back to _offload_max_pending_saves() only when no connector value is available. Route the _publish_state_stores take() through it so the publish cap tracks the same value the connector enforces. --- atom/model_engine/scheduler.py | 33 +++++++++++++++++++++++++++------ 1 file changed, 27 insertions(+), 6 deletions(-) diff --git a/atom/model_engine/scheduler.py b/atom/model_engine/scheduler.py index bb436e6123..340bf70f5c 100644 --- a/atom/model_engine/scheduler.py +++ b/atom/model_engine/scheduler.py @@ -2631,15 +2631,36 @@ def _settle_state_load(self, req_id, ok: bool) -> None: if settle is not None: settle(req_id, ok=ok) + def _state_store_pending_cap(self) -> int: + """The running-plus-queued cap for state-tier stores. + + Read off the connector's own `_max_pending_saves` -- the canonical + `_offload_common.max_pending_saves` value it computed from + `kv_connector_extra_config` and `OFFLOAD_COPY_WORKERS`, the exact bound + the KV leg's `_may_emit_save` enforces. Sharing that number rather than + re-reading the bare `OFFLOAD_MAX_PENDING_SAVES` default of 2 keeps the + two legs from pinning different amounts of the same pool, and honours a + per-connector `"max_pending_saves"` override the env reader never sees. + Falls back to the env reader for a connector (or test double) that never + computed one, and through the delegating shell's `_impl`. + """ + conn = self.kv_connector + cap = getattr(conn, "_max_pending_saves", None) + if cap is None: + cap = getattr(getattr(conn, "_impl", None), "_max_pending_saves", None) + if isinstance(cap, int) and not isinstance(cap, bool) and cap > 0: + return cap + return _offload_max_pending_saves() + def _publish_state_stores(self) -> None: """Hand this pass's ready checkpoints to the connector for the CPU tier. Beside `_publish_state_loads` and on the same schedule, so one pass - makes one decision about the tier. The cap is the same - `OFFLOAD_MAX_PENDING_SAVES` the KV leg uses and for the same reason: - each outstanding transfer holds the bytes it is reading out of the - pool, so the queue depth is also how much of the pool a slow backend - can pin. + makes one decision about the tier. The cap is the connector's own + `_max_pending_saves` (see `_state_store_pending_cap`) -- the same bound + the KV leg's `_may_emit_save` enforces, and for the same reason: each + outstanding transfer holds the bytes it is reading out of the pool, so + the queue depth is also how much of the pool a slow backend can pin. `take_state_stores` pins as it hands over, so anything the connector will not carry has to be settled here rather than left to the @@ -2650,7 +2671,7 @@ def _publish_state_stores(self) -> None: take = getattr(getattr(self, "block_manager", None), "take_state_stores", None) if take is None: return - stores = take(_offload_max_pending_saves()) + stores = take(self._state_store_pending_cap()) if not stores: return accepted = False From b039bc027f284a946c73920188c777ab6d046fd7 Mon Sep 17 00:00:00 2001 From: zejchen Date: Mon, 31 Aug 2026 15:34:58 +0000 Subject: [PATCH 058/151] #8-4f: stop double-counting refused state stores as failures _publish_state_stores bumps state_offload.stores_refused by the whole batch length when a refusal short-circuits publishing, then still walks each op calling settle_state_store(op, ok=False). settle_state_store's else-branch bumps state_offload.stores_failed += 1 per op, so every refused store lands in BOTH counters -- refused and failed -- inflating the failure rate by exactly the refused count. A refusal is not an attempted-then-failed store: the unit must be released, but the op never reached the offload path. Add an attempted=True parameter to settle_state_store and gate the stores_failed bump behind `elif attempted:`; the refusal caller passes attempted=False so it releases the reserved unit without recording a phantom failure. --- atom/model_engine/block_manager.py | 10 ++++++++-- atom/model_engine/scheduler.py | 5 ++++- 2 files changed, 12 insertions(+), 3 deletions(-) diff --git a/atom/model_engine/block_manager.py b/atom/model_engine/block_manager.py index 9a09f73b26..b310caa22c 100644 --- a/atom/model_engine/block_manager.py +++ b/atom/model_engine/block_manager.py @@ -1260,7 +1260,7 @@ def take_state_stores(self, max_inflight: int) -> list[tuple]: self.state_offload.stores_attempted += len(out) return out - def settle_state_store(self, op, ok: bool) -> None: + def settle_state_store(self, op, ok: bool, attempted: bool = True) -> None: """One store reported. Release the units, and index the hash if it landed. Success and failure release the pin identically -- it existed to keep @@ -1268,6 +1268,12 @@ def settle_state_store(self, op, ok: bool) -> None: Only the indexing differs, because only a hash whose bytes are really there may be voted for. + `attempted` is `False` for a store a connector *refused* to carry: the + pin still has to be released here, but the copy never reached a worker, + so it is not a `stores_failed` -- the caller counts the refusal once as + `stores_refused`, and bumping `stores_failed` too would double-count the + same event under two names. + `op` is a `StateStoreOperationId`: the pin is released for that exact generation, so a late report from a superseded attempt settles nothing. The index is keyed by hash, because what a resume asks is whether the @@ -1300,7 +1306,7 @@ def settle_state_store(self, op, ok: bool) -> None: if ok: self.state_offload.stores_completed += 1 self.state_offload.note_stored(int(op.prefix_hash)) - else: + elif attempted: self.state_offload.stores_failed += 1 def release_state_store_source(self, op) -> None: diff --git a/atom/model_engine/scheduler.py b/atom/model_engine/scheduler.py index 340bf70f5c..8f24da9e56 100644 --- a/atom/model_engine/scheduler.py +++ b/atom/model_engine/scheduler.py @@ -2694,7 +2694,10 @@ def _publish_state_stores(self) -> None: len(stores), ) for op, _units in stores: - self.block_manager.settle_state_store(op, ok=False) + # `attempted=False`: this refusal is already counted once above as + # `stores_refused`; the units must still be released, but the store + # never reached a worker, so it is not a `stores_failed`. + self.block_manager.settle_state_store(op, ok=False, attempted=False) def _publish_state_loads(self) -> None: """Hand this pass's state-tier loads to the connector, before it builds From 29ba254a900d89b46a3d7330d1549a6bfe201167 Mon Sep 17 00:00:00 2001 From: zejchen Date: Mon, 31 Aug 2026 15:35:06 +0000 Subject: [PATCH 059/151] #7: make can_allocate a side-effect-free probe of the joint boundary can_allocate() runs _joint_kv_boundary() purely to test whether a sequence fits, but that helper bumps the checkpoint funnel counters (joint_boundaries, state_hbm, state_tier, joint_skips) on every call. The scheduler probes can_allocate() speculatively (scheduler.py:1266 schedules-ahead without committing), so each probe pollutes the funnel that checkpoint_funnel exports to operators -- the counts no longer correspond to real joint-claim decisions. Thread a record flag through _joint_kv_boundary()/_no_joint() (default True, so allocate() and every existing caller keep recording). Guard the four counter bumps behind `if record:`. can_allocate() gains a record parameter it forwards, and the speculative probe call passes record=False so only the committing allocate() path advances the funnel. --- atom/model_engine/block_manager.py | 60 +++++++++++++++++++----------- atom/model_engine/scheduler.py | 6 ++- 2 files changed, 43 insertions(+), 23 deletions(-) diff --git a/atom/model_engine/block_manager.py b/atom/model_engine/block_manager.py index b310caa22c..7794c5241d 100644 --- a/atom/model_engine/block_manager.py +++ b/atom/model_engine/block_manager.py @@ -653,9 +653,18 @@ def _joint_kv_boundary( seq: Sequence, hbm_boundary: int, block_hashes: list[int], + record: bool = True, ) -> int: """Boundary above the HBM hit that BOTH legs can reach, 0 if none. + `record` is `False` for a pure fit probe (`can_allocate` asked only + "does this seq fit?", not admitted): the seq's joint fields are still + set -- a later real `can_allocate` overwrites them anyway -- but the + `joint_boundaries` / `state_hbm` / `state_tier` / `joint_skips` funnel + counters are left alone, so a probe that never admits anything does not + inflate the operator-visible checkpoint funnel. (The demand counters are + already per-seq idempotent and need no such guard.) + `can_allocate` walks the HBM prefix cache and nothing else, so an evicted prefix stops at the first miss even when LMCache holds every block. This looks past that: the KV leg fetches `[hbm, B)` from LMCache @@ -673,7 +682,7 @@ def _joint_kv_boundary( seq.state_joint_kv_tokens = 0 seq.state_joint_claim_tokens = 0 if self.state_offload is None: - return self._no_joint("off") + return self._no_joint("off", record) # PAGE is now the served class, not the excluded one. This used to # read `is not None` and refuse -- correct while a K3 checkpoint was an # Active Slot the tier spilled out of `StateSlotPool`. #2045 moved the @@ -683,7 +692,7 @@ def _joint_kv_boundary( # longer outlive its KV in HBM -- so when LMCache hands the KV back, # nothing hands the state back unless the two are fetched together. if not seq.has_per_req_cache or self.paged_state_checkpoints is None: - return self._no_joint("no_paged_checkpoints") + return self._no_joint("no_paged_checkpoints", record) hbs = self._hash_block_size() # From the connector's config, not the connector object: the scheduler # holds whatever `get_kvconnector` returned, and one without @@ -693,34 +702,34 @@ def _joint_kv_boundary( ) lmc_tokens = int(getattr(seq, "offload_kv_prefix_tokens", 0) or 0) if chunk <= 0: - return self._no_joint("no_chunk_size") + return self._no_joint("no_chunk_size", record) # Floored to the grid everything below compares against, so the # covering-chunk check becomes true by construction rather than by # luck. (`get_num_new_matched_tokens` withholds one token on a # full-prompt hit, which takes the lookup off the grid.) lmc_tokens = (lmc_tokens // chunk) * chunk if lmc_tokens <= hbm_boundary * hbs: - return self._no_joint("lmcache_within_hbm") + return self._no_joint("lmcache_within_hbm", record) # The KV leg moves whole chunks and the blocks below the HBM prefix are # shared, so an unaligned start cannot be rounded down. if (hbm_boundary * hbs) % chunk != 0: - return self._no_joint("hbm_off_chunk_grid") + return self._no_joint("hbm_off_chunk_grid", record) cap = min(lmc_tokens // hbs, self._n_hash_blocks(seq) - 1) if cap <= hbm_boundary: - return self._no_joint("no_room_above_hbm") + return self._no_joint("no_room_above_hbm", record) chain = self._chain_to(seq, block_hashes, cap) # One scan: `_gated_hit` already returns the rightmost rung # `_resumable_from` accepts, so a decrement-and-rescan walk would spend # a fixpoint pass per rung to reach the same answer. candidate = self._gated_hit(seq, cap, chain) if candidate <= hbm_boundary: - return self._no_joint("no_rung_above_hbm") + return self._no_joint("no_rung_above_hbm", record) tokens = candidate * hbs h = chain[candidate - 1] # Cannot fail given the two premises above; kept as their assertion. kv_tokens = -(-tokens // chunk) * chunk if kv_tokens > lmc_tokens: - return self._no_joint("covering_chunk_beyond_lookup") + return self._no_joint("covering_chunk_beyond_lookup", record) seq.state_joint_boundary_tokens = tokens seq.state_joint_boundary_hash = h seq.state_joint_kv_tokens = kv_tokens @@ -740,23 +749,26 @@ def _joint_kv_boundary( # only into blocks below it that nobody else can be reading. claim_tokens = ((claim_blocks * hbs) // chunk) * chunk seq.state_joint_claim_tokens = max(claim_tokens, hbm_boundary * hbs) - self.joint_boundaries += 1 - # Which tier the STATE leg came from. The KV leg's own tier is decided - # separately by `_decide_load_after_alloc`, which is why these are not - # named for the boundary as a whole. - if self.paged_state_checkpoints.contains(h): - self.state_hbm += 1 - else: - self.state_tier += 1 + if record: + self.joint_boundaries += 1 + # Which tier the STATE leg came from. The KV leg's own tier is + # decided separately by `_decide_load_after_alloc`, which is why + # these are not named for the boundary as a whole. + if self.paged_state_checkpoints.contains(h): + self.state_hbm += 1 + else: + self.state_tier += 1 return candidate - def _no_joint(self, reason: str) -> int: + def _no_joint(self, reason: str, record: bool = True) -> int: """Record why this admission got no joint boundary, and return 0. Every reason is counted, including "this build is not even trying", because a silent zero is indistinguishable from a feature that ran and - found nothing. + found nothing. `record=False` is the fit-probe path (see + `_joint_kv_boundary`): compute the answer without touching the funnel. """ - self.joint_skips[reason] = self.joint_skips.get(reason, 0) + 1 + if record: + self.joint_skips[reason] = self.joint_skips.get(reason, 0) + 1 return 0 def _chain_to( @@ -775,9 +787,15 @@ def _chain_to( chain.append(h) return chain - def can_allocate(self, seq: Sequence) -> int: + def can_allocate(self, seq: Sequence, record: bool = True) -> int: """Return number of cache-hit blocks (>=0) if seq fits, else -1. + `record=False` marks a pure fit probe -- a caller asking only whether + the seq *could* be admitted, not admitting it. The answer is identical; + it just does not move the joint-boundary funnel counters, so a probe + that never admits anything cannot inflate operator metrics (see + `_joint_kv_boundary`). + The hit count is the contiguous run of cache hits starting at the prompt's first block. On the first miss we break: subsequent blocks cannot match either (hash is chained, so a divergent token breaks the @@ -831,7 +849,7 @@ def can_allocate(self, seq: Sequence) -> int: # Recorded on the seq rather than returned: what `allocate` claims from # HBM is still `num_cached_blocks`, and the joint boundary only decides # where the two loads are aimed. - self._joint_kv_boundary(seq, num_cached_blocks, block_hashes) + self._joint_kv_boundary(seq, num_cached_blocks, block_hashes, record=record) # Instrumentation: the pre-gate hit, so CacheStats can separate reuse # the gates declined (compressed_hit - num_cached_blocks) from reuse # lost to compressed eviction (everything above compressed_hit). diff --git a/atom/model_engine/scheduler.py b/atom/model_engine/scheduler.py index 8f24da9e56..ed60383b2d 100644 --- a/atom/model_engine/scheduler.py +++ b/atom/model_engine/scheduler.py @@ -1262,8 +1262,10 @@ def _can_admit_head_prefill(self) -> bool: and num_new_tokens > self.max_num_batched_tokens ): continue - # KV-pressured requests definitely cannot prefill. - return self.block_manager.can_allocate(seq) >= 0 + # KV-pressured requests definitely cannot prefill. A fit probe, not + # an admission -- `record=False` so it does not move the joint- + # boundary funnel counters for a seq it may never schedule. + return self.block_manager.can_allocate(seq, record=False) >= 0 return False def _kv_usage(self) -> float: From f35f0f92e97c90ae57ee21c2f21a79787e474707 Mon Sep 17 00:00:00 2001 From: zejchen Date: Mon, 31 Aug 2026 15:37:14 +0000 Subject: [PATCH 060/151] #8-4c: document the three-way offload selector and kimi_k3 modules The offload README still described a two-way dense/hybrid selector and had no module-map rows for the four new hybrid/kimi_k3/ files, even though select_offload_layout() (config.py:100) resolves a third layout: kimi_k3 when hf_config.model_type == "kimi_linear". - Rewrite the selector paragraph to name all three layouts (kimi_k3 / hybrid / dense) and the condition that picks each. - Update the connector.py module-map row to dense/hybrid/kimi_k3. - Add rows for hybrid/kimi_k3/{connector,staging,state_object,state_tier}.py. The OFFLOAD_MAX_PENDING_SAVES row (line 856) already documents the canonical max(2, 2 x OFFLOAD_COPY_WORKERS) default; with the #8-4a reader unification the scheduler now honors that value, so doc and code agree. --- atom/kv_transfer/offload/README.md | 15 +++++++++++---- 1 file changed, 11 insertions(+), 4 deletions(-) diff --git a/atom/kv_transfer/offload/README.md b/atom/kv_transfer/offload/README.md index c95c62322d..7d353e6e6e 100644 --- a/atom/kv_transfer/offload/README.md +++ b/atom/kv_transfer/offload/README.md @@ -13,9 +13,12 @@ A DSV4 boundary is reusable only when both PAGE and SLOT restore successfully. Missing, incompatible, or corrupt sidecar data fails closed to recomputation. The public configuration remains `kv_connector: "lmcache_offload"`. The thin -top-level shell selects `hybrid` when `hf_config.compress_ratios` is present and -`dense` otherwise; `kv_transfer_config.offload_layout` can override that choice -without giving scheduler and worker different connector names. +top-level shell resolves one of three layouts: `kimi_k3` when +`hf_config.model_type == "kimi_linear"` (dense paged MLA KV plus a KDA +per-request state tier), `hybrid` when `hf_config.compress_ratios` is present +(DSV4 PAGE+SLOT), and `dense` otherwise. `kv_transfer_config.offload_layout` can +override that choice without giving scheduler and worker different connector +names. It is the **ATOM-native, in-engine** offload path: the connector plugs straight into ATOM's scheduler/worker via the shared @@ -67,7 +70,7 @@ Four rules carry the module: | File | Role | |------|------| | `__init__.py` | Registers the `lmcache_offload` backend with `KVConnectorFactory`. | -| `connector.py` | Public config-only `dense`/`hybrid` selector and thin worker/scheduler delegation shells. | +| `connector.py` | Public config-only `dense`/`hybrid`/`kimi_k3` selector and thin worker/scheduler delegation shells. | | `_offload_common.py` | Shared LMCache engine construction, role validation, executors, and completion plumbing. | | `_block_gpu_connector.py` | Family-neutral raw-block LMCache `GPUConnectorInterface`; DCP-aware bounded staging and two-stage copies. | | `config.py` | Builds the per-rank `LMCacheEngineConfig` + `LMCacheMetadata` from `LMCACHE_*` env and `kv_transfer_config` extras. | @@ -79,6 +82,10 @@ Four rules carry the module: | `hybrid/dsv4/policy.py` | DSV4 geometry/profile, PAGE/SLOT cadence, prefix hashes, fingerprint, committed-checkpoint policy, and bounded staging-row admission. | | `hybrid/dsv4/codec.py` | `DSV4PageSlotCodec`, `DSV4CheckpointCodec`, and `DSV4CheckpointStore`: unified GPU layout plans plus AOS1 framing/storage. | | `hybrid/dsv4/triton_page_slot.py` | Raw-`uint8` PAGE/SLOT gather/scatter kernels; PAGE is forward-indexed and SLOT is reverse-indexed. | +| `hybrid/kimi_k3/connector.py` | Kimi-K3 worker and scheduler: the dense paged-KV path plus one extra leg for the KDA per-request state tier. | +| `hybrid/kimi_k3/staging.py` | Single-entry bounded GPU staging buffer, D2H/H2D copy stream, and producer event for one flat state entry per transfer. | +| `hybrid/kimi_k3/state_object.py` | One state checkpoint as a single opaque object keyed by ATOM's own hash, bypassing LMCache's `ChunkedTokenDatabase` (state bytes are not token-sliceable). | +| `hybrid/kimi_k3/state_tier.py` | Worker-side store/load driver for the state tier on its own executor; reports store/finished/failed hash sets for the engine-side `StateOffloadIndex` to apply. | | `atom_lmcache_staging.py` | Per-thread CUDA streams, staging buffer, ready/free events, env helpers. | ## Architecture From 7c9015a020eed42066318a5ee26d65825421f99c Mon Sep 17 00:00:00 2001 From: zejchen Date: Mon, 31 Aug 2026 15:41:05 +0000 Subject: [PATCH 061/151] #8-4d: drop StagedTransfer's dead free_event fence writes StagedTransfer._finish records free_event and sets free_event_valid=True before calling producer.synchronize(). The synchronize() is the whole fence -- by the time _finish returns, the producer stream has drained and the buffer is free -- and StagedTransfer never reads free_event or free_event_valid anywhere (its only wait_event is on ready_event, in _handoff). So the free-fence it publishes has no consumer: the record, the True write, and the four defensive `= False` resets in ensure_buffer / release_buffer_if_requested / the pack and unpack except paths are all dead within this class. The maintainer flagged free_event_valid as vestigial. It is NOT globally dead, though: _StagingBuffer.free_event_valid is a live stream-reuse fence in run_staged_pipeline (atom_lmcache_staging.py:123), which waits on free_event before refilling the bounded buffer. StagedTransfer imports that shared _StagingBuffer class but drives its own synchronous pipeline and uses a disjoint per-thread buffer set, so it never reaches that reader. Remove only StagedTransfer's writes; keep the field and its live reader. _finish's docstring now states it is a synchronous fence that deliberately leaves the shared buffer's free_event/free_event_valid untouched. Tests that assert free_event_valid resets all exercise the BlockGPUConnector / run_staged_pipeline path and are unaffected. --- .../kv_transfer/offload/hybrid/kimi_k3/staging.py | 15 +++++++-------- 1 file changed, 7 insertions(+), 8 deletions(-) diff --git a/atom/kv_transfer/offload/hybrid/kimi_k3/staging.py b/atom/kv_transfer/offload/hybrid/kimi_k3/staging.py index 7ca7721b0b..dcb2fceda9 100644 --- a/atom/kv_transfer/offload/hybrid/kimi_k3/staging.py +++ b/atom/kv_transfer/offload/hybrid/kimi_k3/staging.py @@ -113,7 +113,6 @@ def ensure_buffer( dtype=torch.uint8, device=self.device, ) - staging_buffer.free_event_valid = False return staging_buffer.tensor[:nbytes] def release_buffer_if_requested( @@ -123,7 +122,6 @@ def release_buffer_if_requested( if not self._release_after_transfer: return staging_buffer.tensor = None - staging_buffer.free_event_valid = False def memory_tensor(self, memory_obj: Any, nbytes: int) -> torch.Tensor: tensor = getattr(memory_obj, "tensor", None) @@ -221,7 +219,6 @@ def pack(self, segments: list[torch.Tensor], dst: Any) -> None: ) self._finish(state, state.copy_stream) except Exception: - staging_buffer.free_event_valid = False # The caller must release the source units on this path too -- # otherwise a failed store holds a whole image out of the KV # pool. Draining the device first is what makes that safe: the @@ -287,7 +284,6 @@ def unpack(self, src: Any, segments: list[torch.Tensor]) -> None: fused_unpack_chunk_major(device_buf, segments, seg_bytes, [1], [0]) self._finish(state, state.pack_stream) except Exception: - staging_buffer.free_event_valid = False # Mirror of pack()'s except: the unpack kernel scatters into the # caller's destination segments on pack_stream. Returning while # that kernel is still queued would let the owner reuse the @@ -306,8 +302,11 @@ def _handoff(state: _ThreadTransferState, producer, consumer) -> None: @staticmethod def _finish(state: _ThreadTransferState, producer) -> None: - """Publish the result: the buffer is free again once `producer` drains, - and the caller may not observe the bytes until it has.""" - state.staging_buffer.free_event.record(producer) - state.staging_buffer.free_event_valid = True + """Publish the result synchronously: block until `producer` drains, so + the buffer is free and the caller may observe the bytes on return. + + Unlike `run_staged_pipeline`, this path does not hand a `free_event` + to a later consumer -- `producer.synchronize()` is the whole fence -- so + it deliberately leaves the shared buffer's `free_event`/ + `free_event_valid` untouched.""" producer.synchronize() From 169edb453ee7cd05b01929cc6c9bf12d4961e2e7 Mon Sep 17 00:00:00 2001 From: zejchen Date: Mon, 31 Aug 2026 15:51:23 +0000 Subject: [PATCH 062/151] [LMCache][multi] Route the state face by tier ownership, not method presence Review #6a. Under `kv_connector: multi` the composite selected the sub-connector for every state call with `_first_with`, which picks the first sub that *has* the attribute. That is the wrong test for the state tier: every `LMCacheOffloadConnectorScheduler` shell -- dense, hybrid, and kimi_k3 alike -- defines the whole state face (`enqueue_state_stores`, `enqueue_state_loads`, `take_state_reports`, `take_state_source_releases`) unconditionally, delegating through `getattr` to its `_impl` and returning the no-tier default when the impl has none. So with a `[dense_offload, kimi_k3_offload]` (or moriio-first) layout every sub answers `hasattr`, and `_first_with` routes state stores, loads and reports to a dense shell whose `_impl` silently refuses the stores (recording them failed), discards the loads (forcing recompute), drains no reports, and reports the wrong `chunk_size` -- while the sub that actually built the tier never sees them. The CPU/HBM state tier cannot fill or be found. Give the shell a `has_state_tier` property that probes `_impl` for the store entry point (`enqueue_state_stores` is defined only on `KimiK3OffloadScheduler`, so its presence on the impl is the true tier discriminator), and select the state sub in the composite with a new `_state_tier_sub()` that keys on that property. Route `chunk_size` and the four state forwarders through it; `chunk_size` keeps its `_first_with` fallback for non-state offload layouts. `_first_with` still drives the KV-only members (park/prefill-chunk hooks). tests/test_multi_connector.py: teach the mock about `has_state_tier` and add a regression that puts a tier-less offload shell (methods present) first and the tier-carrying shell second, asserting every state call lands on the second. --- .../disaggregation/multi/multi_connector.py | 52 ++++++++++++++++--- atom/kv_transfer/offload/connector.py | 18 +++++++ tests/test_multi_connector.py | 44 ++++++++++++++++ 3 files changed, 107 insertions(+), 7 deletions(-) diff --git a/atom/kv_transfer/disaggregation/multi/multi_connector.py b/atom/kv_transfer/disaggregation/multi/multi_connector.py index 6287fe5b8c..f800770d99 100644 --- a/atom/kv_transfer/disaggregation/multi/multi_connector.py +++ b/atom/kv_transfer/disaggregation/multi/multi_connector.py @@ -411,6 +411,27 @@ def __init__(self, config: Any) -> None: # offload backend (Scheduler._is_offload_connector reads this). self.is_offload = any(getattr(c, "is_offload", False) for c in self._connectors) + def _state_tier_sub(self): + """The one sub-connector that actually hosts the state offload tier. + + Selection is by real capability (`has_state_tier`), never by method + presence: `LMCacheOffloadConnectorScheduler` defines the entire state + face on every layout, so `_first_with(..., "enqueue_state_stores")` + picks a dense offload shell -- whose `_impl` has no tier and returns + False / empty for every state call -- ahead of the `kimi_k3` shell that + owns the tier. The result was that, under + `connectors: [lmcache_offload(dense), lmcache_offload(kimi_k3)]`, stores + settled failed, loads were abandoned and recomputed, and the real + reports were never drained. `_adopt_state_tier` (worker side) already + refuses a config with two tiers, so "first" here is "only"; returns + None when no sub hosts a tier (state forwarders then fall to their + no-tier defaults). + """ + for c in self._connectors: + if getattr(c, "has_state_tier", False): + return c + return None + # -- base interface ----------------------------------------------------- def get_num_new_matched_tokens(self, seq: Any) -> tuple[int, bool]: @@ -460,10 +481,15 @@ def chunk_size(self) -> int | None: composite. Without this forwarder the getattr default (0) wins, so `_joint_kv_boundary` sees a zero chunk grid and refuses every joint KV load with `no_chunk_size` -- the K3 state tier's KV leg is silently off - under `kv_connector: multi`. Delegate to the one offload sub (at most - one hosts the tier, so "first" is "only"). + under `kv_connector: multi`. + + Prefer the state-tier sub's grid: `_joint_kv_boundary` aligns against + it, and with two offload subs (dense + kimi_k3) `_first_with` would have + handed back the dense sub's grid. Fall back to any offload sub's grid + when no tier is configured, so a plain `[moriio, lmcache_offload(dense)]` + still gets its dense KV chunk size. """ - c = _first_with(self._connectors, "chunk_size") + c = self._state_tier_sub() or _first_with(self._connectors, "chunk_size") return getattr(c, "chunk_size", None) if c is not None else None def should_park_for_load_after_alloc(self, seq: Any) -> bool: @@ -485,8 +511,11 @@ def enqueue_state_loads(self, loads) -> bool: The False is not a formality: every load here belongs to a parked request only a report can wake, and the caller's `hasattr` guard cannot catch a swallowed one because this method always exists. + + Select by `has_state_tier`, not method presence -- see + `_state_tier_sub`. """ - c = _first_with(self._connectors, "enqueue_state_loads") + c = self._state_tier_sub() if c is None: return False return bool(c.enqueue_state_loads(loads)) @@ -501,8 +530,11 @@ def enqueue_state_stores(self, stores) -> bool: the CPU tier can never fill under `kv_connector: multi`, even with an offload sub-connector configured. `_adopt_state_tier` already guarantees at most one tier, so "first" is "only" here too. + + Select by `has_state_tier`, not method presence -- see + `_state_tier_sub`. """ - c = _first_with(self._connectors, "enqueue_state_stores") + c = self._state_tier_sub() if c is None: return False return bool(c.enqueue_state_stores(stores)) @@ -514,8 +546,11 @@ def take_state_reports(self): `scheduler._update_from_kv_xfer_finished`). Missing here, the engine never drains store reports, so pins never settle and stored hashes are never indexed -- the CPU tier fills but nothing can be found in it. + + Select by `has_state_tier`, not method presence -- see + `_state_tier_sub`. """ - c = _first_with(self._connectors, "take_state_reports") + c = self._state_tier_sub() if c is None: return set(), set() return c.take_state_reports() @@ -527,8 +562,11 @@ def take_state_source_releases(self) -> set: Its own forwarder rather than a third element of `take_state_reports`, for the reason the offload connector records: that tuple's arity is a contract with the caller, and widening it once already wedged the pool. + + Select by `has_state_tier`, not method presence -- see + `_state_tier_sub`. """ - c = _first_with(self._connectors, "take_state_source_releases") + c = self._state_tier_sub() if c is None: return set() return c.take_state_source_releases() diff --git a/atom/kv_transfer/offload/connector.py b/atom/kv_transfer/offload/connector.py index af28056d22..537cc7ff0d 100644 --- a/atom/kv_transfer/offload/connector.py +++ b/atom/kv_transfer/offload/connector.py @@ -134,6 +134,24 @@ class LMCacheOffloadConnectorScheduler(KVConnectorSchedulerBase): def __init__(self, config) -> None: self._impl = _build_scheduler(config) + @property + def has_state_tier(self) -> bool: + """True when the selected impl actually hosts the KDA state tier. + + This shell defines the whole state face (`enqueue_state_stores`, + `enqueue_state_loads`, `take_state_reports`, + `take_state_source_releases`) unconditionally, delegating through + `getattr` to `_impl` and returning the no-tier default when the impl + does not implement a method. So method *presence on the shell* cannot + tell a tier-hosting `kimi_k3` impl from a `dense`/`hybrid` impl that has + none -- which is exactly how `MultiConnector._first_with`, selecting by + attribute presence, routed every state call to a dense offload shell + whose `_impl` silently returns the no-tier defaults. Probe the impl for + the store entry point instead: the real state face lives there (only + `KimiK3OffloadScheduler` defines `enqueue_state_stores`). + """ + return getattr(self._impl, "enqueue_state_stores", None) is not None + @property def chunk_size(self) -> int | None: """The KV leg's transfer grid, in tokens. diff --git a/tests/test_multi_connector.py b/tests/test_multi_connector.py index c0e73d09fe..43a08fc989 100644 --- a/tests/test_multi_connector.py +++ b/tests/test_multi_connector.py @@ -43,6 +43,7 @@ def __init__( is_producer=False, is_offload=False, offload_methods=False, + has_state_tier=None, ): self._match = match self.is_producer = is_producer @@ -52,6 +53,15 @@ def __init__( self.finished_calls = [] self.meta = ConnectorMetadata() self._offload = offload_methods + # The real `LMCacheOffloadConnectorScheduler` shell defines the whole + # state face unconditionally and exposes `has_state_tier` to say whether + # its `_impl` actually carries the tier. An offload shell always has the + # flag; a producer like moriio has no such attribute at all. Default a + # tier-carrying shell to True so existing offload subs keep routing. + if offload_methods: + self.has_state_tier = ( + offload_methods if has_state_tier is None else has_state_tier + ) if offload_methods: self.park = False @@ -442,6 +452,40 @@ def test_state_stores_and_reports_forward_to_the_owning_sub(): assert not hasattr(moriio, "enqueue_state_stores") +def test_state_calls_route_by_tier_ownership_not_method_presence(): + """The reason `_first_with` was the wrong selector for the state face. + + Every `LMCacheOffloadConnectorScheduler` shell -- dense OR kimi_k3 -- + defines the whole state face unconditionally, delegating through `getattr` + to its `_impl` and returning the no-tier default when the impl has none. So + with `[dense_offload, kimi_k3_offload]` both subs answer `hasattr` for + `enqueue_state_stores`, and `_first_with` picks the dense shell -- whose + `_impl` silently refuses every store, discards every load, and drains no + reports. The tier actually lives on the second sub. + + Selection must follow `has_state_tier` (does the `_impl` really host the + tier), not method presence. Here the dense-like shell is first and carries + no tier; the kimi_k3-like shell is second and does. Every state call must + land on the second. + """ + dense = FakeSchedSub(is_offload=True, offload_methods=True, has_state_tier=False) + k3 = FakeSchedSub(is_offload=True, offload_methods=True, has_state_tier=True) + k3.state_reports = ({7}, {9}) + k3.state_source_releases = {7, 9} + sched = _sched([dense, k3]) + + assert sched.enqueue_state_stores([(111, (1, 2, 3))]) is True + assert sched.enqueue_state_loads([(1, 111, 0)]) is True + assert sched.take_state_reports() == ({7}, {9}) + assert sched.take_state_source_releases() == {7, 9} + + # The tier-carrying sub got everything; the dense shell got nothing. + assert k3.state_stores == [(111, (1, 2, 3))] + assert k3.state_loads == [(1, 111, 0)] + assert dense.state_stores == [] + assert dense.state_loads == [] + + def test_state_stores_default_when_no_sub_implements(): """No tier under the composite: refuse the stores (not silently accept) and drain nothing, matching the offload connector's own no-tier fallbacks. A From 4551281fdfa35b6e342566dbabc3e5b65e51e3db Mon Sep 17 00:00:00 2001 From: zejchen Date: Mon, 31 Aug 2026 15:54:46 +0000 Subject: [PATCH 063/151] [LMCache][multi] Cancel losing subs' armed loads and forward load cancellation Review #6b. The composite's `get_num_new_matched_tokens` is first-hit-wins in its *return value*, but it queries every sub and an offload sub's lookup is not side-effect-free: matching a prefix arms a KV load (LMCache lookup pin plus a `_load_specs`/`_lookup_in_step` entry). Because `update_state_after_alloc` fans to every sub, a losing offload sub's armed load is later flipped to `can_load=True` and recv-queued -- firing into the same block table the winner already owns. Under `[moriio, lmcache_offload]` moriio answers first while the offload tier still arms a second write over moriio's KV, plus a `finished_loading` the scheduler never accounted; under two offload subs the dense shell can win while kimi_k3 arms behind it. Once a winner is chosen, cancel every other sub's pending load. `cancel_pending_load` is idempotent (guarded by `_load_lifecycles`) and only offload subs define it, so a sub that armed nothing -- a miss, or moriio -- is a no-op. The querying pass over all subs is unchanged, so every sub still sees every request; only the harmful armed-load side effect is undone. Also add the `cancel_pending_load` forwarder the composite was missing entirely: the scheduler abandons a parked load by calling it on the connector it holds, which under `multi` is the composite. With no forwarder the offload sub never heard the cancel and leaked its `_load_specs`/`_reqs_need_recv`/lookup pin for the abandoned request. Fan to every sub that arms loads. tests/test_multi_connector.py: mock now arms a load on a matching offload lookup and clears it on cancel; add regressions for loser-cancelled, winner-kept, and the standalone cancel forwarder. --- .../disaggregation/multi/multi_connector.py | 43 ++++++++++++- tests/test_multi_connector.py | 60 +++++++++++++++++++ 2 files changed, 101 insertions(+), 2 deletions(-) diff --git a/atom/kv_transfer/disaggregation/multi/multi_connector.py b/atom/kv_transfer/disaggregation/multi/multi_connector.py index f800770d99..7dbe17e019 100644 --- a/atom/kv_transfer/disaggregation/multi/multi_connector.py +++ b/atom/kv_transfer/disaggregation/multi/multi_connector.py @@ -435,14 +435,53 @@ def _state_tier_sub(self): # -- base interface ----------------------------------------------------- def get_num_new_matched_tokens(self, seq: Any) -> tuple[int, bool]: - """First-hit-wins: the first sub that reports a match owns the load.""" + """First-hit-wins, and undo the losers' armed loads. + + A sub's lookup is not side-effect-free: an offload sub arms a KV load + for any prefix it matches -- it takes an LMCache lookup pin, records a + `_load_specs` entry and a `_lookup_in_step` id, so `update_state_after_ + alloc` (which the composite fans to *every* sub) later flips + `can_load=True` and recv-queues that request. Only the first matching + sub owns the load. If a second sub also matched -- moriio winning ahead + of the offload tier, or a dense offload shell ahead of the kimi_k3 one + -- the loser's armed load fires into the same block table on + `update_state_after_alloc`: a second writer over the winner's KV, plus a + `finished_loading` the scheduler never accounted. So once a winner is + chosen, cancel every other sub's pending load. `cancel_pending_load` is + idempotent and guarded by `_load_lifecycles`, so a sub that armed + nothing (a miss, or moriio which has no such method) is a no-op. + """ result = (0, False) + winner = None for c in self._connectors: toks, needs_load = c.get_num_new_matched_tokens(seq) - if result[0] == 0 and toks > 0: + if winner is None and toks > 0: result = (toks, needs_load) + winner = c + if winner is not None: + for c in self._connectors: + if c is not winner: + fn = getattr(c, "cancel_pending_load", None) + if callable(fn): + fn(seq) return result + def cancel_pending_load(self, seq: Any) -> None: + """Forward a load cancellation to every sub that arms loads. + + The scheduler calls this on `self.kv_connector` -- the composite under + `multi` -- when a parked load is abandoned (park timeout, request + finished before its transfer). Only offload subs implement it, and the + composite had no forwarder, so under `multi` the cancel never reached + the offload sub: its `_load_specs`/`_reqs_need_recv`/lookup pin leaked + and the abandoned request stayed recv-queued. Idempotent per sub (the + `_load_lifecycles` guard), so fanning to all is safe. + """ + for c in self._connectors: + fn = getattr(c, "cancel_pending_load", None) + if callable(fn): + fn(seq) + def build_connector_meta(self) -> MultiConnectorMetadata: return MultiConnectorMetadata( metas=[c.build_connector_meta() for c in self._connectors] diff --git a/tests/test_multi_connector.py b/tests/test_multi_connector.py index 43a08fc989..6b3f8ccd1a 100644 --- a/tests/test_multi_connector.py +++ b/tests/test_multi_connector.py @@ -75,8 +75,15 @@ def __init__( self.state_stores = [] self.state_reports = (set(), set()) self.state_source_releases = set() + self.pending_load = False + self.cancelled = [] def get_num_new_matched_tokens(self, seq): + # Model the offload lookup's side effect: a matching prefix arms a load + # (LMCache pin + _load_specs) that update_state_after_alloc would later + # turn into a real recv. A miss arms nothing. + if self._offload and self._match[0] > 0: + self.pending_load = True return self._match def build_connector_meta(self): @@ -107,6 +114,11 @@ def save_finished(self, req_id): def load_failed(self, req_id): self.load_failed_ids.append(req_id) + def cancel_pending_load(self, seq): + # Idempotent, like the real connector's `_load_lifecycles` guard. + self.cancelled.append(seq) + self.pending_load = False + def process_completions(self, output): # Mirrors OffloadSchedulerMixin: apply the completions, then hand the # scheduler bare request ids. @@ -144,6 +156,7 @@ def __getattribute__(self, name): "should_defer_free", "save_finished", "load_failed", + "cancel_pending_load", "process_completions", "has_pending_work", "enqueue_state_loads", @@ -247,6 +260,53 @@ def test_no_match_returns_zero(): assert sched.get_num_new_matched_tokens(object()) == (0, False) +def test_losing_offload_sub_load_is_cancelled_when_another_sub_wins(): + """Review #6b. The lookup arms a load; only the winner may keep it. + + In `[moriio, lmcache_offload]`, moriio answers the match first, but the + offload sub's own lookup still armed a KV load for the same prefix. Because + `update_state_after_alloc` fans to every sub, that armed load would fire + into the same block table moriio is already filling -- a second writer plus + an unaccounted `finished_loading`. Once moriio wins, the composite must + cancel the offload sub's pending load. + """ + moriio = FakeSchedSub(is_producer=True, match=(5, True)) + off = FakeSchedSub(is_offload=True, offload_methods=True, match=(4, True)) + seq = object() + sched = _sched([moriio, off]) + + assert sched.get_num_new_matched_tokens(seq) == (5, True) # moriio wins + assert off.cancelled == [seq] # its armed load was undone + assert off.pending_load is False + + +def test_winning_offload_sub_keeps_its_load(): + """The mirror: when the offload sub is the first match, its load is the one + that owns the block table, so it must NOT be cancelled.""" + moriio = FakeSchedSub(is_producer=True, match=(0, False)) + off = FakeSchedSub(is_offload=True, offload_methods=True, match=(4, True)) + seq = object() + sched = _sched([moriio, off]) + + assert sched.get_num_new_matched_tokens(seq) == (4, True) # offload wins + assert off.cancelled == [] + assert off.pending_load is True + + +def test_cancel_pending_load_forwards_to_offload_subs(): + """The scheduler abandons a parked load by calling `cancel_pending_load` on + the connector it holds -- the composite under `multi`. With no forwarder the + offload sub never heard it and leaked the load; fan to every sub that arms + loads (idempotent, and moriio has no such method).""" + moriio = FakeSchedSub(is_producer=True) + off = FakeSchedSub(is_offload=True, offload_methods=True) + seq = object() + _sched([moriio, off]).cancel_pending_load(seq) + + assert off.cancelled == [seq] + assert not hasattr(moriio, "cancel_pending_load") + + def test_update_and_finished_fan_out_to_all(): a, b = FakeSchedSub(), FakeSchedSub() sched = _sched([a, b]) From aa8a2b3d5082cdf9ac88cc605549a3c3d28c59a6 Mon Sep 17 00:00:00 2001 From: zejunchen-zejun Date: Tue, 1 Sep 2026 09:34:18 +0800 Subject: [PATCH 064/151] fix(ci): gate the LMCache chunk probe, and rename one unused unpack Two pre-check failures on the current head. **Ruff (RUF059, tests/test_block_manager.py:181).** `holder` is unpacked and never used in `test_disown_fails_when_pool_cannot_back_private_copies` -- the two sibling call sites do use it, so this one is the outlier. Renamed to `_holder`, which is ruff's own suggested fix. **Unit tests (`test_idle_heartbeat_is_available_on_every_scheduler`).** Fails on this branch and passes on main, so it is ours. The test asserts `"Engine" not in caplog.text` after constructing three schedulers on a plain `MockConfig`. `BlockManager.__init__` was reading the LMCache chunk size unconditionally, and that import reaches an optional third-party package: with lmcache absent it logged a full `ModuleNotFoundError` traceback at WARNING, and the traceback text contains `LMCacheEngineConfig` -- which contains "Engine". The brittle substring assertion belongs to main and is not this PR's to change; the unconditional probe is ours and is wrong on its own merits. An engine with no offload connector has no use for the chunk grid, cannot issue a joint KV load, and should not be importing lmcache to find that out -- yet every such start paid an exception, a traceback, and a WARNING line. The same applies to a dense or hybrid offload engine: it has a connector but no per-request state, so a joint load is impossible either way. The probe now runs only when `state_tier_capability` says a tier is hosted -- the same gate the tier itself uses, so the chunk size and the capability cannot disagree -- and is extracted into `_read_joint_chunk_tokens` rather than left inline. Where it does run, a failure still logs with `exc_info`, and now means something: the tier is on, so a missing chunk size really does disable the joint KV load. The startup line `[State Cache] CPU state tier off: ` is narrowed the same way, to models that actually have a checkpoint spec. A dense engine announcing that a tier it could never use is off is true and useless. Tests: an engine with no tier neither probes nor logs anything mentioning lmcache; a dense-layout offload engine likewise; a hosted tier still probes and still says so when it cannot read the size. --- atom/model_engine/block_manager.py | 84 +++++++++++++++++++----------- tests/test_block_manager.py | 61 +++++++++++++++++++++- 2 files changed, 115 insertions(+), 30 deletions(-) diff --git a/atom/model_engine/block_manager.py b/atom/model_engine/block_manager.py index 47c8ec1288..c2fe98b8a8 100644 --- a/atom/model_engine/block_manager.py +++ b/atom/model_engine/block_manager.py @@ -172,7 +172,14 @@ def __init__( # nowhere to go. self.state_tier_capability = state_tier_capability(config) kv_offload_enabled = self.state_tier_capability.hosts_state_tier - if not kv_offload_enabled and self.state_tier_capability.reason: + # Only worth saying for a model that has state to offload. Every dense + # engine would otherwise announce at startup that a tier it could never + # use is off, which is true and useless. + if ( + not kv_offload_enabled + and checkpoint_spec is not None + and self.state_tier_capability.reason + ): logger.info( "[State Cache] CPU state tier off: %s.", self.state_tier_capability.reason, @@ -262,34 +269,13 @@ def __init__( # The LMCache chunk size in tokens, read where the config is rather than # off the connector object. self._joint_chunk_tokens = 0 - try: - from atom.kv_transfer.offload import config as offcfg - from atom.model_engine.state_offload import _offload_subconfig - - kvcfg = getattr(config, "kv_transfer_config", None) or {} - # Under `multi` the lmcache.* keys (chunk_size, offload_layout) live - # on the offload SUB-connector, not the composite. Passing the raw - # composite here read a zero chunk grid, and `_joint_kv_boundary` - # then refused every joint KV load with `no_chunk_size`. Unwrap the - # sub the same way `state_tier_capability` does, so the gate's chunk - # size and the tier the capability check builds stay consistent. - if isinstance(kvcfg, dict) and kvcfg.get("kv_connector") == "multi": - sub, _why = _offload_subconfig(kvcfg) - if sub is not None: - kvcfg = sub - self._joint_chunk_tokens = int( - offcfg.build_lmcache_config(kvcfg).chunk_size - ) - except Exception: - # Blind on purpose: this runs at model load, the import reaches a - # third-party package that may be absent entirely, and the only - # consequence of not knowing the chunk size is that the joint KV - # load stays off. Refusing to start would be worse. - logger.warning( - "state offload: could not read the LMCache chunk size; the " - "joint KV load needs it and stays off", - exc_info=True, - ) + # Only when a joint KV load is possible at all. Reading it reaches into + # LMCache, an optional dependency: probing on an engine that hosts no + # state tier logged a full `ModuleNotFoundError` traceback at WARNING + # on every start, for a number that engine has no use for. The + # capability above is the same gate the tier itself uses. + if kv_offload_enabled: + self._joint_chunk_tokens = self._read_joint_chunk_tokens(config) self.state_offload: StateOffloadIndex | None = None # (req_id, hash, target_group) admitted this pass and not yet handed to # the connector. Kept here rather than in the index because the slot @@ -1041,6 +1027,46 @@ def allocate(self, seq: Sequence, num_cached_blocks: int = 0) -> bool: seq.state_joint_boundary_hash = -1 return True + @staticmethod + def _read_joint_chunk_tokens(config) -> int: + """The LMCache chunk size in tokens, or 0 if it cannot be read. + + Read from the config rather than off the connector object, because the + scheduler holds whatever `get_kvconnector` returned and one without + `chunk_size` would zero this and disable the joint load silently. + + Called only when the capability says a state tier is hosted; see the + caller. + """ + try: + from atom.kv_transfer.offload import config as offcfg + from atom.model_engine.state_offload import _offload_subconfig + + kvcfg = getattr(config, "kv_transfer_config", None) or {} + # Under `multi` the lmcache.* keys (chunk_size, offload_layout) live + # on the offload SUB-connector, not the composite. Passing the raw + # composite here read a zero chunk grid, and `_joint_kv_boundary` + # then refused every joint KV load with `no_chunk_size`. Unwrap the + # sub the same way `state_tier_capability` does, so the gate's chunk + # size and the tier the capability check builds stay consistent. + if isinstance(kvcfg, dict) and kvcfg.get("kv_connector") == "multi": + sub, _why = _offload_subconfig(kvcfg) + if sub is not None: + kvcfg = sub + return int(offcfg.build_lmcache_config(kvcfg).chunk_size) + except Exception: + # Blind on purpose: this runs at model load, the import reaches a + # third-party package that may be absent entirely, and the only + # consequence of not knowing the chunk size is that the joint KV + # load stays off. Refusing to start would be worse. Reached only + # when the tier is on, so a traceback here is worth printing. + logger.warning( + "state offload: could not read the LMCache chunk size; the " + "joint KV load needs it and stays off", + exc_info=True, + ) + return 0 + def _state_leg_secured(self, seq: Sequence) -> bool: """Whether something will really put state behind this seq's boundary. diff --git a/tests/test_block_manager.py b/tests/test_block_manager.py index d9d392e5e7..9977ac4834 100644 --- a/tests/test_block_manager.py +++ b/tests/test_block_manager.py @@ -178,7 +178,9 @@ def test_disown_fails_when_pool_cannot_back_private_copies(self, seq_factory): # 6 blocks: holder takes 4, seq's fresh tail takes 1, leaving 1 free -- # fewer than the 4 private copies the disown needs, so it must refuse # rather than silently reuse the shared blocks. - bm, holder, seq, shared = self._shared_prefix(seq_factory, num_kvcache_blocks=6) + bm, _holder, seq, shared = self._shared_prefix( + seq_factory, num_kvcache_blocks=6 + ) assert bm.kv.has_free(4) is False assert bm.disown_claimed_prefix(seq) is False @@ -1013,3 +1015,60 @@ def test_the_source_release_is_what_normally_returns_them(self): assert bm.paged_state_checkpoints.settled == [op] # ...and it does not touch the index, which the store report owns. assert bm.state_offload.hashes == set() + + +# ── the LMCache chunk probe is gated on the capability ───────────────────── + + +class TestJointChunkProbeIsGated: + """Reading the chunk size imports LMCache, which is an optional dependency. + Doing it unconditionally meant every engine without an offload connector + logged a full `ModuleNotFoundError` traceback at WARNING on startup, for a + number it has no use for -- and any caplog assertion downstream inherited + that noise.""" + + def _bm(self, kv_transfer_config): + cfg = MockConfig( + enable_prefix_caching=True, + kv_transfer_config=kv_transfer_config, + pool_entries={"state": 4}, + pool_entries_per_req={"state": 1}, + ) + return BlockManager(cfg) + + def test_an_engine_with_no_tier_does_not_probe_or_warn(self, caplog): + with caplog.at_level(logging.DEBUG, logger="atom"): + bm = self._bm({"kv_connector": "moriio"}) + assert bm._joint_chunk_tokens == 0 + assert "LMCache chunk size" not in caplog.text + assert "lmcache" not in caplog.text.lower() + + def test_a_dense_offload_engine_does_not_probe_either(self, caplog): + """It has an offload connector but no state to offload, so a joint KV + load is impossible and the chunk grid is not its business.""" + with caplog.at_level(logging.DEBUG, logger="atom"): + bm = self._bm( + { + "kv_connector": "lmcache_offload", + "kv_role": "offload", + "offload_layout": "dense", + } + ) + assert bm._joint_chunk_tokens == 0 + assert "LMCache chunk size" not in caplog.text + + def test_a_hosted_tier_still_probes_and_says_so_when_it_cannot_read(self, caplog): + """The warning is worth printing exactly here: the tier is on, so a + missing chunk size really does disable the joint KV load.""" + with caplog.at_level(logging.WARNING, logger="atom"): + bm = self._bm( + { + "kv_connector": "lmcache_offload", + "kv_role": "offload", + "offload_layout": "kimi_k3", + } + ) + # lmcache is not installed in the unit-test environment, so the probe + # runs and fails -- which is the point: it ran. + assert bm._joint_chunk_tokens == 0 + assert "LMCache chunk size" in caplog.text From 6565f105cc0bdc6c430a7989f7968c0fceeba6ac Mon Sep 17 00:00:00 2001 From: zejchen Date: Tue, 1 Sep 2026 07:09:39 +0000 Subject: [PATCH 065/151] fix(offload): guard the shell's abandon_save forward so DSV4 can't wedge the engine The LMCacheOffloadConnectorScheduler shell forwards abandon_save straight to self._impl (connector.py:191), but not every _impl defines it. DSV4OffloadScheduler (OffloadSchedulerMixin, KVConnectorSchedulerBase) defines save_finished but not abandon_save, and neither of its bases supplies one -- abandon_save exists only on the multi shell, this shell, the dense connector, and the kimi_k3 connector. The engine reaches this path from _reconcile_stalled_deferred_saves -> scheduler.py:980 -> _connector_abandon_save (:910). The getattr guard there (:919) passes because the *shell* always defines abandon_save, so the guard is satisfied one level too high; the unconditional forward inside the shell then raises AttributeError out of the engine loop and wedges the TP workers whenever a DSV4 model is run under lmcache_offload with a stalled save to reconcile. Guard the forward the same way its siblings adjust_prefill_chunk_after_alloc (:197) and cancel_pending_load (:205) already are: resolve the callback via getattr and no-op when the _impl doesn't implement it. --- atom/kv_transfer/offload/connector.py | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/atom/kv_transfer/offload/connector.py b/atom/kv_transfer/offload/connector.py index 537cc7ff0d..10a47cee9d 100644 --- a/atom/kv_transfer/offload/connector.py +++ b/atom/kv_transfer/offload/connector.py @@ -189,7 +189,9 @@ def save_finished(self, req_id) -> None: self._impl.save_finished(req_id) def abandon_save(self, req_id) -> None: - self._impl.abandon_save(req_id) + callback = getattr(self._impl, "abandon_save", None) + if callback is not None: + callback(req_id) def load_failed(self, req_id): return self._impl.load_failed(req_id) From e3ea1ae9e024bc840878770d4a6cfd60411b5b6c Mon Sep 17 00:00:00 2001 From: zejchen Date: Tue, 1 Sep 2026 07:12:07 +0000 Subject: [PATCH 066/151] fix(offload): refuse a joint boundary whose claim isn't hash-block-aligned _joint_kv_boundary floors state_joint_claim_tokens to the LMCache *chunk* grid so the KV leg starts on a chunk boundary. allocate() then claims that prefix in whole hash blocks (state_joint_claim_tokens // hbs). Those two grids only agree when hbs divides chunk. When hbs does not divide chunk -- a DCP world size that makes hbs = block_size * dcp_world_size no longer a divisor of LMCACHE_CHUNK_SIZE -- a chunk-aligned claim is not hash-block-aligned. Worked example: hbs=192, chunk=256, a 384-token match floors claim_tokens to 256, allocate claims 256 // 192 = 1 block (192 tokens), and the [192, 256) tail is handed a fresh unfilled block. Once the KV leg lands, num_cached_tokens is raised to the boundary and counts that fresh block as computed -- 64 tokens are never computed or loaded, silent wrong output with no exception. It is invisible in the shipping configuration, where block_size * dcp_world_size equals the LMCache chunk (1024) so hbs | chunk holds and the claim is aligned by construction -- which is why CI never hit it. Guard it: if the claim is not a multiple of hbs, drop the joint boundary (reset the four joint fields) and recompute the tail via _no_joint("claim_off_hash_grid") rather than resume the KV leg over a gap. --- atom/model_engine/block_manager.py | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/atom/model_engine/block_manager.py b/atom/model_engine/block_manager.py index c2fe98b8a8..82e01de992 100644 --- a/atom/model_engine/block_manager.py +++ b/atom/model_engine/block_manager.py @@ -735,6 +735,23 @@ def _joint_kv_boundary( # only into blocks below it that nobody else can be reading. claim_tokens = ((claim_blocks * hbs) // chunk) * chunk seq.state_joint_claim_tokens = max(claim_tokens, hbm_boundary * hbs) + # `allocate` claims the boundary in whole hash blocks -- + # `state_joint_claim_tokens // hbs` -- but the value above is floored to + # the *chunk* grid, not the hash-block grid. When `hbs` does not divide + # `chunk`, a chunk-aligned claim is not hash-block-aligned: the floor + # drops the `[floor(claim_tokens/hbs)*hbs, claim_tokens)` tail into a + # fresh, unfilled block that `num_cached_tokens` (raised to the boundary + # once the KV leg lands) then counts as computed -- silent wrong output, + # with no exception. It is invisible whenever `hbs | chunk`, which is the + # shipping case (`block_size * dcp_world_size == LMCACHE_CHUNK_SIZE`), so + # it only surfaces at DCP world sizes that take `hbs` off the chunk grid. + # Refuse the joint and recompute the tail rather than resume over a gap. + if seq.state_joint_claim_tokens % hbs != 0: + seq.state_joint_boundary_tokens = 0 + seq.state_joint_boundary_hash = -1 + seq.state_joint_kv_tokens = 0 + seq.state_joint_claim_tokens = 0 + return self._no_joint("claim_off_hash_grid", record) if record: self.joint_boundaries += 1 # Which tier the STATE leg came from. The KV leg's own tier is From 24cd6c434af22cee6070ff6598fe8b55313403c3 Mon Sep 17 00:00:00 2001 From: zejchen Date: Tue, 1 Sep 2026 07:12:37 +0000 Subject: [PATCH 067/151] fix(offload): keep hit_hash on the last cached block when admission breaks early allocate() assigned hit_hash only on the iteration where `i + 1 == num_cached_blocks`. The block_id == -1 branch above it can clamp num_cached_blocks down to an earlier `i` (an eviction detected inside the cached range) and break before that iteration is ever reached -- leaving hit_hash == -1 while num_cached_tokens is still num_cached_blocks * hbs > 0. _attach_state_slots then takes the hit_hash == -1 cold-start exit and requests no state restore at all, even though the request kept a non-empty cached prefix: the forward resumes over a prefix whose recurrent/compressor state was never restored. Silent wrong output. Assign hit_hash on every cached block (`i < num_cached_blocks`) instead of only the last one. In the normal full-claim path the final value is unchanged (the hash of block num_cached_blocks-1); on an early break it is the hash of the last block actually claimed, which is exactly the clamped num_cached_blocks-1. --- atom/model_engine/block_manager.py | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/atom/model_engine/block_manager.py b/atom/model_engine/block_manager.py index 82e01de992..9721780c4a 100644 --- a/atom/model_engine/block_manager.py +++ b/atom/model_engine/block_manager.py @@ -963,7 +963,15 @@ def allocate(self, seq: Sequence, num_cached_blocks: int = 0) -> bool: break self.kv.claim(block_id) seq.block_table.append(block_id) - if i + 1 == num_cached_blocks: + # Track the hash of the last claimed *cached* block as we go, not + # only when the loop reaches num_cached_blocks-1. If the block_id==-1 + # branch above clamps num_cached_blocks to an earlier `i` (an + # eviction inside the cached range), the `i + 1 == num_cached_blocks` + # test never fired, leaving hit_hash == -1 while num_cached_tokens + # stayed > 0 -- a cold-start state restore over a prefix that is in + # fact cached. Assigning on every cached block leaves hit_hash on the + # last one actually claimed, whichever way the loop exits. + if i < num_cached_blocks: hit_hash = h # Pin the restore before fresh blocks can evict its checkpoint. state_holds = True From 43a0a0d3c0f6e77537e4f9628bb9b75f8597d107 Mon Sep 17 00:00:00 2001 From: zejchen Date: Tue, 1 Sep 2026 07:13:15 +0000 Subject: [PATCH 068/151] perf(offload): compute the joint boundary below the pool-fit refusal, not above can_allocate walked _joint_kv_boundary before the _has_page_units pool-fit refusal. That walk is not free: _chain_to continues an xxhash chain up to the LMCache-only cap and _gated_hit rescans it. A request refused for lack of pool blocks discarded the whole result -- so a long prompt queued behind a full pool paid the chain every scheduling pass while never being admitted. This is the same cost the author already deferred for _extend_hash_chain, which sits just below the refusal with a comment explaining exactly this. Move _joint_kv_boundary down to join it. Nothing between the old call site and the refusal reads the seq's joint fields (_record_checkpoint_demand / _record_checkpoint_end / _has_page_units all take their inputs explicitly and never touch state_joint_*), so admitted requests still have the fields set before allocate() runs, and refused requests no longer pay for a boundary they discard. Side benefit that matches the funnel's own philosophy: the joint_boundaries / state_hbm / state_tier counters now move only for admissions that actually happen, not for ones the pool then rejects. --- atom/model_engine/block_manager.py | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-) diff --git a/atom/model_engine/block_manager.py b/atom/model_engine/block_manager.py index 9721780c4a..8573c2ecb8 100644 --- a/atom/model_engine/block_manager.py +++ b/atom/model_engine/block_manager.py @@ -848,11 +848,6 @@ def can_allocate(self, seq: Sequence, record: bool = True) -> int: # `_gated_hit` settles the two gates jointly; neither can be applied to # the other's answer. num_cached_blocks = self._gated_hit(seq, compressed_hit, block_hashes) - # A boundary LMCache and the tier can jointly reach, above this hit. - # Recorded on the seq rather than returned: what `allocate` claims from - # HBM is still `num_cached_blocks`, and the joint boundary only decides - # where the two loads are aimed. - self._joint_kv_boundary(seq, num_cached_blocks, block_hashes, record=record) # Instrumentation: the pre-gate hit, so EngineStats can separate reuse # the gates declined (compressed_hit - num_cached_blocks) from reuse # lost to compressed eviction (everything above compressed_hit). @@ -880,6 +875,16 @@ def can_allocate(self, seq: Sequence, record: bool = True) -> int: self._record_checkpoint_end(seq) if not self._has_page_units(num_new_blocks, protected_hash): return -1 + # A boundary LMCache and the tier can jointly reach, above this hit. + # Recorded on the seq rather than returned: what `allocate` claims from + # HBM is still `num_cached_blocks`, and the joint boundary only decides + # where the two loads are aimed. Below the refusal, not above it, for the + # same reason _extend_hash_chain is: _joint_kv_boundary walks a chained + # xxhash up to the LMCache-only cap (_chain_to) plus a _gated_hit rescan, + # and a refused admission would discard all of it. Nothing between here + # and the refusal reads the seq's joint fields, and the author already + # moved _extend_hash_chain down for this exact cost -- this is its twin. + self._joint_kv_boundary(seq, num_cached_blocks, block_hashes, record=record) # After the refusal, not before it. The chain is O(prompt) xxhash plus # two temporaries per block, and a refused admission discards it — a # 128k prompt queued behind a full pool paid ~2000 rounds per waiting From 2ae5d6edc45fdbf52ebf708415818a49e6fbfcf6 Mon Sep 17 00:00:00 2001 From: zejchen Date: Tue, 1 Sep 2026 07:17:24 +0000 Subject: [PATCH 069/151] fix(offload): arm state-only state loads so their failure reaches the engine _arm_joint_loads armed only loads with req.load_spec is not None (the joint KV+state loads). A state-only load -- a plain HBM-index miss where the KV is resident but the recurrent state is not (block_manager.py requesting a load with no load_spec) -- was never armed. Its success already reached the engine through the _settle_joint passthrough: the tier reports the bare request id, waits_for() is false for the unarmed id, and it falls through into finished_loading, which the engine matches on the bare id. But its failure did not. _fail_state_loads (the no-tier path, and any tier-side load failure routed through it) calls _JointPark.settle_state on an id the park never armed, which _settle ignores -- so nothing was emitted and the request sat in WAITING_FOR_REMOTE_KVS for the life of the process. The prior code's mitigation comment was wrong: reconcile_orphan_load_slots only reclaims slots of already-deallocated seqs; it never wakes a live parked request. Arm state-only loads on the state leg alone (needs_kv=False, needs_state=True). Both outcomes now land on a real park entry: settle_state releases it, and take_ready surfaces success into finished_loading and failure into failed_loading (recompute) -- under the same bare id the passthrough already used, so the engine matches it unchanged. The no-tier get_finished drain already runs _settle_joint before its early return, so a state-only load failed there is reported in the same tick rather than waiting on an abandon window that never fires for a live request. Comment at the no-tier branch updated to match. --- .../offload/hybrid/kimi_k3/connector.py | 53 ++++++++++++++----- 1 file changed, 41 insertions(+), 12 deletions(-) diff --git a/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py b/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py index 4a847c98be..b515968299 100644 --- a/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py +++ b/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py @@ -253,13 +253,37 @@ def _arm_joint_loads(self, metadata) -> None: if not state_ids or not self._do_load: return for req in metadata.requests: - if req.load_spec is not None and req.req_id in state_ids: + if req.req_id not in state_ids: + continue + if req.load_spec is not None: + # Both legs: file the park under the KV identity, because that is + # the one that has to reach the engine on finished/failed_loading. self._joint_park.arm( req.req_id, needs_kv=True, needs_state=True, kv_id=self._load_completion_id(req), ) + else: + # State-only load (no KV leg -- a plain HBM-index miss where the + # KV is resident but the recurrent state is not). Arm it too, on + # the state leg alone. Its SUCCESS already reached the engine via + # the _settle_joint passthrough (the tier reports the bare id and + # the engine waits on it), but its FAILURE did not: _fail_state_ + # loads (including the no-tier path) settled a park that was never + # armed, so nothing was emitted and the request sat in + # WAITING_FOR_REMOTE_KVS for the life of the process -- + # reconcile_orphan_load_slots only reclaims slots of already- + # deallocated seqs, it never wakes a live parked request. Arming + # on the state leg alone makes both outcomes land on a real entry: + # take_ready surfaces success into finished_loading and failure + # into failed_loading (recompute), under the same bare id the + # passthrough already used, so the engine matches it unchanged. + self._joint_park.arm( + req.req_id, + needs_kv=False, + needs_state=True, + ) def _start_state_loads(self, metadata) -> None: """Hand this step's state loads to the tier's executor. @@ -333,17 +357,22 @@ def get_finished(self): ) self._store_failed_no_tier = set() if self._state_tier is None: - # Joint loads are still armed with no tier (see `_arm_joint_loads`) - # and their state leg has already failed (`_fail_state_loads`), so - # the park owes only the KV leg. Drain it here -- before the return - # -- with empty state reports (the tier that would produce them - # never built): a landed KV completion settles the pair into - # `failed_loading` and the request recomputes, instead of flowing - # through as `finished_loading` and being miscounted a state restore - # by `Scheduler._settle_state_load(ok=True)`. This is also what keeps - # the arm from leaking: the KV completion arrives and releases the - # entry. A state-only load (no KV leg) is not armed here and is - # instead reclaimed by `reconcile_orphan_load_slots`' abandon window. + # Both joint and state-only loads were armed by `_arm_joint_loads` + # even with no tier, and `_fail_state_loads` has already failed their + # state leg. Drain the park here -- before the return -- with empty + # tier reports (the tier that would produce them never built). + # * A joint load (needs_kv) now owes only the KV leg: its landed KV + # completion settles the pair into `failed_loading` and the + # request recomputes, instead of flowing through as + # `finished_loading` and being miscounted a state restore by + # `Scheduler._settle_state_load(ok=True)`. The KV completion also + # releases the entry, so the arm cannot leak. + # * A state-only load (no KV leg) was released the moment + # `_fail_state_loads` settled its only leg, so `take_ready` + # (inside `_settle_joint`) surfaces it into `failed_loading` now + # and the request recomputes -- rather than sitting in + # WAITING_FOR_REMOTE_KVS until an abandon window that never wakes + # a live parked request. out.finished_loading, out.failed_loading = self._settle_joint( out.finished_loading, out.failed_loading, set(), set() ) From 17bf523ec2d34820e9d656d7e8368a7f277185b4 Mon Sep 17 00:00:00 2001 From: zejchen Date: Tue, 1 Sep 2026 07:24:51 +0000 Subject: [PATCH 070/151] [Kimi-K3][LMCache] Keep the engine polling while a state store is pinned A dispatched state store had no liveness signal. Once `KimiK3OffloadScheduler.build_connector_meta` drained the store out of `_pending_state_stores` and into the step's metadata, `has_pending_work` went back to False the next step -- the store was in flight in the worker but nothing on the scheduler side reported it as outstanding. If that store was the only pending work, `EngineCore.has_pending_kv_work` read False, the idle loop stopped calling `_poll_kv_transfer_progress`, and the store's source-release / index completions were never drained. The store's PAGE units then stayed pinned out of the KV pool until the `reclaim_stale_offload_pins` abandon window elapsed -- and that reclaim also runs only from `_poll_kv_transfer_progress`, so with the loop idle it never fired either. Net effect: a whole checkpoint's units held out of the pool for no reason, silent capacity loss under decode. Fix it at the liveness gate rather than in the connector, because the authoritative signal already exists in the right layer: `PagedStateCheckpointStore._offload_pins` holds one entry per dispatched store, added when `take_offload_stores` hands it to the worker and popped by `settle_offload_store` (both the source-release and the index completion call it) or by `reclaim_stale_offload_pins`. It is therefore non-empty exactly while a store is dispatched-but-unreported, and -- key point -- it cannot latch the busy loop, because reclaim shares the same dict: a lost report clears the entry when the abandon window fires. Surface it as `has_offload_pins()` on the store and its coordinator, wrap it as `BlockManager.has_pending_state_store_pins()`, and OR it into `EngineCore.has_pending_kv_work` right next to its KV-save twin `deferred_free_blocks`, whose docstring already declares that every liveness condition belongs in that one predicate. The block_manager hop is getattr-guarded for the scheduler doubles that implement only the connector surface. No new timeout and no duplicate bookkeeping in the connector: the signal and its recovery are the same data structure. Co-Authored-By: Claude Opus 4.8 --- atom/model_engine/block_manager.py | 17 +++++++++++++++++ atom/model_engine/engine_core.py | 10 ++++++++++ atom/model_engine/page_unit_checkpoint.py | 20 ++++++++++++++++++++ 3 files changed, 47 insertions(+) diff --git a/atom/model_engine/block_manager.py b/atom/model_engine/block_manager.py index 8573c2ecb8..a46cbaff83 100644 --- a/atom/model_engine/block_manager.py +++ b/atom/model_engine/block_manager.py @@ -1394,6 +1394,23 @@ def release_state_store_source(self, op) -> None: if self.paged_state_checkpoints is not None: self.paged_state_checkpoints.settle_offload_store(op) + def has_pending_state_store_pins(self) -> bool: + """Whether a dispatched state store is still awaiting its report. + + The engine's liveness gate (`EngineCore.has_pending_kv_work`) ORs this + in so a step whose only outstanding work is a state store already handed + to the worker does not read as idle. Without it the store's PAGE units + stay pinned out of the KV pool until they age out, because the report + that would release them -- and the reclaim that is the report's only + backstop -- are both drained by `_poll_kv_transfer_progress`, which the + idle loop stops calling the moment liveness goes False. This is the + unit-store twin of the `deferred_free_blocks` condition the gate already + checks for the KV save leg. + """ + if self.paged_state_checkpoints is None: + return False + return self.paged_state_checkpoints.has_offload_pins() + def reclaim_stale_state_store_pins(self, timeout_s: float) -> int: """Release store pins whose report never came. See the store.""" if self.paged_state_checkpoints is None: diff --git a/atom/model_engine/engine_core.py b/atom/model_engine/engine_core.py index be50309f3b..9c9d439729 100644 --- a/atom/model_engine/engine_core.py +++ b/atom/model_engine/engine_core.py @@ -473,6 +473,16 @@ def has_pending_kv_work(self) -> bool: return False if getattr(self.scheduler, "deferred_free_blocks", None): return True + # Unit-store twin of `deferred_free_blocks`: a state store handed to the + # worker keeps its PAGE units pinned out of the KV pool until its report + # settles them or the reclaim ages them out. Both exits run only from + # `_poll_kv_transfer_progress`, so the loop must keep polling while a + # pin is outstanding or the units leak until the abandon window. Guarded + # for the scheduler doubles that implement only the connector surface. + bm = getattr(self.scheduler, "block_manager", None) + pending_pins = getattr(bm, "has_pending_state_store_pins", None) + if pending_pins is not None and pending_pins(): + return True connector = getattr(self.scheduler, "kv_connector", None) if connector is None or not hasattr(connector, "has_pending_work"): return False diff --git a/atom/model_engine/page_unit_checkpoint.py b/atom/model_engine/page_unit_checkpoint.py index 2ff438f1c7..494c9e9f3b 100644 --- a/atom/model_engine/page_unit_checkpoint.py +++ b/atom/model_engine/page_unit_checkpoint.py @@ -486,6 +486,22 @@ def settle_offload_store(self, op: StateStoreOperationId) -> None: return self._release_offload_pin(entry[0]) + def has_offload_pins(self) -> bool: + """Whether any dispatched store is still holding its PAGE units. + + A pin lives here from the moment `take_offload_stores` hands a store to + the worker until either report path settles it (`settle_offload_store`, + driven by the source-release and index completions) or the last-resort + `reclaim_stale_offload_pins` times it out. It is therefore true exactly + while a store is dispatched-but-unreported, which is the interval the + engine must keep polling across: the completion that clears the pin -- + and the reclaim that is its only other exit -- both run only from + `_poll_kv_transfer_progress`, which the idle loop skips once liveness + reads False. Because reclaim shares this same dict, the signal cannot + latch the busy loop: a lost report clears here when the reclaim fires. + """ + return bool(self._offload_pins) + def reclaim_stale_offload_pins(self, timeout_s: float) -> int: """Release pins whose report never came, after `timeout_s`. @@ -837,6 +853,10 @@ def settle_offload_store(self, op: StateStoreOperationId) -> None: """One store reported, either way; release the units it was holding.""" self.store.settle_offload_store(op) + def has_offload_pins(self) -> bool: + """Whether any dispatched store is still pinning its units. See store.""" + return self.store.has_offload_pins() + def reclaim_stale_offload_pins(self, timeout_s: float) -> int: """Release offload pins whose report never came. See the store.""" return self.store.reclaim_stale_offload_pins(timeout_s) From 4305153d025c5ead395c69e7bd79501786285c95 Mon Sep 17 00:00:00 2001 From: zejchen Date: Tue, 1 Sep 2026 07:27:22 +0000 Subject: [PATCH 071/151] [Kimi-K3][LMCache][multi] Route the 3 load-decision forwarders through the tier sub Round-2 moved the state face (`enqueue_state_*`, `take_state_*`, `chunk_size`) onto `_state_tier_sub()` so `MultiConnectorScheduler` selects the tier by ownership rather than method presence. Three load-decision forwarders were left on plain `_first_with`: `should_park_for_load_after_alloc`, `adjust_prefill_chunk_after_alloc`, and `should_park_partial_prefill_for_load`. All three are answered by the delegating offload shell on every layout (getattr-delegated to `_impl`), so with two offload subs -- e.g. `[lmcache_offload(dense), lmcache_offload(kimi_k3)]` -- `_first_with` returns the dense shell and the request is routed by the dense `_impl`, which returns the unclamped defaults. That skips kimi_k3's `_decide_load_after_alloc` joint-boundary clamp: a partial prefill that should park for its state load proceeds, and the prefill chunk is sized without the joint bound. Prefer `_state_tier_sub()` and fall back to `_first_with`, exactly as the `chunk_size` forwarder already does. When no tier sub is configured the behaviour is unchanged, so a dense-only or `[moriio, dense]` composite still answers from the same sub as before. Co-Authored-By: Claude Opus 4.8 --- .../disaggregation/multi/multi_connector.py | 23 ++++++++++++++++--- 1 file changed, 20 insertions(+), 3 deletions(-) diff --git a/atom/kv_transfer/disaggregation/multi/multi_connector.py b/atom/kv_transfer/disaggregation/multi/multi_connector.py index 7dbe17e019..12502afeeb 100644 --- a/atom/kv_transfer/disaggregation/multi/multi_connector.py +++ b/atom/kv_transfer/disaggregation/multi/multi_connector.py @@ -532,11 +532,23 @@ def chunk_size(self) -> int | None: return getattr(c, "chunk_size", None) if c is not None else None def should_park_for_load_after_alloc(self, seq: Any) -> bool: - c = _first_with(self._connectors, "should_park_for_load_after_alloc") + # Prefer the state-tier sub, then any offload sub: this is a load + # decision routed through `_decide_load_after_alloc`, which kimi_k3 + # overrides with a joint-boundary clamp. With two offload subs + # (dense + kimi_k3) plain `_first_with` hands back the dense sub and the + # clamp is silently skipped. Mirrors the `chunk_size` forwarder. + c = self._state_tier_sub() or _first_with( + self._connectors, "should_park_for_load_after_alloc" + ) return c.should_park_for_load_after_alloc(seq) if c is not None else False def adjust_prefill_chunk_after_alloc(self, seq: Any, chunk: int) -> int: - c = _first_with(self._connectors, "adjust_prefill_chunk_after_alloc") + # State-tier sub first, for the reason in `should_park_for_load_after_alloc`: + # the prefill chunk kimi_k3 hands back is sized to the joint load it will + # arm, and the dense sub would answer with the unclamped chunk. + c = self._state_tier_sub() or _first_with( + self._connectors, "adjust_prefill_chunk_after_alloc" + ) return ( c.adjust_prefill_chunk_after_alloc(seq, chunk) if c is not None else chunk ) @@ -611,7 +623,12 @@ def take_state_source_releases(self) -> set: return c.take_state_source_releases() def should_park_partial_prefill_for_load(self, seq: Any) -> bool: - c = _first_with(self._connectors, "should_park_partial_prefill_for_load") + # State-tier sub first, for the reason in `should_park_for_load_after_alloc`: + # parking a partial prefill for its load is the state tier's decision, and + # a second offload sub must not answer it in the tier sub's place. + c = self._state_tier_sub() or _first_with( + self._connectors, "should_park_partial_prefill_for_load" + ) return c.should_park_partial_prefill_for_load(seq) if c is not None else False def should_defer_free(self, seq: Any) -> bool: From 535b8f89f0952974d07686335e4addbd6dc3eec0 Mon Sep 17 00:00:00 2001 From: zejchen Date: Tue, 1 Sep 2026 07:27:22 +0000 Subject: [PATCH 072/151] [Kimi-K3][LMCache][multi] Read the state-store pending cap from the tier sub `_state_store_pending_cap` probes `conn._max_pending_saves`, then `conn._impl._max_pending_saves`, then the `OFFLOAD_MAX_PENDING_SAVES` env default of 2. Under `kv_connector: multi` the connector is `MultiConnectorScheduler`, which exposes neither attribute, so the state leg silently fell back to 2 while the KV leg's `_may_emit_save` used the sub's real bound (e.g. 16 from `max_pending_saves`). The two legs then disagree on how much of the same pool a slow backend may pin -- the state leg throttles to a quarter of the KV leg for no configured reason, and a per-connector `max_pending_saves` override never reaches the state stores. Probe the state-tier sub as a third source, reached through the same `_state_tier_sub()` accessor the state face already routes on (the sub is itself a delegating shell, so read through its `_impl` too). Factored the conn/_impl walk into a local `_cap_of` so both the direct connector and the multi sub resolve the cap identically. No behaviour change off `multi`. Co-Authored-By: Claude Opus 4.8 --- atom/model_engine/scheduler.py | 22 +++++++++++++++++++--- 1 file changed, 19 insertions(+), 3 deletions(-) diff --git a/atom/model_engine/scheduler.py b/atom/model_engine/scheduler.py index a8dc76dbdd..c29e846301 100644 --- a/atom/model_engine/scheduler.py +++ b/atom/model_engine/scheduler.py @@ -2139,12 +2139,28 @@ def _state_store_pending_cap(self) -> int: two legs from pinning different amounts of the same pool, and honours a per-connector `"max_pending_saves"` override the env reader never sees. Falls back to the env reader for a connector (or test double) that never - computed one, and through the delegating shell's `_impl`. + computed one, and reaches the value through the delegating shell's + `_impl` and, under `kv_connector: multi`, through the state-tier sub -- + `MultiConnectorScheduler` exposes neither `_max_pending_saves` nor + `_impl`, so without the sub probe the composite silently caps the state + leg at the env default of 2 while the KV leg keeps the sub's real bound. """ + + def _cap_of(obj): + cap = getattr(obj, "_max_pending_saves", None) + if cap is None: + cap = getattr(getattr(obj, "_impl", None), "_max_pending_saves", None) + return cap + conn = self.kv_connector - cap = getattr(conn, "_max_pending_saves", None) + cap = _cap_of(conn) if cap is None: - cap = getattr(getattr(conn, "_impl", None), "_max_pending_saves", None) + # Under `multi` the bound lives on the state-tier sub (itself a + # delegating shell), reached the same way the state face is routed. + sub = getattr(conn, "_state_tier_sub", None) + tier = sub() if callable(sub) else None + if tier is not None: + cap = _cap_of(tier) if isinstance(cap, int) and not isinstance(cap, bool) and cap > 0: return cap return _offload_max_pending_saves() From e44cfe3c191a99971646d3ef12f37e36a1132751 Mon Sep 17 00:00:00 2001 From: zejchen Date: Tue, 1 Sep 2026 07:30:23 +0000 Subject: [PATCH 073/151] [Kimi-K3][LMCache] Document why should_defer_free drops the stalled save's tracker entry Review round 3 flagged the `_save_tracker.pop` inside `should_defer_free` as a mutating side-effect in a predicate and asked to delete it as redundant with the `request_finished` / `save_finished` cleanup. Tracing the callers shows it is not redundant: `Scheduler.preempt` reaches `should_defer_free` via `_is_preemptable`, and when the stall-escape returns False it frees the blocks with `block_manager.deallocate(seq)` and never calls `request_finished`. Nothing else then removes the sid from `_save_tracker`. The base save loop does not re-validate that a tracked request is still live before emitting its save; it reads `seq.block_table` straight from the tracker entry. So a lingering entry for a preempted request would, once the stall clears and `_may_emit_save` re-opens, dispatch a save reading blocks that were freed and possibly handed to another request -- a silent cross-prefix corruption, the same class of bug the reviewer is guarding against, reached by a different path. No behaviour change: keep the pop and add a comment stating exactly why it must drop the entry at the point the blocks are surrendered, and that dropping a stalled save (this request's KV goes un-offloaded rather than wedging the engine) is the intended outcome, matching `abandon_save` on the handed-out twin. Co-Authored-By: Claude Opus 4.8 --- .../offload/hybrid/kimi_k3/connector.py | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py b/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py index b515968299..a5280264b4 100644 --- a/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py +++ b/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py @@ -587,6 +587,20 @@ def should_defer_free(self, seq) -> bool: and sid not in self._save_inflight and self._has_pending_save(seq) ): + # Dropping the tracker entry here is load-bearing, not a tidy-up. + # Returning False lets the caller free the blocks, and on the + # preemption path (`Scheduler.preempt`) that free is a + # `block_manager.deallocate` with no `request_finished` call -- so + # nothing else removes this sid from `_save_tracker`. Left in, the + # base save loop would later (once the stall clears and + # `_may_emit_save` re-opens) emit a save reading `seq.block_table`, + # whose blocks are now freed and possibly reused: silent cross-prefix + # corruption. The loop does not re-check liveness, so the entry has + # to go the moment its blocks are surrendered. Dropping a stalled + # save is the intended outcome (this request's KV goes un-offloaded + # rather than wedging the engine), and `abandon_save` drops the + # matching `_save_inflight` entry on the handed-out twin for the same + # reason. self._save_tracker.pop(sid, None) return False return super().should_defer_free(seq) From 5961f0c0c4edf883469652b77b7c5f249d5d8c86 Mon Sep 17 00:00:00 2001 From: zejchen Date: Tue, 1 Sep 2026 07:31:29 +0000 Subject: [PATCH 074/151] [Kimi-K3][LMCache][test] Drop the stale _state_store_failed_locally fixture attr The scheduler test double set `_state_store_failed_locally` on its `SimpleNamespace`, but that attribute no longer exists on `KimiK3OffloadScheduler` -- no-tier store failures ride the worker's STATE_INDEX_CHANNEL completions into `_state_index_failed` now, there is no engine-side "failed locally" set to merge. Nothing reads the fixture attr, so it only misleads a reader into thinking such a field is live. Remove it. Co-Authored-By: Claude Opus 4.8 --- tests/test_lmcache_offload_connector.py | 1 - 1 file changed, 1 deletion(-) diff --git a/tests/test_lmcache_offload_connector.py b/tests/test_lmcache_offload_connector.py index e6278bb9d8..56057eae22 100644 --- a/tests/test_lmcache_offload_connector.py +++ b/tests/test_lmcache_offload_connector.py @@ -4688,7 +4688,6 @@ def _k3_scheduler() -> KimiK3OffloadScheduler: s._warned_save_stalled = False s._state_indexed = set() s._state_index_failed = set() - s._state_store_failed_locally = set() s._active_load_operations = {} s._do_save = True s._max_pending_saves = 4 From cabf99b31c499597352566fb69caf7e5dd6f3074 Mon Sep 17 00:00:00 2001 From: zejchen Date: Tue, 1 Sep 2026 07:31:29 +0000 Subject: [PATCH 075/151] [DSV4] Reject a negative state slot instead of silently wrapping `state_entry_views(slot)` indexed `self._slot_views()[slot]` directly. The slot views are a fixed-length list, so a negative slot -- e.g. the -1 "no slot" sentinel that appears on padded/aborted tokens elsewhere on this path -- does not raise; Python returns the last slot's views, and the gather or scatter then reads or writes another request's state under this request's identity. That is silent cross-request corruption with no exception to point at. Raise IndexError on a negative slot. The upstream invariant is that a real slot is always non-negative, so this only ever fires on a genuine bug, and turns what was a silent wrong-answer into a loud, locatable failure. This is the guard promised in the round-3 review reply for refuted-#4: rather than rely on the upstream invariant holding forever, make the violation observable at the point it would corrupt state. Co-Authored-By: Claude Opus 4.8 --- atom/model_ops/attentions/deepseek_v4_attn.py | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/atom/model_ops/attentions/deepseek_v4_attn.py b/atom/model_ops/attentions/deepseek_v4_attn.py index 5429ecced2..8eff9151a5 100644 --- a/atom/model_ops/attentions/deepseek_v4_attn.py +++ b/atom/model_ops/attentions/deepseek_v4_attn.py @@ -842,6 +842,14 @@ def state_transfer(self) -> StateTransfer: def state_entry_views(self, slot: int) -> list[torch.Tensor]: """One contiguous slice per plane: a V4 slot is one row per plane.""" + # Reject a negative slot explicitly. Indexing a list with -1 silently + # returns the last slot's views rather than raising, so a stray -1 (the + # "no slot" sentinel used elsewhere on this path) would gather or scatter + # another request's state under this one's identity -- silent + # cross-request corruption. Fail loudly instead; the upstream invariant + # is that a real slot is always non-negative. + if slot < 0: + raise IndexError(f"state_entry_views: invalid negative slot {slot}") return self._slot_views()[slot] def relocate_state_slots(self, pairs: Sequence[tuple[int, int]]) -> None: From f722bd820a3da054dd36dac08633a5a0db423218 Mon Sep 17 00:00:00 2001 From: zejchen Date: Tue, 1 Sep 2026 07:34:34 +0000 Subject: [PATCH 076/151] [Kimi-K3][LMCache] Fix the state_stores tuple comment: op id, not state hash The `state_stores` field on LMCacheOffloadMetadata carried a comment saying its tuples were `(state_hash, unit_ids)`. That is wrong about the key: the list is keyed by the StateStoreOperationId, not by a raw prefix hash. By the time a store lands, its originating request is long gone, so the entry cannot be keyed by request. The operation id (prefix hash plus scheduler generation) is precisely what the source-release and index-settle reports carry back -- the generation is what lets a late TP notification for one save be told apart from another save on the same prefix. Correct the comment so a reader doesn't assume a bare hash and reintroduce the cross-generation aliasing the operation id exists to prevent. --- atom/kv_transfer/offload/metadata.py | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/atom/kv_transfer/offload/metadata.py b/atom/kv_transfer/offload/metadata.py index 844b7c9830..3155d1b2df 100644 --- a/atom/kv_transfer/offload/metadata.py +++ b/atom/kv_transfer/offload/metadata.py @@ -178,9 +178,11 @@ def __init__(self) -> None: # token ids, no block ids, no chunking -- only the park/report # lifecycle, which is why it rides the metadata rather than the batch. self.state_loads: list[tuple] = [] - # (state_hash, unit_ids) for checkpoints leaving HBM for the CPU tier. - # Not keyed by request: by the time a store lands its request is long - # gone, and the hash is all the report carries back. + # (StateStoreOperationId, unit_ids) for checkpoints leaving HBM for the + # CPU tier. Keyed by the operation id, not by request: by the time a + # store lands its request is long gone, and the op (prefix hash plus + # generation) is what the source-release and index reports carry back to + # settle the pin and index the hash. self.state_stores: list[tuple] = [] def add_request(self, meta: LMCacheReqMeta) -> None: From bb997030e2a532fb10de9d60fe554a3f03496387 Mon Sep 17 00:00:00 2001 From: zejchen Date: Tue, 1 Sep 2026 07:35:26 +0000 Subject: [PATCH 077/151] [Kimi-K3][LMCache] Document the two offload env vars the scheduler reads raw scheduler.py reads OFFLOAD_MAX_PENDING_SAVES and LMCACHE_EC_PIN_TIMEOUT_SEC directly via os.environ, so neither shows up for anyone auditing the central envs.py registry -- the exact discoverability gap the registry exists to close. Neither belongs in the managed `environment_variables` dict, though: - LMCACHE_EC_PIN_TIMEOUT_SEC is LMCache's own knob. ATOM reads it only to derive the save-abandon window from the same number LMCache uses to unpin a source, which is what keeps the two ordered. Registering it here with an ATOM default would invent a second source of truth that could disagree with LMCache's. - OFFLOAD_MAX_PENDING_SAVES is the offload connector's own knob, defined and defaulted in _offload_common.py and documented in the offload README; the state tier shares it rather than adding a parallel var. So list both in the "documented only, NOT managed here" block alongside the other third-party / subsystem vars, which is exactly the convention that block already codifies for NCCL, FLA, and the vLLM legacy vars. --- atom/utils/envs.py | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/atom/utils/envs.py b/atom/utils/envs.py index 6331c87027..4a77ecdb3e 100644 --- a/atom/utils/envs.py +++ b/atom/utils/envs.py @@ -623,3 +623,14 @@ def __getattr__(name: str): # FLA_TRIL_PRECISION — FLA ops library # VLLM_PP_LAYER_PARTITION — vLLM legacy (still active in models/utils.py) # VLLM_USE_MODELSCOPE — vLLM legacy (benchmarks) +# LMCACHE_EC_PIN_TIMEOUT_SEC — LMCache library's own source-pin timeout; +# read in scheduler.py only to derive the +# engine's save-abandon window from it, so the +# two stay ordered. ATOM does not own the +# knob, hence no default of its own here. +# OFFLOAD_MAX_PENDING_SAVES — offload connector queue-depth bound; +# defined/defaulted in +# kv_transfer/offload/_offload_common.py and +# documented in kv_transfer/offload/README.md. +# The state tier shares it (scheduler.py) +# rather than adding a second knob. From 6a8e8ce736fb93e980354422357bc2487552510d Mon Sep 17 00:00:00 2001 From: zejchen Date: Tue, 1 Sep 2026 07:38:07 +0000 Subject: [PATCH 078/151] [Kimi-K3][LMCache] Annotate _store_failed_no_tier with what it actually holds The set was annotated `set[int]`, but the values put into it are the `op` element of each `state_stores` tuple -- a `StateStoreOperationId` (a frozen `(prefix_hash, generation)` dataclass), not a bare int. The two places that drain it (`get_finished`) hand each element straight to `ConnectorCompletion` as a `ConnectorCompletionId`, which is where the real type matters: a reader who trusts the `int` annotation would think a raw hash flows here and miss that the generation is carried too -- the very distinction that keeps a re-store of the same prefix from being tombstoned as a duplicate. Correct the annotation to `set[StateStoreOperationId]` and import the type. Annotation-only change; `from __future__ import annotations` already makes it lazy, so there is no runtime cost. --- atom/kv_transfer/offload/hybrid/kimi_k3/connector.py | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py b/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py index a5280264b4..a6dde6ab7a 100644 --- a/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py +++ b/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py @@ -14,7 +14,10 @@ import os import time -from atom.kv_transfer.disaggregation.types import ConnectorCompletion +from atom.kv_transfer.disaggregation.types import ( + ConnectorCompletion, + StateStoreOperationId, +) from atom.kv_transfer.offload import config as offcfg from atom.kv_transfer.offload._offload_common import ( max_pending_saves, @@ -73,7 +76,7 @@ def __init__(self, config) -> None: # from the worker, which does not exist here and raised AttributeError # on the no-tier store path, taking the whole step's KV loads/saves down # with it (super().start_load_kv never ran). - self._store_failed_no_tier: set[int] = set() + self._store_failed_no_tier: set[StateStoreOperationId] = set() def register_kv_caches( self, kv_caches: dict, transfer_tensors=None, num_blocks: int | None = None From 1e14a8169db6468a2aafc2b0ffb654fd479fa3e6 Mon Sep 17 00:00:00 2001 From: zejchen Date: Tue, 1 Sep 2026 07:38:29 +0000 Subject: [PATCH 079/151] [stats] Correct the served-reuse invariant in two hit-rate docstrings Both `lmcache_hit_rate` and the [Cache Tiers] log line justified their partition claim with `cached + offload == num_cached <= reusable`. The `== num_cached` clause is simply false: it would force `offload == 0`, and if it held there would be nothing to partition. `num_cached` here is `cached` itself, so the equality is `cached + offload == cached`. The real invariant is the one `update_cache` actually enforces -- its clamp bounds `num_offload_tokens` to `reusable - cached`, giving `cached + offload <= reusable`. That inequality is exactly what makes `cache_hit_rate` and `lmcache_hit_rate` a partition of served reuse against the shared `reusable` denominator, so `combined` cannot exceed 100%. Restate both docstrings to the enforced inequality and point at the clamp. --- atom/model_engine/engine_stats.py | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/atom/model_engine/engine_stats.py b/atom/model_engine/engine_stats.py index b931eb53b6..51f61a0085 100644 --- a/atom/model_engine/engine_stats.py +++ b/atom/model_engine/engine_stats.py @@ -461,8 +461,9 @@ def lmcache_hit_rate(self) -> float: those read as "the HBM number" and only `cache_hit_rate` is a served quantity: `paged_hit_rate`'s numerator is `compressed`, how far the prefix walk reached *before* the state gates cut it. `cached + offload - == num_cached <= reusable` by construction, so `cache_hit_rate` and this - partition the served reuse; `compressed + offload` does not and can + <= reusable` by construction (the clamp in `update_cache` enforces it), + so `cache_hit_rate` and this partition the served reuse; `compressed + + offload` does not and can exceed the denominator. """ return self._rate(self.total_offload_tokens, self.total_reusable_tokens) @@ -625,8 +626,9 @@ def _log_pools(self) -> None: # (`compressed`). Both read as "the HBM number" and only one of them is # a *served* quantity: `compressed` is how far the prefix walk reached # before the state gates cut it, so it is reach, not reuse anybody got. - # `cached + offload == num_cached <= reusable` by construction, so these - # two do partition, and `combined` is exactly the end-to-end rate. + # `cached + offload <= reusable` by construction (clamped in + # `update_cache`), so these two do partition, and `combined` is exactly + # the end-to-end rate. if self.total_offload_tokens: served = self.total_cached_tokens + self.total_offload_tokens logger.info( From 7bca18989c75e29fd50e3edfa20a68c98a0f4001 Mon Sep 17 00:00:00 2001 From: zejchen Date: Tue, 1 Sep 2026 07:42:43 +0000 Subject: [PATCH 080/151] [Kimi-K3][LMCache][docs] Document the offload env vars and the engine-side index The round-3 review noted 7,297 insertions with docs/ untouched. Three concrete gaps, closed here (the README's layout prose already lists all three layouts): - docs/environment_variables.md gains an "LMCache offload tier" subsection for LMCACHE_EC_PIN_TIMEOUT_SEC and OFFLOAD_MAX_PENDING_SAVES. Both are read raw via os.environ (neither is ATOM's default to own -- see the matching note in envs.py), so they were invisible from the central env reference; the subsection makes them discoverable while stating why they bypass the registry. - The offload README's module map documented every kv_transfer/offload file but not state_offload.py, which lives in model_engine. It is the engine-side counterpart the state tier reports to, and state_tier.py's own row already names StateOffloadIndex, so add a short cross-reference after the table rather than pull an out-of-directory file into a directory-scoped map. --- atom/kv_transfer/offload/README.md | 5 +++++ docs/environment_variables.md | 15 +++++++++++++++ 2 files changed, 20 insertions(+) diff --git a/atom/kv_transfer/offload/README.md b/atom/kv_transfer/offload/README.md index 7d353e6e6e..475e2d73d3 100644 --- a/atom/kv_transfer/offload/README.md +++ b/atom/kv_transfer/offload/README.md @@ -88,6 +88,11 @@ Four rules carry the module: | `hybrid/kimi_k3/state_tier.py` | Worker-side store/load driver for the state tier on its own executor; reports store/finished/failed hash sets for the engine-side `StateOffloadIndex` to apply. | | `atom_lmcache_staging.py` | Per-thread CUDA streams, staging buffer, ready/free events, env helpers. | +The engine-side counterpart of the state tier lives outside this directory: +`atom/model_engine/state_offload.py` holds `StateOffloadIndex`, which applies the +store/finished/failed hash sets that `hybrid/kimi_k3/state_tier.py` reports back, +so a later prefix hit knows which checkpoints the CPU tier can actually serve. + ## Architecture The connector is split across two processes, mirroring ATOM's P/D split: diff --git a/docs/environment_variables.md b/docs/environment_variables.md index be971c8a3c..abe5b44b20 100644 --- a/docs/environment_variables.md +++ b/docs/environment_variables.md @@ -183,6 +183,21 @@ flag below. Details in the state-checkpoint section of the |----------|------|---------|-------------| | **ATOM_STATE_CHECKPOINT_DEMAND** | bool | 1 (true) | Set to `0` to stop a prefix hit that was refused for want of a checkpoint from placing a rung of its own, leaving the prompt-end anchor as the only placement. Overrides `--state-checkpoint-demand`, so the policy can be A/B'd without editing a launch script. The rung is most of the checkpoint write traffic and little of the read-back, and every write evicts something — `StateSlotPool.mark_speculative` carries the measurement. | +### LMCache offload tier + +Two knobs that govern the LMCache CPU/NVMe offload connector are read directly +via `os.environ` rather than through `atom.utils.envs`, because ATOM does not +own either default: one belongs to the LMCache library, the other to the +offload connector itself (defined in +`atom/kv_transfer/offload/_offload_common.py` and documented in full in +`atom/kv_transfer/offload/README.md`). They are listed here so they are +discoverable from the central env reference despite bypassing the registry. + +| Variable | Type | Default | Description | +|----------|------|---------|-------------| +| **LMCACHE_EC_PIN_TIMEOUT_SEC** | float | LMCache's own (300) | LMCache's source-pin timeout. ATOM reads it only to derive the engine's save-abandon window (`pin + 30s`), so the two stay ordered — a lost store report is reclaimed only after LMCache would already have force-unpinned its source. Non-positive disables ATOM's reclamation. ATOM sets no default of its own; when unset it assumes LMCache's. | +| **OFFLOAD_MAX_PENDING_SAVES** | int | `max(2, 2 × OFFLOAD_COPY_WORKERS)` | Bound on total in-flight offload transfers (running + queued) held before a SLOT snapshot or executor submission. A KV save and a state store both pin bytes out of the same pool while they run, so the KV leg and the K3 state tier share this one number rather than each carrying its own. | + ## Profiling & debugging | Variable | Type | Default | Description | From 6fd43af10ee540268d90c27d5040a062df38d7df Mon Sep 17 00:00:00 2001 From: zejchen Date: Tue, 1 Sep 2026 07:49:23 +0000 Subject: [PATCH 081/151] [Kimi-K3][LMCache][test] Close two gaps in the shell-coverage sweep The source-scan test that guards the delegating shell had two ways to pass without actually covering the shell: 1. Vacuous partial-pass. The sweep builds `probed` from four regexes over the scheduler source, then asserts only `probed` is non-empty before checking each name resolves on the shell. Hoisting a single `getattr(self.kv_connector, "x")` into a local alias makes that one read stop matching -- `probed` shrinks but stays non-empty, so the test still passes and `x` is silently no longer covered. Anchor on the members we know are read through the shell today (the past incidents plus the flags); a rewrite that hides one of them now fails loudly here instead of vacuously. Each anchored name is currently matched by one of the four patterns, verified against the scheduler source. 2. It only covered one of the shell's three forwarding shapes. The sweep checks that members the scheduler reads exist on the shell, and a sibling test checks the getattr-guarded fallbacks unpack correctly. Neither covered the bare `return self._impl.NAME(...)` / void `self._impl.NAME(...)` forward -- the shape that raises AttributeError inside a scheduler step when the `_impl` chosen for a layout does not define NAME. Because that step runs on every TP rank, one rank dies and the survivors block forever in the next collective. Add `test_every_unconditional_shell_forward_exists_on_every_impl`: it AST-scans the shell's own source for those unconditional forwards (reading the source, not a hand-kept list, for the same reason the existing sweep does) and holds each forwarded name against all three concrete impls (dense / kimi_k3 / dsv4) through their full MRO. There are 11 such forwards today and all 11 resolve on every impl; the test locks that in so a new unconditional forward added without the matching impl method fails here. Co-Authored-By: Claude Opus 4.8 --- tests/test_lmcache_offload_connector.py | 114 ++++++++++++++++++++++++ 1 file changed, 114 insertions(+) diff --git a/tests/test_lmcache_offload_connector.py b/tests/test_lmcache_offload_connector.py index 56057eae22..a317cd5cb0 100644 --- a/tests/test_lmcache_offload_connector.py +++ b/tests/test_lmcache_offload_connector.py @@ -5135,6 +5135,34 @@ def test_every_member_the_scheduler_reads_is_reachable_through_the_shell(): probed.discard("_impl") assert probed, "the scan found nothing -- it has stopped matching the source" + # A regex sweep degrades silently: hoist one `getattr(self.kv_connector, + # "x")` into a local alias and that read simply stops matching, shrinking + # `probed` without emptying it, so `assert probed` above still passes and + # `x` is quietly no longer covered. Anchor on members we know are read + # through the shell today -- the past incidents plus the flags -- so a + # rewrite that hides one of them fails here loudly instead of vacuously. + # Each name below is currently matched by one of the four patterns; if the + # scheduler stops reading it (a real removal), delete it from this set in + # the same change, deliberately. + must_be_seen = { + "should_defer_free", + "abandon_save", + "request_finished", + "build_connector_meta", + "process_completions", + "chunk_size", + "enqueue_state_stores", + "take_state_reports", + "take_state_source_releases", + "is_offload", + } + lost = sorted(must_be_seen - probed) + assert not lost, ( + "the scan no longer sees these known reads -- the scheduler now reaches " + f"them in a form the regex does not match, so coverage silently dropped: " + f"{lost}. Update the patterns (or this anchor set) intentionally." + ) + shell = LMCacheOffloadConnectorScheduler missing = sorted(n for n in probed if not hasattr(shell, n)) assert not missing, ( @@ -5176,6 +5204,92 @@ def test_the_shells_no_impl_fallbacks_match_what_the_caller_unpacks(): assert shell.adjust_prefill_chunk_after_alloc(None, 7) == 7 +def test_every_unconditional_shell_forward_exists_on_every_impl(): + """The other half of the shell contract: a forward that does NOT guard. + + Two tests above cover the members the scheduler reads off the shell, and the + getattr-guarded fallbacks the shell defines. Neither covers the third shape: + a shell method whose body is a bare `return self._impl.NAME(...)`. When the + variant `_impl` chosen for a layout does not define `NAME`, that forward + raises AttributeError inside a scheduler step -- and because the step runs on + every TP rank, one rank dies and the survivors block forever in the next + collective. The shell has three `_impl` types (dense / kimi_k3 / dsv4), so an + unconditional forward is only safe if the method resolves on all three. + + Scan the shell's own source for those bare forwards -- reading the source, + not a hand-kept list, for the same reason the sweep above does -- and hold + each forwarded name against every concrete impl through its full MRO. + """ + import ast + import inspect + + from atom.kv_transfer.offload.connector import LMCacheOffloadConnectorScheduler + from atom.kv_transfer.offload.hybrid.dsv4.connector import DSV4OffloadScheduler + + shell_src = inspect.getsource(LMCacheOffloadConnectorScheduler) + shell_ast = ast.parse(shell_src) + (shell_cls,) = [n for n in shell_ast.body if isinstance(n, ast.ClassDef)] + + def _is_unconditional_forward(fn): + """A method whose ONLY statement (past a docstring) forwards straight to + `self._impl.(...)` -- either `return self._impl.(...)` or the + bare void call `self._impl.(...)`. No getattr, no `if callable`: if + the impl lacks ``, this raises AttributeError at call time.""" + body = [ + s + for s in fn.body + if not (isinstance(s, ast.Expr) and isinstance(s.value, ast.Constant)) + ] + if len(body) != 1: + return None + stmt = body[0] + if isinstance(stmt, ast.Return): + call = stmt.value + elif isinstance(stmt, ast.Expr): # void forward, returns None implicitly + call = stmt.value + else: + return None + if not isinstance(call, ast.Call): + return None + target = call.func # self._impl. + if ( + isinstance(target, ast.Attribute) + and isinstance(target.value, ast.Attribute) + and target.value.attr == "_impl" + and isinstance(target.value.value, ast.Name) + and target.value.value.id == "self" + ): + return target.attr + return None + + forwards = sorted( + { + name + for fn in shell_cls.body + if isinstance(fn, (ast.FunctionDef, ast.AsyncFunctionDef)) + for name in [_is_unconditional_forward(fn)] + if name + } + ) + assert forwards, "found no unconditional forwards -- the scan stopped matching" + + impls = { + "dense": DenseOffloadScheduler, + "kimi_k3": KimiK3OffloadScheduler, + "dsv4": DSV4OffloadScheduler, + } + broken = { + layout: [name for name in forwards if not hasattr(cls, name)] + for layout, cls in impls.items() + } + broken = {layout: names for layout, names in broken.items() if names} + assert not broken, ( + "the shell forwards these unconditionally, but the named `_impl` does " + "not define them, so the forward raises AttributeError mid-step and " + f"wedges the TP group for that layout: {broken}" + ) + + # ── kimi_k3: the two joint legs report different identities ─────────────── From 9a32cff9a13e9a60d380ef1b195c76a9467998ae Mon Sep 17 00:00:00 2001 From: zejchen Date: Tue, 1 Sep 2026 08:10:33 +0000 Subject: [PATCH 082/151] [Kimi-K3][LMCache] Isolate each offload sub in multi.process_completions MultiConnectorScheduler.process_completions fanned a single mutable KVConnectorOutput through every sub in sequence. That is unsafe because OffloadSchedulerMixin.process_completions is destructive: it replaces finished_loading / failed_loading / finished_saving with only the operations it recognises, and .clear()s connector_completions wholesale (_offload_common.py). Run sequentially, the first offload sub therefore drops the loads and clears the completion channels a later sub owns. For the [dense, kimi_k3] composite this silently loses every K3 state completion: dense filters finished_loading down to its own KV loads and clears the k3_state_index / k3_state_source channels before kimi_k3's process_completions ever sees them. The parked joint request never wakes, settle_state_store never runs, and the PAGE units stay pinned until the stale reclaimer marks the checkpoint image untrusted. Fix: give each offload sub a private copy of only the four fields the mixin rewrites (a shallow dataclasses.replace with fresh sets; the sending/recving fields it never touches stay shared), run each sub on its own copy, then merge the four fields back by union. A sub now only ever consumes what it owns, so one sub's .clear() can no longer eat another's completions. The single-handler case -- the common [producer, offload] shape -- keeps the direct, copy-free call, so it is byte-for-byte the old behaviour and cannot regress. Test: tests/test_multi_connector.py adds DestructiveSub, a faithful model of the mixin's destructive rewrite, and asserts that with two destructive offload subs each owning a distinct load id and completion channel, both loads survive the fan-out and each sub settles exactly its own channel (both fail under the old sequential fan). A second test pins the single-sub fast path. Co-Authored-By: Claude Opus 4.8 --- .../disaggregation/multi/multi_connector.py | 55 ++++++++++-- tests/test_multi_connector.py | 86 +++++++++++++++++++ 2 files changed, 136 insertions(+), 5 deletions(-) diff --git a/atom/kv_transfer/disaggregation/multi/multi_connector.py b/atom/kv_transfer/disaggregation/multi/multi_connector.py index 12502afeeb..f8deabacd9 100644 --- a/atom/kv_transfer/disaggregation/multi/multi_connector.py +++ b/atom/kv_transfer/disaggregation/multi/multi_connector.py @@ -61,6 +61,7 @@ from __future__ import annotations import copy +import dataclasses import logging from typing import Any @@ -650,16 +651,60 @@ def has_pending_work(self) -> bool: ) def process_completions(self, output: KVConnectorOutput) -> KVConnectorOutput: - """Let every sub apply its own completions and normalize the output. + """Let every offload sub apply its own completions and normalize output. Only offload defines this. Without the fan-out its save/load bookkeeping never clears and raw operation ids reach the scheduler, which looks requests up by bare id. + + `OffloadSchedulerMixin.process_completions` is *destructive*: it + replaces `finished_loading`/`failed_loading`/`finished_saving` with only + the operations it recognises and `.clear()`s `connector_completions` + wholesale. Fanning one shared `output` through the subs in sequence + therefore lets the first offload sub drop the loads and clear the + completion channels the later subs own -- so a `[dense, kimi_k3]` + composite silently loses every K3 state completion: the parked request + never wakes, `settle_state_store` never runs, and the PAGE units stay + pinned until the stale reclaimer marks the image untrusted. + + Give each offload sub a private copy of the mutated sets and merge the + results, so a sub only ever consumes what it owns. The single-offload + case (the common `[producer, offload]` shape) keeps the direct call -- + no copy, byte-for-byte the old behaviour. """ - for c in self._connectors: - handler = getattr(c, "process_completions", None) - if callable(handler): - output = handler(output) + handlers = [ + handler + for c in self._connectors + if callable(handler := getattr(c, "process_completions", None)) + ] + if not handlers: + return output + if len(handlers) == 1: + return handlers[0](output) + + merged_loading: set = set() + merged_failed_loading: set = set() + merged_saving: set = set() + merged_completions: set = set() + for handler in handlers: + # Only the four fields the mixin rewrites need isolating; the rest + # (sending/recving) it never touches, so they stay shared. + sub_out = dataclasses.replace( + output, + finished_loading=set(output.finished_loading), + failed_loading=set(output.failed_loading), + finished_saving=set(output.finished_saving), + connector_completions=set(output.connector_completions), + ) + sub_out = handler(sub_out) + merged_loading |= sub_out.finished_loading + merged_failed_loading |= sub_out.failed_loading + merged_saving |= sub_out.finished_saving + merged_completions |= sub_out.connector_completions + output.finished_loading = merged_loading + output.failed_loading = merged_failed_loading + output.finished_saving = merged_saving + output.connector_completions = merged_completions return output def save_finished(self, req_id: Any) -> None: diff --git a/tests/test_multi_connector.py b/tests/test_multi_connector.py index 6b3f8ccd1a..c5e694e047 100644 --- a/tests/test_multi_connector.py +++ b/tests/test_multi_connector.py @@ -169,6 +169,39 @@ def __getattribute__(self, name): return object.__getattribute__(self, name) +class DestructiveSub: + """Faithful model of ``OffloadSchedulerMixin.process_completions``. + + Unlike ``FakeSchedSub`` (which only appends to ``finished_saving``), this + reproduces the *destructive* behaviour the real mixin has: it filters + ``finished_loading`` down to the one load it owns and ``.clear()``s + ``connector_completions`` wholesale after skimming its own channel. Two of + these in one composite is the ``[dense, kimi_k3]`` shape from the review — + each sub rewrites the shared output and eats what the other owns. + """ + + is_offload = True + + def __init__(self, *, owned_load, owned_channel): + self._owned_load = owned_load + self._owned_channel = owned_channel + self.settled_channels = [] + + def process_completions(self, output): + # Keep only the load this sub owns; the mixin drops the rest as "not + # mine" (dense/connector.py load_finished returns False on foreign ids). + output.finished_loading = { + v for v in output.finished_loading if v == self._owned_load + } + # Consume only this sub's completion channel... + for completion in output.connector_completions: + if completion.channel == self._owned_channel: + self.settled_channels.append(completion.channel) + # ...but clear ALL of them, exactly as the mixin does. + output.connector_completions.clear() + return output + + class FakeWorkerSub: """Worker-side sub-connector mock.""" @@ -371,6 +404,59 @@ def test_process_completions_reaches_the_offload_sub(): assert out.finished_saving == {9} +def test_process_completions_isolates_each_offload_sub(): + # [dense, kimi_k3]: BOTH subs define process_completions, and each one is + # destructive — it filters finished_loading down to its own and .clear()s + # every completion channel. Fanning one mutable output through them in + # sequence lets the first sub eat the loads and completion channels the + # second owns, so a K3 state completion silently vanishes and the parked + # request never wakes. Each sub must operate on a private copy; the union + # of their results is what reaches the scheduler. + dense = DestructiveSub(owned_load="dense_load", owned_channel="dense_ch") + k3 = DestructiveSub(owned_load="k3_load", owned_channel="k3_state_index") + sched = _sched([dense, k3]) + + out = sched.process_completions( + KVConnectorOutput( + finished_loading={"dense_load", "k3_load"}, + connector_completions={ + ConnectorCompletion("dense_ch", SaveOperationId(1, 0), True), + ConnectorCompletion("k3_state_index", SaveOperationId(2, 0), True), + }, + ) + ) + + # Both loads survive the fan-out (sequential fan would drop k3_load)... + assert out.finished_loading == {"dense_load", "k3_load"} + # ...and each sub settled exactly its own channel (sequential fan would + # leave k3.settled_channels empty — dense cleared them first). + assert dense.settled_channels == ["dense_ch"] + assert k3.settled_channels == ["k3_state_index"] + # Every completion channel was consumed by its owner; none leak onward. + assert out.connector_completions == set() + + +def test_process_completions_single_offload_sub_is_called_directly(): + # The common [producer, offload] shape has exactly one process_completions + # handler. That must stay the direct, copy-free path so its in-place + # rewrites reach the caller unchanged. + only = DestructiveSub(owned_load="x", owned_channel="ch") + sched = _sched([FakeSchedSub(is_producer=True), only]) + + out = sched.process_completions( + KVConnectorOutput( + finished_loading={"x", "y"}, + connector_completions={ + ConnectorCompletion("ch", SaveOperationId(3, 0), True) + }, + ) + ) + + assert out.finished_loading == {"x"} + assert only.settled_channels == ["ch"] + assert out.connector_completions == set() + + def test_offload_methods_default_when_no_sub_implements(): sched = _sched([FakeSchedSub(is_producer=True), FakeSchedSub()]) seq = object() From 367792edb6cad0e5f66f35c5299a62edf605d5b3 Mon Sep 17 00:00:00 2001 From: zejchen Date: Tue, 1 Sep 2026 08:14:20 +0000 Subject: [PATCH 083/151] [scheduler] Re-assert the multimodal atomic-prefill invariant after alloc A multimodal prefill must forward in a single chunk: the vision embeddings are computed for the whole prompt and scattered onto its placeholder positions, so a partial chunk either misses the placeholders or lands them at the wrong offsets (and a later chunk, once `multimodal_data` has been cleared, embeds raw `<|media_pad|>` tokens into the KV cache). The existing guard enforced this only *before* allocation, against the pre-allocation chunk size. But two adjusters run after that guard and after allocate, and both can shorten the chunk: - `_adjust_prefill_chunk_after_alloc` defers part of a prefill for an offload load; - `_finalize_prefill_chunk` lands the chunk on a state-checkpoint rung. So a multimodal prompt that passed the pre-alloc check could still be split by either adjuster and scheduled as a partial chunk -- silent corruption whenever a multimodal model is also served through the offload connector or carries per-request state. Fix: after both adjusters run, re-check the invariant against the final chunk and the post-allocation `num_cached_tokens`. If a multimodal prompt would be split, undo the admission and requeue it whole -- the same deallocate + appendleft + break the joint-disown path already uses after the connector has been notified (re-admission rebuilds the connector's per-request state, so this is safe). Test: tests/test_scheduler.py stubs the post-alloc adjuster to shorten a whole-prompt chunk and asserts a multimodal prompt is requeued whole (not split), with a non-multimodal control that still chunks as usual. Co-Authored-By: Claude Opus 4.8 --- atom/model_engine/scheduler.py | 18 ++++++++++++ tests/test_scheduler.py | 52 ++++++++++++++++++++++++++++++++++ 2 files changed, 70 insertions(+) diff --git a/atom/model_engine/scheduler.py b/atom/model_engine/scheduler.py index c29e846301..98969acb87 100644 --- a/atom/model_engine/scheduler.py +++ b/atom/model_engine/scheduler.py @@ -1457,6 +1457,24 @@ def _schedule(self) -> tuple[ScheduledBatch, dict[int, Sequence]] | None: chunk = self._adjust_prefill_chunk_after_alloc(seq, chunk) chunk = self._finalize_prefill_chunk(seq, seq.num_cached_tokens, chunk) + # Re-assert the atomic-prefill invariant against the *final* chunk. + # The guard above ran on the pre-allocation size, but the two + # adjusters just above can both shorten the chunk after the fact -- + # `_adjust_prefill_chunk_after_alloc` defers part of a prefill for an + # offload load, and `_finalize_prefill_chunk` lands the chunk on a + # state-checkpoint rung. Either can leave a multimodal prompt split + # even though it passed the pre-alloc check, which would scatter the + # vision embeddings against the wrong positions (or, on a later + # chunk once `multimodal_data` is cleared, embed raw `<|media_pad|>` + # into the KV cache). Take it whole or not at all: undo this + # admission and wait for a step that can, exactly as the joint-disown + # path above requeues. `num_cached_tokens` is post-allocation, so + # this stays correct even if the prefix was disowned. + if atomic_prefill and chunk < seq.num_tokens - seq.num_cached_tokens: + self.block_manager.deallocate(seq) + self.waiting.appendleft(seq) + break + self._assert_positive_prefill_chunk(chunk, num_new_tokens, budget_remaining) num_seqs_prefill, num_batched_tokens = self._schedule_prefill_seq( seq, diff --git a/tests/test_scheduler.py b/tests/test_scheduler.py index 03a8a6ca08..f4fb6157a2 100644 --- a/tests/test_scheduler.py +++ b/tests/test_scheduler.py @@ -633,6 +633,58 @@ def test_chunked_prefill_splits_prompt_across_steps(self, seq_factory): assert list(batch2.scheduled_tokens) == list(range(6, 10)) assert list(batch2.num_cached_tokens) == [6] + def test_multimodal_prefill_shortened_after_alloc_is_requeued_whole( + self, seq_factory + ): + # A multimodal prompt must forward in one chunk (its vision embeddings + # are scattered onto placeholder positions for the whole prompt). The + # pre-allocation atomic guard enforces that, but a post-allocation + # adjuster -- offload chunk deferral or state-checkpoint alignment -- + # can shorten the chunk *after* that guard passed. When it does, the + # prompt must be requeued whole, not split into a partial chunk that + # would scatter the embeddings against the wrong positions. + sched = Scheduler( + MockConfig( + max_num_batched_tokens=64, + num_kvcache_blocks=100, + kv_cache_block_size=4, + enable_chunked_prefill=True, + ) + ) + # Whole 8-token prompt clears the pre-alloc guard (budget 64 >= 8); + # only the post-alloc adjuster shortens it. + sched._adjust_prefill_chunk_after_alloc = lambda seq, chunk: 4 + seq = seq_factory(list(range(8)), multimodal_data={"pixel_values": object()}) + sched.add(seq) + + batch, _ = sched.schedule() + + # Not split into a 4-token partial chunk -- deferred whole. + assert batch.total_seqs_num_prefill == 0 + assert seq.is_partial_prefill is False + assert seq in sched.waiting + + def test_text_prefill_shortened_after_alloc_still_splits(self, seq_factory): + # Control for the guard above: a non-multimodal prompt shortened by the + # same post-alloc adjuster is chunked as usual -- the requeue is + # multimodal-only. + sched = Scheduler( + MockConfig( + max_num_batched_tokens=64, + num_kvcache_blocks=100, + kv_cache_block_size=4, + enable_chunked_prefill=True, + ) + ) + sched._adjust_prefill_chunk_after_alloc = lambda seq, chunk: 4 + seq = seq_factory(list(range(8))) + sched.add(seq) + + batch, _ = sched.schedule() + + assert batch.total_seqs_num_prefill == 1 + assert list(batch.num_scheduled_tokens) == [4] + def test_prefill_respects_block_availability(self, seq_factory): sched = Scheduler(MockConfig(num_kvcache_blocks=1, kv_cache_block_size=4)) sched.add(seq_factory([1, 2, 3, 4])) # 1 block From b9b0c03744631134c339160f9ab226428bab4bfc Mon Sep 17 00:00:00 2001 From: zejchen Date: Tue, 1 Sep 2026 08:22:55 +0000 Subject: [PATCH 084/151] [Kimi-K3][LMCache][test] Run the PAGE-unit geometry on the CPU tier MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit §8.2. The PAGE-unit address/view arithmetic is the CPU tier's only seam onto the checkpoint image: given logical block ids it decides which bytes -- and which tensor views -- of the MLA pool every checkpoint copy lands on. Getting it wrong scrambles 127 blocks of state or oversizes the store MemoryObj. It is pure arithmetic over `self.model_runner`, so it can and should be exercised on a runner with no GPU. It was not. The four methods lived on `_KimiMLAGDNCommon` in kimi_mla_gdn_attn.py, whose module imports aiter at load; the test file that covered them was `importorskip`ped whole on CI, so the three PAGE-unit test classes never ran there -- the seam that decides where every checkpoint byte goes had zero CI coverage. Extract the arithmetic into a new aiter-free module, `page_unit_geometry.py`, as `PageUnitGeometryMixin`, and mix it into `_KimiMLAGDNCommon` ahead of `GDNStateMixin` (whose stubs raise NotImplementedError for a pool it does not own -- the mixin must precede it so the real methods win the MRO). Behaviour is byte-for-byte identical; the methods moved verbatim. Move the three PAGE-unit test classes to a new `tests/test_page_unit_geometry.py` that imports the pure mixin directly (no importorskip), so they run on the non-GPU CI runner. `test_kda_checkpoint_slot_copy.py` keeps the store/restore round-trip, which still needs the concrete K3 builder and stays importorskip'd. tests/test_page_unit_geometry.py + tests/test_kda_checkpoint_slot_copy.py 36 passed --- .../model_ops/attentions/kimi_mla_gdn_attn.py | 171 +--------- .../attentions/page_unit_geometry.py | 200 +++++++++++ tests/test_kda_checkpoint_slot_copy.py | 284 ---------------- tests/test_page_unit_geometry.py | 311 ++++++++++++++++++ 4 files changed, 513 insertions(+), 453 deletions(-) create mode 100644 atom/model_ops/attentions/page_unit_geometry.py create mode 100644 tests/test_page_unit_geometry.py diff --git a/atom/model_ops/attentions/kimi_mla_gdn_attn.py b/atom/model_ops/attentions/kimi_mla_gdn_attn.py index f98de4638a..8887aea345 100644 --- a/atom/model_ops/attentions/kimi_mla_gdn_attn.py +++ b/atom/model_ops/attentions/kimi_mla_gdn_attn.py @@ -1,8 +1,6 @@ # SPDX-License-Identifier: MIT # Copyright (C) 2024-2025, Advanced Micro Devices, Inc. All rights reserved. -from collections.abc import Sequence - import numpy as np import torch from aiter import dtypes @@ -16,6 +14,7 @@ from .aiter_mla import AiterMLAMetadataBuilder from .backends import AttentionBackend from .gdn_attn import GDNStateMixin +from .page_unit_geometry import PageUnitGeometryMixin from .sub_pool_spec import SubPoolSpec, page_pool from .triton_mla import TritonMLAMetadataBuilder @@ -36,7 +35,7 @@ def get_impl_cls() -> type["MLAAttention"]: return MLAAttention -class _KimiMLAGDNCommon(GDNStateMixin): +class _KimiMLAGDNCommon(PageUnitGeometryMixin, GDNStateMixin): def __init__(self, model_runner): super().__init__(model_runner=model_runner) self.mla_idx_by_layer = { @@ -158,172 +157,6 @@ def allocate_kv_cache_tensors( ) } - def _page_unit_regions(self) -> tuple[np.ndarray, np.ndarray]: - """Base address and per-unit stride of every region a PAGE id owns. - - The destination side of a checkpoint copy. `GDNStateMixin` knows where - a state slot's bytes are; this knows where a KV block's are, because - this class owns the MLA pool. - - `kv_cache` is `(rows, physical_blocks, physical_block_size, entry)`, - so a block owns one contiguous region per row and the rows are a fixed - stride apart. Affine in the block id, and a property of the pool rather - than of any block, so it is worked out once. - - The units are the trap. `unit_ids` carries **logical** block ids -- what - `BlockPool` hands out and what `sub_pool_specs` priced -- while the - tensor is shaped in **physical** blocks, and K3's `block_ratio` is 128. - So a region is `runner.block_size` tokens wide, not - `physical_block_size`, and the two differ by exactly that ratio. The - assertion below is what makes a mix-up a startup error rather than 127 - blocks of scrambled state: it is the one relation that cannot hold if - the granularity is wrong. - """ - runner = self.model_runner - cache = runner.kv_cache - owner = cache.data_ptr() - cached = getattr(self, "_page_unit_region_cache", None) - if cached is not None and cached[0] == owner: - return cached[1] - - if not cache.is_contiguous(): - raise RuntimeError("the MLA pool must be contiguous to be copied") - item = cache.element_size() - entry = cache.shape[3] - rows = cache.shape[0] - # One logical block's bytes inside one row. - region = runner.block_size * entry * item - row_stride = cache.stride(0) * item - - runtime = getattr(runner, "state_runtime", None) - spec = None if runtime is None else runtime.checkpoint_spec - page_unit_bytes = spec.page_unit_bytes if spec is not None else rows * region - if rows * region != page_unit_bytes: - raise RuntimeError( - f"a PAGE unit is {page_unit_bytes} B but this pool gives a " - f"logical block {rows} rows x {region} B = {rows * region} B; " - "the two disagree about block granularity" - ) - base = np.array( - [owner + row * row_stride for row in range(rows)], dtype=np.int64 - ) - regions = (base, np.full(rows, region, dtype=np.int64)) - self._page_unit_region_cache = (owner, regions) - return regions - - def _page_unit_bases(self, unit_ids: Sequence[Sequence[int]]) -> np.ndarray: - """Start address of every destination segment, one row per image. - - `unit_ids` is `(images, units_per_checkpoint)`. A unit's regions are - each at `base + id * stride`, so one image's worth is an outer product - and a batch's is the same product with an image axis in front. Unit - major, region minor -- the order `_checkpoint_copy_plan` built the - destination stream in. - """ - base, stride = self._page_unit_regions() - ids = np.asarray(unit_ids, dtype=np.int64) - return (base + ids[..., None] * stride).reshape(len(ids), -1) - - def _page_unit_stream_sizes(self, units: int) -> np.ndarray: - """Bytes in each destination segment of an image of `units` units.""" - return np.tile(self._page_unit_regions()[1], units) - - def page_unit_views(self, unit_ids: Sequence[int]) -> list[torch.Tensor]: - """Tensor-view counterpart of `_page_unit_regions`, for the CPU tier. - - `_page_unit_regions` gives raw addresses for the Triton descriptor; the - LMCache packer takes `list[torch.Tensor]`, so the same bytes need naming - as views. This is the tier's only new seam -- a load writes the Active - Slot directly and reuses `state_entry_views`. - - Order is the contract: unit major, row minor, the same ravel as - `_page_unit_bases`. Cross-build safety is `layout_id` in - `StateByteCodec.key`, not this order. - - `unit_ids` carries *logical* block ids while `kv_cache` is shaped in - *physical* ones (K3's ratio is 128). Flattening the two block axes and - indexing by `unit * block_size` is that conversion -- the same - arithmetic `_page_unit_bases` does in addresses, so the two cannot - drift. Range-checked against the logical count for the same reason. - - `_page_unit_regions` runs first for its side effect: the contiguity and - granularity checks live there. - """ - _base, _stride = self._page_unit_regions() - cache = self.model_runner.kv_cache - rows, entry = cache.shape[0], cache.shape[3] - block = self.model_runner.block_size - # Logical blocks the pool holds, which is what a unit id indexes. - # Range-checked against THIS count, not the physical one: the physical - # count is `block_ratio` times smaller, so checking it would pass ids - # that address past the end of the tensor. - logical_blocks = (cache.shape[1] * cache.shape[2]) // block - planes = [cache[row].view(-1, entry) for row in range(rows)] - views: list[torch.Tensor] = [] - for unit in unit_ids: - unit = int(unit) - if not 0 <= unit < logical_blocks: - raise IndexError( - f"PAGE unit {unit} outside the pool's {logical_blocks} " - "logical blocks" - ) - lo = unit * block - views.extend(plane[lo : lo + block] for plane in planes) - - # ---- cut the padding tail off the final unit ---- - # - # An image is `image_bytes`, but it occupies `ceil(image_bytes / - # page_unit_bytes)` WHOLE units, so the last one is mostly padding: at - # K3's 2,138,112 B unit and 58,079,232 B image that is 28 units = - # 59,867,136 B, of which 1,787,904 belong to no image. - # - # `_checkpoint_copy_plan` already knows this -- it hands - # `plan_segmented_copy` the unit stream *and* `spec.image_bytes`, so the - # HBM copy writes only the leading `image_bytes` and leaves the tail - # alone. Gathering whole units for the CPU tier skipped that argument - # and asked the packer for the padding too, against a MemoryObj - # `StateByteCodec.put` sizes at `entry_bytes` -- which IS `image_bytes`. - # Every store died with "MemoryObj tensor is too small for 59867136 - # bytes; got 58079232". - # - # The trim is byte-exact rather than view-aligned because the image does - # not end on one: 58,079,232 leaves 350,208 B of the last unit in use, - # which is 20.96 rows of it. That view is reinterpreted as `uint8` and - # sliced; the packer sizes segments as `numel * element_size`, so it - # costs nothing to accept. The load side needs no counterpart -- it - # scatters into `state_entry_views`, which sums to `entry_bytes`. - # - # Inline rather than a helper method: the unit tests drive this function - # with a `SimpleNamespace` stand-in for `self`, so a second attribute - # would have to be stubbed by every one of them. - runtime = getattr(self.model_runner, "state_runtime", None) - spec = None if runtime is None else runtime.checkpoint_spec - # `getattr`: a fork build carries no spec, and the unit tests hand this - # a stand-in that carries no `image_bytes`. Either way there is no image - # size to trim against, and the whole-unit stream is the right answer. - budget = int(getattr(spec, "image_bytes", 0) or 0) if spec else 0 - if not budget: - return views - out: list[torch.Tensor] = [] - for view in views: - nbytes = view.numel() * view.element_size() - if budget >= nbytes: - out.append(view) - budget -= nbytes - continue - if budget > 0: - out.append(view.reshape(-1).view(torch.uint8)[:budget]) - budget = 0 - break - if budget: - raise RuntimeError( - f"a checkpoint image is {spec.image_bytes} B but its " - f"{len(views)} unit views hold " - f"{int(spec.image_bytes) - budget} B; " - "the unit geometry and the image size disagree" - ) - return out - def build_kv_cache_tensor(self, layer_id: int, module): from atom.config import KVCacheTensor diff --git a/atom/model_ops/attentions/page_unit_geometry.py b/atom/model_ops/attentions/page_unit_geometry.py new file mode 100644 index 0000000000..1b6d16c6fb --- /dev/null +++ b/atom/model_ops/attentions/page_unit_geometry.py @@ -0,0 +1,200 @@ +# SPDX-License-Identifier: MIT +# Copyright (C) 2024-2025, Advanced Micro Devices, Inc. All rights reserved. + +"""Where a checkpoint image's bytes live in the MLA paged pool. + +This is the destination side of a K3 PAGE-copy checkpoint: given logical block +(PAGE unit) ids, it says which addresses -- and which tensor views -- of the KV +pool a checkpoint copy lands on. `_KimiMLAGDNCommon` mixes it in over +`GDNStateMixin`, whose stubs raise `NotImplementedError` for a pool it does not +own. + +Kept in its own module, free of any GPU/aiter import, for one reason: it is the +CPU tier's only seam onto PAGE units, and it is pure arithmetic over +`self.model_runner`. Living here it can be unit-tested on a runner with no GPU +(the K3 builder's own module imports aiter at load and is `importorskip`ped on +CI), so the geometry that decides where every checkpoint byte goes is actually +exercised there. +""" + +from __future__ import annotations + +from collections.abc import Sequence + +import numpy as np +import torch + + +class PageUnitGeometryMixin: + """PAGE-unit address/view arithmetic for the MLA paged pool. + + Reads only `self.model_runner` (`kv_cache`, `block_size`, `state_runtime`) + and caches the pool-derived regions on `self._page_unit_region_cache`. No + aiter, no CUDA -- the concrete K3 builder supplies the pool. + """ + + def _page_unit_regions(self) -> tuple[np.ndarray, np.ndarray]: + """Base address and per-unit stride of every region a PAGE id owns. + + The destination side of a checkpoint copy. `GDNStateMixin` knows where + a state slot's bytes are; this knows where a KV block's are, because + this class owns the MLA pool. + + `kv_cache` is `(rows, physical_blocks, physical_block_size, entry)`, + so a block owns one contiguous region per row and the rows are a fixed + stride apart. Affine in the block id, and a property of the pool rather + than of any block, so it is worked out once. + + The units are the trap. `unit_ids` carries **logical** block ids -- what + `BlockPool` hands out and what `sub_pool_specs` priced -- while the + tensor is shaped in **physical** blocks, and K3's `block_ratio` is 128. + So a region is `runner.block_size` tokens wide, not + `physical_block_size`, and the two differ by exactly that ratio. The + assertion below is what makes a mix-up a startup error rather than 127 + blocks of scrambled state: it is the one relation that cannot hold if + the granularity is wrong. + """ + runner = self.model_runner + cache = runner.kv_cache + owner = cache.data_ptr() + cached = getattr(self, "_page_unit_region_cache", None) + if cached is not None and cached[0] == owner: + return cached[1] + + if not cache.is_contiguous(): + raise RuntimeError("the MLA pool must be contiguous to be copied") + item = cache.element_size() + entry = cache.shape[3] + rows = cache.shape[0] + # One logical block's bytes inside one row. + region = runner.block_size * entry * item + row_stride = cache.stride(0) * item + + runtime = getattr(runner, "state_runtime", None) + spec = None if runtime is None else runtime.checkpoint_spec + page_unit_bytes = spec.page_unit_bytes if spec is not None else rows * region + if rows * region != page_unit_bytes: + raise RuntimeError( + f"a PAGE unit is {page_unit_bytes} B but this pool gives a " + f"logical block {rows} rows x {region} B = {rows * region} B; " + "the two disagree about block granularity" + ) + base = np.array( + [owner + row * row_stride for row in range(rows)], dtype=np.int64 + ) + regions = (base, np.full(rows, region, dtype=np.int64)) + self._page_unit_region_cache = (owner, regions) + return regions + + def _page_unit_bases(self, unit_ids: Sequence[Sequence[int]]) -> np.ndarray: + """Start address of every destination segment, one row per image. + + `unit_ids` is `(images, units_per_checkpoint)`. A unit's regions are + each at `base + id * stride`, so one image's worth is an outer product + and a batch's is the same product with an image axis in front. Unit + major, region minor -- the order `_checkpoint_copy_plan` built the + destination stream in. + """ + base, stride = self._page_unit_regions() + ids = np.asarray(unit_ids, dtype=np.int64) + return (base + ids[..., None] * stride).reshape(len(ids), -1) + + def _page_unit_stream_sizes(self, units: int) -> np.ndarray: + """Bytes in each destination segment of an image of `units` units.""" + return np.tile(self._page_unit_regions()[1], units) + + def page_unit_views(self, unit_ids: Sequence[int]) -> list[torch.Tensor]: + """Tensor-view counterpart of `_page_unit_regions`, for the CPU tier. + + `_page_unit_regions` gives raw addresses for the Triton descriptor; the + LMCache packer takes `list[torch.Tensor]`, so the same bytes need naming + as views. This is the tier's only new seam -- a load writes the Active + Slot directly and reuses `state_entry_views`. + + Order is the contract: unit major, row minor, the same ravel as + `_page_unit_bases`. Cross-build safety is `layout_id` in + `StateByteCodec.key`, not this order. + + `unit_ids` carries *logical* block ids while `kv_cache` is shaped in + *physical* ones (K3's ratio is 128). Flattening the two block axes and + indexing by `unit * block_size` is that conversion -- the same + arithmetic `_page_unit_bases` does in addresses, so the two cannot + drift. Range-checked against the logical count for the same reason. + + `_page_unit_regions` runs first for its side effect: the contiguity and + granularity checks live there. + """ + _base, _stride = self._page_unit_regions() + cache = self.model_runner.kv_cache + rows, entry = cache.shape[0], cache.shape[3] + block = self.model_runner.block_size + # Logical blocks the pool holds, which is what a unit id indexes. + # Range-checked against THIS count, not the physical one: the physical + # count is `block_ratio` times smaller, so checking it would pass ids + # that address past the end of the tensor. + logical_blocks = (cache.shape[1] * cache.shape[2]) // block + planes = [cache[row].view(-1, entry) for row in range(rows)] + views: list[torch.Tensor] = [] + for unit in unit_ids: + unit = int(unit) + if not 0 <= unit < logical_blocks: + raise IndexError( + f"PAGE unit {unit} outside the pool's {logical_blocks} " + "logical blocks" + ) + lo = unit * block + views.extend(plane[lo : lo + block] for plane in planes) + + # ---- cut the padding tail off the final unit ---- + # + # An image is `image_bytes`, but it occupies `ceil(image_bytes / + # page_unit_bytes)` WHOLE units, so the last one is mostly padding: at + # K3's 2,138,112 B unit and 58,079,232 B image that is 28 units = + # 59,867,136 B, of which 1,787,904 belong to no image. + # + # `_checkpoint_copy_plan` already knows this -- it hands + # `plan_segmented_copy` the unit stream *and* `spec.image_bytes`, so the + # HBM copy writes only the leading `image_bytes` and leaves the tail + # alone. Gathering whole units for the CPU tier skipped that argument + # and asked the packer for the padding too, against a MemoryObj + # `StateByteCodec.put` sizes at `entry_bytes` -- which IS `image_bytes`. + # Every store died with "MemoryObj tensor is too small for 59867136 + # bytes; got 58079232". + # + # The trim is byte-exact rather than view-aligned because the image does + # not end on one: 58,079,232 leaves 350,208 B of the last unit in use, + # which is 20.96 rows of it. That view is reinterpreted as `uint8` and + # sliced; the packer sizes segments as `numel * element_size`, so it + # costs nothing to accept. The load side needs no counterpart -- it + # scatters into `state_entry_views`, which sums to `entry_bytes`. + # + # Inline rather than a helper method: the unit tests drive this function + # with a `SimpleNamespace` stand-in for `self`, so a second attribute + # would have to be stubbed by every one of them. + runtime = getattr(self.model_runner, "state_runtime", None) + spec = None if runtime is None else runtime.checkpoint_spec + # `getattr`: a fork build carries no spec, and the unit tests hand this + # a stand-in that carries no `image_bytes`. Either way there is no image + # size to trim against, and the whole-unit stream is the right answer. + budget = int(getattr(spec, "image_bytes", 0) or 0) if spec else 0 + if not budget: + return views + out: list[torch.Tensor] = [] + for view in views: + nbytes = view.numel() * view.element_size() + if budget >= nbytes: + out.append(view) + budget -= nbytes + continue + if budget > 0: + out.append(view.reshape(-1).view(torch.uint8)[:budget]) + budget = 0 + break + if budget: + raise RuntimeError( + f"a checkpoint image is {spec.image_bytes} B but its " + f"{len(views)} unit views hold " + f"{int(spec.image_bytes) - budget} B; " + "the unit geometry and the image size disagree" + ) + return out diff --git a/tests/test_kda_checkpoint_slot_copy.py b/tests/test_kda_checkpoint_slot_copy.py index eb83610b9e..00ae49d7fd 100644 --- a/tests/test_kda_checkpoint_slot_copy.py +++ b/tests/test_kda_checkpoint_slot_copy.py @@ -206,116 +206,6 @@ def test_the_mixin_refuses_to_guess_where_a_page_unit_is(self): ) -class TestPageUnitAddressesAreArithmetic: - """The addresses `_page_unit_regions` computes are the ones slicing gives. - - Replacing a view per row per unit with one multiplication is only safe if - it lands on the same bytes, so this asks both and compares. The slicing - expression is written out here rather than kept in production: it is the - oracle, not a fallback. - """ - - ROWS = 3 # MLA layers - ENTRY = 4 - LOGICAL_BS = 2 # tokens per logical block - RATIO = 4 # block_ratio: physical blocks per logical block - N_LOGICAL = 5 - - def build(self, block_size=None, spec_bytes=None, image_bytes=None, dtype=None): - phys_bs = self.LOGICAL_BS // self.RATIO or 1 - dtype = dtype or torch.uint8 - cache = ( - torch.arange( - self.ROWS * self.N_LOGICAL * self.RATIO * phys_bs * self.ENTRY, - dtype=torch.int64, - ) - .to(dtype) - .reshape(self.ROWS, self.N_LOGICAL * self.RATIO, phys_bs, self.ENTRY) - ) - region = self.LOGICAL_BS * self.ENTRY * cache.element_size() - spec = SimpleNamespace( - page_unit_bytes=self.ROWS * region if spec_bytes is None else spec_bytes - ) - # Left OFF unless a test asks for it: `page_unit_views` trims to - # `image_bytes` only when the spec carries one, and the addressing tests - # above are about whole units. - if image_bytes is not None: - spec.image_bytes = image_bytes - runtime = SimpleNamespace(checkpoint_spec=spec) - runner = SimpleNamespace( - kv_cache=cache, - block_size=self.LOGICAL_BS if block_size is None else block_size, - state_runtime=runtime, - ) - stub = SimpleNamespace(model_runner=runner, _page_unit_region_cache=None) - for name in ( - "_page_unit_regions", - "_page_unit_bases", - "_page_unit_stream_sizes", - ): - method = getattr(K3._KimiMLAGDNCommon, name) - setattr(stub, name, method.__get__(stub, type(stub))) - return stub, cache - - def test_a_unit_addresses_one_region_per_row(self): - stub, cache = self.build() - base, stride = stub._page_unit_regions() - assert len(base) == len(stride) == self.ROWS - # The oracle: flatten each row and slice the logical block out of it. - region = self.LOGICAL_BS * self.ENTRY - for unit in range(self.N_LOGICAL): - addrs = stub._page_unit_bases([[unit]])[0] - for row in range(self.ROWS): - want = cache[row].reshape(-1)[unit * region :].data_ptr() - assert addrs[row] == want - - def test_the_stream_sizes_match_the_addresses_one_for_one(self): - stub, _ = self.build() - units = [0, 3, 1] - addrs = stub._page_unit_bases([units])[0] - sizes = stub._page_unit_stream_sizes(len(units)) - assert len(addrs) == len(sizes) == self.ROWS * len(units) - - def test_an_image_covers_its_units_in_order_unit_major_row_minor(self): - """`_checkpoint_copy_plan` builds the destination stream this way, so - a plan's segment index has to mean the same thing here.""" - stub, _ = self.build() - units = [4, 0, 2] # deliberately not ascending: ids are not a range - addrs = stub._page_unit_bases([units])[0] - base, stride = stub._page_unit_regions() - want = [base[r] + u * stride[r] for u in units for r in range(self.ROWS)] - assert addrs.tolist() == want - - def test_the_region_is_a_logical_block_not_a_physical_one(self): - """The `block_ratio` trap: the tensor is shaped in physical blocks, but - `unit_ids` carries logical ones, and K3's ratio is 128.""" - stub, _ = self.build() - _, stride = stub._page_unit_regions() - assert set(stride.tolist()) == {self.LOGICAL_BS * self.ENTRY} - - def test_a_granularity_mismatch_is_refused_at_startup(self): - """Wrong granularity would not raise on its own -- it would scatter a - checkpoint across the wrong blocks. The one relation that cannot hold - if it is wrong is asserted instead.""" - stub, _ = self.build(block_size=self.LOGICAL_BS * 2) - with pytest.raises(RuntimeError, match="granularity"): - stub._page_unit_regions() - - def test_a_non_contiguous_pool_is_refused_not_mis_addressed(self): - stub, _ = self.build() - stub.model_runner.kv_cache = stub.model_runner.kv_cache.transpose(0, 1) - with pytest.raises(RuntimeError, match="contiguous"): - stub._page_unit_regions() - - def test_the_regions_are_worked_out_once_but_notice_a_moved_pool(self): - stub, _ = self.build() - first = stub._page_unit_regions() - assert stub._page_unit_regions() is first - - stub.model_runner.kv_cache = torch.zeros_like(stub.model_runner.kv_cache) - assert stub._page_unit_regions() is not first - - class TestAStoreRestoreRoundTripMovesExactlyTheImage: """Store a slot into PAGE units, gather it back into a different slot. @@ -451,177 +341,3 @@ def test_a_restore_does_not_read_past_the_image(self): for row in range(self.ROWS): tail = pool[row].reshape(-1)[len(units) * self.LOGICAL_BS * self.ENTRY :] assert int(tail.sum()) == 0 - - -class TestPageUnitViewsNameTheSameBytes: - """`page_unit_views` is the CPU tier's seam onto the PAGE units. - - `_page_unit_regions` gives the Triton descriptor raw addresses; the LMCache - packer wants tensors. Two ways to name one set of bytes is exactly the kind - of pair that drifts, so every test here asks both and compares. - """ - - H = TestPageUnitAddressesAreArithmetic - - def build(self, **kw): - stub, cache = self.H.build(self.H(), **kw) - stub.page_unit_views = K3._KimiMLAGDNCommon.page_unit_views.__get__( - stub, type(stub) - ) - return stub, cache - - def test_a_view_starts_exactly_where_the_address_says(self): - stub, _ = self.build() - units = [4, 0, 2] # not ascending: ids are not a range - views = stub.page_unit_views(units) - addrs = stub._page_unit_bases([units])[0] - assert len(views) == len(addrs) == self.H.ROWS * len(units) - assert [v.data_ptr() for v in views] == addrs.tolist() - - def test_a_view_is_exactly_one_region_long(self): - stub, _ = self.build() - views = stub.page_unit_views([1]) - sizes = stub._page_unit_stream_sizes(1) - assert [v.numel() * v.element_size() for v in views] == sizes.tolist() - - def test_the_order_is_unit_major_row_minor(self): - """The packer indexes segments positionally, so this order IS the blob - layout -- and `_checkpoint_copy_plan` built the HBM stream the same - way. A build that ordered it differently must not read another's blob, - which is what `layout_id` in the key covers.""" - stub, _ = self.build() - units = [3, 1] - base, stride = stub._page_unit_regions() - want = [base[r] + u * stride[r] for u in units for r in range(self.H.ROWS)] - assert [v.data_ptr() for v in stub.page_unit_views(units)] == want - - def test_every_view_is_contiguous(self): - """The Triton packer refuses a strided segment.""" - stub, _ = self.build() - assert all(v.is_contiguous() for v in stub.page_unit_views([0, 2, 4])) - - def test_a_unit_id_past_the_logical_count_is_refused(self): - """The `block_ratio` trap in its most dangerous form. - - The bound is computed from the tensor rather than taken from the - harness's label, because that is the whole point: `unit_ids` counts - logical blocks and the tensor is shaped in physical ones, so a check - written against the wrong axis admits ids that read past the end. - """ - stub, cache = self.build() - logical = (cache.shape[1] * cache.shape[2]) // self.H.LOGICAL_BS - assert logical != cache.shape[1], "the harness must exercise a ratio != 1" - assert stub.page_unit_views([logical - 1]) # the last valid id - with pytest.raises(IndexError, match="logical blocks"): - stub.page_unit_views([logical]) - - def test_the_views_do_not_reach_into_a_neighbouring_unit(self): - """Off-by-one here scrambles someone else's checkpoint rather than - running off the end, so the bound is asserted from the bytes.""" - stub, cache = self.build() - flat = cache.reshape(-1) - seen = set() - logical = (cache.shape[1] * cache.shape[2]) // self.H.LOGICAL_BS - for unit in range(logical): - for v in stub.page_unit_views([unit]): - lo = (v.data_ptr() - flat.data_ptr()) // flat.element_size() - span = set(range(lo, lo + v.numel())) - assert not (span & seen), f"unit {unit} overlaps an earlier one" - seen |= span - - def test_the_granularity_check_runs_before_any_view_is_handed_out(self): - stub, _ = self.build( - block_size=TestPageUnitAddressesAreArithmetic.LOGICAL_BS * 2 - ) - with pytest.raises(RuntimeError, match="granularity"): - stub.page_unit_views([0]) - - -class TestPageUnitViewsStopAtTheImage: - """An image occupies WHOLE units, so the last one is mostly padding. - - `_checkpoint_copy_plan` already knows this — it hands `plan_segmented_copy` - the unit stream *and* `spec.image_bytes`, and that function walks both - streams from offset 0, so the image is by construction the LEADING - `image_bytes` of the unit stream and the tail belongs to nobody. - - `page_unit_views` is the tensor-view counterpart of the same addressing, so - it has to stop at the same place. Gathering whole units instead hands the - packer more bytes than `StateByteCodec.put` allocated, and every store dies - on "MemoryObj tensor is too small". - - None of the addressing tests above reach this: their spec carries no - `image_bytes`, so they take the untrimmed path by construction. - """ - - H = TestPageUnitAddressesAreArithmetic - - def build(self, **kw): - stub, cache = self.H.build(self.H(), **kw) - stub.page_unit_views = K3._KimiMLAGDNCommon.page_unit_views.__get__( - stub, type(stub) - ) - return stub, cache - - @staticmethod - def total(views): - return sum(v.numel() * v.element_size() for v in views) - - def test_the_gathered_stream_is_exactly_the_image(self): - """The invariant the packer depends on, and the one that was broken.""" - stub, _ = self.build(image_bytes=70) # 3 units x 24 B = 72 B available - assert self.total(stub.page_unit_views([0, 1, 2])) == 70 - - def test_a_whole_unit_past_the_image_is_dropped_entirely(self): - stub, _ = self.build(image_bytes=20) - views = stub.page_unit_views([0, 1, 2]) - assert self.total(views) == 20 - assert len(views) == 3, "two whole views plus the one that straddles" - - def test_an_image_that_lands_on_a_boundary_is_not_sliced(self): - stub, _ = self.build(image_bytes=72) - untrimmed, _ = self.build() - assert [v.shape for v in stub.page_unit_views([0, 1, 2])] == [ - v.shape for v in untrimmed.page_unit_views([0, 1, 2]) - ] - - def test_the_kept_bytes_are_the_leading_bytes_and_nothing_else(self): - """Trimming must not reorder or reslice — it truncates. The blob is - read back by scattering into the slot in the same order, so a byte that - moves here lands in the wrong layer there, silently.""" - stub, _ = self.build(image_bytes=70) - untrimmed, _ = self.build() - want = torch.cat( - [ - v.reshape(-1).view(torch.uint8) - for v in untrimmed.page_unit_views([0, 1, 2]) - ] - )[:70] - got = torch.cat( - [v.reshape(-1).view(torch.uint8) for v in stub.page_unit_views([0, 1, 2])] - ) - assert torch.equal(got, want) - - def test_a_multi_byte_dtype_is_sliced_by_bytes_not_elements(self): - """The straddling view is reinterpreted as uint8 before slicing, because - the image does not end on an element boundary in general. A slice taken - in elements would silently keep the wrong amount on any dtype wider than - a byte.""" - stub, cache = self.build(dtype=torch.bfloat16, image_bytes=70) - assert cache.element_size() == 2 - assert self.total(stub.page_unit_views([0, 1, 2])) == 70 - - def test_a_spec_with_no_image_size_keeps_whole_units(self): - """A fork build carries no spec at all, and there the whole-unit stream - is the right answer — there is nothing to trim against.""" - stub, _ = self.build() # no image_bytes on the spec - assert self.total(stub.page_unit_views([0, 1, 2])) == 3 * self.H.ROWS * ( - self.H.LOGICAL_BS * self.H.ENTRY - ) - - def test_units_that_cannot_cover_the_image_raise(self): - """A short blob read back as valid is the one outcome worth crashing - over: it resumes a request onto a truncated state with no exception.""" - stub, _ = self.build(image_bytes=80) # more than the 72 B on offer - with pytest.raises(RuntimeError, match="disagree"): - stub.page_unit_views([0, 1, 2]) diff --git a/tests/test_page_unit_geometry.py b/tests/test_page_unit_geometry.py new file mode 100644 index 0000000000..cc2e5fe758 --- /dev/null +++ b/tests/test_page_unit_geometry.py @@ -0,0 +1,311 @@ +# SPDX-License-Identifier: MIT +# Copyright (C) 2024-2025, Advanced Micro Devices, Inc. All rights reserved. + +"""Where a checkpoint image's bytes land in the MLA paged pool. + +The CPU tier's only seam onto PAGE units is `page_unit_views`, and its address +counterpart is `_page_unit_regions` / `_page_unit_bases`. Both are pure +arithmetic over the runner's KV pool, so they live in +`atom.model_ops.attentions.page_unit_geometry` -- a module with no aiter/GPU +import -- and are tested here, directly, on the non-GPU runner. + +This is deliberately split out of `test_kda_checkpoint_slot_copy.py`: that file +`importorskip`s the K3/GDN builder modules (they import aiter at load), so on a +CI runner without aiter the *whole* module is skipped, and these PAGE-unit tests +-- the ones that decide where every checkpoint byte goes -- never ran there. The +builder mixes `PageUnitGeometryMixin` in over `GDNStateMixin`, so what runs here +is the shipped arithmetic. +""" + +from __future__ import annotations + +from types import SimpleNamespace + +import pytest +import torch + +from atom.model_ops.attentions.page_unit_geometry import PageUnitGeometryMixin + + +class TestPageUnitAddressesAreArithmetic: + """The addresses `_page_unit_regions` computes are the ones slicing gives. + + Replacing a view per row per unit with one multiplication is only safe if + it lands on the same bytes, so this asks both and compares. The slicing + expression is written out here rather than kept in production: it is the + oracle, not a fallback. + """ + + ROWS = 3 # MLA layers + ENTRY = 4 + LOGICAL_BS = 2 # tokens per logical block + RATIO = 4 # block_ratio: physical blocks per logical block + N_LOGICAL = 5 + + def build(self, block_size=None, spec_bytes=None, image_bytes=None, dtype=None): + phys_bs = self.LOGICAL_BS // self.RATIO or 1 + dtype = dtype or torch.uint8 + cache = ( + torch.arange( + self.ROWS * self.N_LOGICAL * self.RATIO * phys_bs * self.ENTRY, + dtype=torch.int64, + ) + .to(dtype) + .reshape(self.ROWS, self.N_LOGICAL * self.RATIO, phys_bs, self.ENTRY) + ) + region = self.LOGICAL_BS * self.ENTRY * cache.element_size() + spec = SimpleNamespace( + page_unit_bytes=self.ROWS * region if spec_bytes is None else spec_bytes + ) + # Left OFF unless a test asks for it: `page_unit_views` trims to + # `image_bytes` only when the spec carries one, and the addressing tests + # above are about whole units. + if image_bytes is not None: + spec.image_bytes = image_bytes + runtime = SimpleNamespace(checkpoint_spec=spec) + runner = SimpleNamespace( + kv_cache=cache, + block_size=self.LOGICAL_BS if block_size is None else block_size, + state_runtime=runtime, + ) + stub = SimpleNamespace(model_runner=runner, _page_unit_region_cache=None) + for name in ( + "_page_unit_regions", + "_page_unit_bases", + "_page_unit_stream_sizes", + ): + method = getattr(PageUnitGeometryMixin, name) + setattr(stub, name, method.__get__(stub, type(stub))) + return stub, cache + + def test_a_unit_addresses_one_region_per_row(self): + stub, cache = self.build() + base, stride = stub._page_unit_regions() + assert len(base) == len(stride) == self.ROWS + # The oracle: flatten each row and slice the logical block out of it. + region = self.LOGICAL_BS * self.ENTRY + for unit in range(self.N_LOGICAL): + addrs = stub._page_unit_bases([[unit]])[0] + for row in range(self.ROWS): + want = cache[row].reshape(-1)[unit * region :].data_ptr() + assert addrs[row] == want + + def test_the_stream_sizes_match_the_addresses_one_for_one(self): + stub, _ = self.build() + units = [0, 3, 1] + addrs = stub._page_unit_bases([units])[0] + sizes = stub._page_unit_stream_sizes(len(units)) + assert len(addrs) == len(sizes) == self.ROWS * len(units) + + def test_an_image_covers_its_units_in_order_unit_major_row_minor(self): + """`_checkpoint_copy_plan` builds the destination stream this way, so + a plan's segment index has to mean the same thing here.""" + stub, _ = self.build() + units = [4, 0, 2] # deliberately not ascending: ids are not a range + addrs = stub._page_unit_bases([units])[0] + base, stride = stub._page_unit_regions() + want = [base[r] + u * stride[r] for u in units for r in range(self.ROWS)] + assert addrs.tolist() == want + + def test_the_region_is_a_logical_block_not_a_physical_one(self): + """The `block_ratio` trap: the tensor is shaped in physical blocks, but + `unit_ids` carries logical ones, and K3's ratio is 128.""" + stub, _ = self.build() + _, stride = stub._page_unit_regions() + assert set(stride.tolist()) == {self.LOGICAL_BS * self.ENTRY} + + def test_a_granularity_mismatch_is_refused_at_startup(self): + """Wrong granularity would not raise on its own -- it would scatter a + checkpoint across the wrong blocks. The one relation that cannot hold + if it is wrong is asserted instead.""" + stub, _ = self.build(block_size=self.LOGICAL_BS * 2) + with pytest.raises(RuntimeError, match="granularity"): + stub._page_unit_regions() + + def test_a_non_contiguous_pool_is_refused_not_mis_addressed(self): + stub, _ = self.build() + stub.model_runner.kv_cache = stub.model_runner.kv_cache.transpose(0, 1) + with pytest.raises(RuntimeError, match="contiguous"): + stub._page_unit_regions() + + def test_the_regions_are_worked_out_once_but_notice_a_moved_pool(self): + stub, _ = self.build() + first = stub._page_unit_regions() + assert stub._page_unit_regions() is first + + stub.model_runner.kv_cache = torch.zeros_like(stub.model_runner.kv_cache) + assert stub._page_unit_regions() is not first + + +class TestPageUnitViewsNameTheSameBytes: + """`page_unit_views` is the CPU tier's seam onto the PAGE units. + + `_page_unit_regions` gives the Triton descriptor raw addresses; the LMCache + packer wants tensors. Two ways to name one set of bytes is exactly the kind + of pair that drifts, so every test here asks both and compares. + """ + + H = TestPageUnitAddressesAreArithmetic + + def build(self, **kw): + stub, cache = self.H.build(self.H(), **kw) + stub.page_unit_views = PageUnitGeometryMixin.page_unit_views.__get__( + stub, type(stub) + ) + return stub, cache + + def test_a_view_starts_exactly_where_the_address_says(self): + stub, _ = self.build() + units = [4, 0, 2] # not ascending: ids are not a range + views = stub.page_unit_views(units) + addrs = stub._page_unit_bases([units])[0] + assert len(views) == len(addrs) == self.H.ROWS * len(units) + assert [v.data_ptr() for v in views] == addrs.tolist() + + def test_a_view_is_exactly_one_region_long(self): + stub, _ = self.build() + views = stub.page_unit_views([1]) + sizes = stub._page_unit_stream_sizes(1) + assert [v.numel() * v.element_size() for v in views] == sizes.tolist() + + def test_the_order_is_unit_major_row_minor(self): + """The packer indexes segments positionally, so this order IS the blob + layout -- and `_checkpoint_copy_plan` built the HBM stream the same + way. A build that ordered it differently must not read another's blob, + which is what `layout_id` in the key covers.""" + stub, _ = self.build() + units = [3, 1] + base, stride = stub._page_unit_regions() + want = [base[r] + u * stride[r] for u in units for r in range(self.H.ROWS)] + assert [v.data_ptr() for v in stub.page_unit_views(units)] == want + + def test_every_view_is_contiguous(self): + """The Triton packer refuses a strided segment.""" + stub, _ = self.build() + assert all(v.is_contiguous() for v in stub.page_unit_views([0, 2, 4])) + + def test_a_unit_id_past_the_logical_count_is_refused(self): + """The `block_ratio` trap in its most dangerous form. + + The bound is computed from the tensor rather than taken from the + harness's label, because that is the whole point: `unit_ids` counts + logical blocks and the tensor is shaped in physical ones, so a check + written against the wrong axis admits ids that read past the end. + """ + stub, cache = self.build() + logical = (cache.shape[1] * cache.shape[2]) // self.H.LOGICAL_BS + assert logical != cache.shape[1], "the harness must exercise a ratio != 1" + assert stub.page_unit_views([logical - 1]) # the last valid id + with pytest.raises(IndexError, match="logical blocks"): + stub.page_unit_views([logical]) + + def test_the_views_do_not_reach_into_a_neighbouring_unit(self): + """Off-by-one here scrambles someone else's checkpoint rather than + running off the end, so the bound is asserted from the bytes.""" + stub, cache = self.build() + flat = cache.reshape(-1) + seen = set() + logical = (cache.shape[1] * cache.shape[2]) // self.H.LOGICAL_BS + for unit in range(logical): + for v in stub.page_unit_views([unit]): + lo = (v.data_ptr() - flat.data_ptr()) // flat.element_size() + span = set(range(lo, lo + v.numel())) + assert not (span & seen), f"unit {unit} overlaps an earlier one" + seen |= span + + def test_the_granularity_check_runs_before_any_view_is_handed_out(self): + stub, _ = self.build( + block_size=TestPageUnitAddressesAreArithmetic.LOGICAL_BS * 2 + ) + with pytest.raises(RuntimeError, match="granularity"): + stub.page_unit_views([0]) + + +class TestPageUnitViewsStopAtTheImage: + """An image occupies WHOLE units, so the last one is mostly padding. + + `_checkpoint_copy_plan` already knows this — it hands `plan_segmented_copy` + the unit stream *and* `spec.image_bytes`, and that function walks both + streams from offset 0, so the image is by construction the LEADING + `image_bytes` of the unit stream and the tail belongs to nobody. + + `page_unit_views` is the tensor-view counterpart of the same addressing, so + it has to stop at the same place. Gathering whole units instead hands the + packer more bytes than `StateByteCodec.put` allocated, and every store dies + on "MemoryObj tensor is too small". + + None of the addressing tests above reach this: their spec carries no + `image_bytes`, so they take the untrimmed path by construction. + """ + + H = TestPageUnitAddressesAreArithmetic + + def build(self, **kw): + stub, cache = self.H.build(self.H(), **kw) + stub.page_unit_views = PageUnitGeometryMixin.page_unit_views.__get__( + stub, type(stub) + ) + return stub, cache + + @staticmethod + def total(views): + return sum(v.numel() * v.element_size() for v in views) + + def test_the_gathered_stream_is_exactly_the_image(self): + """The invariant the packer depends on, and the one that was broken.""" + stub, _ = self.build(image_bytes=70) # 3 units x 24 B = 72 B available + assert self.total(stub.page_unit_views([0, 1, 2])) == 70 + + def test_a_whole_unit_past_the_image_is_dropped_entirely(self): + stub, _ = self.build(image_bytes=20) + views = stub.page_unit_views([0, 1, 2]) + assert self.total(views) == 20 + assert len(views) == 3, "two whole views plus the one that straddles" + + def test_an_image_that_lands_on_a_boundary_is_not_sliced(self): + stub, _ = self.build(image_bytes=72) + untrimmed, _ = self.build() + assert [v.shape for v in stub.page_unit_views([0, 1, 2])] == [ + v.shape for v in untrimmed.page_unit_views([0, 1, 2]) + ] + + def test_the_kept_bytes_are_the_leading_bytes_and_nothing_else(self): + """Trimming must not reorder or reslice — it truncates. The blob is + read back by scattering into the slot in the same order, so a byte that + moves here lands in the wrong layer there, silently.""" + stub, _ = self.build(image_bytes=70) + untrimmed, _ = self.build() + want = torch.cat( + [ + v.reshape(-1).view(torch.uint8) + for v in untrimmed.page_unit_views([0, 1, 2]) + ] + )[:70] + got = torch.cat( + [v.reshape(-1).view(torch.uint8) for v in stub.page_unit_views([0, 1, 2])] + ) + assert torch.equal(got, want) + + def test_a_multi_byte_dtype_is_sliced_by_bytes_not_elements(self): + """The straddling view is reinterpreted as uint8 before slicing, because + the image does not end on an element boundary in general. A slice taken + in elements would silently keep the wrong amount on any dtype wider than + a byte.""" + stub, cache = self.build(dtype=torch.bfloat16, image_bytes=70) + assert cache.element_size() == 2 + assert self.total(stub.page_unit_views([0, 1, 2])) == 70 + + def test_a_spec_with_no_image_size_keeps_whole_units(self): + """A fork build carries no spec at all, and there the whole-unit stream + is the right answer — there is nothing to trim against.""" + stub, _ = self.build() # no image_bytes on the spec + assert self.total(stub.page_unit_views([0, 1, 2])) == 3 * self.H.ROWS * ( + self.H.LOGICAL_BS * self.H.ENTRY + ) + + def test_units_that_cannot_cover_the_image_raise(self): + """A short blob read back as valid is the one outcome worth crashing + over: it resumes a request onto a truncated state with no exception.""" + stub, _ = self.build(image_bytes=80) # more than the 72 B on offer + with pytest.raises(RuntimeError, match="disagree"): + stub.page_unit_views([0, 1, 2]) From 5416b3ca9de573341802e02a89966ba0ff24fdab Mon Sep 17 00:00:00 2001 From: zejchen Date: Tue, 1 Sep 2026 08:38:08 +0000 Subject: [PATCH 085/151] [Kimi-K3][LMCache] Refuse GDN/linear models at offload config resolution MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit valarLip (round-3 review, §7·4): a GDN/linear-attention model (Qwen3-Next, Qwen3.5) that enables `lmcache_offload` falls through `select_offload_layout` to `dense`, because it has neither `model_type == "kimi_linear"` nor `compress_ratios`. Registration then succeeds and the dense path restores a KV prefix while the model's per-request recurrent (mamba/KDA) state stays stale -- silent wrong output. The dense codec does fail closed on the per-request-state tensor, but that ValueError fires deep in `register_kv_caches` and reads as a byte-layout mismatch, not "this model is unsupported". Refuse at config resolution instead, where the message can name the cause and the models. `_GDN_LINEAR_MODEL_TYPES` mirrors `ModelRunner.is_qwen_next()` -- the exact set that turns on `use_gdn`. MiniMax-M2/M3 are deliberately excluded: M3 is lightning-indexer sparse attention, M2 is standard attention; neither registers recurrent state, so both correctly route to dense/hybrid. Tests: parametrized refusal over the four GDN model_types; a companion test pins that MiniMax/llama/None still resolve to dense so the refusal stays narrow. --- atom/kv_transfer/offload/config.py | 21 +++++++++++++++++++++ tests/test_lmcache_offload_config.py | 26 ++++++++++++++++++++++++++ 2 files changed, 47 insertions(+) diff --git a/atom/kv_transfer/offload/config.py b/atom/kv_transfer/offload/config.py index 17a68e1e6d..8f8ece1319 100644 --- a/atom/kv_transfer/offload/config.py +++ b/atom/kv_transfer/offload/config.py @@ -55,6 +55,12 @@ "compress_ratios", "indexer_dtype", } +# GDN/linear model types carrying a per-request recurrent state but NOT in the +# `kimi_linear` family `kimi_k3` owns. Mirrors `ModelRunner.is_qwen_next()`. +# MiniMax-M2/M3 are absent by design (sparse/standard attention, no state). +_GDN_LINEAR_MODEL_TYPES = frozenset( + {"qwen3_next", "qwen3_next_mtp", "qwen3_5_text", "qwen3_5_moe_text"} +) logger = logging.getLogger("atom") @@ -101,6 +107,21 @@ def select_offload_layout(config) -> str: return "kimi_k3" if getattr(hf_config, "compress_ratios", None): return "hybrid" + # GDN/linear models carry a per-request recurrent state no layout owns a + # tier for; `dense` would restore a KV prefix over stale state (silent wrong + # output). The dense codec fails closed on the state tensor, but that fires + # deep in `register_kv_caches` as a byte-layout mismatch -- refuse here where + # the message can name the cause. + if getattr(hf_config, "model_type", None) in _GDN_LINEAR_MODEL_TYPES: + raise ValueError( + "lmcache_offload does not support GDN/linear-attention models " + f"(model_type={getattr(hf_config, 'model_type', None)!r}; e.g. " + "Qwen3-Next, Qwen3.5). These carry a per-request recurrent state " + "that no offload layout owns a tier for -- restoring their KV " + "prefix while that state is stale is silent wrong output. Disable " + "offload for this model, or use a state-owning layout once one " + "exists for it (kimi_k3 owns only the kimi_linear family)." + ) return "dense" diff --git a/tests/test_lmcache_offload_config.py b/tests/test_lmcache_offload_config.py index 6cd92f0db8..4f41739683 100644 --- a/tests/test_lmcache_offload_config.py +++ b/tests/test_lmcache_offload_config.py @@ -150,6 +150,32 @@ def test_unknown_explicit_offload_layout_is_rejected(override): offcfg.select_offload_layout(config) +@pytest.mark.parametrize( + "model_type", + ["qwen3_next", "qwen3_next_mtp", "qwen3_5_text", "qwen3_5_moe_text"], +) +def test_gdn_linear_model_offload_is_refused(model_type): + # A GDN model must be turned away at config resolution, not fall through to + # `dense` and restore a KV prefix over its stale recurrent state. Clear + # `compress_ratios` so the hybrid branch does not claim it first. + config = _config() + config.hf_config.compress_ratios = None + config.hf_config.model_type = model_type + + with pytest.raises(ValueError, match="does not support GDN"): + offcfg.select_offload_layout(config) + + +def test_minimax_and_dense_model_types_still_route_to_dense(): + # The refusal must be narrow: a non-GDN model with no compress_ratios is + # ordinary dense, including MiniMax (sparse/standard attention, no state). + for model_type in ("minimax_m2", "minimax_m3", "llama", None): + config = _config() + config.hf_config.compress_ratios = None + config.hf_config.model_type = model_type + assert offcfg.select_offload_layout(config) == "dense" + + @pytest.mark.parametrize("invalid", [True, 256.0, "256"]) @pytest.mark.parametrize("field", ["block", "chunk", "world", "hf"]) def test_page_namespace_rejects_coerced_integer_geometry(field, invalid): From e7e3781651e343e06bb009c1c912b48d8fc87059 Mon Sep 17 00:00:00 2001 From: zejchen Date: Tue, 1 Sep 2026 08:39:43 +0000 Subject: [PATCH 086/151] [Kimi-K3][LMCache] Dedup the MemoryObj->uint8 validation into one helper MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit valarLip (round-3 review, §3): `StagedTransfer.memory_tensor` (K3 state staging) and `BlockGPUConnector._memory_tensor` (dense block path) were byte-identical -- same tensor extraction, same uint8/contiguous/size checks, same three error messages. Two copies of a validation drift apart the moment one is touched. Extract `memory_object_as_uint8(memory_obj, nbytes)` as a free function in `atom_lmcache_staging.py` -- the connectors' common aiter-free dependency, which both already import for `_StagingBuffer`/`_ThreadTransferState`, so no new import edge or cycle. Delete both methods; the three call sites (staging pack/ unpack, block chunk build) call the shared function. dsv4/codec.py's `_memory_tensor` is intentionally left alone: it is a different helper (raw tensor extraction, no nbytes/uint8/size validation) feeding dsv4's own `_validated_checkpoint_tensor`, not a copy of this one. Tests: existing offload suites (273) cover both call paths and stay green. --- .../offload/_block_gpu_connector.py | 26 ++--------------- .../offload/atom_lmcache_staging.py | 29 +++++++++++++++++++ .../offload/hybrid/kimi_k3/staging.py | 28 ++---------------- 3 files changed, 34 insertions(+), 49 deletions(-) diff --git a/atom/kv_transfer/offload/_block_gpu_connector.py b/atom/kv_transfer/offload/_block_gpu_connector.py index 72586b72f1..a01ba25fc2 100644 --- a/atom/kv_transfer/offload/_block_gpu_connector.py +++ b/atom/kv_transfer/offload/_block_gpu_connector.py @@ -26,6 +26,7 @@ _PipelineStage, _StagingBuffer, _ThreadTransferState, + memory_object_as_uint8, run_staged_pipeline, ) @@ -270,29 +271,6 @@ def _assert_fused_chunk_major_available(self) -> None: "loads successfully" ) - def _memory_tensor(self, memory_obj: Any, nbytes: int) -> torch.Tensor: - tensor = getattr(memory_obj, "tensor", None) - if tensor is None and hasattr(memory_obj, "get_tensor"): - tensor = memory_obj.get_tensor(0) - if tensor is None: - raise RuntimeError("ATOM LMCache connector: invalid MemoryObj tensor") - if tensor.dtype != torch.uint8: - raise TypeError( - "ATOM LMCache connector: MemoryObj tensor must be uint8, " - f"got {tensor.dtype}" - ) - if not tensor.is_contiguous(): - raise RuntimeError( - "ATOM LMCache connector: MemoryObj tensor not contiguous" - ) - flat = tensor.reshape(-1) - if int(flat.numel()) < int(nbytes): - raise ValueError( - "ATOM LMCache connector: MemoryObj tensor is too small " - f"for {nbytes} bytes; got {int(flat.numel())}" - ) - return flat[: int(nbytes)] - def _range_block_ids( self, all_block_ids: list[int], @@ -360,7 +338,7 @@ def _iter_transfer_chunks( _TransferChunk( memory_obj=memory_obj, block_ids=block_ids, - tensor=self._memory_tensor(memory_obj, nbytes), + tensor=memory_object_as_uint8(memory_obj, nbytes), nbytes=nbytes, ) ) diff --git a/atom/kv_transfer/offload/atom_lmcache_staging.py b/atom/kv_transfer/offload/atom_lmcache_staging.py index 685651a8ad..0f1cc8ba6c 100644 --- a/atom/kv_transfer/offload/atom_lmcache_staging.py +++ b/atom/kv_transfer/offload/atom_lmcache_staging.py @@ -13,6 +13,35 @@ import torch +def memory_object_as_uint8(memory_obj: Any, nbytes: int) -> torch.Tensor: + """The leading `nbytes` of a LMCache MemoryObj as a flat uint8 view. + + Shared by every ATOM offload connector (dense block, K3 staging, state + tier): each transfers raw bytes, so all validate the MemoryObj the same + way. Kept here -- the connectors' common aiter-free dependency -- so the + check lives once instead of byte-identically in each. + """ + tensor = getattr(memory_obj, "tensor", None) + if tensor is None and hasattr(memory_obj, "get_tensor"): + tensor = memory_obj.get_tensor(0) + if tensor is None: + raise RuntimeError("ATOM LMCache connector: invalid MemoryObj tensor") + if tensor.dtype != torch.uint8: + raise TypeError( + "ATOM LMCache connector: MemoryObj tensor must be uint8, " + f"got {tensor.dtype}" + ) + if not tensor.is_contiguous(): + raise RuntimeError("ATOM LMCache connector: MemoryObj tensor not contiguous") + flat = tensor.reshape(-1) + if int(flat.numel()) < int(nbytes): + raise ValueError( + "ATOM LMCache connector: MemoryObj tensor is too small " + f"for {nbytes} bytes; got {int(flat.numel())}" + ) + return flat[: int(nbytes)] + + class _NullCtx: def __enter__(self): return None diff --git a/atom/kv_transfer/offload/hybrid/kimi_k3/staging.py b/atom/kv_transfer/offload/hybrid/kimi_k3/staging.py index dcb2fceda9..a636efd58a 100644 --- a/atom/kv_transfer/offload/hybrid/kimi_k3/staging.py +++ b/atom/kv_transfer/offload/hybrid/kimi_k3/staging.py @@ -20,6 +20,7 @@ _NullCtx, _StagingBuffer, _ThreadTransferState, + memory_object_as_uint8, ) # --------------------------------------------------------------------------- @@ -123,29 +124,6 @@ def release_buffer_if_requested( return staging_buffer.tensor = None - def memory_tensor(self, memory_obj: Any, nbytes: int) -> torch.Tensor: - tensor = getattr(memory_obj, "tensor", None) - if tensor is None and hasattr(memory_obj, "get_tensor"): - tensor = memory_obj.get_tensor(0) - if tensor is None: - raise RuntimeError("ATOM LMCache connector: invalid MemoryObj tensor") - if tensor.dtype != torch.uint8: - raise TypeError( - "ATOM LMCache connector: MemoryObj tensor must be uint8, " - f"got {tensor.dtype}" - ) - if not tensor.is_contiguous(): - raise RuntimeError( - "ATOM LMCache connector: MemoryObj tensor not contiguous" - ) - flat = tensor.reshape(-1) - if int(flat.numel()) < int(nbytes): - raise ValueError( - "ATOM LMCache connector: MemoryObj tensor is too small " - f"for {nbytes} bytes; got {int(flat.numel())}" - ) - return flat[: int(nbytes)] - # -- whole-entry transfer (state tier) -------------------------------- @staticmethod @@ -186,7 +164,7 @@ def pack(self, segments: list[torch.Tensor], dst: Any) -> None: segments = list(segments) seg_bytes = self._segment_block_bytes(segments) nbytes = sum(seg_bytes) - dst_tensor = self.memory_tensor(dst, nbytes) + dst_tensor = memory_object_as_uint8(dst, nbytes) with self._device_ctx(): state = self.thread_state() staging_buffer = state.staging_buffer @@ -250,7 +228,7 @@ def unpack(self, src: Any, segments: list[torch.Tensor]) -> None: segments = list(segments) seg_bytes = self._segment_block_bytes(segments) nbytes = sum(seg_bytes) - src_tensor = self.memory_tensor(src, nbytes) + src_tensor = memory_object_as_uint8(src, nbytes) with self._device_ctx(): state = self.thread_state() staging_buffer = state.staging_buffer From 16f991fc20fdc7a799b7bafb9f9e3368eb95ddad Mon Sep 17 00:00:00 2001 From: zejchen Date: Tue, 1 Sep 2026 08:40:03 +0000 Subject: [PATCH 087/151] [Kimi-K3][LMCache] Call _page_unit_regions bare in page_unit_views MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit valarLip (round-3 review, §3): `page_unit_views` opened with `_base, _stride = self._page_unit_regions()` and then used neither name. The call is there only for its side effect -- the contiguity and granularity checks that live in `_page_unit_regions` -- but unpacking a tuple into two discarded locals reads as a leftover, and a linter flags the unused bindings. Call it bare and say in the docstring why: the returned base/stride addresses feed the Triton descriptor path, not this view path, which rebuilds the same geometry as tensor views a few lines down. --- atom/model_ops/attentions/page_unit_geometry.py | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/atom/model_ops/attentions/page_unit_geometry.py b/atom/model_ops/attentions/page_unit_geometry.py index 1b6d16c6fb..f862582384 100644 --- a/atom/model_ops/attentions/page_unit_geometry.py +++ b/atom/model_ops/attentions/page_unit_geometry.py @@ -122,9 +122,10 @@ def page_unit_views(self, unit_ids: Sequence[int]) -> list[torch.Tensor]: drift. Range-checked against the logical count for the same reason. `_page_unit_regions` runs first for its side effect: the contiguity and - granularity checks live there. + granularity checks live there. Its return (base/stride addresses) is for + the Triton descriptor, not this view path -- called bare, not unpacked. """ - _base, _stride = self._page_unit_regions() + self._page_unit_regions() cache = self.model_runner.kv_cache rows, entry = cache.shape[0], cache.shape[3] block = self.model_runner.block_size From f42500a3b38558c5e563dc1e4b9f67a84c4ea13d Mon Sep 17 00:00:00 2001 From: zejchen Date: Tue, 1 Sep 2026 08:45:00 +0000 Subject: [PATCH 088/151] [Kimi-K3][LMCache] Split should_defer_free into a pure query + a free-time mutator MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit valarLip (round-3 review, §5·2): K3's `should_defer_free` mutated inside a predicate -- on the stall escape it popped `_save_tracker` and returned False. `_is_preemptable` and `_maybe_release_deferred` both probe `should_defer_free`, so a query that mutates is a hazard: a probe that never leads to a free still drops the tracker. The pop is genuinely load-bearing, but only on the preempt free: `preempt` runs `block_manager.deallocate` with no `request_finished`, so nothing else removes the sid, and the base save loop could later (once the stall clears) emit a save reading those now-freed, possibly-reused blocks -- silent cross-prefix corruption. The finished path is already covered: it frees via `request_finished`, which pops the tracker under its own `not should_defer_free` guard. Split the two halves: - `should_defer_free` is now pure -- `_save_is_stall_escaped(seq)` factors out the escape predicate; the method only reads it. Safe to probe repeatedly. - `release_stalled_save(seq)` is the mutator: pops the tracker, guarded by the same escape predicate so a non-stalled save is never dropped. Idempotent. - `DenseOffloadScheduler` gets a no-op `release_stalled_save` (dense has no escape, so a pending-save request always defers and is never preemptable); the shell forwards it getattr-guarded, like `abandon_save`. - `Scheduler.preempt` calls `_connector_release_stalled_save(seq)` right before `block_manager.deallocate`, so the tracker drops exactly when the blocks are surrendered, and before the save loop could read them. Tests: the stall-escape connector test now asserts the query is pure (tracker survives a double probe) and the mutator drops it; a new scheduler test proves `preempt` calls `release_stalled_save` BEFORE `deallocate`, with the block table still populated at release time. --- atom/kv_transfer/offload/connector.py | 5 ++ atom/kv_transfer/offload/dense/connector.py | 10 +++ .../offload/hybrid/kimi_k3/connector.py | 90 ++++++++++--------- atom/model_engine/scheduler.py | 19 ++++ tests/test_lmcache_offload_connector.py | 17 +++- tests/test_scheduler.py | 37 ++++++++ 6 files changed, 135 insertions(+), 43 deletions(-) diff --git a/atom/kv_transfer/offload/connector.py b/atom/kv_transfer/offload/connector.py index 10a47cee9d..ac8077d955 100644 --- a/atom/kv_transfer/offload/connector.py +++ b/atom/kv_transfer/offload/connector.py @@ -182,6 +182,11 @@ def should_park_for_load_after_alloc(self, seq) -> bool: def should_defer_free(self, seq) -> bool: return self._impl.should_defer_free(seq) + def release_stalled_save(self, seq) -> None: + callback = getattr(self._impl, "release_stalled_save", None) + if callback is not None: + callback(seq) + def has_pending_work(self) -> bool: return self._impl.has_pending_work() diff --git a/atom/kv_transfer/offload/dense/connector.py b/atom/kv_transfer/offload/dense/connector.py index c35b1ab542..c5fdcd9220 100644 --- a/atom/kv_transfer/offload/dense/connector.py +++ b/atom/kv_transfer/offload/dense/connector.py @@ -706,6 +706,16 @@ def should_defer_free(self, seq) -> bool: sid = str(seq.id) return sid in self._save_inflight or self._has_pending_save(seq) + def release_stalled_save(self, seq) -> None: + """Drop bookkeeping for a stall-escaped save the scheduler is freeing. + + No-op on dense: its `should_defer_free` has no stall escape, so a request + with a pending save always defers and is never preemptable. K3 overrides + this to pop its `_save_tracker`. Defined here so every offload impl + answers the scheduler's `release_stalled_save` forward uniformly. + """ + return None + def has_pending_work(self) -> bool: """True while a load still needs dispatch or a save is unreported. diff --git a/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py b/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py index a6dde6ab7a..3c67150b92 100644 --- a/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py +++ b/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py @@ -560,54 +560,64 @@ def abandon_save(self, req_id) -> None: super().abandon_save(req_id) self._refresh_save_stall() - def should_defer_free(self, seq) -> bool: - """Base behaviour, plus an escape when the backend has stopped. - - A save already handed out is reading these blocks, so freeing them - would let the next request write into them mid-transfer and index the - result under this prefix's hash. One never handed out has no reader, - and holding it is what turns a stopped backend into a stopped engine. - - That leaves the handed-out save with no way out, which is what - `Scheduler._reconcile_stalled_deferred_saves` covers, on a longer clock - tied to LMCache's own force-unpin window. Neither mechanism subsumes the - other: this one asks whether the backend ever took the save, that one - whether it ever answered. - - An active load is checked *first*, before the stall escape: the base's - `should_defer_free` holds blocks while a load is still reading/writing - them (`_has_active_load`), and the escape must not override that. A - request whose save has stalled can still have a live load into the same - block table; releasing those blocks mid-load is the free-while-writing - corruption the escape is not meant to cause. No-load requests are - unaffected -- the base checks the same predicate first anyway. + def _save_is_stall_escaped(self, seq) -> bool: + """Whether a stalled, never-dispatched save lets these blocks go free. + + A save already handed out is reading these blocks, so freeing them would + let the next request write into them mid-transfer and index the result + under this prefix's hash. One never handed out has no reader, and holding + it is what turns a stopped backend into a stopped engine. + + The handed-out save is left to `Scheduler._reconcile_stalled_deferred_saves`, + on a longer clock tied to LMCache's force-unpin window. Neither subsumes + the other: this asks whether the backend ever took the save, that whether + it ever answered. """ - if self._has_active_load(seq): - return True sid = str(seq.id) - if ( + return ( self._save_stalled and sid not in self._save_inflight and self._has_pending_save(seq) - ): - # Dropping the tracker entry here is load-bearing, not a tidy-up. - # Returning False lets the caller free the blocks, and on the - # preemption path (`Scheduler.preempt`) that free is a - # `block_manager.deallocate` with no `request_finished` call -- so - # nothing else removes this sid from `_save_tracker`. Left in, the - # base save loop would later (once the stall clears and - # `_may_emit_save` re-opens) emit a save reading `seq.block_table`, - # whose blocks are now freed and possibly reused: silent cross-prefix - # corruption. The loop does not re-check liveness, so the entry has - # to go the moment its blocks are surrendered. Dropping a stalled - # save is the intended outcome (this request's KV goes un-offloaded - # rather than wedging the engine), and `abandon_save` drops the - # matching `_save_inflight` entry on the handed-out twin for the same - # reason. - self._save_tracker.pop(sid, None) + ) + + def should_defer_free(self, seq) -> bool: + """Pure query: base behaviour, plus the stall escape. + + An active load is checked *first*: the base holds blocks while a load is + still reading/writing them, and the escape must not override that -- a + stalled-save request can still have a live load into the same table, and + releasing mid-load is free-while-writing corruption. + + This is a predicate only. `_is_preemptable`/`_maybe_release_deferred` + probe it, so it must not mutate. The `_save_tracker` cleanup the escape + needs on the preempt free (`block_manager.deallocate`, no + `request_finished`) is done by `release_stalled_save`, which the + scheduler calls at that free; the finished path pops the tracker in + `request_finished` (its `not should_defer_free` guard reads False here). + """ + if self._has_active_load(seq): + return True + if self._save_is_stall_escaped(seq): return False return super().should_defer_free(seq) + def release_stalled_save(self, seq) -> None: + """Drop the tracker for a stall-escaped save whose blocks are being freed. + + The mutator half of the escape in `should_defer_free`. The scheduler + calls this at `preempt`'s `block_manager.deallocate`, which runs no + `request_finished`, so nothing else removes this sid from + `_save_tracker`. Left in, the base save loop would later (once the stall + clears and `_may_emit_save` re-opens) emit a save reading a now-freed, + possibly-reused `seq.block_table`: silent cross-prefix corruption. The + loop does not re-check liveness, so the entry has to go the moment its + blocks are surrendered. Dropping the save is the intended outcome (this + request's KV goes un-offloaded rather than wedging the engine); guarded + by the same escape predicate so a non-stalled save is never dropped. + """ + if self._save_is_stall_escaped(seq): + self._save_tracker.pop(str(seq.id), None) + # -- joint boundary ---------------------------------------------------- def _decide_load_after_alloc(self, seq, ls): """Clamp a hybrid's KV leg to the boundary the state leg is aimed at. diff --git a/atom/model_engine/scheduler.py b/atom/model_engine/scheduler.py index 98969acb87..272f032a8c 100644 --- a/atom/model_engine/scheduler.py +++ b/atom/model_engine/scheduler.py @@ -907,6 +907,21 @@ def _connector_should_defer_free(self, seq: Sequence) -> bool: callback = getattr(self.kv_connector, "should_defer_free", None) return bool(callable(callback) and callback(seq)) + def _connector_release_stalled_save(self, seq: Sequence) -> None: + """Let the connector drop a stall-escaped save this free surrenders. + + `preempt` frees blocks with `block_manager.deallocate` and no + `request_finished`, so a K3 request whose `should_defer_free` returned + False via its stall escape would keep its `_save_tracker` entry -- the + base save loop could later emit a save reading those now-freed blocks. + The connector's `should_defer_free` is a pure predicate, so this mutator + does the cleanup exactly at the free. Guarded because not every connector + offloads saves. + """ + callback = getattr(self.kv_connector, "release_stalled_save", None) + if callable(callback): + callback(seq) + def _connector_abandon_save(self, seq: Sequence) -> None: """Tell the connector to drop a save this reclaim just abandoned. @@ -2333,6 +2348,10 @@ def preempt(self, seq: Sequence) -> bool: if seq.is_partial_prefill: seq.is_partial_prefill = False self._partial_prefill_count -= 1 + # This free runs no `request_finished`; drop any stall-escaped save + # tracker now so the connector's save loop cannot later read these + # surrendered blocks (the mutator half of `should_defer_free`'s escape). + self._connector_release_stalled_save(seq) self.block_manager.deallocate(seq) self.waiting.appendleft(seq) return True diff --git a/tests/test_lmcache_offload_connector.py b/tests/test_lmcache_offload_connector.py index a317cd5cb0..553584942f 100644 --- a/tests/test_lmcache_offload_connector.py +++ b/tests/test_lmcache_offload_connector.py @@ -4887,11 +4887,22 @@ def test_a_stalled_save_releases_blocks_it_never_handed_out(monkeypatch): s._save_inflight_since["9"] -= 2 * SAVE_STALL_SECONDS s._refresh_save_stall() assert s._save_stalled is True - # Never handed out -> dropped, blocks released. + # Never handed out -> the blocks may go free. `should_defer_free` is a pure + # query now: it answers False but does NOT drop the tracker (probed by + # `_is_preemptable`, so it must not mutate). It can be asked twice, safely. assert s.should_defer_free(seq) is False + assert s.should_defer_free(seq) is False + assert "7" in s._save_tracker + # The scheduler drops the tracker at the actual free, via the mutator half. + s.release_stalled_save(seq) assert "7" not in s._save_tracker - # Handed out -> still held, whatever the stall says. - assert s.should_defer_free(SimpleNamespace(id=9)) is True + # Idempotent: dropping an already-dropped save is a no-op. + s.release_stalled_save(seq) + # Handed out -> still held, whatever the stall says, and never dropped. + handed_out = SimpleNamespace(id=9) + assert s.should_defer_free(handed_out) is True + s.release_stalled_save(handed_out) + assert "9" in s._save_inflight def test_state_loads_are_drained_into_the_metadata_exactly_once(monkeypatch): diff --git a/tests/test_scheduler.py b/tests/test_scheduler.py index f4fb6157a2..f504c45eb0 100644 --- a/tests/test_scheduler.py +++ b/tests/test_scheduler.py @@ -1503,6 +1503,43 @@ def test_preempt(self, scheduler, seq_factory): assert seq.status == SequenceStatus.WAITING assert len(seq.block_table) == 0 + def test_preempt_releases_stalled_save_before_freeing_blocks( + self, scheduler, seq_factory + ): + """A stall-escaped save is preemptable, and the free runs no + `request_finished`; the connector must be told to drop its save tracker + at the free, and told BEFORE the blocks are deallocated so its save loop + can never race in and read them.""" + seq = seq_factory([1, 2, 3, 4]) + scheduler.add(seq) + scheduler.schedule() + events: list[str] = [] + block_table_at_release: list = [] + + class _Connector: + def should_defer_free(self, s): + return False # stall-escaped -> preemptable + + def release_stalled_save(self, s): + events.append("release") + block_table_at_release[:] = list(s.block_table) + + scheduler.kv_connector = _Connector() + original_deallocate = scheduler.block_manager.deallocate + + def _recording_deallocate(s): + events.append("deallocate") + return original_deallocate(s) + + scheduler.block_manager.deallocate = _recording_deallocate + + assert scheduler.preempt(seq) is True + # Released, and released first -- the blocks were still held then. + assert events == ["release", "deallocate"] + assert block_table_at_release # non-empty at the moment of release + assert seq.status == SequenceStatus.WAITING + assert len(seq.block_table) == 0 + # ── postprocess ──────────────────────────────────────────────────────────── From a8dfaffe98d71d476e46f007e8042058f6e5177f Mon Sep 17 00:00:00 2001 From: zejchen Date: Tue, 1 Sep 2026 08:47:02 +0000 Subject: [PATCH 089/151] [Kimi-K3][LMCache] Join offload copy threads at worker teardown MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit valarLip (round-3 review, §7·3): `StateOffloadTier.drain()`/`shutdown()` and the worker executors had no teardown caller. The save/load `ThreadPoolExecutor`s and the tier's own store/load pools are non-daemon, so a worker process that exits without joining them either hangs on interpreter shutdown or logs `threads can no longer be started` as atexit tears a pool down under an in-flight copy. Wire a teardown path: - `OffloadWorkerMixin.close()` joins `_save_executor`/`_load_executor` (idempotent, getattr-guarded so a partially-built worker is safe). - `KimiK3OffloadConnector.close()` overrides: it `drain()`s the state tier first (an in-flight PAGE-unit copy finishes against a still-mapped KV pool), then `shutdown()`s the tier's executors, then `super().close()` joins the base pools -- ordering that matters because the tier copies out of the pool the base is about to let go. - `ModelRunner.exit()` calls `get_kvconnector().close()` (getattr-guarded -- moriio and friends define no `close`) as step 0, before `destroy_dist_env()` and before the KV tensors are dropped, so the join happens while everything the copy threads touch is still alive. Tests: close() drains-then-shuts-tier-then-joins-base in that exact order; the no-tier path (PP / non-owning layout) still joins the base pools and is idempotent. --- atom/kv_transfer/offload/_offload_common.py | 16 +++++ .../offload/hybrid/kimi_k3/connector.py | 16 +++++ atom/model_engine/model_runner.py | 9 +++ tests/test_lmcache_offload_connector.py | 62 +++++++++++++++++++ 4 files changed, 103 insertions(+) diff --git a/atom/kv_transfer/offload/_offload_common.py b/atom/kv_transfer/offload/_offload_common.py index f9a04c3084..3a74995a8a 100644 --- a/atom/kv_transfer/offload/_offload_common.py +++ b/atom/kv_transfer/offload/_offload_common.py @@ -136,6 +136,22 @@ def _init_worker_common( self._done_load: set[LoadCompletionId] = set() self._failed_load: set[LoadCompletionId] = set() + def close(self) -> None: + """Join the save/load executors at worker teardown. + + `ThreadPoolExecutor` threads are non-daemon, so a process that exits + without joining them either hangs on interpreter shutdown or logs a + `threads can no longer be started` error as atexit tears the pools down + out from under an in-flight copy. `ModelRunner.exit()` calls this before + it destroys the distributed env. Idempotent -- a second call finds the + executors already shut down. Subclasses that own further resources + (K3's state tier) override and call `super().close()` last. + """ + for name in ("_save_executor", "_load_executor"): + executor = getattr(self, name, None) + if executor is not None: + executor.shutdown(wait=True) + @staticmethod def _load_completion_id(req) -> LoadCompletionId: return getattr(req, "load_operation", None) or req.req_id diff --git a/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py b/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py index 3c67150b92..52ebc22e0c 100644 --- a/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py +++ b/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py @@ -78,6 +78,22 @@ def __init__(self, config) -> None: # with it (super().start_load_kv never ran). self._store_failed_no_tier: set[StateStoreOperationId] = set() + def close(self) -> None: + """Drain then join the state tier before the base executors. + + The tier's store/load threads copy PAGE units out of the KV pool. Draining + first lets an in-flight transfer finish against a pool that is still + mapped; `shutdown` then joins the tier's own executors. Only after that + does the base close its save/load pools. Guarded because the tier is + `None` until `register_kv_caches` builds it (and stays `None` under PP or + on a non-owning layout). + """ + tier = getattr(self, "_state_tier", None) + if tier is not None: + tier.drain() + tier.shutdown() + super().close() + def register_kv_caches( self, kv_caches: dict, transfer_tensors=None, num_blocks: int | None = None ) -> None: diff --git a/atom/model_engine/model_runner.py b/atom/model_engine/model_runner.py index 363cc6ead9..9da2655648 100644 --- a/atom/model_engine/model_runner.py +++ b/atom/model_engine/model_runner.py @@ -1039,6 +1039,15 @@ def exit(self): if not self.still_running: return self.still_running = False + # 0. Join any offload connector's copy threads. Its ThreadPoolExecutors + # are non-daemon, so leaving them running wedges interpreter shutdown + # or races an in-flight copy against atexit. Must run BEFORE the KV + # pool it copies out of is dropped and before the dist env goes away. + # Guarded: only offload workers define close() (moriio etc. do not). + connector = get_kvconnector() + close = getattr(connector, "close", None) if connector is not None else None + if callable(close): + close() # 1. Destroy distributed env (NCCL + CustomAllreduce + process groups) # Must happen while ops module is still alive for CustomAllreduce cleanup. destroy_dist_env() diff --git a/tests/test_lmcache_offload_connector.py b/tests/test_lmcache_offload_connector.py index 553584942f..754afe8bfc 100644 --- a/tests/test_lmcache_offload_connector.py +++ b/tests/test_lmcache_offload_connector.py @@ -5384,6 +5384,68 @@ def test_either_leg_failing_fails_the_pair(self): assert done == set() assert failed == {req.load_operation} + +class _RecordingExecutor: + def __init__(self, log: list, name: str) -> None: + self._log = log + self._name = name + + def shutdown(self, wait: bool = True) -> None: + self._log.append((self._name, "shutdown", wait)) + + +class _RecordingTier: + def __init__(self, log: list) -> None: + self._log = log + + def drain(self) -> None: + self._log.append(("tier", "drain")) + + def shutdown(self) -> None: + self._log.append(("tier", "shutdown")) + + +def test_worker_close_drains_tier_then_joins_all_executors(): + """Teardown order is load-bearing: the state tier drains (an in-flight copy + finishes against a still-mapped KV pool) and joins its own threads before + the base save/load pools go, and all of it before the pool is dropped.""" + log: list = [] + c = KimiK3OffloadConnector.__new__(KimiK3OffloadConnector) + c._state_tier = _RecordingTier(log) + c._save_executor = _RecordingExecutor(log, "save") + c._load_executor = _RecordingExecutor(log, "load") + + c.close() + + assert log == [ + ("tier", "drain"), + ("tier", "shutdown"), + ("save", "shutdown", True), + ("load", "shutdown", True), + ] + + +def test_worker_close_is_a_noop_when_no_tier_was_built(): + """Under PP or a non-owning layout the tier is never built; close must still + join the base executors and not crash on the missing tier.""" + log: list = [] + c = KimiK3OffloadConnector.__new__(KimiK3OffloadConnector) + c._state_tier = None + c._save_executor = _RecordingExecutor(log, "save") + c._load_executor = _RecordingExecutor(log, "load") + + c.close() + + assert log == [("save", "shutdown", True), ("load", "shutdown", True)] + # Idempotent: executors already shut down, no tier -> nothing more recorded. + c.close() + assert log == [ + ("save", "shutdown", True), + ("load", "shutdown", True), + ("save", "shutdown", True), + ("load", "shutdown", True), + ] + def test_the_park_does_not_leak_an_entry_per_joint_load(self): worker = _k3_worker() for i in range(4): From c6c6432f1a74122bf26fb3269e43a7c56004531b Mon Sep 17 00:00:00 2001 From: zejchen Date: Tue, 1 Sep 2026 08:58:56 +0000 Subject: [PATCH 090/151] [Kimi-K3][LMCache] Cut comment density in the new offload modules The maintainer flagged the new hybrid/kimi_k3 modules at ~47% comment+ docstring density. This trims the restating and over-narrated prose while preserving every load-bearing "why" the review relies on -- the findings that justify a non-obvious line stay; the sentences that re-describe the code go. Per-module comment+doc density (measured), before -> after: page_unit_geometry.py 65% -> 45% (201 -> 172 lines) state_object.py 59% -> 42% (257 -> 230 lines) state_tier.py 51% -> 40% (385 -> 368 lines) kimi_k3/connector.py 55% -> 40% (747 -> 702 lines) What was kept (load-bearing, unchanged in substance): * the signed=False xxh64 overflow finding (27/46 stores) in state_object.key * the pack/batched_put leak guards and busy_loop=False store rationale * the double-unpin backstop and two-lane head-of-line rationale in state_tier * the two-legs-different-identity + no-tier-arm rationale in _arm_joint_loads (the reopened-#5 fix) and the get_finished no-tier drain * the should_defer_free pure-query / release_stalled_save mutator split * the connector_completion "return False" contract note * the logical-vs-physical PAGE-unit granularity trap and the bare _page_unit_regions() call note in page_unit_geometry What was cut: worked-example arithmetic kept as one line instead of a paragraph, restated method contracts, and repeated cross-references. No code changed; comment and docstring text only. --- .../offload/hybrid/kimi_k3/connector.py | 203 +++++++----------- .../offload/hybrid/kimi_k3/state_object.py | 144 +++++-------- .../offload/hybrid/kimi_k3/state_tier.py | 116 +++++----- .../attentions/page_unit_geometry.py | 109 ++++------ 4 files changed, 228 insertions(+), 344 deletions(-) diff --git a/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py b/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py index 52ebc22e0c..008b38453b 100644 --- a/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py +++ b/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py @@ -67,14 +67,12 @@ def __init__(self, config) -> None: # Inert until a request has both legs, which only a joint boundary # produces; costs one dict lookup per report otherwise. self._joint_park = _JointPark() - # Stores this worker could not even attempt because the tier never - # built. Drained in `get_finished` into the SAME completion channels a - # real tier failure uses (index-failed + source-release), so the engine - # unpins their PAGE units now. This is a *worker-process* field; the - # scheduler's report path lives in a different process and cannot see a - # set written here -- the earlier code wrote a scheduler-only attribute - # from the worker, which does not exist here and raised AttributeError - # on the no-tier store path, taking the whole step's KV loads/saves down + # Stores this worker could not attempt because the tier never built. + # Drained in `get_finished` into the same completion channels a real tier + # failure uses (index-failed + source-release), so the engine unpins + # their PAGE units now. A *worker-process* field: writing a scheduler-only + # attribute from here (as the earlier code did) raised AttributeError on + # the no-tier store path and took the whole step's KV loads/saves down # with it (super().start_load_kv never ran). self._store_failed_no_tier: set[StateStoreOperationId] = set() @@ -123,9 +121,8 @@ def _build_state_tier(self, transfer_tensors) -> None: ) return # The geometry the bytes are written under, folded into every key so a - # build that changed any of it cannot read another's images. Read from - # the runtime rather than recomputed, so there is one owner of the - # string and the HBM and CPU sides cannot disagree about it. + # changed build cannot read another's images. Read from the runtime, not + # recomputed, so HBM and CPU sides share one owner of the string. spec = getattr(getattr(backend, "model_runner", None), "state_runtime", None) spec = getattr(spec, "checkpoint_spec", None) layout_id = getattr(spec, "layout_id", None) @@ -148,13 +145,11 @@ def _build_state_tier(self, transfer_tensors) -> None: ) return - # The load direction reads `state_entry_views` (validated just above); - # the STORE direction reads `page_unit_views`, a *different* backend - # method (StateByteCodec.put -> backend.page_unit_views). A backend that - # implements one but not the other would build the tier and pass every - # load, then AttributeError on the first store -- and the tier's blind - # `except Exception` masks it as an endlessly "failed" store. Probe the - # store method at construction so the mismatch fails fast and visibly. + # The store reads `page_unit_views`, a *different* method than the load's + # `state_entry_views` (validated above). A backend with one but not the + # other would build the tier, pass every load, then AttributeError on the + # first store -- which the tier's blind `except` masks as an endlessly + # "failed" store. Probe it here so the mismatch fails fast and visibly. if not callable(getattr(backend, "page_unit_views", None)): logger.warning( "kimi_k3 offload: %s has state_entry_views but no callable " @@ -164,12 +159,10 @@ def _build_state_tier(self, transfer_tensors) -> None: ) return - # The store reads the checkpoint's PAGE units and the load writes an - # Active Slot, so the blob has to be the same length both ways. They are - # equal for K3 (a checkpoint covers the whole slot), and a model where - # they differ would silently truncate the store or over-read the load -- - # so it is asserted here rather than assumed, at the one point both - # numbers are in scope. + # The store reads PAGE units and the load writes an Active Slot, so the + # blob must be the same length both ways (equal for K3: a checkpoint + # covers the whole slot). A model where they differ would truncate the + # store or over-read the load -- checked here, where both are in scope. image_bytes = int(getattr(spec, "image_bytes", 0) or 0) if image_bytes and image_bytes != entry_bytes: logger.warning( @@ -206,14 +199,11 @@ def _build_state_tier(self, transfer_tensors) -> None: worker_id=rank, layout_id=layout_id, ) - # ONE pool, shared with paged KV, and one `cache_policy` with it. - # A request writes its KV chunks and its one state object inside the - # same prefill window, so both enter LMCache's LRU together and cool at - # the same rate -- state is retired alongside its own KV, which is what - # we want, since a boundary whose KV is gone is worthless. Two pools - # would buy independent eviction policies that must drift, while a joint - # boundary needs both legs to survive together. - # `LMCACHE_MAX_LOCAL_CPU_SIZE` is the one size to tune. + # ONE pool, shared with paged KV. A request writes its KV chunks and its + # one state object in the same prefill window, so both enter LMCache's + # LRU together and cool at the same rate -- exactly right, since a joint + # boundary needs both legs to survive together and a boundary whose KV is + # gone is worthless. `LMCACHE_MAX_LOCAL_CPU_SIZE` is the one size to tune. codec.bind_storage_manager(self._engine.storage_manager) # No index here: StateOffloadIndex lives in the engine process; both # directions report and the engine applies. @@ -237,35 +227,26 @@ def start_load_kv(self, metadata) -> None: def _arm_joint_loads(self, metadata) -> None: """Hold a request owning both legs until both report. - Both legs surface on the KV completion channel, so an id in both would + Both legs surface on the KV completion channel, so an unheld id would collapse into one wake and resume the suffix prefill while the other transfer is still writing. - **The two legs do not report the same identity.** The KV worker reports - `_load_completion_id(req)`, which is `req.load_operation` whenever the - scheduler issued one -- and `build_connector_meta` issues one for every - load, so in practice it is always the typed `LoadOperationId`. The state - tier is keyed by request and reports the bare id. Arming under the bare - id therefore parked nothing the KV leg could ever settle: `waits_for` - was false for the `LoadOperationId`, the KV completion passed straight - through, and the engine could resume the suffix prefill while the state - H2D was still writing the Active Slot -- silent wrong output, and a park - entry leaked per joint load. The park is filed under the KV identity, - which is also the one that has to reach the engine. + **The two legs report different identities.** The KV worker reports the + typed `LoadOperationId` (always issued for a load); the state tier the + bare req id. So the park is filed under the KV identity (the one that has + to reach the engine), with the bare id aliased onto it. Arming under the + bare id instead parked nothing the KV leg could settle -- it passed + through and the engine resumed prefill while the state H2D was still + writing: silent wrong output, one leaked park per joint load. **No tier is armed exactly like a tier.** An earlier guard skipped the - arm when `_state_tier is None`, reasoning that an unsettleable state leg - left the park owing the KV leg forever. But `_start_state_loads` fails - these loads (`_fail_state_loads` -> `settle_state(ok=False)`), and with - the arm in place that failure lands on a real park entry -- the pair is - marked failed and owes only the KV leg. `get_finished` then drains the - park on the no-tier path too (before its `_state_tier is None` return), - so the KV completion settles the pair into `failed_loading` and the - request recomputes. Skipping the arm instead let the KV leg pass through - as `finished_loading`, and `Scheduler._settle_state_load(ok=True)` - counted a state restore that never happened -- silent wrong output. The - arm is what makes the failure reportable; the drain is what keeps it - from leaking (the KV completion always arrives and releases the entry). + arm when `_state_tier is None`; but `_start_state_loads` fails these + loads, and only an armed park turns that failure into a real + `failed_loading` (recompute). Skipping it let the KV leg pass through as + `finished_loading`, and `Scheduler._settle_state_load(ok=True)` counted a + state restore that never happened -- silent wrong output. `get_finished` + drains the park on the no-tier path too, so the arriving KV completion + settles the pair and the entry cannot leak. """ loads = getattr(metadata, "state_loads", None) or () state_ids = {req_id for req_id, _h, _slot in loads} @@ -284,20 +265,13 @@ def _arm_joint_loads(self, metadata) -> None: kv_id=self._load_completion_id(req), ) else: - # State-only load (no KV leg -- a plain HBM-index miss where the - # KV is resident but the recurrent state is not). Arm it too, on + # State-only load (KV resident, recurrent state not). Arm it on # the state leg alone. Its SUCCESS already reached the engine via - # the _settle_joint passthrough (the tier reports the bare id and - # the engine waits on it), but its FAILURE did not: _fail_state_ - # loads (including the no-tier path) settled a park that was never - # armed, so nothing was emitted and the request sat in - # WAITING_FOR_REMOTE_KVS for the life of the process -- - # reconcile_orphan_load_slots only reclaims slots of already- - # deallocated seqs, it never wakes a live parked request. Arming - # on the state leg alone makes both outcomes land on a real entry: - # take_ready surfaces success into finished_loading and failure - # into failed_loading (recompute), under the same bare id the - # passthrough already used, so the engine matches it unchanged. + # the `_settle_joint` passthrough, but its FAILURE did not: an + # unarmed park emitted nothing, so the request sat in + # WAITING_FOR_REMOTE_KVS forever (the orphan reclaimer never wakes + # a live parked request). Arming makes `take_ready` surface either + # outcome under the same bare id the passthrough already used. self._joint_park.arm( req.req_id, needs_kv=False, @@ -376,22 +350,15 @@ def get_finished(self): ) self._store_failed_no_tier = set() if self._state_tier is None: - # Both joint and state-only loads were armed by `_arm_joint_loads` - # even with no tier, and `_fail_state_loads` has already failed their - # state leg. Drain the park here -- before the return -- with empty - # tier reports (the tier that would produce them never built). - # * A joint load (needs_kv) now owes only the KV leg: its landed KV - # completion settles the pair into `failed_loading` and the - # request recomputes, instead of flowing through as - # `finished_loading` and being miscounted a state restore by - # `Scheduler._settle_state_load(ok=True)`. The KV completion also - # releases the entry, so the arm cannot leak. - # * A state-only load (no KV leg) was released the moment - # `_fail_state_loads` settled its only leg, so `take_ready` - # (inside `_settle_joint`) surfaces it into `failed_loading` now - # and the request recomputes -- rather than sitting in - # WAITING_FOR_REMOTE_KVS until an abandon window that never wakes - # a live parked request. + # `_arm_joint_loads` armed these even with no tier and + # `_fail_state_loads` failed their state leg; drain the park here, + # before the return, with empty tier reports. A joint load then owes + # only the KV leg -- its arriving KV completion settles the pair into + # `failed_loading` (recompute) and releases the entry, instead of + # passing through as a miscounted state restore. A state-only load + # was released when its one leg failed, so `take_ready` surfaces it + # into `failed_loading` now rather than leaving it stuck in + # WAITING_FOR_REMOTE_KVS. out.finished_loading, out.failed_loading = self._settle_joint( out.finished_loading, out.failed_loading, set(), set() ) @@ -401,16 +368,12 @@ def get_finished(self): out.finished_loading, out.failed_loading = self._settle_joint( out.finished_loading, out.failed_loading, state_done, state_failed ) - # Store reports have no request identity -- by the time one lands its - # owner is long gone -- so they ride the connector-owned channel; the - # aggregator's quorum is failure-dominant, which is what resolves a - # partial store instead of pinning the key. - # - # The operation, not the bare hash. `KVOutputAggregator` tombstones - # every `(channel, operation_id)` it has taken quorum on, so a hash - # alone made the second store of a re-evicted prefix a duplicate: it - # was dropped before quorum, its pin waited for stale reclamation, and - # the CPU index never learned the bytes were back. + # Store reports have no request identity (the owner is long gone), so + # they ride the connector-owned channel with its failure-dominant quorum. + # Keyed by operation, not bare hash: `KVOutputAggregator` tombstones each + # `(channel, operation_id)` it takes quorum on, so a bare hash made the + # second store of a re-evicted prefix a dropped duplicate -- its pin + # waited for stale reclamation and the CPU index never learned it was back. for op in indexed: out.connector_completions.add( ConnectorCompletion(STATE_INDEX_CHANNEL, op, True) @@ -599,17 +562,15 @@ def _save_is_stall_escaped(self, seq) -> bool: def should_defer_free(self, seq) -> bool: """Pure query: base behaviour, plus the stall escape. - An active load is checked *first*: the base holds blocks while a load is - still reading/writing them, and the escape must not override that -- a - stalled-save request can still have a live load into the same table, and - releasing mid-load is free-while-writing corruption. - - This is a predicate only. `_is_preemptable`/`_maybe_release_deferred` - probe it, so it must not mutate. The `_save_tracker` cleanup the escape - needs on the preempt free (`block_manager.deallocate`, no - `request_finished`) is done by `release_stalled_save`, which the - scheduler calls at that free; the finished path pops the tracker in - `request_finished` (its `not should_defer_free` guard reads False here). + An active load is checked *first*: the escape must not override the base + holding blocks under a live load, or releasing mid-load is + free-while-writing corruption (a stalled-save request can still have one). + + Predicate only -- `_is_preemptable`/`_maybe_release_deferred` probe it, so + it must not mutate. The `_save_tracker` cleanup the escape needs on the + preempt free is done by `release_stalled_save`; the finished path pops + the tracker in `request_finished` (its `not should_defer_free` guard + reads False here). """ if self._has_active_load(seq): return True @@ -622,14 +583,12 @@ def release_stalled_save(self, seq) -> None: The mutator half of the escape in `should_defer_free`. The scheduler calls this at `preempt`'s `block_manager.deallocate`, which runs no - `request_finished`, so nothing else removes this sid from - `_save_tracker`. Left in, the base save loop would later (once the stall - clears and `_may_emit_save` re-opens) emit a save reading a now-freed, - possibly-reused `seq.block_table`: silent cross-prefix corruption. The - loop does not re-check liveness, so the entry has to go the moment its - blocks are surrendered. Dropping the save is the intended outcome (this - request's KV goes un-offloaded rather than wedging the engine); guarded - by the same escape predicate so a non-stalled save is never dropped. + `request_finished`, so nothing else removes this sid. Left in, the save + loop would later (once the stall clears) emit a save reading a now-freed, + possibly-reused `block_table` -- silent cross-prefix corruption, as the + loop does not re-check liveness. Dropping the save is intended (this KV + goes un-offloaded rather than wedging the engine); guarded by the same + escape predicate so a non-stalled save is never dropped. """ if self._save_is_stall_escaped(seq): self._save_tracker.pop(str(seq.id), None) @@ -654,16 +613,12 @@ def _decide_load_after_alloc(self, seq, ls): joint = int(getattr(seq, "state_joint_boundary_tokens", 0) or 0) if joint <= hbm: return False, "per_req_cache_state_boundary", hbm, lmc, lmc - hbm, chunk - # Where the transfer starts, which is NOT where the request may call - # itself cached. `allocate` claimed every block the prefix walk matched, - # not just the resumable ones, so the KV below this is already in the - # pool and asking LMCache to send it back would move bytes the GPU - # holds -- and land a second copy of them in HBM, since - # `publish_loaded_prefix` keeps the existing canonical mapping and the - # freshly written blocks stay private to this request. - # - # `state_joint_claim_tokens` is floored to the chunk grid by - # `_joint_kv_boundary`, so this start is aligned whenever `hbm` was. + # Where the transfer starts -- NOT where the request may call itself + # cached. `allocate` claimed every matched block, not just resumable + # ones, so the KV below this is already resident; asking LMCache to + # resend it would land a second copy in HBM (`publish_loaded_prefix` + # keeps the canonical mapping, fresh blocks stay private). Floored to the + # chunk grid by `_joint_kv_boundary`, so aligned whenever `hbm` was. start = max(hbm, int(getattr(seq, "state_joint_claim_tokens", 0) or 0)) # The KV leg moves whole chunks and the blocks below `start` are shared, # so an unaligned start cannot be rounded down. diff --git a/atom/kv_transfer/offload/hybrid/kimi_k3/state_object.py b/atom/kv_transfer/offload/hybrid/kimi_k3/state_object.py index 57ca5e2da9..dddf830902 100644 --- a/atom/kv_transfer/offload/hybrid/kimi_k3/state_object.py +++ b/atom/kv_transfer/offload/hybrid/kimi_k3/state_object.py @@ -3,11 +3,10 @@ """One state checkpoint, one opaque object, keyed by ATOM's own hash. -`ChunkedTokenDatabase` is bypassed. `StateSlotPool._resumable_from` looks up -one integer in HBM and in this tier, and a chunker-derived key would not be the -same thing; and state's bytes cannot be sliced by token anyway -- there is no -"first three chunks hit", so chunking would produce N keys useful only -together. It also sidesteps LMCache's chunk-alignment loss. +`ChunkedTokenDatabase` is bypassed: `StateSlotPool._resumable_from` looks up one +integer, and state's bytes cannot be sliced by token anyway (there is no "first +three chunks hit", so chunking would produce N keys useful only together). It +also sidesteps LMCache's chunk-alignment loss. """ from __future__ import annotations @@ -23,15 +22,13 @@ class StateByteCodec: """Pack/unpack one entry's state and move it through `storage_manager`. - The object is an opaque flat uint8 blob: the x-packed / strided / - multi-plane state layouts cannot be expressed in LMCache's token-major - model at all. + An opaque flat uint8 blob: the x-packed / strided / multi-plane state + layouts cannot be expressed in LMCache's token-major model at all. - The two directions read different things, and that is not an oversight: - a store gathers the checkpoint's PAGE units (`page_unit_views`) because - #2045 is where the image lives, while a load scatters into the Active Slot - the resuming forward will read (`state_entry_views`). The blob is the same - ordered byte stream either way. + The two directions read different things by design: a store gathers the + checkpoint's PAGE units (`page_unit_views`, where #2045 keeps the image), a + load scatters into the Active Slot the resuming forward reads + (`state_entry_views`). The blob is the same ordered byte stream either way. """ def __init__( @@ -76,31 +73,26 @@ def key(self, h: int): """ATOM's hash, bound to the geometry the bytes were written under. *Build safety.* One prefix hash maps to a different image under a - different `num_spec`, TP size, or conv/ssm dtype. A size mismatch is - caught by `entry_bytes`; the same size with a different order reads back - silently wrong state, and a request resumed onto it produces wrong - output with nothing raised. `layout_id` already names all of it and is - enforced HBM-side -- this is the CPU side of the same check. + different `num_spec`, TP size, or conv/ssm dtype. `entry_bytes` catches a + size mismatch; the same size in a different order reads back silently + wrong state. `layout_id` names all of it and is enforced HBM-side -- this + is the CPU side of the same check. *Namespace separation.* `CacheEngineKey` has no field saying what an - entry IS, KV and state keys come from different hash functions into one - integer space, and both now share one pool. Folding `layout_id` in makes - the two spaces disjoint by construction. + entry IS, and KV and state keys now share one pool. Folding `layout_id` + in makes the two key spaces disjoint by construction. `xxh64`, not `hash((h, layout_id))`: Python salts `hash` of a str per - process, so a restart would silently orphan every entry the previous run - wrote. + process, so a restart would silently orphan every prior entry. """ import xxhash from lmcache.utils import CacheEngineKey digest = xxhash.xxh64() - # Unsigned, and it must be: an ATOM block hash is `xxh64().intdigest()`, - # which spans the full 0..2**64-1, and `BlockManager.compute_hash` - # chains it with `prefix.to_bytes(8, "little")` -- unsigned too. With - # `signed=True` every hash above 2**63-1 raised `OverflowError: int too - # big to convert` from inside the `batched_put` argument list, which is - # about half of them: a measured 27 of 46 stores. + # Unsigned, and it must be: an ATOM block hash spans the full + # 0..2**64-1. With `signed=True` every hash above 2**63-1 raised + # `OverflowError: int too big to convert` -- about half of them, a + # measured 27 of 46 stores. digest.update(int(h).to_bytes(8, "little", signed=False)) digest.update(self._layout_id.encode()) return CacheEngineKey( @@ -114,43 +106,35 @@ def key(self, h: int): def put(self, h: int, unit_ids, on_source_released=None) -> bool: """Store one checkpoint image. False when nothing was stored. - Reads the checkpoint's PAGE units, while `get` writes an Active Slot. - The asymmetry is deliberate and safe: the copy plan intersects two - *ordered byte streams*, so a blob gathered in unit order is - byte-identical to one gathered in slot order. + Reads PAGE units where `get` writes an Active Slot; safe because the + copy plan intersects two *ordered byte streams*, so a blob gathered in + unit order is byte-identical to one gathered in slot order. - A refusal is not an error -- `_allocate` returns None under CPU memory - pressure, and a whole image is refused sooner than a KV chunk, so the - state leg feels a full pool first. `puts_refused` makes that visible - instead of a silent hit-rate drift. + A refusal is not an error -- `_allocate` returns None under CPU pressure, + and a whole image is refused sooner than a KV chunk, so the state leg + feels a full pool first. `puts_refused` makes that visible. - `on_source_released` fires the moment `pack` returns, which is after - the gather and the D2H have both been synchronized -- i.e. the moment - the GPU has stopped reading the checkpoint's PAGE units. It is separate - from the return value on purpose: the units are the KV pool's, and - holding them across `batched_put` would keep an image out of the pool - for a CPU operation that cannot touch them. + `on_source_released` fires when `pack` returns -- after the gather and + D2H are synchronized, i.e. once the GPU has stopped reading the units. + Separate from the return value on purpose: the units are the KV pool's, + and holding them across `batched_put` would keep an image out of the + pool for a CPU operation that cannot touch them. """ if self._storage is None: return False - # Before the allocation, not inside the `batched_put` argument list. - # `key` is pure and can raise (it did: see the `signed` note there), and - # an argument-position raise happens *after* the allocation and *before* - # `batched_put` can take ownership -- stranding the MemoryObj exactly - # the way a throwing `pack` used to. Computing it first removes the - # window rather than guarding it. + # Computed before the allocation: `key` can raise (see the `signed` + # note), and a raise from inside the `batched_put` argument list would + # strand the MemoryObj the same way a throwing `pack` used to. key = self.key(h) obj = self._allocate(self.entry_bytes) if obj is None: self.puts_refused += 1 return False - # `batched_put` discharges the reference it is handed, so the success - # path owes nothing -- but only if it is reached. A throwing `pack` - # skips it and strands the allocation at ref_count=1, which LMCache - # reports much later as "garbage collected with ref_count=1, - # pin_count=0" and which shrinks the CPU pool by one entry per failure - # until the tier can no longer allocate at all. `get` already guards - # its own reference with `finally` for the same reason. + # `batched_put` discharges the reference it is handed -- but only if + # reached. A throwing `pack` skips it and strands the allocation at + # ref_count=1 (LMCache reports "garbage collected with ref_count=1, + # pin_count=0" much later), shrinking the CPU pool one entry per failure. + # `get` guards its own reference with `finally` for the same reason. try: self._staged.pack(self._backend.page_unit_views(unit_ids), obj) # Source first: `pack` has synchronized the stream that reads the @@ -167,11 +151,10 @@ def get(self, h: int, slot: int) -> bool: """Load one image back into Active Slot `slot`. False on a miss. The reference must be discharged here: `get_blocking` does a - `ref_count_up()` on the caller's behalf, and without the matching down - LRU drops the block from the index but never returns it to the pinned - allocator -- an entry-sized leak per hit. `finally`, so a throwing - unpack does not leak either. `put` is deliberately not symmetric: - `batched_put` discharges its own reference. + `ref_count_up()` for the caller, and without the matching down LRU drops + the block from the index but never returns it to the pinned allocator -- + an entry-sized leak per hit. `finally`, so a throwing unpack does not + leak either. (`put` differs: `batched_put` discharges its own reference.) """ if self._storage is None: return False @@ -180,12 +163,11 @@ def get(self, h: int, slot: int) -> bool: return False size = self._object_bytes(obj) if size is not None and size != self.entry_bytes: - # Unreachable while `layout_id` is in the key, which is the point: - # a hit of the wrong size means two things collided in the shared - # pool, and unpacking it would write another entry's bytes over a - # request's live state. Degrade to a miss -- the caller disowns the - # boundary and recomputes -- and count it, because a nonzero count - # means the key is no longer doing its job. + # Unreachable while `layout_id` is in the key, which is the point: a + # wrong-size hit means two things collided in the shared pool, and + # unpacking would write another entry's bytes over live state. + # Degrade to a miss (caller disowns and recomputes) and count it -- + # a nonzero count means the key is no longer doing its job. self._misfit_reads += 1 logger.warning( "state offload: hash %d came back %d bytes, expected %d; " @@ -208,14 +190,10 @@ def _object_bytes(obj) -> int | None: """Bytes in a `MemoryObj`, or None when it will not say. `get_size()` is the documented accessor; the tensor is the fallback for - the buffer-backed objects that have no size of their own (their - `.tensor` is None). None means "cannot measure", never "size 0" -- - refusing a load we cannot measure would turn an unknown into a - guaranteed miss. - - Neither accessor is wrapped: an allocator that raises when asked its own - object's size is broken in a way this method must not paper over, and - swallowing it here would turn every read into a silent miss. + buffer-backed objects with no size of their own. None means "cannot + measure", never "size 0". Neither accessor is wrapped -- an allocator + that raises when asked its own object's size is broken in a way this + method must not paper over into a silent miss. """ get_size = getattr(obj, "get_size", None) if callable(get_size): @@ -238,16 +216,12 @@ def contains(self, h: int) -> bool: def _allocate(self, nbytes: int) -> Any: from lmcache.v1.memory_management import MemoryFormat - # `fmt` is inert: the shape/dtype above already force a flat opaque - # blob. It is passed only because `MixedMemoryAllocator.allocate` - # rejects anything outside its tensor formats -- same reason the KV path - # sets `self._engine.fmt`. - # - # `busy_loop=False` because this is a *store*. LMCache's own - # `LocalCPUBackend.allocate` says busy_loop "should only be used for - # retrieve", since "many stores happen concurrently (if they busy_loop, - # deadlock happens)" -- and its default is True, under which a full pool - # spins forever instead of returning the None this caller handles. + # `fmt` is inert (shape/dtype force a flat blob); passed only because + # `MixedMemoryAllocator.allocate` rejects anything outside its tensor + # formats. `busy_loop=False` because this is a *store*: LMCache warns + # busy_loop "should only be used for retrieve" (concurrent stores + # deadlock), yet defaults it True -- under which a full pool spins + # forever instead of returning the None this caller handles. return self._storage.allocate( torch.Size([nbytes]), torch.uint8, diff --git a/atom/kv_transfer/offload/hybrid/kimi_k3/state_tier.py b/atom/kv_transfer/offload/hybrid/kimi_k3/state_tier.py index 68626fd55c..a0a167165e 100644 --- a/atom/kv_transfer/offload/hybrid/kimi_k3/state_tier.py +++ b/atom/kv_transfer/offload/hybrid/kimi_k3/state_tier.py @@ -1,15 +1,13 @@ # SPDX-License-Identifier: MIT """Worker-side store and load driver for the state offload tier. -One executor of its own, separate from the KV connector's, so state and KV -transfers cannot block each other -- but a single one shared by this tier's own -stores and loads; see `submit_load`. - -This class **reports, and the engine applies**. `StateOffloadIndex` lives in the -engine process, so nothing here can index a hash directly: `take_store_reports` -and `get_finished` hand their sets to the connector. The failed-hash set exists -to resolve the aggregator's quorum -- without it a partial store pins the hash -forever. +Its own executors, separate from the KV connector's, so state and KV transfers +cannot block each other; see `__init__` for the store/load lane split. + +This class **reports, and the engine applies**: `StateOffloadIndex` lives in the +engine process, so `take_store_reports`/`get_finished` hand their sets to the +connector rather than index a hash here. The failed-hash set resolves the +aggregator's quorum -- without it a partial store pins the hash forever. """ import logging @@ -36,43 +34,37 @@ class StateOffloadTier: def __init__(self, codec, *, max_workers: int = 1, staging_lanes: int = 2) -> None: self.codec = codec # Two lanes, not one queue. A load is on the TTFT critical path and a - # store is not, but a single serial executor made that ordering - # unenforceable: a load submitted in a later scheduler step queued - # behind every store already sitting in front of it, and one store - # stuck in gather/D2H blocked every later load for as long as it ran. - # Putting same-step loads first in the submit order does not help -- - # it cannot overtake work already queued. + # store is not; a single serial executor made that unenforceable -- a + # later-step load queued behind every store in front of it, and one + # store stuck in gather/D2H blocked every later load. Submit order + # cannot overtake work already queued. self._load_executor = ThreadPoolExecutor( max_workers=max_workers, thread_name_prefix="lmc-state-load" ) self._store_executor = ThreadPoolExecutor( max_workers=max_workers, thread_name_prefix="lmc-state-store" ) - # What the lanes share is HBM, not a queue. `StagedTransfer` keeps its - # staging buffer in `threading.local` and sizes it to a whole entry - # (~55 MiB on K3), so each lane inside a transfer holds one. The + # What the lanes share is HBM, not a queue. `StagedTransfer` keeps a + # per-thread staging buffer sized to a whole entry (~55 MiB on K3). The # budget is stated here rather than left implicit in the thread count: - # 2 gives each lane its own buffer, so a load never waits on a store at - # all; 1 makes them share one, halving the standing HBM at the cost of - # a load waiting out the single in-flight store. Either way a load - # never queues behind a *backlog* of stores, which is the actual - # head-of-line problem. + # 2 gives each lane its own buffer (a load never waits on a store); 1 + # shares one, halving standing HBM at the cost of a load waiting out the + # single in-flight store. Either way a load never queues behind a + # *backlog* of stores, which is the actual head-of-line problem. self._staging_budget = threading.BoundedSemaphore(max(1, int(staging_lanes))) self._lock = threading.Lock() self._done: set[str] = set() self._failed: set[str] = set() self._inflight: set = set() - # The store path's reports, drained by `take_store_reports`. Sets of - # `StateStoreOperationId`, not of bare hashes: the engine settles the - # pin for that exact generation, and the aggregator would tombstone a - # bare hash after the first store of it. + # Store reports, drained by `take_store_reports`. Sets of + # `StateStoreOperationId`, not bare hashes: the engine settles the pin + # for that exact generation, and the aggregator would tombstone a bare + # hash after its first store. # - # `_source_released` is a separate report from `_indexed` because the - # two answer different questions and land at different times. The - # source is the KV pool's PAGE units and is free the instant the D2H - # drains; whether the CPU put succeeded is decided afterwards and - # cannot touch them. Reporting only the second would hold an image out - # of the pool across an operation that does not need it. + # `_source_released` is separate from `_indexed`: the source units are + # free the instant the D2H drains, while whether the CPU put succeeded + # is decided afterwards and cannot touch them. Reporting only the second + # would hold an image out of the pool across a CPU-only operation. self._source_released: set = set() self._indexed: set = set() self._index_failed: set = set() @@ -159,15 +151,12 @@ def _do_store(self, op, unit_ids) -> None: released = False def _source_released() -> None: - # Fires from `codec.put` the moment `pack`'s gather+D2H have - # drained (state_object.put), before `batched_put`. Publish the - # release *here*, early: the PAGE units are the KV pool's and the - # GPU has stopped reading them, so hand them back now instead of - # holding a whole image out of the pool across the CPU put. Under - # `self._lock` because `take_source_releases` drains the set from - # the engine thread; the flag makes the end-of-store backstop a - # no-op so the release is emitted exactly once (a second emission - # would double-unpin on the engine side). + # Fires from `codec.put` once `pack`'s gather+D2H drain, before + # `batched_put`: the GPU has stopped reading the units, so hand them + # back now instead of holding a whole image out of the pool across + # the CPU put. Under `self._lock` (the engine thread drains the set); + # the flag makes the end-of-store backstop a no-op so the release is + # emitted once -- a second emission would double-unpin engine-side. nonlocal released with self._lock: if released: @@ -184,11 +173,11 @@ def _source_released() -> None: on_source_released=_source_released, ) ) - except Exception: # deliberately blind, see below - # Deliberately blind. `codec.put` reaches into LMCache, whose - # failure modes are its own, and a store that cannot happen must - # cost one checkpoint's CPU copy -- not this worker thread, whose - # death would strand every request parked on a later load. + except Exception: # deliberately blind + # `codec.put` reaches into LMCache, whose failure modes are its own. + # A store that cannot happen must cost one checkpoint's CPU copy -- + # not this worker thread, whose death would strand every request + # parked on a later load. logger.warning( "state offload: store of hash %d (generation %d) failed", op.prefix_hash, @@ -196,28 +185,25 @@ def _source_released() -> None: exc_info=True, ) with self._lock: - # Backstop, not the primary path: on success the D2H callback above - # already published the release early, so this must NOT add it again - # -- a second emission across two engine drains is the double-unpin. - # It still fires on the paths the callback never reached: a refused - # allocation never read the units, and a throwing `pack` drained the - # device before it propagated (`StagedTransfer._drain_device`), so - # the GPU has stopped reading them there too. Withholding it on those - # paths would hold an image out of the KV pool until the stale - # reclaimer noticed. + # Backstop, not the primary path: on success the callback already + # published the release, so this must NOT re-add it (that second + # emission is the double-unpin). It fires only on paths the callback + # never reached -- a refused allocation never read the units, and a + # throwing `pack` drained the device first + # (`StagedTransfer._drain_device`) -- where withholding it would hold + # an image out of the pool until the stale reclaimer noticed. self._store_submitted_at.pop(op, None) if not released: released = True self._source_released.add(op) # Report, never apply: `StateOffloadIndex` lives in the engine - # process and this runs in a spawned runner. The engine applies - # these via KVConnectorOutput. + # process; the engine applies these via KVConnectorOutput. if stored: self._indexed.add(op) else: # The failure channel lets the aggregator take quorum on - # `indexed | index_failed` instead of waiting for a second - # report that will never come from this rank. + # `indexed | index_failed` rather than await a second report + # that will never come from this rank. self._index_failed.add(op) def take_store_reports(self) -> tuple[set, set]: @@ -296,12 +282,10 @@ def __init__(self) -> None: self._failed: set = set() self._ready: set = set() self._ready_failed: set = set() - # The two legs do not report the same identity. The KV leg reports - # whatever `_load_completion_id` yields -- a typed `LoadOperationId` - # whenever the scheduler issued one, which for an offload load is - # always -- while the state tier is keyed by request and reports the - # bare id. The park is filed under the KV identity, because that is - # what has to reach the engine on `finished_loading`; this maps the + # The two legs report different identities: the KV leg reports its typed + # `LoadOperationId` (always issued for an offload load), the state tier + # the bare request id. The park is filed under the KV identity -- that + # is what reaches the engine on `finished_loading` -- and this maps the # bare id onto it so a state report can find its own park. self._alias: dict = {} self._alias_of: dict = {} diff --git a/atom/model_ops/attentions/page_unit_geometry.py b/atom/model_ops/attentions/page_unit_geometry.py index f862582384..814eae450b 100644 --- a/atom/model_ops/attentions/page_unit_geometry.py +++ b/atom/model_ops/attentions/page_unit_geometry.py @@ -3,18 +3,13 @@ """Where a checkpoint image's bytes live in the MLA paged pool. -This is the destination side of a K3 PAGE-copy checkpoint: given logical block -(PAGE unit) ids, it says which addresses -- and which tensor views -- of the KV -pool a checkpoint copy lands on. `_KimiMLAGDNCommon` mixes it in over -`GDNStateMixin`, whose stubs raise `NotImplementedError` for a pool it does not -own. - -Kept in its own module, free of any GPU/aiter import, for one reason: it is the -CPU tier's only seam onto PAGE units, and it is pure arithmetic over -`self.model_runner`. Living here it can be unit-tested on a runner with no GPU -(the K3 builder's own module imports aiter at load and is `importorskip`ped on -CI), so the geometry that decides where every checkpoint byte goes is actually -exercised there. +The destination side of a K3 PAGE-copy checkpoint: given logical block (PAGE +unit) ids, it gives the addresses and tensor views of the KV pool a checkpoint +copy lands on. `_KimiMLAGDNCommon` mixes it in over `GDNStateMixin`. + +Kept aiter/GPU-import-free and pure over `self.model_runner` so this geometry -- +which decides where every checkpoint byte goes -- can be unit-tested on a +GPU-less runner (the K3 builder's own module is `importorskip`ped on CI). """ from __future__ import annotations @@ -36,23 +31,15 @@ class PageUnitGeometryMixin: def _page_unit_regions(self) -> tuple[np.ndarray, np.ndarray]: """Base address and per-unit stride of every region a PAGE id owns. - The destination side of a checkpoint copy. `GDNStateMixin` knows where - a state slot's bytes are; this knows where a KV block's are, because - this class owns the MLA pool. - - `kv_cache` is `(rows, physical_blocks, physical_block_size, entry)`, - so a block owns one contiguous region per row and the rows are a fixed - stride apart. Affine in the block id, and a property of the pool rather - than of any block, so it is worked out once. - - The units are the trap. `unit_ids` carries **logical** block ids -- what - `BlockPool` hands out and what `sub_pool_specs` priced -- while the - tensor is shaped in **physical** blocks, and K3's `block_ratio` is 128. - So a region is `runner.block_size` tokens wide, not - `physical_block_size`, and the two differ by exactly that ratio. The - assertion below is what makes a mix-up a startup error rather than 127 - blocks of scrambled state: it is the one relation that cannot hold if - the granularity is wrong. + `kv_cache` is `(rows, physical_blocks, physical_block_size, entry)`: a + block owns one contiguous region per row, rows a fixed stride apart. + Affine in the block id and a pool property, so it is computed once. + + The units are the trap: `unit_ids` carries **logical** block ids while + the tensor is shaped in **physical** blocks (K3's `block_ratio` is 128), + so a region is `runner.block_size` tokens wide, not `physical_block_size`. + The granularity assertion below is the one relation that cannot hold if + the two are confused -- a startup error, not 127 blocks of scrambled state. """ runner = self.model_runner cache = runner.kv_cache @@ -108,22 +95,17 @@ def page_unit_views(self, unit_ids: Sequence[int]) -> list[torch.Tensor]: `_page_unit_regions` gives raw addresses for the Triton descriptor; the LMCache packer takes `list[torch.Tensor]`, so the same bytes need naming - as views. This is the tier's only new seam -- a load writes the Active - Slot directly and reuses `state_entry_views`. - - Order is the contract: unit major, row minor, the same ravel as - `_page_unit_bases`. Cross-build safety is `layout_id` in - `StateByteCodec.key`, not this order. + as views. Order is the contract: unit major, row minor, the same ravel + as `_page_unit_bases` (cross-build safety is `layout_id` in + `StateByteCodec.key`, not this order). `unit_ids` carries *logical* block ids while `kv_cache` is shaped in - *physical* ones (K3's ratio is 128). Flattening the two block axes and - indexing by `unit * block_size` is that conversion -- the same - arithmetic `_page_unit_bases` does in addresses, so the two cannot - drift. Range-checked against the logical count for the same reason. - - `_page_unit_regions` runs first for its side effect: the contiguity and - granularity checks live there. Its return (base/stride addresses) is for - the Triton descriptor, not this view path -- called bare, not unpacked. + *physical* ones (K3's ratio is 128); indexing by `unit * block_size` is + that conversion, range-checked against the logical count. + + `_page_unit_regions` runs first only for its side effect (the contiguity + and granularity checks). Its return is for the Triton descriptor, not + this view path -- called bare, not unpacked. """ self._page_unit_regions() cache = self.model_runner.kv_cache @@ -148,35 +130,24 @@ def page_unit_views(self, unit_ids: Sequence[int]) -> list[torch.Tensor]: # ---- cut the padding tail off the final unit ---- # - # An image is `image_bytes`, but it occupies `ceil(image_bytes / - # page_unit_bytes)` WHOLE units, so the last one is mostly padding: at - # K3's 2,138,112 B unit and 58,079,232 B image that is 28 units = - # 59,867,136 B, of which 1,787,904 belong to no image. - # - # `_checkpoint_copy_plan` already knows this -- it hands - # `plan_segmented_copy` the unit stream *and* `spec.image_bytes`, so the - # HBM copy writes only the leading `image_bytes` and leaves the tail - # alone. Gathering whole units for the CPU tier skipped that argument - # and asked the packer for the padding too, against a MemoryObj - # `StateByteCodec.put` sizes at `entry_bytes` -- which IS `image_bytes`. - # Every store died with "MemoryObj tensor is too small for 59867136 - # bytes; got 58079232". - # - # The trim is byte-exact rather than view-aligned because the image does - # not end on one: 58,079,232 leaves 350,208 B of the last unit in use, - # which is 20.96 rows of it. That view is reinterpreted as `uint8` and - # sliced; the packer sizes segments as `numel * element_size`, so it - # costs nothing to accept. The load side needs no counterpart -- it - # scatters into `state_entry_views`, which sums to `entry_bytes`. + # An image occupies `ceil(image_bytes / page_unit_bytes)` WHOLE units, + # so the last one is mostly padding. The HBM copy already trims it + # (`_checkpoint_copy_plan` passes `plan_segmented_copy` both the unit + # stream and `spec.image_bytes`); gathering whole units for the CPU tier + # skipped that and offered the packer the padding too, against a + # MemoryObj `StateByteCodec.put` sizes at `entry_bytes` == `image_bytes` + # -- so every store died with "MemoryObj tensor is too small". # - # Inline rather than a helper method: the unit tests drive this function - # with a `SimpleNamespace` stand-in for `self`, so a second attribute - # would have to be stubbed by every one of them. + # Byte-exact, not view-aligned: an image need not end on a row boundary, + # so the boundary view is reinterpreted as `uint8` and sliced (the + # packer sizes segments as `numel * element_size`). The load side needs + # no counterpart -- it scatters into `state_entry_views`, which sums to + # `entry_bytes`. Inline, not a helper: the tests drive this with a + # `SimpleNamespace` `self`, and a helper would be one more stub each. runtime = getattr(self.model_runner, "state_runtime", None) spec = None if runtime is None else runtime.checkpoint_spec - # `getattr`: a fork build carries no spec, and the unit tests hand this - # a stand-in that carries no `image_bytes`. Either way there is no image - # size to trim against, and the whole-unit stream is the right answer. + # `getattr`: a fork build or a test stand-in carries no `image_bytes` -- + # no image size to trim against, so the whole-unit stream is correct. budget = int(getattr(spec, "image_bytes", 0) or 0) if spec else 0 if not budget: return views From 9bc652ce60729be20a6672e861b255e6a2b374e9 Mon Sep 17 00:00:00 2001 From: zejchen Date: Tue, 1 Sep 2026 10:40:09 +0000 Subject: [PATCH 091/151] [Kimi-K3][LMCache] Enforce the offload save/load lifecycle at construction MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Round-3 review §4(a): DSV4 shipped without `abandon_save`. The delegating shell forwarded it through `getattr(self._impl, "abandon_save", None)`, so a missing method was not an error -- it was a silent no-op. When the scheduler reclaimed a stalled DSV4 save, nothing dropped the inflight entry; `has_pending_work` never cleared and the engine busy-looped with every GPU idle. The shell guard I added earlier only stopped the AttributeError crash; it did not make the save actually get abandoned. Root cause is structural: `DSV4OffloadScheduler` subclasses `OffloadSchedulerMixin` directly (not `DenseOffloadScheduler`), so it silently misses every Dense-only lifecycle method. Nothing forced it to implement them. Fix: make `OffloadSchedulerMixin` inherit `ABC` and declare the six save/load lifecycle methods (`save_finished`, `abandon_save`, `release_stalled_save`, `load_failed`, `load_finished`, `cancel_pending_load`) `@abstractmethod`. Now a missing forwarder is a TypeError at construction, not a no-op behind the shell. (An `@abstractmethod` on a plain class is collected but never enforced -- the `ABC` base is what makes ABCMeta refuse the incomplete subclass.) - Give DSV4 its own `abandon_save`: it tracks a *set* of page save operations plus a SLOT sidecar (dense tracks one save per request), so a dense-style single-pop would strand the rest. Drop the whole request -- page set and sidecar -- cancelling (not finishing) each operation's statistics and dropping the tracker entry so the save loop cannot re-emit against freed blocks. Add an explicit `release_stalled_save` no-op (DSV4's `should_defer_free` has no stall escape, like dense). - Simplify the shell's four now-guaranteed lifecycle forwards (`release_stalled_save`, `abandon_save`, `cancel_pending_load`, `load_finished`) from getattr-guarded to plain forwards. The tier state-face forwards stay guarded -- those are genuinely optional on an impl and are handled separately. Tests: DSV4 abandon_save clears page set + sidecar + tracker without leaking the pending gauge, and is idempotent; DSV4 release_stalled_save is a declared no-op; the mixin refuses an incomplete subclass at construction (naming the missing method). A shared `_OffloadMixinStub` base gives the existing test doubles the now-required lifecycle defaults. The pre-existing shell-forward sweep now also proves all four simplified forwards resolve on dense/kimi_k3/dsv4. Co-Authored-By: Claude Opus 4.8 --- atom/kv_transfer/offload/_offload_common.py | 22 ++- atom/kv_transfer/offload/connector.py | 23 +-- .../offload/hybrid/dsv4/connector.py | 35 +++++ tests/test_lmcache_offload_connector.py | 137 +++++++++++++++++- tests/test_scheduler.py | 32 +++- 5 files changed, 227 insertions(+), 22 deletions(-) diff --git a/atom/kv_transfer/offload/_offload_common.py b/atom/kv_transfer/offload/_offload_common.py index 3a74995a8a..3c416f78ed 100644 --- a/atom/kv_transfer/offload/_offload_common.py +++ b/atom/kv_transfer/offload/_offload_common.py @@ -14,6 +14,7 @@ import logging import os import threading +from abc import ABC, abstractmethod from concurrent.futures import ThreadPoolExecutor from atom.kv_transfer.disaggregation.types import ( @@ -262,7 +263,7 @@ def get_finished_recv_blocks(self) -> list[int]: return [] -class OffloadSchedulerMixin: +class OffloadSchedulerMixin(ABC): """Layout-independent scheduler policy shared by dense and DSV4 offload. Subclasses own lookup construction, metadata serialization, and any @@ -270,6 +271,25 @@ class OffloadSchedulerMixin: handoff mechanics whose invariants are identical for both layouts. """ + # Save/load lifecycle contract. Declared abstract so a missing forwarder is + # a construction-time TypeError, not a silent no-op behind the delegating + # shell -- the failure mode that let DSV4 ship without abandon_save. The + # bodies differ by layout (dense keeps one save per request; DSV4 keeps a + # set plus a SLOT sidecar), so each impl supplies its own; the contract + # detail lives on those concrete overrides. + @abstractmethod + def save_finished(self, req_id) -> None: ... + @abstractmethod + def abandon_save(self, req_id) -> None: ... + @abstractmethod + def release_stalled_save(self, seq) -> None: ... + @abstractmethod + def load_failed(self, req_id) -> bool: ... + @abstractmethod + def load_finished(self, req_id) -> bool: ... + @abstractmethod + def cancel_pending_load(self, seq) -> None: ... + def _init_offload_statistics(self) -> None: """Initialize layout-independent scheduler counters.""" diff --git a/atom/kv_transfer/offload/connector.py b/atom/kv_transfer/offload/connector.py index ac8077d955..44cc7c4bc4 100644 --- a/atom/kv_transfer/offload/connector.py +++ b/atom/kv_transfer/offload/connector.py @@ -183,9 +183,12 @@ def should_defer_free(self, seq) -> bool: return self._impl.should_defer_free(seq) def release_stalled_save(self, seq) -> None: - callback = getattr(self._impl, "release_stalled_save", None) - if callback is not None: - callback(seq) + # Plain forward, not getattr-guarded: OffloadSchedulerMixin declares the + # save/load lifecycle abstract, so every _impl defines all six methods or + # fails at construction. A guard here would silently swallow a genuinely + # missing forwarder -- the exact failure mode the abstract contract exists + # to make loud. + self._impl.release_stalled_save(seq) def has_pending_work(self) -> bool: return self._impl.has_pending_work() @@ -194,9 +197,7 @@ def save_finished(self, req_id) -> None: self._impl.save_finished(req_id) def abandon_save(self, req_id) -> None: - callback = getattr(self._impl, "abandon_save", None) - if callback is not None: - callback(req_id) + self._impl.abandon_save(req_id) def load_failed(self, req_id): return self._impl.load_failed(req_id) @@ -210,13 +211,13 @@ def should_park_partial_prefill_for_load(self, seq) -> bool: return callback(seq) if callback is not None else False def cancel_pending_load(self, seq) -> None: - callback = getattr(self._impl, "cancel_pending_load", None) - if callback is not None: - callback(seq) + # Plain forward: the lifecycle contract (see release_stalled_save) + # guarantees every _impl defines this. + self._impl.cancel_pending_load(seq) def load_finished(self, req_id): - callback = getattr(self._impl, "load_finished", None) - return callback(req_id) if callback is not None else True + # Plain forward: guaranteed by the abstract lifecycle contract. + return self._impl.load_finished(req_id) def process_completions(self, output): return self._impl.process_completions(output) diff --git a/atom/kv_transfer/offload/hybrid/dsv4/connector.py b/atom/kv_transfer/offload/hybrid/dsv4/connector.py index a3f1e91279..b11dac14c5 100644 --- a/atom/kv_transfer/offload/hybrid/dsv4/connector.py +++ b/atom/kv_transfer/offload/hybrid/dsv4/connector.py @@ -2399,6 +2399,41 @@ def save_finished(self, req_id) -> None: self._save_inflight.pop(sid, None) self._finish_save_statistics(req_id) + def abandon_save(self, req_id) -> None: + """Force-drop a save the scheduler reclaimed after it stalled. + + The completion path (`save_finished`) is precise: it discards one exact + `SaveOperationId` and leaves the rest of the request's inflight set. This + is the opposite need. `_reconcile_stalled_deferred_saves` has already + freed the blocks of a save the backend never reported, and holds only + the raw request id, so drop the *whole* request unconditionally -- the + page set and the SLOT sidecar. Without this the entries linger, + `should_defer_free` stays True and `has_pending_work` never clears, and + the engine busy-loops with every GPU idle (the DSV4 twin of the dense + stall). Not a completion: the bytes were never persisted, so every + operation's statistics are *cancelled*, not finished, and the tracker + entry is dropped so the save loop cannot re-emit against freed blocks. + """ + sid = str(req_id.req_id if isinstance(req_id, SaveOperationId) else req_id) + inflight = self._save_inflight.pop(sid, None) + if inflight is not None: + for operation in inflight: + self._cancel_save_statistics(operation) + sidecar = self._sidecar_save_inflight.pop(sid, None) + if sidecar is not None: + self._cancel_save_statistics(sidecar[0]) + self._save_tracker.pop(sid, None) + + def release_stalled_save(self, seq) -> None: + """Drop bookkeeping for a stall-escaped save the scheduler is freeing. + + No-op on DSV4: like dense, its `should_defer_free` has no stall escape, + so a request with a pending page or sidecar save always defers and is + never preemptable. Defined so every offload impl answers the scheduler's + `release_stalled_save` forward uniformly (the mixin declares it abstract). + """ + return None + def connector_completion(self, completion: ConnectorCompletion) -> bool: """Apply one TP-aggregated completion owned by the DSV4 scheduler.""" diff --git a/tests/test_lmcache_offload_connector.py b/tests/test_lmcache_offload_connector.py index 754afe8bfc..232a186cb7 100644 --- a/tests/test_lmcache_offload_connector.py +++ b/tests/test_lmcache_offload_connector.py @@ -91,6 +91,32 @@ def lookup(self, token_ids, lookup_id): raise RuntimeError("lookup failed") +class _OffloadMixinStub(OffloadSchedulerMixin): + """Concrete `OffloadSchedulerMixin` for tests that exercise only frontier and + completion mechanics, not a real save/load lifecycle. + + The mixin now declares the six save/load methods abstract (a missing + forwarder is a construction-time TypeError, not a silent no-op behind the + shell). Test doubles must therefore satisfy the contract; this base fills it + with harmless defaults so the ABC constructs, and each local `_Connector` + overrides the one or two methods it asserts on. + """ + + is_producer = False + is_offload = True + + def save_finished(self, req_id) -> None: ... + def abandon_save(self, req_id) -> None: ... + def release_stalled_save(self, seq) -> None: ... + def load_failed(self, req_id) -> bool: + return False + + def load_finished(self, req_id) -> bool: + return True + + def cancel_pending_load(self, seq) -> None: ... + + def _scheduler() -> LMCacheOffloadConnectorScheduler: sched = LMCacheOffloadConnectorScheduler.__new__(LMCacheOffloadConnectorScheduler) sched._config = SimpleNamespace() @@ -2754,7 +2780,7 @@ def test_collapsed_sidecar_and_save_completion_clears_page_inflight(): def test_offload_completion_processing_calls_load_terminal_once(field, callback): calls = [] - class _Connector(OffloadSchedulerMixin): + class _Connector(_OffloadMixinStub): is_producer = False is_offload = True @@ -2791,7 +2817,7 @@ def test_aborted_parked_load_defers_owned_resources_until_terminal( _counted_as_inflight_load=True, ) - class _Connector(OffloadSchedulerMixin): + class _Connector(_OffloadMixinStub): is_producer = False is_offload = True @@ -2857,7 +2883,7 @@ def test_aborted_parked_load_consumes_already_queued_terminal(queued_field): _counted_as_inflight_load=True, ) - class _Connector(OffloadSchedulerMixin): + class _Connector(_OffloadMixinStub): is_producer = False is_offload = True @@ -2927,7 +2953,7 @@ def test_abort_cleans_load_whose_terminal_was_already_consumed( offload_load_start_tokens=4, ) - class _Connector(OffloadSchedulerMixin): + class _Connector(_OffloadMixinStub): is_producer = False is_offload = True @@ -4095,7 +4121,7 @@ def __init__(self) -> None: def deallocate(self, seq) -> None: self.deallocated.append(seq.id) - class _Connector(OffloadSchedulerMixin): + class _Connector(_OffloadMixinStub): is_producer = False is_offload = True @@ -4905,6 +4931,57 @@ def test_a_stalled_save_releases_blocks_it_never_handed_out(monkeypatch): assert "9" in s._save_inflight +def test_dsv4_abandon_save_drops_page_set_sidecar_and_tracker(): + """A DSV4 save the backend never reported must be droppable in full. + + `_reconcile_stalled_deferred_saves` frees the blocks and calls + `abandon_save(sid)` with only the raw request id. DSV4 tracks a *set* of + page save operations plus a SLOT sidecar (dense tracks one save per + request), so the dense single-pop would strand the rest: `has_pending_work` + would never clear and the engine would busy-loop with every GPU idle -- the + DSV4 twin of the dense stall. Before this method DSV4 had no `abandon_save` + at all, so the shell's guarded forward silently no-op'd and nothing was + dropped. Prove every channel clears and the pending gauge does not leak. + """ + sched = _scheduler() + sid = "7" + op_a = SaveOperationId(7, 0) + op_b = SaveOperationId(7, 1) # shared by the page set and the sidecar + sched._save_inflight[sid] = {op_a, op_b} + sched._sidecar_save_inflight[sid] = (op_b, 8, 0x1234) + sched._save_tracker[sid] = [SimpleNamespace(id=7), 0] + sched._track_save_statistics(op_a, 16) + sched._track_save_statistics(op_b, 24) + assert sched.has_pending_work() is True + + sched.abandon_save(sid) + + assert sid not in sched._save_inflight + assert sid not in sched._sidecar_save_inflight + assert sid not in sched._save_tracker + # Cancelled, not finished: the bytes never persisted, so neither the request + # count nor the inflight-token gauge may bump. + assert sched.get_statistics()["save_requests"] == 0 + assert sched.get_statistics()["saves_pending"] == 0 + assert sched.has_pending_work() is False + # Idempotent: abandoning an already-dropped save is a no-op. + sched.abandon_save(sid) + assert sched.has_pending_work() is False + + +def test_dsv4_release_stalled_save_is_a_declared_no_op(): + """DSV4's `should_defer_free` has no stall escape, so a request with a + pending save always defers and `release_stalled_save` has nothing to drop. + It exists (not inherited, not guarded away) so the abstract lifecycle + contract is satisfied and the shell can forward it unconditionally.""" + sched = _scheduler() + seq = SimpleNamespace(id=7) + sched._save_tracker["7"] = [seq, 0] + sched.release_stalled_save(seq) + # A no-op: it must not touch tracker state the way K3's override does. + assert "7" in sched._save_tracker + + def test_state_loads_are_drained_into_the_metadata_exactly_once(monkeypatch): """A second submission would write the same entry into a group the first transfer is already filling.""" @@ -5193,7 +5270,13 @@ def test_the_shells_no_impl_fallbacks_match_what_the_caller_unpacks(): accepted requests and returned nothing. Drive the shell with an `_impl` that defines nothing, and hold each - fallback against the contract its caller relies on. + remaining fallback against the contract its caller relies on. The set is + exactly the tier/policy methods that are OPTIONAL on an impl: the state face + (`take_state_reports`, `enqueue_state_*`), `chunk_size`, and the two prefill + hints. The save/load lifecycle is no longer here -- it is declared abstract + on `OffloadSchedulerMixin`, so those forwards are unconditional and every + real impl defines them (proven by the two sibling sweeps); a no-`_impl` + stub is not a valid impl for them. """ from types import SimpleNamespace @@ -5210,7 +5293,6 @@ def test_the_shells_no_impl_fallbacks_match_what_the_caller_unpacks(): # `int(getattr(..., "chunk_size", 0) or 0)` assert shell.chunk_size is None or isinstance(shell.chunk_size, int) assert shell.should_park_partial_prefill_for_load(None) is False - assert shell.load_finished(None) is True # Unchanged, so a connector with no opinion does not shrink the chunk. assert shell.adjust_prefill_chunk_after_alloc(None, 7) == 7 @@ -5301,6 +5383,47 @@ def _is_unconditional_forward(fn): ) +def test_offload_mixin_lifecycle_is_enforced_at_construction(): + """The abstract lifecycle contract is enforcement, not documentation. + + `OffloadSchedulerMixin` inherits `ABC`, so ABCMeta refuses to instantiate a + subclass that leaves any of the six save/load methods unimplemented. This is + the mechanism that turns a missing forwarder into a construction-time + TypeError instead of a silent no-op behind the delegating shell -- the + failure mode that let DSV4 ship without `abandon_save`. (On a *plain* class + `@abstractmethod` is collected but never enforced; this test would pass + vacuously, so it also guards the `ABC` base itself.) + """ + from atom.kv_transfer.offload._offload_common import OffloadSchedulerMixin + + class MissingAbandon(OffloadSchedulerMixin): + def save_finished(self, req_id): + ... + + def release_stalled_save(self, seq): + ... + + def load_failed(self, req_id): + return False + + def load_finished(self, req_id): + return True + + def cancel_pending_load(self, seq): + ... + + # abandon_save deliberately left unimplemented. + + with pytest.raises(TypeError, match="abandon_save"): + MissingAbandon() + + class AllPresent(MissingAbandon): + def abandon_save(self, req_id): + ... + + AllPresent() # the full contract constructs cleanly + + # ── kimi_k3: the two joint legs report different identities ─────────────── diff --git a/tests/test_scheduler.py b/tests/test_scheduler.py index f504c45eb0..89e2f7dc88 100644 --- a/tests/test_scheduler.py +++ b/tests/test_scheduler.py @@ -26,6 +26,32 @@ from atom.model_engine.sequence import Sequence, SequenceStatus, SequenceType from atom.sampling_params import SamplingParams + +class _OffloadMixinStub(OffloadSchedulerMixin): + """Concrete `OffloadSchedulerMixin` for scheduler tests. + + `OffloadSchedulerMixin` declares the six save/load lifecycle methods abstract + so a missing forwarder is a construction-time TypeError. These test doubles + exercise only the scheduler's deferred-free / preemption paths, so this base + fills the contract with harmless defaults and each local `_Connector` + overrides the methods it drives. + """ + + is_producer = False + is_offload = True + + def save_finished(self, req_id) -> None: ... + def abandon_save(self, req_id) -> None: ... + def release_stalled_save(self, seq) -> None: ... + def load_failed(self, req_id) -> bool: + return False + + def load_finished(self, req_id) -> bool: + return True + + def cancel_pending_load(self, seq) -> None: ... + + # ── EngineStats: spec section ──────────────────────────────────────────────── @@ -437,7 +463,7 @@ def test_aborted_load_and_save_both_finish_before_release( ) events = [] - class _Connector(OffloadSchedulerMixin): + class _Connector(_OffloadMixinStub): is_offload = True is_producer = False @@ -730,7 +756,7 @@ def test_decode_preemption_skips_save_pinned_victim(self, seq_factory): pinned_victim.append_token(10) operation = SaveOperationId(pinned_victim.id, 50) - class _Connector(OffloadSchedulerMixin): + class _Connector(_OffloadMixinStub): is_producer = False is_offload = True _do_load = False @@ -768,7 +794,7 @@ def test_decode_preemption_stalls_pinned_current_until_save_terminal( pinned.append_token(9) operation = SaveOperationId(pinned.id, 51) - class _Connector(OffloadSchedulerMixin): + class _Connector(_OffloadMixinStub): is_producer = False is_offload = True _do_load = False From 3ca550805b66f94dd9377d6a6b76a2b8ba2cefc2 Mon Sep 17 00:00:00 2001 From: zejchen Date: Tue, 1 Sep 2026 10:42:45 +0000 Subject: [PATCH 092/151] [Kimi-K3][LMCache] Route the state tier by type, not method presence MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Round-3 review §4(b): the shell/`MultiConnector` decide where to send state stores and loads by probing for a state method (`hasattr`/`getattr` on `enqueue_state_stores`). But the delegating shell defines the *entire* state face unconditionally, forwarding through `getattr` to `_impl` and returning the no-tier default when the impl lacks a method. So presence on the shell cannot tell a tier-hosting kimi_k3 impl from a dense/dsv4-page impl that has none, and `MultiConnector._first_with` -- selecting by attribute presence -- routed every state call to a dense offload shell whose `_impl` silently returns the no-tier defaults (stores recorded failed, loads dropped, reports empty). Fix: make the state tier an explicit type. Add `StateOffloadFace(ABC)` in `_offload_common` declaring the four tier methods (`enqueue_state_loads`, `enqueue_state_stores`, `take_state_reports`, `take_state_source_releases`) abstract; only `KimiK3OffloadScheduler` inherits it. The shell's `has_state_tier` now returns `isinstance(self._impl, StateOffloadFace)`, which the shell's own forwards cannot fool. DSV4 keeps its separate sidecar-channel mechanism and does not wear the face, so its routing is unchanged (it was False before and stays False). The face is deliberately narrow -- only the four tier methods. `chunk_size` is an instance attribute set in `__init__`, not a class method, so it cannot be abstract without breaking construction; it stays a plain shell forward. Tests: only kimi_k3 is a `StateOffloadFace` (dense/dsv4 are not); an impl that claims the face but omits a tier method is a construction-time TypeError; and the exact regression -- an `_impl` that merely exposes `enqueue_state_stores` but is not a real host now reads `has_state_tier is False`, where the old presence check said True. Co-Authored-By: Claude Opus 4.8 --- atom/kv_transfer/offload/_offload_common.py | 28 ++++++++++ atom/kv_transfer/offload/connector.py | 9 ++-- .../offload/hybrid/kimi_k3/connector.py | 10 +++- tests/test_lmcache_offload_connector.py | 51 +++++++++++++++++++ 4 files changed, 92 insertions(+), 6 deletions(-) diff --git a/atom/kv_transfer/offload/_offload_common.py b/atom/kv_transfer/offload/_offload_common.py index 3c416f78ed..1530adc4ca 100644 --- a/atom/kv_transfer/offload/_offload_common.py +++ b/atom/kv_transfer/offload/_offload_common.py @@ -263,6 +263,34 @@ def get_finished_recv_blocks(self) -> list[int]: return [] +class StateOffloadFace(ABC): + """The KDA state-tier surface, implemented only by the layout that hosts it. + + `MultiConnector` and the delegating shell must tell a tier-hosting impl + (kimi_k3) from one with no state tier (dense / dsv4-page) so they route + state stores and loads to the right sub. They used to probe by method + *presence* -- but the shell defines the whole face unconditionally, + forwarding through `getattr` to `_impl`, so presence could not distinguish + them and every state call fell through to a dense impl's no-tier defaults + (stores recorded failed, loads dropped, reports empty). Make the face an + explicit type: only `KimiK3OffloadScheduler` inherits it, so + `isinstance(impl, StateOffloadFace)` is the honest predicate. + + Narrow on purpose -- only the four tier methods. `chunk_size` is an instance + attribute set in `__init__`, not a class method, so it cannot be abstract + here without breaking construction; it stays a plain shell forward. + """ + + @abstractmethod + def enqueue_state_loads(self, loads) -> bool: ... + @abstractmethod + def enqueue_state_stores(self, stores) -> bool: ... + @abstractmethod + def take_state_reports(self) -> tuple[set[int], set[int]]: ... + @abstractmethod + def take_state_source_releases(self) -> set: ... + + class OffloadSchedulerMixin(ABC): """Layout-independent scheduler policy shared by dense and DSV4 offload. diff --git a/atom/kv_transfer/offload/connector.py b/atom/kv_transfer/offload/connector.py index 44cc7c4bc4..6c7b83cd3d 100644 --- a/atom/kv_transfer/offload/connector.py +++ b/atom/kv_transfer/offload/connector.py @@ -22,6 +22,7 @@ KVConnectorBase, KVConnectorSchedulerBase, ) +from atom.kv_transfer.offload._offload_common import StateOffloadFace from atom.kv_transfer.offload.config import select_offload_layout logger = logging.getLogger("atom") @@ -146,11 +147,11 @@ def has_state_tier(self) -> bool: tell a tier-hosting `kimi_k3` impl from a `dense`/`hybrid` impl that has none -- which is exactly how `MultiConnector._first_with`, selecting by attribute presence, routed every state call to a dense offload shell - whose `_impl` silently returns the no-tier defaults. Probe the impl for - the store entry point instead: the real state face lives there (only - `KimiK3OffloadScheduler` defines `enqueue_state_stores`). + whose `_impl` silently returns the no-tier defaults. Ask the impl's + *type* instead: only `KimiK3OffloadScheduler` inherits `StateOffloadFace` + (see there), so the check cannot be fooled by the shell's own forwards. """ - return getattr(self._impl, "enqueue_state_stores", None) is not None + return isinstance(self._impl, StateOffloadFace) @property def chunk_size(self) -> int | None: diff --git a/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py b/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py index 008b38453b..96bd4bd457 100644 --- a/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py +++ b/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py @@ -20,6 +20,7 @@ ) from atom.kv_transfer.offload import config as offcfg from atom.kv_transfer.offload._offload_common import ( + StateOffloadFace, max_pending_saves, pp_aware_rank_and_world, ) @@ -414,9 +415,14 @@ def _settle_joint(self, kv_done, kv_failed, state_done, state_failed): return passthrough_done | ready, passthrough_failed | ready_failed -class KimiK3OffloadScheduler(DenseOffloadScheduler): +class KimiK3OffloadScheduler(DenseOffloadScheduler, StateOffloadFace): """Scheduler side: dense KV, plus the state tier's load queue and the - save-stall guard that keeps a stopped backend from stopping the engine.""" + save-stall guard that keeps a stopped backend from stopping the engine. + + Inherits `StateOffloadFace` -- the only offload scheduler that hosts the KDA + state tier -- so routing can select it with `isinstance` rather than probing + for a method the delegating shell defines on every layout (see + `StateOffloadFace` and the shell's `has_state_tier`).""" def __init__(self, config) -> None: super().__init__(config) diff --git a/tests/test_lmcache_offload_connector.py b/tests/test_lmcache_offload_connector.py index 232a186cb7..f4016b7af0 100644 --- a/tests/test_lmcache_offload_connector.py +++ b/tests/test_lmcache_offload_connector.py @@ -5424,6 +5424,57 @@ def abandon_save(self, req_id): AllPresent() # the full contract constructs cleanly +def test_only_kimi_k3_wears_the_state_offload_face(): + """`has_state_tier` routes state stores and loads, so it must key on a + property only the tier-hosting layout has. The shell defines every state + forward, so forward presence cannot be that key -- the face type can. Only + kimi_k3 hosts the state tier; dense and dsv4-page do not.""" + from atom.kv_transfer.offload._offload_common import StateOffloadFace + from atom.kv_transfer.offload.hybrid.dsv4.connector import DSV4OffloadScheduler + + assert issubclass(KimiK3OffloadScheduler, StateOffloadFace) + assert not issubclass(DenseOffloadScheduler, StateOffloadFace) + assert not issubclass(DSV4OffloadScheduler, StateOffloadFace) + + +def test_state_offload_face_is_enforced_at_construction(): + """Like the lifecycle contract: an impl that claims the face but leaves a + tier method unimplemented is a construction-time TypeError, not a shell + forward that silently returns the no-tier default.""" + from atom.kv_transfer.offload._offload_common import StateOffloadFace + + class MissingReports(StateOffloadFace): + def enqueue_state_loads(self, loads): + return True + + def enqueue_state_stores(self, stores): + return True + + def take_state_source_releases(self): + return set() + + # take_state_reports deliberately left unimplemented. + + with pytest.raises(TypeError, match="take_state_reports"): + MissingReports() + + +def test_shell_has_state_tier_tracks_the_face_not_forward_presence(): + """The regression `StateOffloadFace` closes: an `_impl` that merely *exposes* + `enqueue_state_stores` (as any shell forward would) but is not a real state + host must NOT read as tier-hosting. The old presence check said True here and + routed state calls to a no-tier impl; the isinstance check says False.""" + from types import SimpleNamespace + + from atom.kv_transfer.offload.connector import ( + LMCacheOffloadConnectorScheduler as Shell, + ) + + shell = object.__new__(Shell) + shell._impl = SimpleNamespace(enqueue_state_stores=lambda stores: False) + assert shell.has_state_tier is False + + # ── kimi_k3: the two joint legs report different identities ─────────────── From 2de950b752a4f00b36d389581f4eba22378b63b0 Mon Sep 17 00:00:00 2001 From: zejchen Date: Tue, 1 Sep 2026 10:50:59 +0000 Subject: [PATCH 093/151] =?UTF-8?q?[offload]=20=C2=A72a:=20source=20the=20?= =?UTF-8?q?save-abandon=20window=20from=20the=20connector?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The deferred-save reclaim window -- how long a save may sit unreported before the engine frees its blocks -- was derived in scheduler.py from LMCache's `LMCACHE_EC_PIN_TIMEOUT_SEC`. That derivation IS an LMCache fact (reclaim must happen strictly after LMCache force-unpins the source, so the window is `pin_timeout + margin`), yet it lived on the scheduler, away from the rest of the offload connector's LMCache knowledge. Nothing tied the two together; a future edit to either side could silently break the ordering the safety argument rests on. Move the derivation onto the offload connector and have the scheduler ask for the result: * `_offload_common.offload_save_abandon_timeout_s()` now owns the computation, the `_SAVE_ABANDON_MARGIN_S` / `_LMCACHE_PIN_TIMEOUT_ DEFAULT_S` constants, and the process-constant memo. `OffloadScheduler Mixin.save_abandon_timeout_s()` exposes it -- concrete, because every offload variant answers identically. * The shell (`LMCacheOffloadConnectorScheduler`) forwards it plainly, so the sweep in test_lmcache_offload_connector.py picks it up as one more unconditional forward validated against dense/kimi_k3/dsv4. * `Scheduler._save_abandon_timeout_s()` reads it off `self.kv_connector` via getattr, returning 0 (reclamation disabled) when no offload connector is attached -- there is nothing to reclaim. The three call sites (`_reconcile_stalled_deferred_saves`, `reclaim_stale_state_store_ pins`, `reconcile_orphan_load_slots`) now go through it. scheduler.py no longer computes the window or holds its constants. Behaviour is unchanged for the attached-connector case; the only new behaviour is the explicit 0 when no offload connector exists, which the reclaim path already treated as "wait forever". Tests: retarget the derivation test to `offload_common`, source the window from a connector stub in `TestStalledOffloadSaveReclaim`, and add a direct test that a missing/plain connector disables reclamation. Co-Authored-By: Claude Opus 4.8 --- atom/kv_transfer/offload/_offload_common.py | 65 +++++++++++++++++++ atom/kv_transfer/offload/connector.py | 6 ++ atom/model_engine/scheduler.py | 71 +++++---------------- tests/test_lmcache_offload_connector.py | 1 + tests/test_scheduler.py | 51 +++++++++++---- 5 files changed, 126 insertions(+), 68 deletions(-) diff --git a/atom/kv_transfer/offload/_offload_common.py b/atom/kv_transfer/offload/_offload_common.py index 1530adc4ca..74eb3ce95e 100644 --- a/atom/kv_transfer/offload/_offload_common.py +++ b/atom/kv_transfer/offload/_offload_common.py @@ -415,6 +415,18 @@ def get_statistics(self) -> dict[str, int]: "saves_pending": len(self._save_inflight_tokens), } + def save_abandon_timeout_s(self) -> float: + """Seconds a deferred save may sit before the engine reclaims it. + + The value is LMCache knowledge -- it is derived from LMCache's own pin + timeout -- so it lives on the offload connector, and the scheduler asks + the connector for it rather than re-deriving the same env math on its own. + See `offload_save_abandon_timeout_s` for the safety argument. Concrete on + the mixin because every offload variant answers it identically. + """ + + return offload_save_abandon_timeout_s() + def _chunk_floor(self, tokens: int) -> int: chunk = int(self.chunk_size or 256) return (max(0, int(tokens)) // chunk) * chunk @@ -606,3 +618,56 @@ def max_pending_saves(kvc, save_workers: int) -> int: if capacity <= 0: raise ValueError("max pending saves must be a positive integer") return capacity + + +# Seconds added on top of LMCache's own pin timeout before the engine reclaims a +# save that never reported. See `offload_save_abandon_timeout_s` for why the sum +# and not the timeout itself is the safe window. +_SAVE_ABANDON_MARGIN_S = 30.0 + +# What LMCache's pin monitor uses when `LMCACHE_EC_PIN_TIMEOUT_SEC` is unset. +_LMCACHE_PIN_TIMEOUT_DEFAULT_S = 300.0 + +# Memoised result of `offload_save_abandon_timeout_s`; the reconciler asks for it +# on a 1ms poll and the answer is a process constant. +_save_abandon_timeout_s: float | None = None + + +def offload_save_abandon_timeout_s() -> float: + """Seconds a deferred offload save may sit before the engine reclaims it. + + Blocks are freed on `finished_saving`, so a lost report leaves them deferred + forever: `has_pending_kv_work()` never clears and the engine busy-loops with + every GPU idle. + + Reclaiming cannot race a live copy. `OffloadWorkerMixin._guard` reports on + both the success and the exception path, so a report is lost only when + `store()` neither returns nor raises -- it is parked inside LMCache. Then + either the parked save is not copying (LMCache force-unpinned its source + after `pin_timeout_sec`) or a save queued behind it never reached `store()`. + Both cases are safe once that window has passed. + + Derived from LMCache's own `LMCACHE_EC_PIN_TIMEOUT_SEC` rather than a knob of + its own, because that ordering IS the safety argument -- two independent env + vars could be set the wrong way round with nothing to say so. This lives on + the offload connector, not the scheduler: it is LMCache knowledge, and the + scheduler now asks the connector for it (`save_abandon_timeout_s`). + Non-positive disables reclamation. + """ + global _save_abandon_timeout_s + if _save_abandon_timeout_s is not None: + return _save_abandon_timeout_s + pin = _LMCACHE_PIN_TIMEOUT_DEFAULT_S + raw = os.environ.get("LMCACHE_EC_PIN_TIMEOUT_SEC") + if raw is not None: + try: + pin = float(raw) + except ValueError: + logger.warning( + "invalid LMCACHE_EC_PIN_TIMEOUT_SEC=%r; assuming LMCache's %.0fs " + "default for the offload save abandon window", + raw, + _LMCACHE_PIN_TIMEOUT_DEFAULT_S, + ) + _save_abandon_timeout_s = pin + _SAVE_ABANDON_MARGIN_S if pin > 0 else 0.0 + return _save_abandon_timeout_s diff --git a/atom/kv_transfer/offload/connector.py b/atom/kv_transfer/offload/connector.py index 6c7b83cd3d..27f9ec700c 100644 --- a/atom/kv_transfer/offload/connector.py +++ b/atom/kv_transfer/offload/connector.py @@ -253,6 +253,12 @@ def take_state_source_releases(self) -> set: callback = getattr(self._impl, "take_state_source_releases", None) return callback() if callback is not None else set() + def save_abandon_timeout_s(self) -> float: + # Plain forward: the abstract lifecycle contract guarantees every _impl + # defines this (concrete on OffloadSchedulerMixin). The scheduler sources + # the reclaim window from the connector because it is LMCache knowledge. + return self._impl.save_abandon_timeout_s() + def get_statistics(self) -> dict[str, int]: return self._impl.get_statistics() diff --git a/atom/model_engine/scheduler.py b/atom/model_engine/scheduler.py index 272f032a8c..c5d6884dc8 100644 --- a/atom/model_engine/scheduler.py +++ b/atom/model_engine/scheduler.py @@ -55,18 +55,6 @@ # reconciler no-ops until this interval has passed. _SAVE_RECONCILE_INTERVAL_S = 5.0 -# Seconds added on top of LMCache's own pin timeout before the engine reclaims a -# save that never reported. See `_offload_save_abandon_timeout_s` for why the sum -# and not the timeout itself is the safe window. -_SAVE_ABANDON_MARGIN_S = 30.0 - -# What LMCache's pin monitor uses when `LMCACHE_EC_PIN_TIMEOUT_SEC` is unset. -_LMCACHE_PIN_TIMEOUT_DEFAULT_S = 300.0 - -# Memoised result of `_offload_save_abandon_timeout_s`; the reconciler asks for -# it on a 1ms poll and the answer is a process constant. -_SAVE_ABANDON_TIMEOUT_S: float | None = None - # Memoised result of `_offload_max_pending_saves`, a process constant likewise. _MAX_PENDING_OFFLOAD: int | None = None @@ -96,44 +84,6 @@ def _offload_max_pending_saves() -> int: return _MAX_PENDING_OFFLOAD -def _offload_save_abandon_timeout_s() -> float: - """Seconds a deferred offload save may sit before the engine reclaims it. - - Blocks are freed on `finished_saving`, so a lost report leaves them deferred - forever: `has_pending_kv_work()` never clears and the engine busy-loops with - every GPU idle. - - Reclaiming cannot race a live copy. `OffloadWorkerMixin._guard` reports on - both the success and the exception path, so a report is lost only when - `store()` neither returns nor raises -- it is parked inside LMCache. Then - either the parked save is not copying (LMCache force-unpinned its source - after `pin_timeout_sec`) or a save queued behind it never reached `store()`. - Both cases are safe once that window has passed. - - Derived from LMCache's own `LMCACHE_EC_PIN_TIMEOUT_SEC` rather than a knob - of its own, because that ordering IS the safety argument -- two independent - env vars could be set the wrong way round with nothing to say so. - Non-positive disables reclamation. - """ - global _SAVE_ABANDON_TIMEOUT_S - if _SAVE_ABANDON_TIMEOUT_S is not None: - return _SAVE_ABANDON_TIMEOUT_S - pin = _LMCACHE_PIN_TIMEOUT_DEFAULT_S - raw = os.environ.get("LMCACHE_EC_PIN_TIMEOUT_SEC") - if raw is not None: - try: - pin = float(raw) - except ValueError: - logger.warning( - "invalid LMCACHE_EC_PIN_TIMEOUT_SEC=%r; assuming LMCache's %.0fs " - "default for the offload save abandon window", - raw, - _LMCACHE_PIN_TIMEOUT_DEFAULT_S, - ) - _SAVE_ABANDON_TIMEOUT_S = pin + _SAVE_ABANDON_MARGIN_S if pin > 0 else 0.0 - return _SAVE_ABANDON_TIMEOUT_S - - def _prompt_tokens_of(result) -> int: """Prompt tokens in whatever a `_schedule()` returned. @@ -949,13 +899,26 @@ def _maybe_release_deferred(self, seq: Sequence) -> None: self.deferred_free_blocks.pop(seq.id, None) self.block_manager.deallocate(seq) + def _save_abandon_timeout_s(self) -> float: + """Seconds a deferred save may sit before reclamation, or 0 to disable. + + Sourced from the offload connector (`save_abandon_timeout_s`), not + re-derived here: the window is a function of LMCache's own pin timeout, + which is the connector's knowledge, and keeping the derivation in one + place is what stops the safety ordering (reclaim strictly after LMCache + force-unpins) from silently drifting. Returns 0 when no offload connector + is attached, which disables reclamation -- there is nothing to reclaim. + """ + callback = getattr(self.kv_connector, "save_abandon_timeout_s", None) + return callback() if callable(callback) else 0.0 + def _reconcile_stalled_deferred_saves(self) -> int: """Reclaim blocks whose offload save has stalled past the abandon timeout. `_maybe_release_deferred` cannot do this: the connector still reports the save as pending (`should_defer_free` stays True), which is exactly why the blocks were never released. Once the save has been deferred longer - than `_offload_save_abandon_timeout_s()` -- set above LMCache's pin + than `_save_abandon_timeout_s()` -- set above LMCache's pin timeout, so upstream has force-unpinned and released the source bytes -- the blocks are safe to return to the pool. Without this, a single never-reported save keeps `has_pending_kv_work()` True forever and the @@ -977,7 +940,7 @@ def _reconcile_stalled_deferred_saves(self) -> int: this reclaims once the report is not coming. Between them every deferred save has a way out. """ - timeout = _offload_save_abandon_timeout_s() + timeout = self._save_abandon_timeout_s() if timeout <= 0 or not self.deferred_free_blocks: return 0 now = time.monotonic() @@ -3205,14 +3168,14 @@ def _update_from_kv_xfer_finished(self, kv_connector_output: KVConnectorOutput): # the pool forever. Recovery is total -- a leaked pin breaks no # `BlockPool` invariant -- so zeroing the count restores the record # exactly. - bm.reclaim_stale_state_store_pins(_offload_save_abandon_timeout_s()) + bm.reclaim_stale_state_store_pins(self._save_abandon_timeout_s()) # Load-side twin of the same defence: `deallocate` parks a slot when # it tears down a request whose state load is still in flight, on the # promise that `settle_state_load` will hand it back. A crashed # worker or dropped completion breaks that promise and the slot sits # off the free list forever, wedging `can_allocate`'s state gate. # Same window, same "cannot tell lost from slow" caveat as the pins. - bm.reconcile_orphan_load_slots(_offload_save_abandon_timeout_s()) + bm.reconcile_orphan_load_slots(self._save_abandon_timeout_s()) def get_request_counts(self) -> tuple[int, int]: """Returns (num_running_reqs, num_waiting_reqs).""" diff --git a/tests/test_lmcache_offload_connector.py b/tests/test_lmcache_offload_connector.py index f4016b7af0..6c420cee15 100644 --- a/tests/test_lmcache_offload_connector.py +++ b/tests/test_lmcache_offload_connector.py @@ -5235,6 +5235,7 @@ def test_every_member_the_scheduler_reads_is_reachable_through_the_shell(): must_be_seen = { "should_defer_free", "abandon_save", + "save_abandon_timeout_s", "request_finished", "build_connector_meta", "process_completions", diff --git a/tests/test_scheduler.py b/tests/test_scheduler.py index 89e2f7dc88..330e3056d2 100644 --- a/tests/test_scheduler.py +++ b/tests/test_scheduler.py @@ -1894,17 +1894,19 @@ class TestStalledOffloadSaveReclaim: """ @staticmethod - def _sched(monkeypatch, deferred): + def _sched(monkeypatch, deferred, connector=None): import atom.model_engine.scheduler as sched_mod - # The window is memoised as a process constant; tests choose their own. - monkeypatch.setattr(sched_mod, "_SAVE_ABANDON_TIMEOUT_S", 100.0) s = object.__new__(sched_mod.Scheduler) s.deferred_free_blocks = {seq.id: seq for seq in deferred} s._abandoned_saves = 0 s._next_save_reconcile_at = 0.0 - # Production always sets this (possibly None); reclaim now notifies it. - s.kv_connector = None + # The window is now sourced from the connector, not a scheduler constant: + # the scheduler asks `kv_connector.save_abandon_timeout_s()` for it (the + # value is LMCache knowledge). Tests choose their own via the stub. + if connector is None: + connector = SimpleNamespace(save_abandon_timeout_s=lambda: 100.0) + s.kv_connector = connector freed: list[int] = [] s.block_manager = SimpleNamespace(deallocate=lambda q: freed.append(q.id)) return s, freed @@ -1930,9 +1932,12 @@ def test_reclaim_notifies_the_connector_to_drop_the_save(self, monkeypatch): now = _time.monotonic() stale = SimpleNamespace(id=1, _deferred_save_at=now - 500.0) - s, freed = self._sched(monkeypatch, [stale]) abandoned: list = [] - s.kv_connector = SimpleNamespace(abandon_save=lambda sid: abandoned.append(sid)) + connector = SimpleNamespace( + save_abandon_timeout_s=lambda: 100.0, + abandon_save=lambda sid: abandoned.append(sid), + ) + s, freed = self._sched(monkeypatch, [stale], connector=connector) assert s._reconcile_stalled_deferred_saves() == 1 assert freed == [1] @@ -1958,13 +1963,14 @@ def test_a_non_positive_window_restores_wait_forever(self, monkeypatch): import atom.model_engine.scheduler as sched_mod - monkeypatch.setattr(sched_mod, "_SAVE_ABANDON_TIMEOUT_S", 0.0) s = object.__new__(sched_mod.Scheduler) s.deferred_free_blocks = { 1: SimpleNamespace(id=1, _deferred_save_at=_time.monotonic() - 1e6) } s._abandoned_saves = 0 s._next_save_reconcile_at = 0.0 + # A connector reporting a non-positive window disables reclamation. + s.kv_connector = SimpleNamespace(save_abandon_timeout_s=lambda: 0.0) s.block_manager = SimpleNamespace(deallocate=lambda q: pytest.fail("reclaimed")) assert s._reconcile_stalled_deferred_saves() == 0 @@ -1973,17 +1979,34 @@ def test_the_window_sits_above_lmcaches_own_pin_timeout(self, monkeypatch): """Deriving it from LMCache's knob IS the safety argument. Two independent env vars would let ours be set below the timeout it has - to exceed, and nothing would say so. + to exceed, and nothing would say so. The derivation now lives on the + offload connector (`offload_save_abandon_timeout_s`), since the pin + timeout is LMCache knowledge; the scheduler only asks for the result. """ - import atom.model_engine.scheduler as sched_mod + import atom.kv_transfer.offload._offload_common as offload_common - monkeypatch.setattr(sched_mod, "_SAVE_ABANDON_TIMEOUT_S", None) + monkeypatch.setattr(offload_common, "_save_abandon_timeout_s", None) monkeypatch.setenv("LMCACHE_EC_PIN_TIMEOUT_SEC", "900") - assert sched_mod._offload_save_abandon_timeout_s() == 930.0 + assert offload_common.offload_save_abandon_timeout_s() == 930.0 - monkeypatch.setattr(sched_mod, "_SAVE_ABANDON_TIMEOUT_S", None) + monkeypatch.setattr(offload_common, "_save_abandon_timeout_s", None) monkeypatch.delenv("LMCACHE_EC_PIN_TIMEOUT_SEC", raising=False) - assert sched_mod._offload_save_abandon_timeout_s() == 330.0 + assert offload_common.offload_save_abandon_timeout_s() == 330.0 + + def test_no_offload_connector_disables_reclamation(self): + """`_save_abandon_timeout_s` returns 0 when nothing offloads. + + A non-offload connector (or none at all) has no save to reclaim, so the + scheduler must read a non-positive window and skip the scan rather than + raise reaching for a method that is not there. + """ + import atom.model_engine.scheduler as sched_mod + + s = object.__new__(sched_mod.Scheduler) + s.kv_connector = None + assert s._save_abandon_timeout_s() == 0.0 + s.kv_connector = SimpleNamespace() # a connector without the offload face + assert s._save_abandon_timeout_s() == 0.0 class TestTheTierSplitPartitionsServedReuse: From ffbe86f31db3dc6aa62e28b5fd678de2336cc178 Mon Sep 17 00:00:00 2001 From: zejchen Date: Tue, 1 Sep 2026 10:53:51 +0000 Subject: [PATCH 094/151] =?UTF-8?q?[offload]=20=C2=A72b:=20expose=20the=20?= =?UTF-8?q?save=20bound=20publicly,=20stop=20reaching=20through=20=5Fimpl?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `Scheduler._state_store_pending_cap` shares the KV leg's running-plus-queued save bound with the state leg so both pin the same slice of the pool. It read that bound by reaching into the connector's private `_max_pending_saves`, and -- because `self.kv_connector` is the delegating shell, not the impl -- it had to reach one level further into `shell._impl._max_pending_saves`, then a third time into the multi state-tier sub's `_impl`. Three private reaches across two layering boundaries for one number. Expose the number instead: * `OffloadSchedulerMixin.max_pending_saves` (property) is the public read of the per-connector `_max_pending_saves` that `_offload_common.max_pending_ saves(kvc, save_workers)` computes. It returns None when the connector does not bound its save queue (`_may_emit_save` always True, as on dense), which is the scheduler's signal to fall back to the env reader. * The shell (`LMCacheOffloadConnectorScheduler`) forwards it as a property, mirroring `chunk_size`, so the scheduler reads it straight off `self.kv_connector`. * `_state_store_pending_cap` now reads `getattr(conn, "max_pending_saves", None)` and, under `kv_connector: multi`, `getattr(sub, "max_pending_saves", None)` on the state-tier sub -- no `_impl` in sight. This also fixes §7's multi cap: the composite bounds nothing of its own, so before this the state leg silently fell to the env default of 2 while the KV leg kept the sub's real bound; now both read the sub's bound. Behaviour is unchanged for every existing configuration -- the value returned is identical, only the access path is now public. Existing tests that set `connector._max_pending_saves` still drive the property (it reads that attribute). Adds `TestStateStorePendingCap` covering the direct-connector, dense-None-fallback, and multi-sub paths. Co-Authored-By: Claude Opus 4.8 --- atom/kv_transfer/offload/_offload_common.py | 17 ++++++++++ atom/kv_transfer/offload/connector.py | 11 ++++++ atom/model_engine/scheduler.py | 28 ++++++---------- tests/test_scheduler.py | 37 +++++++++++++++++++++ 4 files changed, 76 insertions(+), 17 deletions(-) diff --git a/atom/kv_transfer/offload/_offload_common.py b/atom/kv_transfer/offload/_offload_common.py index 74eb3ce95e..002f347a52 100644 --- a/atom/kv_transfer/offload/_offload_common.py +++ b/atom/kv_transfer/offload/_offload_common.py @@ -427,6 +427,23 @@ def save_abandon_timeout_s(self) -> float: return offload_save_abandon_timeout_s() + @property + def max_pending_saves(self) -> int | None: + """Running-plus-queued save bound this connector enforces, else None. + + The public read of the per-connector `_max_pending_saves` that + `max_pending_saves(kvc, save_workers)` computes from + `kv_connector_extra_config` and `OFFLOAD_COPY_WORKERS`. The state leg + (`Scheduler._state_store_pending_cap`) shares this exact number with the + KV leg's `_may_emit_save` so both legs pin the same slice of the pool, + and honours a per-connector `"max_pending_saves"` override the env reader + never sees. None when the connector does not bound its save queue + (`_may_emit_save` always True, as on dense) -- the scheduler then falls + back to the env reader. Exposed so the scheduler never reaches through + the delegating shell's `_impl` for it. + """ + return getattr(self, "_max_pending_saves", None) + def _chunk_floor(self, tokens: int) -> int: chunk = int(self.chunk_size or 256) return (max(0, int(tokens)) // chunk) * chunk diff --git a/atom/kv_transfer/offload/connector.py b/atom/kv_transfer/offload/connector.py index 27f9ec700c..7d39f06aa7 100644 --- a/atom/kv_transfer/offload/connector.py +++ b/atom/kv_transfer/offload/connector.py @@ -259,6 +259,17 @@ def save_abandon_timeout_s(self) -> float: # the reclaim window from the connector because it is LMCache knowledge. return self._impl.save_abandon_timeout_s() + @property + def max_pending_saves(self) -> int | None: + """The state leg's running-plus-queued save bound, read off the impl. + + `Scheduler._state_store_pending_cap` reads this off `self.kv_connector` + -- which is this shell -- so it must appear here rather than only on the + `_impl`, or the scheduler falls back to the bare env default of 2 and + the two legs pin different amounts of the same pool. Mirrors `chunk_size`. + """ + return getattr(self._impl, "max_pending_saves", None) + def get_statistics(self) -> dict[str, int]: return self._impl.get_statistics() diff --git a/atom/model_engine/scheduler.py b/atom/model_engine/scheduler.py index c5d6884dc8..15566e381e 100644 --- a/atom/model_engine/scheduler.py +++ b/atom/model_engine/scheduler.py @@ -2127,7 +2127,7 @@ def _settle_state_load(self, req_id, ok: bool) -> None: def _state_store_pending_cap(self) -> int: """The running-plus-queued cap for state-tier stores. - Read off the connector's own `_max_pending_saves` -- the canonical + Read off the connector's public `max_pending_saves` -- the canonical `_offload_common.max_pending_saves` value it computed from `kv_connector_extra_config` and `OFFLOAD_COPY_WORKERS`, the exact bound the KV leg's `_may_emit_save` enforces. Sharing that number rather than @@ -2135,28 +2135,22 @@ def _state_store_pending_cap(self) -> int: two legs from pinning different amounts of the same pool, and honours a per-connector `"max_pending_saves"` override the env reader never sees. Falls back to the env reader for a connector (or test double) that never - computed one, and reaches the value through the delegating shell's - `_impl` and, under `kv_connector: multi`, through the state-tier sub -- - `MultiConnectorScheduler` exposes neither `_max_pending_saves` nor - `_impl`, so without the sub probe the composite silently caps the state - leg at the env default of 2 while the KV leg keeps the sub's real bound. + bounded its save queue (`max_pending_saves` is None, as on dense) and, + under `kv_connector: multi`, reaches the bound through the state-tier sub + -- `MultiConnectorScheduler` bounds nothing of its own, so without the + sub probe the composite silently caps the state leg at the env default of + 2 while the KV leg keeps the sub's real bound. The public accessor means + neither read reaches through the delegating shell's `_impl` any more. """ - - def _cap_of(obj): - cap = getattr(obj, "_max_pending_saves", None) - if cap is None: - cap = getattr(getattr(obj, "_impl", None), "_max_pending_saves", None) - return cap - conn = self.kv_connector - cap = _cap_of(conn) + cap = getattr(conn, "max_pending_saves", None) if cap is None: # Under `multi` the bound lives on the state-tier sub (itself a # delegating shell), reached the same way the state face is routed. sub = getattr(conn, "_state_tier_sub", None) tier = sub() if callable(sub) else None if tier is not None: - cap = _cap_of(tier) + cap = getattr(tier, "max_pending_saves", None) if isinstance(cap, int) and not isinstance(cap, bool) and cap > 0: return cap return _offload_max_pending_saves() @@ -2165,8 +2159,8 @@ def _publish_state_stores(self) -> None: """Hand this pass's ready checkpoints to the connector for the CPU tier. Beside `_publish_state_loads` and on the same schedule, so one pass - makes one decision about the tier. The cap is the connector's own - `_max_pending_saves` (see `_state_store_pending_cap`) -- the same bound + makes one decision about the tier. The cap is the connector's public + `max_pending_saves` (see `_state_store_pending_cap`) -- the same bound the KV leg's `_may_emit_save` enforces, and for the same reason: each outstanding transfer holds the bytes it is reading out of the pool, so the queue depth is also how much of the pool a slow backend can pin. diff --git a/tests/test_scheduler.py b/tests/test_scheduler.py index 330e3056d2..0fe7756aa7 100644 --- a/tests/test_scheduler.py +++ b/tests/test_scheduler.py @@ -2009,6 +2009,43 @@ def test_no_offload_connector_disables_reclamation(self): assert s._save_abandon_timeout_s() == 0.0 +class TestStateStorePendingCap: + """`_state_store_pending_cap`: the state leg reads the KV leg's save bound. + + It must share the connector's real `max_pending_saves` so both legs pin the + same slice of the pool -- and read it off the *public* accessor, never by + reaching through the delegating shell's `_impl` (review finding §2b). + """ + + @staticmethod + def _sched(connector): + import atom.model_engine.scheduler as sched_mod + + s = object.__new__(sched_mod.Scheduler) + s.kv_connector = connector + return s + + def test_reads_the_public_bound_off_the_connector(self): + s = self._sched(SimpleNamespace(max_pending_saves=5)) + assert s._state_store_pending_cap() == 5 + + def test_falls_back_to_env_when_the_connector_does_not_bound(self, monkeypatch): + """dense reports None (its save queue is unbounded); use the env reader.""" + import atom.model_engine.scheduler as sched_mod + + monkeypatch.setattr(sched_mod, "_MAX_PENDING_OFFLOAD", None) + monkeypatch.setenv("OFFLOAD_MAX_PENDING_SAVES", "3") + s = self._sched(SimpleNamespace(max_pending_saves=None)) + assert s._state_store_pending_cap() == 3 + + def test_under_multi_reaches_the_bound_on_the_state_tier_sub(self): + """The composite bounds nothing of its own; the sub carries the bound.""" + sub = SimpleNamespace(max_pending_saves=7) + conn = SimpleNamespace(max_pending_saves=None, _state_tier_sub=lambda: sub) + s = self._sched(conn) + assert s._state_store_pending_cap() == 7 + + class TestTheTierSplitPartitionsServedReuse: """`[Cache Tiers]` exists to answer "what does the CPU tier buy", so its two halves have to be two halves of one thing. From 3a38ae8091c6645a0a71de32a56ee1a50d6382c9 Mon Sep 17 00:00:00 2001 From: zejchen Date: Tue, 1 Sep 2026 10:55:00 +0000 Subject: [PATCH 095/151] =?UTF-8?q?[offload]=20=C2=A72d:=20declare=20state?= =?UTF-8?q?=5Fbackend=20on=20KVTransferTensors?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `ModelRunner` publishes the attention metadata builder to the connector by assigning `transfer_tensors.state_backend = self.attn_metadata_builder`, and the kimi_k3 state tier reads it back (`state_runtime.checkpoint_spec.layout_id`) to fold the state geometry into every offload key. But `state_backend` was never a field of `KVTransferTensors` -- it was a loose runtime attribute set in one file and read in another, invisible in the dataclass that is supposed to be the transfer-region contract. A reader of the connector sees `getattr( transfer_tensors, "state_backend", None)` with no declaration to point to, and nothing records that the field is part of the contract. Declare it: `state_backend: object | None = None`. Typed `object` because `disaggregation.types` must not import the model engine, and defaulting to None because every layout without a state tier leaves it unset. Appended after the existing fields, so positional construction is unaffected. No behaviour change -- the assignment and read are unchanged; the field they already used is now part of the declared contract instead of an undocumented attribute. Co-Authored-By: Claude Opus 4.8 --- atom/kv_transfer/disaggregation/types.py | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/atom/kv_transfer/disaggregation/types.py b/atom/kv_transfer/disaggregation/types.py index e0ea297268..5454cfb481 100644 --- a/atom/kv_transfer/disaggregation/types.py +++ b/atom/kv_transfer/disaggregation/types.py @@ -176,6 +176,15 @@ class KVTransferTensors: scatter_slot: Callable[[int, int], None] | None = None # Appended for positional compatibility with existing generic descriptors. expected_full_slot_region_count: int | None = None + # The attention metadata builder, published by `ModelRunner` after the tier + # is built inside `register_kv_caches` -- the one place builder and connector + # are both in scope. The kimi_k3 state tier reads its `state_runtime. + # checkpoint_spec.layout_id` to fold the state geometry into every key. Typed + # `object` because `types` must not import the model engine; None on every + # layout that has no state tier to name. Declared here rather than set as a + # loose runtime attribute so the field the connector reads is part of the + # contract, not an undocumented assignment two layers away. + state_backend: object | None = None @dataclass From aa09dd16553f574aa88181bb2d4337c20aeee5c7 Mon Sep 17 00:00:00 2001 From: zejchen Date: Tue, 1 Sep 2026 11:31:56 +0000 Subject: [PATCH 096/151] =?UTF-8?q?[offload]=20=C2=A72c:=20group=20the=20s?= =?UTF-8?q?even=20joint-transfer=20fields=20into=20OffloadJointRecord?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The KV-transfer offload/joint-load protocol kept seven flat attributes on Sequence -- state_load_hash, offload_kv_prefix_tokens, offload_kv_chunk_tokens, state_joint_boundary_tokens, state_joint_boundary_hash, state_joint_kv_tokens, state_joint_claim_tokens -- written from BlockManager and Scheduler and read back by the offload connector. The four "joint" fields (boundary_tokens, boundary_hash, kv_tokens, claim_tokens) are one logical unit: a boundary is meaningless without the KV span aimed at it and the prefix claimed below it, yet each site that abandoned a joint load re-zeroed its own hand-picked subset. A reset that forgot a field left a stale span the connector would then act on. This is the mechanical, behavior-preserving consolidation from the review. It groups the seven into a single mutable dataclass, OffloadJointRecord, held as Sequence.offload_joint, and adds reset_joint() to drop the four joint fields together. No reset semantics change: every site reproduces its exact prior subset. The two full-boundary drops (block_manager deallocate, scheduler JOINT-DISOWN alignment guard) call reset_joint(); the four partial resets (block_manager clamp + disown, scheduler load-fail + load-complete) stay explicit and each carries a "partial reset by design" comment so a later reader does not "helpfully" fold them into reset_joint() and change behavior. The field is set unconditionally in Sequence.__init__, so it is always present on a real Sequence; the getattr(seq, "state_joint_*", 0) reads in the kimi_k3 connector become direct attribute access. Comment references in offload/connector and multi_connector are updated to the new field path (name-matches-function). Test doubles that stood in for a Sequence gain an offload_joint= OffloadJointRecord() so the direct reads resolve. state_fork_src is deliberately NOT folded in -- it is the paged-state checkpoint fork slot, a different mechanism, not part of this protocol. Not addressed here (flagged to the maintainer as a separate correctness question): whether the four partial resets *should* be full resets. This commit preserves the current asymmetry exactly rather than deciding it. Tests: full offload/scheduler/block_manager/state_checkpoint/page_unit/multi suites green (679 passed); the one remaining failure, TestJointChunkProbeIsGated, is pre-existing and environmental (k3f ships lmcache 0.4.5, contradicting the test's "not installed" premise) and touches none of the renamed fields. Co-Authored-By: Claude Opus 4.8 --- .../disaggregation/multi/multi_connector.py | 2 +- atom/kv_transfer/offload/connector.py | 2 +- .../offload/hybrid/kimi_k3/connector.py | 8 +- atom/model_engine/block_manager.py | 71 +++++++------- atom/model_engine/scheduler.py | 44 +++++---- atom/model_engine/sequence.py | 93 +++++++++++++------ tests/test_block_manager.py | 6 +- tests/test_lmcache_offload_connector.py | 13 ++- tests/test_state_checkpoint.py | 32 +++---- 9 files changed, 155 insertions(+), 116 deletions(-) diff --git a/atom/kv_transfer/disaggregation/multi/multi_connector.py b/atom/kv_transfer/disaggregation/multi/multi_connector.py index f8deabacd9..8395ae682e 100644 --- a/atom/kv_transfer/disaggregation/multi/multi_connector.py +++ b/atom/kv_transfer/disaggregation/multi/multi_connector.py @@ -516,7 +516,7 @@ def chunk_size(self) -> int | None: """The offload sub's KV transfer grid, in tokens. `Scheduler._resolve_waiting_remote_kv` stamps - `seq.offload_kv_chunk_tokens` from `getattr(self.kv_connector, + `seq.offload_joint.kv_chunk_tokens` from `getattr(self.kv_connector, "chunk_size", 0)`, and under `multi` `self.kv_connector` IS this composite. Without this forwarder the getattr default (0) wins, so `_joint_kv_boundary` sees a zero chunk grid and refuses every joint KV diff --git a/atom/kv_transfer/offload/connector.py b/atom/kv_transfer/offload/connector.py index 7d39f06aa7..96239be2e7 100644 --- a/atom/kv_transfer/offload/connector.py +++ b/atom/kv_transfer/offload/connector.py @@ -158,7 +158,7 @@ def chunk_size(self) -> int | None: """The KV leg's transfer grid, in tokens. Read by `Scheduler._resolve_waiting_remote_kv` to stamp - `seq.offload_kv_chunk_tokens`, which is `_joint_kv_boundary`'s fallback + `seq.offload_joint.kv_chunk_tokens`, which is `_joint_kv_boundary`'s fallback when `BlockManager` could not read the LMCache config at startup. Without this forwarder that fallback was dead -- `getattr` returned its 0 default and the joint path refused with `no_chunk_size`. diff --git a/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py b/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py index 96bd4bd457..75665baed4 100644 --- a/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py +++ b/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py @@ -616,7 +616,7 @@ def _decide_load_after_alloc(self, seq, ls): hbm = int(seq.num_cached_tokens) lmc = int(ls.lmcache_cached_tokens) chunk = self.chunk_size or 256 - joint = int(getattr(seq, "state_joint_boundary_tokens", 0) or 0) + joint = int(seq.offload_joint.boundary_tokens or 0) if joint <= hbm: return False, "per_req_cache_state_boundary", hbm, lmc, lmc - hbm, chunk # Where the transfer starts -- NOT where the request may call itself @@ -625,13 +625,13 @@ def _decide_load_after_alloc(self, seq, ls): # resend it would land a second copy in HBM (`publish_loaded_prefix` # keeps the canonical mapping, fresh blocks stay private). Floored to the # chunk grid by `_joint_kv_boundary`, so aligned whenever `hbm` was. - start = max(hbm, int(getattr(seq, "state_joint_claim_tokens", 0) or 0)) + start = max(hbm, int(seq.offload_joint.claim_tokens or 0)) # The KV leg moves whole chunks and the blocks below `start` are shared, # so an unaligned start cannot be rounded down. if start % chunk != 0: return False, "joint_unaligned_hbm_prefill", start, lmc, lmc - start, chunk # Transfer the chunk covering the boundary, claim only the boundary. - kv_target = int(getattr(seq, "state_joint_kv_tokens", 0) or 0) or joint + kv_target = int(seq.offload_joint.kv_tokens or 0) or joint if joint > lmc or kv_target > lmc: return False, "joint_boundary_above_lookup", start, lmc, lmc - start, chunk if kv_target <= start: @@ -659,7 +659,7 @@ def _claim_after_load(self, seq, hbm: int, lmc: int) -> int: claiming the rounded-up figure would have the forward skip tokens the recurrent state does not cover. """ - joint = int(getattr(seq, "state_joint_boundary_tokens", 0) or 0) + joint = int(seq.offload_joint.boundary_tokens or 0) return max(hbm, min(joint, lmc)) if joint else max(hbm, lmc) # -- connector-owned channels ----------------------------------------- diff --git a/atom/model_engine/block_manager.py b/atom/model_engine/block_manager.py index a46cbaff83..1d085ee0c7 100644 --- a/atom/model_engine/block_manager.py +++ b/atom/model_engine/block_manager.py @@ -663,10 +663,7 @@ def _joint_kv_boundary( B: overshooting costs one chunk into blocks the forward is about to rewrite, undershooting would be silent wrong output. """ - seq.state_joint_boundary_tokens = 0 - seq.state_joint_boundary_hash = -1 - seq.state_joint_kv_tokens = 0 - seq.state_joint_claim_tokens = 0 + seq.offload_joint.reset_joint() if self.state_offload is None: return self._no_joint("off", record) # PAGE is now the served class, not the excluded one. This used to @@ -684,9 +681,9 @@ def _joint_kv_boundary( # holds whatever `get_kvconnector` returned, and one without # `chunk_size` would zero this and disable the feature silently. chunk = self._joint_chunk_tokens or int( - getattr(seq, "offload_kv_chunk_tokens", 0) or 0 + seq.offload_joint.kv_chunk_tokens or 0 ) - lmc_tokens = int(getattr(seq, "offload_kv_prefix_tokens", 0) or 0) + lmc_tokens = int(seq.offload_joint.kv_prefix_tokens or 0) if chunk <= 0: return self._no_joint("no_chunk_size", record) # Floored to the grid everything below compares against, so the @@ -716,9 +713,9 @@ def _joint_kv_boundary( kv_tokens = -(-tokens // chunk) * chunk if kv_tokens > lmc_tokens: return self._no_joint("covering_chunk_beyond_lookup", record) - seq.state_joint_boundary_tokens = tokens - seq.state_joint_boundary_hash = h - seq.state_joint_kv_tokens = kv_tokens + seq.offload_joint.boundary_tokens = tokens + seq.offload_joint.boundary_hash = h + seq.offload_joint.kv_tokens = kv_tokens # Everything up to the compressed hit is this prompt's KV and is still # in the pool -- what `_gated_hit` cut was resumability, not residency. # Below the joint boundary those blocks are read-only for this request @@ -734,9 +731,9 @@ def _joint_kv_boundary( # Floored to the chunk grid so the KV leg starts aligned and writes # only into blocks below it that nobody else can be reading. claim_tokens = ((claim_blocks * hbs) // chunk) * chunk - seq.state_joint_claim_tokens = max(claim_tokens, hbm_boundary * hbs) + seq.offload_joint.claim_tokens = max(claim_tokens, hbm_boundary * hbs) # `allocate` claims the boundary in whole hash blocks -- - # `state_joint_claim_tokens // hbs` -- but the value above is floored to + # `claim_tokens // hbs` -- but the value above is floored to # the *chunk* grid, not the hash-block grid. When `hbs` does not divide # `chunk`, a chunk-aligned claim is not hash-block-aligned: the floor # drops the `[floor(claim_tokens/hbs)*hbs, claim_tokens)` tail into a @@ -746,11 +743,8 @@ def _joint_kv_boundary( # shipping case (`block_size * dcp_world_size == LMCACHE_CHUNK_SIZE`), so # it only surfaces at DCP world sizes that take `hbs` off the chunk grid. # Refuse the joint and recompute the tail rather than resume over a gap. - if seq.state_joint_claim_tokens % hbs != 0: - seq.state_joint_boundary_tokens = 0 - seq.state_joint_boundary_hash = -1 - seq.state_joint_kv_tokens = 0 - seq.state_joint_claim_tokens = 0 + if seq.offload_joint.claim_tokens % hbs != 0: + seq.offload_joint.reset_joint() return self._no_joint("claim_off_hash_grid", record) if record: self.joint_boundaries += 1 @@ -930,7 +924,7 @@ def allocate(self, seq: Sequence, num_cached_blocks: int = 0) -> bool: hbs = self._hash_block_size() claim_blocks = max( num_cached_blocks, - int(getattr(seq, "state_joint_claim_tokens", 0) or 0) // hbs, + int(seq.offload_joint.claim_tokens or 0) // hbs, ) h = -1 hit_hash = -1 @@ -947,22 +941,21 @@ def allocate(self, seq: Sequence, num_cached_blocks: int = 0) -> bool: # the old fall-through then recorded `num_cached_blocks * hbs` as # cached over a prefix it could not claim, left `hit_hash` # unassigned (cold-start state exit), and clamped only - # `state_joint_claim_tokens` while `boundary`/`kv` stayed above + # `claim_tokens` while `boundary`/`kv` stayed above # the claimed region -- silent wrong output on both counts. Use # real control flow instead: never treat more than the `i` blocks # actually claimed as cached, and drop any joint boundary that # sits above them so the KV leg cannot resume over a gap. claimed_tokens = i * hbs num_cached_blocks = min(num_cached_blocks, i) - if ( - int(getattr(seq, "state_joint_boundary_tokens", 0) or 0) - > claimed_tokens - ): - seq.state_joint_boundary_tokens = 0 - seq.state_joint_boundary_hash = -1 - seq.state_joint_kv_tokens = 0 - seq.state_joint_claim_tokens = min( - int(getattr(seq, "state_joint_claim_tokens", 0) or 0), + # Partial reset by design: the claim is clamped, not zeroed, + # so this cannot go through `reset_joint`. + if int(seq.offload_joint.boundary_tokens or 0) > claimed_tokens: + seq.offload_joint.boundary_tokens = 0 + seq.offload_joint.boundary_hash = -1 + seq.offload_joint.kv_tokens = 0 + seq.offload_joint.claim_tokens = min( + int(seq.offload_joint.claim_tokens or 0), claimed_tokens, ) break @@ -993,7 +986,7 @@ def allocate(self, seq: Sequence, num_cached_blocks: int = 0) -> bool: # joint boundary and `_claim_after_load` raised `num_cached_tokens` # to it, so the forward resumed over a prefix the state does not # cover. Silent wrong output, no exception. - joint_hash = seq.state_joint_boundary_hash + joint_hash = seq.offload_joint.boundary_hash state_holds = self._attach_state_slots( seq, joint_hash if joint_hash != -1 else hit_hash ) @@ -1012,7 +1005,7 @@ def allocate(self, seq: Sequence, num_cached_blocks: int = 0) -> bool: # above the HBM hit by construction. `num_cached_tokens` stays at # the HBM prefix until the KV leg lands, so the forward covers # `[hbm, num_prompt)` if anything goes wrong from here. - joint_hash = seq.state_joint_boundary_hash + joint_hash = seq.offload_joint.boundary_hash state_holds = self._attach_state_slots( seq, joint_hash if joint_hash != -1 else hit_hash ) @@ -1030,7 +1023,7 @@ def allocate(self, seq: Sequence, num_cached_blocks: int = 0) -> bool: if ( state_holds and seq.has_per_req_cache - and seq.state_joint_boundary_hash != -1 + and seq.offload_joint.boundary_hash != -1 and not self._state_leg_secured(seq) ): self.state_gate_lost_boundary += 1 @@ -1053,8 +1046,10 @@ def allocate(self, seq: Sequence, num_cached_blocks: int = 0) -> bool: # deallocate and requeue for a clean recompute next pass. return False seq.num_cached_tokens = 0 - seq.state_joint_boundary_tokens = 0 - seq.state_joint_boundary_hash = -1 + # Clears only the boundary here, as the pre-record code did -- a + # partial reset, not a full `reset_joint`. + seq.offload_joint.boundary_tokens = 0 + seq.offload_joint.boundary_hash = -1 return True @staticmethod @@ -1107,7 +1102,7 @@ def _state_leg_secured(self, seq: Sequence) -> bool: returns True as well, which is correct with no boundary and wrong with one. """ - if getattr(seq, "state_load_hash", -1) != -1: + if seq.offload_joint.load_hash != -1: return True # a CPU load is in flight for it if getattr(seq, "state_fork_src", -1) != -1: return True # a fork checkpoint is its source @@ -1254,7 +1249,7 @@ def _request_state_load(self, seq: Sequence, hit_hash: int) -> bool: return False if not self.state_offload.request_load(seq.id, hit_hash): return False - seq.state_load_hash = hit_hash + seq.offload_joint.load_hash = hit_hash self._state_loads.append((seq.id, hit_hash, seq.state_slot)) return True @@ -1270,12 +1265,12 @@ def cancel_state_load(self, seq: Sequence) -> bool: caller must abandon this admission rather than resume over the shared prefix. ``True`` when there was nothing to cancel or the disown held. """ - if seq.state_load_hash == -1: + if seq.offload_joint.load_hash == -1: return True self._state_loads = [e for e in self._state_loads if e[0] != seq.id] if self.state_offload is not None: self.state_offload.abandon_load(seq.id) - seq.state_load_hash = -1 + seq.offload_joint.load_hash = -1 disowned = self.disown_claimed_prefix(seq) seq.num_cached_tokens = 0 return disowned @@ -2436,7 +2431,7 @@ def deallocate(self, seq: Sequence): # already true over it. Held until `settle_state_load`, which always # comes: the worker reports every load either way. The rollback # scratch is nobody's destination and goes back regardless. - if seq.state_load_hash != -1 and self.state_offload is not None: + if seq.offload_joint.load_hash != -1 and self.state_offload is not None: self._orphan_load_slots[seq.id] = seq.state_slot self._orphan_load_slots_at[seq.id] = monotonic() # Abandoned, not failed: an abort says nothing about the bytes, @@ -2451,7 +2446,7 @@ def deallocate(self, seq: Sequence): seq.state_slots = [] seq.state_fork_src = -1 - seq.state_load_hash = -1 + seq.offload_joint.load_hash = -1 def can_append(self, seq: Sequence, num_new_tokens: int = 1) -> bool: seq_len = len(seq) diff --git a/atom/model_engine/scheduler.py b/atom/model_engine/scheduler.py index 15566e381e..a2d1a51be4 100644 --- a/atom/model_engine/scheduler.py +++ b/atom/model_engine/scheduler.py @@ -1369,7 +1369,8 @@ def _schedule(self) -> tuple[ScheduledBatch, dict[int, Sequence]] | None: ) if needs_remote_load: - if seq.state_load_hash != -1 and not seq.state_joint_boundary_tokens: + oj = seq.offload_joint + if oj.load_hash != -1 and not oj.boundary_tokens: # Two transfers but one report: the first completion would # unpark the request while the other is still writing. Only # reachable when the legs were decided separately -- a joint @@ -1390,7 +1391,7 @@ def _schedule(self) -> tuple[ScheduledBatch, dict[int, Sequence]] | None: self._park_for_remote_load(seq, skipped_waiting_requests) continue - if seq.state_joint_boundary_tokens: + if seq.offload_joint.boundary_tokens: # The KV leg was refused after the state leg was aimed at the # joint boundary, so the state claims a prefix whose KV nobody # will fetch. Disown it; `_decide_load_after_alloc` logs why. @@ -1398,13 +1399,13 @@ def _schedule(self) -> tuple[ScheduledBatch, dict[int, Sequence]] | None: "[JOINT-DISOWN] seq %s: KV leg refused at boundary %d; " "recomputing from %d", seq.id, - seq.state_joint_boundary_tokens, + seq.offload_joint.boundary_tokens, seq.num_cached_tokens, ) # Privatise the claimed prefix (finding #2) so recompute-from-0 # cannot tear a shared decode. cancel_state_load does it when a # CPU state load was pending; otherwise disown directly. - if seq.state_load_hash != -1: + if seq.offload_joint.load_hash != -1: disowned = self.block_manager.cancel_state_load(seq) else: disowned = self.block_manager.disown_claimed_prefix(seq) @@ -1413,10 +1414,12 @@ def _schedule(self) -> tuple[ScheduledBatch, dict[int, Sequence]] | None: self.waiting.appendleft(seq) break seq.num_cached_tokens = 0 - seq.state_joint_boundary_tokens = 0 - seq.state_joint_boundary_hash = -1 + # Partial reset by design: the KV/claim spans are left as-is, + # matching the pre-record code -- not a full `reset_joint`. + seq.offload_joint.boundary_tokens = 0 + seq.offload_joint.boundary_hash = -1 - if seq.state_load_hash != -1: + if seq.offload_joint.load_hash != -1: # The kept state boundary is in LMCache, not HBM. It parks # exactly as a KV load does -- same queue, same backpressure, # same wake-up -- because to the scheduler both are one event: @@ -1769,9 +1772,7 @@ def _consume_failed_remote_kv(self, seq: Sequence) -> bool: seq.status = SequenceStatus.WAITING if not self._connector_flag("is_offload"): self._uncount_inflight_load(seq) - if getattr(seq, "state_load_hash", -1) != -1 or getattr( - seq, "state_joint_boundary_tokens", 0 - ): + if seq.offload_joint.load_hash != -1 or seq.offload_joint.boundary_tokens: # The state never arrived, so the boundary is not this request's # history. Disown it exactly as `BlockManager.allocate` does at # admission, otherwise the resume runs with `num_cached_tokens > 0` @@ -1783,12 +1784,13 @@ def _consume_failed_remote_kv(self, seq: Sequence) -> bool: # otherwise tear a shared decode. disowned = self.block_manager.disown_claimed_prefix(seq) seq.num_cached_tokens = 0 - seq.state_load_hash = -1 + seq.offload_joint.load_hash = -1 # A joint boundary that failed on either leg is not this request's # history any more than a state-only one is, and leaving it set - # would let the connector clamp a later lookup to it. - seq.state_joint_boundary_tokens = 0 - seq.state_joint_boundary_hash = -1 + # would let the connector clamp a later lookup to it. Partial reset + # by design: the KV/claim spans are left as-is, as before. + seq.offload_joint.boundary_tokens = 0 + seq.offload_joint.boundary_hash = -1 if not disowned: # Cannot back private copies; drop the whole table so the # re-admission rebuilds a clean one via can_allocate/allocate @@ -1874,9 +1876,11 @@ def _mark_offload_load_ready(self, seq: Sequence) -> None: # that is left is to stop calling the load pending. A joint load is the # one case where both halves fire, and `_JointPark` held the wake until # both legs reported. - seq.state_load_hash = -1 - seq.state_joint_boundary_tokens = 0 - seq.state_joint_boundary_hash = -1 + # Partial reset by design: the KV/claim spans are left as-is, as the + # pre-record code did -- not a full `reset_joint`. + seq.offload_joint.load_hash = -1 + seq.offload_joint.boundary_tokens = 0 + seq.offload_joint.boundary_hash = -1 def _is_offload_prefill_resume(self, seq: Sequence) -> bool: """True when offload already owns blocks and should resume suffix prefill. @@ -1905,8 +1909,10 @@ def _query_connector_prefill_match(self, seq: Sequence, *, skip: bool) -> bool: # next and a hybrid's two legs have to be aimed at one boundary, for # which this is the KV leg's ceiling. In tokens, and absolute -- the # connector reports what it found *beyond* what the seq already has. - seq.offload_kv_prefix_tokens = int(ext_tokens) + int(seq.num_cached_tokens) - seq.offload_kv_chunk_tokens = int( + seq.offload_joint.kv_prefix_tokens = int(ext_tokens) + int( + seq.num_cached_tokens + ) + seq.offload_joint.kv_chunk_tokens = int( getattr(self.kv_connector, "chunk_size", 0) or 0 ) return needs_remote_load diff --git a/atom/model_engine/sequence.py b/atom/model_engine/sequence.py index 6935dea97b..78f9d68c8a 100644 --- a/atom/model_engine/sequence.py +++ b/atom/model_engine/sequence.py @@ -3,6 +3,7 @@ import array from collections.abc import Callable +from dataclasses import dataclass from enum import Enum, auto from itertools import count from typing import Any @@ -67,6 +68,64 @@ def get_exit_sequence(): return exit_seq +@dataclass +class OffloadJointRecord: + """The KV-transfer offload/joint-load protocol state for one sequence. + + These seven values were seven flat attributes on `Sequence`, written from + `BlockManager` and `Scheduler` and read back by the offload connector, with + the joint subset re-initialised by hand in several places -- so a reset that + forgot one field left a stale span the connector would then act on. Grouped + here so the whole protocol state is one field (`Sequence.offload_joint`) and + the joint boundary has one place to be dropped from (`reset_joint`). + """ + + # Content hash of a checkpoint the tier was asked to fetch back into + # `state_slot`, or -1. Tells the scheduler to park, and tells the failure + # path that `num_cached_tokens` is claiming undelivered state. + load_hash: int = -1 + # The LMCache-resident KV prefix in tokens, as this admission's lookup + # reported it -- the KV leg's ceiling. Written before `can_allocate`, which + # is where the two legs agree on one boundary. 0 = no lookup. + kv_prefix_tokens: int = 0 + # The connector's LMCache chunk size, in tokens, or 0. The KV leg moves + # whole chunks, so a joint boundary has to be a multiple of it. + kv_chunk_tokens: int = 0 + # The boundary both legs of a joint load are aimed at, or 0. Chosen by + # `can_allocate`: the rightmost checkpoint rung the LMCache KV prefix + # covers. `num_cached_tokens` stays at the HBM prefix until both legs + # report, which keeps the forward honest if either fails. + boundary_tokens: int = 0 + # Content hash of that boundary's last block, for the state leg. + boundary_hash: int = -1 + # How far the KV leg transfers, in tokens: the LMCache chunk that *covers* + # the boundary above, which is at or past it. The request still only claims + # the boundary -- see `_claim_after_load`. + kv_tokens: int = 0 + # How far `allocate` claimed straight out of the HBM prefix cache, in + # tokens. Above `num_cached_tokens`, not below it: the blocks in + # `(hit, compressed_hit]` hold this prompt's KV and are still indexed -- + # the state gate cut the hit, the KV never went anywhere. Claiming them is + # what keeps the KV leg from paying LMCache to resend what the pool already + # has, and it is where that leg starts. + claim_tokens: int = 0 + + def reset_joint(self) -> None: + """Drop the joint boundary and both of its transfer spans. + + The four joint fields move together -- a boundary is meaningless + without the KV span aimed at it and the prefix claimed below it -- so + the sites that abandon a joint load in full clear them through here + rather than by hand. Leaves `load_hash`, `kv_prefix_tokens`, and + `kv_chunk_tokens` untouched: they are set on their own paths and are + not part of the boundary this drops. + """ + self.boundary_tokens = 0 + self.boundary_hash = -1 + self.kv_tokens = 0 + self.claim_tokens = 0 + + class Sequence: counter = count() @@ -207,36 +266,10 @@ def __init__( # forward. -1 = read and write the same slot, the case for every step in # between. Always a single slot: a checkpoint is one slot wide. self.state_fork_src = -1 - # Content hash of a checkpoint the tier was asked to fetch back into - # `state_slot`, or -1. Tells the scheduler to park, and tells - # the failure path that `num_cached_tokens` is claiming undelivered - # state. - self.state_load_hash = -1 - # The LMCache-resident KV prefix in tokens, as this admission's lookup - # reported it -- the KV leg's ceiling. Written before `can_allocate`, - # which is where the two legs agree on one boundary. 0 = no lookup. - self.offload_kv_prefix_tokens = 0 - # The connector's LMCache chunk size, in tokens, or 0. The KV leg moves - # whole chunks, so a joint boundary has to be a multiple of it. - self.offload_kv_chunk_tokens = 0 - # The boundary both legs of a joint load are aimed at, or 0. Chosen by - # `can_allocate`: the rightmost checkpoint rung the LMCache KV prefix - # covers. `num_cached_tokens` stays at the HBM prefix until both legs - # report, which keeps the forward honest if either fails. - self.state_joint_boundary_tokens = 0 - # Content hash of that boundary's last block, for the state leg. - self.state_joint_boundary_hash = -1 - # How far the KV leg transfers, in tokens: the LMCache chunk that - # *covers* the boundary above, which is at or past it. The request still - # only claims the boundary -- see `_claim_after_load`. - self.state_joint_kv_tokens = 0 - # How far `allocate` claimed straight out of the HBM prefix cache, in - # tokens. Above `num_cached_tokens`, not below it: the blocks in - # `(hit, compressed_hit]` hold this prompt's KV and are still indexed -- - # the state gate cut the hit, the KV never went anywhere. Claiming them - # is what keeps the KV leg from paying LMCache to resend what the pool - # already has, and it is where that leg starts. - self.state_joint_claim_tokens = 0 + # The KV-transfer offload/joint-load protocol state -- seven values that + # move together through admission and load. Grouped so a joint reset + # cannot forget a field; see `OffloadJointRecord`. + self.offload_joint = OffloadJointRecord() self.temperature = sampling_params.temperature self.top_k = sampling_params.top_k self.top_p = sampling_params.top_p diff --git a/tests/test_block_manager.py b/tests/test_block_manager.py index 9977ac4834..448cc048d7 100644 --- a/tests/test_block_manager.py +++ b/tests/test_block_manager.py @@ -99,7 +99,7 @@ def test_a_joint_boundary_widens_the_claim_without_widening_the_hit( bm, tokens, resident = self._resident_prefix(seq_factory) seq = seq_factory(tokens) # The state gate cut the hit to 2 blocks; the walk had matched 4. - seq.state_joint_claim_tokens = 16 + seq.offload_joint.claim_tokens = 16 bm.allocate(seq, 2) # All four resident blocks are reused -- that is the transfer the KV @@ -112,7 +112,7 @@ def test_a_joint_boundary_widens_the_claim_without_widening_the_hit( def test_the_widened_claim_still_gives_every_position_a_block(self, seq_factory): bm, tokens, _ = self._resident_prefix(seq_factory) seq = seq_factory(tokens) - seq.state_joint_claim_tokens = 16 + seq.offload_joint.claim_tokens = 16 bm.allocate(seq, 2) assert len(seq.block_table) == 5 assert len(set(seq.block_table)) == 5 @@ -142,7 +142,7 @@ def _shared_prefix(self, seq_factory, num_kvcache_blocks=32): # A second live seq claims the same 4 resident blocks via the joint # boundary -- now every prefix block is held at ref_count == 2. seq = seq_factory(tokens) - seq.state_joint_claim_tokens = 16 + seq.offload_joint.claim_tokens = 16 bm.allocate(seq, 2) shared = list(seq.block_table[:4]) assert all(bm.kv.block(b).ref_count == 2 for b in shared) diff --git a/tests/test_lmcache_offload_connector.py b/tests/test_lmcache_offload_connector.py index 6c420cee15..b5a47c2c8d 100644 --- a/tests/test_lmcache_offload_connector.py +++ b/tests/test_lmcache_offload_connector.py @@ -71,7 +71,7 @@ ) from atom.model_engine.block_manager import BlockManager from atom.model_engine.scheduler import Scheduler -from atom.model_engine.sequence import SequenceStatus +from atom.model_engine.sequence import OffloadJointRecord, SequenceStatus class _LookupClient: @@ -188,6 +188,7 @@ def _stateful_seq( state_slot=group, prefix_hashes_published=True, _state_initialized_after_alloc=True, + offload_joint=OffloadJointRecord(), ) @@ -2951,6 +2952,7 @@ def test_abort_cleans_load_whose_terminal_was_already_consumed( num_tokens=8, offload_loaded_tokens=4, offload_load_start_tokens=4, + offload_joint=OffloadJointRecord(), ) class _Connector(_OffloadMixinStub): @@ -3012,6 +3014,7 @@ def test_consume_failed_remote_kv_does_not_repeat_terminal_callback(): id=730, status=SequenceStatus.WAITING_FOR_REMOTE_KVS, num_cached_tokens=4, + offload_joint=OffloadJointRecord(), ) class _Connector: @@ -4725,9 +4728,11 @@ def _k3_seq(*, hbm: int, joint: int = 0, kv: int = 0, claimed: int = 0): id=1, has_per_req_cache=True, num_cached_tokens=hbm, - state_joint_boundary_tokens=joint, - state_joint_kv_tokens=kv, - state_joint_claim_tokens=claimed, + offload_joint=OffloadJointRecord( + boundary_tokens=joint, + kv_tokens=kv, + claim_tokens=claimed, + ), ) diff --git a/tests/test_state_checkpoint.py b/tests/test_state_checkpoint.py index db871d6813..6a7df1f285 100644 --- a/tests/test_state_checkpoint.py +++ b/tests/test_state_checkpoint.py @@ -1429,7 +1429,7 @@ def test_a_boundary_only_the_tier_has_becomes_a_load(self): second = stateful_seq(list(range(48))) assert bm._attach_state_slots(second, h) is True - assert second.state_load_hash == h + assert second.offload_joint.load_hash == h assert second.state_slot >= 0, "the H2D writes this slot directly" assert index.requested == [(second.id, h)] # No restore queued: there is nothing in HBM to gather from, and the @@ -1452,7 +1452,7 @@ def test_hbm_is_preferred_over_the_tier(self): second = stateful_seq(list(range(48))) assert bm._attach_state_slots(second, h) is True - assert second.state_load_hash == -1 + assert second.offload_joint.load_hash == -1 assert index.requested == [], "a resident image must not pay a park" assert len(bm.take_state_maintenance_ops().checkpoint_restores) == 1 @@ -1471,7 +1471,7 @@ def test_a_tier_that_declines_disowns_rather_than_parking(self): second = stateful_seq(list(range(48))) assert bm._attach_state_slots(second, h) is False - assert second.state_load_hash == -1 + assert second.offload_joint.load_hash == -1 assert bm.checkpoint_funnel()["state_gate_lost_boundary"] == 1 # ── the joint boundary, which #2045 makes the only source of value ── @@ -1553,14 +1553,14 @@ def test_a_page_class_now_gets_a_joint_boundary(self): self._reset_joint_counters(bm) second = stateful_seq(PROMPT) - second.offload_kv_prefix_tokens = len(PROMPT) + second.offload_joint.kv_prefix_tokens = len(PROMPT) hbm_hit = bm.can_allocate(second) assert bm.joint_boundaries == 1, bm.joint_skips - assert second.state_joint_boundary_hash == h - assert second.state_joint_boundary_tokens == 8 * BLOCK + assert second.offload_joint.boundary_hash == h + assert second.offload_joint.boundary_tokens == 8 * BLOCK # The KV leg has real work: the walk stopped well below the boundary. - assert hbm_hit * BLOCK < second.state_joint_kv_tokens + assert hbm_hit * BLOCK < second.offload_joint.kv_tokens def test_the_split_says_which_tier_the_state_leg_came_from(self): """`state_tier` is the only counter here that cannot be non-zero with @@ -1577,14 +1577,14 @@ def test_the_split_says_which_tier_the_state_leg_came_from(self): # The checkpoint outlived the block that broke the chain, so the state # leg is free -- a gather out of resident units, no transfer. resident = stateful_seq(PROMPT) - resident.offload_kv_prefix_tokens = len(PROMPT) + resident.offload_joint.kv_prefix_tokens = len(PROMPT) bm.can_allocate(resident) assert (bm.state_hbm, bm.state_tier) == (1, 0) # Drop it from HBM too: now the state leg costs an image-sized H2D. bm.paged_state_checkpoints.unindex(h) from_cpu = stateful_seq(PROMPT) - from_cpu.offload_kv_prefix_tokens = len(PROMPT) + from_cpu.offload_joint.kv_prefix_tokens = len(PROMPT) bm.can_allocate(from_cpu) assert (bm.state_hbm, bm.state_tier) == (1, 1) @@ -1599,7 +1599,7 @@ def test_no_tier_means_no_joint_boundary(self): self._reset_joint_counters(bm) seq = stateful_seq(PROMPT) - seq.offload_kv_prefix_tokens = len(PROMPT) + seq.offload_joint.kv_prefix_tokens = len(PROMPT) bm.can_allocate(seq) assert bm.joint_boundaries == 0 assert bm.joint_skips.get("off") == 1 @@ -3584,9 +3584,9 @@ def _bm(self, *tier_hashes): def _joint_seq(self, bm, tokens, *, boundary_hash_value, claim_tokens): seq = stateful_seq(list(tokens)) - seq.state_joint_boundary_hash = boundary_hash_value - seq.state_joint_boundary_tokens = claim_tokens - seq.state_joint_claim_tokens = claim_tokens + seq.offload_joint.boundary_hash = boundary_hash_value + seq.offload_joint.boundary_tokens = claim_tokens + seq.offload_joint.claim_tokens = claim_tokens return seq def test_the_state_leg_is_aimed_at_the_boundary_not_the_hbm_hit(self): @@ -3600,7 +3600,7 @@ def test_the_state_leg_is_aimed_at_the_boundary_not_the_hbm_hit(self): assert index.requested == [ (seq.id, 4242) ], "the state leg must be aimed at the joint boundary" - assert seq.state_joint_boundary_hash == 4242, "the boundary still stands" + assert seq.offload_joint.boundary_hash == 4242, "the boundary still stands" def test_a_boundary_with_no_state_behind_it_is_disowned(self): """The backstop. Whatever the gate decided, a request may not resume @@ -3610,8 +3610,8 @@ def test_a_boundary_with_no_state_behind_it_is_disowned(self): before = bm.state_gate_lost_boundary bm.allocate(seq, 0) - assert seq.state_joint_boundary_hash == -1 - assert seq.state_joint_boundary_tokens == 0 + assert seq.offload_joint.boundary_hash == -1 + assert seq.offload_joint.boundary_tokens == 0 assert seq.num_cached_tokens == 0 assert bm.state_gate_lost_boundary > before From f2b54e1aead70501e2ce1dddd674189f6324369d Mon Sep 17 00:00:00 2001 From: zejchen Date: Tue, 1 Sep 2026 11:56:43 +0000 Subject: [PATCH 097/151] =?UTF-8?q?[offload]=20=C2=A71=20slice(1):=20make?= =?UTF-8?q?=20the=20joint-boundary=20computation=20a=20side-effect-free=20?= =?UTF-8?q?value?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The maintainer's "clearest single tell that the layering is wrong is `record: bool`": the flag threaded `can_allocate` -> `_joint_kv_boundary` -> `_no_joint`, so a method whose name says it *computes* a boundary was also deciding whether to *record* one, and a fit probe and a real admission ran the same mutating path with the writes suppressed by a flag. This splits computing from recording, bounded to the two boundary methods and their one caller (no state moves across classes -- that is the deferred coordinator PR): * `_joint_kv_boundary(seq, hbm_boundary, block_hashes) -> JointBoundaryDecision` is now pure. It reads the seq's KV-leg inputs and the caches and returns a frozen `JointBoundaryDecision`, but writes nothing on the seq and moves no funnel counter. `_no_joint(reason)` returns the empty decision naming why. Both drop the `record` parameter. * `_commit_joint_boundary(seq, decision)` is the recording half: `reset_joint` then either write the four joint fields and bump `joint_boundaries` + `state_hbm`/`state_tier`, or bump `joint_skips[reason]`. The tier split is carried on the decision (`state_from_hbm`) so the commit does not re-read the checkpoint cache. * `can_allocate` computes the decision unconditionally and commits it only when `record` is true. `record` stays on `can_allocate` alone -- the legitimate probe/admission split -- and no longer reaches the computation. Behaviour-preserving. A real admission ends with the same seq fields and the same counter movements as before (the old top-of-method `reset_joint` is now the first line of the commit); a fit probe returns the same `num_cached_blocks` and, as before, moves no funnel counter -- it additionally now leaves the seq's joint fields untouched, which no reader observes before the next real `can_allocate` resets them (the sole probe caller reads only `>= 0`). The `can_allocate`/`allocate` double-hash the maintainer flags in §5 is left for the coordinator PR: killing it means threading the decision's chain into `allocate`, which crosses the two-methods bound this slice keeps. Full offload / scheduler / block-manager / state-checkpoint / page-unit / multi-connector suites green (679 passed; the one pre-existing `TestJointChunkProbeIsGated` failure is the k3f-has-lmcache env mismatch, untouched here). Co-Authored-By: Claude Opus 4.8 --- atom/model_engine/block_manager.py | 171 +++++++++++++++++++---------- 1 file changed, 114 insertions(+), 57 deletions(-) diff --git a/atom/model_engine/block_manager.py b/atom/model_engine/block_manager.py index 1d085ee0c7..289a3d7bca 100644 --- a/atom/model_engine/block_manager.py +++ b/atom/model_engine/block_manager.py @@ -3,6 +3,7 @@ import array import logging +from dataclasses import dataclass from math import inf, isinf from time import monotonic @@ -67,6 +68,34 @@ def _make_all_cleared() -> AllBlocksCleared: return AllBlocksCleared() +@dataclass(frozen=True) +class JointBoundaryDecision: + """The outcome of the joint-boundary computation, as a value. + + `_joint_kv_boundary` is a pure function of the prompt and the current pool + state: it reads the seq's KV-leg inputs and the caches, computes where (if + anywhere) both legs can jointly resume, and returns this object WITHOUT + touching the seq or any funnel counter. The caller decides whether to + commit it -- a fit probe discards it, a real admission applies it via + `_commit_joint_boundary`. That is what removed the `record` flag that used + to thread down into the boundary computation: computing and recording are + now two separate steps in two different places. + + `skip_reason is None` marks a real boundary and the four token/hash fields + carry it; otherwise `skip_reason` names why no joint boundary was found and + the other fields are unset. `state_from_hbm` tells the commit which tier the + STATE leg came from (HBM image vs. an image-sized H2D), decided here because + it is a read of the same cache state the boundary was computed against. + """ + + boundary_tokens: int = 0 + boundary_hash: int = -1 + kv_tokens: int = 0 + claim_tokens: int = 0 + state_from_hbm: bool = False + skip_reason: str | None = None + + class BlockManager: def __init__( self, @@ -639,17 +668,15 @@ def _joint_kv_boundary( seq: Sequence, hbm_boundary: int, block_hashes: list[int], - record: bool = True, - ) -> int: - """Boundary above the HBM hit that BOTH legs can reach, 0 if none. + ) -> JointBoundaryDecision: + """The boundary above the HBM hit that BOTH legs can reach, as a value. - `record` is `False` for a pure fit probe (`can_allocate` asked only - "does this seq fit?", not admitted): the seq's joint fields are still - set -- a later real `can_allocate` overwrites them anyway -- but the - `joint_boundaries` / `state_hbm` / `state_tier` / `joint_skips` funnel - counters are left alone, so a probe that never admits anything does not - inflate the operator-visible checkpoint funnel. (The demand counters are - already per-seq idempotent and need no such guard.) + A pure computation: it reads the seq's KV-leg inputs and the caches and + returns a `JointBoundaryDecision`, but writes nothing on the seq and + moves no funnel counter. `can_allocate` decides whether to commit the + result -- a real admission calls `_commit_joint_boundary`, a fit probe + discards it. Separating the two is what let the `record` flag that used + to thread down through here disappear. `can_allocate` walks the HBM prefix cache and nothing else, so an evicted prefix stops at the first miss even when LMCache holds every @@ -663,9 +690,8 @@ def _joint_kv_boundary( B: overshooting costs one chunk into blocks the forward is about to rewrite, undershooting would be silent wrong output. """ - seq.offload_joint.reset_joint() if self.state_offload is None: - return self._no_joint("off", record) + return self._no_joint("off") # PAGE is now the served class, not the excluded one. This used to # read `is not None` and refuse -- correct while a K3 checkpoint was an # Active Slot the tier spilled out of `StateSlotPool`. #2045 moved the @@ -675,7 +701,7 @@ def _joint_kv_boundary( # longer outlive its KV in HBM -- so when LMCache hands the KV back, # nothing hands the state back unless the two are fetched together. if not seq.has_per_req_cache or self.paged_state_checkpoints is None: - return self._no_joint("no_paged_checkpoints", record) + return self._no_joint("no_paged_checkpoints") hbs = self._hash_block_size() # From the connector's config, not the connector object: the scheduler # holds whatever `get_kvconnector` returned, and one without @@ -685,37 +711,34 @@ def _joint_kv_boundary( ) lmc_tokens = int(seq.offload_joint.kv_prefix_tokens or 0) if chunk <= 0: - return self._no_joint("no_chunk_size", record) + return self._no_joint("no_chunk_size") # Floored to the grid everything below compares against, so the # covering-chunk check becomes true by construction rather than by # luck. (`get_num_new_matched_tokens` withholds one token on a # full-prompt hit, which takes the lookup off the grid.) lmc_tokens = (lmc_tokens // chunk) * chunk if lmc_tokens <= hbm_boundary * hbs: - return self._no_joint("lmcache_within_hbm", record) + return self._no_joint("lmcache_within_hbm") # The KV leg moves whole chunks and the blocks below the HBM prefix are # shared, so an unaligned start cannot be rounded down. if (hbm_boundary * hbs) % chunk != 0: - return self._no_joint("hbm_off_chunk_grid", record) + return self._no_joint("hbm_off_chunk_grid") cap = min(lmc_tokens // hbs, self._n_hash_blocks(seq) - 1) if cap <= hbm_boundary: - return self._no_joint("no_room_above_hbm", record) + return self._no_joint("no_room_above_hbm") chain = self._chain_to(seq, block_hashes, cap) # One scan: `_gated_hit` already returns the rightmost rung # `_resumable_from` accepts, so a decrement-and-rescan walk would spend # a fixpoint pass per rung to reach the same answer. candidate = self._gated_hit(seq, cap, chain) if candidate <= hbm_boundary: - return self._no_joint("no_rung_above_hbm", record) + return self._no_joint("no_rung_above_hbm") tokens = candidate * hbs h = chain[candidate - 1] # Cannot fail given the two premises above; kept as their assertion. kv_tokens = -(-tokens // chunk) * chunk if kv_tokens > lmc_tokens: - return self._no_joint("covering_chunk_beyond_lookup", record) - seq.offload_joint.boundary_tokens = tokens - seq.offload_joint.boundary_hash = h - seq.offload_joint.kv_tokens = kv_tokens + return self._no_joint("covering_chunk_beyond_lookup") # Everything up to the compressed hit is this prompt's KV and is still # in the pool -- what `_gated_hit` cut was resumability, not residency. # Below the joint boundary those blocks are read-only for this request @@ -731,7 +754,7 @@ def _joint_kv_boundary( # Floored to the chunk grid so the KV leg starts aligned and writes # only into blocks below it that nobody else can be reading. claim_tokens = ((claim_blocks * hbs) // chunk) * chunk - seq.offload_joint.claim_tokens = max(claim_tokens, hbm_boundary * hbs) + claim_tokens = max(claim_tokens, hbm_boundary * hbs) # `allocate` claims the boundary in whole hash blocks -- # `claim_tokens // hbs` -- but the value above is floored to # the *chunk* grid, not the hash-block grid. When `hbs` does not divide @@ -743,30 +766,57 @@ def _joint_kv_boundary( # shipping case (`block_size * dcp_world_size == LMCACHE_CHUNK_SIZE`), so # it only surfaces at DCP world sizes that take `hbs` off the chunk grid. # Refuse the joint and recompute the tail rather than resume over a gap. - if seq.offload_joint.claim_tokens % hbs != 0: - seq.offload_joint.reset_joint() - return self._no_joint("claim_off_hash_grid", record) - if record: - self.joint_boundaries += 1 - # Which tier the STATE leg came from. The KV leg's own tier is - # decided separately by `_decide_load_after_alloc`, which is why - # these are not named for the boundary as a whole. - if self.paged_state_checkpoints.contains(h): - self.state_hbm += 1 - else: - self.state_tier += 1 - return candidate - - def _no_joint(self, reason: str, record: bool = True) -> int: - """Record why this admission got no joint boundary, and return 0. - Every reason is counted, including "this build is not even trying", - because a silent zero is indistinguishable from a feature that ran and - found nothing. `record=False` is the fit-probe path (see - `_joint_kv_boundary`): compute the answer without touching the funnel. + if claim_tokens % hbs != 0: + return self._no_joint("claim_off_hash_grid") + # Which tier the STATE leg came from. The KV leg's own tier is decided + # separately by `_decide_load_after_alloc`, which is why this is not + # named for the boundary as a whole. Read here, off the same cache the + # boundary was computed against, and carried on the decision so the + # commit does not have to look it up again. + return JointBoundaryDecision( + boundary_tokens=tokens, + boundary_hash=h, + kv_tokens=kv_tokens, + claim_tokens=claim_tokens, + state_from_hbm=self.paged_state_checkpoints.contains(h), + ) + + @staticmethod + def _no_joint(reason: str) -> JointBoundaryDecision: + """The empty decision, naming why no joint boundary was found. + Every reason is counted at commit time, including "this build is not + even trying", because a silent zero is indistinguishable from a feature + that ran and found nothing. """ - if record: - self.joint_skips[reason] = self.joint_skips.get(reason, 0) + 1 - return 0 + return JointBoundaryDecision(skip_reason=reason) + + def _commit_joint_boundary( + self, seq: Sequence, decision: JointBoundaryDecision + ) -> None: + """Apply a `JointBoundaryDecision` to the seq and the funnel counters. + + The recording half of what `_joint_kv_boundary` used to do inline under + `record=True`: a real admission calls this, a fit probe does not, so the + counters move once per admission and never for a seq that only probed. + `reset_joint` first so a boundary that resolved to a skip this pass + leaves no stale span from a prior one -- the same clean-slate the old + code got from resetting at the top of every boundary walk. + """ + seq.offload_joint.reset_joint() + if decision.skip_reason is not None: + self.joint_skips[decision.skip_reason] = ( + self.joint_skips.get(decision.skip_reason, 0) + 1 + ) + return + seq.offload_joint.boundary_tokens = decision.boundary_tokens + seq.offload_joint.boundary_hash = decision.boundary_hash + seq.offload_joint.kv_tokens = decision.kv_tokens + seq.offload_joint.claim_tokens = decision.claim_tokens + self.joint_boundaries += 1 + if decision.state_from_hbm: + self.state_hbm += 1 + else: + self.state_tier += 1 def _chain_to( self, seq: Sequence, block_hashes: list[int], blocks: int @@ -789,9 +839,10 @@ def can_allocate(self, seq: Sequence, record: bool = True) -> int: `record=False` marks a pure fit probe -- a caller asking only whether the seq *could* be admitted, not admitting it. The answer is identical; - it just does not move the joint-boundary funnel counters, so a probe - that never admits anything cannot inflate operator metrics (see - `_joint_kv_boundary`). + it just does not commit the joint boundary (neither the seq's joint + fields nor the funnel counters), so a probe that never admits anything + cannot inflate operator metrics (see `_joint_kv_boundary` / + `_commit_joint_boundary`). The hit count is the contiguous run of cache hits starting at the prompt's first block. On the first miss we break: subsequent blocks @@ -870,15 +921,21 @@ def can_allocate(self, seq: Sequence, record: bool = True) -> int: if not self._has_page_units(num_new_blocks, protected_hash): return -1 # A boundary LMCache and the tier can jointly reach, above this hit. - # Recorded on the seq rather than returned: what `allocate` claims from + # Committed to the seq rather than returned: what `allocate` claims from # HBM is still `num_cached_blocks`, and the joint boundary only decides - # where the two loads are aimed. Below the refusal, not above it, for the - # same reason _extend_hash_chain is: _joint_kv_boundary walks a chained - # xxhash up to the LMCache-only cap (_chain_to) plus a _gated_hit rescan, - # and a refused admission would discard all of it. Nothing between here - # and the refusal reads the seq's joint fields, and the author already - # moved _extend_hash_chain down for this exact cost -- this is its twin. - self._joint_kv_boundary(seq, num_cached_blocks, block_hashes, record=record) + # where the two loads are aimed. `record` is the probe/admission split: + # a real admission commits the decision (seq fields + funnel counters), a + # fit probe computes and drops it, so a seq that never schedules cannot + # inflate the operator-visible funnel. Below the refusal, not above it, + # for the same reason _extend_hash_chain is: _joint_kv_boundary walks a + # chained xxhash up to the LMCache-only cap (_chain_to) plus a _gated_hit + # rescan, and a refused admission would discard all of it. Nothing + # between here and the refusal reads the seq's joint fields, and the + # author already moved _extend_hash_chain down for this exact cost -- + # this is its twin. + decision = self._joint_kv_boundary(seq, num_cached_blocks, block_hashes) + if record: + self._commit_joint_boundary(seq, decision) # After the refusal, not before it. The chain is O(prompt) xxhash plus # two temporaries per block, and a refused admission discards it — a # 128k prompt queued behind a full pool paid ~2000 rounds per waiting From 96eb8ee4cbfe9201a0d5fb7aa519607a7ea585b7 Mon Sep 17 00:00:00 2001 From: zejchen Date: Tue, 1 Sep 2026 12:30:55 +0000 Subject: [PATCH 098/151] =?UTF-8?q?[offload]=20=C2=A71=20slice(1b):=20one?= =?UTF-8?q?=20chunk-size=20producer,=20no=20LMCache=20import=20in=20BlockM?= =?UTF-8?q?anager?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The joint KV chunk size had two producers OR-ed together in `_joint_kv_boundary`: `self._joint_chunk_tokens`, read from config at construction, and `seq.offload_joint.kv_chunk_tokens`, stamped off the connector in `_query_connector_prefill_match`. The review flagged the second as redundant, and the config read pulled an `atom.kv_transfer.offload.config` import into BlockManager -- a layering the coordinator split is meant to remove. Keep the config-derived producer (the one the surrounding docstrings already argue for: the connector object may lack `chunk_size` and would zero the grid silently) and delete the connector-fed path end to end: - Move the config read to `state_offload.state_tier_chunk_tokens`, the module that already resolves offload config for the engine (`state_tier_capability`, same `_offload_subconfig` unwrap). BlockManager loses the LMCache import and the `_read_joint_chunk_tokens` staticmethod; `__init__` calls the new helper. Kept out of `state_tier_capability` because that check is pure and has a second caller (`hosts_state_tier`) that must not reach into LMCache. - `_joint_kv_boundary` now reads `self._joint_chunk_tokens` alone. - Drop the dead second producer: the `kv_chunk_tokens` seq field, its write in the scheduler, and the two connector `chunk_size` forwarders that existed only to feed it (their docstrings say so). Update the shell-coverage anchor set and the no-impl fallback assertion that named `chunk_size`. Behavior-preserving: wherever the joint gate needs the chunk size it also needs `kv_prefix_tokens`, and both were stamped together, so nothing that could fire a joint load loses its chunk grid. Net -36 lines. UT: tests/test_block_manager.py tests/test_state_checkpoint.py tests/test_multi_connector.py tests/test_lmcache_offload_connector.py tests/test_scheduler.py tests/test_page_unit_checkpoint.py tests/test_dense_offload_connector.py tests/test_lmcache_offload_config.py all green (737 passed); the sole failure, TestJointChunkProbeIsGated hosted-tier "cannot read", is the pre-existing k3f-env case where lmcache IS installed so the probe reads 256 not 0 -- unchanged by this commit. Co-Authored-By: Claude Opus 4.8 --- .../disaggregation/multi/multi_connector.py | 23 +------- atom/kv_transfer/offload/connector.py | 12 ---- atom/model_engine/block_manager.py | 58 ++++--------------- atom/model_engine/scheduler.py | 3 - atom/model_engine/sequence.py | 9 +-- atom/model_engine/state_offload.py | 44 ++++++++++++++ tests/test_lmcache_offload_connector.py | 3 - 7 files changed, 58 insertions(+), 94 deletions(-) diff --git a/atom/kv_transfer/disaggregation/multi/multi_connector.py b/atom/kv_transfer/disaggregation/multi/multi_connector.py index 8395ae682e..2ae8e00b47 100644 --- a/atom/kv_transfer/disaggregation/multi/multi_connector.py +++ b/atom/kv_transfer/disaggregation/multi/multi_connector.py @@ -511,33 +511,12 @@ def abandon_save(self, req_id: Any) -> None: # The scheduler guards every one of these with hasattr(), so MultiConnector # only needs to expose them when a sub-connector implements them. - @property - def chunk_size(self) -> int | None: - """The offload sub's KV transfer grid, in tokens. - - `Scheduler._resolve_waiting_remote_kv` stamps - `seq.offload_joint.kv_chunk_tokens` from `getattr(self.kv_connector, - "chunk_size", 0)`, and under `multi` `self.kv_connector` IS this - composite. Without this forwarder the getattr default (0) wins, so - `_joint_kv_boundary` sees a zero chunk grid and refuses every joint KV - load with `no_chunk_size` -- the K3 state tier's KV leg is silently off - under `kv_connector: multi`. - - Prefer the state-tier sub's grid: `_joint_kv_boundary` aligns against - it, and with two offload subs (dense + kimi_k3) `_first_with` would have - handed back the dense sub's grid. Fall back to any offload sub's grid - when no tier is configured, so a plain `[moriio, lmcache_offload(dense)]` - still gets its dense KV chunk size. - """ - c = self._state_tier_sub() or _first_with(self._connectors, "chunk_size") - return getattr(c, "chunk_size", None) if c is not None else None - def should_park_for_load_after_alloc(self, seq: Any) -> bool: # Prefer the state-tier sub, then any offload sub: this is a load # decision routed through `_decide_load_after_alloc`, which kimi_k3 # overrides with a joint-boundary clamp. With two offload subs # (dense + kimi_k3) plain `_first_with` hands back the dense sub and the - # clamp is silently skipped. Mirrors the `chunk_size` forwarder. + # clamp is silently skipped. c = self._state_tier_sub() or _first_with( self._connectors, "should_park_for_load_after_alloc" ) diff --git a/atom/kv_transfer/offload/connector.py b/atom/kv_transfer/offload/connector.py index 96239be2e7..4564dabc41 100644 --- a/atom/kv_transfer/offload/connector.py +++ b/atom/kv_transfer/offload/connector.py @@ -153,18 +153,6 @@ def has_state_tier(self) -> bool: """ return isinstance(self._impl, StateOffloadFace) - @property - def chunk_size(self) -> int | None: - """The KV leg's transfer grid, in tokens. - - Read by `Scheduler._resolve_waiting_remote_kv` to stamp - `seq.offload_joint.kv_chunk_tokens`, which is `_joint_kv_boundary`'s fallback - when `BlockManager` could not read the LMCache config at startup. - Without this forwarder that fallback was dead -- `getattr` returned its - 0 default and the joint path refused with `no_chunk_size`. - """ - return getattr(self._impl, "chunk_size", None) - def get_num_new_matched_tokens(self, seq): return self._impl.get_num_new_matched_tokens(seq) diff --git a/atom/model_engine/block_manager.py b/atom/model_engine/block_manager.py index 289a3d7bca..bb818021b3 100644 --- a/atom/model_engine/block_manager.py +++ b/atom/model_engine/block_manager.py @@ -191,7 +191,10 @@ def __init__( # coordinator has to know at construction whether anything can carry a # store: without a sink it must nominate nothing, since a pin nobody # releases holds a whole image out of the pool forever. - from atom.model_engine.state_offload import state_tier_capability + from atom.model_engine.state_offload import ( + state_tier_capability, + state_tier_chunk_tokens, + ) # A capability derived from the whole config, not the connector's name. # The name only says which class is constructed; whether that class @@ -304,7 +307,7 @@ def __init__( # on every start, for a number that engine has no use for. The # capability above is the same gate the tier itself uses. if kv_offload_enabled: - self._joint_chunk_tokens = self._read_joint_chunk_tokens(config) + self._joint_chunk_tokens = state_tier_chunk_tokens(config) self.state_offload: StateOffloadIndex | None = None # (req_id, hash, target_group) admitted this pass and not yet handed to # the connector. Kept here rather than in the index because the slot @@ -703,12 +706,11 @@ def _joint_kv_boundary( if not seq.has_per_req_cache or self.paged_state_checkpoints is None: return self._no_joint("no_paged_checkpoints") hbs = self._hash_block_size() - # From the connector's config, not the connector object: the scheduler - # holds whatever `get_kvconnector` returned, and one without - # `chunk_size` would zero this and disable the feature silently. - chunk = self._joint_chunk_tokens or int( - seq.offload_joint.kv_chunk_tokens or 0 - ) + # Resolved once from config at construction (`state_tier_chunk_tokens`), + # not off the connector object: the scheduler holds whatever + # `get_kvconnector` returned, and one without `chunk_size` would zero + # this and disable the feature silently. + chunk = self._joint_chunk_tokens lmc_tokens = int(seq.offload_joint.kv_prefix_tokens or 0) if chunk <= 0: return self._no_joint("no_chunk_size") @@ -1109,46 +1111,6 @@ def allocate(self, seq: Sequence, num_cached_blocks: int = 0) -> bool: seq.offload_joint.boundary_hash = -1 return True - @staticmethod - def _read_joint_chunk_tokens(config) -> int: - """The LMCache chunk size in tokens, or 0 if it cannot be read. - - Read from the config rather than off the connector object, because the - scheduler holds whatever `get_kvconnector` returned and one without - `chunk_size` would zero this and disable the joint load silently. - - Called only when the capability says a state tier is hosted; see the - caller. - """ - try: - from atom.kv_transfer.offload import config as offcfg - from atom.model_engine.state_offload import _offload_subconfig - - kvcfg = getattr(config, "kv_transfer_config", None) or {} - # Under `multi` the lmcache.* keys (chunk_size, offload_layout) live - # on the offload SUB-connector, not the composite. Passing the raw - # composite here read a zero chunk grid, and `_joint_kv_boundary` - # then refused every joint KV load with `no_chunk_size`. Unwrap the - # sub the same way `state_tier_capability` does, so the gate's chunk - # size and the tier the capability check builds stay consistent. - if isinstance(kvcfg, dict) and kvcfg.get("kv_connector") == "multi": - sub, _why = _offload_subconfig(kvcfg) - if sub is not None: - kvcfg = sub - return int(offcfg.build_lmcache_config(kvcfg).chunk_size) - except Exception: - # Blind on purpose: this runs at model load, the import reaches a - # third-party package that may be absent entirely, and the only - # consequence of not knowing the chunk size is that the joint KV - # load stays off. Refusing to start would be worse. Reached only - # when the tier is on, so a traceback here is worth printing. - logger.warning( - "state offload: could not read the LMCache chunk size; the " - "joint KV load needs it and stays off", - exc_info=True, - ) - return 0 - def _state_leg_secured(self, seq: Sequence) -> bool: """Whether something will really put state behind this seq's boundary. diff --git a/atom/model_engine/scheduler.py b/atom/model_engine/scheduler.py index a2d1a51be4..0fa7fe8806 100644 --- a/atom/model_engine/scheduler.py +++ b/atom/model_engine/scheduler.py @@ -1912,9 +1912,6 @@ def _query_connector_prefill_match(self, seq: Sequence, *, skip: bool) -> bool: seq.offload_joint.kv_prefix_tokens = int(ext_tokens) + int( seq.num_cached_tokens ) - seq.offload_joint.kv_chunk_tokens = int( - getattr(self.kv_connector, "chunk_size", 0) or 0 - ) return needs_remote_load def _schedule_first_decode_after_remote_kv(self, seq: Sequence) -> None: diff --git a/atom/model_engine/sequence.py b/atom/model_engine/sequence.py index 78f9d68c8a..2b46ebbeb7 100644 --- a/atom/model_engine/sequence.py +++ b/atom/model_engine/sequence.py @@ -88,9 +88,6 @@ class OffloadJointRecord: # reported it -- the KV leg's ceiling. Written before `can_allocate`, which # is where the two legs agree on one boundary. 0 = no lookup. kv_prefix_tokens: int = 0 - # The connector's LMCache chunk size, in tokens, or 0. The KV leg moves - # whole chunks, so a joint boundary has to be a multiple of it. - kv_chunk_tokens: int = 0 # The boundary both legs of a joint load are aimed at, or 0. Chosen by # `can_allocate`: the rightmost checkpoint rung the LMCache KV prefix # covers. `num_cached_tokens` stays at the HBM prefix until both legs @@ -116,9 +113,9 @@ def reset_joint(self) -> None: The four joint fields move together -- a boundary is meaningless without the KV span aimed at it and the prefix claimed below it -- so the sites that abandon a joint load in full clear them through here - rather than by hand. Leaves `load_hash`, `kv_prefix_tokens`, and - `kv_chunk_tokens` untouched: they are set on their own paths and are - not part of the boundary this drops. + rather than by hand. Leaves `load_hash` and `kv_prefix_tokens` + untouched: they are set on their own paths and are not part of the + boundary this drops. """ self.boundary_tokens = 0 self.boundary_hash = -1 diff --git a/atom/model_engine/state_offload.py b/atom/model_engine/state_offload.py index 02e632a7f1..c961528d91 100644 --- a/atom/model_engine/state_offload.py +++ b/atom/model_engine/state_offload.py @@ -293,6 +293,50 @@ def state_tier_capability(config) -> StateTierCapability: return StateTierCapability(can_store, can_load, layout) +def state_tier_chunk_tokens(config) -> int: + """The LMCache chunk size in tokens for the joint KV leg, or 0. + + Decided from config for the same reason `state_tier_capability` is: this + runs in the engine process at `BlockManager.__init__`, and the scheduler + holds whatever `get_kvconnector` returned -- a connector object without + `chunk_size` would zero this and disable the joint KV load silently. The KV + leg moves whole LMCache chunks, so a joint boundary has to be a multiple of + this. + + Kept out of `state_tier_capability` on purpose: that check is pure and has a + second caller (`hosts_state_tier`), and this reaches into LMCache -- an + optional dependency whose absence must not warn on an engine that hosts no + tier. Call this only when the capability says a tier is hosted; the WARNING + below is then worth printing, because a missing chunk size really does + disable the joint KV load. + """ + try: + from atom.kv_transfer.offload.config import build_lmcache_config + + kvcfg = getattr(config, "kv_transfer_config", None) or {} + # Under `multi` the lmcache.* keys (chunk_size, offload_layout) live on + # the offload SUB-connector, not the composite. Passing the raw + # composite here read a zero chunk grid. Unwrap the sub the same way + # `state_tier_capability` does, so the gate's chunk size and the tier + # the capability check builds stay consistent. + if isinstance(kvcfg, dict) and kvcfg.get("kv_connector") == "multi": + sub, _why = _offload_subconfig(kvcfg) + if sub is not None: + kvcfg = sub + return int(build_lmcache_config(kvcfg).chunk_size) + except Exception: + # Blind on purpose: this runs at model load, the import reaches a + # third-party package that may be absent entirely, and the only + # consequence of not knowing the chunk size is that the joint KV load + # stays off. Refusing to start would be worse. + logger.warning( + "state offload: could not read the LMCache chunk size; the joint " + "KV load needs it and stays off", + exc_info=True, + ) + return 0 + + #: The connector backends whose worker half can build a `StateOffloadTier`. #: `multi` qualifies when it lists exactly one. _STATE_TIER_BACKENDS = frozenset({"lmcache_offload"}) diff --git a/tests/test_lmcache_offload_connector.py b/tests/test_lmcache_offload_connector.py index b5a47c2c8d..dde9983e86 100644 --- a/tests/test_lmcache_offload_connector.py +++ b/tests/test_lmcache_offload_connector.py @@ -5244,7 +5244,6 @@ def test_every_member_the_scheduler_reads_is_reachable_through_the_shell(): "request_finished", "build_connector_meta", "process_completions", - "chunk_size", "enqueue_state_stores", "take_state_reports", "take_state_source_releases", @@ -5296,8 +5295,6 @@ def test_the_shells_no_impl_fallbacks_match_what_the_caller_unpacks(): assert shell.enqueue_state_stores([]) is False # `bool(enqueue(stores))` assert shell.enqueue_state_loads([]) is False - # `int(getattr(..., "chunk_size", 0) or 0)` - assert shell.chunk_size is None or isinstance(shell.chunk_size, int) assert shell.should_park_partial_prefill_for_load(None) is False # Unchanged, so a connector with no opinion does not shrink the chunk. assert shell.adjust_prefill_chunk_after_alloc(None, 7) == 7 From c2bf4253b3ed896d092217cbb41e5237715800d7 Mon Sep 17 00:00:00 2001 From: zejchen Date: Tue, 1 Sep 2026 12:44:42 +0000 Subject: [PATCH 099/151] [offload] scrub dangling chunk_size docstring refs left by slice(1b) slice(1b) (96eb8ee4) deleted the two `chunk_size` shell forwarders but left three docstrings still pointing at them, which read as live cross-references to a member that no longer exists: - `connector.py` `_state_tier`: "Mirrors the scheduler shell's `chunk_size` forwarder" -> point at the surviving "every probed member appears on the shell" contract instead. - `connector.py` `max_pending_saves`: drop the trailing "Mirrors `chunk_size`." - `_offload_common.py` `StateOffloadFace`: the note explaining why `chunk_size` is not on the abstract face -> reworded to `max_pending_saves`, the attribute that is still a plain shell forward. Comments only; no behavior change. UT: tests/test_lmcache_offload_connector.py tests/test_multi_connector.py green (261 passed). Co-Authored-By: Claude Opus 4.8 --- atom/kv_transfer/offload/_offload_common.py | 6 +++--- atom/kv_transfer/offload/connector.py | 8 ++++---- 2 files changed, 7 insertions(+), 7 deletions(-) diff --git a/atom/kv_transfer/offload/_offload_common.py b/atom/kv_transfer/offload/_offload_common.py index 002f347a52..935c70103d 100644 --- a/atom/kv_transfer/offload/_offload_common.py +++ b/atom/kv_transfer/offload/_offload_common.py @@ -276,9 +276,9 @@ class StateOffloadFace(ABC): explicit type: only `KimiK3OffloadScheduler` inherits it, so `isinstance(impl, StateOffloadFace)` is the honest predicate. - Narrow on purpose -- only the four tier methods. `chunk_size` is an instance - attribute set in `__init__`, not a class method, so it cannot be abstract - here without breaking construction; it stays a plain shell forward. + Narrow on purpose -- only the four tier methods. Other impl attributes the + scheduler reads (e.g. `max_pending_saves`) stay plain shell forwards rather + than joining this face, which is exclusively the KDA state-tier contract. """ @abstractmethod diff --git a/atom/kv_transfer/offload/connector.py b/atom/kv_transfer/offload/connector.py index 4564dabc41..66e89a31a7 100644 --- a/atom/kv_transfer/offload/connector.py +++ b/atom/kv_transfer/offload/connector.py @@ -95,9 +95,9 @@ def _state_tier(self): Under the `multi` shell the sub IS this object, not the `kimi_k3` impl behind `_impl`, so without this forwarder the probe reads None on every member and the composite never adopts the tier that was actually built - -- state spills silently go nowhere. Mirrors the scheduler shell's - `chunk_size` forwarder and the "every probed member appears on the - shell" contract on `LMCacheOffloadConnectorScheduler`. + -- state spills silently go nowhere. The same "every probed member + appears on the shell" contract enforced on + `LMCacheOffloadConnectorScheduler`. """ return getattr(self._impl, "_state_tier", None) @@ -254,7 +254,7 @@ def max_pending_saves(self) -> int | None: `Scheduler._state_store_pending_cap` reads this off `self.kv_connector` -- which is this shell -- so it must appear here rather than only on the `_impl`, or the scheduler falls back to the bare env default of 2 and - the two legs pin different amounts of the same pool. Mirrors `chunk_size`. + the two legs pin different amounts of the same pool. """ return getattr(self._impl, "max_pending_saves", None) From 784b7cdb40064601ff0a61532f9f9332817a0c48 Mon Sep 17 00:00:00 2001 From: zejchen Date: Tue, 1 Sep 2026 12:57:11 +0000 Subject: [PATCH 100/151] [offload] fix can_allocate docstring: the fit probe is not "pure" The `can_allocate` docstring called `record=False` a "pure fit probe" that "just does not commit the joint boundary". It is not pure: the instrumentation write (`num_compressed_hit_blocks`) and the checkpoint-demand/-end records (`num_wanted_hit_blocks`, `checkpoint_demand_pos`, `checkpoint_end_pos`) run unconditionally, before the `record` gate. That is safe -- the sole probe caller (Scheduler's pre-admission fit test) reads only the `>= 0` return and never those fields, and the next real `can_allocate` overwrites them before any other reader -- so this corrects the wording rather than gating the writes (gating them would add risk to the checkpoint-ladder invariants for a value nobody reads on the probe path). The docstring now states the unconditional writes and scopes `record` to what it actually gates: only the joint-boundary commit (seq joint fields + funnel counters). Docstring only; no behaviour change. Co-Authored-By: Claude Opus 4.8 --- atom/model_engine/block_manager.py | 18 +++++++++++++----- 1 file changed, 13 insertions(+), 5 deletions(-) diff --git a/atom/model_engine/block_manager.py b/atom/model_engine/block_manager.py index bb818021b3..5a72493603 100644 --- a/atom/model_engine/block_manager.py +++ b/atom/model_engine/block_manager.py @@ -839,11 +839,19 @@ def _chain_to( def can_allocate(self, seq: Sequence, record: bool = True) -> int: """Return number of cache-hit blocks (>=0) if seq fits, else -1. - `record=False` marks a pure fit probe -- a caller asking only whether - the seq *could* be admitted, not admitting it. The answer is identical; - it just does not commit the joint boundary (neither the seq's joint - fields nor the funnel counters), so a probe that never admits anything - cannot inflate operator metrics (see `_joint_kv_boundary` / + `record=False` marks a fit probe -- a caller asking only whether the seq + *could* be admitted, not admitting it. The fit answer is identical. It is + not side-effect-free, though: the instrumentation write + (`num_compressed_hit_blocks`, line ~901) and the checkpoint-demand/-end + records (`_record_checkpoint_demand` / `_record_checkpoint_end`, which set + `num_wanted_hit_blocks` / `checkpoint_demand_pos` / `checkpoint_end_pos`) + run unconditionally, *before* the `record` gate. That is safe because the + sole probe caller -- `Scheduler`'s pre-admission fit test -- reads only + the `>= 0` return and never those fields, and the next real + `can_allocate` overwrites them before any other reader sees them. What + `record` actually gates is narrower: only the joint-boundary commit (the + seq's joint fields and the funnel counters), so a probe that never admits + cannot inflate the operator-visible funnel (see `_joint_kv_boundary` / `_commit_joint_boundary`). The hit count is the contiguous run of cache hits starting at the From db9a7dfac7c748230f3b81e11e7a1b4820bff878 Mon Sep 17 00:00:00 2001 From: zejunchen-zejun Date: Tue, 1 Sep 2026 21:01:08 +0800 Subject: [PATCH 101/151] style: satisfy Black's stub-body rule and three ruff findings Both pre-check style jobs were red on the current head. Black (26.5.1) collapses a function whose whole body is `...` onto one line. `tests/test_lmcache_offload_connector.py` had five such stubs in the ABC-contract test, written across two lines each. Ruff, on lines this branch touches: * RET501 x2 -- `release_stalled_save` on the dense and DSV4 schedulers ends in a bare `return None` that is the only possible return. The docstring on both already says the method is a no-op that exists so every offload impl answers the scheduler's forward uniformly, so the statement carried no information the reader did not already have. * SIM114 in the shell-coverage sweep -- the `ast.Return` and `ast.Expr` branches assigned `stmt.value` identically. Inverted into a single guard, with the void-forward note kept as a comment since that was the only thing the split branches were saying. No behaviour change; 4163 unit tests pass. --- atom/kv_transfer/offload/dense/connector.py | 1 - .../offload/hybrid/dsv4/connector.py | 1 - tests/test_lmcache_offload_connector.py | 20 +++++++------------ 3 files changed, 7 insertions(+), 15 deletions(-) diff --git a/atom/kv_transfer/offload/dense/connector.py b/atom/kv_transfer/offload/dense/connector.py index c5fdcd9220..e797550b66 100644 --- a/atom/kv_transfer/offload/dense/connector.py +++ b/atom/kv_transfer/offload/dense/connector.py @@ -714,7 +714,6 @@ def release_stalled_save(self, seq) -> None: this to pop its `_save_tracker`. Defined here so every offload impl answers the scheduler's `release_stalled_save` forward uniformly. """ - return None def has_pending_work(self) -> bool: """True while a load still needs dispatch or a save is unreported. diff --git a/atom/kv_transfer/offload/hybrid/dsv4/connector.py b/atom/kv_transfer/offload/hybrid/dsv4/connector.py index b11dac14c5..b0888beff9 100644 --- a/atom/kv_transfer/offload/hybrid/dsv4/connector.py +++ b/atom/kv_transfer/offload/hybrid/dsv4/connector.py @@ -2432,7 +2432,6 @@ def release_stalled_save(self, seq) -> None: never preemptable. Defined so every offload impl answers the scheduler's `release_stalled_save` forward uniformly (the mixin declares it abstract). """ - return None def connector_completion(self, completion: ConnectorCompletion) -> bool: """Apply one TP-aggregated completion owned by the DSV4 scheduler.""" diff --git a/tests/test_lmcache_offload_connector.py b/tests/test_lmcache_offload_connector.py index dde9983e86..555ded72e6 100644 --- a/tests/test_lmcache_offload_connector.py +++ b/tests/test_lmcache_offload_connector.py @@ -5339,12 +5339,10 @@ def _is_unconditional_forward(fn): if len(body) != 1: return None stmt = body[0] - if isinstance(stmt, ast.Return): - call = stmt.value - elif isinstance(stmt, ast.Expr): # void forward, returns None implicitly - call = stmt.value - else: + # `ast.Expr` is a void forward, which returns None implicitly. + if not isinstance(stmt, (ast.Return, ast.Expr)): return None + call = stmt.value if not isinstance(call, ast.Call): return None target = call.func # self._impl. @@ -5400,11 +5398,9 @@ def test_offload_mixin_lifecycle_is_enforced_at_construction(): from atom.kv_transfer.offload._offload_common import OffloadSchedulerMixin class MissingAbandon(OffloadSchedulerMixin): - def save_finished(self, req_id): - ... + def save_finished(self, req_id): ... - def release_stalled_save(self, seq): - ... + def release_stalled_save(self, seq): ... def load_failed(self, req_id): return False @@ -5412,8 +5408,7 @@ def load_failed(self, req_id): def load_finished(self, req_id): return True - def cancel_pending_load(self, seq): - ... + def cancel_pending_load(self, seq): ... # abandon_save deliberately left unimplemented. @@ -5421,8 +5416,7 @@ def cancel_pending_load(self, seq): MissingAbandon() class AllPresent(MissingAbandon): - def abandon_save(self, req_id): - ... + def abandon_save(self, req_id): ... AllPresent() # the full contract constructs cleanly From f432fdfb3b561952072f0ff4a37775e13358ccfa Mon Sep 17 00:00:00 2001 From: zejchen Date: Tue, 1 Sep 2026 13:01:35 +0000 Subject: [PATCH 102/151] [offload] tighten the can_allocate fit-probe docstring Trim the note added in 784b7cdb: drop the exhaustive field-name list and the stale `line ~901` reference, keeping the honest core -- the probe is not side-effect-free (instrumentation + checkpoint-demand/-end writes run before the `record` gate), it is safe because the sole probe caller reads only the `>= 0` return, and `record` gates only the joint-boundary commit. Docstring only. Co-Authored-By: Claude Opus 4.8 --- atom/model_engine/block_manager.py | 22 ++++++++-------------- 1 file changed, 8 insertions(+), 14 deletions(-) diff --git a/atom/model_engine/block_manager.py b/atom/model_engine/block_manager.py index 5a72493603..f3027771ac 100644 --- a/atom/model_engine/block_manager.py +++ b/atom/model_engine/block_manager.py @@ -839,20 +839,14 @@ def _chain_to( def can_allocate(self, seq: Sequence, record: bool = True) -> int: """Return number of cache-hit blocks (>=0) if seq fits, else -1. - `record=False` marks a fit probe -- a caller asking only whether the seq - *could* be admitted, not admitting it. The fit answer is identical. It is - not side-effect-free, though: the instrumentation write - (`num_compressed_hit_blocks`, line ~901) and the checkpoint-demand/-end - records (`_record_checkpoint_demand` / `_record_checkpoint_end`, which set - `num_wanted_hit_blocks` / `checkpoint_demand_pos` / `checkpoint_end_pos`) - run unconditionally, *before* the `record` gate. That is safe because the - sole probe caller -- `Scheduler`'s pre-admission fit test -- reads only - the `>= 0` return and never those fields, and the next real - `can_allocate` overwrites them before any other reader sees them. What - `record` actually gates is narrower: only the joint-boundary commit (the - seq's joint fields and the funnel counters), so a probe that never admits - cannot inflate the operator-visible funnel (see `_joint_kv_boundary` / - `_commit_joint_boundary`). + `record=False` marks a fit probe -- asking only whether the seq *could* + be admitted. The fit answer is identical, but the probe is not + side-effect-free: the instrumentation and checkpoint-demand/-end writes + below run before the `record` gate. That is safe -- the sole probe caller + reads only the `>= 0` return, and the next real `can_allocate` overwrites + those fields first. `record` gates only the joint-boundary commit (seq + joint fields + funnel counters), so a probe cannot inflate the + operator-visible funnel (see `_commit_joint_boundary`). The hit count is the contiguous run of cache hits starting at the prompt's first block. On the first miss we break: subsequent blocks From bc3bbc8e7def3b914fbecc4e80148a2c52f3e865 Mon Sep 17 00:00:00 2001 From: zejchen Date: Tue, 1 Sep 2026 16:10:39 +0000 Subject: [PATCH 103/151] [offload] declare the shell-forwarding axis across both shells and the worker side Round 1 flagged chunk_size/_state_tier missing on a shell; round 3 flagged abandon_save; this round flags close, save_abandon_timeout_s, release_stalled_save and get_statistics. These are not four bugs -- they are successive instances of one missing declaration: the connector shells forward every member by hand, so a member added to an _impl but not hand-forwarded is a silent fallthrough (getattr takes its default forever, no fault to point at). There are two forwarding axes and four shells, and each review round has found the defect on whichever the last point patch did not cover: - scheduler axis: LMCacheOffloadConnectorScheduler + MultiConnectorScheduler - worker axis: LMCacheOffloadConnector + MultiConnector The existing generated sweep only iterated the plain scheduler shell. Extend it to iterate BOTH shells on BOTH axes, and add the forwarders the extended sweep now demands: Findings 1 (+ its multi twin): close was added to every _impl and to the scheduler shell but to neither worker shell, so model_runner.py:1048 `getattr(connector, "close", None)` resolved to None and worker teardown never ran -- destroy_dist_env releases the KV pool while non-daemon lmc-state-store / lmc-state-load / offload-save threads still copy out of it. Add close on LMCacheOffloadConnector (plain forward, every _impl has it) and on MultiConnector (getattr-guarded fan-out; a producer sub need not implement it). Finding 2 + finding 9: MultiConnectorScheduler was missing save_abandon_timeout_s, release_stalled_save and get_statistics. Scheduler._save_abandon_timeout_s getattr-defaults to 0.0 when absent, silently switching off all three leak reclaimers (stalled-save reconcile, stale-store-pin reclaim, orphan-load-slot reconcile); get_statistics reads as {} so the whole offload metrics block vanishes under multi. Route the timeout through the tier sub then any offload sub, fan release_stalled_save to every sub that tracks saves, merge get_statistics across subs. Tests: _shell_exposes() helper resolves a member against a class counting constructor-set instance attributes (is_offload/is_producer are class attrs on the plain shells but instance attrs on the composites). Sweep #1 now loops both scheduler shells; a new sibling sweep loops both worker shells over the members model_runner/forward_context read off `connector` (anchored on close). The multi allow-list is deliberately empty so a member the composite legitimately routes via _state_tier_sub (as max_pending_saves is, via an aliased getattr the regex does not match) forces a deliberate decision rather than a default. Co-Authored-By: Claude Opus 4.8 --- .../disaggregation/multi/multi_connector.py | 70 +++++++++++ atom/kv_transfer/offload/connector.py | 16 +++ tests/test_lmcache_offload_connector.py | 114 +++++++++++++++++- 3 files changed, 194 insertions(+), 6 deletions(-) diff --git a/atom/kv_transfer/disaggregation/multi/multi_connector.py b/atom/kv_transfer/disaggregation/multi/multi_connector.py index 2ae8e00b47..c480b09358 100644 --- a/atom/kv_transfer/disaggregation/multi/multi_connector.py +++ b/atom/kv_transfer/disaggregation/multi/multi_connector.py @@ -394,6 +394,21 @@ def get_finished_recv_blocks(self) -> list[int]: blocks.extend(c.get_finished_recv_blocks()) return blocks + def close(self) -> None: + """Tear down every sub-connector at worker teardown. + + `ModelRunner.exit()` resolves `getattr(connector, "close", None)` on the + composite under `multi`; without this forwarder it returns None and no + sub is joined -- the offload sub's non-daemon `lmc-state-store` / + `lmc-state-load` / `offload-save` threads keep copying out of the KV pool + that `destroy_dist_env()` is about to release. Guard per sub (`getattr`): + a producer sub such as moriio need not implement `close`. + """ + for c in self._connectors: + fn = getattr(c, "close", None) + if callable(fn): + fn() + # --------------------------------------------------------------------------- # Scheduler side @@ -695,3 +710,58 @@ def load_failed(self, req_id: Any) -> None: for c in self._connectors: if hasattr(c, "load_failed"): c.load_failed(req_id) + + def save_abandon_timeout_s(self) -> float: + """The reclaim window for the offload leg, else 0.0 (disabled). + + `Scheduler._save_abandon_timeout_s` reads this off `self.kv_connector` + -- the composite under `multi` -- and `getattr`-defaults to 0.0 when it + is absent. 0.0 silently switches off all three leak reclaimers + (`_reconcile_stalled_deferred_saves`, `reclaim_stale_state_store_pins`, + `reconcile_orphan_load_slots`), so a missing forwarder here is not a + no-op: one lost save report then keeps `has_pending_kv_work()` True + forever, one dropped store completion pins a checkpoint image out of the + pool, one orphaned load slot wedges `can_allocate`'s state gate -- with + no fault to point at. The window is a global LMCache property + (`LMCACHE_EC_PIN_TIMEOUT_SEC + 30`), identical across offload subs; + prefer the tier sub, then any offload sub. Select by capability, never + method presence -- see `_state_tier_sub`. + """ + c = self._state_tier_sub() or _first_with( + self._connectors, "save_abandon_timeout_s" + ) + return c.save_abandon_timeout_s() if c is not None else 0.0 + + def release_stalled_save(self, seq: Any) -> None: + """Forward a stalled-save release to every sub that tracks saves. + + `Scheduler._connector_release_stalled_save` calls this on + `self.kv_connector` when `_reconcile_stalled_deferred_saves` reclaims a + save the connector never reported done. Only the offload sub tracks + `_save_inflight`; fan to whichever subs implement it (idempotent + pop-with-default, like `abandon_save`), so a producer sub without saves + is a no-op. Absent here, the connector's stall clock never advances and + the offload save loop wedges permanently. + """ + for c in self._connectors: + fn = getattr(c, "release_stalled_save", None) + if callable(fn): + fn(seq) + + def get_statistics(self) -> dict[str, int]: + """Merge offload metrics across subs, summing shared counters. + + `engine_utility` reads this off `self.kv_connector` under an `hasattr` + guard and renders `{}` when it is absent -- so under `multi` the whole + offload metrics block silently reads empty. Sum overlapping int keys + across every sub that reports (only offload subs do), rather than + first-hit, so a `[dense, kimi_k3]` composite reports both legs' totals. + """ + merged: dict[str, int] = {} + for c in self._connectors: + fn = getattr(c, "get_statistics", None) + if not callable(fn): + continue + for k, v in fn().items(): + merged[k] = merged.get(k, 0) + v + return merged diff --git a/atom/kv_transfer/offload/connector.py b/atom/kv_transfer/offload/connector.py index 66e89a31a7..568b3e3ecc 100644 --- a/atom/kv_transfer/offload/connector.py +++ b/atom/kv_transfer/offload/connector.py @@ -115,6 +115,22 @@ def get_finished(self): def get_finished_recv_blocks(self): return self._impl.get_finished_recv_blocks() + def close(self) -> None: + """Join the impl's save/load executors at worker teardown. + + `ModelRunner.exit()` resolves this through + `getattr(connector, "close", None)` on `self.kv_connector` -- which is + this shell. Without the forwarder that getattr returns None, so + `OffloadWorkerMixin.close()` / `KimiK3OffloadConnector.close()` never + run: `destroy_dist_env()` then releases the KV pool while non-daemon + `lmc-state-store` / `lmc-state-load` / `offload-save` threads are still + copying out of it -- use-after-free on the KV tensors, or the + interpreter-shutdown hang `close()` exists to prevent. Every `_impl` + defines `close` (`OffloadWorkerMixin`), so this is a plain forward, the + same "every probed member appears on the shell" contract as `_state_tier`. + """ + self._impl.close() + class LMCacheOffloadConnectorScheduler(KVConnectorSchedulerBase): """Scheduler-side shell delegating to the selected implementation. diff --git a/tests/test_lmcache_offload_connector.py b/tests/test_lmcache_offload_connector.py index 555ded72e6..64ed172b6d 100644 --- a/tests/test_lmcache_offload_connector.py +++ b/tests/test_lmcache_offload_connector.py @@ -5195,7 +5195,41 @@ def test_a_step_whose_only_work_is_a_state_store_reaches_the_worker(): assert connector_metadata_has_work(meta) -# ── the delegating shell must expose everything the scheduler reads ──────── +# ── the delegating shells must expose everything their driver reads ──────── +# +# There are two forwarding axes, not one, and every round of review has found +# the same defect on whichever axis the last patch did not cover: the scheduler +# reads members off `self.kv_connector` (the scheduler-side shell), the worker +# reads members off `connector` (the worker-side shell), and BOTH axes have a +# plain `LMCacheOffloadConnector*` shell AND a `Multi*` composite. A member +# added to an `_impl` but not hand-forwarded on one of these four classes is a +# silent fallthrough -- `close` (worker shells), `save_abandon_timeout_s` / +# `release_stalled_save` / `get_statistics` (the multi scheduler composite) are +# the most recent instances. The two sweeps below iterate both shells on each +# axis so a fifth instance fails a test instead of shipping. + + +def _shell_exposes(cls, name: str) -> bool: + """Whether `name` resolves on `cls`, counting attributes the constructor + sets on the instance (`self. = ...`). + + `hasattr(cls, name)` alone is not enough: `is_offload` / `is_producer` are + class attributes on the `LMCacheOffload*` shells but instance attributes on + the `Multi*` composites (assigned in `__init__` from the sub-connectors), so + a class-level `hasattr` would false-flag them as missing on the composite. + Parse the class source for `self. =` to cover that case without + constructing the class (which needs a live config and a worker build). + """ + import inspect + import re + + if hasattr(cls, name): + return True + try: + src = inspect.getsource(cls) + except (OSError, TypeError): + return False + return bool(re.search(r"\bself\.%s\s*=" % re.escape(name), src)) def test_every_member_the_scheduler_reads_is_reachable_through_the_shell(): @@ -5217,6 +5251,9 @@ def test_every_member_the_scheduler_reads_is_reachable_through_the_shell(): import re import atom.model_engine.scheduler as sched_mod + from atom.kv_transfer.disaggregation.multi.multi_connector import ( + MultiConnectorScheduler, + ) from atom.kv_transfer.offload.connector import LMCacheOffloadConnectorScheduler src = inspect.getsource(sched_mod) @@ -5241,6 +5278,7 @@ def test_every_member_the_scheduler_reads_is_reachable_through_the_shell(): "should_defer_free", "abandon_save", "save_abandon_timeout_s", + "release_stalled_save", "request_finished", "build_connector_meta", "process_completions", @@ -5256,13 +5294,77 @@ def test_every_member_the_scheduler_reads_is_reachable_through_the_shell(): f"{lost}. Update the patterns (or this anchor set) intentionally." ) - shell = LMCacheOffloadConnectorScheduler - missing = sorted(n for n in probed if not hasattr(shell, n)) - assert not missing, ( - "Scheduler reads these off `kv_connector`, but the delegating shell " - f"does not expose them, so each silently takes its default: {missing}" + # Both scheduler-side shells must expose every probed member: the plain + # `LMCacheOffloadConnectorScheduler` AND the `MultiConnectorScheduler` + # composite the engine holds under `kv_transfer_config: multi`. The + # composite's misses are the same silent default -- `Scheduler` reads it off + # `self.kv_connector`, which under `multi` IS the composite. The allow-list + # below is deliberately empty: a probed member the composite legitimately + # routes through `_state_tier_sub` instead of exposing (as `max_pending_saves` + # is, via an aliased `getattr` the regex does not match, so it never enters + # `probed`) would land here and SHOULD force a deliberate decision, not a + # default. If one ever does, add it here with the routing that covers it. + multi_routes_via_sub: set[str] = set() + for shell in (LMCacheOffloadConnectorScheduler, MultiConnectorScheduler): + allow = multi_routes_via_sub if shell is MultiConnectorScheduler else set() + missing = sorted( + n for n in probed if n not in allow and not _shell_exposes(shell, n) + ) + assert not missing, ( + f"Scheduler reads these off `kv_connector`, but {shell.__name__} does " + f"not expose them, so each silently takes its default: {missing}" + ) + + +def test_every_member_the_worker_reads_is_reachable_through_the_shell(): + """The worker forwarding axis, swept the same way as the scheduler axis. + + `ModelRunner` / `forward_context` hold the WORKER shell in `connector`, and + read members off it by hand exactly as the scheduler does off + `self.kv_connector`. This axis has its own two shells -- + `LMCacheOffloadConnector` and the `MultiConnector` composite -- and the same + silent-fallthrough failure: `close` was added to every `_impl` and to the + scheduler shell but not to either worker shell, so `getattr(connector, + "close", None)` resolved to None and worker teardown never ran (use-after- + free on the KV pool at `destroy_dist_env`, or the interpreter-shutdown hang + `close` exists to prevent). Read the driver's source, not a hand-kept list, + for the same reason the scheduler sweep does. + """ + import inspect + import re + + import atom.model_engine.model_runner as mr_mod + import atom.utils.forward_context as fc_mod + from atom.kv_transfer.disaggregation.multi.multi_connector import MultiConnector + from atom.kv_transfer.offload.connector import LMCacheOffloadConnector + + probed: set[str] = set() + for mod in (mr_mod, fc_mod): + src = inspect.getsource(mod) + probed |= set(re.findall(r'getattr\(\s*connector,\s*"([a-z_]+)"', src)) + probed |= set(re.findall(r"\bconnector\.([a-z_]+)\(", src)) + probed |= set(re.findall(r"\bconnector\.([a-z_]+)\b", src)) + probed.discard("_impl") + assert probed, "the scan found nothing -- it has stopped matching the source" + + # Same regex-rot guard as the scheduler sweep: anchor on reads we know the + # worker driver makes today, so a pattern that stops matching fails loudly + # here rather than shrinking `probed` in silence. `close` is the whole point + # of this sweep -- the member whose absence this test was written to catch. + must_be_seen = {"close", "register_kv_caches", "start_load_kv", "get_finished"} + lost = sorted(must_be_seen - probed) + assert not lost, ( + "the scan no longer sees these known worker reads -- coverage silently " + f"dropped: {lost}. Update the patterns (or this anchor set) intentionally." ) + for shell in (LMCacheOffloadConnector, MultiConnector): + missing = sorted(n for n in probed if not _shell_exposes(shell, n)) + assert not missing, ( + f"The worker reads these off `connector`, but {shell.__name__} does " + f"not expose them, so each silently takes its default: {missing}" + ) + def test_the_shells_no_impl_fallbacks_match_what_the_caller_unpacks(): """Existence is not enough -- a fallback of the wrong SHAPE is worse. From a05b15992cddc0c34060fdd6c3b75e7668a331dd Mon Sep 17 00:00:00 2001 From: zejchen Date: Tue, 1 Sep 2026 16:11:15 +0000 Subject: [PATCH 104/151] [offload] filter _state_loads in deallocate, matching cancel_state_load BlockManager.deallocate abandons the state-offload index entry for an aborted admission but left self._state_loads untouched, while cancel_state_load (block_manager.py:1291) filters it. So a requeued or preempted admission leaves its queued load behind, and _publish_state_loads later hands the connector a load for a request no longer in metadata.requests: the tier writes into a slot already released or orphaned, and the completion can settle that request's next-generation NEW park -- a state restore attributed to the wrong generation, silently. Sweep the same one line cancel_state_load uses into deallocate's load_hash != -1 branch. A no-op once the load was taken (in flight): the entry is already gone and the orphan-slot machinery holds the slot until settle_state_load. Co-Authored-By: Claude Opus 4.8 --- atom/model_engine/block_manager.py | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/atom/model_engine/block_manager.py b/atom/model_engine/block_manager.py index f3027771ac..b96dcd479f 100644 --- a/atom/model_engine/block_manager.py +++ b/atom/model_engine/block_manager.py @@ -2459,6 +2459,17 @@ def deallocate(self, seq: Sequence): # and forgetting the hash would cost the next request over this # prefix a full recompute. self.state_offload.abandon_load(seq.id) + # Drop any not-yet-taken load queued for this request, exactly as + # `cancel_state_load` does (finding #3). Without this, a requeued + # or preempted admission leaves its entry in `_state_loads`, and + # `_publish_state_loads` later hands the connector a load for a + # request no longer in `metadata.requests` -- the tier writes into + # a released or orphaned slot, and the completion can settle this + # request's NEXT-generation park, attributing a state restore to + # the wrong generation silently. A no-op once the load was taken + # (in flight): the entry is already gone and the orphan-slot path + # above holds the slot until `settle_state_load`. + self._state_loads = [e for e in self._state_loads if e[0] != seq.id] self.state.release_many(seq.state_slots[1:]) else: self.state.release_many(seq.state_slots) From 07b9f70a019ded8df6df5e4687bcff40da8e9fd5 Mon Sep 17 00:00:00 2001 From: zejchen Date: Tue, 1 Sep 2026 16:12:56 +0000 Subject: [PATCH 105/151] [offload][kimi_k3] isolate the pre-super() state submits so super() always runs start_load_kv ran _arm_joint_loads / _start_state_loads / _start_state_stores before super().start_load_kv(). ThreadPoolExecutor.submit raises RuntimeError if the executor was shut down by a racing close() or the OS refuses a thread, and ModelRunner.process_kvconnector_output has no handler -- so the escape dropped every KV load and save in that metadata AND left the parks armed a few lines earlier owing a KV leg nobody would run, so those requests never woke. This also re-opens a finding refuted only conditionally: "_JointPark leaks on abort" was REFUTED because "the KV leg always reports", which holds only while super().start_load_kv cannot be skipped. Isolate each submit inside its helper -- a failed load settles its state leg False (the armed park resolves to a recompute), a failed store routes into the no-tier failure channel get_finished already drains (units released now, not on the reconciler's timeout) -- and run super() in a finally so a KV leg is never dropped and no armed park is left owing one. Fixes the shape, not just the instance the constructor comment already records. Co-Authored-By: Claude Opus 4.8 --- .../offload/hybrid/kimi_k3/connector.py | 49 +++++++++++++++++-- 1 file changed, 44 insertions(+), 5 deletions(-) diff --git a/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py b/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py index 75665baed4..97cfd7ecb4 100644 --- a/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py +++ b/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py @@ -219,11 +219,27 @@ def _build_state_tier(self, transfer_tensors) -> None: # -- per-step ---------------------------------------------------------- def start_load_kv(self, metadata) -> None: - if isinstance(metadata, LMCacheOffloadMetadata): - self._arm_joint_loads(metadata) + if not isinstance(metadata, LMCacheOffloadMetadata): + super().start_load_kv(metadata) + return + # The three state helpers run before super() so the KV leg is submitted + # against a park that already exists. They must not be able to skip + # super(): _JointPark's correctness -- and the refutation of the + # "park leaks on abort" candidate -- both rest on "the KV leg always + # reports", which holds only while super().start_load_kv() cannot be + # skipped. `ThreadPoolExecutor.submit` raises RuntimeError if the + # executor was shut down by a racing close() or the OS refuses a thread, + # and ModelRunner.process_kvconnector_output has no handler -- so each + # submit is isolated inside its helper (a failure fails just that item, + # into the same failed-load / failed-store channels a no-tier step uses, + # so the armed park resolves to a recompute instead of hanging), and + # super() runs in a finally so a KV load or save is never dropped. + self._arm_joint_loads(metadata) + try: self._start_state_loads(metadata) self._start_state_stores(metadata) - super().start_load_kv(metadata) + finally: + super().start_load_kv(metadata) def _arm_joint_loads(self, metadata) -> None: """Hold a request owning both legs until both report. @@ -299,7 +315,19 @@ def _start_state_loads(self, metadata) -> None: self._fail_state_loads(loads) return for req_id, h, group in loads: - self._state_tier.submit_load(req_id, int(h), int(group)) + # Isolate each submit (finding #4): a shut-down executor or an OS + # thread refusal fails only this load -- settle its state leg False + # so the armed park resolves to a recompute -- and cannot take down + # the remaining loads or, via an escape past start_load_kv, the KV + # legs super() still owes. + try: + self._state_tier.submit_load(req_id, int(h), int(group)) + except Exception: + logger.exception( + "kimi_k3 offload: submit_load failed for %s; failing it.", + req_id, + ) + self._joint_park.settle_state(req_id, False) def _start_state_stores(self, metadata) -> None: """Hand this step's ready checkpoints to the tier's executor. @@ -325,7 +353,18 @@ def _start_state_stores(self, metadata) -> None: self._store_failed_no_tier.add(op) return for op, unit_ids in stores: - self._state_tier.submit_store(op, tuple(int(u) for u in unit_ids)) + # Isolate each submit (finding #4): route a submit failure into the + # same no-tier failure channel (get_finished drains it) so the engine + # releases the store's pinned PAGE units now rather than on the + # reconciler's full timeout, without escaping past super(). + try: + self._state_tier.submit_store(op, tuple(int(u) for u in unit_ids)) + except Exception: + logger.exception( + "kimi_k3 offload: submit_store failed for %s; failing it.", + op, + ) + self._store_failed_no_tier.add(op) def _fail_state_loads(self, loads) -> None: for req_id, _h, _group in loads: From 49ef7e4c347e46235e7144334e2fd28f66f8e710 Mon Sep 17 00:00:00 2001 From: zejchen Date: Tue, 1 Sep 2026 16:13:46 +0000 Subject: [PATCH 106/151] [offload][kimi_k3] allocate the staging buffer on the first consumer stream staging.py's pack() and unpack() called ensure_buffer (which may torch.empty device storage) inside _device_ctx(), which sets the device but not the stream, and before the first stream_ctx -- so the allocation ran on the default stream. A caching allocator can hand back a freshly-freed ~55 MiB block whose previous default-stream user is still running; the Triton gather on pack_stream then overwrites it early, and the CPU-tier checkpoint holds a mix of the image and unrelated KV bytes filed under a valid prefix hash -- a later resume is silent wrong output. The sibling atom_lmcache_staging.py:155 already allocates inside the first consumer stream with this exact hazard spelled out, and _block_gpu_connector.py follows it; staging.py was the only divergence (grep -rn record_stream across the subsystem returns zero). Move ensure_buffer inside the first stream_ctx: pack_stream for the pack gather, copy_stream for the unpack H2D. Bounded today (one buffer per thread, OFFLOAD_RELEASE_GPU_STAGING_AFTER_TRANSFER defaults 0) but becomes per-transfer the moment that flag is set. Co-Authored-By: Claude Opus 4.8 --- atom/kv_transfer/offload/hybrid/kimi_k3/staging.py | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/atom/kv_transfer/offload/hybrid/kimi_k3/staging.py b/atom/kv_transfer/offload/hybrid/kimi_k3/staging.py index a636efd58a..705174768f 100644 --- a/atom/kv_transfer/offload/hybrid/kimi_k3/staging.py +++ b/atom/kv_transfer/offload/hybrid/kimi_k3/staging.py @@ -185,9 +185,17 @@ def pack(self, segments: list[torch.Tensor], dst: Any) -> None: self._drain_device() raise return - device_buf = self.ensure_buffer(staging_buffer, nbytes) try: with state.stream_ctx(state.pack_stream): + # `ensure_buffer` may allocate device storage. Allocate it on + # the first consumer stream (the gather below), not the + # default stream, so a caching allocator cannot hand back + # storage whose previous default-stream user is still running + # and then let this side stream overwrite it early -- the CPU + # tier would otherwise hold a mix of the image and unrelated + # KV bytes under a valid prefix hash. Mirrors the hazard note + # on `atom_lmcache_staging.py`'s `ensure_buffer` call. + device_buf = self.ensure_buffer(staging_buffer, nbytes) fused_pack_chunk_major(segments, seg_bytes, [1], [0], device_buf) self._handoff(state, state.pack_stream, state.copy_stream) with state.stream_ctx(state.copy_stream): @@ -250,9 +258,11 @@ def unpack(self, src: Any, segments: list[torch.Tensor]) -> None: self._drain_device() raise return - device_buf = self.ensure_buffer(staging_buffer, nbytes) try: with state.stream_ctx(state.copy_stream): + # Allocate on the first consumer stream (the H2D copy below), + # not the default stream -- see the matching note in `pack`. + device_buf = self.ensure_buffer(staging_buffer, nbytes) device_buf.copy_( src_tensor, non_blocking=src_tensor.device.type != "cpu", From 3fa6bbb423e6f80d6132814e62c90840726abc8d Mon Sep 17 00:00:00 2001 From: zejchen Date: Tue, 1 Sep 2026 16:31:05 +0000 Subject: [PATCH 107/151] [offload][kimi_k3] abandon (not fail) a state load whose only failed leg was KV A joint load pairs a recurrent-state H2D with a KV-chunk load under one park. When the state bytes land intact but LMCache's LRU has since dropped the KV chunk, the park settles failure-dominant (`_failed`), the KV leg reports in `failed_loading`, and the engine ran `_settle_state_load(req_id, ok=False)` -> `BlockManager.settle_state_load` -> `StateOffloadIndex.fail_load` -> `forget(h)`. That permanently un-advertises a state image whose bytes are still present: the next request over that prefix can no longer reload it and recomputes from scratch. The maintainer's note is exact -- `abandon_load` exists for this "neither outcome" case (release the reservation, keep the hash loadable) and was not being used. Distinguish the two failure shapes at the source and carry the distinction to the engine on the same step the KV report arrives: - `_JointPark` now tracks `_state_failed` separately from the single failure-dominant `_failed`. `_settle` adds a key to `_state_failed` only when the *state* leg itself misses; a KV-leg failure leaves it clear. `_release` files a per-key disposition (`_dispositions[key] = key not in _state_failed`) for *every* released key -- success and failure alike -- so the worker always emits a completion the TP aggregator can bring to quorum alongside the KV load report instead of leaking a partially-reported key. `take_dispositions` drains them next to `take_ready`. - `KimiK3OffloadConnector.get_finished` emits one `ConnectorCompletion(STATE_LOAD_DISPOSITION_CHANNEL, key, state_intact)` per released load (both the no-tier and tier `_settle_joint` paths). The channel is TP-aggregated failure-dominant like the load report, so a survivor is advertised only when all workers agree the state leg held. `connector_completion` records survivors into `_state_load_survived`, normalising the KV identity to the bare request id exactly as `process_completions` normalises `finished/failed_loading`, so the engine's membership test lines up. `take_state_load_survived` drains it. - `Scheduler` drains the survivor set before the `failed_loading` loop and branches: a survivor gets the new `_abandon_state_load` (guarded delegate to `BlockManager.abandon_state_load` -- pops the reservation, releases the orphan slot, keeps the hash) instead of `_settle_state_load(ok=False)`. A genuine state-leg miss, or a connector that never advertises the channel, still fails and forgets as before. Co-Authored-By: Claude Opus 4.8 --- .../offload/hybrid/kimi_k3/connector.py | 61 +++++++++++++++++++ .../offload/hybrid/kimi_k3/state_tier.py | 41 +++++++++++++ atom/model_engine/scheduler.py | 35 ++++++++++- 3 files changed, 136 insertions(+), 1 deletion(-) diff --git a/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py b/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py index 97cfd7ecb4..2830fc559c 100644 --- a/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py +++ b/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py @@ -49,6 +49,15 @@ #: CPU put succeeded is decided afterwards and cannot touch them. STATE_SOURCE_CHANNEL = "k3_state_source" +#: The per-request state-load verdict, one completion per released joint/state +#: load (finding #6). `succeeded=True` means the state H2D landed and the index +#: must keep the hash even when the pair failed (a dropped KV chunk): +#: `abandon_load`, not `fail_load`. `succeeded=False` means the state leg itself +#: missed and the hash must be forgotten. Emitted for successful loads too so +#: the aggregator reaches quorum on the same step the KV load report does, +#: rather than leaving a partially-reported key pending forever. +STATE_LOAD_DISPOSITION_CHANNEL = "k3_state_load_disposition" + #: A save outstanding longer than this is a backend that stopped, not one that #: is busy: a 4096-token store costs ~65ms. SAVE_STALL_SECONDS = 120.0 @@ -402,12 +411,14 @@ def get_finished(self): out.finished_loading, out.failed_loading = self._settle_joint( out.finished_loading, out.failed_loading, set(), set() ) + self._emit_load_dispositions(out) return out indexed, index_failed = self._state_tier.take_store_reports() state_done, state_failed = self._state_tier.get_finished() out.finished_loading, out.failed_loading = self._settle_joint( out.finished_loading, out.failed_loading, state_done, state_failed ) + self._emit_load_dispositions(out) # Store reports have no request identity (the owner is long gone), so # they ride the connector-owned channel with its failure-dominant quorum. # Keyed by operation, not bare hash: `KVOutputAggregator` tombstones each @@ -453,6 +464,26 @@ def _settle_joint(self, kv_done, kv_failed, state_done, state_failed): ready, ready_failed = park.take_ready() return passthrough_done | ready, passthrough_failed | ready_failed + def _emit_load_dispositions(self, out) -> None: + """Turn this step's park verdicts into state-index completions (finding #6). + + One `STATE_LOAD_DISPOSITION_CHANNEL` completion per released joint/state + load, keyed by the same identity the KV leg reported on + `finished/failed_loading` so the engine can correlate them. `succeeded` + is the state leg's own outcome: the engine keeps the hash (`abandon_load`) + for a failed pair whose state bytes are intact, and forgets it + (`fail_load`) only when the state leg itself missed. Emitting for the + successful loads too is deliberate -- the failure-dominant TP quorum + needs every worker that reported the load to report a verdict, or the + key never drains. + """ + for key, state_intact in self._joint_park.take_dispositions().items(): + out.connector_completions.add( + ConnectorCompletion( + STATE_LOAD_DISPOSITION_CHANNEL, key, bool(state_intact) + ) + ) + class KimiK3OffloadScheduler(DenseOffloadScheduler, StateOffloadFace): """Scheduler side: dense KV, plus the state tier's load queue and the @@ -489,6 +520,10 @@ def __init__(self, config) -> None: self._state_indexed: set = set() self._state_index_failed: set = set() self._state_source_released: set = set() + # Requests whose state H2D landed even though the joint load failed + # (finding #6). Drained by the engine before it settles `failed_loading`, + # to abandon rather than forget their still-present state hash. + self._state_load_survived: set = set() # -- state load queue -------------------------------------------------- def enqueue_state_loads(self, loads) -> bool: @@ -714,6 +749,21 @@ def connector_completion(self, completion) -> bool: ) target.add(completion.operation_id) return True + if completion.channel == STATE_LOAD_DISPOSITION_CHANNEL: + # Finding #6: a state load whose bytes survived a failed pair. Record + # only the survivors -- their `failed_loading` report must abandon + # the index (keep the hash) instead of failing it (forget). A + # genuine state miss drains `succeeded=False` and is left out, so the + # engine's default `fail_load` still forgets it. Normalise the KV + # identity to the bare request id here, matching how + # `process_completions` normalises `finished/failed_loading`, so the + # engine's set membership test lines up. + if completion.succeeded: + op = completion.operation_id + self._state_load_survived.add( + op.req_id if hasattr(op, "req_id") else op + ) + return True # Channels this connector does not own. `DenseOffloadConnector` and the # rest of the MRO define no `connector_completion`, so `super().` would # raise AttributeError; `False` is the caller's contract for "unhandled" @@ -745,3 +795,14 @@ def take_state_reports(self) -> tuple[set[int], set[int]]: self._state_indexed = set() self._state_index_failed = set() return indexed, failed + + def take_state_load_survived(self) -> set: + """Drain the requests whose state bytes outlived a failed joint load. + + Finding #6. The engine consults this before settling `failed_loading`: + a member here abandons its state-index entry (keeps the loadable hash) + instead of failing it (forgetting a hash whose bytes are present). + """ + survived = self._state_load_survived + self._state_load_survived = set() + return survived diff --git a/atom/kv_transfer/offload/hybrid/kimi_k3/state_tier.py b/atom/kv_transfer/offload/hybrid/kimi_k3/state_tier.py index a0a167165e..1e4c38a045 100644 --- a/atom/kv_transfer/offload/hybrid/kimi_k3/state_tier.py +++ b/atom/kv_transfer/offload/hybrid/kimi_k3/state_tier.py @@ -282,6 +282,19 @@ def __init__(self) -> None: self._failed: set = set() self._ready: set = set() self._ready_failed: set = set() + # Per-leg outcome, kept apart from the single failure-dominant `_failed` + # flag (finding #6). `_failed` fires when *either* leg fails, which is + # right for the KV wake -- half a load must recompute -- but wrong for + # the state index: a KV chunk that LMCache's LRU dropped fails the pair + # while the state H2D that landed is still intact, and routing that + # through `fail_load` would `forget` a hash whose bytes are present, + # denying it to every later request over the prefix. `_state_failed` + # records the keys whose *state* leg specifically missed, so `_release` + # can tell "keep the state hash, only KV recompute" (abandon) from "the + # state bytes are really gone" (fail/forget). `_dispositions` carries + # that verdict per released key out to the worker's completion channel. + self._state_failed: set = set() + self._dispositions: dict = {} # The two legs report different identities: the KV leg reports its typed # `LoadOperationId` (always issued for an offload load), the state tier # the bare request id. The park is filed under the KV identity -- that @@ -335,6 +348,12 @@ def _settle(self, ident, leg: str, ok: bool) -> None: need.discard(leg) if not ok: self._failed.add(key) + if leg == "state": + # Only a state-leg miss is evidence the bytes are gone (finding + # #6). A KV-leg failure leaves `_state_failed` untouched, so the + # disposition `_release` files says "state intact" and the index + # abandons rather than forgets. + self._state_failed.add(key) if need: return self._release(key) @@ -344,6 +363,15 @@ def _release(self, key) -> None: bare = self._alias_of.pop(key, None) if bare is not None: self._alias.pop(bare, None) + # File the state-index verdict for this key before dropping the per-leg + # bookkeeping (finding #6): True means the state bytes are intact (the + # index should keep the hash -- `abandon_load`), False means the state + # leg itself missed (the index should drop it -- `fail_load`). Every + # released key gets one, success or failure, so the worker emits a + # completion the TP aggregator can bring to quorum alongside the KV load + # report rather than leaking a partially-reported key. + self._dispositions[key] = key not in self._state_failed + self._state_failed.discard(key) if key in self._failed: self._failed.discard(key) self._ready_failed.add(key) @@ -366,3 +394,16 @@ def take_ready(self) -> tuple[set[str], set[str]]: self._ready.clear() self._ready_failed.clear() return ready, failed + + def take_dispositions(self) -> dict: + """State-index verdict per released key since the last drain (finding #6). + + `{key: state_intact}` -- True keeps the hash (abandon), False forgets it + (fail). Drained together with `take_ready`: the worker turns each into a + `STATE_LOAD_DISPOSITION_CHANNEL` completion so the engine can override + the failure-dominant default of `fail_load` for a load whose only failed + leg was the KV chunk. + """ + out = self._dispositions + self._dispositions = {} + return out diff --git a/atom/model_engine/scheduler.py b/atom/model_engine/scheduler.py index 0fa7fe8806..06d98aba03 100644 --- a/atom/model_engine/scheduler.py +++ b/atom/model_engine/scheduler.py @@ -2127,6 +2127,24 @@ def _settle_state_load(self, req_id, ok: bool) -> None: if settle is not None: settle(req_id, ok=ok) + def _abandon_state_load(self, req_id) -> None: + """Release a state group a load reserved *without* voting the outcome. + + The 'neither outcome' release, for a joint load whose state H2D landed + intact but whose KV leg failed (an LMCache LRU miss on the KV chunk). + Settling that as a failure would call `StateOffloadIndex.fail_load` and + `forget(h)`, permanently un-advertising a state image whose bytes are + still present -- the next request over that prefix could have reloaded + it. `abandon_load` pops the pending reservation and releases the orphan + slot but keeps the hash loadable. Same guards as `_settle_state_load`: + no-op for an id the state index never issued and for a scheduler built + without a block manager (the connector test doubles).""" + abandon = getattr( + getattr(self, "block_manager", None), "abandon_state_load", None + ) + if abandon is not None: + abandon(req_id) + def _state_store_pending_cap(self) -> int: """The running-plus-queued cap for state-tier stores. @@ -3095,6 +3113,18 @@ def _update_from_kv_xfer_finished(self, kv_connector_output: KVConnectorOutput): continue self.finished_recving_kv_req_ids.append(req_id) + # A joint load can fail on its KV leg alone while its state H2D landed + # intact (an LMCache LRU miss dropping the KV chunk). The connector + # advertises those survivors on a failure-dominant, TP-quorumed + # disposition channel, drained here on the same step the KV failure + # report arrives: a survivor is *abandoned* (pending released, hash + # kept loadable) rather than *failed* (hash forgotten). Everything else + # -- a genuine state-leg miss, or a connector that never advertises the + # channel -- still settles as a failure. + state_survived = getattr( + self.kv_connector, "take_state_load_survived", None + ) + state_survived = state_survived() if state_survived is not None else set() for req_id in kv_connector_output.failed_loading or (): assert ( is_offload @@ -3103,7 +3133,10 @@ def _update_from_kv_xfer_finished(self, kv_connector_output: KVConnectorOutput): "Offload KV load failed for request %s; falling back to prefill.", req_id, ) - self._settle_state_load(req_id, ok=False) + if req_id in state_survived: + self._abandon_state_load(req_id) + else: + self._settle_state_load(req_id, ok=False) if self._finish_aborted_load_cleanup(req_id): continue self.failed_recving_kv_req_ids.append(req_id) From 18819a2559c95abd4a08a3050a16e7816061b5b8 Mon Sep 17 00:00:00 2001 From: zejchen Date: Tue, 1 Sep 2026 16:32:30 +0000 Subject: [PATCH 108/151] [offload][kimi_k3] guard state_entry_views against a negative slot, matching DSV4 `deepseek_v4_attn.py:852` gained an explicit `if slot < 0: raise IndexError` guard in this same PR, with a comment on the silent cross-request corruption a stray -1 sentinel would cause (list/tensor indexing at -1 returns the last slot's views instead of raising). The GDN twin `state_entry_views` did the bare `cache[layer, slot : slot + 1]` with no such check, and `_KimiMLAGDNCommon(PageUnitGeometryMixin, GDNStateMixin)` does not override it, so K3 inherited the unguarded gather. The invariant holds structurally today -- `StateSlotPool.pop` never returns negative and both `_attach_state_slots` branches pop before requesting the load -- so this is the fix-then-sweep parity gap, not a demonstrated failure. One line, and the two twins now fail identically loud instead of one corrupting silently. Co-Authored-By: Claude Opus 4.8 --- atom/model_ops/attentions/gdn_attn.py | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/atom/model_ops/attentions/gdn_attn.py b/atom/model_ops/attentions/gdn_attn.py index aa31968bcd..6748e70e20 100644 --- a/atom/model_ops/attentions/gdn_attn.py +++ b/atom/model_ops/attentions/gdn_attn.py @@ -836,6 +836,17 @@ def state_entry_views(self, slot: int) -> list[torch.Tensor]: committed state (#2045), and the speculation scratch beside it is this request's own and resumable by nobody. """ + # Reject a negative slot explicitly, for parity with the DSV4 twin + # (`deepseek_v4_attn.py`). `cache[layer, slot : slot + 1]` on a stray -1 + # (the "no slot" sentinel used elsewhere on this path) silently gathers + # or scatters the last slot's state under this request's identity -- + # silent cross-request corruption. The upstream invariant is that a real + # slot is non-negative (`StateSlotPool.pop` never returns negative, and + # `_attach_state_slots` pops before requesting the load), so this guards + # the shape rather than a demonstrated path; fail loudly if it is ever + # violated instead of corrupting silently. + if slot < 0: + raise IndexError(f"state_entry_views: invalid negative slot {slot}") views = [] for cache in ( self.model_runner.mamba_k_cache, From 8337571c98f67c9e9bc02207db8c403c26acb3b1 Mon Sep 17 00:00:00 2001 From: zejchen Date: Tue, 1 Sep 2026 16:33:17 +0000 Subject: [PATCH 109/151] [offload][kimi_k3] key the page-unit region cache on full layout, not data_ptr `_page_unit_regions` cached its result keyed on `cache.data_ptr()` alone and returned early on an address match, before the `is_contiguous()` and the `rows * region == page_unit_bytes` granularity checks. Address equality does not imply geometry equality, and `model_runner.kv_cache` is reassigned in two places: the P/D IPC import (`model_runner.py:4472`) and the rollout sleep/wake path (`rollout/memory_manager.py` deletes it, then `_resume_kv_cache` -> `allocate_kv_cache` with a possibly reduced block count). `rows` and `region` survive such a reallocation; `row_stride = stride(0) * item` does not. A same-start-address realloc therefore yields stale bases for every row > 0, and the granularity assertion could not catch it even if the early return had not skipped it -- it never involves `row_stride`. Key on `(data_ptr, shape, stride, element_size)` so any layout change is a cache miss that rebuilds, and argue the case in a comment as the DSV4 sibling (`deepseek_v4_attn.py:1196`, which keys on the same evidence) already does. The base computation now reads the raw address separately (`base_addr`) since the key is a tuple. Co-Authored-By: Claude Opus 4.8 --- .../attentions/page_unit_geometry.py | 23 +++++++++++++++++-- 1 file changed, 21 insertions(+), 2 deletions(-) diff --git a/atom/model_ops/attentions/page_unit_geometry.py b/atom/model_ops/attentions/page_unit_geometry.py index 814eae450b..87da3fca2d 100644 --- a/atom/model_ops/attentions/page_unit_geometry.py +++ b/atom/model_ops/attentions/page_unit_geometry.py @@ -43,7 +43,25 @@ def _page_unit_regions(self) -> tuple[np.ndarray, np.ndarray]: """ runner = self.model_runner cache = runner.kv_cache - owner = cache.data_ptr() + # Key on the full layout, not `data_ptr()` alone. `model_runner.kv_cache` + # is reassigned in two places -- the P/D IPC import + # (`model_runner.py:4472`) and the rollout sleep/wake path + # (`rollout/memory_manager.py` deletes it, then `_resume_kv_cache` -> + # `allocate_kv_cache` with a possibly *reduced* block count). Address + # equality does not imply geometry equality: a same-start-address + # reallocation keeps `data_ptr()` but changes `stride(0)`, so `row_stride` + # (and thus every base for row > 0) goes stale while `rows`/`region` + # survive -- and the granularity assertion below could not catch it even + # if the early return did not skip it, since it never involves + # `row_stride`. Keying on `(data_ptr, shape, stride, element_size)` makes + # any of those changing a cache miss that rebuilds. Mirrors the DSV4 + # sibling (`deepseek_v4_attn.py`), which keys on the same evidence. + owner = ( + cache.data_ptr(), + tuple(cache.shape), + tuple(cache.stride()), + cache.element_size(), + ) cached = getattr(self, "_page_unit_region_cache", None) if cached is not None and cached[0] == owner: return cached[1] @@ -66,8 +84,9 @@ def _page_unit_regions(self) -> tuple[np.ndarray, np.ndarray]: f"logical block {rows} rows x {region} B = {rows * region} B; " "the two disagree about block granularity" ) + base_addr = cache.data_ptr() base = np.array( - [owner + row * row_stride for row in range(rows)], dtype=np.int64 + [base_addr + row * row_stride for row in range(rows)], dtype=np.int64 ) regions = (base, np.full(rows, region, dtype=np.int64)) self._page_unit_region_cache = (owner, regions) From b1b12274daee03fdff9677650c071b67757d287a Mon Sep 17 00:00:00 2001 From: zejchen Date: Tue, 1 Sep 2026 16:33:59 +0000 Subject: [PATCH 110/151] [offload][kimi_k3] gate _start_state_loads on _do_load, matching _arm_joint_loads `_arm_joint_loads` returns early on `not self._do_load`, but `_start_state_loads` did not -- so on a step that does not load (e.g. `kv_role: kv_consumer`) the park was never armed yet the loads were still submitted/failed. The reachable half is a state-only load (KV HBM-resident, state slot evicted): with the tier None, `_fail_state_loads` -> `_joint_park.settle_state(req_id, False)` finds no armed park and returns early at `if need is None`, so nothing is emitted and the request sits in WAITING_FOR_REMOTE_KVS with nothing to wake it (the orphan reclaimer frees the slot but never wakes a live parked request). Gate both halves on the one predicate so the arm and the submit are never split. The gating asymmetry's other half is defence-in-depth already covered by `StateOffloadIndex.can_load` on the producer-role `kv_role` string; this closes the reachable state-only-load hang. Co-Authored-By: Claude Opus 4.8 --- atom/kv_transfer/offload/hybrid/kimi_k3/connector.py | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py b/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py index 2830fc559c..d0f92e8a19 100644 --- a/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py +++ b/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py @@ -314,6 +314,18 @@ def _start_state_loads(self, metadata) -> None: loads = getattr(metadata, "state_loads", None) if not loads: return + if not self._do_load: + # Symmetric with `_arm_joint_loads`' gate. Without it, a step that + # does not load (e.g. `kv_role: kv_consumer`) leaves the park unarmed + # -- `_arm_joint_loads` returned early on the same predicate -- yet + # still submits/fails these loads here. For a state-only load with no + # tier, `_fail_state_loads` -> `settle_state(req_id, False)` finds no + # armed park (`need is None`) and emits nothing, so the request sits + # in WAITING_FOR_REMOTE_KVS with nothing to wake it (the orphan + # reclaimer frees the slot but never wakes a live parked request). + # Gate both halves on the one predicate so neither runs without the + # other. + return if self._state_tier is None: # The engine's index can outlive a tier that refused to build. Fail # them so the requests recompute rather than park forever. From 3cfae8333e6be0bf560e384053699a3d42af51aa Mon Sep 17 00:00:00 2001 From: zejchen Date: Tue, 1 Sep 2026 16:37:50 +0000 Subject: [PATCH 111/151] test: restore the import-time torch stub so it cannot leak across collection `tests/test_lmcache_offload_connector.py` installed a bare `types.ModuleType` under `sys.modules["torch"]` at module scope on a `ModuleNotFoundError` and never removed it. `run_unit_tests.sh` collects all of `tests/` in one pytest process with unpinned import order, so on a runner without torch this empty module becomes *the* `torch` for every module collected afterwards: a later module's `import torch` gets a namespace with no `uint8`/`Tensor`/`cuda` and fails with AttributeError at collection time -- which aborts the whole ~4100-test run rather than skipping one file. (The repo already excludes `tests/plugin` from the unit run for exactly this "module-level stubs leak into and break native tests" reason.) The stub is genuinely needed at import time -- the atom modules imported here do `import torch` at their own module scope -- so `monkeypatch.setitem`, which every other stub in this file uses, is unavailable at import scope. Do its restore manually instead: keep a handle to the stub object, and once the torch-dependent imports have bound their references, delete it iff our exact stub is still installed, restoring the prior absent state. On CI (CPU torch present) line 17 imports real torch and none of this fires. Co-Authored-By: Claude Opus 4.8 --- tests/test_lmcache_offload_connector.py | 25 ++++++++++++++++++++++++- 1 file changed, 24 insertions(+), 1 deletion(-) diff --git a/tests/test_lmcache_offload_connector.py b/tests/test_lmcache_offload_connector.py index 64ed172b6d..0ec04643ad 100644 --- a/tests/test_lmcache_offload_connector.py +++ b/tests/test_lmcache_offload_connector.py @@ -13,10 +13,25 @@ import pytest +_torch_stub = None try: import torch except ModuleNotFoundError: - sys.modules["torch"] = types.ModuleType("torch") + # Import-time only. The atom modules imported below do `import torch` at + # their own module scope, so `torch` has to be resolvable while this file is + # collected; a bare stub satisfies that. The torch-*using* tests here still + # need real torch (CI runs on CPU torch) and error without it -- the stub + # only keeps collection from aborting. Keep a handle so it can be removed the + # moment those imports have bound their references (see below): left in + # place, this empty module becomes *the* `torch` for every module collected + # afterwards, and since `run_unit_tests.sh` collects all of `tests/` in one + # process with unpinned order, a later module's `import torch` then finds a + # namespace with no `uint8`/`Tensor`/`cuda` and fails at collection -- which + # aborts the whole ~4100-test run rather than one file. This is the manual + # equivalent of the `monkeypatch.setitem` restore every other stub in this + # file uses; `monkeypatch` is unavailable at import scope. + _torch_stub = types.ModuleType("torch") + sys.modules["torch"] = _torch_stub from conftest import MockConfig @@ -73,6 +88,14 @@ from atom.model_engine.scheduler import Scheduler from atom.model_engine.sequence import OffloadJointRecord, SequenceStatus +if _torch_stub is not None and sys.modules.get("torch") is _torch_stub: + # The torch-dependent atom imports above have bound their `torch` references, + # so the import-time stub has done its job. Restore the prior absent state -- + # only if our exact stub is still installed -- so a later-collected module + # re-resolves `torch` itself (real on CI, its own guard otherwise) rather + # than inheriting an empty namespace from this file. See the install site. + del sys.modules["torch"] + class _LookupClient: def __init__(self, hit: int) -> None: From ea2cb10190d354943dcac40950ae0da2d0f9b25a Mon Sep 17 00:00:00 2001 From: zejchen Date: Tue, 1 Sep 2026 16:39:04 +0000 Subject: [PATCH 112/151] [offload][kimi_k3] correct the staging_lanes HBM claim and let its metric see contention `staging_lanes` gates concurrency between the load and store lanes, not standing HBM. `StagedTransfer._tls` keys the ~55 MiB staging buffer per thread, both executors are `max_workers=1`, and `release_after_transfer` defaults off -- so standing HBM is two buffers whether the knob is 1 or 2. The comment claimed "1 shares one, halving standing HBM"; setting 1 in fact saves zero HBM and only serialises the lanes (re-introducing the head-of-line coupling the two-executor split exists to remove). At the default 2 the semaphore is never contended. Reword the comment to describe it as the lane-serialisation switch it is, and fix the test docstring + name (`test_one_shared_staging_buffer...` -> `test_one_lane_serialises_the_load_behind_the_inflight_store`) that repeated the false claim while asserting only ordering. Also move the `waited_ms` warning inside the `with self._staging_budget:` acquire. The semaphore acquire is the lane contention the metric exists to surface -- under `staging_lanes=1` a load blocks there for a full gather+D2H behind an in-flight store -- but the warning was computed above the `with`, so it timed only the executor-queue wait and was structurally blind to the acquire. Measured after acquiring, `waited_ms` now spans the lane wait too. Co-Authored-By: Claude Opus 4.8 --- .../offload/hybrid/kimi_k3/state_tier.py | 43 ++++++++++++------- tests/test_lmcache_offload_connector.py | 9 ++-- 2 files changed, 33 insertions(+), 19 deletions(-) diff --git a/atom/kv_transfer/offload/hybrid/kimi_k3/state_tier.py b/atom/kv_transfer/offload/hybrid/kimi_k3/state_tier.py index 1e4c38a045..f873fbe904 100644 --- a/atom/kv_transfer/offload/hybrid/kimi_k3/state_tier.py +++ b/atom/kv_transfer/offload/hybrid/kimi_k3/state_tier.py @@ -44,13 +44,17 @@ def __init__(self, codec, *, max_workers: int = 1, staging_lanes: int = 2) -> No self._store_executor = ThreadPoolExecutor( max_workers=max_workers, thread_name_prefix="lmc-state-store" ) - # What the lanes share is HBM, not a queue. `StagedTransfer` keeps a - # per-thread staging buffer sized to a whole entry (~55 MiB on K3). The - # budget is stated here rather than left implicit in the thread count: - # 2 gives each lane its own buffer (a load never waits on a store); 1 - # shares one, halving standing HBM at the cost of a load waiting out the - # single in-flight store. Either way a load never queues behind a - # *backlog* of stores, which is the actual head-of-line problem. + # `staging_lanes` gates *concurrency* between the load and store lanes, + # not standing HBM. `StagedTransfer._tls = threading.local()` keys the + # ~55 MiB staging buffer per thread, both executors are `max_workers=1`, + # and `release_after_transfer` defaults off -- so standing HBM is exactly + # two buffers (one per executor thread) whether this is 1 or 2. The knob + # only decides whether a load and a store may run at once: + # 2 (default): they overlap; the semaphore is never contended. + # 1: a load waits out any in-flight store, re-introducing the + # head-of-line coupling the two-executor split exists to remove, + # and saves zero HBM. + # Do not read this as an HBM lever; it is a lane-serialisation switch. self._staging_budget = threading.BoundedSemaphore(max(1, int(staging_lanes))) self._lock = threading.Lock() self._done: set[str] = set() @@ -240,18 +244,25 @@ def _do_load(self, req_id: str, h: int, slot: int, submitted_at=None) -> None: # under a hash the engine's index still advertises. Retracting that # claim is the engine's job -- it owns the index -- and it does it from # the report below. - if submitted_at is not None: - waited_ms = max(0.0, monotonic() - submitted_at) * 1000.0 - if waited_ms >= _LOAD_WAIT_WARN_MS: - logger.warning( - "state offload: a state load waited %.0fms for its lane " - "(oldest store outstanding %.1fs). This is TTFT.", - waited_ms, - self.oldest_store_age_s(), - ) ok = False try: with self._staging_budget: + # Measure the wait *after* acquiring the lane, not before. The + # semaphore acquire is the lane contention this metric exists to + # surface: under `staging_lanes=1` a load blocks here for a full + # ~55 MiB gather+D2H behind an in-flight store. Sampling + # `submitted_at` above the `with` timed only the executor-queue + # wait and was structurally blind to the acquire -- the one thing + # it was added to see. Now `waited_ms` spans both. + if submitted_at is not None: + waited_ms = max(0.0, monotonic() - submitted_at) * 1000.0 + if waited_ms >= _LOAD_WAIT_WARN_MS: + logger.warning( + "state offload: a state load waited %.0fms for its " + "lane (oldest store outstanding %.1fs). This is TTFT.", + waited_ms, + self.oldest_store_age_s(), + ) ok = bool(self.codec.get(h, slot)) except Exception: # a failed load is a normal path # Same reasoning as `_do_store`, and here a miss is expected: diff --git a/tests/test_lmcache_offload_connector.py b/tests/test_lmcache_offload_connector.py index 0ec04643ad..dbd0ef1d81 100644 --- a/tests/test_lmcache_offload_connector.py +++ b/tests/test_lmcache_offload_connector.py @@ -5898,9 +5898,12 @@ def test_a_load_lands_while_a_store_is_still_stuck(self): codec.gate.set() tier.shutdown() - def test_one_shared_staging_buffer_still_separates_the_queues(self): - """`staging_lanes=1` halves the standing HBM: a load then waits out the - single in-flight store, but still not the backlog behind it.""" + def test_one_lane_serialises_the_load_behind_the_inflight_store(self): + """`staging_lanes=1` serialises the two lanes: a load waits out the + single in-flight store, but still not the backlog behind it. It does + *not* change standing HBM -- the staging buffer is per-thread and both + executors are `max_workers=1`, so HBM is two buffers either way; this + knob only gates load/store concurrency, which is what this asserts.""" codec = self._BlockingCodec() tier = self._tier(codec, staging_lanes=1) try: From e63113c7db6374459eaaf3f974dd6800ac60f280 Mon Sep 17 00:00:00 2001 From: zejchen Date: Tue, 1 Sep 2026 16:44:01 +0000 Subject: [PATCH 113/151] [offload][kimi_k3] derive the save-stall clock from the abandon window so their ordering cannot invert The K3 connector's save-stall escape and the scheduler's deferred-save reconciler are complements on a single deferred save: the connector releases the blocks of a save the backend never took, and `Scheduler._reconcile_stalled_deferred_saves` reclaims a save already handed out once its report stops coming. The scheduler docstring asserts the connector fires "on a shorter clock" than the abandon window -- an invariant that keeps the connector from declaring a save dead while the scheduler still considers it live. A hardcoded `SAVE_STALL_SECONDS = 120.0` broke that invariant silently the moment `LMCACHE_EC_PIN_TIMEOUT_SEC` moved the abandon window (`offload_save_abandon_timeout_s()` = pin + margin) below 120: pin=60 gives an abandon window of 90 < 120, inverting the order with nothing to detect it. Derive both clocks from the one source. `save_stall_seconds()` fires at LMCache's pin timeout itself (abandon minus the margin -- the point upstream has force-unpinned the source) but never above the 120 s default, so the ordering holds for every pin value and the default behaviour is unchanged. When reclamation is disabled (abandon <= 0) there is no window to stay under, so the standalone default is used. Updates the scheduler docstring cross-reference and the unit test's import and call site to the derived function. Co-Authored-By: Claude Opus 4.8 --- .../offload/hybrid/kimi_k3/connector.py | 36 ++++++++++++++++--- atom/model_engine/scheduler.py | 2 +- tests/test_lmcache_offload_connector.py | 4 +-- 3 files changed, 35 insertions(+), 7 deletions(-) diff --git a/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py b/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py index d0f92e8a19..d5fa57c33e 100644 --- a/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py +++ b/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py @@ -20,8 +20,10 @@ ) from atom.kv_transfer.offload import config as offcfg from atom.kv_transfer.offload._offload_common import ( + _SAVE_ABANDON_MARGIN_S, StateOffloadFace, max_pending_saves, + offload_save_abandon_timeout_s, pp_aware_rank_and_world, ) from atom.kv_transfer.offload.dense.connector import ( @@ -58,9 +60,35 @@ #: rather than leaving a partially-reported key pending forever. STATE_LOAD_DISPOSITION_CHANNEL = "k3_state_load_disposition" -#: A save outstanding longer than this is a backend that stopped, not one that -#: is busy: a 4096-token store costs ~65ms. -SAVE_STALL_SECONDS = 120.0 +#: The connector's standalone stall clock, used only when reclamation is +#: disabled (`LMCACHE_EC_PIN_TIMEOUT_SEC <= 0`) so there is no abandon window to +#: stay under. A save outstanding longer than this is a backend that stopped, +#: not one that is busy: a 4096-token store costs ~65ms. +_SAVE_STALL_DEFAULT_S = 120.0 + + +def save_stall_seconds() -> float: + """Seconds a save may sit before this connector calls the path stalled. + + Kept strictly under the scheduler's abandon window by construction. The two + are complements on one deferred save: this connector releases the blocks of + a save the backend never took, and `Scheduler._reconcile_stalled_deferred_ + saves` reclaims a save already handed out once its report is not coming -- + the scheduler docstring asserts this connector fires "on a shorter clock". + A hardcoded 120 s broke that the moment `LMCACHE_EC_PIN_TIMEOUT_SEC` put the + abandon window (`offload_save_abandon_timeout_s()` = pin + margin) below 120 + -- e.g. pin=60 gives abandon 90 < 120, inverting the order with nothing to + detect it. Derive both from the one source instead: fire at LMCache's pin + timeout itself (abandon minus the margin -- the point upstream has + force-unpinned the source), but never above the 120 s default, so the + ordering holds for every pin value and the default behaviour is unchanged. + When reclamation is disabled (abandon <= 0) there is no window to stay + under, so use the default. + """ + abandon = offload_save_abandon_timeout_s() + if abandon <= 0: + return _SAVE_STALL_DEFAULT_S + return min(_SAVE_STALL_DEFAULT_S, abandon - _SAVE_ABANDON_MARGIN_S) class KimiK3OffloadConnector(DenseOffloadConnector): @@ -609,7 +637,7 @@ def _refresh_save_stall(self) -> None: self._warned_save_stalled = False return oldest = min(self._save_inflight_since.values()) - self._save_stalled = (now - oldest) > SAVE_STALL_SECONDS + self._save_stalled = (now - oldest) > save_stall_seconds() if self._save_stalled and not self._warned_save_stalled: self._warned_save_stalled = True logger.warning( diff --git a/atom/model_engine/scheduler.py b/atom/model_engine/scheduler.py index 06d98aba03..eb5a303e7f 100644 --- a/atom/model_engine/scheduler.py +++ b/atom/model_engine/scheduler.py @@ -934,7 +934,7 @@ def _reconcile_stalled_deferred_saves(self) -> int: that drop. The complement of the K3 connector's stall escape - (`kimi_k3.connector.SAVE_STALL_SECONDS`), not a duplicate of it: that + (`kimi_k3.connector.save_stall_seconds()`), not a duplicate of it: that one releases the blocks of a save the backend never took, on a shorter clock, and leaves a save already handed out alone -- precisely the case this reclaims once the report is not coming. Between them every deferred diff --git a/tests/test_lmcache_offload_connector.py b/tests/test_lmcache_offload_connector.py index dbd0ef1d81..c788d01545 100644 --- a/tests/test_lmcache_offload_connector.py +++ b/tests/test_lmcache_offload_connector.py @@ -71,10 +71,10 @@ _chained_prefix_hashes, ) from atom.kv_transfer.offload.hybrid.kimi_k3.connector import ( - SAVE_STALL_SECONDS, STATE_INDEX_CHANNEL, KimiK3OffloadConnector, KimiK3OffloadScheduler, + save_stall_seconds, ) from atom.kv_transfer.offload.hybrid.kimi_k3.state_tier import _JointPark from atom.kv_transfer.offload.metadata import ( @@ -4938,7 +4938,7 @@ def test_a_stalled_save_releases_blocks_it_never_handed_out(monkeypatch): # Age the outstanding save rather than the process clock: `time.monotonic` # is the stdlib's, and patching it reaches every other test in the run. - s._save_inflight_since["9"] -= 2 * SAVE_STALL_SECONDS + s._save_inflight_since["9"] -= 2 * save_stall_seconds() s._refresh_save_stall() assert s._save_stalled is True # Never handed out -> the blocks may go free. `should_defer_free` is a pure From e147175c7676ecc96debaae13ad3618332631fc2 Mon Sep 17 00:00:00 2001 From: zejchen Date: Tue, 1 Sep 2026 16:47:22 +0000 Subject: [PATCH 114/151] [docs] correct the offload env-var documentation to match the code that reads it Two docs-match-code drifts in the offload env surface, both flagged in review: envs.py: the central registry comment for LMCACHE_EC_PIN_TIMEOUT_SEC said it is "read in scheduler.py only". Repo-wide the only os.environ.get for that name is in _offload_common.offload_save_abandon_timeout_s(); the scheduler never reads the env, reaching the value only through the connector's save_abandon_timeout_s. The same PR's _offload_common comment already says this plainly ("this lives on the offload connector, not the scheduler"), so the registry -- the file people grep first -- was the one contradicting it. Point the comment at the real reader. environment_variables.md: the OFFLOAD_MAX_PENDING_SAVES entry listed only the KV leg's canonical default (max(2, 2 x OFFLOAD_COPY_WORKERS), raises on a bad value) and omitted the scheduler's separate state-tier reader (_offload_max_pending_saves), which has a simpler fallback -- a flat default of 2 with no worker scaling, and warns-then-uses-2 rather than raising on an unparseable value. Document the second reader so the two paths' differing default and error behaviour are visible. Co-Authored-By: Claude Opus 4.8 --- atom/utils/envs.py | 7 +++++-- docs/environment_variables.md | 2 +- 2 files changed, 6 insertions(+), 3 deletions(-) diff --git a/atom/utils/envs.py b/atom/utils/envs.py index 4a77ecdb3e..0a319683c7 100644 --- a/atom/utils/envs.py +++ b/atom/utils/envs.py @@ -624,9 +624,12 @@ def __getattr__(name: str): # VLLM_PP_LAYER_PARTITION — vLLM legacy (still active in models/utils.py) # VLLM_USE_MODELSCOPE — vLLM legacy (benchmarks) # LMCACHE_EC_PIN_TIMEOUT_SEC — LMCache library's own source-pin timeout; -# read in scheduler.py only to derive the +# read in kv_transfer/offload/_offload_common.py +# (offload_save_abandon_timeout_s) to derive the # engine's save-abandon window from it, so the -# two stay ordered. ATOM does not own the +# two stay ordered. The scheduler never reads the +# env itself -- it asks the connector, via +# save_abandon_timeout_s. ATOM does not own the # knob, hence no default of its own here. # OFFLOAD_MAX_PENDING_SAVES — offload connector queue-depth bound; # defined/defaulted in diff --git a/docs/environment_variables.md b/docs/environment_variables.md index abe5b44b20..8667949372 100644 --- a/docs/environment_variables.md +++ b/docs/environment_variables.md @@ -196,7 +196,7 @@ discoverable from the central env reference despite bypassing the registry. | Variable | Type | Default | Description | |----------|------|---------|-------------| | **LMCACHE_EC_PIN_TIMEOUT_SEC** | float | LMCache's own (300) | LMCache's source-pin timeout. ATOM reads it only to derive the engine's save-abandon window (`pin + 30s`), so the two stay ordered — a lost store report is reclaimed only after LMCache would already have force-unpinned its source. Non-positive disables ATOM's reclamation. ATOM sets no default of its own; when unset it assumes LMCache's. | -| **OFFLOAD_MAX_PENDING_SAVES** | int | `max(2, 2 × OFFLOAD_COPY_WORKERS)` | Bound on total in-flight offload transfers (running + queued) held before a SLOT snapshot or executor submission. A KV save and a state store both pin bytes out of the same pool while they run, so the KV leg and the K3 state tier share this one number rather than each carrying its own. | +| **OFFLOAD_MAX_PENDING_SAVES** | int | `max(2, 2 × OFFLOAD_COPY_WORKERS)`; **2** for the state tier's own reader | Bound on total in-flight offload transfers (running + queued) held before a SLOT snapshot or executor submission. A KV save and a state store both pin bytes out of the same pool while they run, so the KV leg and the K3 state tier share this one number rather than each carrying its own. Two readers compute it, though: the KV leg's canonical `_offload_common.max_pending_saves` derives the shown default from `OFFLOAD_COPY_WORKERS` and **raises** on an unparseable value, while the scheduler's state-tier reader (`_offload_max_pending_saves`) has a simpler fallback — a flat default of **2** (no `OFFLOAD_COPY_WORKERS` scaling) that **warns and uses 2** on an unparseable value rather than raising. Set the env to an explicit integer to pin both. | ## Profiling & debugging From 9d46949951407fc6ee7c54955841a869ff243d63 Mon Sep 17 00:00:00 2001 From: zejchen Date: Tue, 1 Sep 2026 16:47:30 +0000 Subject: [PATCH 115/151] [offload] fix OffloadJointRecord docstring/init comment: six fields, not seven OffloadJointRecord carries six fields (load_hash, kv_prefix_tokens, boundary_tokens, boundary_hash, kv_tokens, claim_tokens), but both its own docstring and the Sequence.__init__ comment that constructs it said "seven values". Correct both to six so the count matches the dataclass. Co-Authored-By: Claude Opus 4.8 --- atom/model_engine/sequence.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/atom/model_engine/sequence.py b/atom/model_engine/sequence.py index 2b46ebbeb7..ffa958bf6d 100644 --- a/atom/model_engine/sequence.py +++ b/atom/model_engine/sequence.py @@ -72,7 +72,7 @@ def get_exit_sequence(): class OffloadJointRecord: """The KV-transfer offload/joint-load protocol state for one sequence. - These seven values were seven flat attributes on `Sequence`, written from + These six values were six flat attributes on `Sequence`, written from `BlockManager` and `Scheduler` and read back by the offload connector, with the joint subset re-initialised by hand in several places -- so a reset that forgot one field left a stale span the connector would then act on. Grouped @@ -263,7 +263,7 @@ def __init__( # forward. -1 = read and write the same slot, the case for every step in # between. Always a single slot: a checkpoint is one slot wide. self.state_fork_src = -1 - # The KV-transfer offload/joint-load protocol state -- seven values that + # The KV-transfer offload/joint-load protocol state -- six values that # move together through admission and load. Grouped so a joint reset # cannot forget a field; see `OffloadJointRecord`. self.offload_joint = OffloadJointRecord() From f6c4130e8601168c7e3b9f0e56393151e0b0e3ca Mon Sep 17 00:00:00 2001 From: zejchen Date: Tue, 1 Sep 2026 16:47:30 +0000 Subject: [PATCH 116/151] [config] drop the "V4 compressor ring" from the per_request_state comment; no V4 path sets it The per_request_state field comment cited "(GDN/KDA recurrent state, the V4 compressor ring)" as the tensors it flags, but no DeepSeek-V4 path sets per_request_state=True -- the only setters are the GDN and KDA linear-attn forwards (gdn_attn.py, kimi_mla_gdn_attn.py, and the rtpllm/sglang plugin twins). Remove the wrong V4 reference and name the actual setters so the comment matches who writes the flag. Co-Authored-By: Claude Opus 4.8 --- atom/config.py | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/atom/config.py b/atom/config.py index 40fd4ff918..40a464e2b5 100644 --- a/atom/config.py +++ b/atom/config.py @@ -55,7 +55,9 @@ class KVCacheTensor: replay_buf_u: torch.Tensor = None replay_buf_g: torch.Tensor = None # True when the tensors above are a hybrid's PER-REQUEST state (GDN/KDA - # recurrent state, the V4 compressor ring) rather than paged KV. They belong + # recurrent state) rather than paged KV. Only the GDN and KDA linear-attn + # forwards set it (`gdn_attn.py`, `kimi_mla_gdn_attn.py`, and the rtpllm/ + # sglang plugin twins); no DeepSeek-V4 path does. They belong # in ``kv_cache_data`` -- the linear-attention forward reads them from there # -- but are addressed by request slot, so no block-addressed mover may # touch them. The dense codec skips them; the state tier reaches the same From 47584788ea88789d5e1c8b91e2ab68adf909fc4e Mon Sep 17 00:00:00 2001 From: zejchen Date: Tue, 1 Sep 2026 16:50:53 +0000 Subject: [PATCH 117/151] [offload] MiniMax-M3 is sparse paged attention, not GDN: correct the four texts that disagree Review raised that four texts disagree about MiniMax-M3, and asked for an answer rather than a blind edit. Traced it to ground truth in the model: - `MiniMaxM3SparseForCausalLM` (model_runner registry) -> `MiniMaxM3SparseAttention` -> `SparseMHAPagedAttentionImpl` (attention_mha.py): lightning-*indexer* sparse attention over PAGED KV. - The M3 model file has zero mamba / GDN / conv_state / recurrent / per_request_state markers; the only "lightning" is the sparse indexer, not linear-attention recurrent state. - The `per_request_state=True` at sglang attention_gdn.py:436 is the GDN mamba backend (`mamba_map` / `conv` / `temporal`) for Qwen3-Next/Qwen3.5, not an M3 path. So M3 carries no per-request recurrent state; it is safe on the dense offload path as ordinary paged KV, and must NOT be refused by the layout guard. That makes `_GDN_LINEAR_MODEL_TYPES` (which omits M3) and its "M2/M3 absent by design" comment the correct texts, and the three offload guard/codec texts that list "MiniMax-M3" among "GDN: Qwen3-Next, MiniMax-M3, Qwen3.5" the wrong ones. Drop M3 from all three (dense/connector.py comment, kv_byte_codec.py docstring and ValueError text) -- comment/message-only, no behaviour change. Also document the second half of the finding in the `_GDN_LINEAR_MODEL_TYPES` comment: the set matches `is_qwen_next()` but the guard reads `hf_config.model_type` where is_qwen_next reads `hf_text_config.model_type`. They agree only because `get_hf_config` flattens the text type up on the native path; `is_vllm()` drops qwen3_5/qwen3_5_moe from that flatten, so the guard must move to the text config if offload is ever wired into plugin mode. Left as a documented caveat rather than a runtime change because the plugin path is unreachable today (no plugin caller passes a kv_transfer_config). Co-Authored-By: Claude Opus 4.8 --- atom/kv_transfer/offload/config.py | 13 +++++++++++-- atom/kv_transfer/offload/dense/connector.py | 2 +- atom/kv_transfer/offload/dense/kv_byte_codec.py | 6 +++--- 3 files changed, 15 insertions(+), 6 deletions(-) diff --git a/atom/kv_transfer/offload/config.py b/atom/kv_transfer/offload/config.py index 8f8ece1319..394d395a10 100644 --- a/atom/kv_transfer/offload/config.py +++ b/atom/kv_transfer/offload/config.py @@ -56,8 +56,17 @@ "indexer_dtype", } # GDN/linear model types carrying a per-request recurrent state but NOT in the -# `kimi_linear` family `kimi_k3` owns. Mirrors `ModelRunner.is_qwen_next()`. -# MiniMax-M2/M3 are absent by design (sparse/standard attention, no state). +# `kimi_linear` family `kimi_k3` owns. Same set as `ModelRunner.is_qwen_next()`, +# but note the attribute skew: the guard below reads `hf_config.model_type` +# while `is_qwen_next` reads `hf_text_config.model_type`. On the native path +# they agree only because `get_hf_config` flattens the text type up; `is_vllm()` +# drops `qwen3_5`/`qwen3_5_moe` from that flatten, so if offload is ever wired +# into plugin mode this guard must switch to the text config to stay in sync. +# Unreachable today -- no plugin caller passes a `kv_transfer_config`. +# MiniMax-M2/M3 are absent by design: M3 is sparse *paged* attention +# (`MiniMaxM3SparseForCausalLM` -> `SparseMHAPagedAttentionImpl`), no recurrent +# state, so the dense offload path handles it as ordinary paged KV -- it is NOT +# a GDN/linear model despite some sibling comments once listing it as one. _GDN_LINEAR_MODEL_TYPES = frozenset( {"qwen3_next", "qwen3_next_mtp", "qwen3_5_text", "qwen3_5_moe_text"} ) diff --git a/atom/kv_transfer/offload/dense/connector.py b/atom/kv_transfer/offload/dense/connector.py index e797550b66..e6d7d5e183 100644 --- a/atom/kv_transfer/offload/dense/connector.py +++ b/atom/kv_transfer/offload/dense/connector.py @@ -72,7 +72,7 @@ class DenseOffloadConnector(OffloadWorkerMixin, KVConnectorBase): # Whether a per-request recurrent-state tensor in the registered kv_caches is # tolerated. The plain dense path has no rule keeping a restored KV prefix # aligned with linear-attention state, so it must reject such a model - # (GDN: Qwen3-Next, MiniMax-M3, Qwen3.5) and fail fast. A hybrid connector + # (GDN: Qwen3-Next, Qwen3.5) and fail fast. A hybrid connector # that owns a state tier (kimi_k3) overrides this to True. _permit_per_request_state = False diff --git a/atom/kv_transfer/offload/dense/kv_byte_codec.py b/atom/kv_transfer/offload/dense/kv_byte_codec.py index b0220fb921..2e311c6feb 100644 --- a/atom/kv_transfer/offload/dense/kv_byte_codec.py +++ b/atom/kv_transfer/offload/dense/kv_byte_codec.py @@ -72,8 +72,8 @@ def __init__( ``permit_per_request_state`` gates whether a per-request recurrent-state tensor in ``kv_caches`` is skipped or rejected. A hybrid connector that moves state through its own tier (kimi_k3) sets it True to skip. The - plain dense path leaves it False: a GDN model (Qwen3-Next, MiniMax-M3, - Qwen3.5) registered on ``lmcache_offload`` puts its slot-indexed mamba + plain dense path leaves it False: a GDN model (Qwen3-Next, Qwen3.5) + registered on ``lmcache_offload`` puts its slot-indexed mamba caches in this dict, and the dense path has no rule keeping its KV prefix aligned with that linear state -- restoring KV while the state is stale is silent wrong output. Skipping the tensor here used to hide that @@ -95,7 +95,7 @@ def __init__( "DenseKVByteCodec: per-request recurrent state was " "registered on the plain dense offload path. This is a " "GDN/linear-attention model (e.g. Qwen3-Next, " - "MiniMax-M3, Qwen3.5); the dense path would restore its " + "Qwen3.5); the dense path would restore its " "KV prefix while the recurrent state stayed stale -- " "silent wrong output. Use a connector that owns a state " "tier (kimi_k3), which passes " From 27fe4a8ffd4ffebad2b560fa3eebd715067b1715 Mon Sep 17 00:00:00 2001 From: zejchen Date: Wed, 2 Sep 2026 01:11:25 +0000 Subject: [PATCH 118/151] [offload] kimi_k3 staging: guard _handoff/_finish against the non-CUDA None event `_StagingBuffer.ready_event` and `_ThreadTransferState.pack_stream`/ `copy_stream` are created iff `use_cuda` (atom_lmcache_staging.py:59,107) -- they are None on a non-CUDA device. `_handoff` did `state.staging_buffer.ready_event.record(producer)` and `_finish` did `producer.synchronize()` unconditionally, so both dereference None off the GPU. `pack`'s general path never reaches them off-GPU: its gather `fused_pack_chunk_major` (a Triton kernel) raises first. But `unpack`'s general path does a plain `device_buf.copy_(src_tensor)` H2D leg that succeeds on CPU, then calls `_handoff` at staging.py:270 -- so on a non-CUDA device the None-deref fires there and masks the meaningful `fused_unpack_chunk_major` (Triton needs CUDA) error that would otherwise follow. Guard both helpers to no-op on the non-CUDA state (`ready_event is None` / `producer is None`), which is the exact non-CUDA test since the event and streams are allocated together under `use_cuda`. Off-GPU every copy on this path already ran synchronously on the calling thread, so there is nothing to fence or drain; the guard just lets the honest Triton error surface instead of an opaque `NoneType has no attribute 'record'`. Co-Authored-By: Claude Opus 4.8 --- .../offload/hybrid/kimi_k3/staging.py | 22 +++++++++++++++++-- 1 file changed, 20 insertions(+), 2 deletions(-) diff --git a/atom/kv_transfer/offload/hybrid/kimi_k3/staging.py b/atom/kv_transfer/offload/hybrid/kimi_k3/staging.py index 705174768f..7dc3bb3437 100644 --- a/atom/kv_transfer/offload/hybrid/kimi_k3/staging.py +++ b/atom/kv_transfer/offload/hybrid/kimi_k3/staging.py @@ -284,7 +284,19 @@ def unpack(self, src: Any, segments: list[torch.Tensor]) -> None: @staticmethod def _handoff(state: _ThreadTransferState, producer, consumer) -> None: - """Record the producer event and make the consumer stream wait on it.""" + """Record the producer event and make the consumer stream wait on it. + + No-op on a non-CUDA device: `_ThreadTransferState` leaves both streams + and `_StagingBuffer.ready_event` at None there (see + `atom_lmcache_staging.py`), and every copy on this path already ran + synchronously on the calling thread, so there is nothing to fence. The + event is created iff `use_cuda`, so `ready_event is None` is the exact + non-CUDA test. Guarding here keeps `unpack`'s general path -- whose H2D + leg is a plain `copy_` that succeeds on CPU -- from dereferencing that + None and masking the meaningful `fused_unpack_chunk_major` (Triton needs + CUDA) error that follows.""" + if state.staging_buffer.ready_event is None: + return state.staging_buffer.ready_event.record(producer) consumer.wait_event(state.staging_buffer.ready_event) @@ -296,5 +308,11 @@ def _finish(state: _ThreadTransferState, producer) -> None: Unlike `run_staged_pipeline`, this path does not hand a `free_event` to a later consumer -- `producer.synchronize()` is the whole fence -- so it deliberately leaves the shared buffer's `free_event`/ - `free_event_valid` untouched.""" + `free_event_valid` untouched. + + No-op when `producer` is None (non-CUDA: `_ThreadTransferState` has no + streams), matching `_handoff` -- the copies already completed + synchronously, so there is nothing to drain.""" + if producer is None: + return producer.synchronize() From 774765cfd2c57e67fa0e3b052e5e2634e49a5aa6 Mon Sep 17 00:00:00 2001 From: zejchen Date: Wed, 2 Sep 2026 01:12:31 +0000 Subject: [PATCH 119/151] [offload] _JointPark.arm: evict stale prior-park bookkeeping on re-arm `arm` filed the new park but never cleared what a prior park for the same `req_id` left behind. A preempt/requeue re-arms the request, usually under a fresh `kv_id`, so the old key keeps its `_need` entry and its reverse `_alias_of[old_key] -> req_id`. A stray late `settle` for that old key then `_release(old_key)`, whose `_alias_of` -> `_alias` cleanup pops `_alias[req_id]` -- the live mapping the re-armed park installed. The re-armed park's state leg reports under the bare `req_id`, which now `_resolve`s to itself instead of the new `kv_id`, finds no `_need`, and is silently dropped: the park never unparks and rides out the abandon window. Add `_purge(key)`, an eviction (not a settle) that drops a key from every park structure -- `_need`, `_alias`/`_alias_of`, `_failed`, `_state_failed`, `_ready`, `_ready_failed`, `_dispositions` -- without emitting a ready/failed/disposition, so an evicted load can never masquerade as a completed one. `arm` calls it for both the previously-aliased key and the bare `req_id` (the no-`kv_id` prior-arm case) before installing the new mapping. No-op on None and on an unknown key, so a first-time arm is unchanged. Co-Authored-By: Claude Opus 4.8 --- .../offload/hybrid/kimi_k3/state_tier.py | 31 +++++++++++++++++++ 1 file changed, 31 insertions(+) diff --git a/atom/kv_transfer/offload/hybrid/kimi_k3/state_tier.py b/atom/kv_transfer/offload/hybrid/kimi_k3/state_tier.py index f873fbe904..e27541ec4b 100644 --- a/atom/kv_transfer/offload/hybrid/kimi_k3/state_tier.py +++ b/atom/kv_transfer/offload/hybrid/kimi_k3/state_tier.py @@ -328,6 +328,18 @@ def arm( `finished_loading`/`failed_loading` for this load, because that report is matched by equality and nothing translates it on the way in. """ + # Evict any bookkeeping a prior park for this same `req_id` left behind + # before filing the new one. A preempt/requeue re-arms the request, + # usually under a fresh `kv_id`; without this the old key's `_need` + # entry and its reverse `_alias_of[old_key] -> req_id` survive. A stray + # late `settle` for the old key would then `_release(old_key)`, whose + # `_alias_of` -> `_alias` cleanup pops `_alias[req_id]` -- the live + # mapping the re-armed park's state leg needs -- so that park's state + # report resolves to the bare id, finds no `_need`, and never unparks. + # `_purge` drops the stale key everywhere without emitting a + # ready/failed/disposition (an eviction, not a settled outcome). + self._purge(self._alias.pop(req_id, None)) + self._purge(req_id) need = set() if needs_kv: need.add("kv") @@ -341,6 +353,25 @@ def arm( if not need: self._release(key) + def _purge(self, key) -> None: + """Drop a key from every park structure without settling it. + + Used only by re-arm to evict a stale prior park; unlike `_release` it + produces no `_ready`/`_ready_failed`/`_dispositions` entry, so an evicted + load never masquerades as a completed one. No-op on None (nothing was + aliased) and on an unknown key.""" + if key is None or key not in self._need: + return + self._need.pop(key, None) + bare = self._alias_of.pop(key, None) + if bare is not None: + self._alias.pop(bare, None) + self._failed.discard(key) + self._state_failed.discard(key) + self._ready.discard(key) + self._ready_failed.discard(key) + self._dispositions.pop(key, None) + def settle_kv(self, ident, ok: bool) -> None: self._settle(ident, "kv", ok) From 8093ac3f3da0f845a1bb0c7d0136deb06a2fb4da Mon Sep 17 00:00:00 2001 From: zejchen Date: Wed, 2 Sep 2026 01:15:27 +0000 Subject: [PATCH 120/151] [offload] PP: aggregate connector_completions across stages instead of dropping them The head's `_poll_kv_transfer_progress` split the TP-aggregated `kvoutput` into a mooncake `non_offload` slice (recv) and an `offload_local` slice (finished/failed_loading, finished_saving) but never carried `connector_completions`. Off the PP path (engine_core.py) the whole `kvoutput` -- completions included -- reaches `_update_from_kv_xfer_finished` and thus the connector's `process_completions`; under PP those events were silently dropped. That strands every connector-owned channel: kimi_k3's state-index dispositions (finding #6: abandon-vs-forget verdict) and dsv4's checkpoint-save boundaries never reach the scheduler, so a joint load's disposition is lost and a checkpoint boundary never commits under pipeline parallel. connector_completions are per-stage the same way saves are -- each PP stage holds a slice of the layers a checkpoint/state boundary spans -- so they need the same all-stage quorum with failure-dominance, not a direct hand to the scheduler. Extend PPKVAggregator to track them by `(channel, operation_id)` key: a key is terminal once every stage has reported it, and its verdict is the AND across stages (any stage's `succeeded=False` sinks the boundary), mirroring the TP aggregator's failure-dominant rule one level up. Feed the head's own completions in via `offload_local`, count them in `has_offload` so the poll does not early-return and strand them, and include them in `has_pending` so the head keeps polling downstream until quorum. The downstream transport already pickles the full KVConnectorOutput, so stage reports carried the completions over the wire already -- only the head-side aggregation was missing. Co-Authored-By: Claude Opus 4.8 --- .../disaggregation/pp_kv_aggregator.py | 56 +++++++++++++++++-- atom/model_engine/pp_engine_core.py | 7 +++ 2 files changed, 57 insertions(+), 6 deletions(-) diff --git a/atom/kv_transfer/disaggregation/pp_kv_aggregator.py b/atom/kv_transfer/disaggregation/pp_kv_aggregator.py index fe126c24ff..deb551bc88 100644 --- a/atom/kv_transfer/disaggregation/pp_kv_aggregator.py +++ b/atom/kv_transfer/disaggregation/pp_kv_aggregator.py @@ -10,19 +10,28 @@ from __future__ import annotations -from atom.kv_transfer.disaggregation.types import KVConnectorOutput, ReqId +from atom.kv_transfer.disaggregation.types import ( + ConnectorCompletion, + ConnectorCompletionKey, + KVConnectorOutput, + ReqId, +) class PPKVAggregator: """Aggregate offload ``finished_loading / failed_loading / finished_saving`` - across PP stages. + and ``connector_completions`` across PP stages. Call :meth:`ingest` once per (pp_rank, output) pair. The method returns a - :class:`KVConnectorOutput` containing only the request IDs that have - reached a terminal state across all stages. + :class:`KVConnectorOutput` containing only the request IDs (and connector + completion keys) that have reached a terminal state across all stages. A load fails the request if any stage reports a failure, but the failure is - only emitted once every stage has reported either success or failure. + only emitted once every stage has reported either success or failure. A + connector completion is failure-dominant the same way: each stage holds a + slice of the layers a checkpoint/state boundary spans, so the boundary + commits (``succeeded=True``) only if every stage succeeded, and any stage's + failure sinks it -- but, like a load, only once every stage has reported. Only offload-specific fields are tracked. Mooncake P/D fields (``finished_sending``, ``finished_recving``) have their own PP-aware @@ -36,6 +45,13 @@ def __init__(self, pp_size: int) -> None: self._loading: dict[ReqId, set[int]] = {} self._saving: dict[ReqId, set[int]] = {} self._failed_loading: dict[ReqId, set[int]] = {} + # Per connector-completion key: which stages have reported it, and + # whether any stage reported it failed. The head's own TP aggregation + # already collapsed each stage's TP ranks to one succeeded/failed + # verdict per key (aggregator.py), so a stage reports a key at most once + # per generation and PP quorum is just stage coverage. + self._connector: dict[ConnectorCompletionKey, set[int]] = {} + self._connector_failed: set[ConnectorCompletionKey] = set() def ingest(self, pp_rank: int, output: KVConnectorOutput) -> KVConnectorOutput: for rid in output.finished_loading: @@ -44,6 +60,10 @@ def ingest(self, pp_rank: int, output: KVConnectorOutput) -> KVConnectorOutput: self._failed_loading.setdefault(rid, set()).add(pp_rank) for rid in output.finished_saving: self._saving.setdefault(rid, set()).add(pp_rank) + for completion in output.connector_completions: + self._connector.setdefault(completion.key, set()).add(pp_rank) + if not completion.succeeded: + self._connector_failed.add(completion.key) # A load is only terminal once every stage has reported one way or the # other. Reporting the failure at the first failing stage would wake @@ -61,17 +81,37 @@ def ingest(self, pp_rank: int, output: KVConnectorOutput) -> KVConnectorOutput: done_saving = { rid for rid, stages in self._saving.items() if len(stages) >= self._pp_size } + # A connector completion is terminal once every stage has reported it + # (same all-stage quorum as a save); its verdict is the AND across + # stages -- failed if any stage failed, else succeeded. + done_connector = { + key + for key, stages in self._connector.items() + if len(stages) >= self._pp_size + } + connector_completions = { + ConnectorCompletion( + channel=key[0], + operation_id=key[1], + succeeded=key not in self._connector_failed, + ) + for key in done_connector + } for rid in done_loading | failed: self._loading.pop(rid, None) self._failed_loading.pop(rid, None) for rid in done_saving: self._saving.pop(rid, None) + for key in done_connector: + self._connector.pop(key, None) + self._connector_failed.discard(key) return KVConnectorOutput( finished_loading=done_loading, failed_loading=failed, finished_saving=done_saving, + connector_completions=connector_completions, ) def has_pending(self) -> bool: @@ -80,9 +120,13 @@ def has_pending(self) -> bool: The head's busy loop keeps polling downstream stages while this holds; the tallies only drain when the missing stages report in. """ - return bool(self._loading or self._saving or self._failed_loading) + return bool( + self._loading or self._saving or self._failed_loading or self._connector + ) def reset(self) -> None: self._loading.clear() self._saving.clear() self._failed_loading.clear() + self._connector.clear() + self._connector_failed.clear() diff --git a/atom/model_engine/pp_engine_core.py b/atom/model_engine/pp_engine_core.py index 5959277c18..c0be34ddbb 100644 --- a/atom/model_engine/pp_engine_core.py +++ b/atom/model_engine/pp_engine_core.py @@ -268,6 +268,12 @@ def _poll_kv_transfer_progress(self): kvoutput.finished_loading or kvoutput.failed_loading or kvoutput.finished_saving + # connector_completions are offload channel events (kimi_k3 state + # dispositions, dsv4 checkpoint boundaries). They too span all PP + # stages, so they must reach the aggregator rather than the + # scheduler directly -- and count as "offload work" so this poll + # does not early-return and strand them. + or kvoutput.connector_completions ) pp_messages = self.pp_transport.recv_kv_status(timeout_ms=0) @@ -317,6 +323,7 @@ def _poll_kv_transfer_progress(self): finished_loading=kvoutput.finished_loading, failed_loading=kvoutput.failed_loading, finished_saving=kvoutput.finished_saving, + connector_completions=kvoutput.connector_completions, ) if not offload_local.is_empty(): self._ingest_and_release(offload_local, 0) From 1a127bbf9cc2337ef9ecae3c2bdaead6dc2fd02a Mon Sep 17 00:00:00 2001 From: zejchen Date: Wed, 2 Sep 2026 01:16:24 +0000 Subject: [PATCH 121/151] [offload] _profile_enabled: read empty OFFLOAD_PROFILE as off, matching _env_flag `_env_flag` (atom_lmcache_staging.py) was fixed to strip its value and treat the empty string as OFF: `VAR=` is how a shell clears a flag inline, and a bare membership test reads "" as ON (it is not in the false set), the opposite of what the operator wrote; `VAR="off "` had the same trap. `_profile_enabled` still carried the un-swept version -- `os.environ.get("OFFLOAD_PROFILE","0") .lower() not in {...}` with no strip and no empty guard -- so `OFFLOAD_PROFILE=` or `OFFLOAD_PROFILE="off "` silently turned profiling ON. Apply the same semantics. Kept inline with a cross-ref comment rather than importing `_env_flag`, because that module imports torch and `_offload_common` is deliberately torch-free. Co-Authored-By: Claude Opus 4.8 --- atom/kv_transfer/offload/_offload_common.py | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/atom/kv_transfer/offload/_offload_common.py b/atom/kv_transfer/offload/_offload_common.py index 935c70103d..ff81d318d4 100644 --- a/atom/kv_transfer/offload/_offload_common.py +++ b/atom/kv_transfer/offload/_offload_common.py @@ -183,12 +183,14 @@ def _lookup_unpin(self, req_id) -> None: @staticmethod def _profile_enabled() -> bool: - return os.environ.get("OFFLOAD_PROFILE", "0").lower() not in { - "0", - "false", - "no", - "off", - } + # Same env-flag semantics as `atom_lmcache_staging._env_flag` (kept + # inline rather than imported: that module pulls in torch and this one + # is torch-free). Strip, and read the empty string as OFF -- `VAR=` is + # how a shell clears a flag inline, and a bare membership test would + # read "" as ON (not in the false set), the opposite of what the + # operator wrote; `VAR="off "` had the same trap. + raw = os.environ.get("OFFLOAD_PROFILE", "0").strip().lower() + return bool(raw) and raw not in {"0", "false", "no", "off"} def _last_gpu_connector_transfer_stats(self) -> dict[str, int | float]: gpu_connector = getattr(getattr(self, "_engine", None), "gpu_connector", None) From 36d9e332d2a957bc27e0d7c9109f2bb94097f2e4 Mon Sep 17 00:00:00 2001 From: zejchen Date: Wed, 2 Sep 2026 01:17:31 +0000 Subject: [PATCH 122/151] [offload] block_manager: fold _extend_hash_chain's duplicated loop onto _chain_to `_extend_hash_chain` and `_chain_to` carried byte-identical chain-continuation loops -- `list(block_hashes)`, resume `h` from the last appended hash (or -1), then `compute_hash(self._hash_block_tokens(seq, i), h)` appended up to a target width. The only differences are `_extend_hash_chain`'s midstep gate, its target of `_n_hash_blocks(seq)` (vs `_chain_to`'s explicit `blocks` cap), and that it writes `seq.block_hashes` instead of returning. Delegate the loop: `_extend_hash_chain` now calls `_chain_to(seq, block_hashes, self._n_hash_blocks(seq))` after its gate. Behaviour is identical -- `_chain_to` already resumes from the last appended hash exactly as the inlined loop did -- and the "resume, don't reuse the loop's `h`" invariant now lives in one place, so the two can't drift. Co-Authored-By: Claude Opus 4.8 --- atom/model_engine/block_manager.py | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/atom/model_engine/block_manager.py b/atom/model_engine/block_manager.py index b96dcd479f..a425de3158 100644 --- a/atom/model_engine/block_manager.py +++ b/atom/model_engine/block_manager.py @@ -1877,15 +1877,15 @@ def _extend_hash_chain(self, seq: Sequence, block_hashes: list[int]) -> None: if not self.state.readable_midstep or not self.state.applies(seq): seq.block_hashes = [] return - full = list(block_hashes) - # Resume from the last APPENDED hash, not from the loop's `h`: on a miss - # that holds the hash of the block that failed to match, which was never - # appended and is not part of this chain. - h = full[-1] if full else -1 - for i in range(len(full), self._n_hash_blocks(seq)): - h = self.compute_hash(self._hash_block_tokens(seq, i), h) - full.append(h) - seq.block_hashes = full + # The continuation itself -- resume from the last APPENDED hash (on a + # miss the loop's `h` holds the hash of the block that failed to match, + # which was never appended and is not part of this chain) -- is exactly + # `_chain_to`, run to the full prompt width. Delegate rather than + # duplicate the loop; the only thing local to this path is the midstep + # gate above and writing the result to `seq`. + seq.block_hashes = self._chain_to( + seq, block_hashes, self._n_hash_blocks(seq) + ) def midstep_positions(self, seq: Sequence, start: int, end: int) -> list[tuple]: """`(position, hash)` for every checkpoint a forward over `(start, end]` From 81b06a5d23b01b052f1648e8881c2dcd78b47e64 Mon Sep 17 00:00:00 2001 From: zejchen Date: Wed, 2 Sep 2026 01:18:41 +0000 Subject: [PATCH 123/151] [offload] state_offload: drop dead kv_connector_hosts_state_tier and StateTierCapability.layout Two never-read symbols. `kv_connector_hosts_state_tier` -- the thin `state_tier_capability(config).hosts_state_tier` wrapper -- has no callers anywhere in the tree; the one place that needs the yes/no (`BlockManager.__init__`) reads `state_tier_capability(...).hosts_state_tier` directly. `StateTierCapability.layout` is set at both construction sites but never read: block_manager consumes `hosts_state_tier`, `reason`, `can_store_state`, and `can_load_state`, never `layout`, and the layout value is already used locally in `state_tier_capability` for the `_STATE_TIER_LAYOUTS` gate before it would be stored. Remove the wrapper function and the field; drop the now-unused positional at the two `StateTierCapability(...)` call sites. `state_tier_capability` and its live fields are unchanged. Co-Authored-By: Claude Opus 4.8 --- atom/model_engine/state_offload.py | 16 ++-------------- 1 file changed, 2 insertions(+), 14 deletions(-) diff --git a/atom/model_engine/state_offload.py b/atom/model_engine/state_offload.py index c961528d91..c469a2d3a4 100644 --- a/atom/model_engine/state_offload.py +++ b/atom/model_engine/state_offload.py @@ -189,7 +189,6 @@ class StateTierCapability: can_store_state: bool can_load_state: bool - layout: str | None reason: str = "" @property @@ -251,7 +250,7 @@ def state_tier_capability(config) -> StateTierCapability: """ from atom.kv_transfer.offload.config import select_offload_layout - none = StateTierCapability(False, False, None, "") + none = StateTierCapability(False, False, "") cfg = getattr(config, "kv_transfer_config", None) or {} if not isinstance(cfg, dict): return replace(none, reason="no kv_transfer_config") @@ -290,7 +289,7 @@ def state_tier_capability(config) -> StateTierCapability: can_load = role in _LOAD_ROLES if not (can_store or can_load): return replace(none, reason=f"kv_role {role!r} neither saves nor loads") - return StateTierCapability(can_store, can_load, layout) + return StateTierCapability(can_store, can_load) def state_tier_chunk_tokens(config) -> int: @@ -340,14 +339,3 @@ def state_tier_chunk_tokens(config) -> int: #: The connector backends whose worker half can build a `StateOffloadTier`. #: `multi` qualifies when it lists exactly one. _STATE_TIER_BACKENDS = frozenset({"lmcache_offload"}) - - -def kv_connector_hosts_state_tier(config) -> bool: - """Whether the configured connector will really run the state tier. - - Thin wrapper over `state_tier_capability` for callers that only need the - yes/no. Takes the whole config: the layout and the pipeline depth are as - much a part of the answer as the connector name, and passing only - `kv_transfer_config` is what made this a name check in the first place. - """ - return state_tier_capability(config).hosts_state_tier From 27c22f3c396a3136d1515d666186cf7bf6e6a7e1 Mon Sep 17 00:00:00 2001 From: zejchen Date: Wed, 2 Sep 2026 01:19:40 +0000 Subject: [PATCH 124/151] [offload] get_cache_statistics: include offload_tokens in the DP-aggregated sum Each rank's `_handle_get_cache_statistics` returns `offload_tokens` (from `EngineStats.cache_statistics()`), but the cross-rank aggregation in `LLMEngine.get_cache_statistics` summed a fixed key tuple that omitted it. So the DP-aggregated snapshot -- the one a client polls to measure reuse over a handful of requests -- silently dropped the single counter the CPU offload tier exists to move, while faithfully summing the HBM and checkpoint-funnel series around it. Add `offload_tokens` to the summed keys and expose `offload_hit` = rate over the same `reusable` denominator as `hit`, mirroring `EngineStats.offload_hit_rate`. It is scored on its own, not folded into the `hit`/`compressed_hit` product: the offload tier serves what HBM missed, so its numerator is disjoint from the paged/state split. Co-Authored-By: Claude Opus 4.8 --- atom/model_engine/llm_engine.py | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/atom/model_engine/llm_engine.py b/atom/model_engine/llm_engine.py index 669603c759..c4f536e041 100644 --- a/atom/model_engine/llm_engine.py +++ b/atom/model_engine/llm_engine.py @@ -388,6 +388,12 @@ def get_cache_statistics(self, timeout: float = 30.0) -> dict[str, Any]: "wanted_tokens", "reusable_tokens", "full_tokens", + # Reuse served from the CPU offload tier. Every rank reports it + # in `cache_statistics()`, but it was missing from this sum, so + # the DP-aggregated snapshot dropped the one counter the offload + # feature exists to move. Shares the `reusable` denominator with + # the HBM series but no numerator -- scored separately below. + "offload_tokens", "checkpoints_kept", "checkpoints_dropped", "checkpoints_evicted", @@ -440,6 +446,12 @@ def rate(num: int, den: int = reusable) -> float: "state_recoverable_loss": rate( totals["wanted_tokens"] - totals["cached_tokens"], compressed ), + # CPU-tier reuse against the same `reusable` denominator as `hit`, + # matching `EngineStats.offload_hit_rate`. Disjoint from the HBM + # `hit` numerator (the offload tier serves what HBM missed), so it + # is not part of the `hit`/`compressed_hit` product and stands on + # its own. + "offload_hit": rate(totals["offload_tokens"]), } def get_metrics_statistics(self) -> dict[str, Any]: From 7a2684b36487695ee0464fc7fe038b611ff8df18 Mon Sep 17 00:00:00 2001 From: zejchen Date: Wed, 2 Sep 2026 01:26:50 +0000 Subject: [PATCH 125/151] [block_manager] skip joint-boundary computation on record=False fit probe The `record=False` fit probe (scheduler `is_mixed_batch`, block_manager.py can_allocate) only reads the `>= 0` return, yet it still ran `_joint_kv_boundary` in full and threw the result away. That call is the one the maintainer flagged as "a full O(prompt) hash chain per scheduler pass": `_joint_kv_boundary` walks a chained xxhash up to the LMCache-only cap (`_chain_to`) plus a `_gated_hit` rescan, and its only consumer is `_commit_joint_boundary`, already gated behind `if record:`. Previously the boundary was computed unconditionally and only the *commit* was gated -- so a probe paid the entire O(prompt) chain to build a `decision` that was immediately discarded. `is_mixed_batch` peeks up to four waiting seqs every scheduling pass, so a 128k prompt queued behind a full pool paid ~2000 xxhash rounds per pass for nothing. Move the `_joint_kv_boundary` call inside the `if record:` block alongside the commit. The return value (`num_cached_blocks`, from the Step-1/`_gated_hit` walk) does not depend on the joint decision, so this is behaviour-preserving: the probe's fit answer is unchanged, it just no longer computes a boundary it cannot use. Comment rewritten to say the probe now skips the computation rather than "computes and drops it". Runner-up from review round 4 (comment 5496038968). Co-Authored-By: Claude Opus 4.8 --- atom/model_engine/block_manager.py | 23 +++++++++++++---------- 1 file changed, 13 insertions(+), 10 deletions(-) diff --git a/atom/model_engine/block_manager.py b/atom/model_engine/block_manager.py index a425de3158..0fd9b096ae 100644 --- a/atom/model_engine/block_manager.py +++ b/atom/model_engine/block_manager.py @@ -928,17 +928,20 @@ def can_allocate(self, seq: Sequence, record: bool = True) -> int: # Committed to the seq rather than returned: what `allocate` claims from # HBM is still `num_cached_blocks`, and the joint boundary only decides # where the two loads are aimed. `record` is the probe/admission split: - # a real admission commits the decision (seq fields + funnel counters), a - # fit probe computes and drops it, so a seq that never schedules cannot - # inflate the operator-visible funnel. Below the refusal, not above it, - # for the same reason _extend_hash_chain is: _joint_kv_boundary walks a - # chained xxhash up to the LMCache-only cap (_chain_to) plus a _gated_hit - # rescan, and a refused admission would discard all of it. Nothing - # between here and the refusal reads the seq's joint fields, and the - # author already moved _extend_hash_chain down for this exact cost -- - # this is its twin. - decision = self._joint_kv_boundary(seq, num_cached_blocks, block_hashes) + # a real admission computes the boundary and commits it (seq fields + + # funnel counters); a fit probe skips it entirely. The `num_cached_blocks` + # returned below does not depend on the decision, so gating the + # computation -- not just the commit -- spares a probe the O(prompt) work + # for a value it would only discard: `_joint_kv_boundary` walks a chained + # xxhash up to the LMCache-only cap (`_chain_to`) plus a `_gated_hit` + # rescan. A 128k prompt at the front of a KV-pressured queue paid that + # full chain on every scheduling pass (`is_mixed_batch` peeks up to four + # waiting seqs with `record=False`) for a decision nothing read. Placed + # below the refusal, not above it, for the same reason `_extend_hash_chain` + # is: a refused admission would discard it, and nothing between here and + # the refusal reads the seq's joint fields -- this is that move's twin. if record: + decision = self._joint_kv_boundary(seq, num_cached_blocks, block_hashes) self._commit_joint_boundary(seq, decision) # After the refusal, not before it. The chain is O(prompt) xxhash plus # two temporaries per block, and a refused admission discards it — a From e4e70802baa714ff4da9530247e7c17416f704f7 Mon Sep 17 00:00:00 2001 From: zejchen Date: Wed, 2 Sep 2026 01:28:09 +0000 Subject: [PATCH 126/151] [block_manager] rename state_hbm/state_tier int counters to *_boundaries `BlockManager.state_tier` was an int counter (joint boundaries whose state leg came from the CPU tier), but bare `state_tier` names the state-tier object and module everywhere else in this PR -- `kimi_k3.state_tier`, `_JointPark`, the connector's `_state_tier` index. Reading `self.state_tier` mid-scheduler looks like a handle to that object, not a running total; the maintainer flagged the collision. Rename the counter pair for name-matches-function: state_hbm -> state_hbm_boundaries state_tier -> state_tier_boundaries Both are the two arms of `joint_boundaries` (state_from_hbm vs state_from_tier), so they move together and keep reading as a pair. The operator-facing surface is unchanged: the exported `checkpoint_funnel()` keys stay "state_hbm"/"state_tier" and the scheduler log label stays `state_hbm=%d state_tier=%d` -- only the Python attribute is renamed. Updated the two scheduler reads and the three test_state_checkpoint assertions accordingly. Runner-up from review round 4 (comment 5496038968). Co-Authored-By: Claude Opus 4.8 --- atom/model_engine/block_manager.py | 16 ++++++++++------ atom/model_engine/scheduler.py | 4 ++-- tests/test_state_checkpoint.py | 6 +++--- 3 files changed, 15 insertions(+), 11 deletions(-) diff --git a/atom/model_engine/block_manager.py b/atom/model_engine/block_manager.py index 0fd9b096ae..be9d13f4ab 100644 --- a/atom/model_engine/block_manager.py +++ b/atom/model_engine/block_manager.py @@ -285,11 +285,15 @@ def __init__( # resident checkpoint, `tier` paid an entry-sized H2D and a park. Almost # all `tier` means the state pool is too small for the concurrency. self.joint_boundaries = 0 - self.state_hbm = 0 + self.state_hbm_boundaries = 0 # The number that says the tier is doing anything at all. Every other # counter here can be non-zero with the CPU tier switched off; this one # cannot, which makes it the only honest test of "did this feature run". - self.state_tier = 0 + # Suffixed `_boundaries` so an int counter no longer shares the bare name + # `state_tier` with the state-tier object/module used across this PR + # (`kimi_k3.state_tier`, `_JointPark`, `_state_tier`). The exported funnel + # key and the log label keep the short `state_tier` string. + self.state_tier_boundaries = 0 # Admissions whose gated boundary neither tier could produce by the time # `allocate` ran. Non-zero is expected under pressure (the CPU index is # optimistic, and an HBM checkpoint can be unindexed inside the same @@ -816,9 +820,9 @@ def _commit_joint_boundary( seq.offload_joint.claim_tokens = decision.claim_tokens self.joint_boundaries += 1 if decision.state_from_hbm: - self.state_hbm += 1 + self.state_hbm_boundaries += 1 else: - self.state_tier += 1 + self.state_tier_boundaries += 1 def _chain_to( self, seq: Sequence, block_hashes: list[int], blocks: int @@ -2091,8 +2095,8 @@ def checkpoint_funnel(self) -> dict[str, int]: "chunks_cut_for_demand": self.chunks_cut_for_demand, "chunks_cut_for_end": self.chunks_cut_for_end, "joint_boundaries": self.joint_boundaries, - "state_hbm": self.state_hbm, - "state_tier": self.state_tier, + "state_hbm": self.state_hbm_boundaries, + "state_tier": self.state_tier_boundaries, "state_gate_lost_boundary": self.state_gate_lost_boundary, "orphan_load_slots_reclaimed": self._orphan_load_slots_reclaimed, } | self.state_checkpoint_fates() diff --git a/atom/model_engine/scheduler.py b/atom/model_engine/scheduler.py index eb5a303e7f..91cfc73d8a 100644 --- a/atom/model_engine/scheduler.py +++ b/atom/model_engine/scheduler.py @@ -1508,8 +1508,8 @@ def _schedule(self) -> tuple[ScheduledBatch, dict[int, Sequence]] | None: logger.info( "joint kv: boundaries=%d (state_hbm=%d state_tier=%d) | %s", chosen, - self.block_manager.state_hbm, - self.block_manager.state_tier, + self.block_manager.state_hbm_boundaries, + self.block_manager.state_tier_boundaries, " ".join(f"{k}={v}" for k, v in sorted(skips.items())), ) diff --git a/tests/test_state_checkpoint.py b/tests/test_state_checkpoint.py index 6a7df1f285..ea0eb9eac8 100644 --- a/tests/test_state_checkpoint.py +++ b/tests/test_state_checkpoint.py @@ -1528,7 +1528,7 @@ def _break_the_kv_chain_at(bm, block_id: int) -> None: bm.kv.allocate(block_id) # takes it for fresh content, dropping the hash def _reset_joint_counters(self, bm): - bm.joint_boundaries = bm.state_hbm = bm.state_tier = 0 + bm.joint_boundaries = bm.state_hbm_boundaries = bm.state_tier_boundaries = 0 bm.joint_skips.clear() def test_a_page_class_now_gets_a_joint_boundary(self): @@ -1579,14 +1579,14 @@ def test_the_split_says_which_tier_the_state_leg_came_from(self): resident = stateful_seq(PROMPT) resident.offload_joint.kv_prefix_tokens = len(PROMPT) bm.can_allocate(resident) - assert (bm.state_hbm, bm.state_tier) == (1, 0) + assert (bm.state_hbm_boundaries, bm.state_tier_boundaries) == (1, 0) # Drop it from HBM too: now the state leg costs an image-sized H2D. bm.paged_state_checkpoints.unindex(h) from_cpu = stateful_seq(PROMPT) from_cpu.offload_joint.kv_prefix_tokens = len(PROMPT) bm.can_allocate(from_cpu) - assert (bm.state_hbm, bm.state_tier) == (1, 1) + assert (bm.state_hbm_boundaries, bm.state_tier_boundaries) == (1, 1) def test_no_tier_means_no_joint_boundary(self): """`state_offload is None` short-circuits before anything else: a From 01a6fe551995c4a8ed02a97af7fdf489cfb81d08 Mon Sep 17 00:00:00 2001 From: zejchen Date: Wed, 2 Sep 2026 01:33:05 +0000 Subject: [PATCH 127/151] [offload] bound the two unbounded state-offload structures Two offload-side structures grew without limit over a server's lifetime: 1. `StateOffloadIndex.hashes` -- the optimistic "believed in LMCache" set. It only grows on `note_stored` and shrinks on `forget` (a failed load), so a long-running server that stored many distinct prefixes accumulated a hash per prefix forever. Cap it with a companion insertion-ordered `_hash_lru` (OrderedDict used as an ordered set) that drops the coldest hash once the index passes `_hash_cap` (~1M, a few tens of MB). `hashes` stays a plain set so every caller and test keeps `in` / `len` / `== set()` / `discard`; the two move in lockstep on add and forget, so they never diverge in length. Dropping a hash is safe for the same reason the index is optimistic: a false "not stored" costs one recompute, never wrong output. Surfaced as `hashes_evicted`. 2. `PageUnitCheckpointStore._offload_ready` -- the READY-checkpoint nomination backlog. Under sustained tier backpressure the drain loop in `take_offload_stores` exits at its first iteration (pins already at `max_inflight`) and never reaches the stale-drop, so a steady nomination stream grew the deque without limit with ids of long-evicted checkpoints. The `checkpoint_fates` gauge even documented the queue as "unbounded". Cap it at `_offload_backlog_cap` (8192), dropping the oldest (coldest, most likely already spent) nomination on overflow -- a missed offload, never a lost image, since a nomination is unpinned and the pool may spend its checkpoint anyway. popleft-on-overflow rather than `deque(maxlen=)` because `take_offload_stores` re-queues deferred nominations with `extendleft` and a maxlen would evict them from the wrong end. Warns once and counts drops in `state_offload_nominations_dropped`; comment corrected from "unbounded". Runner-up from review round 4 (comment 5496038968). Co-Authored-By: Claude Opus 4.8 --- atom/model_engine/page_unit_checkpoint.py | 40 ++++++++++++++++++++--- atom/model_engine/state_offload.py | 35 +++++++++++++++++++- 2 files changed, 70 insertions(+), 5 deletions(-) diff --git a/atom/model_engine/page_unit_checkpoint.py b/atom/model_engine/page_unit_checkpoint.py index 494c9e9f3b..42b56504fa 100644 --- a/atom/model_engine/page_unit_checkpoint.py +++ b/atom/model_engine/page_unit_checkpoint.py @@ -146,6 +146,18 @@ def __init__( # space and `_next_victim` may spend it. The pin is taken later, in # `take_offload_stores`, and only for the few actually in flight. self._offload_ready: deque[int] = deque() + # Ceiling on that backlog. Under sustained tier backpressure the drain + # loop in `take_offload_stores` exits at its first iteration -- pins + # already at `max_inflight` -- and so never reaches the stale-drop that + # otherwise clears spent nominations. An unbounded nomination stream + # would then grow `_offload_ready` without limit with ids whose + # checkpoints have long since been evicted. A nomination is unpinned and + # the pool may spend its checkpoint anyway, so dropping the oldest + # (coldest) on overflow costs a missed offload, never a lost image: the + # checkpoint stays HBM-resident until it is evicted. Sized well above + # any real in-flight-plus-waiting count so it only bites a runaway. + self._offload_backlog_cap = 8192 + self.offload_nominations_dropped = 0 # StateStoreOperationId -> (checkpoint_id, monotonic when pinned). # A store's report carries no request -- by the time one lands its owner # is long gone -- so the hash is the only thing left to name it by. The @@ -392,6 +404,22 @@ def _queue_offload_store(self, checkpoint_id: int, record) -> None: if not self._offload_sink: return self._offload_ready.append(checkpoint_id) + # Bound the backlog (see `_offload_backlog_cap`). popleft-on-overflow + # rather than a `deque(maxlen=)`: `take_offload_stores` re-queues + # deferred nominations with `extendleft`, and a maxlen deque would evict + # from whichever end an append pushed against -- dropping the freshly + # re-queued oldest entries we take pains to preserve there. + if len(self._offload_ready) > self._offload_backlog_cap: + if self.offload_nominations_dropped == 0: + logger.warning( + "state offload nomination backlog exceeded %d; dropping " + "oldest nominations. The CPU tier is draining checkpoints " + "slower than they reach READY -- offload is falling behind.", + self._offload_backlog_cap, + ) + while len(self._offload_ready) > self._offload_backlog_cap: + self._offload_ready.popleft() + self.offload_nominations_dropped += 1 @property def store_backlog(self) -> deque: @@ -919,9 +947,13 @@ def checkpoint_fates(self) -> dict[str, int]: if callable(stats): fates.update((f"state_offload_{k}", v) for k, v in stats().items()) # A gauge, not a counter: how many nominations are waiting for a - # slot under `OFFLOAD_MAX_PENDING_SAVES`. The queue is unbounded and - # drained oldest-first, so a value that climbs and stays high means - # checkpoints are reaching READY faster than the tier drains them - # and fresh nominations are queueing behind stale ones. + # slot under `OFFLOAD_MAX_PENDING_SAVES`. The queue is bounded by + # `_offload_backlog_cap` and drained oldest-first, so a value that + # climbs toward the cap means checkpoints are reaching READY faster + # than the tier drains them and fresh nominations are queueing behind + # stale ones; at the cap the oldest are dropped, counted next to it. fates["state_offload_store_backlog"] = len(self.store.store_backlog) + fates["state_offload_nominations_dropped"] = ( + self.store.offload_nominations_dropped + ) return fates diff --git a/atom/model_engine/state_offload.py b/atom/model_engine/state_offload.py index c469a2d3a4..201c5d4b54 100644 --- a/atom/model_engine/state_offload.py +++ b/atom/model_engine/state_offload.py @@ -12,6 +12,7 @@ """ import logging +from collections import OrderedDict from dataclasses import dataclass, replace logger = logging.getLogger("atom") @@ -33,7 +34,24 @@ def __init__(self, *, can_store: bool = True, can_load: bool = True) -> None: # load the worker will not serve parks the request that took it. self.can_store = bool(can_store) self.can_load = bool(can_load) + # The optimistic membership index (see the class docstring). It only + # grows on `note_stored` and shrinks on `forget` (a failed load), so a + # long-lived server that stored many distinct prefixes would grow it + # without bound. Cap it: `hashes` stays a plain set -- membership stays + # O(1) and every caller/test keeps `in` / `len` / `== set()` / + # `discard` -- and `_hash_lru` records insertion order beside it purely + # so the oldest hash can be dropped on overflow. The two move in + # lockstep (every add and every forget touches both), so `_hash_lru` + # never diverges in length from `hashes`. Dropping the oldest hash is + # safe for the same reason the index is optimistic: a false "not stored" + # only costs one recompute, never wrong output. self.hashes: set[int] = set() + self._hash_lru: OrderedDict[int, None] = OrderedDict() + # ~1M hashes -> a few tens of MB. Well above any working set that a + # tier of realistic capacity actually holds, so it bounds a leak + # without evicting live entries in normal operation. + self._hash_cap = 1 << 20 + self.hashes_evicted = 0 # req_id -> hash, for loads offered and not yet settled. Keyed by # request because that is what comes back, on the same # `finished_loading`/`failed_loading` channel a KV load uses. @@ -68,11 +86,21 @@ def note_stored(self, h: int) -> None: hash advertised before its bytes exist parks the next request over that prefix against a `get` that must miss. """ - self.hashes.add(int(h)) + h = int(h) + self.hashes.add(h) + # Reinsert at the young end whether or not it was already present, so a + # re-stored prefix is treated as freshly used, then trim the cold end. + self._hash_lru.pop(h, None) + self._hash_lru[h] = None + while len(self._hash_lru) > self._hash_cap: + old, _ = self._hash_lru.popitem(last=False) + self.hashes.discard(old) + self.hashes_evicted += 1 def forget(self, h: int) -> None: """Drop a hash whose load failed, so the next request does not retry.""" self.hashes.discard(h) + self._hash_lru.pop(h, None) # -------------------------------- loads -------------------------------- # def request_load(self, req_id, h: int) -> bool: @@ -156,6 +184,11 @@ def stats(self) -> dict[str, int]: "loads_completed": self.loads_completed, "loads_failed": self.loads_failed, "indexed": len(self.hashes), + # Non-zero means the index hit its `_hash_cap` and is dropping the + # coldest hashes. Harmless (a dropped hash just misses one reuse), + # but a climbing value means the cap is smaller than the live prefix + # working set and reuse is being left on the table. + "hashes_evicted": self.hashes_evicted, } From a0fae1f537d4eaac39ae90366ce485cdbf3af7c1 Mon Sep 17 00:00:00 2001 From: zejunchen-zejun Date: Wed, 2 Sep 2026 09:41:59 +0800 Subject: [PATCH 128/151] fix(offload): forward take_state_load_survived, and unbreak the worker sweep Four pre-check jobs were red: Black, Ruff, and the non-GPU unit tests (two failures). The style ones are trivial; the test failures were not. **Black** wanted two call sites collapsed onto one line now that they fit in 88 columns -- `_chain_to(...)` in `block_manager.py` and the `take_state_load_survived` `getattr` in `scheduler.py`. **Ruff UP031** on the shell-coverage helper: `r"...%s..." % re.escape(name)` became an f-string. **`take_state_load_survived` was missing from BOTH shells.** The sweep test caught a real defect, not a test problem. `KimiK3OffloadScheduler` defines it and `Scheduler._update_from_kv_xfer_finished` reads it off `self.kv_connector` -- which is the shell, never the impl. Neither `LMCacheOffloadConnectorScheduler` nor `MultiConnectorScheduler` exposed it, so `getattr(..., None)` took its default on every pass and the survivor set was always empty. The consequence is precisely the one that channel exists to prevent: a joint load whose KV leg failed while its state H2D landed intact settled as *failed* rather than *abandoned*, so `fail_load` forgot a hash whose bytes are sitting in LMCache. Every later request over that prefix then parks, misses, and recomputes. Silent, and it degrades the hit rate rather than raising. This is the fourth instance of the forwarder axis in this branch's history (`enqueue_state_stores`, `take_state_reports` arity, `chunk_size` under multi, now this one). Both shells are fixed, and the multi one routes by `_state_tier_sub()` like its neighbours rather than by method presence. **The worker sweep imported `model_runner`, which imports aiter at module load.** aiter is absent on the non-GPU runner -- the machine that runs this test -- so the sweep added for the worker axis could only ever fail there. It wants the source text, not the module, so it now reads both driver files off disk via `atom.__file__` and asserts they exist. That keeps the sweep honest (a moved file fails loudly) without dragging a GPU dependency into a CPU-only job. --- .../disaggregation/multi/multi_connector.py | 18 +++++++++++++++ atom/kv_transfer/offload/connector.py | 12 ++++++++++ atom/model_engine/block_manager.py | 4 +--- atom/model_engine/scheduler.py | 4 +--- tests/test_lmcache_offload_connector.py | 22 ++++++++++++++----- 5 files changed, 48 insertions(+), 12 deletions(-) diff --git a/atom/kv_transfer/disaggregation/multi/multi_connector.py b/atom/kv_transfer/disaggregation/multi/multi_connector.py index c480b09358..75bdbb4687 100644 --- a/atom/kv_transfer/disaggregation/multi/multi_connector.py +++ b/atom/kv_transfer/disaggregation/multi/multi_connector.py @@ -601,6 +601,24 @@ def take_state_reports(self): return set(), set() return c.take_state_reports() + def take_state_load_survived(self) -> set: + """Requests whose state bytes outlived a failed joint load, from the + tier sub; empty set if none. + + `Scheduler._update_from_kv_xfer_finished` reads this off the composite, + and its `getattr(..., None)` default is indistinguishable from "nothing + survived" -- so without the forwarder every survivor under + `kv_connector: multi` settled as a failure, forgetting a hash whose + bytes are present. + + Select by `has_state_tier`, not method presence -- see + `_state_tier_sub`. + """ + c = self._state_tier_sub() + if c is None: + return set() + return c.take_state_load_survived() + def take_state_source_releases(self) -> set: """Stores whose PAGE units the GPU has finished reading, from the tier sub; empty set if none. diff --git a/atom/kv_transfer/offload/connector.py b/atom/kv_transfer/offload/connector.py index 568b3e3ecc..68c47ad4d6 100644 --- a/atom/kv_transfer/offload/connector.py +++ b/atom/kv_transfer/offload/connector.py @@ -246,6 +246,18 @@ def take_state_reports(self): callback = getattr(self._impl, "take_state_reports", None) return callback() if callback is not None else (set(), set()) + def take_state_load_survived(self) -> set: + """Requests whose state bytes outlived a failed joint load. + + `Scheduler._update_from_kv_xfer_finished` reads this off the SHELL, so + without the forwarder the `getattr(..., None)` default won and every + survivor settled as a failure -- forgetting a hash whose bytes are + present, which is the opposite of what the channel exists to do. An + impl that does not advertise the channel survives nothing. + """ + callback = getattr(self._impl, "take_state_load_survived", None) + return callback() if callback is not None else set() + def take_state_source_releases(self) -> set: """Stores whose PAGE units the GPU has finished reading. diff --git a/atom/model_engine/block_manager.py b/atom/model_engine/block_manager.py index be9d13f4ab..da8a36f200 100644 --- a/atom/model_engine/block_manager.py +++ b/atom/model_engine/block_manager.py @@ -1890,9 +1890,7 @@ def _extend_hash_chain(self, seq: Sequence, block_hashes: list[int]) -> None: # `_chain_to`, run to the full prompt width. Delegate rather than # duplicate the loop; the only thing local to this path is the midstep # gate above and writing the result to `seq`. - seq.block_hashes = self._chain_to( - seq, block_hashes, self._n_hash_blocks(seq) - ) + seq.block_hashes = self._chain_to(seq, block_hashes, self._n_hash_blocks(seq)) def midstep_positions(self, seq: Sequence, start: int, end: int) -> list[tuple]: """`(position, hash)` for every checkpoint a forward over `(start, end]` diff --git a/atom/model_engine/scheduler.py b/atom/model_engine/scheduler.py index 91cfc73d8a..e1c0480aeb 100644 --- a/atom/model_engine/scheduler.py +++ b/atom/model_engine/scheduler.py @@ -3121,9 +3121,7 @@ def _update_from_kv_xfer_finished(self, kv_connector_output: KVConnectorOutput): # kept loadable) rather than *failed* (hash forgotten). Everything else # -- a genuine state-leg miss, or a connector that never advertises the # channel -- still settles as a failure. - state_survived = getattr( - self.kv_connector, "take_state_load_survived", None - ) + state_survived = getattr(self.kv_connector, "take_state_load_survived", None) state_survived = state_survived() if state_survived is not None else set() for req_id in kv_connector_output.failed_loading or (): assert ( diff --git a/tests/test_lmcache_offload_connector.py b/tests/test_lmcache_offload_connector.py index c788d01545..29a23179c4 100644 --- a/tests/test_lmcache_offload_connector.py +++ b/tests/test_lmcache_offload_connector.py @@ -5252,7 +5252,7 @@ class attributes on the `LMCacheOffload*` shells but instance attributes on src = inspect.getsource(cls) except (OSError, TypeError): return False - return bool(re.search(r"\bself\.%s\s*=" % re.escape(name), src)) + return bool(re.search(rf"\bself\.{re.escape(name)}\s*=", src)) def test_every_member_the_scheduler_reads_is_reachable_through_the_shell(): @@ -5353,17 +5353,27 @@ def test_every_member_the_worker_reads_is_reachable_through_the_shell(): `close` exists to prevent). Read the driver's source, not a hand-kept list, for the same reason the scheduler sweep does. """ - import inspect + import pathlib import re - import atom.model_engine.model_runner as mr_mod - import atom.utils.forward_context as fc_mod + import atom from atom.kv_transfer.disaggregation.multi.multi_connector import MultiConnector from atom.kv_transfer.offload.connector import LMCacheOffloadConnector + # Read the driver sources off disk rather than importing them. + # `model_runner` imports aiter at module load, which is absent on the + # non-GPU CI runner -- importing it here would turn this sweep into a + # collection error on exactly the machine that runs it. The sweep only ever + # wanted the text. + root = pathlib.Path(atom.__file__).parent + sources = ( + root / "model_engine" / "model_runner.py", + root / "utils" / "forward_context.py", + ) probed: set[str] = set() - for mod in (mr_mod, fc_mod): - src = inspect.getsource(mod) + for path in sources: + assert path.is_file(), f"driver source moved: {path}" + src = path.read_text() probed |= set(re.findall(r'getattr\(\s*connector,\s*"([a-z_]+)"', src)) probed |= set(re.findall(r"\bconnector\.([a-z_]+)\(", src)) probed |= set(re.findall(r"\bconnector\.([a-z_]+)\b", src)) From 6b59c9ba5fcbc59459f4358a96700c319d478011 Mon Sep 17 00:00:00 2001 From: zejchen Date: Wed, 2 Sep 2026 06:40:44 +0000 Subject: [PATCH 129/151] fix(offload): do not double-discharge the state MemoryObj on a post-handoff raise Round-5 review finding #2. `StateByteCodec.put`'s `try` spanned pack + on_source_released + batched_put, and the `except: obj.ref_count_down()` could not tell "never reached batched_put" from "reached it, it took ownership, then something raised" (an NVMe/L3 backend error or an eviction-path raise inside LMCache's put, after the CPU insert). The second case drove the ref count to -1 and returned a still-indexed buffer to the allocator, so a later get for a different key handed that buffer back -- wrong bytes unpacked into a request's Active Slot with no exception. Draw the ownership line with a `handed_off` flag set immediately before batched_put: down the ref only when the handoff never happened. Co-Authored-By: Claude Opus 4.8 --- .../offload/hybrid/kimi_k3/state_object.py | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-) diff --git a/atom/kv_transfer/offload/hybrid/kimi_k3/state_object.py b/atom/kv_transfer/offload/hybrid/kimi_k3/state_object.py index dddf830902..971ded5581 100644 --- a/atom/kv_transfer/offload/hybrid/kimi_k3/state_object.py +++ b/atom/kv_transfer/offload/hybrid/kimi_k3/state_object.py @@ -135,15 +135,28 @@ def put(self, h: int, unit_ids, on_source_released=None) -> bool: # ref_count=1 (LMCache reports "garbage collected with ref_count=1, # pin_count=0" much later), shrinking the CPU pool one entry per failure. # `get` guards its own reference with `finally` for the same reason. + # + # `handed_off` draws the line the bare `except` could not: it cannot tell + # "never reached `batched_put`" (must down the ref) from "reached it, it + # took ownership, then something raised" (must NOT down it). The second + # case is real -- an NVMe/L3 backend error or an eviction-path raise + # inside LMCache's put, after the CPU insert. Downing there drives the + # count to -1 / returns a still-indexed buffer to the allocator, and a + # later `get` for a DIFFERENT key hands that buffer back: wrong bytes + # unpacked into a request's Active Slot, no exception anywhere. Once the + # reference is passed to `batched_put`, ownership is its, success or raise. + handed_off = False try: self._staged.pack(self._backend.page_unit_views(unit_ids), obj) # Source first: `pack` has synchronized the stream that reads the # units, so nothing on the device touches them from here. if on_source_released is not None: on_source_released() + handed_off = True self._storage.batched_put([key], [obj]) except Exception: - obj.ref_count_down() + if not handed_off: + obj.ref_count_down() raise return True From bfa969e35ef8fd7911d62ff5906c1254698e7f9e Mon Sep 17 00:00:00 2001 From: zejchen Date: Wed, 2 Sep 2026 06:40:53 +0000 Subject: [PATCH 130/151] fix(offload): resolve offload layout off the text config, not the wrapper Round-5 review finding #7. `select_offload_layout` read `hf_config.model_type`, but two of the four names in `_GDN_LINEAR_MODEL_TYPES` -- qwen3_5_text, qwen3_5_moe_text -- are text-config model types, and the model-side predicates this selector mirrors (`ModelRunner.is_qwen_next`/`is_kimi_linear`) read `self.hf_text_config`. On the bare wrapper type, Qwen3.5 matched neither the kimi_linear nor the GDN branch, fell through to dense, and died deep in register_kv_caches with the DenseKVByteCodec byte-layout ValueError -- exactly the failure the refusal was added to pre-empt. Symmetrically a kimi_linear checkpoint wrapped in a text_config resolved to dense with the state tier silently off. Resolve model_type via get_hf_text_config(hf_config) so both the kimi_k3 selection and the GDN refusal see the unwrapped type. Co-Authored-By: Claude Opus 4.8 --- atom/kv_transfer/offload/config.py | 21 ++++++++++++++++++--- 1 file changed, 18 insertions(+), 3 deletions(-) diff --git a/atom/kv_transfer/offload/config.py b/atom/kv_transfer/offload/config.py index 394d395a10..0b11658523 100644 --- a/atom/kv_transfer/offload/config.py +++ b/atom/kv_transfer/offload/config.py @@ -110,9 +110,24 @@ def select_offload_layout(config) -> str: hf_config = getattr(config, "hf_config", None) if hf_config is None: return "dense" + # Resolve the family off the *text* config, exactly as the model-side + # predicates this mirrors do (`ModelRunner.is_qwen_next`/`is_kimi_linear` + # read `self.hf_text_config`). Two names in `_GDN_LINEAR_MODEL_TYPES` -- + # `qwen3_5_text`, `qwen3_5_moe_text` -- are literally text-config model + # types, so the bare wrapper `model_type` would be the multimodal wrapper's + # and neither the `kimi_linear` nor the GDN branch would fire: Qwen3.5 would + # fall through to `dense` and die deep in `register_kv_caches` (the exact + # failure the refusal below exists to pre-empt), and a `kimi_linear` + # checkpoint wrapped in a `text_config` would silently run with no state tier. + # Local import: this module is on the early config path, and `atom.utils` + # pulls in transformers -- keep it out of module import time. + from atom.utils import get_hf_text_config + + text_config = get_hf_text_config(hf_config) + model_type = getattr(text_config, "model_type", None) # K3's paged KV is ordinary dense MLA; what makes it its own family is the # KDA per-request state that a reusable prefix also needs. - if getattr(hf_config, "model_type", None) == "kimi_linear": + if model_type == "kimi_linear": return "kimi_k3" if getattr(hf_config, "compress_ratios", None): return "hybrid" @@ -121,10 +136,10 @@ def select_offload_layout(config) -> str: # output). The dense codec fails closed on the state tensor, but that fires # deep in `register_kv_caches` as a byte-layout mismatch -- refuse here where # the message can name the cause. - if getattr(hf_config, "model_type", None) in _GDN_LINEAR_MODEL_TYPES: + if model_type in _GDN_LINEAR_MODEL_TYPES: raise ValueError( "lmcache_offload does not support GDN/linear-attention models " - f"(model_type={getattr(hf_config, 'model_type', None)!r}; e.g. " + f"(model_type={model_type!r}; e.g. " "Qwen3-Next, Qwen3.5). These carry a per-request recurrent state " "that no offload layout owns a tier for -- restoring their KV " "prefix while that state is stale is silent wrong output. Disable " From 5e4edd5c59ab119a4185ef4b34ec66682b15274b Mon Sep 17 00:00:00 2001 From: zejchen Date: Wed, 2 Sep 2026 06:50:44 +0000 Subject: [PATCH 131/151] fix(offload): make the state-store pin two-phase so a source release does not read as unreported The store's source release fired from inside `codec.put` (after `pack`'s gather+D2H drain, before `batched_put`) and the engine drained it through `release_state_store_source` -> `settle_offload_store`, which popped the pin outright. `_offload_pins` carries two unrelated facts on one entry, and the pop falsified both the moment the gather drained rather than when the store reported: (a) `has_offload_pins()` went False while `_indexed`/`_index_failed` for that op had not been produced. `EngineCore.has_pending_kv_work()` could then read idle, the loop stopped calling `_poll_kv_transfer_progress` -- the only drain of `take_state_reports()` -- and the success report sat in the worker's set indefinitely. `note_stored` never ran, the checkpoint was never votable, and `stores_attempted - stores_completed` grew silently. (b) `_hash_in_flight` went False, so a second generation of the same hash could be dispatched while the first's CPU put was still running -- the concurrency the generation field is documented not to license. Split the pin into two phases via `_OffloadPin(checkpoint_id, pinned_at, source_released)`: - `release_offload_store_source(op)` (phase one) hands the PAGE units back the instant the gather drains -- holding an image out of the pool across the CPU put would cost reuse for nothing -- but leaves the op in `_offload_pins`, marked source-released, so `has_offload_pins`/`_hash_in_flight` keep seeing it. Idempotent. - `settle_offload_store(op)` (phase two) retires the op on the index report, and frees the units only as the backstop for a store that failed or reported before its source was released. `reclaim_stale_offload_pins` splits the same way: a source-released op that never reports is only a lost report -- the reader provably stopped, so its units are not released again (no underflow) and it is NOT forfeited (a late report may still index trustworthy bytes); a never-released op keeps the old forfeit path, since the worker may still be inside the gather. `BlockManager.release_state_store_source` now calls the phase-one method. Tests: source release frees units but keeps the pin; it is idempotent and never double-frees; a source-released lost report is reclaimed without forfeit. Co-Authored-By: Claude Opus 4.8 --- atom/model_engine/block_manager.py | 10 +- atom/model_engine/page_unit_checkpoint.py | 125 +++++++++++++++++----- tests/test_block_manager.py | 9 +- tests/test_page_unit_checkpoint.py | 46 ++++++++ 4 files changed, 162 insertions(+), 28 deletions(-) diff --git a/atom/model_engine/block_manager.py b/atom/model_engine/block_manager.py index da8a36f200..68a8ebe190 100644 --- a/atom/model_engine/block_manager.py +++ b/atom/model_engine/block_manager.py @@ -1413,9 +1413,17 @@ def release_state_store_source(self, op) -> None: drains, while whether the CPU put succeeded is decided afterwards and cannot touch them. Holding an image out of the pool across that would cost reuse for nothing. + + Phase one of the two-phase pin: it releases the units but leaves the + operation pinned as dispatched-but-unreported, so `has_offload_pins` + (and the liveness gate that ORs it in) keeps the engine polling until + the index report actually lands. Retiring the pin here -- as it once + did -- let liveness read idle while the store's report still sat + undrained in the worker, so `note_stored` never ran and the checkpoint + was never votable. """ if self.paged_state_checkpoints is not None: - self.paged_state_checkpoints.settle_offload_store(op) + self.paged_state_checkpoints.release_offload_store_source(op) def has_pending_state_store_pins(self) -> bool: """Whether a dispatched state store is still awaiting its report. diff --git a/atom/model_engine/page_unit_checkpoint.py b/atom/model_engine/page_unit_checkpoint.py index 42b56504fa..ddfcc9027a 100644 --- a/atom/model_engine/page_unit_checkpoint.py +++ b/atom/model_engine/page_unit_checkpoint.py @@ -116,6 +116,26 @@ class CheckpointRecord: pin_count: int = 0 +@dataclass +class _OffloadPin: + """A store that has been dispatched to the worker but not yet indexed. + + Two-phase, because a store's life has two independent ends. The PAGE units + it gathers are the KV pool's and are free the moment the D2H gather drains + (`source_released`); whether the CPU `batched_put` then succeeded is decided + strictly afterwards and cannot touch them. The pin therefore holds the units + only until the source is released, but the *record itself* stays in + `_offload_pins` until the index report lands -- so `has_offload_pins` and + `_hash_in_flight`, which mean "dispatched but unreported", keep answering + from dispatch all the way to the report, across the gap the source release + opens in the middle. + """ + + checkpoint_id: int + pinned_at: float + source_released: bool = False + + class PageUnitCheckpointStore: """Content index and ownership table for split state images.""" @@ -158,7 +178,8 @@ def __init__( # any real in-flight-plus-waiting count so it only bites a runaway. self._offload_backlog_cap = 8192 self.offload_nominations_dropped = 0 - # StateStoreOperationId -> (checkpoint_id, monotonic when pinned). + # StateStoreOperationId -> _OffloadPin (checkpoint, when pinned, whether + # the source units are already back). # A store's report carries no request -- by the time one lands its owner # is long gone -- so the hash is the only thing left to name it by. The # hash ALONE is not an identity though: the same prefix is stored again @@ -169,7 +190,7 @@ def __init__( # generation separates the attempts, and keying the pins by the whole # operation is also what stops a late report from an earlier attempt # from settling a newer pin for the same hash. - self._offload_pins: dict[StateStoreOperationId, tuple[int, float]] = {} + self._offload_pins: dict[StateStoreOperationId, _OffloadPin] = {} self._offload_generation = 0 self.offload_pins_reclaimed = 0 # Operations whose pin the stale reclaimer took back. Bounded, because @@ -475,7 +496,7 @@ def take_offload_stores( int(record.prefix_hash), self._offload_generation ) record.pin_count += 1 - self._offload_pins[op] = (checkpoint_id, monotonic()) + self._offload_pins[op] = _OffloadPin(checkpoint_id, monotonic()) out.append((op, record.unit_ids)) # Re-queue the deferred nominations for the next drain, once the # colliding in-flight generation has had a chance to settle. The drain @@ -497,36 +518,63 @@ def _hash_in_flight(self, prefix_hash: int) -> bool: """ return any(op.prefix_hash == prefix_hash for op in self._offload_pins) + def release_offload_store_source(self, op: StateStoreOperationId) -> None: + """Phase one: the gather drained, so hand the PAGE units back now. + + This is the earlier of a store's two ends. It returns the units to the + KV pool the instant the D2H copy is done reading them -- holding an + image out of the pool across the subsequent CPU `batched_put` would cost + reuse for a step that cannot touch the units. But it does NOT remove the + operation from `_offload_pins`: the store is still dispatched-but- + unreported, so `has_offload_pins` and `_hash_in_flight` must keep + seeing it until the index report lands. It only marks the source gone, + so the report (or a reclaim) does not release the same units twice. + + Idempotent, and matched on the whole operation: a release for a + generation no longer here -- already reported, or reclaimed -- does + nothing. + """ + pin = self._offload_pins.get(op) + if pin is None or pin.source_released: + return + pin.source_released = True + self._release_offload_pin(pin.checkpoint_id) + def settle_offload_store(self, op: StateStoreOperationId) -> None: - """One store reported, either way. Release the units it was holding. + """Phase two: the store reported, either way. Retire the operation. - Success and failure release identically: the pin exists to keep the - bytes still during the copy, and the copy is over either way. Whether - the hash becomes reachable is `StateOffloadIndex`'s business, not this - one's. + Removes the pin so `has_offload_pins`/`_hash_in_flight` stop seeing this + operation -- the report is the true end of "dispatched but unreported". + The units are normally already back (`release_offload_store_source` ran + first); this is the backstop for a store that failed or reported before + its source was released, so the units are freed here only when they were + not freed there. Whether the hash becomes reachable is + `StateOffloadIndex`'s business, not this one's. Matched on the whole operation: a report from an attempt whose pin was already reclaimed names a generation no longer here and settles nothing, rather than releasing a newer attempt's pin. """ - entry = self._offload_pins.pop(op, None) - if entry is None: + pin = self._offload_pins.pop(op, None) + if pin is None: return - self._release_offload_pin(entry[0]) + if not pin.source_released: + self._release_offload_pin(pin.checkpoint_id) def has_offload_pins(self) -> bool: - """Whether any dispatched store is still holding its PAGE units. + """Whether any dispatched store is still awaiting its index report. A pin lives here from the moment `take_offload_stores` hands a store to - the worker until either report path settles it (`settle_offload_store`, - driven by the source-release and index completions) or the last-resort - `reclaim_stale_offload_pins` times it out. It is therefore true exactly - while a store is dispatched-but-unreported, which is the interval the - engine must keep polling across: the completion that clears the pin -- - and the reclaim that is its only other exit -- both run only from - `_poll_kv_transfer_progress`, which the idle loop skips once liveness - reads False. Because reclaim shares this same dict, the signal cannot - latch the busy loop: a lost report clears here when the reclaim fires. + the worker until `settle_offload_store` retires it on the index report, + or the last-resort `reclaim_stale_offload_pins` times it out. The source + release in between hands the PAGE units back but leaves the pin, so this + stays true across it: it is therefore true exactly while a store is + dispatched-but-unreported, which is the interval the engine must keep + polling across -- the report that clears the pin, and the reclaim that + is its only other exit, both run only from `_poll_kv_transfer_progress`, + which the idle loop skips once liveness reads False. Because reclaim + shares this same dict, the signal cannot latch the busy loop: a lost + report clears here when the reclaim fires. """ return bool(self._offload_pins) @@ -562,14 +610,35 @@ def reclaim_stale_offload_pins(self, timeout_s: float) -> int: if timeout_s <= 0 or not self._offload_pins: return 0 cutoff = monotonic() - timeout_s - stale = [op for op, (_cid, at) in self._offload_pins.items() if at <= cutoff] + stale = [ + op for op, pin in self._offload_pins.items() if pin.pinned_at <= cutoff + ] for op in stale: - checkpoint_id, _at = self._offload_pins.pop(op) - self._release_offload_pin(checkpoint_id) + pin = self._offload_pins.pop(op) + self.offload_pins_reclaimed += 1 + if pin.source_released: + # The gather already drained and the units are back; only the + # index report was lost. Nothing to release again -- doing so + # would underflow the record -- and nothing to forfeit: the + # bytes are trustworthy because the reader provably stopped, so + # a late report may still index the hash. This is a lost report, + # not a taken-back source. + logger.warning( + "state offload: store %s never reported after %.1fs; its " + "units were already returned at source release, so only " + "the index report is lost.", + op, + timeout_s, + ) + continue + # The source was never released, so the worker may still be inside + # the gather. Taking the units back now means they cannot be + # trusted: the pool may hand them to another request whose writes + # this gather then picks up. Release and forfeit the entry. + self._release_offload_pin(pin.checkpoint_id) self._offload_reclaimed[op] = None while len(self._offload_reclaimed) > _RECLAIMED_MEMORY: self._offload_reclaimed.popitem(last=False) - self.offload_pins_reclaimed += 1 logger.warning( "state offload: reclaimed the units of store %s after %.1fs " "with no report. The worker may still be reading them, so the " @@ -877,8 +946,12 @@ def take_offload_stores( """`(operation, unit_ids)` to hand the tier now. See the store.""" return self.store.take_offload_stores(max_inflight) + def release_offload_store_source(self, op: StateStoreOperationId) -> None: + """The gather drained; hand the units back but keep the pin. See store.""" + self.store.release_offload_store_source(op) + def settle_offload_store(self, op: StateStoreOperationId) -> None: - """One store reported, either way; release the units it was holding.""" + """One store reported, either way; retire the operation. See store.""" self.store.settle_offload_store(op) def has_offload_pins(self) -> bool: diff --git a/tests/test_block_manager.py b/tests/test_block_manager.py index 448cc048d7..99df2fd762 100644 --- a/tests/test_block_manager.py +++ b/tests/test_block_manager.py @@ -963,6 +963,7 @@ class _Coordinator: def __init__(self, reclaimed): self._reclaimed = reclaimed self.settled = [] + self.released = [] def was_reclaimed(self, op): return op in self._reclaimed @@ -970,6 +971,9 @@ def was_reclaimed(self, op): def settle_offload_store(self, op): self.settled.append(op) + def release_offload_store_source(self, op): + self.released.append(op) + def _bm(self, reclaimed=()): from atom.model_engine.state_offload import StateOffloadIndex @@ -1012,7 +1016,10 @@ def test_the_source_release_is_what_normally_returns_them(self): bm = self._bm() op = StateStoreOperationId(11, 1) bm.release_state_store_source(op) - assert bm.paged_state_checkpoints.settled == [op] + # Phase one hands the units back but leaves the pin in place, so the + # store still reads as dispatched-but-unreported until its report lands. + assert bm.paged_state_checkpoints.released == [op] + assert bm.paged_state_checkpoints.settled == [], "the pin is not retired" # ...and it does not touch the index, which the store report owns. assert bm.state_offload.hashes == set() diff --git a/tests/test_page_unit_checkpoint.py b/tests/test_page_unit_checkpoint.py index 93799a18c8..7c08b0a839 100644 --- a/tests/test_page_unit_checkpoint.py +++ b/tests/test_page_unit_checkpoint.py @@ -481,6 +481,52 @@ def test_a_report_for_a_hash_never_sent_is_a_no_op(self): store.settle_offload_store(StateStoreOperationId(11, 1)) assert store.records[cid].pin_count == 0 + def test_source_release_frees_units_but_keeps_the_pin(self): + """Phase one: the gather drained, so the units go back -- but the store + is still dispatched-but-unreported, so the engine must keep polling for + its report. Retiring the pin here (as it once did) let liveness read + idle while the report still sat undrained, so it was never votable.""" + _pool, store = offload_store() + cid, _op = ready(store, 11) + [(sent, _u)] = store.take_offload_stores(max_inflight=4) + + store.release_offload_store_source(sent) + assert store.records[cid].pin_count == 0, "no longer pinned for the copy" + assert store._is_evictable(cid), "the units are the pool's again" + assert store.has_offload_pins(), "still unreported: keep polling" + assert store._hash_in_flight(11), "no second store of 11 yet" + + store.settle_offload_store(sent) + assert not store.has_offload_pins() + assert not store._hash_in_flight(11) + + def test_source_release_is_idempotent_and_does_not_double_free(self): + """A second release -- or a release after the report -- must not + decrement the record a second time and underflow the pin count.""" + _pool, store = offload_store() + cid, _op = ready(store, 11) + [(sent, _u)] = store.take_offload_stores(max_inflight=4) + store.release_offload_store_source(sent) + store.release_offload_store_source(sent) # idempotent + store.settle_offload_store(sent) + store.release_offload_store_source(sent) # after the report: no pin left + assert store.records[cid].pin_count == 0 + + def test_a_source_released_store_that_never_reports_is_reclaimed_but_kept(self): + """Its units are already back and the reader provably stopped, so a lost + report is not the taken-back-source case: nothing is released again (no + underflow) and the image is NOT forfeited -- a late report may index.""" + _pool, store = offload_store() + cid, _op = ready(store, 11) + [(sent, _u)] = store.take_offload_stores(max_inflight=4) + store.release_offload_store_source(sent) + assert store.records[cid].pin_count == 0 + + assert store.reclaim_stale_offload_pins(timeout_s=1e-9) == 1 + assert store.records[cid].pin_count == 0, "not released twice" + assert not store.was_reclaimed(sent), "source-released bytes are trusted" + assert store.offload_pins_reclaimed == 1 + def test_unindex_during_the_copy_holds_the_units_to_the_end(self): """The tier's whole point: the CPU copy outliving the HBM one. So `unindex` must not pull the source out from under a running D2H.""" From ef79437799b3c29c81f3de104d89fd825082b8d2 Mon Sep 17 00:00:00 2001 From: zejchen Date: Wed, 2 Sep 2026 06:56:19 +0000 Subject: [PATCH 132/151] fix(offload): park the orphan load slot in one dict of (slot, stamp) An orphan load slot -- one whose request was deallocated before its state bytes landed -- was tracked across two dicts keyed the same way, `_orphan_load_slots` (req_id -> slot) and `_orphan_load_slots_at` (req_id -> stamp), assigned and popped in pairs at four sites. Two dicts on one key are a desync waiting to happen: `reconcile_orphan_load_slots` iterated the stamp dict while the release paths read the slot dict, so any state where the two disagreed -- a half-applied pop, or a future edit touching one and not the other -- would let the reconciler miss a slot that was still off the `StateSlotPool` free list. Enough stranded slots and `can_allocate`'s state gate refuses every hybrid admission: the pool wedge the reconciler exists to prevent, reached through a gap in its own bookkeeping. Collapse to one dict of `(slot, stamp)`. An entry is now present with its slot and stamp together or absent entirely -- the two can no longer disagree -- and every site mutates a single key atomically. (This does not by itself cover a same-req_id double-park: `seq.id` is per-request, so a preempt/re-admit can park a second slot under an id whose first load is still in flight, and either representation overwrites the first. That residual is called out separately for follow-up; one dict removes the desync class, which is what this change is scoped to.) Tests cover settle/abandon/reconcile freeing the parked slot and clearing the entry, the window guard sparing a fresh park, and a late report after reconcile releasing nothing. Co-Authored-By: Claude Opus 4.8 --- atom/model_engine/block_manager.py | 56 +++++++++---------- tests/test_block_manager.py | 89 ++++++++++++++++++++++++++++++ 2 files changed, 117 insertions(+), 28 deletions(-) diff --git a/atom/model_engine/block_manager.py b/atom/model_engine/block_manager.py index 68a8ebe190..da05e44941 100644 --- a/atom/model_engine/block_manager.py +++ b/atom/model_engine/block_manager.py @@ -317,19 +317,21 @@ def __init__( # the connector. Kept here rather than in the index because the slot # is this object's fact. self._state_loads: list[tuple] = [] - # req_id -> slot, for loads whose request was deallocated before the - # bytes landed. The slot is off the free list until the report comes - # back; see `deallocate`. - self._orphan_load_slots: dict = {} - # req_id -> monotonic stamp, parallel to `_orphan_load_slots`. The hold - # above trusts `settle_state_load` to "always come": it does not, if the - # worker that owed the report crashed or its completion was dropped. Then - # the slot sits off the free list forever and the state gate in - # `can_allocate` wedges the pool once every slot is stranded. This stamps - # the wait so `reconcile_orphan_load_slots` can reclaim it after the same - # abandon window the store pins use -- the load-side twin of + # req_id -> (slot, monotonic stamp), for loads whose request was + # deallocated before the bytes landed. The slot is off the free list + # until the report comes back; see `deallocate`. One dict, not two + # parallel ones keyed the same way: `seq.id` is per-request not + # per-admission, so a preempt/re-admit can park the same id twice, and + # two dicts let an overwrite of one desync from the other -- stranding a + # slot the reconciler can no longer find. The stamp trusts + # `settle_state_load` to "always come": it does not, if the worker that + # owed the report crashed or its completion was dropped. Then the slot + # would sit off the free list forever and the state gate in + # `can_allocate` would wedge the pool once every slot is stranded, so + # `reconcile_orphan_load_slots` reclaims it after the same abandon + # window the store pins use -- the load-side twin of # `reclaim_stale_state_store_pins`. - self._orphan_load_slots_at: dict = {} + self._orphan_load_slots: dict[object, tuple[int, float]] = {} self._orphan_load_slots_reclaimed: int = 0 # `kv_offload_enabled` is the whole switch: without a connector to # carry a transfer there is nothing beneath the pool, and a hash whose @@ -1312,10 +1314,9 @@ def abandon_state_load(self, req_id) -> None: if self.state_offload is None: return self.state_offload.abandon_load(req_id) - slot = self._orphan_load_slots.pop(req_id, None) - self._orphan_load_slots_at.pop(req_id, None) - if slot is not None: - self.state.release(slot) + parked = self._orphan_load_slots.pop(req_id, None) + if parked is not None: + self.state.release(parked[0]) def settle_state_load(self, req_id, ok: bool) -> None: """Apply one worker load report. Keyed by request, like the KV load. @@ -1332,10 +1333,9 @@ def settle_state_load(self, req_id, ok: bool) -> None: self.state_offload.complete_load(req_id) else: self.state_offload.fail_load(req_id) - slot = self._orphan_load_slots.pop(req_id, None) - self._orphan_load_slots_at.pop(req_id, None) - if slot is not None: - self.state.release(slot) + parked = self._orphan_load_slots.pop(req_id, None) + if parked is not None: + self.state.release(parked[0]) def take_state_stores(self, max_inflight: int) -> list[tuple]: """`(operation, unit_ids)` for checkpoints to hand the CPU tier. @@ -1469,17 +1469,18 @@ def reconcile_orphan_load_slots(self, timeout_s: float) -> int: index leg was abandoned back in `deallocate`, so nothing here touches it; a late report finds the slot already popped and releases nothing. """ - if timeout_s <= 0 or not self._orphan_load_slots_at: + if timeout_s <= 0 or not self._orphan_load_slots: return 0 cutoff = monotonic() - timeout_s stale = [ - req_id for req_id, at in self._orphan_load_slots_at.items() if at <= cutoff + req_id + for req_id, (_slot, at) in self._orphan_load_slots.items() + if at <= cutoff ] for req_id in stale: - self._orphan_load_slots_at.pop(req_id, None) - slot = self._orphan_load_slots.pop(req_id, None) - if slot is not None: - self.state.release(slot) + parked = self._orphan_load_slots.pop(req_id, None) + if parked is not None: + self.state.release(parked[0]) # The index leg was already abandoned in `deallocate` when the slot # was parked; only the slot itself outlived the report. Do not touch # the index again here -- a second `abandon_load` would double-count @@ -2466,8 +2467,7 @@ def deallocate(self, seq: Sequence): # comes: the worker reports every load either way. The rollback # scratch is nobody's destination and goes back regardless. if seq.offload_joint.load_hash != -1 and self.state_offload is not None: - self._orphan_load_slots[seq.id] = seq.state_slot - self._orphan_load_slots_at[seq.id] = monotonic() + self._orphan_load_slots[seq.id] = (seq.state_slot, monotonic()) # Abandoned, not failed: an abort says nothing about the bytes, # and forgetting the hash would cost the next request over this # prefix a full recompute. diff --git a/tests/test_block_manager.py b/tests/test_block_manager.py index 99df2fd762..5efb28932d 100644 --- a/tests/test_block_manager.py +++ b/tests/test_block_manager.py @@ -1024,6 +1024,95 @@ def test_the_source_release_is_what_normally_returns_them(self): assert bm.state_offload.hashes == set() +# ── the orphan load slot lives in one dict of (slot, stamp) ──────────────── + + +class TestOrphanLoadSlotSingleDict: + """A load slot whose request was torn down before its bytes landed is + parked in one dict of `(slot, stamp)`. It used to live across two parallel + dicts keyed the same way, mutated in pairs at four sites -- a shape where an + overwrite or a half-applied pop could desync them, so the reconciler + (iterating the stamp dict) and the release (reading the slot dict) could + disagree and strand a slot off the pool free list. One dict makes that + unrepresentable: an entry is present with its slot and stamp together, or it + is gone.""" + + class _Pool: + def __init__(self): + self.released = [] + + def release(self, slot): + self.released.append(slot) + + class _Index: + def __init__(self): + self.abandoned, self.completed, self.failed = [], [], [] + + def abandon_load(self, req_id): + self.abandoned.append(req_id) + + def complete_load(self, req_id): + self.completed.append(req_id) + + def fail_load(self, req_id): + self.failed.append(req_id) + + def _bm(self): + bm = object.__new__(BlockManager) + bm.state = self._Pool() + bm.state_offload = self._Index() + bm._orphan_load_slots = {} + bm._orphan_load_slots_reclaimed = 0 + return bm + + def test_settle_frees_the_parked_slot_and_clears_the_entry(self): + from time import monotonic + + bm = self._bm() + bm._orphan_load_slots["r"] = (7, monotonic()) + bm.settle_state_load("r", ok=True) + assert bm.state.released == [7] + assert "r" not in bm._orphan_load_slots + assert bm.state_offload.completed == ["r"] + + def test_abandon_frees_the_parked_slot(self): + from time import monotonic + + bm = self._bm() + bm._orphan_load_slots["r"] = (7, monotonic()) + bm.abandon_state_load("r") + assert bm.state.released == [7] + assert "r" not in bm._orphan_load_slots + + def test_reconcile_frees_a_slot_whose_report_never_came(self): + from time import monotonic + + bm = self._bm() + bm._orphan_load_slots["r"] = (7, monotonic()) + assert bm.reconcile_orphan_load_slots(timeout_s=1e-9) == 1 + assert bm.state.released == [7] + assert bm._orphan_load_slots_reclaimed == 1 + assert "r" not in bm._orphan_load_slots + + def test_reconcile_spares_a_slot_still_inside_its_window(self): + from time import monotonic + + bm = self._bm() + bm._orphan_load_slots["r"] = (7, monotonic()) + assert bm.reconcile_orphan_load_slots(timeout_s=3600) == 0 + assert bm.state.released == [] + assert "r" in bm._orphan_load_slots + + def test_a_late_report_after_reconcile_releases_nothing(self): + from time import monotonic + + bm = self._bm() + bm._orphan_load_slots["r"] = (7, monotonic()) + bm.reconcile_orphan_load_slots(timeout_s=1e-9) + bm.settle_state_load("r", ok=True) # the report finally arrives + assert bm.state.released == [7], "freed once by reconcile, not twice" + + # ── the LMCache chunk probe is gated on the capability ───────────────────── From ca4708a0987ebaf17eefd55d2b442b27077c6add Mon Sep 17 00:00:00 2001 From: zejchen Date: Wed, 2 Sep 2026 07:11:11 +0000 Subject: [PATCH 133/151] fix(offload): give _JointPark an expiry/eviction exit for lost reports Finding #3 (review round 5): `_JointPark` had `arm`/`settle_*`/`waits_for`/ `take_ready` and no way out for a park whose report never comes. A joint KV+state load is released only when both legs report. Three cases strand one leg forever: - the request is aborted mid-load: the KV leg is cancelled scheduler-side (`request_finished`/`cancel_pending_load` on the scheduler connector) and never reports, while the state tier's leg still lands (its worker always ends in `_done`/`_failed`) -- but half a report cannot release the pair; - a worker thread is killed mid-transfer; - a completion is dropped between worker and engine. The key then sat in `_need` (with its `_alias`/`_alias_of`) for the life of the process, and worse: `_settle_joint` swallowed every later KV completion that reused the stale `kv_id` (`waits_for` True -> `settle` consumes it -> `take_ready` returns nothing), wedging that request in WAITING_FOR_REMOTE_KVS with its blocks held. Nothing engine-side rescued it -- `reconcile_orphan_ load_slots` frees only the slot, `_reconcile_stalled_deferred_saves` only saves. The maintainer asked for the abort/expiry/eviction exit the KV leg's `_active_load_operations` and DSV4's slot reservations already have. The park is worker-side and the abort signal is scheduler-side with no synchronous path to it, so the exit is an expiry sweep, not a cross-side abort call: - `_JointPark.reclaim_stale_parks(max_age_s)` stamps each key at `arm` time (`_armed_at`, cleared in `_purge`/`_release`) and `_purge`-evicts any armed longer than the window ago -- no ready/failed emitted, so an evicted load never masquerades as completed; - `KimiK3OffloadConnector.get_finished` sweeps it every step (its one per-step worker driver), on LMCache's own `offload_save_abandon_timeout_s` -- the same clock the engine's pin and orphan-slot reconcilers use, and `<= 0` disables it exactly as they do. The second shape the finding raises -- a re-admitted state-only request whose stale `_alias[req_id]` mis-resolves a new park -- is already covered by `arm`'s pre-file `_purge` (added earlier this round), so it is not re-fixed here. A report that truly arrives after eviction finds its key gone and passes through `_settle_joint` as an unarmed id, dropped for the long-departed request rather than manufacturing a restore. Tests: TestJointParkExpiry -- window eviction spares fresh parks, a reused kv_id wakes a new load instead of being swallowed, a post-reclaim report passes through, and the worker sweep derives its window from the save-abandon timeout and disables on `<= 0`. Co-Authored-By: Claude Opus 4.8 --- .../offload/hybrid/kimi_k3/connector.py | 44 ++++++ .../offload/hybrid/kimi_k3/state_tier.py | 40 +++++ tests/test_lmcache_offload_connector.py | 137 ++++++++++++++++++ 3 files changed, 221 insertions(+) diff --git a/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py b/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py index d5fa57c33e..b006bb215f 100644 --- a/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py +++ b/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py @@ -422,6 +422,13 @@ def _fail_state_loads(self, loads) -> None: # -- completions ------------------------------------------------------- def get_finished(self): out = super().get_finished() + # Sweep parks whose report never came before draining this step's, on + # every path including no-tier (finding #3). Cheap: `_armed_at` holds + # only in-flight joint loads and is usually empty. Ordering-independent + # -- a park stale enough to evict is one whose window already elapsed, so + # a report arriving this same step for it is exactly the late-report case + # `reclaim_stale_parks` is built to pass through harmlessly. + self._reclaim_stale_parks() # No-tier store failures must reach the engine even when the tier never # built -- the engine pinned their PAGE units and is holding them # against a report. Emit the tier's own failure pairing: index-failed so @@ -524,6 +531,43 @@ def _emit_load_dispositions(self, out) -> None: ) ) + def _reclaim_stale_parks(self) -> None: + """Evict joint parks whose report never came (finding #3). + + A joint park is released only by both legs reporting. Three cases leave + one leg forever unreported: the request is aborted mid-load (the KV leg + is cancelled scheduler-side and never reports, while the state tier's leg + still lands -- half a report cannot release the pair), a worker thread is + killed mid-transfer, or a completion is dropped between worker and + engine. Without an exit the key -- and the `_alias`/`_alias_of` under it + -- sits in `_need` for the life of the process, and worse, `_settle_ + joint` swallows every later KV completion that reuses the stale `kv_id`, + wedging that request in WAITING_FOR_REMOTE_KVS with its blocks held. + + The park is worker-side, so it is swept here -- `get_finished` is its one + per-step worker driver. The abort signal itself is scheduler-side + (`request_finished`/`cancel_pending_load` on the scheduler connector, + which cannot reach this object), so an expiry sweep, not a synchronous + abort call, is what closes the gap; the same-request re-admission shape + is already handled by `arm`'s pre-file `_purge`. The window is LMCache's + own save-abandon timeout -- the same clock the engine's pin and orphan- + slot reconcilers use -- and shares their caveat: a report that truly + arrives after it finds the key gone and passes through `_settle_joint` + harmlessly for the long-departed request. `<= 0` disables reclamation + (the operator turned the pin timeout off), matching those reconcilers. + """ + window = offload_save_abandon_timeout_s() + if window <= 0: + return + reclaimed = self._joint_park.reclaim_stale_parks(window) + if reclaimed: + logger.warning( + "kimi_k3 offload: reclaimed %d joint load park(s) whose report " + "never came (aborted request, killed worker, or dropped " + "completion); those requests recompute.", + reclaimed, + ) + class KimiK3OffloadScheduler(DenseOffloadScheduler, StateOffloadFace): """Scheduler side: dense KV, plus the state tier's load queue and the diff --git a/atom/kv_transfer/offload/hybrid/kimi_k3/state_tier.py b/atom/kv_transfer/offload/hybrid/kimi_k3/state_tier.py index e27541ec4b..309dfa9252 100644 --- a/atom/kv_transfer/offload/hybrid/kimi_k3/state_tier.py +++ b/atom/kv_transfer/offload/hybrid/kimi_k3/state_tier.py @@ -313,6 +313,14 @@ def __init__(self) -> None: # bare id onto it so a state report can find its own park. self._alias: dict = {} self._alias_of: dict = {} + # Monotonic stamp per armed key, for `reclaim_stale_parks` (finding #3). + # A park is only ever released by both legs reporting; a report lost to + # an aborted request, a killed worker thread, or a dropped completion + # would otherwise leave the key in `_need`/`_alias`/`_alias_of` for the + # life of the process -- and `_settle_joint` would swallow every later + # KV completion that reuses the stale `kv_id`. `abort` is the prompt + # exit; this stamp is the backstop the reconciler sweeps. + self._armed_at: dict = {} def arm( self, @@ -347,6 +355,7 @@ def arm( need.add("state") key = req_id if kv_id is None else kv_id self._need[key] = need + self._armed_at[key] = monotonic() if key != req_id: self._alias[req_id] = key self._alias_of[key] = req_id @@ -363,6 +372,7 @@ def _purge(self, key) -> None: if key is None or key not in self._need: return self._need.pop(key, None) + self._armed_at.pop(key, None) bare = self._alias_of.pop(key, None) if bare is not None: self._alias.pop(bare, None) @@ -372,6 +382,35 @@ def _purge(self, key) -> None: self._ready_failed.discard(key) self._dispositions.pop(key, None) + def reclaim_stale_parks(self, max_age_s: float) -> int: + """Evict parks armed longer than `max_age_s` ago; return the count. + + The park's abort/expiry/eviction exit (finding #3). A joint park is + released only by both legs reporting; a report lost to an aborted request + (the KV leg cancelled scheduler-side, never reporting, while the state + tier's leg still lands -- half a report cannot release the pair), a + worker thread killed mid-transfer, or a completion dropped between worker + and engine would otherwise leave the key in `_need` (with its + `_alias`/`_alias_of`) forever, and `_settle_joint` would swallow every + later KV completion reusing that stale `kv_id`. The abort signal is + scheduler-side and cannot reach this worker-side park synchronously, so + expiry, not a synchronous abort call, is the exit; the same-request + re-admission shape is separately handled by `arm`'s pre-file `_purge`. + + Like the engine's pin and orphan-slot reconcilers it cannot tell a lost + report from a merely slow one, so the caller passes the same + save-abandon window; a park that outlives it is treated as never going + to complete. Eviction (no ready/failed) is safe: a load whose report + truly arrives after this finds its key gone and passes through + `_settle_joint` as an unarmed id, which the engine drops for the + long-departed request rather than manufacturing a restore. + """ + cutoff = monotonic() - max_age_s + stale = [key for key, at in self._armed_at.items() if at <= cutoff] + for key in stale: + self._purge(key) + return len(stale) + def settle_kv(self, ident, ok: bool) -> None: self._settle(ident, "kv", ok) @@ -402,6 +441,7 @@ def _settle(self, ident, leg: str, ok: bool) -> None: def _release(self, key) -> None: self._need.pop(key, None) + self._armed_at.pop(key, None) bare = self._alias_of.pop(key, None) if bare is not None: self._alias.pop(bare, None) diff --git a/tests/test_lmcache_offload_connector.py b/tests/test_lmcache_offload_connector.py index 29a23179c4..5b5d125c2a 100644 --- a/tests/test_lmcache_offload_connector.py +++ b/tests/test_lmcache_offload_connector.py @@ -5691,6 +5691,143 @@ def test_either_leg_failing_fails_the_pair(self): assert failed == {req.load_operation} +class TestJointParkExpiry: + """`_JointPark`'s abort/expiry/eviction exit (finding #3). + + A joint park is released only by both legs reporting. Three cases leave one + leg forever unreported: the request is aborted mid-load (the KV leg is + cancelled scheduler-side and never reports, while the state tier's leg still + lands -- half a report cannot release the pair), a worker thread is killed + mid-transfer, or a completion is dropped. Without an exit the key -- with its + `_alias`/`_alias_of` -- sat in `_need` for the process's life, and worse, + `_settle_joint` swallowed every later KV completion reusing the stale + `kv_id`. The abort signal is scheduler-side and cannot reach this worker-side + park, so `reclaim_stale_parks` (swept from `get_finished`, on LMCache's + save-abandon window) is the exit; re-admission is handled by `arm`'s purge. + """ + + def test_reclaim_evicts_parks_past_the_window_and_spares_fresh_ones( + self, monkeypatch + ): + clock = {"t": 1000.0} + monkeypatch.setattr( + "atom.kv_transfer.offload.hybrid.kimi_k3.state_tier.monotonic", + lambda: clock["t"], + ) + park = _JointPark() + old = LoadOperationId("old", 0) + park.arm("old", needs_kv=True, needs_state=True, kv_id=old) + clock["t"] = 1100.0 # 100s later + fresh = LoadOperationId("fresh", 0) + park.arm("fresh", needs_kv=True, needs_state=True, kv_id=fresh) + + clock["t"] = 1100.0 + 30.0 # 130s: old is 130s stale, fresh 30s + evicted = park.reclaim_stale_parks(60.0) + + assert evicted == 1 + assert not park.waits_for(old) + assert park.waits_for(fresh) + # Eviction, not settlement: no ready/failed manufactured for `old`. + assert park.take_ready() == (set(), set()) + assert "old" not in park._alias and old not in park._alias_of + + def test_a_report_after_reclaim_passes_through_instead_of_wedging( + self, monkeypatch + ): + clock = {"t": 500.0} + monkeypatch.setattr( + "atom.kv_transfer.offload.hybrid.kimi_k3.state_tier.monotonic", + lambda: clock["t"], + ) + park = _JointPark() + kv_id = LoadOperationId("r1", 0) + park.arm("r1", needs_kv=True, needs_state=True, kv_id=kv_id) + clock["t"] = 500.0 + 120.0 + assert park.reclaim_stale_parks(60.0) == 1 + + # The lost report finally lands: the key is gone, so it is not held. + assert not park.waits_for("r1") + + def test_reclaim_lets_a_reused_kv_id_wake_a_new_load(self, monkeypatch): + """The wedge finding #3 names: a stranded park swallowing later loads. + + Age a park past the window and reclaim it, then let a fresh load reuse + the same `LoadOperationId`. Its KV completion must pass through as a real + wake, not be absorbed by a surviving `waits_for` from the lost load. + """ + clock = {"t": 0.0} + monkeypatch.setattr( + "atom.kv_transfer.offload.hybrid.kimi_k3.state_tier.monotonic", + lambda: clock["t"], + ) + worker = _k3_worker() + req = _k3_load_req("r1") + worker._arm_joint_loads( + SimpleNamespace(state_loads=[("r1", 99, 0)], requests=[req]) + ) + assert worker._joint_park.waits_for(req.load_operation) + + clock["t"] = 999.0 # long past any window + assert worker._joint_park.reclaim_stale_parks(60.0) == 1 + assert not worker._joint_park.waits_for(req.load_operation) + + # A later load reuses the same operation id (id reuse across admissions). + done, failed = worker._settle_joint({req.load_operation}, set(), set(), set()) + assert done == {req.load_operation}, "reused kv_id must not be swallowed" + assert failed == set() + + def test_worker_reclaim_uses_the_save_abandon_window(self, monkeypatch): + """`get_finished`'s sweep derives its window from LMCache's own + save-abandon timeout, not a hardcoded constant.""" + clock = {"t": 0.0} + monkeypatch.setattr( + "atom.kv_transfer.offload.hybrid.kimi_k3.state_tier.monotonic", + lambda: clock["t"], + ) + monkeypatch.setattr( + "atom.kv_transfer.offload.hybrid.kimi_k3.connector." + "offload_save_abandon_timeout_s", + lambda: 60.0, + ) + worker = _k3_worker() + req = _k3_load_req("r1") + worker._arm_joint_loads( + SimpleNamespace(state_loads=[("r1", 99, 0)], requests=[req]) + ) + + clock["t"] = 30.0 # inside the window: spared + worker._reclaim_stale_parks() + assert worker._joint_park.waits_for(req.load_operation) + + clock["t"] = 90.0 # past the 60s window: evicted + worker._reclaim_stale_parks() + assert not worker._joint_park.waits_for(req.load_operation) + + def test_worker_reclaim_is_disabled_when_the_window_is_nonpositive( + self, monkeypatch + ): + """`<= 0` means the operator turned the pin timeout off; matching the + engine's pin/orphan-slot reconcilers, the sweep then does nothing.""" + clock = {"t": 0.0} + monkeypatch.setattr( + "atom.kv_transfer.offload.hybrid.kimi_k3.state_tier.monotonic", + lambda: clock["t"], + ) + monkeypatch.setattr( + "atom.kv_transfer.offload.hybrid.kimi_k3.connector." + "offload_save_abandon_timeout_s", + lambda: 0.0, + ) + worker = _k3_worker() + req = _k3_load_req("r1") + worker._arm_joint_loads( + SimpleNamespace(state_loads=[("r1", 99, 0)], requests=[req]) + ) + clock["t"] = 10_000.0 # arbitrarily old + worker._reclaim_stale_parks() + assert worker._joint_park.waits_for(req.load_operation) + + class _RecordingExecutor: def __init__(self, log: list, name: str) -> None: self._log = log From 9b5b1bf4ad91ca48610d697288dc5695dbd9f546 Mon Sep 17 00:00:00 2001 From: zejchen Date: Wed, 2 Sep 2026 07:24:33 +0000 Subject: [PATCH 134/151] fix(offload): one consistent answer for the [dense, kimi_k3] composite Findings 0, 1 and 6 (review round 5) are three views of the same composite: the multi-routing rework did not agree with itself about whether two offload sub-connectors are legal, and each disagreement was its own bug. Finding 0 -- refuse it loudly, in one place. `_offload_subconfig` returned a no-tier fallback for two `lmcache_offload` subs, `_state_tier_sub`'s docstring cited `[dense, kimi_k3]` as the target shape, and `_confirm_remote_load_after_ alloc` was unaffected by the fallback entirely. So on that config the state tier was silently off while the KV load still ran: dense won `get_num_new_matched_ tokens`, queued its H2D, and the prefill forward ran over the block table dense's worker was still writing -- torn KV, silent wrong output, and a `finished_loading` the scheduler never accounted. `_offload_subconfig` now raises a `ValueError` at startup (via `state_tier_capability` in `BlockManager.__init__`) naming the cause, so the half-configured composite is unreachable. A composite with no offload sub stays a soft no-tier. Finding 1 -- load ownership is answered by the load's owner, not the tier's. `should_park_for_load_after_alloc` / `adjust_prefill_chunk_after_alloc` / `should_park_partial_prefill_for_load` had been routed to `_state_tier_sub()`, which names the tier sub even when a different sub won the load. Under `[moriio, kimi_k3]` moriio owns the KV load and kimi_k3 armed nothing (its load was cancelled), so kimi_k3's `_load_specs.get(sid)` was empty and it answered "don't park" -- the forward ran over moriio's in-flight KV. `get_num_new_ matched_tokens` already picks the winner and cancels the losers; it now records that winner under `seq.id` (`_load_winner`, cleared by `request_finished` / `cancel_pending_load`) and the three questions route to it via `_load_owner`. An owner that armed a load but does not refine parking (a P/D producer) parks -- the scheduler's own absent-hook default -- so the forward waits. Finding 6 -- with two offload subs refused, `process_completions` can no longer be handed a composite it cannot partition. `OffloadSchedulerMixin.process_ completions` is destructive over the full sets it receives and there is no shared key to split them per sub, so two handlers meant one sub retiring the other's `_save_inflight` (both key by `str(seq.id)`) and freeing blocks the other was still reading, plus a WARNING per foreign completion. Only `lmcache_offload` subs define the method, so `handlers` is now always 0 or 1 and the direct call is byte-for-byte the single-offload behaviour; `>1` is a loud guard for a should-never-happen. The dead merge/isolation machinery (and its now unused `dataclasses` import) is removed. Tests: two offload subs raise at BlockManager startup (test_block_manager); the load-side methods follow the load owner not the tier sub, and route to a producer winner as a park; process_completions raises on two offload handlers; the tier-face selector test keeps its coverage with a note that the shape is refused upstream. Co-Authored-By: Claude Opus 4.8 --- .../disaggregation/multi/multi_connector.py | 191 ++++++++++-------- atom/model_engine/state_offload.py | 20 +- tests/test_block_manager.py | 26 ++- tests/test_multi_connector.py | 80 +++++--- 4 files changed, 193 insertions(+), 124 deletions(-) diff --git a/atom/kv_transfer/disaggregation/multi/multi_connector.py b/atom/kv_transfer/disaggregation/multi/multi_connector.py index 75bdbb4687..b274820d6c 100644 --- a/atom/kv_transfer/disaggregation/multi/multi_connector.py +++ b/atom/kv_transfer/disaggregation/multi/multi_connector.py @@ -61,7 +61,6 @@ from __future__ import annotations import copy -import dataclasses import logging from typing import Any @@ -426,6 +425,30 @@ def __init__(self, config: Any) -> None: # Opt into the scheduler's offload suffix-prefill path if any sub is the # offload backend (Scheduler._is_offload_connector reads this). self.is_offload = any(getattr(c, "is_offload", False) for c in self._connectors) + # The sub that won `get_num_new_matched_tokens` for each in-flight + # request, keyed by `seq.id`. The load-side decisions + # (`should_park_for_load_after_alloc` and the two chunk/partial-prefill + # siblings) belong to whichever sub armed the load, not to whichever sub + # hosts the state tier -- see `_load_owner` (review round 5, finding 1). + self._load_winner: dict = {} + + def _load_owner(self, seq: Any): + """The sub that armed this request's KV load, or None. + + Recorded at `get_num_new_matched_tokens` time (first-hit-wins, losers + cancelled). The three load-side questions below -- + `should_park_for_load_after_alloc`, `adjust_prefill_chunk_after_alloc`, + `should_park_partial_prefill_for_load` -- ask "is there a load in flight + for this sequence, and must the forward wait for it?" That answer is the + winner's alone. Routing them through `_state_tier_sub()` instead (an + earlier fix) named the tier sub even when a *different* sub won the load + -- e.g. moriio winning ahead of `kimi_k3` under `[moriio, kimi_k3]`: the + tier sub armed nothing, its `_load_specs.get(sid)` was None, so it + answered "don't park" and the prefill ran over moriio's in-flight KV. + The tier sub still owns the state-face forwarders; only these three + load-side questions follow the load's owner. + """ + return self._load_winner.get(getattr(seq, "id", None)) def _state_tier_sub(self): """The one sub-connector that actually hosts the state offload tier. @@ -433,15 +456,18 @@ def _state_tier_sub(self): Selection is by real capability (`has_state_tier`), never by method presence: `LMCacheOffloadConnectorScheduler` defines the entire state face on every layout, so `_first_with(..., "enqueue_state_stores")` - picks a dense offload shell -- whose `_impl` has no tier and returns - False / empty for every state call -- ahead of the `kimi_k3` shell that - owns the tier. The result was that, under - `connectors: [lmcache_offload(dense), lmcache_offload(kimi_k3)]`, stores - settled failed, loads were abandoned and recomputed, and the real - reports were never drained. `_adopt_state_tier` (worker side) already - refuses a config with two tiers, so "first" here is "only"; returns - None when no sub hosts a tier (state forwarders then fall to their - no-tier defaults). + would pick a non-tier offload shell -- whose `_impl` has no tier and + returns False / empty for every state call -- ahead of the shell that + owns the tier. At most one sub can host the tier: two `lmcache_offload` + sub-connectors are refused at startup (`_offload_subconfig`), so "first" + here is "only". Returns None when no sub hosts a tier -- the legal + `[producer]`-only shape -- and the state-face forwarders then fall to + their no-tier defaults. + + This selects the tier owner for the *state-face* forwarders only + (`enqueue_state_*`, `take_state_*`, `save_abandon_timeout_s`). The + load-side questions route through `_load_owner` instead -- the sub that + armed the load, which need not be the tier owner. """ for c in self._connectors: if getattr(c, "has_state_tier", False): @@ -458,14 +484,19 @@ def get_num_new_matched_tokens(self, seq: Any) -> tuple[int, bool]: `_load_specs` entry and a `_lookup_in_step` id, so `update_state_after_ alloc` (which the composite fans to *every* sub) later flips `can_load=True` and recv-queues that request. Only the first matching - sub owns the load. If a second sub also matched -- moriio winning ahead - of the offload tier, or a dense offload shell ahead of the kimi_k3 one - -- the loser's armed load fires into the same block table on - `update_state_after_alloc`: a second writer over the winner's KV, plus a - `finished_loading` the scheduler never accounted. So once a winner is - chosen, cancel every other sub's pending load. `cancel_pending_load` is - idempotent and guarded by `_load_lifecycles`, so a sub that armed - nothing (a miss, or moriio which has no such method) is a no-op. + sub owns the load. If a second sub also matched -- e.g. moriio winning + ahead of an offload sub -- the loser's armed load fires into the same + block table on `update_state_after_alloc`: a second writer over the + winner's KV, plus a `finished_loading` the scheduler never accounted. So + once a winner is chosen, cancel every other sub's pending load. + `cancel_pending_load` is idempotent and guarded by `_load_lifecycles`, + so a sub that armed nothing (a miss, or moriio which has no such method) + is a no-op. + + Record the winner under `seq.id`: the load-side decisions + (`should_park_for_load_after_alloc` and siblings) belong to it, not to + the tier sub (finding 1). Cleared by `request_finished` / + `cancel_pending_load`. """ result = (0, False) winner = None @@ -474,12 +505,16 @@ def get_num_new_matched_tokens(self, seq: Any) -> tuple[int, bool]: if winner is None and toks > 0: result = (toks, needs_load) winner = c + sid = getattr(seq, "id", None) if winner is not None: + self._load_winner[sid] = winner for c in self._connectors: if c is not winner: fn = getattr(c, "cancel_pending_load", None) if callable(fn): fn(seq) + else: + self._load_winner.pop(sid, None) return result def cancel_pending_load(self, seq: Any) -> None: @@ -493,6 +528,7 @@ def cancel_pending_load(self, seq: Any) -> None: and the abandoned request stayed recv-queued. Idempotent per sub (the `_load_lifecycles` guard), so fanning to all is safe. """ + self._load_winner.pop(getattr(seq, "id", None), None) for c in self._connectors: fn = getattr(c, "cancel_pending_load", None) if callable(fn): @@ -508,6 +544,7 @@ def update_state_after_alloc(self, seq: Any) -> None: c.update_state_after_alloc(seq) def request_finished(self, seq: Any) -> None: + self._load_winner.pop(getattr(seq, "id", None), None) for c in self._connectors: if hasattr(c, "request_finished"): c.request_finished(seq) @@ -527,26 +564,26 @@ def abandon_save(self, req_id: Any) -> None: # only needs to expose them when a sub-connector implements them. def should_park_for_load_after_alloc(self, seq: Any) -> bool: - # Prefer the state-tier sub, then any offload sub: this is a load - # decision routed through `_decide_load_after_alloc`, which kimi_k3 - # overrides with a joint-boundary clamp. With two offload subs - # (dense + kimi_k3) plain `_first_with` hands back the dense sub and the - # clamp is silently skipped. - c = self._state_tier_sub() or _first_with( - self._connectors, "should_park_for_load_after_alloc" - ) - return c.should_park_for_load_after_alloc(seq) if c is not None else False + # Route to the sub that armed this request's load (`_load_owner`), not + # the tier sub: the question is whether the forward must wait for the + # in-flight load, and only its owner knows. An offload owner refines via + # `_decide_load_after_alloc` (kimi_k3's joint-boundary clamp); an owner + # that armed a load but does not refine (a P/D producer) parks -- the + # scheduler's own absent-hook default -- so the forward waits rather than + # running over the load's blocks. No owner means no load: don't park. + c = self._load_owner(seq) + if c is not None and hasattr(c, "should_park_for_load_after_alloc"): + return c.should_park_for_load_after_alloc(seq) + return c is not None def adjust_prefill_chunk_after_alloc(self, seq: Any, chunk: int) -> int: - # State-tier sub first, for the reason in `should_park_for_load_after_alloc`: - # the prefill chunk kimi_k3 hands back is sized to the joint load it will - # arm, and the dense sub would answer with the unclamped chunk. - c = self._state_tier_sub() or _first_with( - self._connectors, "adjust_prefill_chunk_after_alloc" - ) - return ( - c.adjust_prefill_chunk_after_alloc(seq, chunk) if c is not None else chunk - ) + # The prefill chunk is sized to the load its owner armed (kimi_k3 clamps + # it to the joint boundary); a sub that armed nothing must not resize it. + # Route to the load owner; unchanged if it does not resize. + c = self._load_owner(seq) + if c is not None and hasattr(c, "adjust_prefill_chunk_after_alloc"): + return c.adjust_prefill_chunk_after_alloc(seq, chunk) + return chunk def enqueue_state_loads(self, loads) -> bool: """First sub that can carry them owns them; False if none can. @@ -636,13 +673,13 @@ def take_state_source_releases(self) -> set: return c.take_state_source_releases() def should_park_partial_prefill_for_load(self, seq: Any) -> bool: - # State-tier sub first, for the reason in `should_park_for_load_after_alloc`: - # parking a partial prefill for its load is the state tier's decision, and - # a second offload sub must not answer it in the tier sub's place. - c = self._state_tier_sub() or _first_with( - self._connectors, "should_park_partial_prefill_for_load" - ) - return c.should_park_partial_prefill_for_load(seq) if c is not None else False + # Called for every running sequence, so most have no load owner -> False + # (don't park). Parking a partial prefill to await its load is the + # load owner's decision; a sub that armed nothing must not answer it. + c = self._load_owner(seq) + if c is not None and hasattr(c, "should_park_partial_prefill_for_load"): + return c.should_park_partial_prefill_for_load(seq) + return False def should_defer_free(self, seq: Any) -> bool: # Defer if ANY sub wants to defer (so neither a pending save nor a @@ -663,26 +700,28 @@ def has_pending_work(self) -> bool: ) def process_completions(self, output: KVConnectorOutput) -> KVConnectorOutput: - """Let every offload sub apply its own completions and normalize output. + """Let the one offload sub apply its own completions and normalize output. Only offload defines this. Without the fan-out its save/load bookkeeping never clears and raw operation ids reach the scheduler, which looks requests up by bare id. - `OffloadSchedulerMixin.process_completions` is *destructive*: it - replaces `finished_loading`/`failed_loading`/`finished_saving` with only - the operations it recognises and `.clear()`s `connector_completions` - wholesale. Fanning one shared `output` through the subs in sequence - therefore lets the first offload sub drop the loads and clear the - completion channels the later subs own -- so a `[dense, kimi_k3]` - composite silently loses every K3 state completion: the parked request - never wakes, `settle_state_store` never runs, and the PAGE units stay - pinned until the stale reclaimer marks the image untrusted. - - Give each offload sub a private copy of the mutated sets and merge the - results, so a sub only ever consumes what it owns. The single-offload - case (the common `[producer, offload]` shape) keeps the direct call -- - no copy, byte-for-byte the old behaviour. + `OffloadSchedulerMixin.process_completions` is *destructive* and cannot + partition: it replaces `finished_loading`/`failed_loading`/ + `finished_saving` with only the operations it recognises and `.clear()`s + `connector_completions` wholesale, over the full sets it is handed. Two + offload subs would each be handed the other's completions -- one sub + retiring the other's `_save_inflight` (both key by `str(seq.id)`), whose + `_maybe_release_deferred` then frees blocks the other is still reading, + plus a WARNING per foreign completion at steady state. There is no shared + key by which the composite could split the sets per sub. + + That case is now unrepresentable: `_offload_subconfig` refuses two + `lmcache_offload` sub-connectors at startup, and only `lmcache_offload` + subs define `process_completions`, so `handlers` is 0 or 1. The direct + call is byte-for-byte the single-offload (`[producer, offload]`) + behaviour. `>1` is guarded loudly in case that refusal is ever bypassed + -- silently corrupting saves is the worse failure. """ handlers = [ handler @@ -691,33 +730,13 @@ def process_completions(self, output: KVConnectorOutput) -> KVConnectorOutput: ] if not handlers: return output - if len(handlers) == 1: - return handlers[0](output) - - merged_loading: set = set() - merged_failed_loading: set = set() - merged_saving: set = set() - merged_completions: set = set() - for handler in handlers: - # Only the four fields the mixin rewrites need isolating; the rest - # (sending/recving) it never touches, so they stay shared. - sub_out = dataclasses.replace( - output, - finished_loading=set(output.finished_loading), - failed_loading=set(output.failed_loading), - finished_saving=set(output.finished_saving), - connector_completions=set(output.connector_completions), + if len(handlers) > 1: + raise RuntimeError( + "multi has >1 offload sub-connector with process_completions; " + "their completion sets cannot be partitioned per sub and this " + "composite is refused at startup (_offload_subconfig)." ) - sub_out = handler(sub_out) - merged_loading |= sub_out.finished_loading - merged_failed_loading |= sub_out.failed_loading - merged_saving |= sub_out.finished_saving - merged_completions |= sub_out.connector_completions - output.finished_loading = merged_loading - output.failed_loading = merged_failed_loading - output.finished_saving = merged_saving - output.connector_completions = merged_completions - return output + return handlers[0](output) def save_finished(self, req_id: Any) -> None: for c in self._connectors: @@ -773,7 +792,9 @@ def get_statistics(self) -> dict[str, int]: guard and renders `{}` when it is absent -- so under `multi` the whole offload metrics block silently reads empty. Sum overlapping int keys across every sub that reports (only offload subs do), rather than - first-hit, so a `[dense, kimi_k3]` composite reports both legs' totals. + first-hit. At most one offload sub is legal (`_offload_subconfig`), but + summing stays correct for that one and for any future producer that + grows int counters. """ merged: dict[str, int] = {} for c in self._connectors: diff --git a/atom/model_engine/state_offload.py b/atom/model_engine/state_offload.py index 201c5d4b54..3c136e7d32 100644 --- a/atom/model_engine/state_offload.py +++ b/atom/model_engine/state_offload.py @@ -241,13 +241,31 @@ def _offload_subconfig(cfg: dict) -> tuple[dict | None, str]: half and its completions ride that connector's `get_finished`, so two providers would each hold half an answer and a request parked on the wrong one would never be reported. + + Two offload sub-connectors (e.g. `[lmcache_offload(dense), + lmcache_offload(kimi_k3)]`) is refused *loudly* here, the one place the + composite is inspected, rather than degraded to a no-tier fallback. A silent + fallback disabled the state tier but left the KV load path live: dense won + `get_num_new_matched_tokens` and queued its H2D, the tier never armed, and + the prefill forward ran over the block table dense's worker was still + writing -- torn KV, silent wrong output, and a `finished_loading` the + scheduler never accounted (review round 5, finding 0). Making it a startup + `ValueError` keeps that half-configured composite unreachable and names the + cause where the operator can act on it. A composite with *no* offload sub is + still a soft no-tier (`None`), the legal `[producer]`-only shape. """ subs = [s for s in (cfg.get("connectors") or ()) if isinstance(s, dict)] offload = [s for s in subs if s.get("kv_connector") in _STATE_TIER_BACKENDS] if not offload: return None, "multi lists no offload connector" if len(offload) > 1: - return None, f"multi lists {len(offload)} offload connectors; expected one" + raise ValueError( + f"kv_transfer_config: `multi` lists {len(offload)} offload " + "connectors, but the state tier's bytes and completions ride one " + "connector's worker half -- two would each hold half an answer and " + "a request parked on the wrong one would never be reported. Use at " + "most one `lmcache_offload` sub-connector." + ) return offload[0], "" diff --git a/tests/test_block_manager.py b/tests/test_block_manager.py index 5efb28932d..4e97c6c9e9 100644 --- a/tests/test_block_manager.py +++ b/tests/test_block_manager.py @@ -915,16 +915,22 @@ def test_a_consumer_role_votes_but_never_hands_over_a_store(monkeypatch): def test_a_multi_listing_two_offload_connectors_is_refused(monkeypatch): """The tier's bytes ride one connector's worker half and its completions ride that connector's `get_finished`, so two providers would each hold half - an answer.""" - bm = _bm_with_state_tier( - monkeypatch, - { - "kv_connector": "multi", - "connectors": [_OFFLOAD_KVC, dict(_OFFLOAD_KVC)], - }, - ) - assert bm.state_offload is None - assert "expected one" in bm.state_tier_capability.reason + an answer. Refused *loudly* at startup, not degraded to a no-tier fallback: + the silent fallback left the KV load path live over a block table the other + sub was writing (review round 5, finding 0).""" + try: + _bm_with_state_tier( + monkeypatch, + { + "kv_connector": "multi", + "connectors": [_OFFLOAD_KVC, dict(_OFFLOAD_KVC)], + }, + ) + except ValueError as exc: + assert "offload connectors" in str(exc) + assert "at most one" in str(exc) + else: + raise AssertionError("two offload sub-connectors must raise at startup") def test_the_index_is_built_for_a_multi_that_lists_the_offload_backend( diff --git a/tests/test_multi_connector.py b/tests/test_multi_connector.py index c5e694e047..1b6f39cc70 100644 --- a/tests/test_multi_connector.py +++ b/tests/test_multi_connector.py @@ -230,6 +230,7 @@ def _sched(connectors): obj._connectors = connectors obj.is_producer = any(getattr(c, "is_producer", False) for c in connectors) obj.is_offload = any(getattr(c, "is_offload", False) for c in connectors) + obj._load_winner = {} return obj @@ -370,14 +371,18 @@ def test_role_attrs_aggregate(): def test_offload_methods_forwarded_to_owning_sub(): - moriio = FakeSchedSub(is_producer=True) # no offload methods - off = FakeSchedSub(is_offload=True, offload_methods=True) + # The load-side decisions route to the sub that armed the load, recorded at + # `get_num_new_matched_tokens` time. Here the offload sub is the sole match, + # so it owns the load and answers all three. + moriio = FakeSchedSub(is_producer=True) # no offload methods, no match + off = FakeSchedSub(is_offload=True, offload_methods=True, match=(4, True)) off.park = True off.partial_park = True off.defer = True off.chunk_ret = 7 sched = _sched([moriio, off]) - seq = object() + seq = SimpleNamespace(id="s1") + assert sched.get_num_new_matched_tokens(seq) == (4, True) # off owns the load assert sched.should_park_for_load_after_alloc(seq) is True assert sched.should_park_partial_prefill_for_load(seq) is True assert sched.should_defer_free(seq) is True @@ -388,6 +393,35 @@ def test_offload_methods_forwarded_to_owning_sub(): assert off.load_failed_ids == ["r2"] +def test_load_side_methods_follow_the_load_owner_not_the_tier_sub(): + """Finding 1. The load-side questions belong to whichever sub armed the + load, which need not be the tier sub. + + `[moriio, kimi_k3]`: moriio matches first and owns the KV load; kimi_k3 + hosts the state tier but armed nothing (its load was cancelled). Routing + `should_park_for_load_after_alloc` through the tier sub asked kimi_k3, whose + `_load_specs.get(sid)` is empty, so it answered "don't park" and the prefill + ran over moriio's in-flight KV. It must route to moriio -- the load owner -- + which does not refine parking, so the composite parks (the scheduler's own + absent-hook default) and the forward waits. + """ + moriio = FakeSchedSub(is_producer=True, match=(5, True)) # wins, no offload API + k3 = FakeSchedSub( + is_offload=True, offload_methods=True, has_state_tier=True, match=(4, True) + ) + k3.park = False # the tier sub would say "don't park" + sched = _sched([moriio, k3]) + seq = SimpleNamespace(id="s2") + + assert sched.get_num_new_matched_tokens(seq) == (5, True) # moriio owns it + assert k3.cancelled == [seq] # kimi_k3's armed load was undone + assert k3.pending_load is False + # Routes to moriio (owner), which lacks the hook -> park, NOT to k3's False. + assert sched.should_park_for_load_after_alloc(seq) is True + # And the tier sub still owns the state face regardless of load ownership. + assert sched._state_tier_sub() is k3 + + def test_process_completions_reaches_the_offload_sub(): # The scheduler calls process_completions and nothing else — it is the only # caller of the offload sub's save_finished. Without the fan-out the sub's @@ -404,36 +438,22 @@ def test_process_completions_reaches_the_offload_sub(): assert out.finished_saving == {9} -def test_process_completions_isolates_each_offload_sub(): - # [dense, kimi_k3]: BOTH subs define process_completions, and each one is - # destructive — it filters finished_loading down to its own and .clear()s - # every completion channel. Fanning one mutable output through them in - # sequence lets the first sub eat the loads and completion channels the - # second owns, so a K3 state completion silently vanishes and the parked - # request never wakes. Each sub must operate on a private copy; the union - # of their results is what reaches the scheduler. +def test_process_completions_refuses_two_offload_subs(): + # [dense, kimi_k3]: BOTH subs define process_completions, and the mixin is + # destructive over the FULL sets it is handed -- it cannot partition. There + # is no shared key by which the composite could split the completions per + # sub, so one sub would retire the other's saves and clear its channels. + # That composite is refused at startup (`_offload_subconfig`); reaching this + # method with two offload handlers is a should-never-happen guarded loudly, + # because silently corrupting saves is the worse failure. dense = DestructiveSub(owned_load="dense_load", owned_channel="dense_ch") k3 = DestructiveSub(owned_load="k3_load", owned_channel="k3_state_index") sched = _sched([dense, k3]) - out = sched.process_completions( - KVConnectorOutput( - finished_loading={"dense_load", "k3_load"}, - connector_completions={ - ConnectorCompletion("dense_ch", SaveOperationId(1, 0), True), - ConnectorCompletion("k3_state_index", SaveOperationId(2, 0), True), - }, + with pytest.raises(RuntimeError, match="cannot be partitioned"): + sched.process_completions( + KVConnectorOutput(finished_loading={"dense_load", "k3_load"}) ) - ) - - # Both loads survive the fan-out (sequential fan would drop k3_load)... - assert out.finished_loading == {"dense_load", "k3_load"} - # ...and each sub settled exactly its own channel (sequential fan would - # leave k3.settled_channels empty — dense cleared them first). - assert dense.settled_channels == ["dense_ch"] - assert k3.settled_channels == ["k3_state_index"] - # Every completion channel was consumed by its owner; none leak onward. - assert out.connector_completions == set() def test_process_completions_single_offload_sub_is_called_directly(): @@ -613,6 +633,10 @@ def test_state_calls_route_by_tier_ownership_not_method_presence(): tier), not method presence. Here the dense-like shell is first and carries no tier; the kimi_k3-like shell is second and does. Every state call must land on the second. + + A two-offload composite is refused at startup (`_offload_subconfig`, finding + 0), so this shape does not reach a running engine; the test exercises the + selector directly to keep it robust if that refusal is ever bypassed. """ dense = FakeSchedSub(is_offload=True, offload_methods=True, has_state_tier=False) k3 = FakeSchedSub(is_offload=True, offload_methods=True, has_state_tier=True) From d45c545c2de98ef64bdd06320966f5b44778eff1 Mon Sep 17 00:00:00 2001 From: zejchen Date: Wed, 2 Sep 2026 07:30:09 +0000 Subject: [PATCH 135/151] docs(offload): match round-5 docs to code (README GDN refusal, env default, reason count) Review round 5, Docs-match-code: - offload/README.md: the resolver has a fourth outcome the prose omitted -- GDN/linear-attention models (qwen3_next, qwen3_5_*) are refused at startup by select_offload_layout with a ValueError, not resolved to `dense`. Name it. - docs/environment_variables.md: lead the OFFLOAD_MAX_PENDING_SAVES default with the engine-side/state-tier reader's flat 2 (scheduler.py), matching the code, then the KV-leg reader's scaled max(2, 2xOFFLOAD_COPY_WORKERS). The two-reader split (raise vs warn) is intentional; the column no longer leads with the formula the engine-side reader does not use. - state_offload.py: drop the stale "four conditions"/"three conditions" counts (there are seven reason= sites) rather than pin a new number that goes stale; the second site now enumerates the checks the name cannot see. sequence.py "seven"->"six" and the MiniMax-M3 GDN mislabel were already fixed. Co-Authored-By: Claude Opus 4.8 --- atom/kv_transfer/offload/README.md | 7 +++++++ atom/model_engine/state_offload.py | 11 +++++++---- docs/environment_variables.md | 2 +- 3 files changed, 15 insertions(+), 5 deletions(-) diff --git a/atom/kv_transfer/offload/README.md b/atom/kv_transfer/offload/README.md index 475e2d73d3..f16cb02e02 100644 --- a/atom/kv_transfer/offload/README.md +++ b/atom/kv_transfer/offload/README.md @@ -20,6 +20,13 @@ per-request state tier), `hybrid` when `hf_config.compress_ratios` is present override that choice without giving scheduler and worker different connector names. +GDN/linear-attention models (`qwen3_next`, `qwen3_5_*`; e.g. Qwen3-Next, +Qwen3.5) are the one family the resolver does **not** map to a layout: they carry +a per-request recurrent state that no offload layout owns a tier for, so +restoring their KV prefix while that state stays stale is silent wrong output. +`select_offload_layout` refuses them at startup with a `ValueError` that names +the cause, rather than falling through to `dense`. + It is the **ATOM-native, in-engine** offload path: the connector plugs straight into ATOM's scheduler/worker via the shared [`KVConnectorFactory`](../disaggregation/factory.py), with no vLLM in the loop. diff --git a/atom/model_engine/state_offload.py b/atom/model_engine/state_offload.py index 3c136e7d32..a288318c0f 100644 --- a/atom/model_engine/state_offload.py +++ b/atom/model_engine/state_offload.py @@ -216,8 +216,8 @@ class StateTierCapability: mismatch is what left stores emitted with nowhere to go, and it is the common root of the tier-none fallback paths. - `reason` is filled in whenever the tier is off, so the log says which of the - four conditions refused it rather than leaving the operator to guess. + `reason` is filled in whenever the tier is off, so the log names which check + refused it rather than leaving the operator to guess. """ can_store_state: bool @@ -288,8 +288,11 @@ def state_tier_capability(config) -> StateTierCapability: """What the worker will do for the state tier, decided from config alone. A capability descriptor rather than a name check, because the two are not - the same question and the worker refuses on three conditions the name - cannot see. Config alone because this runs in the engine process at + the same question and the worker refuses on conditions the name cannot see + (no transfer config, a `multi` composite with no offload sub, `pp_size > 1`, + an unknown explicit override, a layout that offloads no state, or a `kv_role` + that neither saves nor loads). Config alone because this runs in the engine + process at `BlockManager.__init__`, before any worker connector exists -- so it must agree with the worker by construction, which is why the layout comes from the same `select_offload_layout` the worker uses and the roles from the diff --git a/docs/environment_variables.md b/docs/environment_variables.md index 8667949372..09281e118a 100644 --- a/docs/environment_variables.md +++ b/docs/environment_variables.md @@ -196,7 +196,7 @@ discoverable from the central env reference despite bypassing the registry. | Variable | Type | Default | Description | |----------|------|---------|-------------| | **LMCACHE_EC_PIN_TIMEOUT_SEC** | float | LMCache's own (300) | LMCache's source-pin timeout. ATOM reads it only to derive the engine's save-abandon window (`pin + 30s`), so the two stay ordered — a lost store report is reclaimed only after LMCache would already have force-unpinned its source. Non-positive disables ATOM's reclamation. ATOM sets no default of its own; when unset it assumes LMCache's. | -| **OFFLOAD_MAX_PENDING_SAVES** | int | `max(2, 2 × OFFLOAD_COPY_WORKERS)`; **2** for the state tier's own reader | Bound on total in-flight offload transfers (running + queued) held before a SLOT snapshot or executor submission. A KV save and a state store both pin bytes out of the same pool while they run, so the KV leg and the K3 state tier share this one number rather than each carrying its own. Two readers compute it, though: the KV leg's canonical `_offload_common.max_pending_saves` derives the shown default from `OFFLOAD_COPY_WORKERS` and **raises** on an unparseable value, while the scheduler's state-tier reader (`_offload_max_pending_saves`) has a simpler fallback — a flat default of **2** (no `OFFLOAD_COPY_WORKERS` scaling) that **warns and uses 2** on an unparseable value rather than raising. Set the env to an explicit integer to pin both. | +| **OFFLOAD_MAX_PENDING_SAVES** | int | **2**, flat, for the engine-side/state-tier reader (`scheduler.py`); `max(2, 2 × OFFLOAD_COPY_WORKERS)` for the KV-leg reader (`_offload_common.py`) | Bound on total in-flight offload transfers (running + queued) held before a SLOT snapshot or executor submission. A KV save and a state store both pin bytes out of the same pool while they run, so the KV leg and the K3 state tier share this one number rather than each carrying its own. Two readers compute it, though: the KV leg's canonical `_offload_common.max_pending_saves` derives the shown default from `OFFLOAD_COPY_WORKERS` and **raises** on an unparseable value, while the scheduler's state-tier reader (`_offload_max_pending_saves`) has a simpler fallback — a flat default of **2** (no `OFFLOAD_COPY_WORKERS` scaling) that **warns and uses 2** on an unparseable value rather than raising. Set the env to an explicit integer to pin both. | ## Profiling & debugging From c5cba1481754ce823cc0f9067cd84f4a8da59151 Mon Sep 17 00:00:00 2001 From: zejchen Date: Wed, 2 Sep 2026 07:35:05 +0000 Subject: [PATCH 136/151] refactor(offload): drop dead code flagged in round 5 Review round 5, Smaller/dead code -- zero callers/readers repo-wide: - StateByteCodec.contains (state_object.py): no caller. - StateByteCodec.puts_refused: incremented on a refused put, never surfaced anywhere. Remove the field, its increment, and the docstring claim that it "makes that visible" (it did not); the refusal-is-not-an-error note stays. - StagedTransfer.staging_buffer_bytes property (staging.py): no reader; the __init__ param and the internal self._staging_buffer_bytes stay. kv_connector_hosts_state_tier and StateTierCapability.layout were already gone. Co-Authored-By: Claude Opus 4.8 --- atom/kv_transfer/offload/hybrid/kimi_k3/staging.py | 4 ---- .../offload/hybrid/kimi_k3/state_object.py | 12 +----------- 2 files changed, 1 insertion(+), 15 deletions(-) diff --git a/atom/kv_transfer/offload/hybrid/kimi_k3/staging.py b/atom/kv_transfer/offload/hybrid/kimi_k3/staging.py index 7dc3bb3437..d406c5b608 100644 --- a/atom/kv_transfer/offload/hybrid/kimi_k3/staging.py +++ b/atom/kv_transfer/offload/hybrid/kimi_k3/staging.py @@ -71,10 +71,6 @@ def __init__( self._release_after_transfer = release_after_transfer self._tls = threading.local() - @property - def staging_buffer_bytes(self) -> int: - return self._staging_buffer_bytes - def _use_cuda(self) -> bool: return self.device.type == "cuda" diff --git a/atom/kv_transfer/offload/hybrid/kimi_k3/state_object.py b/atom/kv_transfer/offload/hybrid/kimi_k3/state_object.py index 971ded5581..5595091cf2 100644 --- a/atom/kv_transfer/offload/hybrid/kimi_k3/state_object.py +++ b/atom/kv_transfer/offload/hybrid/kimi_k3/state_object.py @@ -54,7 +54,6 @@ def __init__( self._worker_id = int(worker_id) self._layout_id = layout_id self._misfit_reads = 0 - self.puts_refused = 0 self._storage = None # Never hard-code a size: V4 keeps six compressor fields across # n_csa/n_hca layers plus an optional window; GDN keeps @@ -112,7 +111,7 @@ def put(self, h: int, unit_ids, on_source_released=None) -> bool: A refusal is not an error -- `_allocate` returns None under CPU pressure, and a whole image is refused sooner than a KV chunk, so the state leg - feels a full pool first. `puts_refused` makes that visible. + feels a full pool first. `on_source_released` fires when `pack` returns -- after the gather and D2H are synchronized, i.e. once the GPU has stopped reading the units. @@ -128,7 +127,6 @@ def put(self, h: int, unit_ids, on_source_released=None) -> bool: key = self.key(h) obj = self._allocate(self.entry_bytes) if obj is None: - self.puts_refused += 1 return False # `batched_put` discharges the reference it is handed -- but only if # reached. A throwing `pack` skips it and strands the allocation at @@ -218,14 +216,6 @@ def _object_bytes(obj) -> int | None: return int(tensor.numel()) * tensor.element_size() return None - def contains(self, h: int) -> bool: - """Ask storage, never a local set -- a set goes stale the moment - LMCache's LRU evicts. `contains` answers with a location name or None, - so the truthiness test is the membership test.""" - if self._storage is None: - return False - return bool(self._storage.contains(self.key(h))) - def _allocate(self, nbytes: int) -> Any: from lmcache.v1.memory_management import MemoryFormat From d6e204dfec1c2005aff46890ae7805149f06690a Mon Sep 17 00:00:00 2001 From: zejchen Date: Wed, 2 Sep 2026 07:35:05 +0000 Subject: [PATCH 137/151] fix(offload): guard _SubConnectorView.__getattr__ against unbound _config Review round 5, Smaller. __getattr__ only fires on names normal lookup misses, so if _config is ever unbound (a pickle/deepcopy probe, or a raise between __new__ and the __init__ assignment) reading self._config inside __getattr__ re-enters for "_config" and recurses to RecursionError. The view is built at BlockManager.__init__, so the blast radius is engine startup. Raise AttributeError for "_config" so it fails as an ordinary missing attribute. Co-Authored-By: Claude Opus 4.8 --- atom/model_engine/state_offload.py | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/atom/model_engine/state_offload.py b/atom/model_engine/state_offload.py index a288318c0f..364c6bded4 100644 --- a/atom/model_engine/state_offload.py +++ b/atom/model_engine/state_offload.py @@ -281,6 +281,14 @@ def __init__(self, config, sub: dict) -> None: self.kv_transfer_config = sub def __getattr__(self, name): + # __getattr__ only fires for names normal lookup misses, so if `_config` + # itself is unbound -- a pickle/deepcopy probe, or a raise between + # `__new__` and the assignment in `__init__` -- reading `self._config` + # here re-enters __getattr__ for "_config" and recurses to + # RecursionError. This view is built at `BlockManager.__init__`, so that + # blast radius is engine startup. Fail as an ordinary missing attribute. + if name == "_config": + raise AttributeError(name) return getattr(self._config, name) From 0c833baafb84064b51cd9d34875fca8a3aebdd4f Mon Sep 17 00:00:00 2001 From: zejchen Date: Wed, 2 Sep 2026 07:44:20 +0000 Subject: [PATCH 138/151] fix(offload): back out the optimistic stores_attempted bump on a refusal take_state_stores bumps stores_attempted when it hands an op over, but a connector refusal never reaches a worker. Counting it once as stores_refused and leaving it in stores_attempted made the refusal linger forever in the attempted - completed - failed in-flight funnel. settle_state_store now decrements stores_attempted on the refusal path. Co-Authored-By: Claude Opus 4.8 --- atom/model_engine/block_manager.py | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/atom/model_engine/block_manager.py b/atom/model_engine/block_manager.py index da05e44941..a2609765cc 100644 --- a/atom/model_engine/block_manager.py +++ b/atom/model_engine/block_manager.py @@ -1368,7 +1368,9 @@ def settle_state_store(self, op, ok: bool, attempted: bool = True) -> None: pin still has to be released here, but the copy never reached a worker, so it is not a `stores_failed` -- the caller counts the refusal once as `stores_refused`, and bumping `stores_failed` too would double-count the - same event under two names. + same event under two names. For the same reason it backs the optimistic + `stores_attempted` bump `take_state_stores` made at hand-out back out, so + a refusal does not linger in `attempted - completed - failed`. `op` is a `StateStoreOperationId`: the pin is released for that exact generation, so a late report from a superseded attempt settles nothing. @@ -1404,6 +1406,13 @@ def settle_state_store(self, op, ok: bool, attempted: bool = True) -> None: self.state_offload.note_stored(int(op.prefix_hash)) elif attempted: self.state_offload.stores_failed += 1 + else: + # A refusal. `take_state_stores` bumped `stores_attempted` when it + # handed this op over, but it never reached a worker, so back that + # out: the event is counted once as `stores_refused`, and leaving it + # in `stores_attempted` would read as permanently in-flight in the + # `attempted - completed - failed` funnel. + self.state_offload.stores_attempted -= 1 def release_state_store_source(self, op) -> None: """The GPU has finished reading this store's PAGE units; hand them back. From f8423a1bc62949a6478a8749584b80d7d2e07268 Mon Sep 17 00:00:00 2001 From: zejchen Date: Wed, 2 Sep 2026 07:44:29 +0000 Subject: [PATCH 139/151] fix(offload): do not publish new state work while draining KV at exit _drain_kv_work_at_exit loops while has_pending_kv_work() is True. Its _advance_idle_kv_transfer -> _dispatch_idle_offload_work path published new state loads/stores every iteration, and each fresh store kept has_pending_kv_work() True -- so the drain manufactured its own exit condition and could only ever end at the timeout. Thread a dispatch_new flag (default True preserves normal idle) through _advance_idle_kv_transfer and _dispatch_idle_offload_work; the exit drain passes dispatch_new=False so it flushes in-flight transfers via build_connector_meta/process_kvconnector_output but starts no new ones. The PP override accepts the flag for signature parity (it never publishes -- the state tier refuses pp_size > 1). Adds two regression tests. Co-Authored-By: Claude Opus 4.8 --- atom/model_engine/engine_core.py | 32 ++++++++++++++++------ atom/model_engine/pp_engine_core.py | 8 +++++- tests/test_kv_drain_liveness.py | 42 +++++++++++++++++++++++++++++ 3 files changed, 73 insertions(+), 9 deletions(-) diff --git a/atom/model_engine/engine_core.py b/atom/model_engine/engine_core.py index 9c9d439729..f134da8f36 100644 --- a/atom/model_engine/engine_core.py +++ b/atom/model_engine/engine_core.py @@ -488,14 +488,16 @@ def has_pending_kv_work(self) -> bool: return False return bool(connector.has_pending_work()) - def _advance_idle_kv_transfer(self) -> None: + def _advance_idle_kv_transfer(self, dispatch_new: bool = True) -> None: # No forward batch will run this tick, but offload load/save work may # still need to be dispatched or reported back to the scheduler. + # `dispatch_new=False` on the shutdown drain: report in-flight work + # back, but start no new transfers (see `_dispatch_idle_offload_work`). now = time.monotonic() if now < self._next_idle_kv_drain: return self._next_idle_kv_drain = now + KV_IDLE_DRAIN_INTERVAL_S - self._dispatch_idle_offload_work() + self._dispatch_idle_offload_work(dispatch_new=dispatch_new) self._poll_kv_transfer_progress() def _drain_kv_work_at_exit(self) -> None: @@ -504,6 +506,12 @@ def _drain_kv_work_at_exit(self) -> None: The loop exits as soon as its queues are empty, so a save dispatched by the final batch would otherwise be abandoned with its completion unrecorded and its blocks still deferred. + + Drains with `dispatch_new=False`: it must let already-dispatched + transfers finish and report, but must not publish new state loads/stores. + A fresh store dispatched here spills bytes nothing will read back, and -- + worse -- keeps `has_pending_kv_work()` True, so this very loop could + manufacture its own work and never converge before the deadline. """ if not self.kv_transfer_enabled: return @@ -517,7 +525,7 @@ def _drain_kv_work_at_exit(self) -> None: KV_SHUTDOWN_DRAIN_TIMEOUT_S, ) break - self._advance_idle_kv_transfer() + self._advance_idle_kv_transfer(dispatch_new=False) time.sleep(KV_IDLE_DRAIN_INTERVAL_S) except Exception: logger.exception("KV transfer drain during shutdown failed") @@ -534,7 +542,7 @@ def _poll_kv_transfer_progress(self) -> None: if callable(reconcile): reconcile() - def _dispatch_idle_offload_work(self) -> None: + def _dispatch_idle_offload_work(self, dispatch_new: bool = True) -> None: if not self.kv_transfer_enabled: return connector = getattr(self.scheduler, "kv_connector", None) @@ -543,10 +551,18 @@ def _dispatch_idle_offload_work(self) -> None: # getattr for the same reason as `kv_connector` above: this path is # reached with scheduler doubles that implement only the connector # surface. - for name in ("_publish_state_loads", "_publish_state_stores"): - publish = getattr(self.scheduler, name, None) - if publish is not None: - publish() + # + # `dispatch_new=False` on the shutdown drain: publishing new state + # loads/stores there hands the connector work whose bytes nothing will + # read back, and -- worse -- each fresh store keeps `has_pending_kv_work` + # True, so the drain loop that waits on it manufactures its own exit + # condition and never converges. Still build/process the meta so + # already-dispatched transfers finish and report. + if dispatch_new: + for name in ("_publish_state_loads", "_publish_state_stores"): + publish = getattr(self.scheduler, name, None) + if publish is not None: + publish() meta = connector.build_connector_meta() if not connector_metadata_has_work(meta): return diff --git a/atom/model_engine/pp_engine_core.py b/atom/model_engine/pp_engine_core.py index c0be34ddbb..ab863e8fde 100644 --- a/atom/model_engine/pp_engine_core.py +++ b/atom/model_engine/pp_engine_core.py @@ -196,7 +196,7 @@ def has_pending_kv_work(self) -> bool: self._pp_kv_aggregator is not None and self._pp_kv_aggregator.has_pending() ) - def _dispatch_idle_offload_work(self) -> None: + def _dispatch_idle_offload_work(self, dispatch_new: bool = True) -> None: """Override: fan the idle connector metadata out to every PP stage. ``Scheduler.schedule()`` returns None once waiting and running are @@ -204,7 +204,13 @@ def _dispatch_idle_offload_work(self) -> None: materializes while draining. Build the metadata directly instead, and ship it downstream too — otherwise the stages never save their layers and ``PPKVAggregator`` cannot reach a quorum. + + ``dispatch_new`` exists only to match the base signature the shutdown + drain calls through: this override never publishes new state loads/ + stores (the state tier refuses ``pp_size > 1`` outright, so there are + none), so it has nothing to gate and the flag is inert here. """ + del dispatch_new if not self.kv_transfer_enabled: return connector = getattr(self.scheduler, "kv_connector", None) diff --git a/tests/test_kv_drain_liveness.py b/tests/test_kv_drain_liveness.py index daeb9da9ac..dfa3f14690 100644 --- a/tests/test_kv_drain_liveness.py +++ b/tests/test_kv_drain_liveness.py @@ -42,6 +42,13 @@ def __init__(self, deferred=None, connector=None): self.deferred_free_blocks = dict(deferred or {}) self.kv_connector = connector self.outputs = [] + self.state_publishes = 0 + + def _publish_state_loads(self): + self.state_publishes += 1 + + def _publish_state_stores(self): + self.state_publishes += 1 def _update_from_kv_xfer_finished(self, out): self.outputs.append(out) @@ -249,3 +256,38 @@ def test_exit_drain_is_a_noop_without_kv_transfer(): proc = _engine(enabled=False, deferred={7: object()}) proc._drain_kv_work_at_exit() assert proc.runner_mgr.polls == 0 + + +# -- new state work is dispatched at idle, but NOT while draining at exit --- + + +def test_idle_drain_publishes_new_state_work(): + # Normal idle: publish new state loads/stores so a lull between batches + # still makes forward progress on the offload tiers. + proc = _engine(connector=FakeConnector(pending=True)) + proc._advance_idle_kv_transfer() + assert proc.scheduler.state_publishes == 2 # both loads and stores + + +def test_exit_drain_does_not_publish_new_state_work(): + # Shutdown drain must let in-flight transfers finish and report, but must + # not publish new state work: a fresh store keeps has_pending_kv_work True, + # so the loop that waits on it would manufacture its own exit condition and + # never converge. It still builds/processes the meta to flush what is in + # flight. + connector = FakeConnector(pending=True) + + original = connector.has_pending_work + + def clear_after_first_poll(): + pending = original() + connector.pending = False + return pending + + connector.has_pending_work = clear_after_first_poll + proc = _engine(connector=connector) + proc._drain_kv_work_at_exit() + + assert proc.scheduler.state_publishes == 0 + assert connector.builds == 1 # in-flight work still flushed + assert proc.runner_mgr.dispatched == ["process_kvconnector_output"] From f88dbf3cbd3de68173134ff3aafd8b66a7da210c Mon Sep 17 00:00:00 2001 From: zejchen Date: Wed, 2 Sep 2026 07:48:20 +0000 Subject: [PATCH 140/151] test(offload): make the shell sweep alias-aware so aliased reads stay covered test_every_member_the_scheduler_reads_is_reachable_through_the_shell scanned only self.kv_connector.x reads, so _state_store_pending_cap's hoist into a local alias (conn = self.kv_connector; getattr(conn, "max_pending_saves")) matched none of the four patterns -- the exact silent degradation the test's own docstring warns about, with max_pending_saves uncovered. Collect alias names bound directly from self.kv_connector and sweep reads through each; discard _state_tier_sub (a multi-only internal delegation helper, like _impl); anchor max_pending_saves in must_be_seen; and list it in multi_routes_via_sub since the composite routes it through its state-tier sub rather than exposing it directly. Co-Authored-By: Claude Opus 4.8 --- tests/test_lmcache_offload_connector.py | 51 ++++++++++++++++++++----- 1 file changed, 41 insertions(+), 10 deletions(-) diff --git a/tests/test_lmcache_offload_connector.py b/tests/test_lmcache_offload_connector.py index 5b5d125c2a..4a7b3dc5d5 100644 --- a/tests/test_lmcache_offload_connector.py +++ b/tests/test_lmcache_offload_connector.py @@ -5280,12 +5280,36 @@ def test_every_member_the_scheduler_reads_is_reachable_through_the_shell(): from atom.kv_transfer.offload.connector import LMCacheOffloadConnectorScheduler src = inspect.getsource(sched_mod) - probed = set(re.findall(r'getattr\(\s*self\.kv_connector,\s*"([a-z_]+)"', src)) - probed |= set(re.findall(r"self\.kv_connector\.([a-z_]+)\(", src)) - probed |= set(re.findall(r"self\.kv_connector\.([a-z_]+)\b", src)) + + # The Scheduler reads the shell two ways: directly (`self.kv_connector.x`) + # and through a local alias it hoists first (`conn = self.kv_connector; ... + # getattr(conn, "x")`). A scan that only matched `self.kv_connector` would + # silently drop every aliased read -- which is exactly how `max_pending_saves` + # escaped coverage before: `_state_store_pending_cap` reads it off a `conn` + # alias. So collect the alias names too -- any ` = self.kv_connector` + # that binds the connector itself, not a `.method()` result (those end in a + # call, so the `\s*$` anchor excludes them) -- and sweep reads through each. + readers = [r"self\.kv_connector"] + readers += [ + re.escape(a) + for a in re.findall( + r"^\s*([A-Za-z_]\w*)\s*=\s*self\.kv_connector\s*$", src, re.M + ) + ] + probed: set[str] = set() + for r in readers: + probed |= set(re.findall(rf'getattr\(\s*{r},\s*"([a-z_]+)"', src)) + probed |= set(re.findall(rf"{r}\.([a-z_]+)\(", src)) + probed |= set(re.findall(rf"{r}\.([a-z_]+)\b", src)) # `_connector_flag(name)` is the same read one indirection along. probed |= set(re.findall(r'_connector_flag\(\s*"([a-z_]+)"', src)) + # `_impl` and `_state_tier_sub` are internal delegation helpers, not members + # of the forwarded face every shell must expose: `_impl` is the shell's own + # wrapped connector, and `_state_tier_sub` is the `multi` composite's private + # route to the state-tier sub, read only when the public `max_pending_saves` + # is None (so never on the plain shell, which always answers it). probed.discard("_impl") + probed.discard("_state_tier_sub") assert probed, "the scan found nothing -- it has stopped matching the source" # A regex sweep degrades silently: hoist one `getattr(self.kv_connector, @@ -5309,6 +5333,10 @@ def test_every_member_the_scheduler_reads_is_reachable_through_the_shell(): "take_state_reports", "take_state_source_releases", "is_offload", + # Read through the `conn` alias in `_state_store_pending_cap`; only the + # alias-aware sweep above sees it, so anchor it so a regression to a + # `self.kv_connector`-only scan fails here loudly. + "max_pending_saves", } lost = sorted(must_be_seen - probed) assert not lost, ( @@ -5321,13 +5349,16 @@ def test_every_member_the_scheduler_reads_is_reachable_through_the_shell(): # `LMCacheOffloadConnectorScheduler` AND the `MultiConnectorScheduler` # composite the engine holds under `kv_transfer_config: multi`. The # composite's misses are the same silent default -- `Scheduler` reads it off - # `self.kv_connector`, which under `multi` IS the composite. The allow-list - # below is deliberately empty: a probed member the composite legitimately - # routes through `_state_tier_sub` instead of exposing (as `max_pending_saves` - # is, via an aliased `getattr` the regex does not match, so it never enters - # `probed`) would land here and SHOULD force a deliberate decision, not a - # default. If one ever does, add it here with the routing that covers it. - multi_routes_via_sub: set[str] = set() + # `self.kv_connector`, which under `multi` IS the composite. + # + # `max_pending_saves` is the one deliberate exception. The plain shell + # answers it as a public property; the composite bounds nothing of its own + # and routes the read through its state-tier sub (`_state_tier_sub`), so it + # does not expose the member directly. That routing is intentional -- it is + # listed here. Anything ELSE the composite fails to expose is NOT routed and + # lands as a failure below, forcing a deliberate decision rather than a + # silent default; add it here only with the routing that covers it. + multi_routes_via_sub: set[str] = {"max_pending_saves"} for shell in (LMCacheOffloadConnectorScheduler, MultiConnectorScheduler): allow = multi_routes_via_sub if shell is MultiConnectorScheduler else set() missing = sorted( From a5e18bc429bbfd9ce13e835e378f0bbb0e11cbb0 Mon Sep 17 00:00:00 2001 From: zejchen Date: Wed, 2 Sep 2026 08:13:37 +0000 Subject: [PATCH 141/151] style(offload): spell re.MULTILINE in full to satisfy ruff FURB167 The alias-aware shell sweep added last commit passed re.M to re.findall; ruff's FURB167 wants the non-abbreviated re.MULTILINE. Only lint-affecting change -- the pattern and behaviour are identical. Co-Authored-By: Claude Opus 4.8 --- tests/test_lmcache_offload_connector.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/test_lmcache_offload_connector.py b/tests/test_lmcache_offload_connector.py index 4a7b3dc5d5..5330b7f410 100644 --- a/tests/test_lmcache_offload_connector.py +++ b/tests/test_lmcache_offload_connector.py @@ -5293,7 +5293,7 @@ def test_every_member_the_scheduler_reads_is_reachable_through_the_shell(): readers += [ re.escape(a) for a in re.findall( - r"^\s*([A-Za-z_]\w*)\s*=\s*self\.kv_connector\s*$", src, re.M + r"^\s*([A-Za-z_]\w*)\s*=\s*self\.kv_connector\s*$", src, re.MULTILINE ) ] probed: set[str] = set() From 0513b7201b76eea64c55f218da8148b3dec262b0 Mon Sep 17 00:00:00 2001 From: zejunchen-zejun Date: Thu, 3 Sep 2026 01:25:18 +0000 Subject: [PATCH 142/151] [Kimi-K3][LMCache] Collect the three joint-park tests that never ran These three tests were nested inside the body of `test_worker_close_is_a_noop_when_no_tier_was_built`, so pytest never collected them -- they were dead assertions, not a passing suite. They are the acceptance gate for the no-tier / state-only joint-load paths, so dedent them to module level and drop the unused `self` params. All three now run and pass. --- tests/test_lmcache_offload_connector.py | 111 ++++++++++++------------ 1 file changed, 57 insertions(+), 54 deletions(-) diff --git a/tests/test_lmcache_offload_connector.py b/tests/test_lmcache_offload_connector.py index 5330b7f410..d54e169350 100644 --- a/tests/test_lmcache_offload_connector.py +++ b/tests/test_lmcache_offload_connector.py @@ -5920,61 +5920,64 @@ def test_worker_close_is_a_noop_when_no_tier_was_built(): ("load", "shutdown", True), ] - def test_the_park_does_not_leak_an_entry_per_joint_load(self): - worker = _k3_worker() - for i in range(4): - req = _k3_load_req(f"r{i}") - worker._arm_joint_loads( - SimpleNamespace(state_loads=[(req.req_id, 99, 0)], requests=[req]) - ) - worker._settle_joint({req.load_operation}, set(), {req.req_id}, set()) - park = worker._joint_park - assert park._need == {} - assert park._alias == {} - assert park._alias_of == {} - - def test_with_no_tier_the_joint_load_fails_for_recompute(self): - """No tier means the state leg cannot be served, so the pair must reach - the engine as `failed_loading` (recompute), never as a phantom success. - - Arming a joint load with no tier and then failing its state leg - (`_fail_state_loads`) leaves the park owing only the KV leg; when the KV - completion lands, `_settle_joint` releases the pair into `failed_loading` - -- and the park does not leak the entry, because that same KV completion - is what releases it. Skipping the arm instead let the KV leg pass through - as `finished_loading`, which `Scheduler._settle_state_load(ok=True)` - miscounts as a state restore that never happened.""" - worker = _k3_worker(tier=False) - req = _k3_load_req("r1") - meta = SimpleNamespace(state_loads=[("r1", 99, 0)], requests=[req]) - worker._arm_joint_loads(meta) - assert worker._joint_park.waits_for(req.load_operation) - # `_start_state_loads` on the no-tier path fails the state leg. - worker._start_state_loads(meta) - # The KV leg has not landed yet: the pair is still held, not passed. - done, failed = worker._settle_joint(set(), set(), set(), set()) - assert done == set() - assert failed == set() - assert worker._joint_park.waits_for(req.load_operation) - - # KV completion lands -> the pair resolves to failed, and nothing leaks. - done, failed = worker._settle_joint({req.load_operation}, set(), set(), set()) - assert done == set(), "no phantom finished_loading with no tier" - assert failed == {req.load_operation} - assert not worker._joint_park.waits_for(req.load_operation) - assert worker._joint_park._need == {} - assert worker._joint_park._alias == {} - assert worker._joint_park._alias_of == {} - - def test_a_single_leg_request_still_passes_straight_through(self): - """Nothing armed it, so neither channel may be held back.""" - worker = _k3_worker() - kv_only = LoadOperationId("r9", 0) - done, _failed = worker._settle_joint({kv_only}, set(), set(), set()) - assert done == {kv_only} - done, _failed = worker._settle_joint(set(), set(), {"r8"}, set()) - assert done == {"r8"} +def test_the_park_does_not_leak_an_entry_per_joint_load(): + worker = _k3_worker() + for i in range(4): + req = _k3_load_req(f"r{i}") + worker._arm_joint_loads( + SimpleNamespace(state_loads=[(req.req_id, 99, 0)], requests=[req]) + ) + worker._settle_joint({req.load_operation}, set(), {req.req_id}, set()) + park = worker._joint_park + assert park._need == {} + assert park._alias == {} + assert park._alias_of == {} + + +def test_with_no_tier_the_joint_load_fails_for_recompute(): + """No tier means the state leg cannot be served, so the pair must reach + the engine as `failed_loading` (recompute), never as a phantom success. + + Arming a joint load with no tier and then failing its state leg + (`_fail_state_loads`) leaves the park owing only the KV leg; when the KV + completion lands, `_settle_joint` releases the pair into `failed_loading` + -- and the park does not leak the entry, because that same KV completion + is what releases it. Skipping the arm instead let the KV leg pass through + as `finished_loading`, which `Scheduler._settle_state_load(ok=True)` + miscounts as a state restore that never happened.""" + worker = _k3_worker(tier=False) + req = _k3_load_req("r1") + meta = SimpleNamespace(state_loads=[("r1", 99, 0)], requests=[req]) + worker._arm_joint_loads(meta) + assert worker._joint_park.waits_for(req.load_operation) + + # `_start_state_loads` on the no-tier path fails the state leg. + worker._start_state_loads(meta) + # The KV leg has not landed yet: the pair is still held, not passed. + done, failed = worker._settle_joint(set(), set(), set(), set()) + assert done == set() + assert failed == set() + assert worker._joint_park.waits_for(req.load_operation) + + # KV completion lands -> the pair resolves to failed, and nothing leaks. + done, failed = worker._settle_joint({req.load_operation}, set(), set(), set()) + assert done == set(), "no phantom finished_loading with no tier" + assert failed == {req.load_operation} + assert not worker._joint_park.waits_for(req.load_operation) + assert worker._joint_park._need == {} + assert worker._joint_park._alias == {} + assert worker._joint_park._alias_of == {} + + +def test_a_single_leg_request_still_passes_straight_through(): + """Nothing armed it, so neither channel may be held back.""" + worker = _k3_worker() + kv_only = LoadOperationId("r9", 0) + done, _failed = worker._settle_joint({kv_only}, set(), set(), set()) + assert done == {kv_only} + done, _failed = worker._settle_joint(set(), set(), {"r8"}, set()) + assert done == {"r8"} # ── kimi_k3: a re-stored prefix must survive the aggregator's tombstone ──── From 96b881b448d324c76a8fd748b34408e1e1224cbc Mon Sep 17 00:00:00 2001 From: zejunchen-zejun Date: Thu, 3 Sep 2026 01:25:18 +0000 Subject: [PATCH 143/151] [Kimi-K3][LMCache] Arm joint loads from state ids, not metadata.requests `_arm_joint_loads` iterated `metadata.requests` and armed the park only for req_ids that also appeared in `state_loads`. But the two sets have independent sources: `state_loads` comes from `BlockManager.take_state_loads()` via `Scheduler._publish_state_loads`, while `metadata.requests` is populated by the base KV scheduler. A state-only load -- KV resident, recurrent state not, the `_park_for_remote_load` path taken when `needs_remote_load` is False -- carries no `LMCacheReqMeta` that step, so it is absent from `metadata.requests`. Iterating that set skipped exactly the state-only requests the `else` branch exists to arm: unarmed, their failure emitted nothing and they hung in WAITING_FOR_REMOTE_KVS forever (the orphan reclaimer never wakes a live parked request). Drive the loop off `state_ids` and look the KV leg up per state id via a `by_id` map instead of the other way round. Both legs still file under the KV identity; the state-only leg arms on the state channel alone. Covered by the three joint-park tests dedented in the prior commit. --- .../offload/hybrid/kimi_k3/connector.py | 30 +++++++++++++------ 1 file changed, 21 insertions(+), 9 deletions(-) diff --git a/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py b/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py index b006bb215f..40a0177528 100644 --- a/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py +++ b/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py @@ -306,28 +306,40 @@ def _arm_joint_loads(self, metadata) -> None: state_ids = {req_id for req_id, _h, _slot in loads} if not state_ids or not self._do_load: return - for req in metadata.requests: - if req.req_id not in state_ids: - continue - if req.load_spec is not None: + # Drive the loop off the state loads, not `metadata.requests`. The two + # sets have independent sources: `state_loads` arrives via + # `BlockManager.take_state_loads()` -> `Scheduler._publish_state_loads`, + # while `metadata.requests` is populated by the base KV scheduler. A + # state-only load -- KV resident, recurrent state not; the + # `_park_for_remote_load` path reached exactly when `needs_remote_load` + # came back False -- has no `LMCacheReqMeta` this step, so it is absent + # from `metadata.requests`. Iterating that set skipped precisely the + # state-only requests the `else` branch below exists for: unarmed, their + # failure emitted nothing and they hung in WAITING_FOR_REMOTE_KVS + # forever. Look the KV leg up per state id instead of the other way + # round. + by_id = {req.req_id: req for req in metadata.requests} + for req_id in state_ids: + req = by_id.get(req_id) + if req is not None and req.load_spec is not None: # Both legs: file the park under the KV identity, because that is # the one that has to reach the engine on finished/failed_loading. self._joint_park.arm( - req.req_id, + req_id, needs_kv=True, needs_state=True, kv_id=self._load_completion_id(req), ) else: - # State-only load (KV resident, recurrent state not). Arm it on - # the state leg alone. Its SUCCESS already reached the engine via - # the `_settle_joint` passthrough, but its FAILURE did not: an + # State-only load (KV resident, or no KV meta this step). Arm it + # on the state leg alone. Its SUCCESS already reached the engine + # via the `_settle_joint` passthrough, but its FAILURE did not: an # unarmed park emitted nothing, so the request sat in # WAITING_FOR_REMOTE_KVS forever (the orphan reclaimer never wakes # a live parked request). Arming makes `take_ready` surface either # outcome under the same bare id the passthrough already used. self._joint_park.arm( - req.req_id, + req_id, needs_kv=False, needs_state=True, ) From 4d18af0217b6ec76fed0c8eb3a4488e3ea48e0fe Mon Sep 17 00:00:00 2001 From: zejunchen-zejun Date: Thu, 3 Sep 2026 01:25:19 +0000 Subject: [PATCH 144/151] [Kimi-K3][LMCache] Bound the state slot in both directions in gdn_attn `state_entry_views` rejected only a negative slot. Its DSV4 twin (`deepseek_v4_attn.py`) indexes a Python list (`self._slot_views()[slot]`) and so raises `IndexError` on a stray -1 *or* an out-of-range positive; here `cache[layer, slot : slot + 1]` clamps silently instead. A -1 (the "no slot" sentinel used elsewhere on this path) gathers/scatters the last slot's state under this request's identity, and a too-large slot yields a zero-element view (`numel() == 0`, no error) that the staging packer moves as 0 bytes -- `StateByteCodec.get` then reports success on state never written and the request resumes on whatever the slot already held. Either way: silent cross-request corruption. Reject `not 0 <= slot < num_slots` (num_slots from `mamba_k_cache.shape[1]`) for full parity with the DSV4 twin. This guards the shape invariant rather than a demonstrated path -- `StateSlotPool.pop` never returns negative and `_attach_state_slots` pops before requesting the load -- so it should never fire; if it ever does, fail loudly instead of corrupting silently. --- atom/model_ops/attentions/gdn_attn.py | 25 +++++++++++++++++-------- 1 file changed, 17 insertions(+), 8 deletions(-) diff --git a/atom/model_ops/attentions/gdn_attn.py b/atom/model_ops/attentions/gdn_attn.py index 6748e70e20..6d43d0ad0c 100644 --- a/atom/model_ops/attentions/gdn_attn.py +++ b/atom/model_ops/attentions/gdn_attn.py @@ -836,17 +836,26 @@ def state_entry_views(self, slot: int) -> list[torch.Tensor]: committed state (#2045), and the speculation scratch beside it is this request's own and resumable by nobody. """ - # Reject a negative slot explicitly, for parity with the DSV4 twin - # (`deepseek_v4_attn.py`). `cache[layer, slot : slot + 1]` on a stray -1 - # (the "no slot" sentinel used elsewhere on this path) silently gathers - # or scatters the last slot's state under this request's identity -- - # silent cross-request corruption. The upstream invariant is that a real - # slot is non-negative (`StateSlotPool.pop` never returns negative, and + # Reject an out-of-range slot in BOTH directions, for parity with the + # DSV4 twin (`deepseek_v4_attn.py`), whose `self._slot_views()[slot]` is + # a list index and raises `IndexError` on a stray -1 *or* an + # out-of-range positive. Here `cache[layer, slot : slot + 1]` clamps + # silently instead: a -1 (the "no slot" sentinel used elsewhere on this + # path) gathers/scatters the last slot's state under this request's + # identity, and a too-large slot yields a zero-element view + # (`numel() == 0`, no error) that the staging packer moves as 0 bytes -- + # `StateByteCodec.get` then returns True on state never written and the + # request resumes on whatever the slot already held. Either way: silent + # cross-request corruption. The upstream invariant is a real, in-range + # slot (`StateSlotPool.pop` never returns negative, and # `_attach_state_slots` pops before requesting the load), so this guards # the shape rather than a demonstrated path; fail loudly if it is ever # violated instead of corrupting silently. - if slot < 0: - raise IndexError(f"state_entry_views: invalid negative slot {slot}") + num_slots = self.model_runner.mamba_k_cache.shape[1] + if not 0 <= slot < num_slots: + raise IndexError( + f"state_entry_views: slot {slot} out of range [0, {num_slots})" + ) views = [] for cache in ( self.model_runner.mamba_k_cache, From fa2facbb954e9b08e25963089682d15a7ff3c12a Mon Sep 17 00:00:00 2001 From: zejunchen-zejun Date: Thu, 3 Sep 2026 01:25:19 +0000 Subject: [PATCH 145/151] [Kimi-K3][LMCache] Down the state ref on any store exception `StateByteCodec.store` guarded the reference it hands to `batched_put` with a `handed_off` flag set *before* the call, meaning to distinguish "never reached `batched_put`" (must down) from "adopted, then a later step raised" (must not down). But there is no such later step: in LMCache `storage_manager.py`, `batched_put` discharges the reference in its terminal tail loop (one `ref_count_down()` per object), which runs after every step that can raise -- the scheduler-role guard, `get_allocator_backend`, `allocate_and_copy_objects`, `batched_submit_put_task`. So the down is `batched_put`'s last action: on a successful return the reference is already discharged, and no window exists where the object is adopted yet a subsequent step raises. Setting `handed_off = True` before the call therefore only suppressed the down on a *pre-adoption* `batched_put` raise, stranding the allocation at ref_count=1 (LMCache later reports "garbage collected with ref_count=1, pin_count=0") and shrinking the CPU pool one entry per failure. Drop the flag: on any exception -- from `pack`, from `on_source_released`, or from a pre-adoption raise inside `batched_put` -- we still own the reference and down it exactly once. Matches the `finally` guard in `get`. --- .../offload/hybrid/kimi_k3/state_object.py | 36 +++++++++---------- 1 file changed, 18 insertions(+), 18 deletions(-) diff --git a/atom/kv_transfer/offload/hybrid/kimi_k3/state_object.py b/atom/kv_transfer/offload/hybrid/kimi_k3/state_object.py index 5595091cf2..8f949a1242 100644 --- a/atom/kv_transfer/offload/hybrid/kimi_k3/state_object.py +++ b/atom/kv_transfer/offload/hybrid/kimi_k3/state_object.py @@ -128,33 +128,33 @@ def put(self, h: int, unit_ids, on_source_released=None) -> bool: obj = self._allocate(self.entry_bytes) if obj is None: return False - # `batched_put` discharges the reference it is handed -- but only if - # reached. A throwing `pack` skips it and strands the allocation at - # ref_count=1 (LMCache reports "garbage collected with ref_count=1, - # pin_count=0" much later), shrinking the CPU pool one entry per failure. - # `get` guards its own reference with `finally` for the same reason. + # `batched_put` discharges the reference it is handed, but only in its + # terminal tail loop (LMCache `storage_manager.py`, one + # `ref_count_down()` per object), which runs *after* every step that can + # raise -- the scheduler-role guard, `get_allocator_backend`, + # `allocate_and_copy_objects`, `batched_submit_put_task`. So the down is + # `batched_put`'s last action: on a successful return the reference is + # already discharged, and there is no window where the object is adopted + # yet a later step raises. # - # `handed_off` draws the line the bare `except` could not: it cannot tell - # "never reached `batched_put`" (must down the ref) from "reached it, it - # took ownership, then something raised" (must NOT down it). The second - # case is real -- an NVMe/L3 backend error or an eviction-path raise - # inside LMCache's put, after the CPU insert. Downing there drives the - # count to -1 / returns a still-indexed buffer to the allocator, and a - # later `get` for a DIFFERENT key hands that buffer back: wrong bytes - # unpacked into a request's Active Slot, no exception anywhere. Once the - # reference is passed to `batched_put`, ownership is its, success or raise. - handed_off = False + # That means on ANY exception below -- from `pack`, from + # `on_source_released`, or from a pre-adoption raise inside + # `batched_put` -- the reference was never downed and we still own it, so + # we down it exactly once. The earlier `handed_off = True` set *before* + # the call did the opposite: it suppressed the down on a pre-adoption + # `batched_put` raise, stranding the allocation at ref_count=1 (LMCache + # reports "garbage collected with ref_count=1, pin_count=0" much later) + # and shrinking the CPU pool one entry per failure. `get` guards its own + # reference with `finally` for the same reason. try: self._staged.pack(self._backend.page_unit_views(unit_ids), obj) # Source first: `pack` has synchronized the stream that reads the # units, so nothing on the device touches them from here. if on_source_released is not None: on_source_released() - handed_off = True self._storage.batched_put([key], [obj]) except Exception: - if not handed_off: - obj.ref_count_down() + obj.ref_count_down() raise return True From 930252aa13b62f769c05c4e7c89c319de41649f2 Mon Sep 17 00:00:00 2001 From: zejunchen-zejun Date: Thu, 3 Sep 2026 01:25:19 +0000 Subject: [PATCH 146/151] [Kimi-K3][LMCache] Reclaim stalled deferred saves in the PP engine core `PPEngineCoreProc._poll_kv_transfer_progress` fully overrides the base `EngineCore` method, and the base version's getattr-guarded call to `Scheduler._reconcile_stalled_deferred_saves` is that reclaimer's only caller repo-wide. So under `pp_size > 1` the override silently dropped it: an offload save whose completion report never came (worker crash, dropped completion, LMCache force-unpin) stayed in `Scheduler.deferred_free_blocks` forever. `has_pending_kv_work()` then stays True, so the engine busy-loops with every GPU idle, the blocks never return to the pool, and `_drain_kv_work_at_exit` spins to `KV_SHUTDOWN_DRAIN_TIMEOUT_S` on every shutdown. Mirror the reclaimer call in the override, above the has_offload / pp_messages early-returns so a quiet poll still reclaims. It is self-throttled, so calling it each poll is cheap. --- atom/model_engine/pp_engine_core.py | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/atom/model_engine/pp_engine_core.py b/atom/model_engine/pp_engine_core.py index ab863e8fde..50f0f0c6e3 100644 --- a/atom/model_engine/pp_engine_core.py +++ b/atom/model_engine/pp_engine_core.py @@ -256,6 +256,21 @@ def _poll_kv_transfer_progress(self): if not self.kv_transfer_enabled: return + # Reclaim any offload save whose completion report never came (worker + # crash, dropped completion, LMCache force-unpin). This override fully + # replaces the base `_poll_kv_transfer_progress`, whose getattr-guarded + # call is the reclaimer's only caller repo-wide; without mirroring it + # here, under `pp_size > 1` a stalled save stays in + # `Scheduler.deferred_free_blocks` forever -- `has_pending_kv_work()` + # stays True, so the engine busy-loops with every GPU idle, the blocks + # never return to the pool, and `_drain_kv_work_at_exit` spins to + # `KV_SHUTDOWN_DRAIN_TIMEOUT_S` on every shutdown. Self-throttled, so + # calling it each poll is cheap; placed above the has_offload / + # pp_messages early-returns so a quiet poll still reclaims. + reconcile = getattr(self.scheduler, "_reconcile_stalled_deferred_saves", None) + if callable(reconcile): + reconcile() + # Collect local TP-aggregated output. kvoutput = self.runner_mgr.call_func_with_aggregation("async_proc_aggregation") if kvoutput is None: From cb179e2e5766a8ee8d9f53b3761146bb5b4c2002 Mon Sep 17 00:00:00 2001 From: zejunchen-zejun Date: Thu, 3 Sep 2026 01:25:19 +0000 Subject: [PATCH 147/151] [Kimi-K3][LMCache] Gate tier-hit votes on could_serve, not bare hashes The two admission-path voters -- `PagedStateCheckpointCoordinator. _reachable` and `BlockManager._tier_can_serve` -- tested `h in offload.hashes`, dropping the `can_load` half that `request_load` itself enforces. A store-only role (`kv_producer`) populates `hashes` from its own stores but cannot load them back, so it voted a tier hit it would then refuse: the right-to-left resumable scan stopped at the tier rung, skipped a still-resident HBM rung, and `request_load` returned False -- the boundary disowned and the HBM checkpoint forfeited to a full recompute. Factor the membership+capability predicate onto the index as `StateOffloadIndex.could_serve(h) = can_load and h in hashes` (the guard of `request_load` minus its per-request in-flight check) and route both voters and `request_load` through it, so all three test exactly the same capability and cannot drift -- fix-then-sweep across the three read sites. Test doubles for the index gain the mirror method. --- atom/model_engine/block_manager.py | 11 +++++++--- atom/model_engine/page_unit_checkpoint.py | 8 ++++++- atom/model_engine/state_offload.py | 26 ++++++++++++++++++----- tests/test_page_unit_checkpoint.py | 3 +++ tests/test_state_checkpoint.py | 6 ++++++ 5 files changed, 45 insertions(+), 9 deletions(-) diff --git a/atom/model_engine/block_manager.py b/atom/model_engine/block_manager.py index a2609765cc..1e06e58557 100644 --- a/atom/model_engine/block_manager.py +++ b/atom/model_engine/block_manager.py @@ -1247,16 +1247,21 @@ def _attach_state_slots(self, seq: Sequence, hit_hash: int) -> bool: return True def _tier_can_serve(self, hit_hash: int) -> bool: - """Whether the tier believes it holds `hit_hash`. + """Whether the tier could actually serve a load for `hit_hash`. Asked twice per admission -- once by `_attach_state_slots` to decide whether to try a load at all, and again inside `_request_state_load`. - One predicate so the two cannot drift. + Both, and `request_load` itself, go through `StateOffloadIndex. + could_serve`, so all three test the same capability+membership predicate + and cannot drift. A bare `hit_hash in hashes` here dropped the `can_load` + half: a store-only role (`kv_producer`) voted a hit it would then refuse, + the resumable scan stopped at the tier rung and skipped a still-resident + HBM rung, and the boundary was disowned into a full recompute. """ return ( hit_hash != -1 and self.state_offload is not None - and hit_hash in self.state_offload.hashes + and self.state_offload.could_serve(hit_hash) ) def _request_state_load(self, seq: Sequence, hit_hash: int) -> bool: diff --git a/atom/model_engine/page_unit_checkpoint.py b/atom/model_engine/page_unit_checkpoint.py index ddfcc9027a..fb11d6ad70 100644 --- a/atom/model_engine/page_unit_checkpoint.py +++ b/atom/model_engine/page_unit_checkpoint.py @@ -807,10 +807,16 @@ def _reachable(self, h: int) -> bool: once", never "is still there". A false positive costs one park plus a recompute and retracts itself (`fail_load` -> `forget`); being certain would cost a synchronous cross-process lookup on the admission path. + + The tier's half goes through `could_serve`, not a bare `h in hashes`, so + it also honours `can_load`: a store-only role (`kv_producer`) populates + `hashes` from its own stores but cannot load them back, and voting a hit + it would then refuse strands the scan at the tier rung -- skipping a + still-resident HBM rung -- and forfeits the checkpoint to a recompute. """ if self.store.contains(h): return True - return self.offload is not None and h in self.offload.hashes + return self.offload is not None and self.offload.could_serve(h) def checkpoint(self, seq: Sequence, boundary_blocks: int, h: int) -> None: """File a boundary to be stored, keyed by hash rather than by seq. diff --git a/atom/model_engine/state_offload.py b/atom/model_engine/state_offload.py index 364c6bded4..812aae09f1 100644 --- a/atom/model_engine/state_offload.py +++ b/atom/model_engine/state_offload.py @@ -102,6 +102,23 @@ def forget(self, h: int) -> None: self.hashes.discard(h) self._hash_lru.pop(h, None) + def could_serve(self, h: int) -> bool: + """Whether a load for `h` could be offered at all: this tier can load + AND believes it holds `h`. + + The membership+capability half of `request_load`'s guard (minus its + per-request in-flight check), factored out so the admission-path voters + -- `PageUnitCheckpointCoordinator._reachable` and + `BlockManager._tier_can_serve` -- test exactly what `request_load` will + accept and cannot drift from it. Mirroring only `h in hashes` there, + without `can_load`, made a `kv_producer` (can_load=False, but `hashes` + populated from its own stores) vote a tier hit it would then refuse: the + right-to-left resumable scan stopped at the tier rung, skipped a + still-resident HBM rung, and `request_load` returned False -- the + boundary disowned and the HBM checkpoint forfeited to a full recompute. + """ + return self.can_load and h in self.hashes + # -------------------------------- loads -------------------------------- # def request_load(self, req_id, h: int) -> bool: """Offer to fetch `h` back for `req_id`. False if this tier cannot. @@ -110,11 +127,10 @@ def request_load(self, req_id, h: int) -> bool: a worker report, so offering one for a hash never stored would park the request against bytes no `get` can produce. """ - if not self.can_load: - # A store-only role. Voting for a hash whose load nothing will - # serve parks the request against a report that never comes. - return False - if h not in self.hashes: + if not self.could_serve(h): + # A store-only role (can_load False) or a hash never stored. Voting + # for either parks the request against a report that never comes; + # refuse so the boundary is disowned and recomputed. return False if req_id in self.pending_loads: # One request, one outstanding load: reports are keyed by request diff --git a/tests/test_page_unit_checkpoint.py b/tests/test_page_unit_checkpoint.py index 7c08b0a839..5e23f4408a 100644 --- a/tests/test_page_unit_checkpoint.py +++ b/tests/test_page_unit_checkpoint.py @@ -643,6 +643,9 @@ class _Index: def __init__(self, *hashes): self.hashes = set(hashes) + def could_serve(self, h): + return h in self.hashes + @staticmethod def coordinator(num_units=40, offload=None): from atom.model_engine.page_unit_checkpoint import ( diff --git a/tests/test_state_checkpoint.py b/tests/test_state_checkpoint.py index ea0eb9eac8..5adc12914f 100644 --- a/tests/test_state_checkpoint.py +++ b/tests/test_state_checkpoint.py @@ -1400,6 +1400,9 @@ def __init__(self, *hashes): self.pending_loads = {} self.requested = [] + def could_serve(self, h): + return h in self.hashes + def request_load(self, req_id, h): if h not in self.hashes: return False @@ -3568,6 +3571,9 @@ def __init__(self, *hashes): self.can_store = True self.can_load = True + def could_serve(self, h): + return self.can_load and h in self.hashes + def request_load(self, req_id, h): if h not in self.hashes: return False From 576a0b64651c66c1ee6171d3e59465b6b1b22300 Mon Sep 17 00:00:00 2001 From: zejchen Date: Thu, 3 Sep 2026 08:30:36 +0000 Subject: [PATCH 148/151] [LMCache][round6-F3] arm joint loads inside the try so the KV-leg guarantee holds Finding 3: start_load_kv() called self._arm_joint_loads(metadata) OUTSIDE the try/finally that guarantees super().start_load_kv() runs. _arm_joint_loads builds park state and can raise; if it did, the finally-super() KV-leg fallback was skipped and the KV load was silently dropped. Move the arm inside the try next to the two state helpers so every state-side helper is covered by the finally. Co-Authored-By: Claude Opus 4.8 --- atom/kv_transfer/offload/hybrid/kimi_k3/connector.py | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py b/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py index 40a0177528..ea37277913 100644 --- a/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py +++ b/atom/kv_transfer/offload/hybrid/kimi_k3/connector.py @@ -259,8 +259,8 @@ def start_load_kv(self, metadata) -> None: if not isinstance(metadata, LMCacheOffloadMetadata): super().start_load_kv(metadata) return - # The three state helpers run before super() so the KV leg is submitted - # against a park that already exists. They must not be able to skip + # The arm + three state helpers run before super() so the KV leg is + # submitted against a park that already exists. None of them may skip # super(): _JointPark's correctness -- and the refutation of the # "park leaks on abort" candidate -- both rest on "the KV leg always # reports", which holds only while super().start_load_kv() cannot be @@ -271,8 +271,13 @@ def start_load_kv(self, metadata) -> None: # into the same failed-load / failed-store channels a no-tier step uses, # so the armed park resolves to a recompute instead of hanging), and # super() runs in a finally so a KV load or save is never dropped. - self._arm_joint_loads(metadata) + # `_arm_joint_loads` is inside the try too: it builds park state and can + # raise (a MemoryError filing the dicts, say), and the same "KV leg must + # still report" invariant that puts super() in a finally requires the + # KV submit to run even when arming half-completed -- an arriving KV + # completion then settles or evicts whatever the arm did file. try: + self._arm_joint_loads(metadata) self._start_state_loads(metadata) self._start_state_stores(metadata) finally: From 15ead445064227bdda8903ee1955972454e13d4b Mon Sep 17 00:00:00 2001 From: zejchen Date: Thu, 3 Sep 2026 08:30:43 +0000 Subject: [PATCH 149/151] [LMCache][round6-F2] refuse offload_layout override that downgrades a state-owning model Finding 2: select_offload_layout() mapped an explicit offload_layout override and returned it BEFORE inspecting the model, so an override could (a) bypass the GDN refusal and (b) downgrade a kimi_linear (kimi_k3) checkpoint to a layout that owns no tier for its per-request KDA recurrent state -> a KV prefix restored over stale state = silent wrong output. Fix: always compute the natural layout via the extracted _layout_from_model() first (which still refuses GDN), then, if an override is present, refuse mapping a state-owning natural layout (_STATE_OWNING_LAYOUTS = {"kimi_k3"}) down to a non-state layout. dense<->hybrid remains a legitimate operator choice (namespace separation), so the guard is deliberately narrow. Co-Authored-By: Claude Opus 4.8 --- atom/kv_transfer/offload/config.py | 48 ++++++++++++++++++++++++---- tests/test_lmcache_offload_config.py | 38 ++++++++++++++++++++++ 2 files changed, 80 insertions(+), 6 deletions(-) diff --git a/atom/kv_transfer/offload/config.py b/atom/kv_transfer/offload/config.py index 0b11658523..d564ec08a3 100644 --- a/atom/kv_transfer/offload/config.py +++ b/atom/kv_transfer/offload/config.py @@ -71,6 +71,15 @@ {"qwen3_next", "qwen3_next_mtp", "qwen3_5_text", "qwen3_5_moe_text"} ) +# Layouts that own a tier for a model's per-request *recurrent* state. Today only +# `kimi_k3` (the kimi_linear KDA state); `hybrid` (DSV4 sparse-attention +# checkpoints) is not recurrent-state and dense<->hybrid is a legitimate operator +# override for namespace separation. An explicit `offload_layout` override may not +# downgrade a recurrent-state model to a layout that owns no tier for it (silent +# wrong output) -- see `select_offload_layout`. GDN/linear model types are refused +# outright in `_layout_from_model`, so no override can resurrect them. +_STATE_OWNING_LAYOUTS = frozenset({"kimi_k3"}) + logger = logging.getLogger("atom") @@ -96,17 +105,44 @@ def select_offload_layout(config) -> str: kvc = getattr(config, "kv_transfer_config", {}) or {} override = kvc.get("offload_layout") - if override is not None: - mapped = ( - _OFFLOAD_LAYOUT_ALIASES.get(override) if isinstance(override, str) else None - ) - if mapped is not None: - return mapped + # Inspect the model *first*, always -- even with an override present. An + # override returning before this ran let `offload_layout: dense` on a + # kimi_linear/Qwen3.5 checkpoint skip the GDN refusal below and restore a KV + # prefix over stale recurrent state (silent wrong output). The natural layout + # is also what an override is validated against. + natural = _layout_from_model(config) + if override is None: + return natural + mapped = ( + _OFFLOAD_LAYOUT_ALIASES.get(override) if isinstance(override, str) else None + ) + if mapped is None: raise ValueError( f"lmcache_offload: unknown offload_layout={override!r}; " f"expected one of {sorted(_OFFLOAD_LAYOUT_ALIASES)}" ) + # An override may pick between compatible layouts, but it may not strip a + # state-owning model down to a layout with no tier for its per-request state + # -- the same silent-wrong-output the GDN refusal pre-empts, reached by a + # different door (an explicit `dense` rather than a fall-through). + if natural in _STATE_OWNING_LAYOUTS and mapped not in _STATE_OWNING_LAYOUTS: + raise ValueError( + f"lmcache_offload: offload_layout={override!r} would run a " + f"{natural!r}-family model (per-request recurrent state) on the " + f"{mapped!r} layout, which owns no tier for that state -- its KV " + "prefix would be restored over stale state (silent wrong output). " + "Drop the override, or disable offload for this model." + ) + return mapped + +def _layout_from_model(config) -> str: + """The dense/hybrid/kimi_k3 layout the model itself implies (no override). + + May raise: a GDN/linear model that no layout owns a state tier for is + refused here (the message names the cause) rather than left to die deep in + `register_kv_caches` as a byte-layout mismatch. + """ hf_config = getattr(config, "hf_config", None) if hf_config is None: return "dense" diff --git a/tests/test_lmcache_offload_config.py b/tests/test_lmcache_offload_config.py index 4f41739683..f814b70370 100644 --- a/tests/test_lmcache_offload_config.py +++ b/tests/test_lmcache_offload_config.py @@ -166,6 +166,44 @@ def test_gdn_linear_model_offload_is_refused(model_type): offcfg.select_offload_layout(config) +@pytest.mark.parametrize( + "model_type", + ["qwen3_next", "qwen3_next_mtp", "qwen3_5_text", "qwen3_5_moe_text"], +) +def test_gdn_refusal_is_not_bypassed_by_offload_layout_override(model_type): + # The refusal must run *before* an explicit override is honoured. Otherwise + # `offload_layout: dense` on a GDN checkpoint returns `dense` early and + # restores a KV prefix over stale recurrent state (silent wrong output). + config = _config() + config.hf_config.compress_ratios = None + config.hf_config.model_type = model_type + config.kv_transfer_config = {"offload_layout": "dense"} + + with pytest.raises(ValueError, match="does not support GDN"): + offcfg.select_offload_layout(config) + + +def test_offload_layout_override_cannot_downgrade_kimi_k3_to_dense(): + # kimi_linear's KDA per-request state is owned only by the kimi_k3 layout; + # an override to a layout with no tier for it is silent wrong output, so it + # must be refused rather than silently honoured. + config = _config() + config.hf_config.compress_ratios = None + config.hf_config.model_type = "kimi_linear" + config.kv_transfer_config = {"offload_layout": "dense"} + + with pytest.raises(ValueError, match="owns no tier"): + offcfg.select_offload_layout(config) + + +def test_offload_layout_override_still_allows_dense_hybrid_choice(): + # The downgrade guard is narrow: dense<->hybrid is a legitimate operator + # choice (namespace separation), not a recurrent-state downgrade. + config = _config() # compress_ratios set -> natural "hybrid" + config.kv_transfer_config = {"offload_layout": "dense"} + assert offcfg.select_offload_layout(config) == "dense" + + def test_minimax_and_dense_model_types_still_route_to_dense(): # The refusal must be narrow: a non-GDN model with no compress_ratios is # ordinary dense, including MiniMax (sparse/standard attention, no state). From 92dcef82220e941fdee708c7ab1e92d42ec9ce25 Mon Sep 17 00:00:00 2001 From: zejchen Date: Thu, 3 Sep 2026 08:30:51 +0000 Subject: [PATCH 150/151] [LMCache][round6-F1] admit multimodal prefill whole instead of forcing a checkpoint-rung cut Finding 1: _finalize_prefill_chunk() always aligned the prefill chunk to a state-checkpoint rung via checkpoint_cut(). For a multimodal prompt (taken as one atomic unit) whose length is off the rung grid, the deterministic cut kept shaving the chunk to the same sub-rung length every pass -> the request could never make prefill progress (livelock). Fix: when seq.multimodal_data is present, return the whole chunk before checkpoint_cut(). A checkpoint is only ever kept where a forward ends on a rung, so an off-grid whole prompt simply keeps no mid-prompt checkpoint; forks do not arise on a fresh multimodal request, so no correctness is lost. Co-Authored-By: Claude Opus 4.8 --- atom/model_engine/scheduler.py | 14 +++++++++ tests/test_scheduler.py | 52 ++++++++++++++++++++++++++++++++++ 2 files changed, 66 insertions(+) diff --git a/atom/model_engine/scheduler.py b/atom/model_engine/scheduler.py index e1c0480aeb..1ecf0905f6 100644 --- a/atom/model_engine/scheduler.py +++ b/atom/model_engine/scheduler.py @@ -2002,6 +2002,20 @@ def _finalize_prefill_chunk(self, seq: Sequence, start: int, chunk: int) -> int: than one checkpoint interval. """ bm = self.block_manager + # A multimodal prompt is prefilled whole -- chunked multimodal prefill is + # unsupported (see the `atomic_prefill` guard in `_schedule_prefills`), so + # it is either admitted entire or requeued entire. Landing it on a state + # checkpoint rung shortens it deterministically, and the post-alloc atomic + # re-assert then requeues it whole; the very next pass produces the same + # shortened chunk and requeues again -- a livelock with idle GPUs and + # head-of-line blocking. Skip the rung cut here: the checkpoint machinery + # keeps a checkpoint only where a forward ends exactly on a rung, so a + # whole prompt that ends off-grid simply keeps no mid-prompt checkpoint, + # which is correct for a single-shot prefill (there is no later chunk to + # resume from anyway). Forks do not arise on a fresh multimodal prompt + # (`state_fork_src < 0`), so the fork-vetting below is likewise moot. + if getattr(seq, "multimodal_data", None) is not None: + return chunk target = bm.checkpoint_cut(seq, start, start + chunk) if target: chunk = target - start diff --git a/tests/test_scheduler.py b/tests/test_scheduler.py index 0fe7756aa7..5addafff5d 100644 --- a/tests/test_scheduler.py +++ b/tests/test_scheduler.py @@ -711,6 +711,58 @@ def test_text_prefill_shortened_after_alloc_still_splits(self, seq_factory): assert batch.total_seqs_num_prefill == 1 assert list(batch.num_scheduled_tokens) == [4] + def test_multimodal_prefill_spanning_checkpoint_rung_admits_whole( + self, seq_factory + ): + # The requeue test above forces the shortening through the *offload* + # adjuster, whose real-world shortening goes away once the load lands -- + # so a single pass is enough to prove the requeue. The state-checkpoint + # rung cut in `_finalize_prefill_chunk` is different: it is deterministic, + # so a multimodal prompt spanning one interval is shortened the *same* way + # every pass, and requeue-whole then loops forever (idle GPUs, head-of- + # line blocking). A one-pass test cannot see that. Drive two passes with a + # fixed rung cut in place and assert the prompt is admitted whole and + # prefill actually completes -- the cut must be suppressed for multimodal. + sched = Scheduler( + MockConfig( + max_num_batched_tokens=64, + num_kvcache_blocks=100, + kv_cache_block_size=4, + enable_chunked_prefill=True, + ) + ) + # A rung that always lands 4 tokens short of the chunk end -- the same + # shortening on every pass, exactly as a real ladder cuts a prompt that + # spans an interval. If the cut were honoured for multimodal, the atomic + # re-assert would requeue whole and this seq would never make progress. + sched.block_manager.checkpoint_cut = lambda seq, start, end: end - 4 + seq = seq_factory(list(range(8)), multimodal_data={"pixel_values": object()}) + sched.add(seq) + + # Pass 1: admitted whole, not shortened to a 4-token partial, not requeued. + batch1, _ = sched.schedule() + assert batch1.total_seqs_num_prefill == 1 + assert list(batch1.num_scheduled_tokens) == [8] + assert seq.is_partial_prefill is False + assert seq not in sched.waiting + + sched.postprocess( + list(sched.running), + ScheduledBatchOutput( + req_ids=[], + token_ids=[], + num_rejected=None, + num_bonus=None, + draft_token_ids=None, + ), + batch=batch1, + ) + + # Pass 2: prefill is done, so the seq is not bounced back to waiting to + # be re-shortened -- the livelock would show here as the seq reappearing + # in `waiting` with no forward progress. + assert seq not in sched.waiting + def test_prefill_respects_block_availability(self, seq_factory): sched = Scheduler(MockConfig(num_kvcache_blocks=1, kv_cache_block_size=4)) sched.add(seq_factory([1, 2, 3, 4])) # 1 block From adb1e346f089bdc98ad716ecdfcd9d2f534464d9 Mon Sep 17 00:00:00 2001 From: zejchen Date: Thu, 3 Sep 2026 08:30:58 +0000 Subject: [PATCH 151/151] [LMCache][round6-open2] track orphan load slots as a per-id list so a re-admission cannot drop a slot Still-open 2: _orphan_load_slots mapped req id -> a single (slot, ts) tuple. If a request was preempted and re-admitted while a state load was still in flight, the second park overwrote the first entry and leaked the first slot; settle/abandon also released the wrong (only) slot. Fix: store a list of (slot, ts) per id, append on each park, and release FIFO via _release_oldest_orphan_slot() (matching worker report order). settle_state_load and abandon_state_load release the oldest; reconcile_orphan_load_slots expires stale entries per-slot by age while keeping fresh ones. The entry is deleted when its list empties. Co-Authored-By: Claude Opus 4.8 --- atom/model_engine/block_manager.py | 92 +++++++++++++++++++----------- tests/test_block_manager.py | 49 ++++++++++++++-- 2 files changed, 103 insertions(+), 38 deletions(-) diff --git a/atom/model_engine/block_manager.py b/atom/model_engine/block_manager.py index 1e06e58557..ba3faf1ce1 100644 --- a/atom/model_engine/block_manager.py +++ b/atom/model_engine/block_manager.py @@ -317,21 +317,25 @@ def __init__( # the connector. Kept here rather than in the index because the slot # is this object's fact. self._state_loads: list[tuple] = [] - # req_id -> (slot, monotonic stamp), for loads whose request was + # req_id -> [(slot, monotonic stamp), ...], for loads whose request was # deallocated before the bytes landed. The slot is off the free list - # until the report comes back; see `deallocate`. One dict, not two - # parallel ones keyed the same way: `seq.id` is per-request not - # per-admission, so a preempt/re-admit can park the same id twice, and - # two dicts let an overwrite of one desync from the other -- stranding a - # slot the reconciler can no longer find. The stamp trusts - # `settle_state_load` to "always come": it does not, if the worker that - # owed the report crashed or its completion was dropped. Then the slot - # would sit off the free list forever and the state gate in + # until the report comes back; see `deallocate`. A *list* per id, not a + # single tuple: `seq.id` is per-request not per-admission, so a + # preempt/re-admit can park the same id twice while the first load is + # still in flight. A single-value entry would overwrite -- silently + # dropping the first slot (leaked forever) and, worse, releasing the + # wrong slot when the first report arrives (a slot still being written). + # Append instead and release oldest-first: the worker submits and reports + # a request's loads in admission order, so the FIFO pop in + # `settle_state_load`/`abandon_state_load` matches slot to report. The + # stamp trusts `settle_state_load` to "always come": it does not, if the + # worker that owed the report crashed or its completion was dropped. Then + # the slot would sit off the free list forever and the state gate in # `can_allocate` would wedge the pool once every slot is stranded, so # `reconcile_orphan_load_slots` reclaims it after the same abandon # window the store pins use -- the load-side twin of # `reclaim_stale_state_store_pins`. - self._orphan_load_slots: dict[object, tuple[int, float]] = {} + self._orphan_load_slots: dict[object, list[tuple[int, float]]] = {} self._orphan_load_slots_reclaimed: int = 0 # `kv_offload_enabled` is the whole switch: without a connector to # carry a transfer there is nothing beneath the pool, and a hash whose @@ -1319,9 +1323,23 @@ def abandon_state_load(self, req_id) -> None: if self.state_offload is None: return self.state_offload.abandon_load(req_id) - parked = self._orphan_load_slots.pop(req_id, None) - if parked is not None: - self.state.release(parked[0]) + self._release_oldest_orphan_slot(req_id) + + def _release_oldest_orphan_slot(self, req_id) -> None: + """Hand back the oldest orphan-parked slot for `req_id`, if any. + + FIFO: a preempt/re-admit can park a request's slot more than once, and + the worker reports its loads in admission order, so the oldest park is + the one this report settles. Releasing newest-first would free a slot a + later load is still writing. + """ + parked = self._orphan_load_slots.get(req_id) + if not parked: + return + slot, _at = parked.pop(0) + if not parked: + del self._orphan_load_slots[req_id] + self.state.release(slot) def settle_state_load(self, req_id, ok: bool) -> None: """Apply one worker load report. Keyed by request, like the KV load. @@ -1338,9 +1356,7 @@ def settle_state_load(self, req_id, ok: bool) -> None: self.state_offload.complete_load(req_id) else: self.state_offload.fail_load(req_id) - parked = self._orphan_load_slots.pop(req_id, None) - if parked is not None: - self.state.release(parked[0]) + self._release_oldest_orphan_slot(req_id) def take_state_stores(self, max_inflight: int) -> list[tuple]: """`(operation, unit_ids)` for checkpoints to hand the CPU tier. @@ -1486,22 +1502,30 @@ def reconcile_orphan_load_slots(self, timeout_s: float) -> int: if timeout_s <= 0 or not self._orphan_load_slots: return 0 cutoff = monotonic() - timeout_s - stale = [ - req_id - for req_id, (_slot, at) in self._orphan_load_slots.items() - if at <= cutoff - ] - for req_id in stale: - parked = self._orphan_load_slots.pop(req_id, None) - if parked is not None: - self.state.release(parked[0]) - # The index leg was already abandoned in `deallocate` when the slot - # was parked; only the slot itself outlived the report. Do not touch - # the index again here -- a second `abandon_load` would double-count - # the miss. A late report that still calls in finds the slot popped - # and releases nothing. - self._orphan_load_slots_reclaimed += 1 - return len(stale) + reclaimed = 0 + for req_id in list(self._orphan_load_slots): + parked = self._orphan_load_slots[req_id] + # Each admission is stamped and expires on its own age -- a stale + # park is dropped even when a newer one for the same id is still + # inside the window (kept in place). + fresh = [] + for slot, at in parked: + if at <= cutoff: + # The index leg was already abandoned in `deallocate` when the + # slot was parked; only the slot itself outlived the report. + # Do not touch the index again here -- a second `abandon_load` + # would double-count the miss. A late report that still calls + # in finds this slot gone and releases the next parked one. + self.state.release(slot) + reclaimed += 1 + else: + fresh.append((slot, at)) + if fresh: + self._orphan_load_slots[req_id] = fresh + else: + del self._orphan_load_slots[req_id] + self._orphan_load_slots_reclaimed += reclaimed + return reclaimed def take_state_loads(self) -> list[tuple]: """`(req_id, hash, target_slot)` for loads admitted since the last call. @@ -2481,7 +2505,9 @@ def deallocate(self, seq: Sequence): # comes: the worker reports every load either way. The rollback # scratch is nobody's destination and goes back regardless. if seq.offload_joint.load_hash != -1 and self.state_offload is not None: - self._orphan_load_slots[seq.id] = (seq.state_slot, monotonic()) + self._orphan_load_slots.setdefault(seq.id, []).append( + (seq.state_slot, monotonic()) + ) # Abandoned, not failed: an abort says nothing about the bytes, # and forgetting the hash would cost the next request over this # prefix a full recompute. diff --git a/tests/test_block_manager.py b/tests/test_block_manager.py index 4e97c6c9e9..c2ed6f1ad6 100644 --- a/tests/test_block_manager.py +++ b/tests/test_block_manager.py @@ -1075,7 +1075,7 @@ def test_settle_frees_the_parked_slot_and_clears_the_entry(self): from time import monotonic bm = self._bm() - bm._orphan_load_slots["r"] = (7, monotonic()) + bm._orphan_load_slots["r"] = [(7, monotonic())] bm.settle_state_load("r", ok=True) assert bm.state.released == [7] assert "r" not in bm._orphan_load_slots @@ -1085,7 +1085,7 @@ def test_abandon_frees_the_parked_slot(self): from time import monotonic bm = self._bm() - bm._orphan_load_slots["r"] = (7, monotonic()) + bm._orphan_load_slots["r"] = [(7, monotonic())] bm.abandon_state_load("r") assert bm.state.released == [7] assert "r" not in bm._orphan_load_slots @@ -1094,7 +1094,7 @@ def test_reconcile_frees_a_slot_whose_report_never_came(self): from time import monotonic bm = self._bm() - bm._orphan_load_slots["r"] = (7, monotonic()) + bm._orphan_load_slots["r"] = [(7, monotonic())] assert bm.reconcile_orphan_load_slots(timeout_s=1e-9) == 1 assert bm.state.released == [7] assert bm._orphan_load_slots_reclaimed == 1 @@ -1104,7 +1104,7 @@ def test_reconcile_spares_a_slot_still_inside_its_window(self): from time import monotonic bm = self._bm() - bm._orphan_load_slots["r"] = (7, monotonic()) + bm._orphan_load_slots["r"] = [(7, monotonic())] assert bm.reconcile_orphan_load_slots(timeout_s=3600) == 0 assert bm.state.released == [] assert "r" in bm._orphan_load_slots @@ -1113,11 +1113,50 @@ def test_a_late_report_after_reconcile_releases_nothing(self): from time import monotonic bm = self._bm() - bm._orphan_load_slots["r"] = (7, monotonic()) + bm._orphan_load_slots["r"] = [(7, monotonic())] bm.reconcile_orphan_load_slots(timeout_s=1e-9) bm.settle_state_load("r", ok=True) # the report finally arrives assert bm.state.released == [7], "freed once by reconcile, not twice" + def test_a_preempt_readmit_parks_the_same_id_twice_without_dropping_a_slot( + self, + ): + # `seq.id` is per-request, not per-admission: a preempt/re-admit can park + # the same id twice while the first load is still in flight. A single + # tuple would overwrite -- leaking slot 7 forever and, worse, releasing + # slot 8 (still being written) when the first report lands. Appended and + # released oldest-first, each report frees the slot its load actually + # settled. + from time import monotonic + + bm = self._bm() + first = (7, monotonic()) + second = (8, monotonic()) + bm._orphan_load_slots["r"] = [first] # first admission + bm._orphan_load_slots["r"].append(second) # re-admission, load in flight + + bm.settle_state_load("r", ok=True) # first (oldest) load reports + assert bm.state.released == [7] + assert bm._orphan_load_slots["r"] == [second] # slot 8 still parked + + bm.settle_state_load("r", ok=True) # second load reports + assert bm.state.released == [7, 8] + assert "r" not in bm._orphan_load_slots # entry cleared when empty + + def test_reconcile_expires_one_admission_and_keeps_the_fresh_one(self): + # Each parked admission ages on its own stamp: a stale one is reclaimed + # even while a newer park for the same id is still inside the window. + from time import monotonic + + bm = self._bm() + now = monotonic() + bm._orphan_load_slots["r"] = [(7, now - 3600), (8, now)] # old, fresh + + assert bm.reconcile_orphan_load_slots(timeout_s=1.0) == 1 + assert bm.state.released == [7] + assert bm._orphan_load_slots["r"] == [(8, now)] + assert bm._orphan_load_slots_reclaimed == 1 + # ── the LMCache chunk probe is gated on the capability ─────────────────────