Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
155 commits
Select commit Hold shift + click to select a range
5bb6c10
fix(offload): the generic fixes #1960 was holding behind the state tier
zejunchen-zejun Aug 27, 2026
4208e3e
lmcache(k3): port the state offload tier onto #2045's per-slot pool
zejunchen-zejun Aug 27, 2026
51b530a
refactor(offload): delete the staging ring, which #2045 left nothing …
zejunchen-zejun Aug 27, 2026
bc7b61c
lmcache(k3): one CPU pool for KV and state, not two
zejunchen-zejun Aug 27, 2026
b9829ba
lmcache(k3): bind the state key to its layout, and refuse a misfit read
zejunchen-zejun Aug 27, 2026
c039fc1
k3: name a checkpoint's PAGE units as tensors, for the CPU tier's packer
zejunchen-zejun Aug 27, 2026
9d270b2
lmcache(k3): store a checkpoint when it becomes READY, not when it is…
zejunchen-zejun Aug 27, 2026
24afbce
fix(state-cache): a gated boundary neither tier has is disowned, not …
zejunchen-zejun Aug 27, 2026
ca04f3d
lmcache(k3): give the CPU tier a vote, and let a PAGE resume come fro…
zejunchen-zejun Aug 27, 2026
7e9a87b
lmcache(k3): the joint boundary now serves PAGE, which is its only re…
zejunchen-zejun Aug 27, 2026
5c7e516
fix(offload): the delegating shell never forwarded `enqueue_state_sto…
Aug 27, 2026
9fabaa2
feat(offload): counters for the store leg, and stop reporting a pool …
Aug 27, 2026
445693b
fix(offload): the K3 state packer imported a module that does not exist
Aug 27, 2026
29a7cac
fix(offload): the same two wiring gaps, in the places the fixes did n…
zejunchen-zejun Aug 27, 2026
8b849e2
fix(state-cache): the CPU tier gathered a checkpoint's padding as if …
Aug 27, 2026
06cbdd4
fix(offload): the state key treated an unsigned block hash as signed
Aug 27, 2026
1d201dc
test(state-cache): pin the image trim, which nothing reached
zejunchen-zejun Aug 28, 2026
457332a
style: clear the ruff findings CI reports on this PR
zejunchen-zejun Aug 28, 2026
515260b
fix(offload): the delegating shell's take_state_reports fallback retu…
Aug 28, 2026
88a4b36
feat(offload): log an HBM-vs-lmcache tier split so the value of the C…
Aug 28, 2026
b36f3a3
fix(offload): the tier split paired a reach against a served count
zejunchen-zejun Aug 28, 2026
d895905
style: clear CI's ruff findings, and bring the comments down to packa…
zejunchen-zejun Aug 28, 2026
9164ae1
style: drop three `noqa: BLE001` that suppressed a rule CI does not e…
zejunchen-zejun Aug 28, 2026
118e0b9
docs+style: drop the env-var section and the V4 commentary this PR di…
zejunchen-zejun Aug 28, 2026
4cf7305
Merge branch 'main' into zejun/lmcache_with_2045
zejunchen-zejun Aug 30, 2026
190c544
fix(offload): give the joint load's two legs one completion identity
zejunchen-zejun Aug 30, 2026
9f8c50e
fix(offload): key state-store completions by a generation, not a bare…
zejunchen-zejun Aug 30, 2026
5f8bdee
fix(offload): split a state store's source release from its index result
zejunchen-zejun Aug 30, 2026
3c59ebd
perf(offload): give state loads their own lane, and measure the wait
zejunchen-zejun Aug 30, 2026
ad31789
fix(offload): decide the state tier from a capability, not the connec…
zejunchen-zejun Aug 30, 2026
fb4a069
fix(offload): aim the PAGE state leg at the joint boundary, and gate …
zejunchen-zejun Aug 30, 2026
8ad9249
style: sort the import block and drop two unused unpacks
zejunchen-zejun Aug 30, 2026
24dfb9a
fix(offload): forward the state-store leg through the multi connector
Aug 30, 2026
0d70420
fix(offload): publish the state-store source release once, at the D2H…
Aug 30, 2026
951dab4
fix(k3-offload): repair state-tier lifecycle & liveness in the worker…
Aug 31, 2026
18a9869
fix(k3-offload): drain the device on StagedTransfer pack/unpack failu…
Aug 31, 2026
ccddb8f
fix(offload): forward _state_tier and chunk_size through the multi co…
Aug 31, 2026
73eccab
fix(offload): read joint-chunk under multi, and reclaim stranded load…
Aug 31, 2026
7252da6
fix(offload): defer, don't drop, store nominations whose hash is in f…
Aug 31, 2026
c531ddb
fix(offload): surface orphan_load_slots_reclaimed on the checkpoint f…
Aug 31, 2026
c88f46b
refactor(offload): remove the dead worker-tier stats() hook
Aug 31, 2026
7e08f59
fix(ci): satisfy the pinned Black version, and re-fix a stale joint t…
zejunchen-zejun Aug 31, 2026
95f940e
fix(offload/kimi_k3): return False for unowned connector completions
Aug 31, 2026
1745d8f
fix(scheduler): guard block_manager deref in _publish_state_loads
Aug 31, 2026
7a6534a
fix(block_manager): replace -O-stripped assert in allocate() claim loop
Aug 31, 2026
0386c6d
fix(multi_connector): release state-store completions instead of leaking
Aug 31, 2026
46d7e70
fix(block_manager): pool_pressure fates via aggregator, not a single …
Aug 31, 2026
d9a731d
fix(page_unit_checkpoint): requeue deferred nominations oldest-first
Aug 31, 2026
8acac20
test: drop two brittle source-text assertions
Aug 31, 2026
9c86417
docs: correct docstrings that named dead _do_spill / _submit_state_sp…
Aug 31, 2026
4b813c6
fix(offload): abandon_save hook so a reclaimed stalled save cannot we…
Aug 31, 2026
56e244d
fix(offload/kimi_k3): check active load before the save-stall free es…
Aug 31, 2026
7d343dc
fix(offload/kimi_k3): report no-tier joint loads as failed, not phant…
Aug 31, 2026
e9f593a
fix(offload/dense): fail closed when a GDN model registers state on t…
Aug 31, 2026
77ebcf3
fix(offload/dense): round-robin the save scan so bounded slots are sh…
Aug 31, 2026
2889a2d
[LMCache] #2: privatise claimed prefix on disown to stop shared-block…
Aug 31, 2026
2ac39f4
#8-4e: clamp num_offload_tokens into the CacheStats reuse chain
Aug 31, 2026
3f4df31
#8-4a: honor the per-connector max_pending_saves cap in the scheduler
Aug 31, 2026
b039bc0
#8-4f: stop double-counting refused state stores as failures
Aug 31, 2026
29ba254
#7: make can_allocate a side-effect-free probe of the joint boundary
Aug 31, 2026
f35f0f9
#8-4c: document the three-way offload selector and kimi_k3 modules
Aug 31, 2026
7c9015a
#8-4d: drop StagedTransfer's dead free_event fence writes
Aug 31, 2026
169edb4
[LMCache][multi] Route the state face by tier ownership, not method p…
Aug 31, 2026
4551281
[LMCache][multi] Cancel losing subs' armed loads and forward load can…
Aug 31, 2026
e74b8c5
Merge origin/main into lmc_copilot_fixes
Aug 31, 2026
aa8a2b3
fix(ci): gate the LMCache chunk probe, and rename one unused unpack
zejunchen-zejun Sep 1, 2026
6565f10
fix(offload): guard the shell's abandon_save forward so DSV4 can't we…
Sep 1, 2026
e3ea1ae
fix(offload): refuse a joint boundary whose claim isn't hash-block-al…
Sep 1, 2026
24cd6c4
fix(offload): keep hit_hash on the last cached block when admission b…
Sep 1, 2026
43a0a0d
perf(offload): compute the joint boundary below the pool-fit refusal,…
Sep 1, 2026
2ae5d6e
fix(offload): arm state-only state loads so their failure reaches the…
Sep 1, 2026
17bf523
[Kimi-K3][LMCache] Keep the engine polling while a state store is pinned
Sep 1, 2026
4305153
[Kimi-K3][LMCache][multi] Route the 3 load-decision forwarders throug…
Sep 1, 2026
535b8f8
[Kimi-K3][LMCache][multi] Read the state-store pending cap from the t…
Sep 1, 2026
e44cfe3
[Kimi-K3][LMCache] Document why should_defer_free drops the stalled s…
Sep 1, 2026
5961f0c
[Kimi-K3][LMCache][test] Drop the stale _state_store_failed_locally f…
Sep 1, 2026
cabf99b
[DSV4] Reject a negative state slot instead of silently wrapping
Sep 1, 2026
f722bd8
[Kimi-K3][LMCache] Fix the state_stores tuple comment: op id, not sta…
Sep 1, 2026
bb99703
[Kimi-K3][LMCache] Document the two offload env vars the scheduler re…
Sep 1, 2026
6a8e8ce
[Kimi-K3][LMCache] Annotate _store_failed_no_tier with what it actual…
Sep 1, 2026
1e14a81
[stats] Correct the served-reuse invariant in two hit-rate docstrings
Sep 1, 2026
7bca189
[Kimi-K3][LMCache][docs] Document the offload env vars and the engine…
Sep 1, 2026
6fd43af
[Kimi-K3][LMCache][test] Close two gaps in the shell-coverage sweep
Sep 1, 2026
9a32cff
[Kimi-K3][LMCache] Isolate each offload sub in multi.process_completions
Sep 1, 2026
367792e
[scheduler] Re-assert the multimodal atomic-prefill invariant after a…
Sep 1, 2026
b9b0c03
[Kimi-K3][LMCache][test] Run the PAGE-unit geometry on the CPU tier
Sep 1, 2026
5416b3c
[Kimi-K3][LMCache] Refuse GDN/linear models at offload config resolution
Sep 1, 2026
e7e3781
[Kimi-K3][LMCache] Dedup the MemoryObj->uint8 validation into one helper
Sep 1, 2026
16f991f
[Kimi-K3][LMCache] Call _page_unit_regions bare in page_unit_views
Sep 1, 2026
f42500a
[Kimi-K3][LMCache] Split should_defer_free into a pure query + a free…
Sep 1, 2026
a8dfaff
[Kimi-K3][LMCache] Join offload copy threads at worker teardown
Sep 1, 2026
c6c6432
[Kimi-K3][LMCache] Cut comment density in the new offload modules
Sep 1, 2026
9bc652c
[Kimi-K3][LMCache] Enforce the offload save/load lifecycle at constru…
Sep 1, 2026
3ca5508
[Kimi-K3][LMCache] Route the state tier by type, not method presence
Sep 1, 2026
2de950b
[offload] §2a: source the save-abandon window from the connector
Sep 1, 2026
ffbe86f
[offload] §2b: expose the save bound publicly, stop reaching through …
Sep 1, 2026
3a38ae8
[offload] §2d: declare state_backend on KVTransferTensors
Sep 1, 2026
aa09dd1
[offload] §2c: group the seven joint-transfer fields into OffloadJoin…
Sep 1, 2026
f2b54e1
[offload] §1 slice(1): make the joint-boundary computation a side-eff…
Sep 1, 2026
96eb8ee
[offload] §1 slice(1b): one chunk-size producer, no LMCache import in…
Sep 1, 2026
c2bf425
[offload] scrub dangling chunk_size docstring refs left by slice(1b)
Sep 1, 2026
784b7cd
[offload] fix can_allocate docstring: the fit probe is not "pure"
Sep 1, 2026
db9a7df
style: satisfy Black's stub-body rule and three ruff findings
zejunchen-zejun Sep 1, 2026
f432fdf
[offload] tighten the can_allocate fit-probe docstring
Sep 1, 2026
bc3bbc8
[offload] declare the shell-forwarding axis across both shells and th…
Sep 1, 2026
a05b159
[offload] filter _state_loads in deallocate, matching cancel_state_load
Sep 1, 2026
07b9f70
[offload][kimi_k3] isolate the pre-super() state submits so super() a…
Sep 1, 2026
49ef7e4
[offload][kimi_k3] allocate the staging buffer on the first consumer …
Sep 1, 2026
3fa6bbb
[offload][kimi_k3] abandon (not fail) a state load whose only failed …
Sep 1, 2026
18819a2
[offload][kimi_k3] guard state_entry_views against a negative slot, m…
Sep 1, 2026
8337571
[offload][kimi_k3] key the page-unit region cache on full layout, not…
Sep 1, 2026
b1b1227
[offload][kimi_k3] gate _start_state_loads on _do_load, matching _arm…
Sep 1, 2026
3cfae83
test: restore the import-time torch stub so it cannot leak across col…
Sep 1, 2026
ea2cb10
[offload][kimi_k3] correct the staging_lanes HBM claim and let its me…
Sep 1, 2026
e63113c
[offload][kimi_k3] derive the save-stall clock from the abandon windo…
Sep 1, 2026
e147175
[docs] correct the offload env-var documentation to match the code th…
Sep 1, 2026
9d46949
[offload] fix OffloadJointRecord docstring/init comment: six fields, …
Sep 1, 2026
f6c4130
[config] drop the "V4 compressor ring" from the per_request_state com…
Sep 1, 2026
4758478
[offload] MiniMax-M3 is sparse paged attention, not GDN: correct the …
Sep 1, 2026
27fe4a8
[offload] kimi_k3 staging: guard _handoff/_finish against the non-CUD…
Sep 2, 2026
774765c
[offload] _JointPark.arm: evict stale prior-park bookkeeping on re-arm
Sep 2, 2026
8093ac3
[offload] PP: aggregate connector_completions across stages instead o…
Sep 2, 2026
1a127bb
[offload] _profile_enabled: read empty OFFLOAD_PROFILE as off, matchi…
Sep 2, 2026
36d9e33
[offload] block_manager: fold _extend_hash_chain's duplicated loop on…
Sep 2, 2026
81b06a5
[offload] state_offload: drop dead kv_connector_hosts_state_tier and …
Sep 2, 2026
27c22f3
[offload] get_cache_statistics: include offload_tokens in the DP-aggr…
Sep 2, 2026
7a2684b
[block_manager] skip joint-boundary computation on record=False fit p…
Sep 2, 2026
e4e7080
[block_manager] rename state_hbm/state_tier int counters to *_boundaries
Sep 2, 2026
01a6fe5
[offload] bound the two unbounded state-offload structures
Sep 2, 2026
a0fae1f
fix(offload): forward take_state_load_survived, and unbreak the worke…
zejunchen-zejun Sep 2, 2026
6b59c9b
fix(offload): do not double-discharge the state MemoryObj on a post-h…
Sep 2, 2026
bfa969e
fix(offload): resolve offload layout off the text config, not the wra…
Sep 2, 2026
5e4edd5
fix(offload): make the state-store pin two-phase so a source release …
Sep 2, 2026
ef79437
fix(offload): park the orphan load slot in one dict of (slot, stamp)
Sep 2, 2026
ca4708a
fix(offload): give _JointPark an expiry/eviction exit for lost reports
Sep 2, 2026
9b5b1bf
fix(offload): one consistent answer for the [dense, kimi_k3] composite
Sep 2, 2026
d45c545
docs(offload): match round-5 docs to code (README GDN refusal, env de…
Sep 2, 2026
c5cba14
refactor(offload): drop dead code flagged in round 5
Sep 2, 2026
d6e204d
fix(offload): guard _SubConnectorView.__getattr__ against unbound _co…
Sep 2, 2026
0c833ba
fix(offload): back out the optimistic stores_attempted bump on a refusal
Sep 2, 2026
f8423a1
fix(offload): do not publish new state work while draining KV at exit
Sep 2, 2026
f88dbf3
test(offload): make the shell sweep alias-aware so aliased reads stay…
Sep 2, 2026
a5e18bc
style(offload): spell re.MULTILINE in full to satisfy ruff FURB167
Sep 2, 2026
a19828e
Merge branch 'main' into zejun/lmcache_with_2045
zejunchen-zejun Sep 2, 2026
0513b72
[Kimi-K3][LMCache] Collect the three joint-park tests that never ran
zejunchen-zejun Sep 3, 2026
96b881b
[Kimi-K3][LMCache] Arm joint loads from state ids, not metadata.requests
zejunchen-zejun Sep 3, 2026
4d18af0
[Kimi-K3][LMCache] Bound the state slot in both directions in gdn_attn
zejunchen-zejun Sep 3, 2026
fa2facb
[Kimi-K3][LMCache] Down the state ref on any store exception
zejunchen-zejun Sep 3, 2026
930252a
[Kimi-K3][LMCache] Reclaim stalled deferred saves in the PP engine core
zejunchen-zejun Sep 3, 2026
cb179e2
[Kimi-K3][LMCache] Gate tier-hit votes on could_serve, not bare hashes
zejunchen-zejun Sep 3, 2026
84ed7d9
Merge branch 'main' into zejun/lmcache_with_2045
Sep 3, 2026
576a0b6
[LMCache][round6-F3] arm joint loads inside the try so the KV-leg gua…
Sep 3, 2026
15ead44
[LMCache][round6-F2] refuse offload_layout override that downgrades a…
Sep 3, 2026
92dcef8
[LMCache][round6-F1] admit multimodal prefill whole instead of forcin…
Sep 3, 2026
adb1e34
[LMCache][round6-open2] track orphan load slots as a per-id list so a…
Sep 3, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions atom/config.py
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,15 @@ class KVCacheTensor:
replay_buf_k: torch.Tensor = None
replay_buf_u: torch.Tensor = None
replay_buf_g: torch.Tensor = None
# True when the tensors above are a hybrid's PER-REQUEST state (GDN/KDA
# recurrent state) rather than paged KV. Only the GDN and KDA linear-attn
# forwards set it (`gdn_attn.py`, `kimi_mla_gdn_attn.py`, and the rtpllm/
# sglang plugin twins); no DeepSeek-V4 path does. They belong
# in ``kv_cache_data`` -- the linear-attention forward reads them from there
# -- but are addressed by request slot, so no block-addressed mover may
# touch them. The dense codec skips them; the state tier reaches the same
# bytes through ``state_entry_views``.
per_request_state: bool = False


@dataclass
Expand Down
2 changes: 2 additions & 0 deletions atom/kv_transfer/disaggregation/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@
LoadOperationId,
ReqMeta,
SaveOperationId,
StateStoreOperationId,
)

__all__ = [
Expand All @@ -34,4 +35,5 @@
"LoadOperationId",
"ReqMeta",
"SaveOperationId",
"StateStoreOperationId",
]
405 changes: 388 additions & 17 deletions atom/kv_transfer/disaggregation/multi/multi_connector.py

Large diffs are not rendered by default.

56 changes: 50 additions & 6 deletions atom/kv_transfer/disaggregation/pp_kv_aggregator.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,19 +10,28 @@

from __future__ import annotations

from atom.kv_transfer.disaggregation.types import KVConnectorOutput, ReqId
from atom.kv_transfer.disaggregation.types import (
ConnectorCompletion,
ConnectorCompletionKey,
KVConnectorOutput,
ReqId,
)


class PPKVAggregator:
"""Aggregate offload ``finished_loading / failed_loading / finished_saving``
across PP stages.
and ``connector_completions`` across PP stages.

Call :meth:`ingest` once per (pp_rank, output) pair. The method returns a
:class:`KVConnectorOutput` containing only the request IDs that have
reached a terminal state across all stages.
:class:`KVConnectorOutput` containing only the request IDs (and connector
completion keys) that have reached a terminal state across all stages.

A load fails the request if any stage reports a failure, but the failure is
only emitted once every stage has reported either success or failure.
only emitted once every stage has reported either success or failure. A
connector completion is failure-dominant the same way: each stage holds a
slice of the layers a checkpoint/state boundary spans, so the boundary
commits (``succeeded=True``) only if every stage succeeded, and any stage's
failure sinks it -- but, like a load, only once every stage has reported.

Only offload-specific fields are tracked. Mooncake P/D fields
(``finished_sending``, ``finished_recving``) have their own PP-aware
Expand All @@ -36,6 +45,13 @@ def __init__(self, pp_size: int) -> None:
self._loading: dict[ReqId, set[int]] = {}
self._saving: dict[ReqId, set[int]] = {}
self._failed_loading: dict[ReqId, set[int]] = {}
# Per connector-completion key: which stages have reported it, and
# whether any stage reported it failed. The head's own TP aggregation
# already collapsed each stage's TP ranks to one succeeded/failed
# verdict per key (aggregator.py), so a stage reports a key at most once
# per generation and PP quorum is just stage coverage.
self._connector: dict[ConnectorCompletionKey, set[int]] = {}
self._connector_failed: set[ConnectorCompletionKey] = set()

def ingest(self, pp_rank: int, output: KVConnectorOutput) -> KVConnectorOutput:
for rid in output.finished_loading:
Expand All @@ -44,6 +60,10 @@ def ingest(self, pp_rank: int, output: KVConnectorOutput) -> KVConnectorOutput:
self._failed_loading.setdefault(rid, set()).add(pp_rank)
for rid in output.finished_saving:
self._saving.setdefault(rid, set()).add(pp_rank)
for completion in output.connector_completions:
self._connector.setdefault(completion.key, set()).add(pp_rank)
if not completion.succeeded:
self._connector_failed.add(completion.key)

# A load is only terminal once every stage has reported one way or the
# other. Reporting the failure at the first failing stage would wake
Expand All @@ -61,17 +81,37 @@ def ingest(self, pp_rank: int, output: KVConnectorOutput) -> KVConnectorOutput:
done_saving = {
rid for rid, stages in self._saving.items() if len(stages) >= self._pp_size
}
# A connector completion is terminal once every stage has reported it
# (same all-stage quorum as a save); its verdict is the AND across
# stages -- failed if any stage failed, else succeeded.
done_connector = {
key
for key, stages in self._connector.items()
if len(stages) >= self._pp_size
}
connector_completions = {
ConnectorCompletion(
channel=key[0],
operation_id=key[1],
succeeded=key not in self._connector_failed,
)
for key in done_connector
}

for rid in done_loading | failed:
self._loading.pop(rid, None)
self._failed_loading.pop(rid, None)
for rid in done_saving:
self._saving.pop(rid, None)
for key in done_connector:
self._connector.pop(key, None)
self._connector_failed.discard(key)

return KVConnectorOutput(
finished_loading=done_loading,
failed_loading=failed,
finished_saving=done_saving,
connector_completions=connector_completions,
)

def has_pending(self) -> bool:
Expand All @@ -80,9 +120,13 @@ def has_pending(self) -> bool:
The head's busy loop keeps polling downstream stages while this holds;
the tallies only drain when the missing stages report in.
"""
return bool(self._loading or self._saving or self._failed_loading)
return bool(
self._loading or self._saving or self._failed_loading or self._connector
)

def reset(self) -> None:
self._loading.clear()
self._saving.clear()
self._failed_loading.clear()
self._connector.clear()
self._connector_failed.clear()
95 changes: 81 additions & 14 deletions atom/kv_transfer/disaggregation/types.py
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,31 @@ def __post_init__(self) -> None:

LoadCompletionId = ReqId | LoadOperationId

ConnectorCompletionId = ReqId | SaveOperationId | LoadOperationId

@dataclass(frozen=True)
class StateStoreOperationId:
"""Exact identity of one hand-out of a state checkpoint to the CPU tier.

Not keyed by request: by the time a state store lands, the request that
produced the checkpoint is long gone and only the prefix hash remains. But
the hash alone is not an identity -- the same prefix is stored again after
an eviction or a load miss, and `KVOutputAggregator` tombstones every
`(channel, operation_id)` it has taken quorum on, so a second store under a
bare hash is dropped as a duplicate: its pin is never settled and its
bytes are never re-indexed. `generation` is what separates the attempts.
"""

prefix_hash: int
generation: int

def __post_init__(self) -> None:
if self.generation < 0:
raise ValueError("state store generation must be nonnegative")


ConnectorCompletionId = (
ReqId | SaveOperationId | LoadOperationId | StateStoreOperationId
)
ConnectorCompletionKey = tuple[str, ConnectorCompletionId]

# ---------------------------------------------------------------------------
Expand Down Expand Up @@ -152,6 +176,15 @@ class KVTransferTensors:
scatter_slot: Callable[[int, int], None] | None = None
# Appended for positional compatibility with existing generic descriptors.
expected_full_slot_region_count: int | None = None
# The attention metadata builder, published by `ModelRunner` after the tier
# is built inside `register_kv_caches` -- the one place builder and connector
# are both in scope. The kimi_k3 state tier reads its `state_runtime.
# checkpoint_spec.layout_id` to fold the state geometry into every key. Typed
# `object` because `types` must not import the model engine; None on every
# layout that has no state tier to name. Declared here rather than set as a
# loose runtime attribute so the field the connector reads is part of the
# contract, not an undocumented assignment two layers away.
state_backend: object | None = None


@dataclass
Expand Down Expand Up @@ -271,6 +304,22 @@ class ConnectorMetadata:
and the worker-side connector consumes it in ``start_load_kv``.
"""

#: Attributes whose truthiness means "the worker has something to do this
#: step". A subclass **extends** this rather than replacing it, and owns
#: only its own fields -- which is the point: the engine drops metadata
#: that reports no work, so a field added to a subclass and not listed
#: here leaves every request parked against it waiting for a report nobody
#: was asked to produce. Keeping the list next to the fields it names is
#: what makes that omission local instead of a shared table three
#: connector families have to remember to edit.
WORK_FIELDS: tuple[str, ...] = (
"reqs_to_recv",
"reqs_to_save",
"reqs_to_send",
"reqs_in_batch",
"reqs_not_processed",
)

def __init__(self) -> None:
self.reqs_to_recv: dict[ReqId, ReqMeta] = {}
self.reqs_to_save: dict[ReqId, ReqMeta] = {}
Expand All @@ -279,6 +328,10 @@ def __init__(self) -> None:
self.reqs_not_processed: set[ReqId] = set()
self.request_id_to_transfer_id: dict[ReqId, int] = {}

def has_work(self) -> bool:
"""Whether the worker has anything to do with this snapshot."""
return any(bool(getattr(self, name, None)) for name in self.WORK_FIELDS)

@staticmethod
def _build_req_meta(
req_id: ReqId,
Expand Down Expand Up @@ -343,19 +396,33 @@ def completion_req_key(completion: ConnectorCompletionId) -> str:
return str(getattr(completion, "req_id", completion))


#: Fallback for objects that are not `ConnectorMetadata` -- test doubles and
#: duck-typed sub-metas. Real metadata answers through `has_work`; this list is
#: deliberately not the place to register a new field.
_DUCK_TYPED_WORK_FIELDS = (
"requests",
"state_loads",
"state_stores",
"lookup_requests_in_step",
"reqs_to_recv",
"reqs_to_save",
"reqs_to_send",
"reqs_in_batch",
"reqs_not_processed",
)


def connector_metadata_has_work(metadata: object | None) -> bool:
"""Return whether connector metadata contains dispatchable work."""
"""Return whether connector metadata contains dispatchable work.

Asks the metadata rather than inspecting it, so that each connector family
declares its own work fields next to where it defines them. The engine
drops a snapshot that reports nothing, so an unreported field is a
permanently parked request, not a wasted step.
"""
if metadata is None:
return False
return any(
bool(getattr(metadata, name, None))
for name in (
"requests",
"lookup_requests_in_step",
"reqs_to_recv",
"reqs_to_save",
"reqs_to_send",
"reqs_in_batch",
"reqs_not_processed",
)
)
probe = getattr(metadata, "has_work", None)
if callable(probe):
return bool(probe())
return any(bool(getattr(metadata, name, None)) for name in _DUCK_TYPED_WORK_FIELDS)
27 changes: 23 additions & 4 deletions atom/kv_transfer/offload/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,9 +13,19 @@ A DSV4 boundary is reusable only when both PAGE and SLOT restore successfully.
Missing, incompatible, or corrupt sidecar data fails closed to recomputation.

The public configuration remains `kv_connector: "lmcache_offload"`. The thin
top-level shell selects `hybrid` when `hf_config.compress_ratios` is present and
`dense` otherwise; `kv_transfer_config.offload_layout` can override that choice
without giving scheduler and worker different connector names.
top-level shell resolves one of three layouts: `kimi_k3` when
`hf_config.model_type == "kimi_linear"` (dense paged MLA KV plus a KDA
per-request state tier), `hybrid` when `hf_config.compress_ratios` is present
(DSV4 PAGE+SLOT), and `dense` otherwise. `kv_transfer_config.offload_layout` can
override that choice without giving scheduler and worker different connector
names.

GDN/linear-attention models (`qwen3_next`, `qwen3_5_*`; e.g. Qwen3-Next,
Qwen3.5) are the one family the resolver does **not** map to a layout: they carry
a per-request recurrent state that no offload layout owns a tier for, so
restoring their KV prefix while that state stays stale is silent wrong output.
`select_offload_layout` refuses them at startup with a `ValueError` that names
the cause, rather than falling through to `dense`.

It is the **ATOM-native, in-engine** offload path: the connector plugs straight
into ATOM's scheduler/worker via the shared
Expand Down Expand Up @@ -67,7 +77,7 @@ Four rules carry the module:
| File | Role |
|------|------|
| `__init__.py` | Registers the `lmcache_offload` backend with `KVConnectorFactory`. |
| `connector.py` | Public config-only `dense`/`hybrid` selector and thin worker/scheduler delegation shells. |
| `connector.py` | Public config-only `dense`/`hybrid`/`kimi_k3` selector and thin worker/scheduler delegation shells. |
| `_offload_common.py` | Shared LMCache engine construction, role validation, executors, and completion plumbing. |
| `_block_gpu_connector.py` | Family-neutral raw-block LMCache `GPUConnectorInterface`; DCP-aware bounded staging and two-stage copies. |
| `config.py` | Builds the per-rank `LMCacheEngineConfig` + `LMCacheMetadata` from `LMCACHE_*` env and `kv_transfer_config` extras. |
Expand All @@ -79,8 +89,17 @@ Four rules carry the module:
| `hybrid/dsv4/policy.py` | DSV4 geometry/profile, PAGE/SLOT cadence, prefix hashes, fingerprint, committed-checkpoint policy, and bounded staging-row admission. |
| `hybrid/dsv4/codec.py` | `DSV4PageSlotCodec`, `DSV4CheckpointCodec`, and `DSV4CheckpointStore`: unified GPU layout plans plus AOS1 framing/storage. |
| `hybrid/dsv4/triton_page_slot.py` | Raw-`uint8` PAGE/SLOT gather/scatter kernels; PAGE is forward-indexed and SLOT is reverse-indexed. |
| `hybrid/kimi_k3/connector.py` | Kimi-K3 worker and scheduler: the dense paged-KV path plus one extra leg for the KDA per-request state tier. |
| `hybrid/kimi_k3/staging.py` | Single-entry bounded GPU staging buffer, D2H/H2D copy stream, and producer event for one flat state entry per transfer. |
| `hybrid/kimi_k3/state_object.py` | One state checkpoint as a single opaque object keyed by ATOM's own hash, bypassing LMCache's `ChunkedTokenDatabase` (state bytes are not token-sliceable). |
| `hybrid/kimi_k3/state_tier.py` | Worker-side store/load driver for the state tier on its own executor; reports store/finished/failed hash sets for the engine-side `StateOffloadIndex` to apply. |
| `atom_lmcache_staging.py` | Per-thread CUDA streams, staging buffer, ready/free events, env helpers. |

The engine-side counterpart of the state tier lives outside this directory:
`atom/model_engine/state_offload.py` holds `StateOffloadIndex`, which applies the
store/finished/failed hash sets that `hybrid/kimi_k3/state_tier.py` reports back,
so a later prefix hit knows which checkpoints the CPU tier can actually serve.

## Architecture

The connector is split across two processes, mirroring ATOM's P/D split:
Expand Down
26 changes: 2 additions & 24 deletions atom/kv_transfer/offload/_block_gpu_connector.py
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@
_PipelineStage,
_StagingBuffer,
_ThreadTransferState,
memory_object_as_uint8,
run_staged_pipeline,
)

Expand Down Expand Up @@ -270,29 +271,6 @@ def _assert_fused_chunk_major_available(self) -> None:
"loads successfully"
)

def _memory_tensor(self, memory_obj: Any, nbytes: int) -> torch.Tensor:
tensor = getattr(memory_obj, "tensor", None)
if tensor is None and hasattr(memory_obj, "get_tensor"):
tensor = memory_obj.get_tensor(0)
if tensor is None:
raise RuntimeError("ATOM LMCache connector: invalid MemoryObj tensor")
if tensor.dtype != torch.uint8:
raise TypeError(
"ATOM LMCache connector: MemoryObj tensor must be uint8, "
f"got {tensor.dtype}"
)
if not tensor.is_contiguous():
raise RuntimeError(
"ATOM LMCache connector: MemoryObj tensor not contiguous"
)
flat = tensor.reshape(-1)
if int(flat.numel()) < int(nbytes):
raise ValueError(
"ATOM LMCache connector: MemoryObj tensor is too small "
f"for {nbytes} bytes; got {int(flat.numel())}"
)
return flat[: int(nbytes)]

def _range_block_ids(
self,
all_block_ids: list[int],
Expand Down Expand Up @@ -360,7 +338,7 @@ def _iter_transfer_chunks(
_TransferChunk(
memory_obj=memory_obj,
block_ids=block_ids,
tensor=self._memory_tensor(memory_obj, nbytes),
tensor=memory_object_as_uint8(memory_obj, nbytes),
nbytes=nbytes,
)
)
Expand Down
Loading
Loading