Repository navigation
feat(plugin): DeepSeek-V4.1-Flash on the vLLM plugin, with LMCache byte offload - #2357
Closed
PerryZhang01 wants to merge 232 commits into
Closed
PerryZhang01 wants to merge 232 commits into
PerryZhang01 wants to merge 232 commits into
Conversation
Contributor
🏷️ CI GuideRuns automatically on every eligible PR before approval:
Heavy model tests:
|
3 of 32 tasks
The shared scheduler writes seq._deferred_save_at when a finished request is parked and restarts it for every later chunk. Sequence absorbs that write; SeqView did not, so the first deferred save raised, and a clock that is not restarted frees the source blocks while the copy is still running. Co-authored-by: Cursor <cursoragent@cursor.com>
split_kv_caches_by_group only stripped the MiniMax ".index_cache" suffix, so GLM's "<p>.indexer.k_cache" was unmapped once a model had more than one KV group. The entry now follows index_cache_owner, the same fold the single-group path already uses. Pin the checkpoint's real indexer layers (full on 0,1,2 then every fourth layer), the 132-byte packed scale round trip, and a 16384/32768 prefill that must store the second budget. Co-authored-by: Cursor <cursoragent@cursor.com>
LMCache 0.5.5rc3 unpins every object inside retrieve, and lookup_unpin unpins those same keys again. The second decrement drives the pin count negative and returns the buffer to the allocator. Hold the pin across retrieve so lookup_unpin is the single release, after the copy has synchronized. Co-authored-by: Cursor <cursoragent@cursor.com>
The sparse-MLA path slices every input to `num_actual_tokens` and sizes its q_out from that, so the whole attention compute runs at the live width M. The buffer it returns, however, is the full CUDA-graph width W: rows [M:W] are never written and still hold whatever the previous tenant of that allocation left there. Inside a captured graph the ops downstream cannot tell a padded row from a live one. This only becomes reachable with prefix caching, which is what puts M at a value that is not on the capture ladder: a request whose prefix is a cache hit recomputes only its tail, and any tail that is not itself a ladder size is padded up to one. Tails that land exactly on a ladder size (8/16/24/32/40) have no padding and are unaffected. Measured on GLM-5.3-MXFP4 TP4, plugin backend, PIECEWISE cudagraphs, forcing a specific recompute width per request and checking that a marker planted in the reused prefix is recalled: - undefined tail: 16 of 48 forced widths return wrong text - defined tail: 48 of 48 correct, and 10 of 10 on a second arm - cudagraph_mode=NONE (no padding at all): 0 of 10 wrong, either way The consumer that turns a stale padded row into a wrong live row has not been identified yet. The fix does not depend on that identification: an op that returns a CUDA-graph-padded buffer owns all of it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Black reformats two of the new test files, and Ruff flags an unsorted import block, two `noqa: E402` directives for a rule that is not enabled, and an `if` that is a `max()`. All four are in files this branch adds. No behaviour change: the four affected test files still pass (27 passed, 1 skipped). One Ruff finding is deliberately left for a follow-up commit -- `SeqView.__slots__` is not sorted (RUF023). Sorting it edits a file under `atom/`, and an accuracy measurement is currently running against this tree with a fingerprint gate over exactly that directory. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
forward_impl and forward_impl_sparse have the same shape: both slice the output buffer to num_actual_tokens, write only those rows, and return the full CUDA-graph-width allocation. The sparse path was fixed; the dense one still handed back rows [M:W] holding whatever last used that allocation, and o_proj projects all W rows, so the undefined tail enters the residual stream and reaches the MoE on every layer that has one. GLM-5.3 runs 57 dense MLA layers against 21 sparse ones, so most of this class of leak was never closed. This does NOT fix the accuracy collapse seen when the CPU tier serves a prompt: gsm8k n=140 with the tier supplying ~99% of prompt tokens scores 0.4929 without this commit and 0.4857 with it, against 0.9857 on the same questions without reuse. The carrier of that collapse is still open. This commit stands on its own -- reading undefined memory is a defect whether or not it is the one being chased -- but it should not be read as the fix for it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The staging pipeline runs both of its legs on private side streams. The handshake around them orders stage_a against stage_b and nothing else, so neither leg was ordered against the model's forward -- which is the producer of the KV a save reads and the consumer of the KV a load overwrites. A save's gather was therefore free to run before the attention write it was meant to capture had retired, and it published whatever the block held from its previous occupant under this prompt's hash. Pass 1 cannot see this: nothing reads the stored bytes back, so the store pass scores clean. Pass 2 restores the corrupted image and the model decodes against a prompt it never saw. Measured on GLM-5.3-MXFP4 TP4, gsm8k 20-shot n=140, HBM pool 4096 blocks so the load pass can only come back from the CPU tier: tier supply accuracy gen tokens wall 99.09% 0.4857 970139 6240s before 98.99% 0.9714 40593 1531s after The store pass is 0.9929 either way, which is what made this look like a restore bug for as long as it did. An HBM-only control at the same reuse fraction (99.11% local prefix hits, connector off) scores 0.9929, so prefix reuse itself was never the carrier -- only the tier path was. The K3 recurrent path already fences this way (see `KdaTransferPool.submit_store`, whose docstring spells out the same hazard); the dense block connector never got the edge. `wait_stream` keeps it device-side, so no host thread blocks and offload stays off the critical path. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The fence added in 9354da2 read `state.device` unconditionally. Every unit test that drives `run_staged_pipeline` does so with stream doubles and a state object that has no device -- 28 of them failed with `AttributeError: ... has no attribute 'device'`. The doubles also have no `wait_stream`, so giving them a device would only have moved the error. Resolve the compute stream through `_compute_stream_for`, which returns None when there is no CUDA context to take an edge against. On a real worker `_ThreadTransferState` always carries the device its streams were created on, so production behaviour is unchanged. A fence that is allowed to be absent is a fence that can be deleted quietly, so this also adds the two tests that supply the device and the compute stream the other pipeline tests do not, and assert the edge is taken -- once up front, and exactly once in single-stream mode. Verified as a negative control: reverting the fence fails both and nothing else. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…multiple On a hybrid model vLLM does not use the operator's `--block-size`: it raises the attention block until one attention page holds one recurrent-state page. Kimi-K3 goes 128 -> 1536 (`128 * cdiv(814080, 128 * 576)`), so an operator who passes `LMCACHE_CHUNK_SIZE=128` alongside `--block-size 128` gets an abort naming only the two values that do not work, with no way to derive the one that does short of reading vLLM's platform code. Name the required multiple in the message and say where the block size came from. Behaviour is unchanged -- a chunk size that is not a whole number of blocks still aborts the boot, which is correct: rounding it silently would move every LMCache key. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…uns on
`run_staged_pipeline` fences its staging streams against the stream named by
`_compute_stream_for(state)`, which answered `torch.cuda.current_stream(device)`.
That is wrong in a way that leaves no trace:
* vLLM does not run the forward on the default stream. `vllm.utils.
current_stream` installs a dedicated per-process stream on first use.
* `torch.cuda.current_stream()` is thread-local, and `_compute_stream_for`
runs on a save/load worker thread that never called `set_stream`, so it
read that thread's default stream instead.
* PyTorch creates streams with `cudaStreamNonBlocking`, so the legacy default
stream's implicit synchronization does not paper over the difference.
`stage_a.stream.wait_stream(compute_stream)` therefore ordered the staging
streams against a stream nothing is ever enqueued on: not a weaker fence, no
fence. The hazard it exists to stop -- a pack reading a block whose attention
write is still queued -- was never stopped. At conc16 with restores enabled
this surfaced as a HIP illegal memory access after ~88 restores.
The stream is now recorded on the forward thread, at the top of `start_load_kv`
rather than in `_track_job`: Python evaluates `submit()` before the
`_track_job()` call wrapping it, so a worker could otherwise reach the staging
pipeline first and cache `compute_stream=None` for the life of the process.
`_thread_state` refreshes the slot on every call so an early None cannot become
permanent. Dense and dsv4 are separate class hierarchies and each records it;
KimiK3 inherits the dense path.
An unrecorded stream must not silently degrade to the old behaviour, because
"degrades to the default stream" IS the defect, so `_compute_stream_for` now
returns None and logs once when what it holds is missing or is the default
stream.
Measured on GLM-5.3-MXFP4 TP4, conc16, restores on: unpatched arms fault at
64/76/92/120 restores (four independent runs). Patched, the run reached 4256
restores before hitting a second, distinct fault -- P(surviving the first at
its measured rate) = e^-48. That second fault is deterministic (two unrelated
interventions agree to 4 s of wall clock) and is not addressed here.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ocal one The two fence tests stubbed `torch.cuda.current_stream` to hand `_compute_stream_for` the stream they wanted it to fence against. That is the lookup the fix removes: it is thread-local, and the staging pipeline runs on a transfer worker where it answers the default stream, so the fence ordered nothing. A test that supplies the stream through it can only ever confirm the old, broken contract -- and it would keep passing if the recording on the forward thread were deleted again. They now put the stream on the state, where `note_compute_stream` puts it, and stub `default_stream` to something distinct: `_compute_stream_for` treats "the default stream" as a missing recording, precisely because that is the value the old lookup produced, so a test whose fake compute stream *is* the fake default would assert the absence of the fence it exists to check. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… site The docstring said `_track_job` calls it. That was true of an earlier draft and is not true of the code: the call moved to `start_load_kv`, because Python evaluates the `submit()` argument before the `_track_job()` wrapping it, so a job can already be queued on a transfer worker by the time `_track_job` runs -- too late for a recording whose whole point is to happen first. Doc-only; the requirement the docstring exists to state (must run on the forward thread, since `torch.cuda.current_stream` is thread-local) is unchanged. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…itialized `init_aiter_topK_meta_data` allocates one process-wide `(max_num_tokens, top_k + n_shared + is_EP)` buffer that every MoE layer and every step reuses, then initializes only the shared-expert columns. The `top_k` routed columns are never written here, so for any row the routing kernel does not write in a given step they hold whatever the caching allocator last left there. Those entries are expert *ids*. `moe_sorting` and the scatter/reduce kernel index expert-sized storage with them, so a stale value is an out-of-range access rather than a wrong number -- it surfaces as HSA_STATUS_ERROR_MEMORY_APERTURE_VIOLATION, attributed to whatever kernel happens to be running when the fault retires. Rows past the current step's token count are exactly the rows at risk: under FULL cudagraph capture the padded region is traversed, because the graph was captured at a fixed shape. Zero is safe to leave there: expert 0 with weight 0 (the weights buffer is zeroed for the same reason) contributes nothing to the reduction. vLLM carries a byte-identical copy of this function in `vllm/model_executor/layers/fused_moe/experts/rocm_aiter_moe.py`; it has the same two `torch.empty` calls and the same partial initialization. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`run_staged_pipeline` fenced the compute stream -> staging direction once, at pipeline entry, and never fenced staging -> compute at all. Two gaps followed from that. Per-group, not per-pipeline: a save runs on a transfer worker thread while the *next* forward step is already enqueuing work. An entry-only fence covers only what the forward had queued at that instant, so a later group could pack a block whose attention write was enqueued afterwards. The fence is now taken at the top of each group. The reverse edge was missing entirely. `stage_b.stream.synchronize()` is a host-side wait on the transfer worker thread; it orders the forward only if the forward thread joins this transfer's future before it touches the blocks. Loads write KV the forward then reads, and saves read KV the forward may overwrite once the request's blocks are recycled. Both are now recorded as a device-side edge: an event on each staging stream, waited on the compute stream, enqueued *before* the synchronize (after it the event is already complete and the wait is a no-op). Scope note: this is necessary, not sufficient. On its own it does not stop the aperture violation this PR chases -- an arm carrying only this change still faulted -- and an arm carrying the topK meta-buffer zeroing but not this fence faulted at 324 KV restores. Both changes are load-bearing; this one is the memory-ordering half. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
One arm, connector ON, 1800 s, the aiperf shape published by inference-benchmarking/aiperf-cache-validation. Also drops the conc-16 row, whose ON half was never measured, and refreshes the tree head. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
xgrammar 0.2.8 passes extra_lib_paths to load_lib_module, which needs apache-tvm-ffi>=0.1.11. The OOT image installs xgrammar with --no-deps and then pins apache-tvm-ffi==0.1.10 from vLLM's rocm requirements, so the floating >=0.2.1 range crashes every accuracy job at import. Co-authored-by: Cursor <cursoragent@cursor.com>
The only cell the sweep never produced. 1800 s, seed 530419, the same client line as the other rungs, zero illegal-memory faults. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The old section reported a single conc-16 pair whose two rulers straddled zero and whose validity control failed, so it quoted no steady-state row. It is replaced by the four-rung ON/OFF sweep (conc 16/20/24/32) and the 1319-question gsm8k points taken on this branch. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… read it Both LMCache byte-offload recipes carried test material beyond what a reader needs to reproduce the published numbers. GLM-5.3: drop the agentic multi-turn (cache-validation) section. It was a single ON-only arm with no OFF counterpart, and its CPU tier share was 0.00%, so it reported nothing about the connector -- the reuse it generates stays inside HBM. Kimi-K3: the Client section documented an agentic trace replay that no arm in Measured used, with the client that was actually used demoted to "the second client". Promote the controlled-prefix synthetic pool to be the client, and drop the trace-only flag notes (--max-context-length, --warmup-requests-per-lane) that do not apply to it. Both: add "How this run measures LMCache" under Accuracy, stating the three conditions that make the accuracy run a tier measurement rather than a plain accuracy check -- two passes over one corpus, an HBM pool too small to hold it, and server-side confirmation that pass 2 actually read the tier. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The Server block carried no --port, so vLLM took its default and listened on
8000, while the Client, lm_eval and metrics blocks of the same recipe all
target 8330. Run verbatim, the client connects to nothing.
The failure does not look like a port problem. aiperf does not exit when it
cannot connect -- it retries at full rate and appends one ClientConnectorError
record per attempt to profile_export.jsonl, which reached 5.0M and 8.1M lines
(5.1 GB / 8.2 GB) in two runs and drove the host out of memory, at which point
the OOM killer took a vLLM worker and vLLM reported
Worker proc VllmWorker-2 died unexpectedly (exit code: -9)
RuntimeError: Executor failed. -> EngineDeadError
with nothing that names memory or the port. A healthy conc-16 arm writes 2208
records / 19 MB over the same 1802 s.
Kimi-K3's recipe already binds --host 127.0.0.1 --port 8713; this brings
GLM-5.3 in line with it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ve mode Both recipes so far only cover the in-process tier pinned inside each TP worker by AtomLMCacheOffloadConnector. LMCache can also hold the tier in a separate process reached over ZMQ (LMCacheMPConnector). Record how to run it: the `lmcache server` launch, the connector swap, and the environment that stops applying. The verification steps are not the same and silently so: in MP mode the worker's Stored/Retrieved counters are structurally zero because the worker never touches the tier, so the existing checks would read as "nothing is happening" on a perfectly healthy run. Point at the tier's own /status instead, and note that memory_total_bytes is a lazily grown pool rather than the capacity -- memory_configured_bytes is the ceiling. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The recipe update restored the floating >=0.2.1 range. xgrammar 0.2.8 passes extra_lib_paths to load_lib_module, which needs apache-tvm-ffi>=0.1.11, while this image still installs 0.1.10. Co-authored-by: Cursor <cursoragent@cursor.com>
The multiprocess section launched `lmcache server` on its defaults, which leave one worker thread for every STORE/RETRIEVE across all eight ranks and grow the pinned pool during the run. Pin the four flags that have to be sized -- worker pools to TP, `--l1-init-size-gb` to the tier's plateau, and `--l1-align-bytes` to the DMA granularity -- and turn on the two `lmcache.mp.*` booleans that move lookup and store submission off the critical path. Also record the conc-16 OFF / in-process / multiprocess triplet those flags were measured with, and fix the tier-capacity paragraph, which still quoted `--l1-size-gb 192` against a 720 GiB launch line. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The plugin connector could only drive per-rank in-process LMCache engines. This lets it address ATOM's multiprocess tier instead: the paged MLA KV of every rank lives in one `lmcache server` pool, reached over IPC by block id, while Kimi-K3's KDA recurrent state stays on per-rank in-process engines because MP addresses by paged block and state is addressed by request slot. - `mp_page_layout.py` publishes each rank's PAGE layout to the tier, and `mp_worker_adapter.py` drives the MP worker from the connector, turning its `failed_loading` into the block ids vLLM's recompute policy needs. - `hybrid_invalid_blocks_patch.py` makes that recompute policy work on a model with more than one KV cache group; vLLM's own implementation unpacks a single group and raises inside `update_from_output`, killing the engine. - `_canonical_head_dim` derives the attention head dim from checkpoint fields so the scheduler and the workers hash the same PAGE namespace. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A hybrid model's recurrent layers were the one thing the multiprocess offload could not move. The plugin worked around it with a second, per-rank in-process LMCache engine, which put the KV and the state that continues it in two tiers that evict independently: a reader could find a prefix whose attention pages were present and whose recurrent state was gone, and nothing in either tier could notice. LMCache's MP protocol already describes this case -- `EngineGroupInfo` has carried `recurrent_state` and `sw_size_tokens` all along, and the server drops an all-null chunk rather than copying it. Use it: publish the recurrent planes as `KVTransferTensors.recurrent_page_groups`, register ordinal `j` as engine group `1 + j` one block wide, and emit `[-1] * (n - 1) + [boundary]` beside the attention block ids in the same `AtomMPTransferSpec`. KV and state then commit and restore as one object, which is the property the second tier could not have. This is the numbering `native_state_layout` already uses for PAGE-backed checkpoint images, and it needs the same server flags: `--null-block-id -1 --separate-object-groups`. Transport only. Nothing publishes `recurrent_page_groups` yet, so every existing layout registers exactly the groups it did before. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Under `atom.offload.backend=mp` the recurrent state had a transport of its own: a per-rank in-process LMCache engine with a second host pool, sized by `lmcache.mp.state_cpu_size_gb`. Two tiers that evict independently, so a reader can find a prefix whose attention pages are present and whose recurrent state is gone -- and neither tier can notice. This publishes each mamba group to the MP server as a recurrent engine group of its own, so the state rides the same transfer object as the KV it continues and the two commit and restore together. - `build_mp_recurrent_groups` turns `KdaPageViews` into `RecurrentPageGroup`s, sized off each group's own block axis (a recurrent group counts in a different block id space than the attention block table). - `KdaPageViews.groups` is now public: the MP layout must register the tensors in exactly the order the codec gathers them. - `lmcache.mp.state_transport` selects `mp` (default) or `own-pool`; the latter is the earlier arrangement, kept because the numbers to date were taken on it, and it is what still requires the second pool size. - Under `mp` the second engine, pool and leg are not built at all. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… that cannot work Reverts the previous commit's attempt to normalize vLLM's resolved config. It booted in a probe and died in a worker, which is the difference between `AutoConfig.from_pretrained` and what vLLM's own `ModelConfig` produces: measured through the latter, the shim carries `text_config=None` and no `vision_config`, and `normalize_hf_config` is built from exactly those two. No amount of fixing the shim's serialization reaches that -- vLLM rebuilds the config for the fields it reads and does not keep the nested blocks. So the finding stands and the fix does not. ATOM goes back to reading the checkpoint, and `--hf-overrides` is now refused at startup by name rather than silently ignored: an operator who passes one is told, instead of being served a model that answers as though they had not. The serializable text-config class is kept. It is correct on its own terms (`to_dict` takes `model_type` from the class, so a bare PretrainedConfig serializes an empty one) and costs nothing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…vllm-plugin-nolmcache
A sibling session could not start DSpark on this path at all. The cause is that vLLM drives the draft with its own V4.1 attention, which accepts only a packed `*_ds_mla` layout: given `auto` it resolves to `fp8_ds_mla` and writes that back onto the `cache_config` it was handed. Without a draft-level override that is the *target's* config, so ATOM's pool dtype silently became a spelling it has no pool for, and the run died naming an `fp8_ds_mla` the operator never typed. `--speculative-config`'s `kv_cache_dtype` is applied to the draft's own copy (`_DRAFT_VLLM_CONFIG_OVERRIDES`), which is the separation this needs -- the draft gets its packed layout and the target pool stays bf16 or fp4. The CI cell has always passed it, which is why CI stayed green while the obvious invocation failed. Refused at startup now, with the flag, the value and the fact that the target's own --kv-cache-dtype is separate. Also comments `validate_speculative_config`'s BF16 requirement, which the same session read as contradicting vLLM's fp8 draft requirement. It does not -- they are two different caches -- but nothing said so. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ure DSpark Resolving the connector's group by eliminating every other one needs a new clause for each group anybody adds. DSpark's draft KV is its own group, so the engine refused to start with `expected exactly one non-recurrent KV cache group, found [0, 1]`. The codec registers the proxy's planes, so the proxy's group is the one it owns: it is claimed by layer name instead, and a V4.1 run with no proxy group says so rather than picking something. `_is_deepseek_v41` is also assigned before that resolution rather than after. It is a class attribute defaulting to False, so the branch that reads it was dead and nothing said so -- found by the branch having no effect, not by an error. With those, DSpark alongside this connector runs for the first time, and the refusal's prediction holds. Measured: the engine boots, the three-arm accuracy harness scores blind 0/16, cold 16/16, warm 16/16 with zero stale-cursor refusals and `cursor_mismatch` 0 -- and the tier stores 32 objects and retrieves none. External prefix cache hit rate 0.0%, against 44.1% for the same configuration without drafts, with the lookups happening and missing: a draft moves the frontier a store is keyed on, so nothing later asks for the key that was written. So the combination is safe and useless rather than unmeasured, and the refusal now says that instead of listing what it feared. Fixing it means making the boundary keys speculation-aware. The draft also needs its own `kv_cache_dtype` inside `--speculative-config` (#2494 now refuses the configuration without it). Without that key the draft runs vLLM's V4.1 attention against the TARGET's `cache_config` and writes the resolved `fp8_ds_mla` back into that same object, so the target's dtype changes under ATOM's gate between startup and the check. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…-nolmcache' into perzhang/dsv41-lmcache-031 # Conflicts: # atom/model_ops/attentions/deepseek_v41/cache.py # atom/plugin/vllm/deepseek_v41_bridge.py # tests/plugin/test_vllm_deepseek_v41_state_reserve.py
Two files went out with raw `<<<<<<< HEAD` in them and four workers failed to start on `IndentationError`. I read the merge's output through `tail -8`, took the last line for the whole conflict list, resolved the one file it showed and ran `git add -A` over the rest. The unit suite stayed green throughout, because nothing in it imports either module. `tests/test_tree_is_importable.py` closes that: every module under atom/, tests/ and tools/ has to parse, and no file may carry a merge marker. The second check is separate from the first because a marker inside a docstring parses and is still an unfinished merge. The resolutions take the sibling branch's refactors rather than restoring mine on top of them: * `_report_stale_state` now takes `fresh` and filters on `~fresh` instead of `starts != 0`. `_restore_stale_rows` takes the same `fresh` and uses the same filter, so the restore and the contract that checks it cannot drift apart, and it hooks into their call site in the `histories` branch. * The row-alignment block is theirs wholesale: the V2 `InputBatch` has `req_ids` and no block table, and reading both under one `try` threw the good request ids away with the failed lookup -- on the runner V4.1 picks by default. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`_check_row_alignment` went out with their three-argument call against my four-argument definition. The call was inside the conflict region and the definition was not, so resolving the conflict changed one of them. Every unit test passed -- none reaches that path -- and four workers failed to start on `missing 1 required positional argument`. That is the third failure in this merge found by booting an engine rather than by a test, so `test_tree_is_importable.py` grows a third check: calls to the handful of bridge helpers both branches edit must match their definitions' arity, walked from the AST so it stays cheap enough that nobody skips it. The row-alignment guard is theirs wholesale, which restores raising where this branch had downgraded it to a one-shot log. The downgrade had a reason -- the guard was measured firing on batches that were NOT misordered -- so the tests here now assert their behaviour and say that whether it still fires spuriously on the offload path is a question for a live run. That run is going. `log_stats` also takes its interval from `ATOM_V41_STATS_INTERVAL`. At the fixed 60 s a workload shorter than one interval printed its last line BEFORE the load and none during it, which reads exactly like a load in which nothing happened -- the counter I need for the DSpark question was unreadable for that reason alone. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`cache_config.block_size` is engine-wide and becomes the SMALLEST group's once a second group exists. Every block table this scheduler indexes belongs to the attention group, so with DSpark the draft's 32 replaced the proxy's 256 and the two arithmetics that read it both went wrong the same way: * `covered = len(block_table) * block_size` priced a 1345-token prompt's six blocks at 192 tokens, so `_save_frontier` floored to zero chunks and no save was ever emitted; * `_save_source_is_backed` then needed 40 blocks to cover [0, 1280) and found six, so even a save that got past the first check was skipped. Neither raised anything. The tier was simply never written, and everything downstream -- lookups missing, `cap_hit` never firing, 0% supply -- was that one number. Measured as `computed=192` stuck for the life of every request, against 1280 for the same prompts without drafts. Fixed at the source: the connector tells the scheduler the block size of the group it actually strides, carrying the DCP factor across. Patching each site would leave the next reader of that value wrong in the same way and just as quietly. The call sits after the scheduler is built. Placed before it, the early return on `_scheduler is None` made it a no-op that logged nothing -- the second bug of that exact shape in this file, after `_is_deepseek_v41` being assigned after the resolution that reads it. A test pins the ordering. With this the spec arm goes from `save built=0` to 33, `unbacked` 5888 to 0, and the tier from holding nothing to holding 1280 tokens per prompt. It is not yet served: the hit is now declined inside the cap hook, which is the next thing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
They were one number, and they coincide while there is one kv cache group. `hash_block_size` is the unit vLLM's `request.block_hashes` are in and is engine-wide; the grid a STATE boundary may sit on is V4.1's PAGE block, because state is only restorable at a whole PAGE edge. A second group makes the first the smallest group's -- with DSpark, 32 against the proxy's 256 -- and one variable was then wrong on both sides at once. Measured: `cap_hit` accepted a boundary at 96 tokens for a hit of 1280. 96 is a legal 32-grid point, is not a PAGE edge, and floors to zero chunks at the caller, so the hit was thrown away and the arm supplied nothing. Positions now use the grid (the anchor, both caps, the store sweep, the interval check); `boundary_hash` keeps the hash unit, because it indexes a list vLLM built. The constructor refuses a grid that is not a multiple of the unit: a boundary on such a grid could not be hashed, and the two halves would disagree about which positions exist. Single-group runs are unchanged by construction -- the grid defaults to the unit -- and measured so: blind 0/16, cold 16/16, warm 16/16, 95.2% supplied, `boundary=1280`, byte for byte what it was. The speculative arm now reaches the right question instead of a wrong answer. Its boundary is 0 rather than 96: the store sweep stores when the frontier LANDS on the grid, and a draft step advances by one to six tokens and steps over it, where a plain decode advances by one and cannot. That is the hypothesis this branch falsified earlier -- it was masked by the grid being 32, where landing is easy. The refusal now says so. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…he hash unit Fourth site of the same conflation, and the one that was still costing the speculative arm everything. `apply_vllm_v41_prefill_alignment_patch` exists to land the frontier on a boundary; it clipped to `planner.hash_block_size`, which a second kv cache group makes the draft's 32 rather than the proxy's 256. The chunk then ended somewhere that is a legal 32-grid point and not a PAGE edge, so the prompt-end anchor -- the one boundary a later request with the same prompt asks about -- was never stored. With it, DSpark alongside this connector serves from the tier for the first time: `cap_hit` goes from `hit=1280 boundary=0` to `boundary=1280`, twelve lookups served, eleven loads completed, zero stale-cursor refusals. The single-group arm is unchanged, byte for byte: 95.2% supplied, VALID. Also corrects that patch's own reasoning about decode. It says every boundary above the prompt is reached anyway because decode walks one token at a time -- true without drafts, and exactly what drafts break. Not finished: the speculative arm now hangs instead. One load is attempted and never completes (`loads_attempted=12, loads_completed=11, loads_failed=0`), the engine ends on `Running: 0, Waiting: 1`, and the client times out. That is the next thing, and it is a different bug from the four this branch has fixed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`_check_promised_loads` covers the step before this one -- a promise that never becomes a dispatched load -- and settles its entry at dispatch. So nothing watched the gap between a load going out and its result coming back, and a load that is dispatched, performed and then never reported left the request parked for good: measured, one request re-examined 617k times while the client waited, and 12 state loads armed against 11 completed on three speculative runs out of three, where every run without drafts balanced exactly. Two earlier attempts at this guard could not fire, and the reason is worth keeping: both aged their counters inside `_join_kda`, and the stall is QUIESCENT. The request is parked, nothing runs, the worker produces no output and the join is never called, so a bound counted there advances only while the engine is busy -- which is exactly when it is not needed. The abandonment counter read zero through a run that hung. This one runs from `build_connector_meta`, which the scheduler calls every step whether or not anything is running. With it, DSpark alongside this connector completes a run for the first time: blind 0/16, cold 16/16, warm 16/16, VALID, 89.6% of the prompt supplied by the tier, zero stale-cursor refusals, nine loads attempted and nine completed, and the withdrawal firing exactly once. The refusal is kept and now says why: the withdrawal is a cost, not a cure. One dispatched load per run still goes missing, and why the report disappears under drafts and never without them is not known. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`load_finished` discards a completion whose `LoadOperationId` does not match the active one. That is correct -- it belongs to a lifecycle that has been cancelled or replaced -- but it is the only place a load that was performed can vanish, and it returned False in silence, which reads downstream exactly like a load that was never performed. Counted and warned now. It is NOT the cause of the dangling load this branch is chasing: measured zero drops across a speculative run that still had one load armed and never reported. That makes five hypotheses this branch has raised and killed on this one symptom, and narrows what is left to the worker's own `finished_loading` never containing it. Unit suite: 40 failures against the base branch's 35. The five are all in `tests/test_paged_attention_dispatch.py`, whose test file and module under test (`atom/model_ops/base_attention.py`) are both byte-identical with the sibling branch -- they arrived with its merge of main and fail there too. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
PerryZhang01
force-pushed
the
perzhang/dsv41-flash-vllm-plugin-nolmcache
branch
from
October 13, 2026 14:59
23ef886 to
2a89e29
Compare
…e transport Keep failed receive votes until PAGE and STATE have both terminated, fence unclaimed restore slots before release, and reject slot exhaustion without changing ownership. Remove poll-count withdrawal of dispatched transfers. Resolve target-group block geometry before freezing in-process or MP transport namespaces. Add lifecycle, destination, capacity-refusal and constructor regressions. Co-Authored-By: Claude Code <noreply@anthropic.com>
Inspect vLLM 0.31's KV-holding waiting queue before scheduling running drafts. Preserve the active-request cap and the bridge's mixed-step refusal. Cover the two-step remote completion path and retain legacy scheduler behavior. Co-Authored-By: Claude Code <noreply@anthropic.com>
…ntity Reject missing or invalid cache metrics, use actual per-arm prompt tokens, and retain per-question output for concurrent recall probes. Verify readiness against the expected model and socket-owning process tree, detect explicit server death and stalled completed requests, and document current graph DSpark results without carrying forward old performance claims. Co-Authored-By: Claude Code <noreply@anthropic.com>
Keep running replay tails out of speculative verification even after they cross the original prompt boundary. Preserve steady decode speculation and cover the pre-schedule ordering with CPU red/green tests. Record FULL-graph DSpark MP c1 and c4 revalidation, the tested MP startup recipe, and separate shutdown errors from inference results. The additional output-history guard still needs graph service validation. Co-Authored-By: Claude Code <noreply@anthropic.com>
Only cold-arm nonces change per invocation; the fixed-seed warm document is reused within the boot. Do not describe these runs as independent primes. Co-Authored-By: Claude Code <noreply@anthropic.com>
Represent each opaque snapshot as one synthetic KV_2LTD token without changing its uint8 payload. Keep nonblocking allocation and exercise real LMCache token accounting and backend cleanup on CPU. Co-Authored-By: Claude Code <noreply@anthropic.com>
Use vLLM 0.31's constant-K dynamic lookup path only during static V4.1 scheduling, preventing newly admitted padded decode rows from mixing with prefills. Preserve real drafts, async bonus accounting, existing dynamic schedules, and non-V4.1 behavior; restore the lookup in finally. Cover the real sync/async scheduler admission and pin the two upstream lookup consumers. CPU scoped and legacy connector regression: 811 passed plus 69 subtests. Co-Authored-By: Claude Code <noreply@anthropic.com>
Record the two distinct-document MP concurrency-4 probes at f29878c and the 811-test CPU regression. Keep remaining AITER shutdown and recovery limitations explicit. Co-Authored-By: Claude Code <noreply@anthropic.com>
Track borrowed ProcessGroups and owned AITER communicators explicitly. Close plugin resources before the vLLM TP owner and retain dependencies when cleanup raises. Release temporary resources before initialization fallback and preserve repeated-init/destroy semantics. Add ten CPU lifecycle cases. A graph DSpark MP recall smoke passes and all four explicit cleanup hooks execute; later AITER destructor errors and worker SIGKILL escalation remain unresolved. Co-Authored-By: Claude Code <noreply@anthropic.com>
Bind the real SpeculativeConfig drafting-slot property instead of assuming K slots. Cover sync and async input budgets at 5 and 4 with K=5; the ample-budget admission regressions remain unchanged. Co-Authored-By: Claude Code <noreply@anthropic.com>
Record the new graph DSpark MP recall result and 825-case CPU suite. Explicit AITER cleanup executes on all four ranks, but destructor exceptions and default-timeout SIGKILL escalation remain. Keep full accuracy, local-HBM recovery and performance claims pending. Co-Authored-By: Claude Code <noreply@anthropic.com>
Add fixed-manifest official GSM8K and fixed-request streaming performance clients with CPU regressions and explicit source-accounting gates. Record all four full GSM8K passes and six valid ON performance windows at fdb24d5. Preserve length-gate failures and the OFF illegal-memory-access failure; do not claim a paired speedup or clean shutdown. Keep pending runner-shutdown changes outside this commit. Co-Authored-By: Claude Code <noreply@anthropic.com>
PerryZhang01
force-pushed
the
perzhang/dsv41-flash-vllm-plugin-nolmcache
branch
2 times, most recently
from
October 15, 2026 15:14
85ad5fe to
6099de0
Compare
Base automatically changed from
perzhang/dsv41-flash-vllm-plugin-nolmcache
to
main
October 16, 2026 05:29
Contributor
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #2494 (
perzhang/dsv41-flash-vllm-plugin-nolmcache): this PR adds the LMCache offload layer for DeepSeek-V4.1-Flash on vLLM 0.31.Scope and current status
own-poolSTATE have passed graph + DSpark recall smoke tests. 2026-10-14 acceptance atfdb24d5d2is not passed: full fixed-manifest GSM8K recovered 84.45% from CPU on ON pass 2, but ON/OFF both had length-truncated repetition; six ON performance windows passed while the first OFF window crashed with HIP illegal memory access. No paired speedup is available. Graceful shutdown remains broken. Details below; earlier smoke results are retained as history.ATOM_V41_STATE_IMAGES,ATOM_V41_LOCAL_PREFIX_HITSandATOM_V41_IMAGE_LOCAL_HITSto0. Speculative offload remains opt-in throughATOM_V41_ALLOW_SPECULATIVE_OFFLOAD=1.Fixes in this update
Preserve failed ranks'
finished_recvingvotes while separately reportingfailed_recving; join PAGE and STATE before releasing a receive. Cover staggered TP completion, MP failures, abandoned promises and retry handling.Remove the poll-count reaper for already dispatched loads. A poll count does not fence a writer or prove cancellation. Detection of promises that never became dispatched loads remains.
Treat V4.1 slot zero as valid without changing KDA's NULL block zero. Reject invalid/full-pool destinations, fence pending STATE writers before releasing unclaimed reservations, and fail before mutating allocator ownership on over-capacity. This prevents silent slot overwrite; it is not recoverable capacity admission.
Resolve the target group's block size in the private offload configuration before either transport freezes its PAGE namespace. MP scheduler and workers now agree on block 256 and the same namespace, without changing the engine-wide/draft block 32 or legal TP-collapsed wire world size 1.
Include vLLM 0.31's
kv_holding_waitingqueue in the pre-schedule V4.1 mixed-step guard. A restored tail can be admitted from that queue after running drafts have been scheduled. Preserve the active cap and bridge rejection. CPU red/green and graph service revalidation are complete: one long single-client run and two long concurrency-4 runs passed in the same boot. Cold-arm nonces were fresh; the fixed-seed warm document was reused across invocations.Cover running output-history replay using
is_prefill_chunk, rather than only comparing computed tokens with the original prompt length. Preserve ordinary decode drafts.Suppress vLLM 0.31's new-admission placeholder padding during V4.1 scheduling (
f29878cab): a call-scoped constant-Kdynamic_sd_lookupprevents a new prefill from mixing with a newly padded decode, without changing next-step K, real decode drafts, or async bonus-token accounting. Restore the field infinally; preserve an existing dynamic lookup unchanged. Real sync/async scheduler regressions and a source-contract test cover this private-API adaptation.Fix LMCache STATE-object token accounting (
d1704f7f2): use logical uint8 shape(1, 1, 1, nbytes)withKV_2LTD, counting each opaque snapshot as one synthetic token. Physical bytes and flat payload are unchanged. Real LMCacheget_num_tokens,clearandcloseregressions cover the previous one-dimensional-shapeIndexError.Harden recall and wait tools: actual per-arm prompt-token denominator, required cache metrics, unique cold prefixes, concurrent requests, per-question JSON, PID lifetime/socket ownership checks, and server-death/fault detection independent of launcher exit status.
Attach plugin-owned AITER ca/qr cleanup to the concrete vLLM TP owner's
destroy(3a97b1657). Mark borrowed ProcessGroups and owned communicators explicitly; close plugin resources before the owner, clear only identity-matching globals, preserve errors and retry state, and clean temporary resources before initialization fallback. Ten CPU lifecycle cases passed. This fixes missing ownership/teardown wiring, not all remaining shutdown defects.Correct the DSpark admission-test substitute (
10228cb99) to use the realSpeculativeConfigdrafting-slot property. With K=5, input budget 5 admits one query token and budget 4 admits none. Four new sync/async boundary cases bring admission coverage to 29; no scheduler production change is included in this test correction.Graph + DSpark acceptance — 2026-10-14
Result: acceptance is not complete. Four full GSM8K passes were executed, with genuine CPU recovery in ON pass 2, but repeated generation hit the predeclared length gate in both ON and OFF. Six ON performance windows passed their measurement gates; the first OFF window crashed, so no ON/OFF throughput ratio or speedup is reported.
Frozen configuration and measurement contract
fdb24d5d2de617ce9b47c496368961ca3b88026b, archived before startup. The pending runner-shutdown patch was not included. All 1,885 expected source files and four frozen client/manifest/wait files were checked independently; this checks those expected files, not arbitrary extra files in the directory.0.31.1.dev0+gdb9527a46.d20261008.rocm724, LMCache0.5.5rc3+rocm7.2.4.torch2.10.git3d3aa833.cxx11abi1. Both ON and OFF completed target PIECEWISE 53 / FULL 41 and DSpark FULL 38 captures. Target KVauto(ATOM BF16), draft explicitlyfp8_ds_mla, DSpark K=5; context 32768, sequences 64, batched tokens 8192, GPU utilization 0.78.delta vllm:prompt_tokens_by_source_total{source="external_kv_transfer"} / sum response.usage.prompt_tokens, not lookup hits alone. Local/CPU/computed source deltas must sum to the same prompt-token total. All completed GSM8K passes and ON performance windows had zero local-HBM supply.Full fixed-manifest GSM8K
Official
lm_eval 0.4.13GSM8K test1319, 5-shot, rendered once and replayed without resampling exemplars. Raw/v1/completions, concurrency 16, temperature 0, seed 0, max output 4096; official strict/flexible filters. Manifest SHA256:3060c04950a49c348c637a242b514df6e870990767b87e808322df777bf2ca45.Every pass completed and officially scored all 1319 responses, with zero HTTP failures/cancellations. Source counters were complete and conserved against response usage. ON pass 2 computed 179,483 prompt tokens and recovered 975,104 from CPU, so the >=10% warm-readback requirement genuinely passed. No inference engine/GPU error appeared within those four measurement log prefixes.
ON pass 2 doc 881 repeatedly enumerated years until 4096 tokens; OFF pass 1 doc 298 repeatedly enumerated auction bids until 4096 tokens. OFF pass 2 truncated both docs 298 and 881 with repeated enumeration. These are retained failures, not silently retried or removed. OFF pass 1 doc 881 ended with the correct answer 16; ON pass 1 ended with the incorrect answer 4. Repetition also occurring without offload does not prove that ON's occurrence is unrelated to offload. The observed ON repeat score change is -0.3791 pp and OFF is -0.5307 pp; neither pair establishes a transferable noise floor or absence of regression. The no-truncation gate was fixed before results and remains in force.
Per-question transitions below use the official flexible filter on the same 1319 IDs. Direction is left pass → right pass; the counts describe these runs, not an equivalence test or a causal attribution.
Performance attempt
Per point/round: 12 distinct documents, each primed serially with one output token outside the measurement window; then 240 streaming requests with unique tails, temperature/seed 0, exactly 64 output tokens and
ignore_eos=true. Every round has a new namespace; matching ON/OFF uses the same manifest. Actual mean input length across ON rounds was 7,631–7,635 tokens/request.The timed window is first measured dispatch through final complete SSE response, including ramp/drain but excluding prime and metrics. Every request must have
[DONE], usage and 64 output tokens; ON actual CPU supply must exceed 50%. The client refuses partial-success statistics. Total tokens/s includes reused input tokens and is not GPU-computed-token throughput. TTFT includes the first nonempty content/reasoning event; TPOT is(last_content-first_content)/(completion_tokens-1), not SSE chunk ITL.All six ON windows completed 240/240 requests with 15,360 output tokens each and passed client, drain and independent source/log checks. Per-concurrency total-throughput medians are 102,137 and 119,290 tokens/s; observed ranges are 100,184–103,430 and 116,619–125,360 tokens/s. These are descriptive same-boot results, not an established noise floor.
OFF 16/1 failed: all 12 prime requests succeeded, but only 160/240 measured requests completed successfully; 80 failed. At 06:13:29 UTC workers reported HIP illegal memory access, surfaced asynchronously at
deepseek_v41_bridge.py:1029(seq_lens.cpu()); that stack is not identification of the offending kernel. The API then reported EngineDeadError. Client exited 1, drain exited 2, and the summary deliberately containsperformance=null,supply=null. The other five OFF rounds were not run after the engine failure. There is no valid paired result and no speedup claim.Lifecycle and deliverables
ON graceful shutdown still failed: four explicit TP-close markers and four successful CPU-backend closes were followed by four CA + eight IPC destructor exception frames, four-worker SIGKILL escalation and API EngineDeadError. OFF terminated after its inference fault, not an operator shutdown. Both model launchers returned 0; that does not override either failure. ON's model/MP processes and KFD resources were released. Two host samples after OFF found no KFD holder in its container and closed run ports; four original TP processes remained zombies without KFD entries. This is resource release, not clean teardown.
Delivery commit:
0d04026bb, containing only the two clients, their two CPU test files and the concise recipe; pending runner-shutdown changes are excluded. The measured server source remainsfdb24d5d2.New standalone tools provide fixed-manifest GSM8K and fixed-request streaming performance. These are post-measurement client fixes, not the client hashes used for the service results above. Request rendering, budgets and official scoring are unchanged; the GSM manifest still matches the frozen reference byte-for-byte. The frozen GSM8K client's missing-source loophole was covered by the independent coordinator on every measured pass; real counters were present and conserved.
Client review found and fixed late-final-batch deadline acceptance, repaired-UTF8 scoring, inconsistent lookup counters, missing-metric preflight, manifest hash/parse double reads, unbounded response evidence, and performance comparison trusting an old
gates_passedflag. Streaming validation now reads through HTTP EOF, rejects content after a preceding terminal event, recognizesdelta.reasoning, and permits a one-token prime with no visible text. New performance summaries use validator version 2;comparerejects legacy versions and rechecks source gates and request denominators. The absolute work/round deadlines also catch an event-loop timer delayed by synchronous work. Original summaries were not overwritten, re-versioned or retrospectively called version-2 passes.Final scoped CPU regression: 78 passed, 1 skipped, 189 subtests passed (
pytest --noconftest -p no:cacheprovider, 2.54s). The skipped test was separately run explicitly with the existing official offline cache: 1 passed, with two independently rendered 1319-row manifests identical to the frozen SHA above. All four tool/test files pass Black (--target-version py310) and Ruff. No GPU/model endpoint was used in these client tests; this is not a new production-engine regression run.A separate read-only audit of the saved experiment found: all 5,276 GSM response bodies are strict UTF-8 and match their saved JSON; all four recorded pass elapsed times are below 158s against the 1,500s deadline; lookup bounds and token-source conservation hold in all four GSM passes and six complete ON windows. Across 54,341 saved SSE events (seven attempted rounds, including prime), no nonempty
delta.reasoningor content following an earlier finish event was found; the saved-event reconstruction matches TTFT/TPOT for all 1,684 successful requests. The old client stopped reading at[DONE]: absent saved post-DONE events do not prove that the unobserved HTTP suffix was empty. Failed OFF lacks a complete after-metrics snapshot and remains unvalidated. This audit does not turn the old runs into validator-2 passes. Audit file:delivery/historical-client-boundary-audit-20261014T072717849128Z.json, SHA256ee5c1eb9a98650a1a2d6b07f04a0f6680d93afb2c8d08f03daf7348396c664a2(64 input fingerprints).Remaining client-method limits are explicit: serial prime completion does not wait for a PAGE/STATE save-completion barrier; synchronous evidence writes can delay other requests and measured client latency; an operator-supplied offline cache's provenance is still trusted, not authenticated merely by its row count or a self-computed hash. Post-measurement guards do not resolve those limits or the service failures.
Evidence is retained under
dsv41_lmc/acceptance_graph_dspark_mp_20261014/:validation_plan.json,source_manifest.json,harness_frozen.json, all raw requests/responses/scores/metric snapshots, per-window*_verified.jsonwith measurement-log prefix lengths/hashes, and ON teardown / OFF post-crash release records. Measurement prefixes are kept separate from later shutdown/fault tails. The independentacceptance_results.jsonrecords all four passes, per-question transitions, six recomputed ON windows, failed OFF requests, lifecycle boundaries and 93 input-file fingerprints (SHA256b8db8a53ffcfb137d139eac3467cae6148aaf818cd0f70cc30b6cc879c2b2bee). Recipe: startup, test commands and results.The earlier 2.5x claims remain withdrawn. Still open: the OFF long-prompt GPU fault and repetition/truncation, clean shutdown, STATE byte equality, cancellation/load-failure recomputation, recoverable STATE capacity, and local-HBM PAGE retention with CPU-only missing-tail plus matching-STATE recovery.
Accuracy and recovery validation — 2026-10-13
Environment: TP4 on MI355, vLLM
0.31.1.dev0+gdb9527a46.d20261008, LMCache0.5.5rc3+rocm7.2.4.torch2.10.git3d3aa833.cxx11abi1; FULL decode graphs + DSpark 5 (FULL_AND_PIECEWISEwith breakable capture; target and DSpark FULL captures verified). Target KV dtypeauto(ATOM BF16 pool), draft explicitlyfp8_ds_mla. Maximum context 32768, maximum sequences 64, batched tokens 8192, GPU utilization 0.78, chunk 256, STATE interval 4096.tools/kv_offload_accuracy_arms.pyruns blind / unique-prefix cold / shared-prefix warm arms. Each arm has 16 questions. A passing run requires correct answers and at least 50% warm-tier supply measured against all actual prompt tokens in that arm. These are synthetic recall smoke tests, not full GSM8K or STATE byte-equality validation.f29878cab, seed 2026101301f29878cab, seed 202610130210228cb99, seed 2026101303Completed cold arms had zero tier supply; reported local-HBM supply was zero. In-process used 16 GiB/rank. MP used a shared 32 GiB PAGE pool and 4 GiB/rank own-pool STATE. These pool configurations are not a backend performance comparison.
The concurrency-4 warm run raised the bridge's safety check for rows
(position, length, drafts) = [(17462, 6, 5), (17408, 51, 0)]and returned HTTP 500. Its client exited 1 and drain exited 2, even though the API launcher eventually exited 0. This is recorded as a failure, not a passing run.The post-fix c1 and c4×2 runs used one boot and the same fixed-seed warm document at source
1e3ecb590. Only the cold-arm nonces changed per invocation; these are repeated restores, not three independent primes. All client/drain exits were zero and 1,051 tracked ATOM source files were unchanged. Shutdown afterward logged LMCacheLocalCPUBackendclose errors and AITERfree_meta_bufferdestructor errors; vLLM escalated its four workers to SIGKILL. All seven recorded model/MP processes and their host KFD entries were confirmed absent. The shutdown issues are not resolved by the passing inference probes.The earlier validation at
f29878cabincludes the running replay, new-admission padding, and STATE-shape fixes. Both concurrency-4 probes used different warm documents, in one boot, with seeds2026101301and2026101302(document SHA256 prefixes7b0c411192e3andc52cd1471b16). All four TP ranks completed graph capture. Both client/drain exits were zero; per-question prompt/correct sums matched each arm's totals. No workload error matched the explicit fault scan, all 12 loggedcursor_mismatchvalues were zero, and 1,051 tracked ATOM source files were unchanged through teardown. These are service smoke results for the new source, not independent boots or proof that each CPU corner case occurred in the service.Shutdown recheck at the same source: four
LocalCPUBackend closed successfullylines and zeroError closinglines; the prior one-dimensional STATE-shape close error did not recur. AITERfree_meta_bufferdestructor errors persisted, and vLLM still escalated all four workers to SIGKILL. These are separate observations; neither proves the cause of that escalation. The model launcher exited 0; the separately identity-verified MP service was intentionally sent SIGTERM and exited 143. All seven recorded model/MP processes and their host KFD entries were absent and ports 8021/5561/8086/9006 were closed afterward. Graceful shutdown is still not clean.2026-10-13 TP-lifecycle validation,
10228cb99: a new boot ran seed2026101303, concurrency 4, 900 facts and maximum 640 output tokens. Blind/cold/warm scored 0/16, 16/16 and 16/16; warm tier supplied 278,528 of 279,264 prompt tokens (99.736%). Cold tier and all local-HBM supply were zero. All 49 HTTP requests completed, client/drain exits were 0, per-question correctness/token sums were independently checked, and four ranks completed target and DSpark FULL graph captures. The source was frozen from startup through model/MP teardown: all 1,051 tracked ATOM files and five validation files match the subsequently committed10228cb99byte-for-byte.2026-10-13 shutdown acceptance FAILED: after identity-verified API SIGTERM, four explicit
ATOM plugin: closed reused AITER TP collectivesmarkers appeared. Four STATELocalCPUBackendcloses succeeded with noError closing. Nevertheless, fourCustomAllreduce.__del__and eightIPCBuffer.__del__exceptions followed, allUnicodeDecodeErrorfrom the nativefree_meta_buffercall; vLLM sent SIGTERM to four workers after its unchanged 5-second grace, then SIGKILL to four workers after another 4 seconds. These observations do not prove that the adapter's native frees succeeded or that the destructor errors alone caused escalation. Model launcher exit 0 does not override failed shutdown acceptance. MP was separately identity-verified and deliberately terminated (exit 143). At 19:53:29 UTC, all seven recorded processes and host KFD entries were absent and all four ports were closed. Evidence:dsv41_lmc/tp_cleanup_graph_spec_mp_20261013/{accuracy_seed1_verified,teardown_verified,source_before,source_after}.jsonand the preserved server log. No emergency outer timeout fired.Targeted CPU red/green coverage:
[6, 32]; theis_prefill_chunkguard produces[1, 32]before scheduling while preserving ordinary decode drafts.allocate_slots, while preserving next-step K=5, real drafts, existing dynamic lookups and async bonus accounting. Source-contract tests pin the two vLLM 0.31 lookup read sites.clear/closefail with the old one-dimensional shape and pass with the new shape; byte count, aliasing and flat payload remain unchanged, including the 5,971,968-byte DSpark entry. Full-poolNonebehavior is preserved.CPU-focused receive/STATE/namespace/scheduling/codec/probe/wait regression: 825 passed + 69 subtests, with 90 OpenTelemetry deprecation warnings (22.54s, exit 0). This adds ten TP lifecycle cases and four DSpark input-budget cases to the preceding 811-case suite. The 10 TP cases all failed against the old implementation; the corrected admission suite has 29 passing cases. Black and Ruff passed. This is a scoped suite, not a claim that the entire repository suite passes or that every unrelated failure matches the base.
Commands and current results: recipe.
Remaining validation and functionality
Performance
Six valid ON measurements are reported in the 2026-10-14 section above. No paired ON/OFF result or speedup is claimed: the first OFF window failed, and the other five were not run. The previous 2.5x figures and bounded-withdrawal results remain withdrawn. Without this connector, the current V4.1 plugin requires prefix caching off, so the comparison is reuse versus no reuse, not CPU tier versus local-HBM reuse.
🤖 Generated with Claude Code