Skip to content

offload/dsv4: give PAGE saves a failure channel and roll the watermark back - #2204

Merged
ZhangLirong-amd merged 1 commit into
mainfrom
zhanglirong/dsv4-page-save-failure-channel
Sep 14, 2026
Merged

ZhangLirong-amd merged 1 commit into
mainfrom
zhanglirong/dsv4-page-save-failure-channel

Conversation

@ZhangLirong-amd

@ZhangLirong-amd ZhangLirong-amd commented Sep 12, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

DSV4OffloadScheduler advances a sequence's saved watermark where the save is
emitted — before admission, before execution — and reads it one step later as
SaveSpec.skip_leading_tokens. The advance is a prediction, and nothing checked it.

Save admission is a non-blocking BoundedSemaphore, so a save rejected on
max_pending_saves is dropped and never retried while the watermark has already
moved past it. A save that dies inside the executor is worse: _do_save_req
catches its own exceptions and does not re-raise, so the operation is reported as
success. Either way every later incremental save skips that range, leaving a
permanent hole in the persisted prefix. Nothing on the save side ever lowers
entry[1] — the only lowering lives in load_failed.

One hole is enough to disable the sequence. _page_boundary_visible requires
lookup(tokens[:boundary]) >= boundary, and lookup returns a contiguous
prefix length, so a gap anywhere at or before the boundary truncates the hit
there and SLOT publication times out for the rest of that sequence's life.

This is why raising CPU capacity cannot help: capacity only changes how often the
queue saturates, and one saturation poisons that sequence permanently. Higher
concurrency saturates more often, which is why 256-concurrency failed and low
concurrency did not.

Why the rollback needed new plumbing first

The scheduler could not tell a save that landed from one that did not.
get_finished reports PAGE saves only through finished_saving — one
undifferentiated set — and save_finished takes no success flag. The
connector_completions channel, which does carry succeeded and is TP
aggregated, was wired only for the SLOT sidecar. PAGE had no failure channel at
all, so there was nothing a rollback could hang off.

Change

Give PAGE saves their own completion channel, alongside the sidecar's.

Worker — DSV4_PAGE_SAVE_CHANNEL, reusing the existing generic completion
mechanism: channel dispatched, TP aggregated, failure dominant (if any rank did
not persist its shard, the range is not durable). No shared structures change. A
verdict is reported on every terminal path — rejection before admission,
rejection at submission, and out of the executor via a page_stored flag — plus
a backstop in _guard, because an operation that reports nothing would sit in
the TP aggregator forever.

Scheduler — record what each advance was contingent on, keyed by save
operation. connector_completion releases the record on success and, on failure,
rolls the watermark back and cancels the save statistics so bytes that never
landed do not inflate total_saved_tokens.

Ordering works because process_completions applies connector_completions
before finished_saving: the watermark is corrected while the operation is still
in _save_inflight, and the next build_connector_meta re-emits the range.
Rolling back past a later save that did land costs one redundant write and
nothing else — stores are idempotent, and a prefix has to be contiguous anyway.

Verification

Measured on DeepSeek-V4-Pro, PD-disaggregated, concurrency 256 with 8 x 128 GiB
LocalCPUBackend, against the same configuration on the unfixed code:

signal before after
page_visibility_timeout 1,874 0
SLOT sidecar failure rate 83.7% 26.0%
SLOT loads restored 23 1,075
bytes read back from CPU 0.58 GiB 27.17 GiB
outcome operator-aborted in warmup completed

The failure mode is worth naming precisely, because it is easy to miss: the
unfixed run did not error and did not crash. Client-side errors stayed
at 0 and every request returned normally. What happened is that each SLOT save
waits on _page_boundary_visible, a hole makes that wait unsatisfiable, and the
save burns its full timeout before giving up — so the run simply never finished
warmup. It reached 1,293 of 2,845 warmup requests in 2,250 s, work the fixed
build does in 1,800 s, and was stopped by hand. The only outward signal is 1,874
timeout lines in the server log.

(Both builds log a handful of KV transfer drain during shutdown failed
tracebacks at teardown; those are unrelated to this change and present either
way.)

In one 3,600 s run the fix rolled back 6,723 advances covering 29,989,120
tokens
across 6,183 sequences — all of which the unfixed code would have
declared saved and never written. 1,148 of those holes sat at offset 0, where the
prefix is worth nothing at all; the rest had a median offset of 79,616 tokens.

Rollback tracks rejection 1:1 and is a load-step transient, not a runaway: the
rate spikes on each concurrency ramp and decays within two minutes.
PAGE save did not land stayed at 0, so the executor-internal path is covered
but not yet exercised.

Tests

556 passed, 1 skipped across the kv_transfer suites. The new rollback test is
not vacuous — neutering _rollback_save_watermark makes it fail with
assert 8 == 0, which is exactly the hole.

  • test_failed_page_save_rolls_the_watermark_back_and_re_emits
  • test_page_watermark_records_do_not_outlive_their_request
  • test_worker_reports_a_page_verdict_on_every_terminal_path

The hand-rolled __new__ fixtures mirror __init__ field by field, so they
carry the two new sets.

Not covered here

SLOT sidecar saves still fail at roughly 15-25% under load, and are deliberately
not retried. That is a separate issue: OFFLOAD_SLOT_STAGING_SLOTS defaults to 1,
so each rank has a single staging row contended by saves and by loads that hold it
for a whole batch. Unlike the PAGE hole this costs cache hit rate, not
correctness — a missing sidecar is a miss on a later load, not a corrupt prefix.

Copilot AI lite review requested due to automatic review settings September 12, 2026 02:47
@github-actions

Copy link
Copy Markdown
Contributor

🏷️ CI Guide

Runs automatically on every eligible PR before approval:

  • ✅ Pre Checkin: Black, Ruff, catalog schema validation, non-GPU unit tests

Heavy model tests:

  • ✅ Run after the PR is approved and Pre Checkin passes
  • ✅ Run immediately when an approval review is submitted
  • ✅ Can be requested before approval with labels
Label Tests
ci:full Run all heavy PR model tests: native ATOM, vLLM, and SGLang
ci:atom Run native ATOM model accuracy tests
ci:vllm Run ATOM vLLM OOT model accuracy tests
ci:sglang Run ATOM SGLang model accuracy tests

Heavy jobs are skipped when the PR is not approved and no matching ci:* label is present.
Add labels via the sidebar or gh pr edit 2204 --add-label <label>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

A PAGE failure after request completion may not be re-emitted, leaving the persisted prefix incomplete.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

Adds failure-aware PAGE save reporting for DSV4 and rolls back speculative watermarks when saves fail.

Changes:

  • Adds TP-aggregated PAGE completion reporting.
  • Rolls back failed save watermarks and statistics.
  • Extends scheduler, worker, and fixture tests.
File summaries
File Summary
tests/test_lmcache_offload_v4_page_slot.py Initializes PAGE completion state in fixtures.
tests/test_lmcache_offload_connector.py Tests rollback, cleanup, and terminal reporting; some PAGE failure paths lack direct assertions.
atom/kv_transfer/offload/hybrid/dsv4/connector.py Implements PAGE completion handling and watermark rollback; a post-FINISHED failure can still release the retry tracker and leave a prefix hole.
Review details

Suppressed comments (2)

atom/kv_transfer/offload/hybrid/dsv4/connector.py:1693

  • The executor-side PAGE outcome is not covered: _do_save_req derives this verdict from page_stored after _engine.store and reports it here, but the new worker test only exercises _finish_unadmitted_save; the existing exception tests use save_spec=None and therefore cannot catch a regression in the PAGE channel. Add a PAGE request test that drives _run_save_req with store raising and asserts get_finished().connector_completions contains the failed operation (plus the successful store case).
                self._report_page_save_locked(req, succeeded=page_stored)

tests/test_lmcache_offload_connector.py:2651

  • This test does not cover all of the new PAGE terminal paths despite its name: it exercises only pre-admission rejection and a SLOT-only request. The submission-rejection and executor-exception cases still use existing tests that assert only the checkpoint channel (or use page=False), so a regression that stops emitting DSV4_PAGE_SAVE_CHANNEL for those paths would pass. Add assertions for a PAGE request in both _finish_rejected_save and _do_save_req/_guard failure cases.

    sched._save_inflight.clear()
  • Files reviewed: 3/3 changed files
  • Comments generated: 1
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Copilot AI review requested due to automatic review settings September 12, 2026 02:52

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

PAGE visibility failures may still commit watermarks, and key completion paths lack regression coverage.

Get a fresh assessment by requesting another Copilot review.

Review details

Suppressed comments (3)

atom/kv_transfer/offload/hybrid/dsv4/connector.py:1798

  • The new PAGE channel conversion is not exercised end-to-end: the worker test only inspects _failed_page_save, while the scheduler test constructs ConnectorCompletion directly. Existing page-store failure tests also do not assert this output, so an omission or wrong succeeded value here would leave the rollback path apparently green. Add a focused get_finished()/TP-aggregation assertion for both PAGE success and failure (including failure dominance).
        connector_completions.update(
            ConnectorCompletion(
                channel=DSV4_PAGE_SAVE_CHANNEL,
                operation_id=completion_id,
                succeeded=True,
            )
            for completion_id in dps

atom/kv_transfer/offload/hybrid/dsv4/connector.py:1599

  • Please add a regression for the executor-internal PAGE failure, not only admission rejection. The new page_stored flag is set around _engine.store, but the current tests do not run _do_save_req with store() raising and then verify that get_finished() emits a DSV4_PAGE_SAVE_CHANNEL failure. This is the path that prevents an executor failure from leaving the watermark advanced, so a regression here would recreate the permanent hole while the existing tests still pass.
            page_stored = True

tests/test_lmcache_offload_connector.py:2657

  • [verified] This test only calls _finish_unadmitted_save() and inspects the private sets; it never calls get_finished() to verify that DSV4_PAGE_SAVE_CHANNEL is emitted, nor exercises _finish_rejected_save() or _do_save_req() when store() fails. A regression in the submission/executor paths or in channel draining would therefore still pass while the TP aggregator receives no PAGE verdict. Author must add assertions for the emitted PAGE completion and cover the remaining terminal paths.
def test_worker_reports_a_page_verdict_on_every_terminal_path():
    """A missing report strands the operation in the TP aggregator forever."""
  • Files reviewed: 3/3 changed files
  • Comments generated: 1
  • Review effort level: Lite

…k back

The scheduler advances a sequence's saved watermark where the save is
*emitted* -- before admission, before execution -- and reads it one step later
as SaveSpec.skip_leading_tokens. The advance is therefore a prediction, and
nothing ever checked it. Save admission is a non-blocking BoundedSemaphore, so
a save rejected on max_pending_saves is dropped and never retried while the
watermark has already moved past it; a save that dies inside the executor is
worse still, because _do_save_req swallows its own exceptions and reports the
operation as done. Either way every later incremental save skips the range,
leaving a permanent hole in the persisted prefix.

That single hole explains the whole 256C+DPA failure: _page_boundary_visible
waits for page_hit >= boundary_tokens, which a hole caps below the boundary
forever, so SLOT publication times out repeatedly and no prefix is ever
complete, so CPU readback stays zero. The max_pending_saves rejections are the
trigger, not the bug -- rejecting a save is fine, keeping the advance is not.
This is why more CPU capacity cannot help: capacity only changes how often the
semaphore saturates, and one saturation poisons that sequence for good, which
is also why high concurrency shows it and low concurrency does not.

The rollback could not be written until the scheduler could tell a save that
landed from one that did not, and it could not: finished_saving is a single
undifferentiated set, and both outcomes arrive through it identically. So give
PAGE saves their own completion channel, next to the SLOT sidecar's. It is
TP-aggregated and failure-dominant, exactly right here -- if any rank failed to
persist its shard the range is not durable. Every terminal path reports a
verdict: rejected before admission, rejected at submission, and out of the
executor, with a backstop in _guard, because an operation that reports nothing
would sit in the aggregator forever. process_completions dispatches connector
completions before finished_saving, so the watermark is corrected while the
operation is still inflight and the very next build_connector_meta re-emits the
range.

Rolling back past a later save that did land costs one redundant write and
nothing else: stores are idempotent, and a prefix has to be contiguous, so
everything beyond the hole was unusable anyway. Bytes that never landed are
cancelled rather than counted, so a drop no longer inflates total_saved_tokens.

The hand-rolled __new__ fixtures mirror __init__ field by field, so they carry
the two new sets.
@ZhangLirong-amd
ZhangLirong-amd force-pushed the zhanglirong/dsv4-page-save-failure-channel branch from 0707c77 to e898295 Compare September 12, 2026 03:50
Copilot AI review requested due to automatic review settings September 12, 2026 03:50

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Exclude non-saving PAGE specs from successful verdicts and add coverage for _guard executor failures.

Get a fresh assessment by requesting another Copilot review.

Review details
  • Files reviewed: 3/3 changed files
  • Comments generated: 2
  • Review effort level: Lite

Comment on lines +774 to +775
if getattr(req, "save_spec", None) is None:
return
Comment on lines +1074 to +1078
if kind == "save":
# Backstop. ``_do_save_req`` normally reports its own PAGE
# verdict and swallows its errors, so this only fires if it
# died outside that handling -- but a save that reports
# nothing would sit in the TP aggregator forever, so the
@valarLip

Copy link
Copy Markdown
Collaborator

Reviewed at e898295e1.

Three claims in the PR body check out and are not flagged below: process_completions really does apply connector_completions before finished_saving (_offload_common.py:363 vs :378); the TP aggregator really is failure-dominant per worker (aggregator.py:64); and _lock really is a plain Lock with no deadlock — every _report_page_save caller is outside it.

The new tests also have real revert power, which is worth recording so nobody re-checks: neutering _rollback_save_watermark gives exactly assert 8 == 0, neutering the record-keeping fails 2 tests, neutering the worker verdict fails 6.

Two findings below I would treat as blocking. The rest are ordered after them.


1. page_stored = True is set unconditionally — a PAGE success is certified for a range nothing wrote

page_plan = self._page_save_plan(req)
if page_plan is not None:
    ...
    self._engine.store(...)          # only runs when page_plan is not None
if slot_spec is None:
    # PAGE-only save. `store` returning is the only durability
    # signal that exists here, so it is the verdict; a raise above
    # leaves this False and the advance gets taken back.
    page_stored = True               # runs regardless

The comment states the contract — "store returning is the only durability signal" — and nothing checks that store ran.

_page_save_plan returns None in two cases (:1461-1470):

if ss is None or not bool(getattr(ss, "can_save", True)):
    return None                       # (a) the save was refused
...
if skip >= len(toks):
    return None                       # (b) empty after chunk-floor truncation

(a) is a genuine hole. The watermark was already advanced to aligned at emit time, nothing was stored, and page_stored = True then drives _report_page_save_locked(succeeded=True), which confirms the advance and lets _release_save_watermark retire the rollback record. The range is permanently skipped and actively certified as landed — the exact failure this channel exists to prevent, now with the recovery path deliberately closed. (b) may be harmless depending on how aligned relates to the chunk-floored end, but the code cannot tell the two apart.

One line:

page_stored = page_plan is not None

2. The dense connector has the identical bug, and already has the verdict in hand

Same emit-time advance:

# dense/connector.py:791
entry[1] = aligned

The failure verdict already arrives:

# dense/connector.py:1024
self._store_finished(operation, succeeded=completion.succeeded)

And the failure branch does not restore it:

# dense/connector.py:1086-1090
else:
    self._cancel_save_statistics(operation)
    # Do not infer source-safety from a failed store. Any ranges whose
    # GPU staging callback already reached quorum were released above;
    # the rest stay leased until the conservative abandon timeout.

So the next build_connector_meta sees aligned <= saved and emits nothing for that range forever — the same permanent prefix hole, on the connector with more production mileage, with the signal already wired.

The load side of the same file already knows the pattern:

# dense/connector.py:1176-1179
# The LMCache hit was not actually loaded. Let the recomputed
# [HBM, LMC) chunks be saved again instead of permanently treating
# them as already persisted.
entry[1] = self._chunk_floor(floor)

Per Fix-then-sweep this belongs in the same pass. DENSE_PAGE_STORE_CHANNEL already carries succeeded=False all the way to _store_finished; the store side just never restored the watermark.


3. The channel is inert on the legacy completion shape — and that is the only shape the tests use

DSV4's PAGE branch accepts any operation_id:

if completion.channel == DSV4_PAGE_SAVE_CHANNEL:
    if completion.succeeded:
        self._release_save_watermark(completion.operation_id)
    else:
        self._rollback_save_watermark(completion.operation_id)
        self._cancel_save_statistics(completion.operation_id)
    return True

The sibling has the guard this one is missing:

# dense/connector.py:1022
if not isinstance(operation, SaveOperationId):
    return False

_save_completion_id is getattr(req, "save_operation", None) or req.req_id (_offload_common.py:163), and this class explicitly supports the operation-less legacy shape — _finish_save_operation branches on it and _finish_unadmitted_save reads _pending_legacy_save_ops. But _save_watermark_rollback[sid] is keyed exclusively by SaveOperationId, so with a bare req_id: _forget_save_watermark misses, _rollback_save_watermark returns at if previous is None, and the method still returns True. No rollback, no warning, indistinguishable from a success.

_watermark_sid's getattr(operation, "req_id", operation) is written to tolerate exactly that shape, so the two halves of the function disagree about what an operation is.

Compounding this: tests/test_lmcache_offload_v4_page_slot.py:648

def _save_request(*, page: bool = True) -> LMCacheReqMeta:
    return LMCacheReqMeta(
        req_id=17,
        ...
    )                      # no save_operation

so all five new PAGE tests assert against {17: ...} / {71: ...} — the bare-req_id shape. build_connector_meta always sets save_operation=self._next_save_operation(seq), and that generation is precisely what _save_watermark_rollback keys on. The only shape under test is the one where the feature does nothing.

Either add the isinstance guard (and test the real shape), or narrow the type and drop the duck-typing.

4. Fix-then-sweep on the third teardown site, plus a missing owner check

The PR added _save_watermark_rollback.pop(sid, None) to two of three teardown paths:

:2075-2076   new seq takes over an sid   -> pops _sidecar_hash_cache, _failed_sidecar_saves, NOT the rollback record
:2587        abandon_save                -> pops it
:2711-2712   request_finished            -> pops it (the line right after _failed_sidecar_saves.pop)

request_finished only pops when not self.should_defer_free(seq), which is False for exactly the case that matters — a save still inflight — so a request torn down mid-save leaves its record behind for the takeover path to inherit.

And _rollback_save_watermark has no owner check:

sid = self._watermark_sid(operation)
entry = self._save_tracker.get(sid)
if entry is None:
    return
current = int(entry[1])
if current <= previous:
    return
entry[1] = previous          # no `entry[0] is seq` guard

Every other tracker consumer in this file has one (:2706 cached[0] is seq, :2709 entry[0] is seq). So a late PAGE failure for a dead operation can stamp a new sequence's watermark with a dead request's number; with a large initial_saved from a prefix hit and a stale previous of 0, it re-stores a prefix LMCache already holds.

5. Moving the slot-error raise below the visibility wait makes a doomed save burn the full 5 s

raise slot_preparation_error / _require_slot_components() / _require_checkpoint_codec() now run after _wait_for_session_publication.

OFFLOAD_PUBLICATION_TIMEOUT_S defaults to 5.0 s with a 0.01 s poll (connector.py:212-219) — up to ~500 lookup probes. OFFLOAD_COPY_WORKERS defaults to 1, so max_pending_saves is max(2, 2*workers) = 2. And slot_preparation_error is set by the broad except at :1571, which covers _prepare_slot_save returning _SlotSaveSnapshot(None, None, False) when SLOT staging admission is exhausted — precisely the saturated regime this PR is about.

Before this PR that raised instantly and freed the permit. Now two such saves wedge the rank's save leg for 5 s each, which drives more requests into _finish_unadmitted_save, which this PR turns into rollbacks and re-emissions against the same stalled worker. A permanently misconfigured _require_slot_components() makes every composite save pay 5 s forever instead of failing instantly, and the staging row is held 5 s longer since staging_terminalized is False on this path.

Related, same move: when the probe times out first, save_failure_reason = "page_visibility_timeout" raises a fresh RuntimeError and the slot_preparation_error captured at :1571 — an _SlotStagingSyncError from a failed GPU fence, say — is never raised, never chained, never logged. The except _SlotStagingSyncError arm downstream is unreachable whenever the probe loses the race. At minimum raise RuntimeError(...) from slot_preparation_error.

6. _guard can report FAILURE for an operation the same rank already reported SUCCESS

The profiling block at :1743-1772 sits outside _do_save_req's try/finally and eagerly evaluates ~20 int(transfer_stats.get(...)) / float(...) conversions. If it raises, the exception escapes into _guard, which adds the completion_id to _failed_page_save when the finally at :1707 already added it to _done_page_save.

_report_page_save_locked (:767-780) does a bare add on each side. Compare _complete_load_locked (:1085): if succeeded and completion_id not in self._failed_load: ...; self._done_load.discard(...).

Within one drain the TP aggregator AND-folds to failure — a landed save is rolled back and re-stored, and _cancel_save_statistics permanently undercounts. Across drains it is worse in both directions: if the success reached quorum first, _TPCompletionGroup tombstones the key (aggregator.py:161) and the later failure is silently swallowed; under PP, pp_kv_aggregator has no tombstone, so the orphan re-enters self._connector, never reaches pp_size, and has_pending() — read by pp_engine_core.py:196 — never clears, busy-looping the head with GPUs idle.

7. Rollback turns admission saturation into an unbounded retry loop

_has_pending_save is self._save_frontier(seq) > int(entry[1]) (_offload_common.py:660), so lowering entry[1] makes should_defer_free True and re-arms page_save_due. Each retry re-emits from the old, lower skip_leading_tokens, so it is a larger store than the increment originally due, competing for the same exhausted semaphore, with blocks pinned in deferred_free_blocks for the whole loop. No backoff, no cap.

dense/connector.py:654 and kimi_k3/connector.py:659 both define _may_emit_save, applied as a break in the emit loop (dense:746). DSV4's emit loop (:2368) has no bound at all and lets the worker's non-blocking BoundedSemaphore drop instead. Giving DSV4 the existing _may_emit_save seam removes the dominant rollback cause — the PR's own 6,723 rollbacks / 30M tokens — rather than compensating for it. The new test exercises exactly one retry that succeeds; nothing covers a save that keeps being rejected.

8. The PAGE verdict is scoped to the whole prefix, not the range this save stored

_page_boundary_visible requires lookup(req.token_ids[:boundary_tokens]) >= boundary_tokens, and lookup returns a contiguous prefix length.

A sequence has saved [0, 8192); LMCache evicts chunk [0, 256) under CPU-tier pressure. The next boundary save stores [8192, 16384) cleanly, the probe can never reach the boundary, it burns the 5 s timeout, page_stored stays False, and the watermark drops to 8192. The next step re-stores [8192, 16384) — already present — while the actual hole below 8192 is never re-emitted, so the redundant full re-store buys nothing. Pre-PR this timeout only failed the sidecar, which _failed_sidecar_saves suppressed; now it additionally costs a full PAGE re-store per boundary.

The verdict should be scoped to what this save owns.

9. The verdict and the terminal _done_save are written under two separate lock acquisitions

_do_save_req's finally takes _lock, records the verdict, releases; _done_save is added later by _finish_save_operation (:1115) under a fresh acquisition. _finish_unadmitted_save is worse: _report_page_save(...) at :800 takes and releases, then :802 immediately takes it again for the _done_save block — two acquisitions where one would do, on the path that by construction only runs when the system is saturated. Same shape at :924.

The comment at :2601 ("the watermark is corrected while the operation is still in _save_inflight; clearing it there then lets the very next build_connector_meta re-emit the range") assumes both ride the same drain. When they split, the rollback lands at step N but page_save_due's sid not in self._save_inflight guard suppresses re-emission until step N+2.

dense/connector.py:210-221 _record_store_terminal adds _done_save and the ConnectorCompletion in one lock block. Use _report_page_save_locked inside the existing blocks and delete the lock-taking wrapper.

10. A failed PAGE save is statistically invisible

_offload_common.py:337 has total_load_failures, bumped by _finish_load_statistics(succeeded=False). There is no total_save_failures, and _finish_save_statistics takes no success flag.

On the PAGE failure branch, _cancel_save_statistics pops _save_inflight_tokens so the later _finish_save_statistics no-ops: total_save_requests and total_saved_tokens both stay flat and saves_pending drops — statistically identical to a save that was never emitted. Combined with the uncapped rollback→re-emit loop in #7, a sustained max_pending_saves saturation produces a silent retry storm that get_statistics() renders as a healthy flat save line, with only a per-occurrence WARNING as evidence.

Making PAGE failure observable is the point of the channel; it currently stops at the log line. The new test at tests/test_lmcache_offload_connector.py:2596 asserts total_saved_tokens == 0, which locks the blindness in.

11. Altitude: advancing on completion removes the need for the undo

The fix keeps the emit-time advance and bolts a compensating undo onto it. Advancing on completion needs the same dict and the same channel but no undo, and nothing appears to block it: page_save_due (:2384) is already gated by sid not in self._save_inflight, so a not-yet-applied advance cannot cause per-step duplicate emission while a save is inflight. The one exception, boundary_needs_page, emits an overlapping range whose cost the PR's own docstring already declares acceptable ("one redundant write and nothing else: stores are idempotent").

Recording the pending aligned and applying it in connector_completion on success would delete _rollback_save_watermark, _release_save_watermark, _watermark_sid, the current <= previous guard, the cross-operation clobber reasoning, and the ordering contract against save_finished.

Decisively, it inverts the default. Today a save that reports nothing leaves a permanent prefix hole, so the invariant "every terminal path must report" has to hold forever across six exits in a 2,720-line file — which is why a _guard backstop was needed on day one. Under advance-on-completion, a missing verdict costs one redundant chunk write.

12. Test gaps beyond the shape mismatch in #3

  • test_page_verdict_survives_a_sidecar_failure_after_coverage_is_visible (:584) sets _slot_store.put_result = False, but with slot_snapshot=None the save raises "SLOT snapshot was not acquired successfully" into slot_preparation_error, which is re-raised right after the probe and before slot_store.put. put_result is dead code, put_calls stays empty, and the post-probe SLOT-put failure the test is named for never executes.
  • _verdicts (:533) does by_channel[c.channel][c.operation_id] = c.succeeded over a set, so a contradictory True+False pair for one id collapses last-write-wins and flakes instead of failing. That helper should be the detector for the _guard double-report in update readme for deepseek #6, not the thing that hides it.
  • All four _do_save_req tests pass slot_snapshot=None, so the composite happy path (PAGE True + SLOT True) and the staging-cleanup finally — the one place sidecar_published is downgraded after page_stored latched True — are never covered.

13. Reuse and conventions

  • Four hand-rolled drain blocks where the base class owns the mechanism. OffloadWorkerMixin.__init__ creates self._connector_completions (_offload_common.py:140) and its get_finished drains it (:250); DenseOffloadConnector._record_store_terminal uses exactly that, and kimi_k3 emits straight into out.connector_completions. DSV4 is the only connector that ignores the inherited accumulator, and this PR adds a third per-channel set pair plus a third copy of the drain/clear/generator triple — so anyone following house style and writing self._connector_completions.add(...) in DSV4 gets a silently dropped completion and a wedged request.
  • DSV4_PAGE_SAVE_CHANNEL is the same concept as DENSE_PAGE_STORE_CHANNEL = "dense.page.store" (dense:65), with the same per-SaveOperationId semantics and the same _cancel_save_statistics-on-failure handling, under a name that follows neither existing convention.
  • _watermark_sid (:2488) re-implements completion_req_key in atom/kv_transfer/disaggregation/types.py:477 — byte-for-byte str(getattr(completion, "req_id", completion)), already used at six call sites — under a name that promises a watermark and returns a request id.
  • _release_save_watermark (:2502) is a one-line pass-through to _forget_save_watermark with one caller and no added behaviour, and "release" already means "free a held resource" in this module (release_stalled_save, admission.release).
  • Explanatory-only comments. The 6-line rationale on a one-line constant (:93-98), the 5-line field comment (:233-237), the 8-line block at :1602-1609 and the 11-line _rollback_save_watermark docstring all restate the same incident post-mortem. Per CLAUDE.md, keep the two that actually constrain future edits — the page_stored placement note at :1624-1627 and the ordering note at :2601 — and cut the rest.
  • Docs-match-code. get_finished's header at :1776-1779 enumerates the completion states it emits and the diff adds a whole new channel drained by that exact method without touching the list. The module docstring's "completions are polled in get_finished" bullet is likewise unupdated, and no docs/ file covers this connector, so these comments are the only documentation.
  • Typing. self._done_page_save: set = set() / self._failed_page_save: set = set() (:237-238) versus self._done_sidecar_save: set[SaveCompletionId] = set() nine lines below. Bare set is set[Any], so a type checker cannot catch a bare req_id being mixed in — which is exactly the confusion behind update the link to the repo #3.
  • The file is now 2,720 lines (2,543 before), 3.4x the 800-line soft ceiling, with no extraction and no recorded exception.

@ZhangLirong-amd
ZhangLirong-amd merged commit ae51a34 into main Sep 14, 2026
71 of 80 checks passed
@ZhangLirong-amd
ZhangLirong-amd deleted the zhanglirong/dsv4-page-save-failure-channel branch September 14, 2026 05:16
sajandhy pushed a commit to sajandhy/ATOM that referenced this pull request Sep 17, 2026
…k back (ROCm#2204)

The scheduler advances a sequence's saved watermark where the save is
*emitted* -- before admission, before execution -- and reads it one step later
as SaveSpec.skip_leading_tokens. The advance is therefore a prediction, and
nothing ever checked it. Save admission is a non-blocking BoundedSemaphore, so
a save rejected on max_pending_saves is dropped and never retried while the
watermark has already moved past it; a save that dies inside the executor is
worse still, because _do_save_req swallows its own exceptions and reports the
operation as done. Either way every later incremental save skips the range,
leaving a permanent hole in the persisted prefix.

That single hole explains the whole 256C+DPA failure: _page_boundary_visible
waits for page_hit >= boundary_tokens, which a hole caps below the boundary
forever, so SLOT publication times out repeatedly and no prefix is ever
complete, so CPU readback stays zero. The max_pending_saves rejections are the
trigger, not the bug -- rejecting a save is fine, keeping the advance is not.
This is why more CPU capacity cannot help: capacity only changes how often the
semaphore saturates, and one saturation poisons that sequence for good, which
is also why high concurrency shows it and low concurrency does not.

The rollback could not be written until the scheduler could tell a save that
landed from one that did not, and it could not: finished_saving is a single
undifferentiated set, and both outcomes arrive through it identically. So give
PAGE saves their own completion channel, next to the SLOT sidecar's. It is
TP-aggregated and failure-dominant, exactly right here -- if any rank failed to
persist its shard the range is not durable. Every terminal path reports a
verdict: rejected before admission, rejected at submission, and out of the
executor, with a backstop in _guard, because an operation that reports nothing
would sit in the aggregator forever. process_completions dispatches connector
completions before finished_saving, so the watermark is corrected while the
operation is still inflight and the very next build_connector_meta re-emits the
range.

Rolling back past a later save that did land costs one redundant write and
nothing else: stores are idempotent, and a prefix has to be contiguous, so
everything beyond the hole was unusable anyway. Bytes that never landed are
cancelled rather than counted, so a drop no longer inflates total_saved_tokens.

The hand-rolled __new__ fixtures mirror __init__ field by field, so they carry
the two new sets.

Co-authored-by: ZhangLirong-amd <ZhangLirong-amd@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants