Repository navigation
offload/dsv4: give PAGE saves a failure channel and roll the watermark back - #2204
Conversation
🏷️ CI GuideRuns automatically on every eligible PR before approval:
Heavy model tests:
|
b74615c to
0707c77
Compare
There was a problem hiding this comment.
🟡 Changes recommended
A PAGE failure after request completion may not be re-emitted, leaving the persisted prefix incomplete.
Get a fresh assessment by requesting another Copilot review.
Pull request overview
Adds failure-aware PAGE save reporting for DSV4 and rolls back speculative watermarks when saves fail.
Changes:
- Adds TP-aggregated PAGE completion reporting.
- Rolls back failed save watermarks and statistics.
- Extends scheduler, worker, and fixture tests.
File summaries
| File | Summary |
|---|---|
tests/test_lmcache_offload_v4_page_slot.py |
Initializes PAGE completion state in fixtures. |
tests/test_lmcache_offload_connector.py |
Tests rollback, cleanup, and terminal reporting; some PAGE failure paths lack direct assertions. |
atom/kv_transfer/offload/hybrid/dsv4/connector.py |
Implements PAGE completion handling and watermark rollback; a post-FINISHED failure can still release the retry tracker and leave a prefix hole. |
Review details
Suppressed comments (2)
atom/kv_transfer/offload/hybrid/dsv4/connector.py:1693
- The executor-side PAGE outcome is not covered:
_do_save_reqderives this verdict frompage_storedafter_engine.storeand reports it here, but the new worker test only exercises_finish_unadmitted_save; the existing exception tests usesave_spec=Noneand therefore cannot catch a regression in the PAGE channel. Add a PAGE request test that drives_run_save_reqwithstoreraising and assertsget_finished().connector_completionscontains the failed operation (plus the successful store case).
self._report_page_save_locked(req, succeeded=page_stored)
tests/test_lmcache_offload_connector.py:2651
- This test does not cover all of the new PAGE terminal paths despite its name: it exercises only pre-admission rejection and a SLOT-only request. The submission-rejection and executor-exception cases still use existing tests that assert only the checkpoint channel (or use
page=False), so a regression that stops emittingDSV4_PAGE_SAVE_CHANNELfor those paths would pass. Add assertions for a PAGE request in both_finish_rejected_saveand_do_save_req/_guardfailure cases.
sched._save_inflight.clear()
- Files reviewed: 3/3 changed files
- Comments generated: 1
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
There was a problem hiding this comment.
🟡 Changes recommended
PAGE visibility failures may still commit watermarks, and key completion paths lack regression coverage.
Get a fresh assessment by requesting another Copilot review.
Review details
Suppressed comments (3)
atom/kv_transfer/offload/hybrid/dsv4/connector.py:1798
- The new PAGE channel conversion is not exercised end-to-end: the worker test only inspects
_failed_page_save, while the scheduler test constructsConnectorCompletiondirectly. Existing page-store failure tests also do not assert this output, so an omission or wrongsucceededvalue here would leave the rollback path apparently green. Add a focusedget_finished()/TP-aggregation assertion for both PAGE success and failure (including failure dominance).
connector_completions.update(
ConnectorCompletion(
channel=DSV4_PAGE_SAVE_CHANNEL,
operation_id=completion_id,
succeeded=True,
)
for completion_id in dps
atom/kv_transfer/offload/hybrid/dsv4/connector.py:1599
- Please add a regression for the executor-internal PAGE failure, not only admission rejection. The new
page_storedflag is set around_engine.store, but the current tests do not run_do_save_reqwithstore()raising and then verify thatget_finished()emits aDSV4_PAGE_SAVE_CHANNELfailure. This is the path that prevents an executor failure from leaving the watermark advanced, so a regression here would recreate the permanent hole while the existing tests still pass.
page_stored = True
tests/test_lmcache_offload_connector.py:2657
- [verified] This test only calls
_finish_unadmitted_save()and inspects the private sets; it never callsget_finished()to verify thatDSV4_PAGE_SAVE_CHANNELis emitted, nor exercises_finish_rejected_save()or_do_save_req()whenstore()fails. A regression in the submission/executor paths or in channel draining would therefore still pass while the TP aggregator receives no PAGE verdict. Author must add assertions for the emitted PAGE completion and cover the remaining terminal paths.
def test_worker_reports_a_page_verdict_on_every_terminal_path():
"""A missing report strands the operation in the TP aggregator forever."""
- Files reviewed: 3/3 changed files
- Comments generated: 1
- Review effort level: Lite
…k back The scheduler advances a sequence's saved watermark where the save is *emitted* -- before admission, before execution -- and reads it one step later as SaveSpec.skip_leading_tokens. The advance is therefore a prediction, and nothing ever checked it. Save admission is a non-blocking BoundedSemaphore, so a save rejected on max_pending_saves is dropped and never retried while the watermark has already moved past it; a save that dies inside the executor is worse still, because _do_save_req swallows its own exceptions and reports the operation as done. Either way every later incremental save skips the range, leaving a permanent hole in the persisted prefix. That single hole explains the whole 256C+DPA failure: _page_boundary_visible waits for page_hit >= boundary_tokens, which a hole caps below the boundary forever, so SLOT publication times out repeatedly and no prefix is ever complete, so CPU readback stays zero. The max_pending_saves rejections are the trigger, not the bug -- rejecting a save is fine, keeping the advance is not. This is why more CPU capacity cannot help: capacity only changes how often the semaphore saturates, and one saturation poisons that sequence for good, which is also why high concurrency shows it and low concurrency does not. The rollback could not be written until the scheduler could tell a save that landed from one that did not, and it could not: finished_saving is a single undifferentiated set, and both outcomes arrive through it identically. So give PAGE saves their own completion channel, next to the SLOT sidecar's. It is TP-aggregated and failure-dominant, exactly right here -- if any rank failed to persist its shard the range is not durable. Every terminal path reports a verdict: rejected before admission, rejected at submission, and out of the executor, with a backstop in _guard, because an operation that reports nothing would sit in the aggregator forever. process_completions dispatches connector completions before finished_saving, so the watermark is corrected while the operation is still inflight and the very next build_connector_meta re-emits the range. Rolling back past a later save that did land costs one redundant write and nothing else: stores are idempotent, and a prefix has to be contiguous, so everything beyond the hole was unusable anyway. Bytes that never landed are cancelled rather than counted, so a drop no longer inflates total_saved_tokens. The hand-rolled __new__ fixtures mirror __init__ field by field, so they carry the two new sets.
0707c77 to
e898295
Compare
There was a problem hiding this comment.
🟡 Changes recommended
Exclude non-saving PAGE specs from successful verdicts and add coverage for _guard executor failures.
Get a fresh assessment by requesting another Copilot review.
Review details
- Files reviewed: 3/3 changed files
- Comments generated: 2
- Review effort level: Lite
| if getattr(req, "save_spec", None) is None: | ||
| return |
| if kind == "save": | ||
| # Backstop. ``_do_save_req`` normally reports its own PAGE | ||
| # verdict and swallows its errors, so this only fires if it | ||
| # died outside that handling -- but a save that reports | ||
| # nothing would sit in the TP aggregator forever, so the |
|
Reviewed at Three claims in the PR body check out and are not flagged below: The new tests also have real revert power, which is worth recording so nobody re-checks: neutering Two findings below I would treat as blocking. The rest are ordered after them. 1.
|
…k back (ROCm#2204) The scheduler advances a sequence's saved watermark where the save is *emitted* -- before admission, before execution -- and reads it one step later as SaveSpec.skip_leading_tokens. The advance is therefore a prediction, and nothing ever checked it. Save admission is a non-blocking BoundedSemaphore, so a save rejected on max_pending_saves is dropped and never retried while the watermark has already moved past it; a save that dies inside the executor is worse still, because _do_save_req swallows its own exceptions and reports the operation as done. Either way every later incremental save skips the range, leaving a permanent hole in the persisted prefix. That single hole explains the whole 256C+DPA failure: _page_boundary_visible waits for page_hit >= boundary_tokens, which a hole caps below the boundary forever, so SLOT publication times out repeatedly and no prefix is ever complete, so CPU readback stays zero. The max_pending_saves rejections are the trigger, not the bug -- rejecting a save is fine, keeping the advance is not. This is why more CPU capacity cannot help: capacity only changes how often the semaphore saturates, and one saturation poisons that sequence for good, which is also why high concurrency shows it and low concurrency does not. The rollback could not be written until the scheduler could tell a save that landed from one that did not, and it could not: finished_saving is a single undifferentiated set, and both outcomes arrive through it identically. So give PAGE saves their own completion channel, next to the SLOT sidecar's. It is TP-aggregated and failure-dominant, exactly right here -- if any rank failed to persist its shard the range is not durable. Every terminal path reports a verdict: rejected before admission, rejected at submission, and out of the executor, with a backstop in _guard, because an operation that reports nothing would sit in the aggregator forever. process_completions dispatches connector completions before finished_saving, so the watermark is corrected while the operation is still inflight and the very next build_connector_meta re-emits the range. Rolling back past a later save that did land costs one redundant write and nothing else: stores are idempotent, and a prefix has to be contiguous, so everything beyond the hole was unusable anyway. Bytes that never landed are cancelled rather than counted, so a drop no longer inflates total_saved_tokens. The hand-rolled __new__ fixtures mirror __init__ field by field, so they carry the two new sets. Co-authored-by: ZhangLirong-amd <ZhangLirong-amd@users.noreply.github.com>
Problem
DSV4OffloadScheduleradvances a sequence's saved watermark where the save isemitted — before admission, before execution — and reads it one step later as
SaveSpec.skip_leading_tokens. The advance is a prediction, and nothing checked it.Save admission is a non-blocking
BoundedSemaphore, so a save rejected onmax_pending_savesis dropped and never retried while the watermark has alreadymoved past it. A save that dies inside the executor is worse:
_do_save_reqcatches its own exceptions and does not re-raise, so the operation is reported as
success. Either way every later incremental save skips that range, leaving a
permanent hole in the persisted prefix. Nothing on the save side ever lowers
entry[1]— the only lowering lives inload_failed.One hole is enough to disable the sequence.
_page_boundary_visiblerequireslookup(tokens[:boundary]) >= boundary, andlookupreturns a contiguousprefix length, so a gap anywhere at or before the boundary truncates the hit
there and SLOT publication times out for the rest of that sequence's life.
This is why raising CPU capacity cannot help: capacity only changes how often the
queue saturates, and one saturation poisons that sequence permanently. Higher
concurrency saturates more often, which is why 256-concurrency failed and low
concurrency did not.
Why the rollback needed new plumbing first
The scheduler could not tell a save that landed from one that did not.
get_finishedreports PAGE saves only throughfinished_saving— oneundifferentiated set — and
save_finishedtakes no success flag. Theconnector_completionschannel, which does carrysucceededand is TPaggregated, was wired only for the SLOT sidecar. PAGE had no failure channel at
all, so there was nothing a rollback could hang off.
Change
Give PAGE saves their own completion channel, alongside the sidecar's.
Worker —
DSV4_PAGE_SAVE_CHANNEL, reusing the existing generic completionmechanism: channel dispatched, TP aggregated, failure dominant (if any rank did
not persist its shard, the range is not durable). No shared structures change. A
verdict is reported on every terminal path — rejection before admission,
rejection at submission, and out of the executor via a
page_storedflag — plusa backstop in
_guard, because an operation that reports nothing would sit inthe TP aggregator forever.
Scheduler — record what each advance was contingent on, keyed by save
operation.
connector_completionreleases the record on success and, on failure,rolls the watermark back and cancels the save statistics so bytes that never
landed do not inflate
total_saved_tokens.Ordering works because
process_completionsappliesconnector_completionsbefore
finished_saving: the watermark is corrected while the operation is stillin
_save_inflight, and the nextbuild_connector_metare-emits the range.Rolling back past a later save that did land costs one redundant write and
nothing else — stores are idempotent, and a prefix has to be contiguous anyway.
Verification
Measured on DeepSeek-V4-Pro, PD-disaggregated, concurrency 256 with 8 x 128 GiB
LocalCPUBackend, against the same configuration on the unfixed code:page_visibility_timeoutThe failure mode is worth naming precisely, because it is easy to miss: the
unfixed run did not error and did not crash. Client-side
errorsstayedat 0 and every request returned normally. What happened is that each SLOT save
waits on
_page_boundary_visible, a hole makes that wait unsatisfiable, and thesave burns its full timeout before giving up — so the run simply never finished
warmup. It reached 1,293 of 2,845 warmup requests in 2,250 s, work the fixed
build does in 1,800 s, and was stopped by hand. The only outward signal is 1,874
timeout lines in the server log.
(Both builds log a handful of
KV transfer drain during shutdown failedtracebacks at teardown; those are unrelated to this change and present either
way.)
In one 3,600 s run the fix rolled back 6,723 advances covering 29,989,120
tokens across 6,183 sequences — all of which the unfixed code would have
declared saved and never written. 1,148 of those holes sat at offset 0, where the
prefix is worth nothing at all; the rest had a median offset of 79,616 tokens.
Rollback tracks rejection 1:1 and is a load-step transient, not a runaway: the
rate spikes on each concurrency ramp and decays within two minutes.
PAGE save did not landstayed at 0, so the executor-internal path is coveredbut not yet exercised.
Tests
556 passed, 1 skipped across the
kv_transfersuites. The new rollback test isnot vacuous — neutering
_rollback_save_watermarkmakes it fail withassert 8 == 0, which is exactly the hole.test_failed_page_save_rolls_the_watermark_back_and_re_emitstest_page_watermark_records_do_not_outlive_their_requesttest_worker_reports_a_page_verdict_on_every_terminal_pathThe hand-rolled
__new__fixtures mirror__init__field by field, so theycarry the two new sets.
Not covered here
SLOT sidecar saves still fail at roughly 15-25% under load, and are deliberately
not retried. That is a separate issue:
OFFLOAD_SLOT_STAGING_SLOTSdefaults to 1,so each rank has a single staging row contended by saves and by loads that hold it
for a whole batch. Unlike the PAGE hole this costs cache hit rate, not
correctness — a missing sidecar is a miss on a later load, not a corrupt prefix.