Skip to content

[HiCache] Wait for load-back before deferred Mamba COW - #36743

Open
jpezzulli wants to merge 1 commit into
sgl-project:mainfrom
jpezzulli:john/upstream-mamba-cow-load-order
Open

jpezzulli wants to merge 1 commit into
sgl-project:mainfrom
jpezzulli:john/upstream-mamba-cow-load-order

Conversation

@jpezzulli

@jpezzulli jpezzulli commented Aug 27, 2026 •

Copy link
Copy Markdown

Motivation

HiCache restores Mamba state layer by layer on its H2D transfer stream. A
restored radix slot can be selected as the source of deferred Mamba copy-on-
write before that layerwise transfer has finished. Deferred COW consumes the
entire source slot at once, before the model's existing per-layer waits can
order those reads, so it can copy a partially restored state into the request-
owned destination.

Modifications

  • Add HybridReqToTokenPool.wait_for_hicache_load_complete(), which waits for
    the final event in the active layer transfer counter and is a no-op when no
    HiCache load is active.
  • Call that fence immediately before deferred whole-slot Mamba COW. The
    clear-only path and ordinary per-layer Mamba reads are unchanged.
  • Add focused tests for both the final-layer fence and wait-before-copy
    ordering.

This is complementary to #36738:

The two changes protect opposite sides of the load and can merge independently.

Accuracy Tests

The two focused regressions fail on dc10483592:

  • the final-load fence method is absent;
  • deferred COW copies before waiting.

They pass with this change:

2 passed, 10 deselected

The complete registered Mamba unit file also passes:

12 passed

Speed Tests and Profiling

No benchmark was run. The wait is reached only for deferred whole-slot Mamba
COW while a HiCache layer transfer counter is active. It adds no work when
there is no active load; when there is one, waiting for the final transfer is
the required ordering.

Checklist


CI States

Latest PR Test (Base): ❌ Run #33125188060
Latest PR Test (Extra): ❌ Run #33125187898
Latest PR Test (AMD ROCm 7.2): ❌ Run #33125188020

Signed-off-by: John Pezzulli <38448408+jpezzulli@users.noreply.github.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants