Repository navigation
[Fix] hicache: Order write-back host copies after the forward that produced the KV - #44800
Merged
Merged
Conversation
metamergebot
requested review from
Jialin,
Ying1123,
alphabetc1,
hanming-lu,
hnyls2002,
huangtingwei9988,
hzh0425,
ispobock,
merrymercy,
xiezhq-hermann and
yizhang2077
as code owners
October 16, 2026 21:52
This comment has been minimized.
This comment has been minimized.
Collaborator
Author
|
/rerun-test test/registered/unit/mem_cache/test_unified_post_capture_sizing.py |
Contributor
|
🚀 |
…d the KV Under `--hicache-write-policy write_back`, two paths copied KV pages to host before the forward that wrote them had finished, so a later load-back could serve stale KV to another request: - Insert-time SWA window backups. Write-back defers Full and Mamba backups to eviction, but the tree still published an unbacked SWA window when a request's nodes were inserted. That copy was queued while the request's own forward could still be writing the window. - The blocking write-back drain. Eviction submits everything queued in the tick on the device-to-host stream with no dependency on the forward stream, and an evicted SWA page can belong to a checkpoint whose producer has not completed. Stop publishing SWA windows at insert time under write-back, in both the Python and the Rust tree cores, and give the blocking drain a dependency on the scheduler's forward stream, the same stream the load-back H2D path already fences on (`load_fence_stream`). Only a non-empty blocking drain waits; write-through and the asynchronous write-back flush are unchanged. Co-authored-by: xiezhq-hermann <xiezhq-hermann@users.noreply.github.com>
…backups Write-back insertion no longer publishes the SWA window backup; it is deferred to eviction. These four tests covered the removed behavior.
metamergebot
force-pushed
the
oss-sync-015db9f41c
branch
from
October 17, 2026 03:35
723bb63 to
3a845c1
Compare
Collaborator
Author
|
This port clears our bar on gate 23✓✗1 lint✓ build 7✓✗1✂3 A✓ B✓ C 0/15⏳ other 8✓✗1 · accel ✗amd,npu,xpu @fei-xx @merrymercy @xiezhq-hermann — could one of you take a look and merge it if you are happy? Nothing else is waiting on our side. |
This was referenced Oct 17, 2026
detain
pushed a commit
to detain/sglang
that referenced
this pull request
Oct 19, 2026
…oduced the KV (sgl-project#44800) Co-authored-by: metamergebot <metamergebot@users.noreply.github.com> Co-authored-by: xiezhq-hermann <xiezhq-hermann@users.noreply.github.com> (cherry picked from commit f75232c) (cherry picked from commit cb8cbe7902c86f7f23c66fd0500331b07ab6732d)
detain
added a commit
to detain/sglang
that referenced
this pull request
Oct 19, 2026
… test on a visible device `test_queued_full_backed_parent_owns_its_ack_ahead_of_a_queued_child` arrived with 7e9dda1 (commit in 323c1c07eb) without a device guard, so it added ONE new failure node to test_unified_radix_cache_unittest.py (1300F -> 1301F at CVD=9): `_build()` creates a real cache/host pool, which raises `RuntimeError: No accelerator ... or platform plugin is available`. Uses the file's own in-class convention — its 4 fixture-based siblings already carry `@unittest.skipUnless(torch.cuda.is_available(), "cache fixtures need CUDA")` (added in stack entry 199's test fixes). Node-list check of the whole file after the gate: identical to the base at 1275 unique FAILED nodes, with the 2 expected renames from sgl-project#44800 (`test_write_back_swa_publish_leaves_an_ancestor_pending_under_another_ack_alone` -> `test_write_back_defers_swa_window_backup_until_eviction`, x2 parametrizations). (cherry picked from commit f2e627253a604823a56839aeb76b3877ae63d8ec)
4 of 5 tasks
jumanzii
pushed a commit
to fractalyze/sglang
that referenced
this pull request
Oct 19, 2026
Resolve against sgl-project#44800: start_writing fences every device-to-host copy on write_fence_stream, so the wait_for_forward flag and the separate load_fence_stream wait are no longer needed.
This was referenced Oct 21, 2026
detain
added a commit
to detain/sglang
that referenced
this pull request
Oct 29, 2026
…20 commits since 12dc623 @ merge-sgl-project#4) Conflicts: 167 files (149 content / 14 add/add / 4 modify/delete) at the settled 309 tip d28e7cd -- dossier predicted 146 +/-25; the +21 shift is entries 275-309 landing qsa-ring/hicache-ctrl/gdn/moe/llguard deltas into the conflict zone. By class: A 13 (auto-takes-upstream, dedupe-verified), B 80, C 56, D 18. 102 TODO(merge5) re-plant/fold markers planted: B1 x15, B2 x29, B3 x12, B4 x23, B5 x23. Re-plant stock: 117 files under /tmp/opencode/merge5-dossier/replant/ (76 fork-hunks patches, 19 upstream-side patches, 14 AA fork copies, 8 preremove snapshots) -- sessions 2/3 re-plant from real code, not memory. D-list outcomes (must-not-auto-resolve, handled personally): - sgl-project#45289-vs-sgl-project#44858 (entry 301): VERIFIED, not re-resolved -- the gate/up-order final form is in origin/main (5b1dbc2 in range); scales test taken as upstream oracle, fork 191-line version saved to replant/ for B5 diffing. sgl-project#44858's call-site shape CANNOT survive (our base never had it). - entry-230 dispatcher vs sgl-project#44993/sgl-project#44994: took upstream structure in token_dispatcher/standard.py; published-value CONTRACT lives upstream via get_moe_runner_backend() in the LoRA-EP guard; fork's warn-once mismatch diagnostic re-plants (B5 marker). UNION GATE: standard_dispatcher 16P (was 12P, suite grew), model_overrides 121P/1S/210sub (was 112P/1S/181sub), fold probe 6P/2F (upstream added a runner_backend arg to the predicate -- B5 re-pins the probe). - entry-219 gauge names vs sgl-project#43727 return + sgl-project#44968: PER-POOL NAMES KEPT as the serving schema -- hicache_host_pool_{used,total}_tokens{pool} dataclass fields, Gauge registrations and labels(pool=) emission re-restored over the #4697x structure; hicache_host_pool_evicted_tokens_total already survived auto-merge; producer dict restored by union in metrics_reporter. ANTI-NEEDLE SATISFIED: old mamba-only gauges are NOT the serving schema. - entry-221 eager-direct add/add: DEDUPED to upstream (bodies identical except est_time 15->9); MEASUREMENT DEBT RECORDED: prefill t/s must be re-measured post-B4 before trusting the est_time. - entry-168 D2H fence vs sgl-project#44800: the fence is ALREADY UPSTREAM (both load_fence_stream + write_fence_stream with the load-thread-corruption comment in origin/main's cache_controller.py) -- nothing re-planted; the two real regions resolved as import-union + comment wording (merge sgl-project#4 stale-tail pattern avoided). Fork hunks still saved to replant/ (belt). - entry-209 clamp + 216/218 ordering + 288-295 hicache-ctrl-era vs #46970x KV rewrite: ADOPTED-UPSTREAM across unified_radix_cache/hicache_storage/ hybrid_cache_controller/memory_pool/kv_cache_builder; EVERY fork hunk extracted to replant/*.fork-hunks.patch; B3 re-plant markers planted. sgl-project#45703's short-full-match seam arrives with upstream. - sgl-project#45090 SGLANG_ENABLE_UNIFIED_RADIX_TREE: REMOVAL TAKEN (fork had already moved it to _DeprecatedEnv; now gone). grep: ZERO .py readers repo-wide; 2 upstream-owned docs jsx snippets still export it (deepseek-v4, glm-5.2 -- upstream's own debt). OPERATOR-OWED CHECK: confirm no /root harness exports it (unreadable from this seat). - sgl-project#39727 unified L3 key: VERIFIED OPT-IN -- merged hicache_storage.py keeps the DEFAULT legacy suffix composition _<model>_<tp_rank>_<tp_size> (content-anchor: HiCacheFile._init_config_suffix, raise-guarded when _unified_chunk_spans is set). ANTI-NEEDLE: no silent key-scheme switch; warm /root/hicache_store contents stay addressable. Modify/delete (4, all victims of upstream sgl-project#45672 test prune + URC split): - test_unified_radix_cache_unittest.py: REMOVED (adopted split) -- its bare unified_tree_core_inspector imports are unreachable post-sgl-project#45236 re-home; fork source + hunks saved; B2 marker planted in the new core test. - test_fused_moe_triton_config.py: REMOVED -- imports common_utils, absent from both sides post-sgl-project#45672; the 2 live entry-3/235 tests re-plant in B2. - test_scheduler_batch_is_full_latch.py: REMOVED -- fork delta was 3 stub-attr lines documenting sgl-project#46643's absorbed arms; upstream deletion correct. - test_qsa_kv_pool.py: KEPT -- fork's 147 added lines are entry-187/200 sharding + 285-287 QSA-ring tests with NO upstream replacement; qsa_kv_pool.py sides converged byte-identical; collects clean. B2 marker. Cascading silent-auto-merge repairs (classes not in the conflict set but broken by auto-merge + union splices -- all found by gates, now fixed): kernels/ops/activation/__init__.py (dead _CUDA registration dup), kernels/ops/ diffusion/__init__.py (5-tuple _SPECS entry), kernels ops embeddings/moe/ attention/mamba __init__ (KernelSpec fields removed upstream: description/ format_signature stripped, CapabilityRequirement->CUDA_ONLY), base_runner/ eager_runner cp_extend + shared_read_event imports retargeted per upstream's "no imports in runner_utils/__init__" contract, and 4 fork helpers grafted back onto their take-upstream homes because merged code/tests still import them (memory_pool.spec_intermediate_bytes_per_req, kv_cache_builder. _adjust_hicache_size_for_draft + _device_pool_size_per_token, hybrid_cache_controller.local_restorable_mask, validation_hook. validate_replayssm_spec_algorithm, arg_groups/moe_hook b12x validator). flashinfer_cutlass.py flipped to FORK-OWNED (b12x route is fork-only, 760 vs 95 lines; upstream sgl-project#44993 hunks = B5 fold patch saved). Silent-semantics notes: sgl-project#46127/sgl-project#44912 riders arrived via per-PR picks (union); sgl-project#45520 (entry 302) is STILL OPEN upstream -- entry-230 fail-safe layer stays as entry 303 records; if a future sync brings it, re-apply the fail-safe reads. qwen4_exp holder-contract fused (ctor takes optional batch). Gates at commit: import sglang OK (CVD=9, prod venv); check_registered_tests/ check_no_bare_pytest_main/check_no_registered_tests_in_package rc=0; ruff F821 +E9+format clean on the resolution set; collect-only over unit/{mem_cache, model_executor,distributed,layers/moe} = 4337 nodes, 3 ERRORS: 2x cargo/torch- sys libtorch probe (REPRODUCED IDENTICALLY AT THE PRE-MERGE TIP -- env issue, not merge damage) + test_hicache_async_ready_sync.py (entry-253-era attribute gone with the URC rewrite -- EXPECTED pre-B3). sessions 2/3 owed: B2 test re-home, B3 mem_cache+rust re-plant (+hicache-ctrl re-ports), B4 scheduler/admission, B5 moe/quant/dispatcher, B6 environ+C0a+ CAVEATS re-pin. PROD BOOT FORBIDDEN until B3 completes (209 clamp + 218 ordering re-plant).
detain
added a commit
to detain/sglang
that referenced
this pull request
Oct 29, 2026
…),199,217,218,219,253,290d,293,294 Re-plants the fork-hunks patch (1013 lines, the deepest file) onto the #46970x-rewritten URC. B3 marker consumed in unified_radix_cache.py. What the merged tree had ALREADY absorbed (deduped, not duplicated — per-hunk verdicts recorded): the whole deadline/stop-policy family (_StoragePrefetchWait msgspec/PollTimer, _start_storage_prefetch_wait, _prefetch_deadline_expired REPLACING _prefetch_timeout_check_linear_func, _mark_prefetch_deadline_expired, the reason="deadline" bookkeeping, belief-heal suppression on deadline, the _invalidate_absent_from_hit_query move to the drain head + KV-verdict guard + deadline gate, the release_aborted_request / finish_storage_prefetch_admission wait pops, and the _should_backup_after_insert write-back half of sgl-project#44800 (unified_tree_core.py:1447). The sgl-project#45703 short-full-match seam did NOT obsolete entry 209's clamp (it lives controller-side; B3-2 gate 17P). Re-planted here: - entry 294 (sgl-project#45795): the FULL deferred ready-count sync family (_deferred_ready_sync_group gate ladder incl. the runtime-attach fallback, _check_hicache_events_deferred one-op-per-step on its own gloo group, _consume_deferred_ready_sync with the digest MIN/-MAX assert, _deferred_consume_count, _drop_deferred_ready_sync) + the check_hicache_events branch + the reset() settle. This CLEARS the test_hicache_async_ready_sync collection error (the B3-caused 1err in the merge gates). - entries 199+218: the batched write-through backup family VERBATIM from the patch (_NodeBackupSpec, _queue_write_through_backup, _drain_queued_backups, _ancestors_first incl. the sgl-project#40363 delta walked-set, _is_unbacked_node, _parent_backed_up, _release_queued_backup, _backup_queued_batch, _build_node_backup_spec, _backup_specs, _merge_backup_transfers, the node_ids= ack-list through _execute_kv_backup into the controller write (B3-2), the flush_pending_backups guard, and the writing_check(write_back) drain + start_writing(wait_for_forward=True) — the ONE-fence ordering rule). _batched_backup reads SGLANG_ENABLE_HICACHE_BATCHED_BACKUP, DEFAULT FALSE (upstream's 0ab8654 default-ON stays NOT taken, entry-218's record). - entry 293 (sgl-project#46451): the opt-in host-capacity WAIT: fields + init_hicache reads with the nonnegative-finite ValueError, wait_full, the retry cadence (host_capacity_deadline/next_retry/retries — fields planted on the operation in B3-2), allow_partial=not wait_full, pending_storage_hits park/drain with identity snapshots, the info.operation-is-not-operation late-query guards, and the revoke/alloc-already-owned guards. DEFAULT 0.0 = byte-identical. - entry 290(d): _alloc_prefetch_aux_staging now sizes to alloc_len, the endpoint the KV span ACTUALLY got (mamba-host-full prod boots stop declining prefetches a smaller endpoint would fit). - entry 219 producer: _record_host_pool_evictions on evict_host (before _apply_eviction_result) feeding hicache_host_pool_evicted_tokens_total{pool}. - entry 292 (sgl-project#43110): _build_sidecar_transfers resolves derived pools through merge_source_transfers (every destination, not matching_sources[0]; the two AssertionErrors it subsumes are structurally guaranteed by the resolver). - entry 30 rider: load_back unwinds (drop both locks, report miss) when any component PREPARED-REJECTS (the sgl-project#39830 poison/degrade contract the mamba component sets -- component side rides B3-5's mamba.py cluster). - entry 253: the per-call-site file_storage_path fold DELETED here (it moved into _generate_storage_config, every attach path, B3-1-verified). - test_chunked_publish_release_disjointness.py (entry 307's tripwire): re-anchored CacheInitParams(tree_components=...) -> tree_slots=tree_slots_of(...) per the upstream slots refactor; the tree_core_backend='rust' SUBFAILED line is the documented cargo/torch-sys ENV class (fails identically at base). ACTUALS (CVD=9, prod venv, PYTHONPATH=python, per-file, damage -> after): test_hicache_async_ready_sync 1err -> 7P/0F ** B3-caused error CLEARED ** test_hybrid_storage_sparse_allocation 12P/1F -> 13P/0F/14sub ** entry-290(d) closed ** test_storage_prefetch_lifecycle -> 16P/0F/13sub test_hicache_trailing_endpoint (gate) -> 17P/0F (unmoved by this cluster; controller-side B3-2) test_buffer_mode_sidecar -> 6P/0F/6sub test_chunked_publish_release_disjointness 4P/2F -> 4P+1sub/1F (the F now the rust-cargo ENV class; the python-backend half RE-ANCHORED green) test_mamba_path_state_cap -> 5P/5S (skips = honest, env) test_decode_retraction_backup 2P/2F -> 2P/2F (both F = No-CUDA ENV class, byte-identical at base; file is B2's per the resolution log) test_hicache_mamba_sizing -> 30P/0F/17sub (unmoved) URC topic dir (unified_radix_cache/*.py): 1056F/40P/199S -> 1056F/40P/199S UNMOVED — 100% of the errors are 'No accelerator ... available' from the upstream kits' fixtures.py build_fixture calling get_device() (entry 310's "GREEN, honest skips" claim does NOT reproduce at this seat; the battery's own per-file numbers (core 27P/465F etc.) match the damage totals exactly; this is a kit no-CUDA skip DEFECT = B2/kit scope, recorded not fixed here). Lint: ruff format + F821/E9 clean on both files; py_compile + import OK. No GPU executed; no push.
detain
added a commit
to detain/sglang
that referenced
this pull request
Oct 30, 2026
…en at upstream-new InsertParams site (B3-7 rust re-plant fallout, first-compiling seat)
The operator's cargo/maturin-style release build at /home/my/sglang/sglang was
the FIRST build in the merge era to get past torch-sys, and it turned entry
311 / B3-7's "compilation = ENV DEBT" (the CAVEATS + entry-310 unverified-cargo
marker) into three hard, reproducible compile errors. Reproduced at this seat
BEFORE any edit (same 3, same lines):
error[E0599] no method named `parent_node_id` found for struct
`TreeCoreBinding<Core>` -> src/python_bindings.rs:3253 (x2)
error[E0063] missing field `mamba_value_seqlen` in initializer of
`InsertParams<'_, _>` -> src/unified_tree_core.rs:2105
ROOT CAUSE 1 -- parent_node_id: WRONG CFG BLOCK, not a missing method.
The core side is fine and correct (unified_tree_core.rs:4283: arena.resolve ->
node.try_parent() -> parent's external id, Ok(None) on a root -- matching the
interface doc "The parent node id, or None for the root" and its pure-python
twin, and deliberately using try_parent rather than the panicking parent()).
What is wrong is WHERE the binding-side delegate was replanted: B3-7 put
TreeCoreBinding::parent_node_id inside the `#[cfg(feature = "inspection")]`
impl (python_bindings.rs:2314-2324) while the pyclass wrapper that exposes it
to Python (3252, emitted by tree_core_binding!) is UNCONDITIONAL. History
settles which side is authoritative:
* `git log -S parent_node_id -- rust/sglang-radix-tree` shows the method is
FORK-ONLY -- it does NOT exist at the MERGE sgl-project#5 base (ae6ee52) at all, so
there was no upstream rename to track and nothing to "take from a snapshot".
* The fork's own pre-merge copy (2f31a7e, entry-199 era) had it in the
ALWAYS-ON impl (nearest impl header at line 1030; the two cfg(inspection)
markers are far above at 328/482).
* The shipped extension (python/.../rust_tree_core/mem_cache...so, Oct 28)
exposes `parent_node_id` and contains NO `inspect_contains_node`, i.e. a
NON-inspection build that still published the method.
* The consumer is production, not tests: unified_radix_cache.py:4388/4393/4408
(the hash-chain walk) -> rust_tree_core/adapter.py:1082 get_parent_node_id
-> binding.parent_node_id, and it is an abstractmember of the tree-core
INTERFACE (unified_cache/unified_tree_core_interface.py:226) with a pure-
python twin at unified_tree_core.py:583 -- so BOTH backends must serve it,
and the python-side name is get_parent_* while the rust binding is
parent_node_id (the sibling adapter.py inspect_get_parent_node_id kwarg fix
B3-7 made is the other side of this same seam).
FIX: move the delegate into the always-on accessor impl beside its siblings
node_backuped/is_root (1824/1830), with a comment recording why it must not be
gated. Verified equivalently: the fresh non-inspection build again exposes
`parent_node_id` and contains no `inspect_*` name.
ROOT CAUSE 2 -- mamba_value_seqlen: a genuine OMITTED FIELD, semantics read
from the guard it feeds. insert_overlap_step_ (2105) is an upstream-new / B3
re-plant construction site that copies 12 of InsertParams' 13 fields from the
walk state and drops exactly this one. Filling it with `state.mamba_value_seqlen`
is the only self-consistent choice and matches BOTH sibling sites verbatim
(2058 in insert_walk_step_, 2251 in insert_commit_step_; the field is seeded on
the walk state at 1891 from the caller's params). A `None` here would have
COMPILED and SILENTLY DISARMED the entry-196/sgl-project#38815 alignment guard: the
consumer components/mamba.rs:174-183 returns early -- declining to stamp --
whenever params.mamba_value_seqlen is Some(seqlen) and key.atom_len()+
K::IS_BIGRAM != seqlen, which is precisely "a leaf truncated to a shorter
component boundary is not the position the donated state was produced at".
Since this same params carries state.mamba_value, pairing it with a None seqlen
would assert "produced nowhere in particular" about a real donated tensor.
Call-chain honesty note, measured not assumed: at this site the params is passed
ONLY to update_component_on_insert_overlap, which mamba does NOT override (only
swa.rs:436 does, reading prev_prefix_len/swa_evicted_seqlen), and the binding is
not reused later in the function -- so today the omission is compile-breaking and
LATENT-semantically, not yet an observable misstamp. Fixed at the root anyway.
ROOT CAUSE 3 (found by VERIFYING, not by the operator's report) -- the crate's
TEST TARGET did not compile either, so the era's cargo-test pin was never
provable: src/tests/components/swa.rs::
write_back_reinsert_defers_swa_backup_while_write_through_publishes carried a
half-converted replant: an `assert!(... matches!(action, CacheAction::BackupKV(_)))`
on the RETIRED enum architecture (CacheAction is not in scope -- E0433) plus a
STRAY ORPHAN LINE `.any(|action| action.as_any().is::<BackupKV>())` with no head,
between two statements (E0061-class syntax error). The converted expression was
the right one and is the file-wide idiom (used verbatim 3x in
tests/components/mamba.rs and by the filter_map four lines below); the authority
version (a7d333d, sgl-project#44800) proves the test's intent was: write-back branch
asserts NO BackupKV, write-through branch downcasts them and asserts
backups == vec![vec![leaf]]. FIX: assert converted to as_any().is::<BackupKV>()
and the orphan removed -- NO test weakened, none #[ignore]d.
VERIFICATION (this seat can compile, so it was verified, unlike every prior
claim in this era). Build environment is the crate README's documented recipe --
NOTHING installed, NO venv touched:
LIBTORCH_USE_PYTORCH=1 LIBTORCH_BYPASS_VERSION_CHECK=1 \
CXXFLAGS="-std=c++20 -include $PWD/torch_compat.h" \
LD_LIBRARY_PATH=$(python -c 'import torch,pathlib;print(pathlib.Path(torch.__file__).parent)')/lib
(plain `cargo check` without LIBTORCH_USE_PYTORCH dies in torch-sys's build
script with "Cannot find a libtorch install"; the venv's activate exports NO
LIBTORCH/LD_LIBRARY_PATH, so the README env IS the equivalent of the operator's
setup -- recorded because it is the missing ingredient for all future rust debt.)
cargo check -p sglang-radix-tree --features python-extension -> exit 0 (was 3 errors)
cargo check ... --features 'python-extension inspection' -> exit 0 (no duplicate defn)
cargo clippy -p sglang-radix-tree --features python-extension -> clean
cargo fmt --check -> rc 0 (crate is fmt-gated)
cargo test -p sglang-radix-tree --locked (dev profile) -> 944 collected
test result: ok. 943 passed; 0 failed; 1 ignored
(re-run after `touch` to prove the test target really recompiled; the ONE
ignored test is upstream's own pre-merge #[ignore] from a77283f /
sgl-project#37290 -- insert_host_panics_on_a_colliding_page, documented
"unconstructible today ... the assert is defensive-only". Not added here,
not a suppressed failure.)
==> TRUE POST-MERGE RUST PIN: 943P/0F/1ignored, REPLACING the long-quoted
"932P/0F" of the pre-merge era.
OPERATOR'S EXACT COMMAND, RUN VERBATIM, NOW GREEN:
cargo rustc --lib --manifest-path ../rust/sglang-radix-tree/Cargo.toml \
--release --features 'pyo3/extension-module python-extension' --locked \
--crate-type cdylib
-> exit 0; "Finished `release` profile [optimized] target(s) in 55.20s"
(also re-run green from the other CWD the relative manifest path admits: 51.97s)
-> artifact target/release/libmem_cache.so, 3,321,640 bytes, and it embeds
`parent_node_id` while containing NO `inspect_contains_node`.
ALSO ANSWERS THE OPEN PYTHON-SIDE QUESTION: python/sglang/srt/mem_cache/
rust_tree_core/extension.py runs `bindings = load_tree_core_extension()` at
MODULE IMPORT, and rust_extensions/loader.py:402-407 shells out to EXACTLY
`cargo build --release --locked --package sglang-radix-tree --features
python-extension` -- the command named in the mem_cache dir's 2 collection
errors and in the draft_gate 4F. That feature set has NO inspection, i.e. those
python-side failures WERE this crate break, not independent env debt. Re-running
that loader command here: exit 0, "Finished `release` profile ... in 9.65s".
=> The python pins for those suites WILL MOVE on the operator's next full build;
deliberately NOT re-run or re-pinned in this commit (one-line forward note in
CAVEATS entry 319 instead).
Files: 3 source files, +29/-8. No artifact committed, no target/ churn, the
shipped extension binary untouched, nothing installed, NOTHING PUSHED.
Co-reported with the operator's verbatim failure lines.
2 of 3 tasks
detain
added a commit
to detain/sglang
that referenced
this pull request
Nov 11, 2026
…r/URC markers -- 3/18 census consumed, 3 gates in one CASCADE (start_writing(wait_for_forward) TypeError class, 68 distinct nodes): upstream sgl-project#48012's NEW fixture hicache_write_back_fixtures.InstantWriteBackController mirrored the pre-sgl-project#44800 no-kwarg surface while the fork's writing_check drain (entries 199/218: the batched-backup flush=False + ONE fenced start_writing) calls the real API. Def-site fixes, no fence weakened: - fixture start_writing gains the keyword (instant copies -> accepted-ignored, documented); - test_write_back_eviction_batching: record_submission wrapper FORWARDS the flag (it measures the very wait, so swallowing it would understate the fence); _FencedController gains load_fence_stream = forward (the fork's unconditional entry-168 fence reads it first; prod: both streams are the same forward stream) -- also silences the 16-test Rust TreeCore panic class whose "victims not demoted" assert was the aborted submission downstream. MARKERS (3): managers/cache_controller.py:1202 ADOPTED -- the dropped set_hicache_transfer_done_event write line is superseded by upstream c24285b(sgl-project#48008 per-band page ledger)+2cbb66f563(sgl-project#48012): record_host_transfer publishes the same signal per-page BOTH directions (load twin writes=True). unified_radix_cache.py:627 ADOPTED -- the buffer-mode threshold floor (resolve_prefetch_threshold) was DELIBERATELY deleted upstream by dfaca56(sgl-project#46934, restores sub-window SWA L3 hits); window sizing now lives per-allocation in BufferModePipeline; the per-call-site file_storage_path fold is consistent with the entry-253 funnel (cache_controller.py:948, intact). unified_radix_cache.py:3917 RE-PLANTED -- entry-293's cadence gate (park, do not re-enter allocate_storage_hit/evict_host inside the retry window) + the expired no-retry arm were dropped; wait_full folded into upstream's comp_xfers allow_partial rule (min_tokens form already equivalent). URC context guard: _deferred_ready_sync_group read get_memory() unguarded -- upstream's NEW _arm_tick CPU/spawn fixtures (their restructured timeout tests, sgl-project#44614) reach it without a published server config. Now published-check (schedule_policy.py:615 idiom): default-off CPU path, production unchanged. Deadline-anchor retargets (fork B3-9 family vs upstream's sgl-project#44614 vote-before- poll restructure; every assertion preserved): _vote_prefetch_deadlines drives the verdict before the expiry polls; failed-arm polled inside-budget so the storage failure stays the SOLE cut reason; fixture cc gains the sgl-project#44342 liveness attrs + pp_prefetch_command_thread; _arm_tick stub returns the fork's 5-tuple (health bit True); release_aborted_request -> finish(ABORT) per 8898429(sgl-project#48491). Battery actuals (CPU, CVD=9, prod venv, PYTHONPATH=wt/python): - the 5 cascade files 68F/39P -> 3F/70P (survivors = the sgl-project#48491/pp-thread/ budget-anchor fixes committed here; after them): test_storage_prefetch_ lifecycle ** 31P/31sub ** (entry-311 pin 28P/31sub, 295's >=23P/28sub EXCEEDED); decode_retraction_backup 6F + chunked_publish 2F = No-CUDA device class (env, reported). - combined gate run: trailing_endpoint + batching + parity + async_ready_sync + staged_write_back = 89P/1S/28sub 0F. Gates: ruff repo ruleset+format clean on touched files; registration trio rc=0.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
With
--hicache-write-policy write_back, KV pages could be copied to host before the forward pass that produced them had finished. The host copy then held whatever the page contained at submit time, and a later load-back served that stale KV to another request. Two paths reached the device-to-host copy without any dependency on the forward stream:Modifications
UnifiedTreeCore(Python) andunified_tree_core.rs(Rust): under write-back, insertion never returns aBackupKVaction, so SWA windows are no longer published at insert time. The incremental-component check no longer special-cases SWA under write-back.HiCacheController.start_writing(wait_for_forward=...): a new keyword-only flag. When set and awrite_fence_streamis configured, the device-to-host stream waits on it before the copies are submitted. The scheduler setswrite_fence_streamto the model runner's forward stream next to the existingload_fence_stream.UnifiedRadixCache.writing_check(write_back=True)passeswait_for_forward=Trueon the blocking drain. Write-through and the asynchronous write-back flush are unchanged, and the default keeps every other caller as it was.test_hicache_write_back_ordering.py(new): a checkpointed request whose forward is still writing on a separate stream; evicting an older SWA window of that request must copy the values the forward wrote, not the pre-forward contents.test_unified_hicache_regressions.py: the blocking drain waits for the fence stream before copying both a tick's queued backup and the eviction victim.test_unified_radix_cache_unittest.pyand the Rust crate tests: write-back insertion of backed Full nodes publishes no SWA window and leaves no pending ack; the Mamba pending-ack eviction test now queues its window backup explicitly instead of relying on insert-time publication.Accuracy Tests
cargo testforrust/sglang-radix-tree(--no-default-features, aspr-test-rust-workspace.ymlruns it): 931 passed, 0 failed.SGLANG_UNIFIED_RADIX_TREE_CORE_BACKEND=rustand=python:test_hicache_write_back_ordering.py,test_unified_hicache_regressions.py::TestTransferStreamOrdering, and the two touched cases intest_unified_radix_cache_unittest.pypass.test_hicache_write_back_ordering.pyfails on both backends: the host copy holds the pre-forward contents instead of what the forward wrote.Speed Tests and Profiling
The new wait is a single
wait_streamon the blocking write-back drain, which already blocks on every ack; the asynchronous paths do not take it.Checklist
CI States
Latest PR Test (Base): 🚫 Run #38999355934
Latest PR Test (Extra): ❌ Run #38999355753
Latest PR Test (AMD ROCm 10): 🚫 Run #38999355752