Repository navigation
[Feat] radix-tree: Report the last visited node and a shorter FULL device match in match results - #45703
Merged
Conversation
…vice match in match results A match walk can step into a node and then reject it: a host-only node without HiCache, a node past the SWA window, or a node a component validator declines. The result so far reports only the deepest accepted node (`best_match_node`) and the device anchor, so a cache layered on the core cannot tell where the walk actually stopped, and a core whose FULL device values are not one pool row per token has no place to report a reusable prefix that is shorter than the node-boundary match. `MatchResult` gains two optional fields on both cores and the binding: - `last_visited_node_id`: the deepest non-root node the walk stepped into, whether or not a validator accepted it; `None` when the walk never left the root (empty key, empty tree, dead first child). - `full_device_match_len`: the reusable prefix length seen by the FULL device-value scan before node-boundary or auxiliary constraints; `None` when no shorter prefix was reported. The stock cores always report `None`; the field is a slot for a core that computes one. The Rust walk returns the visited node alongside its other walk state, the post-processor and the empty result fill both fields, and the PyO3 binding exposes them and accepts them on inspection inputs. The Python `UnifiedTreeCore._match_prefix_helper` now takes the match params and returns the finished `MatchResult`, passing the visited node and the device validators to `_match_post_processor`, so the two cores build the result in the same place. `zero_match_result` clears both fields and the Rust adapter copies them across. Tests: the Rust match tests assert the visited node on full hits, partial matches, misses, dead nodes, host-only nodes with and without HiCache and the SWA tombstone case; the Python integration tests run on both backends and cover the empty tree, an exact hit, a split match (checked against `collect_path_info`), a host-only node independent of device acceptance, `zero_match_result`, the stale-finalizer path and the SWA allocator layout. Co-authored-by: Jialin <Jialin@users.noreply.github.com>
metamergebot
requested review from
Jialin,
Ying1123,
alphabetc1,
hanming-lu,
hnyls2002,
huangtingwei9988,
hzh0425,
ispobock,
merrymercy,
xiezhq-hermann and
yizhang2077
as code owners
October 22, 2026 02:17
Collaborator
Author
|
This port clears our bar on gate 21✓✗4✂3 lint✓ build✓ A✓ B✓ C 8/11⏳ other 12✓✂9 · accel ✗amd,npu @luccafong @merrymercy @Jialin — could one of you take a look and merge it if you are happy? Nothing else is waiting on our side. |
detain
added a commit
to detain/sglang
that referenced
this pull request
Oct 29, 2026
…20 commits since 12dc623 @ merge-sgl-project#4) Conflicts: 167 files (149 content / 14 add/add / 4 modify/delete) at the settled 309 tip d28e7cd -- dossier predicted 146 +/-25; the +21 shift is entries 275-309 landing qsa-ring/hicache-ctrl/gdn/moe/llguard deltas into the conflict zone. By class: A 13 (auto-takes-upstream, dedupe-verified), B 80, C 56, D 18. 102 TODO(merge5) re-plant/fold markers planted: B1 x15, B2 x29, B3 x12, B4 x23, B5 x23. Re-plant stock: 117 files under /tmp/opencode/merge5-dossier/replant/ (76 fork-hunks patches, 19 upstream-side patches, 14 AA fork copies, 8 preremove snapshots) -- sessions 2/3 re-plant from real code, not memory. D-list outcomes (must-not-auto-resolve, handled personally): - sgl-project#45289-vs-sgl-project#44858 (entry 301): VERIFIED, not re-resolved -- the gate/up-order final form is in origin/main (5b1dbc2 in range); scales test taken as upstream oracle, fork 191-line version saved to replant/ for B5 diffing. sgl-project#44858's call-site shape CANNOT survive (our base never had it). - entry-230 dispatcher vs sgl-project#44993/sgl-project#44994: took upstream structure in token_dispatcher/standard.py; published-value CONTRACT lives upstream via get_moe_runner_backend() in the LoRA-EP guard; fork's warn-once mismatch diagnostic re-plants (B5 marker). UNION GATE: standard_dispatcher 16P (was 12P, suite grew), model_overrides 121P/1S/210sub (was 112P/1S/181sub), fold probe 6P/2F (upstream added a runner_backend arg to the predicate -- B5 re-pins the probe). - entry-219 gauge names vs sgl-project#43727 return + sgl-project#44968: PER-POOL NAMES KEPT as the serving schema -- hicache_host_pool_{used,total}_tokens{pool} dataclass fields, Gauge registrations and labels(pool=) emission re-restored over the #4697x structure; hicache_host_pool_evicted_tokens_total already survived auto-merge; producer dict restored by union in metrics_reporter. ANTI-NEEDLE SATISFIED: old mamba-only gauges are NOT the serving schema. - entry-221 eager-direct add/add: DEDUPED to upstream (bodies identical except est_time 15->9); MEASUREMENT DEBT RECORDED: prefill t/s must be re-measured post-B4 before trusting the est_time. - entry-168 D2H fence vs sgl-project#44800: the fence is ALREADY UPSTREAM (both load_fence_stream + write_fence_stream with the load-thread-corruption comment in origin/main's cache_controller.py) -- nothing re-planted; the two real regions resolved as import-union + comment wording (merge sgl-project#4 stale-tail pattern avoided). Fork hunks still saved to replant/ (belt). - entry-209 clamp + 216/218 ordering + 288-295 hicache-ctrl-era vs #46970x KV rewrite: ADOPTED-UPSTREAM across unified_radix_cache/hicache_storage/ hybrid_cache_controller/memory_pool/kv_cache_builder; EVERY fork hunk extracted to replant/*.fork-hunks.patch; B3 re-plant markers planted. sgl-project#45703's short-full-match seam arrives with upstream. - sgl-project#45090 SGLANG_ENABLE_UNIFIED_RADIX_TREE: REMOVAL TAKEN (fork had already moved it to _DeprecatedEnv; now gone). grep: ZERO .py readers repo-wide; 2 upstream-owned docs jsx snippets still export it (deepseek-v4, glm-5.2 -- upstream's own debt). OPERATOR-OWED CHECK: confirm no /root harness exports it (unreadable from this seat). - sgl-project#39727 unified L3 key: VERIFIED OPT-IN -- merged hicache_storage.py keeps the DEFAULT legacy suffix composition _<model>_<tp_rank>_<tp_size> (content-anchor: HiCacheFile._init_config_suffix, raise-guarded when _unified_chunk_spans is set). ANTI-NEEDLE: no silent key-scheme switch; warm /root/hicache_store contents stay addressable. Modify/delete (4, all victims of upstream sgl-project#45672 test prune + URC split): - test_unified_radix_cache_unittest.py: REMOVED (adopted split) -- its bare unified_tree_core_inspector imports are unreachable post-sgl-project#45236 re-home; fork source + hunks saved; B2 marker planted in the new core test. - test_fused_moe_triton_config.py: REMOVED -- imports common_utils, absent from both sides post-sgl-project#45672; the 2 live entry-3/235 tests re-plant in B2. - test_scheduler_batch_is_full_latch.py: REMOVED -- fork delta was 3 stub-attr lines documenting sgl-project#46643's absorbed arms; upstream deletion correct. - test_qsa_kv_pool.py: KEPT -- fork's 147 added lines are entry-187/200 sharding + 285-287 QSA-ring tests with NO upstream replacement; qsa_kv_pool.py sides converged byte-identical; collects clean. B2 marker. Cascading silent-auto-merge repairs (classes not in the conflict set but broken by auto-merge + union splices -- all found by gates, now fixed): kernels/ops/activation/__init__.py (dead _CUDA registration dup), kernels/ops/ diffusion/__init__.py (5-tuple _SPECS entry), kernels ops embeddings/moe/ attention/mamba __init__ (KernelSpec fields removed upstream: description/ format_signature stripped, CapabilityRequirement->CUDA_ONLY), base_runner/ eager_runner cp_extend + shared_read_event imports retargeted per upstream's "no imports in runner_utils/__init__" contract, and 4 fork helpers grafted back onto their take-upstream homes because merged code/tests still import them (memory_pool.spec_intermediate_bytes_per_req, kv_cache_builder. _adjust_hicache_size_for_draft + _device_pool_size_per_token, hybrid_cache_controller.local_restorable_mask, validation_hook. validate_replayssm_spec_algorithm, arg_groups/moe_hook b12x validator). flashinfer_cutlass.py flipped to FORK-OWNED (b12x route is fork-only, 760 vs 95 lines; upstream sgl-project#44993 hunks = B5 fold patch saved). Silent-semantics notes: sgl-project#46127/sgl-project#44912 riders arrived via per-PR picks (union); sgl-project#45520 (entry 302) is STILL OPEN upstream -- entry-230 fail-safe layer stays as entry 303 records; if a future sync brings it, re-apply the fail-safe reads. qwen4_exp holder-contract fused (ctor takes optional batch). Gates at commit: import sglang OK (CVD=9, prod venv); check_registered_tests/ check_no_bare_pytest_main/check_no_registered_tests_in_package rc=0; ruff F821 +E9+format clean on the resolution set; collect-only over unit/{mem_cache, model_executor,distributed,layers/moe} = 4337 nodes, 3 ERRORS: 2x cargo/torch- sys libtorch probe (REPRODUCED IDENTICALLY AT THE PRE-MERGE TIP -- env issue, not merge damage) + test_hicache_async_ready_sync.py (entry-253-era attribute gone with the URC rewrite -- EXPECTED pre-B3). sessions 2/3 owed: B2 test re-home, B3 mem_cache+rust re-plant (+hicache-ctrl re-ports), B4 scheduler/admission, B5 moe/quant/dispatcher, B6 environ+C0a+ CAVEATS re-pin. PROD BOOT FORBIDDEN until B3 completes (209 clamp + 218 ordering re-plant).
detain
added a commit
to detain/sglang
that referenced
this pull request
Oct 29, 2026
…),199,217,218,219,253,290d,293,294 Re-plants the fork-hunks patch (1013 lines, the deepest file) onto the #46970x-rewritten URC. B3 marker consumed in unified_radix_cache.py. What the merged tree had ALREADY absorbed (deduped, not duplicated — per-hunk verdicts recorded): the whole deadline/stop-policy family (_StoragePrefetchWait msgspec/PollTimer, _start_storage_prefetch_wait, _prefetch_deadline_expired REPLACING _prefetch_timeout_check_linear_func, _mark_prefetch_deadline_expired, the reason="deadline" bookkeeping, belief-heal suppression on deadline, the _invalidate_absent_from_hit_query move to the drain head + KV-verdict guard + deadline gate, the release_aborted_request / finish_storage_prefetch_admission wait pops, and the _should_backup_after_insert write-back half of sgl-project#44800 (unified_tree_core.py:1447). The sgl-project#45703 short-full-match seam did NOT obsolete entry 209's clamp (it lives controller-side; B3-2 gate 17P). Re-planted here: - entry 294 (sgl-project#45795): the FULL deferred ready-count sync family (_deferred_ready_sync_group gate ladder incl. the runtime-attach fallback, _check_hicache_events_deferred one-op-per-step on its own gloo group, _consume_deferred_ready_sync with the digest MIN/-MAX assert, _deferred_consume_count, _drop_deferred_ready_sync) + the check_hicache_events branch + the reset() settle. This CLEARS the test_hicache_async_ready_sync collection error (the B3-caused 1err in the merge gates). - entries 199+218: the batched write-through backup family VERBATIM from the patch (_NodeBackupSpec, _queue_write_through_backup, _drain_queued_backups, _ancestors_first incl. the sgl-project#40363 delta walked-set, _is_unbacked_node, _parent_backed_up, _release_queued_backup, _backup_queued_batch, _build_node_backup_spec, _backup_specs, _merge_backup_transfers, the node_ids= ack-list through _execute_kv_backup into the controller write (B3-2), the flush_pending_backups guard, and the writing_check(write_back) drain + start_writing(wait_for_forward=True) — the ONE-fence ordering rule). _batched_backup reads SGLANG_ENABLE_HICACHE_BATCHED_BACKUP, DEFAULT FALSE (upstream's 0ab8654 default-ON stays NOT taken, entry-218's record). - entry 293 (sgl-project#46451): the opt-in host-capacity WAIT: fields + init_hicache reads with the nonnegative-finite ValueError, wait_full, the retry cadence (host_capacity_deadline/next_retry/retries — fields planted on the operation in B3-2), allow_partial=not wait_full, pending_storage_hits park/drain with identity snapshots, the info.operation-is-not-operation late-query guards, and the revoke/alloc-already-owned guards. DEFAULT 0.0 = byte-identical. - entry 290(d): _alloc_prefetch_aux_staging now sizes to alloc_len, the endpoint the KV span ACTUALLY got (mamba-host-full prod boots stop declining prefetches a smaller endpoint would fit). - entry 219 producer: _record_host_pool_evictions on evict_host (before _apply_eviction_result) feeding hicache_host_pool_evicted_tokens_total{pool}. - entry 292 (sgl-project#43110): _build_sidecar_transfers resolves derived pools through merge_source_transfers (every destination, not matching_sources[0]; the two AssertionErrors it subsumes are structurally guaranteed by the resolver). - entry 30 rider: load_back unwinds (drop both locks, report miss) when any component PREPARED-REJECTS (the sgl-project#39830 poison/degrade contract the mamba component sets -- component side rides B3-5's mamba.py cluster). - entry 253: the per-call-site file_storage_path fold DELETED here (it moved into _generate_storage_config, every attach path, B3-1-verified). - test_chunked_publish_release_disjointness.py (entry 307's tripwire): re-anchored CacheInitParams(tree_components=...) -> tree_slots=tree_slots_of(...) per the upstream slots refactor; the tree_core_backend='rust' SUBFAILED line is the documented cargo/torch-sys ENV class (fails identically at base). ACTUALS (CVD=9, prod venv, PYTHONPATH=python, per-file, damage -> after): test_hicache_async_ready_sync 1err -> 7P/0F ** B3-caused error CLEARED ** test_hybrid_storage_sparse_allocation 12P/1F -> 13P/0F/14sub ** entry-290(d) closed ** test_storage_prefetch_lifecycle -> 16P/0F/13sub test_hicache_trailing_endpoint (gate) -> 17P/0F (unmoved by this cluster; controller-side B3-2) test_buffer_mode_sidecar -> 6P/0F/6sub test_chunked_publish_release_disjointness 4P/2F -> 4P+1sub/1F (the F now the rust-cargo ENV class; the python-backend half RE-ANCHORED green) test_mamba_path_state_cap -> 5P/5S (skips = honest, env) test_decode_retraction_backup 2P/2F -> 2P/2F (both F = No-CUDA ENV class, byte-identical at base; file is B2's per the resolution log) test_hicache_mamba_sizing -> 30P/0F/17sub (unmoved) URC topic dir (unified_radix_cache/*.py): 1056F/40P/199S -> 1056F/40P/199S UNMOVED — 100% of the errors are 'No accelerator ... available' from the upstream kits' fixtures.py build_fixture calling get_device() (entry 310's "GREEN, honest skips" claim does NOT reproduce at this seat; the battery's own per-file numbers (core 27P/465F etc.) match the damage totals exactly; this is a kit no-CUDA skip DEFECT = B2/kit scope, recorded not fixed here). Lint: ruff format + F821/E9 clean on both files; py_compile + import OK. No GPU executed; no push.
This was referenced Oct 29, 2026
ll2088
added a commit
to ll2088/sglang
that referenced
this pull request
Nov 2, 2026
…atchResult (PR #3) feat(mem_cache): add last visited node and full device match len to MatchResult (from sgl-project#45703)
ll2088
added a commit
to ll2088/sglang
that referenced
this pull request
Nov 6, 2026
Ported from sgl-project#45703 Generated by autojump (AI); review before merging.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
A match walk can step into a node and then reject it: a host-only node without HiCache, a node past the SWA window, or a node a component validator declines.
MatchResultreports only the deepest accepted node (best_match_node) and the device anchor, so a cache layered on the tree core cannot tell where the walk actually stopped. A core whose FULL device values are not laid out one pool row per token also has no place to report a reusable prefix that is shorter than the node-boundary match. Both cores and the binding should report these the same way so a caller behaves identically on either backend.Modifications
MatchResultgains two optional fields, on the PythonNamedTuple, the Rust struct and the PyO3 binding:last_visited_node_id: the deepest non-root node the walk stepped into, whether or not a validator accepted it.Nonewhen the walk never left the root (empty key, empty tree, a dead first child).full_device_match_len: the reusable prefix length seen by the FULL device-value scan before node-boundary or auxiliary constraints.Nonewhen no shorter prefix was reported. The stock cores always reportNone; the field is a slot for a core that computes one.Rust crate (
rust/sglang-radix-tree):match_prefix_helper_returns the visited node with the rest of its walk state,match_post_processor_andempty_match_resultfill both fields,MatchResultBindingexposes them andInspectionMatchResultInputreads them.Python:
UnifiedTreeCore._match_prefix_helper(key, params)now returns the finishedMatchResultand calls_match_post_processoritself, passing the visited node and the device validators, so both cores build the result in one place;match_prefixbecomes a thin wrapper.zero_match_resultclears both fields, andRustUnifiedTreeCorecopies them from the binding.Tests: the Rust match tests assert
last_visited_node_idon full hits, partial matches, misses, dead nodes, host-only nodes with and without HiCache, the SWA tombstone case and the borrowed-validator walk.test_rust_tree_core_integration.pyruns the match cases on both backends and covers the empty tree (page size 2, keys of length 0, 1 and 3), an exact hit, a split match checked againstcollect_path_info, a host-only node whose visited node is independent of device acceptance,zero_match_resultclearing the fields, the stale-finalizer path preserving them, and the SWA allocator-layout case.Accuracy Tests
Not applicable: no model or kernel numerics change.
Benchmarking and Profiling
Not applicable: the walk records one node id it already holds; no extra tree traversal.
Checklist
rust/sglang-radix-tree:cargo test --locked --no-default-features(942 passed, 1 ignored),cargo clippy --all-targets --no-default-features --features tch/doc-only -- -D warnings,cargo fmt -- --check.test/registered/unit/mem_cache/test_rust_tree_core_integration.pyand theunified_radix_cache/unit tests.CI States
Latest PR Test (Base): 🚫 Run #39710581212
Latest PR Test (Extra): ❌ Run #39710581173
Latest PR Test (AMD ROCm 10): ❌ Run #39710581268