Skip to content

[Feat] radix-tree: Report the last visited node and a shorter FULL device match in match results - #45703

Merged
hanming-lu merged 1 commit into
mainfrom
oss-sync-e1edc9c9f6
Oct 22, 2026
Merged

hanming-lu merged 1 commit into
mainfrom
oss-sync-e1edc9c9f6

Conversation

@metamergebot

@metamergebot metamergebot commented Oct 22, 2026 •

Copy link
Copy Markdown
Collaborator

Motivation

A match walk can step into a node and then reject it: a host-only node without HiCache, a node past the SWA window, or a node a component validator declines. MatchResult reports only the deepest accepted node (best_match_node) and the device anchor, so a cache layered on the tree core cannot tell where the walk actually stopped. A core whose FULL device values are not laid out one pool row per token also has no place to report a reusable prefix that is shorter than the node-boundary match. Both cores and the binding should report these the same way so a caller behaves identically on either backend.

Modifications

MatchResult gains two optional fields, on the Python NamedTuple, the Rust struct and the PyO3 binding:

  • last_visited_node_id: the deepest non-root node the walk stepped into, whether or not a validator accepted it. None when the walk never left the root (empty key, empty tree, a dead first child).
  • full_device_match_len: the reusable prefix length seen by the FULL device-value scan before node-boundary or auxiliary constraints. None when no shorter prefix was reported. The stock cores always report None; the field is a slot for a core that computes one.

Rust crate (rust/sglang-radix-tree): match_prefix_helper_ returns the visited node with the rest of its walk state, match_post_processor_ and empty_match_result fill both fields, MatchResultBinding exposes them and InspectionMatchResultInput reads them.

Python: UnifiedTreeCore._match_prefix_helper(key, params) now returns the finished MatchResult and calls _match_post_processor itself, passing the visited node and the device validators, so both cores build the result in one place; match_prefix becomes a thin wrapper. zero_match_result clears both fields, and RustUnifiedTreeCore copies them from the binding.

Tests: the Rust match tests assert last_visited_node_id on full hits, partial matches, misses, dead nodes, host-only nodes with and without HiCache, the SWA tombstone case and the borrowed-validator walk. test_rust_tree_core_integration.py runs the match cases on both backends and covers the empty tree (page size 2, keys of length 0, 1 and 3), an exact hit, a split match checked against collect_path_info, a host-only node whose visited node is independent of device acceptance, zero_match_result clearing the fields, the stale-finalizer path preserving them, and the SWA allocator-layout case.

Accuracy Tests

Not applicable: no model or kernel numerics change.

Benchmarking and Profiling

Not applicable: the walk records one node id it already holds; no extra tree traversal.

Checklist

  • Rust, in rust/sglang-radix-tree: cargo test --locked --no-default-features (942 passed, 1 ignored), cargo clippy --all-targets --no-default-features --features tch/doc-only -- -D warnings, cargo fmt -- --check.
  • Python, on a CUDA host: test/registered/unit/mem_cache/test_rust_tree_core_integration.py and the unified_radix_cache/ unit tests.
  • pre-commit (isort, ruff, ruff-format, codespell, test registry checks) clean on the touched files.

CI States

Latest PR Test (Base): 🚫 Run #39710581212
Latest PR Test (Extra): ❌ Run #39710581173
Latest PR Test (AMD ROCm 10): ❌ Run #39710581268

…vice match in match results

A match walk can step into a node and then reject it: a host-only node
without HiCache, a node past the SWA window, or a node a component
validator declines. The result so far reports only the deepest accepted
node (`best_match_node`) and the device anchor, so a cache layered on the
core cannot tell where the walk actually stopped, and a core whose FULL
device values are not one pool row per token has no place to report a
reusable prefix that is shorter than the node-boundary match.

`MatchResult` gains two optional fields on both cores and the binding:

- `last_visited_node_id`: the deepest non-root node the walk stepped
  into, whether or not a validator accepted it; `None` when the walk
  never left the root (empty key, empty tree, dead first child).
- `full_device_match_len`: the reusable prefix length seen by the FULL
  device-value scan before node-boundary or auxiliary constraints;
  `None` when no shorter prefix was reported. The stock cores always
  report `None`; the field is a slot for a core that computes one.

The Rust walk returns the visited node alongside its other walk state,
the post-processor and the empty result fill both fields, and the PyO3
binding exposes them and accepts them on inspection inputs. The Python
`UnifiedTreeCore._match_prefix_helper` now takes the match params and
returns the finished `MatchResult`, passing the visited node and the
device validators to `_match_post_processor`, so the two cores build the
result in the same place. `zero_match_result` clears both fields and the
Rust adapter copies them across.

Tests: the Rust match tests assert the visited node on full hits, partial
matches, misses, dead nodes, host-only nodes with and without HiCache and
the SWA tombstone case; the Python integration tests run on both backends
and cover the empty tree, an exact hit, a split match (checked against
`collect_path_info`), a host-only node independent of device acceptance,
`zero_match_result`, the stale-finalizer path and the SWA allocator layout.

Co-authored-by: Jialin <Jialin@users.noreply.github.com>
@metamergebot

Copy link
Copy Markdown
Collaborator Author

This port clears our bar on e52c667f689231af6f3147c33c9957ec0b45c20e: every base-a and base-b job is green.
base-c and the accelerator suites are reported but not gated, because they run
on scarce hardware and upstream merges through them routinely.

gate 21✓✗4✂3 lint✓ build✓ A✓ B✓ C 8/11⏳ other 12✓✂9 · accel ✗amd,npu

@luccafong @merrymercy @Jialin — could one of you take a look and merge it if you are happy? Nothing else is waiting on our side.

@hanming-lu
hanming-lu merged commit aae2145 into main Oct 22, 2026
200 of 228 checks passed
@hanming-lu
hanming-lu deleted the oss-sync-e1edc9c9f6 branch October 22, 2026 20:11
detain added a commit to detain/sglang that referenced this pull request Oct 29, 2026
…20 commits since 12dc623 @ merge-sgl-project#4)

Conflicts: 167 files (149 content / 14 add/add / 4 modify/delete) at the settled
309 tip d28e7cd -- dossier predicted 146 +/-25; the +21 shift is entries
275-309 landing qsa-ring/hicache-ctrl/gdn/moe/llguard deltas into the conflict
zone. By class: A 13 (auto-takes-upstream, dedupe-verified), B 80, C 56, D 18.
102 TODO(merge5) re-plant/fold markers planted: B1 x15, B2 x29, B3 x12, B4 x23,
B5 x23. Re-plant stock: 117 files under /tmp/opencode/merge5-dossier/replant/
(76 fork-hunks patches, 19 upstream-side patches, 14 AA fork copies, 8
preremove snapshots) -- sessions 2/3 re-plant from real code, not memory.

D-list outcomes (must-not-auto-resolve, handled personally):
- sgl-project#45289-vs-sgl-project#44858 (entry 301): VERIFIED, not re-resolved -- the gate/up-order
  final form is in origin/main (5b1dbc2 in range); scales test taken as
  upstream oracle, fork 191-line version saved to replant/ for B5 diffing.
  sgl-project#44858's call-site shape CANNOT survive (our base never had it).
- entry-230 dispatcher vs sgl-project#44993/sgl-project#44994: took upstream structure in
  token_dispatcher/standard.py; published-value CONTRACT lives upstream via
  get_moe_runner_backend() in the LoRA-EP guard; fork's warn-once mismatch
  diagnostic re-plants (B5 marker). UNION GATE: standard_dispatcher 16P (was
  12P, suite grew), model_overrides 121P/1S/210sub (was 112P/1S/181sub),
  fold probe 6P/2F (upstream added a runner_backend arg to the predicate --
  B5 re-pins the probe).
- entry-219 gauge names vs sgl-project#43727 return + sgl-project#44968: PER-POOL NAMES KEPT as the
  serving schema -- hicache_host_pool_{used,total}_tokens{pool} dataclass
  fields, Gauge registrations and labels(pool=) emission re-restored over the
  #4697x structure; hicache_host_pool_evicted_tokens_total already survived
  auto-merge; producer dict restored by union in metrics_reporter. ANTI-NEEDLE
  SATISFIED: old mamba-only gauges are NOT the serving schema.
- entry-221 eager-direct add/add: DEDUPED to upstream (bodies identical except
  est_time 15->9); MEASUREMENT DEBT RECORDED: prefill t/s must be re-measured
  post-B4 before trusting the est_time.
- entry-168 D2H fence vs sgl-project#44800: the fence is ALREADY UPSTREAM (both
  load_fence_stream + write_fence_stream with the load-thread-corruption
  comment in origin/main's cache_controller.py) -- nothing re-planted; the two
  real regions resolved as import-union + comment wording (merge sgl-project#4 stale-tail
  pattern avoided). Fork hunks still saved to replant/ (belt).
- entry-209 clamp + 216/218 ordering + 288-295 hicache-ctrl-era vs #46970x KV
  rewrite: ADOPTED-UPSTREAM across unified_radix_cache/hicache_storage/
  hybrid_cache_controller/memory_pool/kv_cache_builder; EVERY fork hunk
  extracted to replant/*.fork-hunks.patch; B3 re-plant markers planted.
  sgl-project#45703's short-full-match seam arrives with upstream.
- sgl-project#45090 SGLANG_ENABLE_UNIFIED_RADIX_TREE: REMOVAL TAKEN (fork had already
  moved it to _DeprecatedEnv; now gone). grep: ZERO .py readers repo-wide;
  2 upstream-owned docs jsx snippets still export it (deepseek-v4, glm-5.2 --
  upstream's own debt). OPERATOR-OWED CHECK: confirm no /root harness exports
  it (unreadable from this seat).
- sgl-project#39727 unified L3 key: VERIFIED OPT-IN -- merged hicache_storage.py keeps
  the DEFAULT legacy suffix composition _<model>_<tp_rank>_<tp_size>
  (content-anchor: HiCacheFile._init_config_suffix, raise-guarded when
  _unified_chunk_spans is set). ANTI-NEEDLE: no silent key-scheme switch;
  warm /root/hicache_store contents stay addressable.

Modify/delete (4, all victims of upstream sgl-project#45672 test prune + URC split):
- test_unified_radix_cache_unittest.py: REMOVED (adopted split) -- its bare
  unified_tree_core_inspector imports are unreachable post-sgl-project#45236 re-home;
  fork source + hunks saved; B2 marker planted in the new core test.
- test_fused_moe_triton_config.py: REMOVED -- imports common_utils, absent
  from both sides post-sgl-project#45672; the 2 live entry-3/235 tests re-plant in B2.
- test_scheduler_batch_is_full_latch.py: REMOVED -- fork delta was 3 stub-attr
  lines documenting sgl-project#46643's absorbed arms; upstream deletion correct.
- test_qsa_kv_pool.py: KEPT -- fork's 147 added lines are entry-187/200
  sharding + 285-287 QSA-ring tests with NO upstream replacement;
  qsa_kv_pool.py sides converged byte-identical; collects clean. B2 marker.

Cascading silent-auto-merge repairs (classes not in the conflict set but
broken by auto-merge + union splices -- all found by gates, now fixed):
kernels/ops/activation/__init__.py (dead _CUDA registration dup), kernels/ops/
diffusion/__init__.py (5-tuple _SPECS entry), kernels ops embeddings/moe/
attention/mamba __init__ (KernelSpec fields removed upstream: description/
format_signature stripped, CapabilityRequirement->CUDA_ONLY), base_runner/
eager_runner cp_extend + shared_read_event imports retargeted per upstream's
"no imports in runner_utils/__init__" contract, and 4 fork helpers grafted
back onto their take-upstream homes because merged code/tests still import
them (memory_pool.spec_intermediate_bytes_per_req, kv_cache_builder.
_adjust_hicache_size_for_draft + _device_pool_size_per_token,
hybrid_cache_controller.local_restorable_mask, validation_hook.
validate_replayssm_spec_algorithm, arg_groups/moe_hook b12x validator).
flashinfer_cutlass.py flipped to FORK-OWNED (b12x route is fork-only, 760 vs
95 lines; upstream sgl-project#44993 hunks = B5 fold patch saved).

Silent-semantics notes: sgl-project#46127/sgl-project#44912 riders arrived via per-PR picks (union);
sgl-project#45520 (entry 302) is STILL OPEN upstream -- entry-230 fail-safe layer stays
as entry 303 records; if a future sync brings it, re-apply the fail-safe reads.
qwen4_exp holder-contract fused (ctor takes optional batch).

Gates at commit: import sglang OK (CVD=9, prod venv); check_registered_tests/
check_no_bare_pytest_main/check_no_registered_tests_in_package rc=0; ruff F821
+E9+format clean on the resolution set; collect-only over unit/{mem_cache,
model_executor,distributed,layers/moe} = 4337 nodes, 3 ERRORS: 2x cargo/torch-
sys libtorch probe (REPRODUCED IDENTICALLY AT THE PRE-MERGE TIP -- env issue,
not merge damage) + test_hicache_async_ready_sync.py (entry-253-era attribute
gone with the URC rewrite -- EXPECTED pre-B3).

sessions 2/3 owed: B2 test re-home, B3 mem_cache+rust re-plant (+hicache-ctrl
re-ports), B4 scheduler/admission, B5 moe/quant/dispatcher, B6 environ+C0a+
CAVEATS re-pin. PROD BOOT FORBIDDEN until B3 completes (209 clamp + 218
ordering re-plant).
detain added a commit to detain/sglang that referenced this pull request Oct 29, 2026
…),199,217,218,219,253,290d,293,294

Re-plants the fork-hunks patch (1013 lines, the deepest file) onto the
#46970x-rewritten URC. B3 marker consumed in unified_radix_cache.py.

What the merged tree had ALREADY absorbed (deduped, not duplicated — per-hunk
verdicts recorded): the whole deadline/stop-policy family (_StoragePrefetchWait
msgspec/PollTimer, _start_storage_prefetch_wait, _prefetch_deadline_expired
REPLACING _prefetch_timeout_check_linear_func, _mark_prefetch_deadline_expired,
the reason="deadline" bookkeeping, belief-heal suppression on deadline, the
_invalidate_absent_from_hit_query move to the drain head + KV-verdict guard +
deadline gate, the release_aborted_request / finish_storage_prefetch_admission
wait pops, and the _should_backup_after_insert write-back half of sgl-project#44800
(unified_tree_core.py:1447). The sgl-project#45703 short-full-match seam did NOT
obsolete entry 209's clamp (it lives controller-side; B3-2 gate 17P).

Re-planted here:
- entry 294 (sgl-project#45795): the FULL deferred ready-count sync family
  (_deferred_ready_sync_group gate ladder incl. the runtime-attach fallback,
  _check_hicache_events_deferred one-op-per-step on its own gloo group,
  _consume_deferred_ready_sync with the digest MIN/-MAX assert,
  _deferred_consume_count, _drop_deferred_ready_sync) + the check_hicache_events
  branch + the reset() settle. This CLEARS the test_hicache_async_ready_sync
  collection error (the B3-caused 1err in the merge gates).
- entries 199+218: the batched write-through backup family VERBATIM from the
  patch (_NodeBackupSpec, _queue_write_through_backup, _drain_queued_backups,
  _ancestors_first incl. the sgl-project#40363 delta walked-set, _is_unbacked_node,
  _parent_backed_up, _release_queued_backup, _backup_queued_batch,
  _build_node_backup_spec, _backup_specs, _merge_backup_transfers, the
  node_ids= ack-list through _execute_kv_backup into the controller write
  (B3-2), the flush_pending_backups guard, and the writing_check(write_back)
  drain + start_writing(wait_for_forward=True) — the ONE-fence ordering rule).
  _batched_backup reads SGLANG_ENABLE_HICACHE_BATCHED_BACKUP, DEFAULT FALSE
  (upstream's 0ab8654 default-ON stays NOT taken, entry-218's record).
- entry 293 (sgl-project#46451): the opt-in host-capacity WAIT: fields + init_hicache
  reads with the nonnegative-finite ValueError, wait_full, the retry cadence
  (host_capacity_deadline/next_retry/retries — fields planted on the operation
  in B3-2), allow_partial=not wait_full, pending_storage_hits park/drain with
  identity snapshots, the info.operation-is-not-operation late-query guards,
  and the revoke/alloc-already-owned guards. DEFAULT 0.0 = byte-identical.
- entry 290(d): _alloc_prefetch_aux_staging now sizes to alloc_len, the
  endpoint the KV span ACTUALLY got (mamba-host-full prod boots stop declining
  prefetches a smaller endpoint would fit).
- entry 219 producer: _record_host_pool_evictions on evict_host (before
  _apply_eviction_result) feeding hicache_host_pool_evicted_tokens_total{pool}.
- entry 292 (sgl-project#43110): _build_sidecar_transfers resolves derived pools through
  merge_source_transfers (every destination, not matching_sources[0]; the two
  AssertionErrors it subsumes are structurally guaranteed by the resolver).
- entry 30 rider: load_back unwinds (drop both locks, report miss) when any
  component PREPARED-REJECTS (the sgl-project#39830 poison/degrade contract the mamba
  component sets -- component side rides B3-5's mamba.py cluster).
- entry 253: the per-call-site file_storage_path fold DELETED here (it moved
  into _generate_storage_config, every attach path, B3-1-verified).
- test_chunked_publish_release_disjointness.py (entry 307's tripwire):
  re-anchored CacheInitParams(tree_components=...) -> tree_slots=tree_slots_of(...)
  per the upstream slots refactor; the tree_core_backend='rust' SUBFAILED line
  is the documented cargo/torch-sys ENV class (fails identically at base).

ACTUALS (CVD=9, prod venv, PYTHONPATH=python, per-file, damage -> after):
  test_hicache_async_ready_sync          1err -> 7P/0F   ** B3-caused error CLEARED **
  test_hybrid_storage_sparse_allocation  12P/1F -> 13P/0F/14sub  ** entry-290(d) closed **
  test_storage_prefetch_lifecycle                -> 16P/0F/13sub
  test_hicache_trailing_endpoint (gate)          -> 17P/0F (unmoved by this cluster; controller-side B3-2)
  test_buffer_mode_sidecar                        -> 6P/0F/6sub
  test_chunked_publish_release_disjointness 4P/2F -> 4P+1sub/1F (the F now the rust-cargo ENV class; the python-backend half RE-ANCHORED green)
  test_mamba_path_state_cap                      -> 5P/5S (skips = honest, env)
  test_decode_retraction_backup             2P/2F -> 2P/2F (both F = No-CUDA ENV class, byte-identical at base; file is B2's per the resolution log)
  test_hicache_mamba_sizing                      -> 30P/0F/17sub (unmoved)
  URC topic dir (unified_radix_cache/*.py): 1056F/40P/199S -> 1056F/40P/199S
    UNMOVED — 100% of the errors are 'No accelerator ... available' from the
    upstream kits' fixtures.py build_fixture calling get_device() (entry 310's
    "GREEN, honest skips" claim does NOT reproduce at this seat; the battery's
    own per-file numbers (core 27P/465F etc.) match the damage totals exactly;
    this is a kit no-CUDA skip DEFECT = B2/kit scope, recorded not fixed here).
Lint: ruff format + F821/E9 clean on both files; py_compile + import OK.
No GPU executed; no push.
ll2088 added a commit to ll2088/sglang that referenced this pull request Nov 2, 2026
…atchResult (PR #3)

feat(mem_cache): add last visited node and full device match len to MatchResult (from sgl-project#45703)
ll2088 added a commit to ll2088/sglang that referenced this pull request Nov 6, 2026
Ported from sgl-project#45703
Generated by autojump (AI); review before merging.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bypass-fail-fast CI: a failing job no longer aborts its siblings (lint still gates) run-ci CI: run the baseline test suite on this PR unified-radix-cache

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants