Repository navigation
fixed an error handling in bench_latency.py - #904
Conversation
- see before and after below.
There was a problem hiding this comment.
Currently, it is impossible to see if self.tree_cache is None. The bench_latency file is a hacky way of running model inference using low-level APIs, which skips the step of creating tree_cache.
However, it is also a concern that the --disable-radix option only skips the computation at the lowest level and does not remove everything like tree_cache. It might be better to turn off everything, but in that case, there might not be only one place to change. We are also planning to refactor the tree_cache in a more modular way.
Regardless, the current change is correct and could be part of the complete change. What do you think? @merrymercy
|
for benchmarking, maybe it is actually good to be able to run without the cache? That should make the benchmark more deterministic and not affected by caching? IMHO, it is better to have the ability to run things with least dependencies unless tree cache is such a core thing and srt simply can't run without it? |
Yes, agree. |
Modify Multimodal Pressure Test Parameters in the Adaptation Scripts.
…d never joined the transfer that fills it) into the flip train
…, so 411 declines have no verdict Routing from the write-side agent (sgl-project#872) narrowed Posten 0 correctly: the store is healthy, 24277 canonical pages live during the R7 window, and the historic "store is empty" readings probed a path this boot does not use. So the 671 refused walks are a MATCH problem, not a content problem. The question that follows immediately is the one nobody could answer: a walk that matches nothing should fall through to an L3 prefetch, and prefetch was attempted on only 264 of 675 sampled walks. THE OBVIOUS ANSWER IS WRONG, and the code says why. It is tempting to read the 411 as the same MambaComponent refusal declining twice -- one root, two numbers. It is not: * The prefetch anchor IS mamba-gated, but by an aliasing the base contract does not describe. base_prefix_cache.py:201-210 separates `last_host_node` (the L3 prefetch anchor) from `best_match_node` (the validator-consensus L2 anchor) -- two fields, two purposes. UnifiedRadixCache sets them equal under HiCache (unified_radix_cache.py:1443-1449). Deliberate and commented, but it contradicts the field's own documentation, and two fields that are documented different and implemented identical are a drift source (sgl-project#747's shape). Named, not changed here. * Even so, the refusal does NOT decline the prefetch. Total refusal leaves best_match_node at the root, and the caller admits the root on purpose: `locally_eligible = last_host_node.backuped or last_host_node is self.tree_cache.root_node` (scheduler.py:4933-4935). A cold request with no match is supposed to prefetch. The first term holds. * The second term (>= 256 new tokens) holds for agent prompts of thousands. That leaves the rate limiter -- AND THERE THE EVIDENCE STOPS, which is the actual finding. None of the three terms is counted or logged. The 411 declines are silent. Nobody can attribute them from this boot, and no amount of re-reading the log will change that. eligible = (locally_eligible and prefetch_length >= self.prefetch_threshold and not self.cache_controller.prefetch_rate_limited()) Three unrelated verdicts wearing one boolean, remedies in three different files. The sgl-project#914 shape one module over: blame without a defect. So this is an instrument and NOT a policy change. The three terms are the same three, evaluated in the same order, with the same outcome; `eligible` is now `reason is None`, one expression, so the predicate and its explanation cannot drift apart the way sgl-project#747 records these very lineages drifting. The first failing term is named rather than all of them -- a request can trip several, and summing them would double-count exactly as `refused_tokens_by_component` is documented to. Not env-gated, unlike the sgl-project#904 match census. That one builds an object and walks validators a second time, so it pays for itself only when armed. This is one integer increment on a path that already builds a RadixKey and takes a host lock -- and a counter you must remember to arm cannot answer "was it ever tried", which is the whole question. `attempted` is counted alongside the refusals so the denominator is local (sgl-project#873: a denominator reconstructed from another log is how a narrowed candidate set reads as a decomposition), and an unrecorded gate says "no observation" rather than a silent zero (sgl-project#829). HYPOTHESIS, RECORDED AS ONE. prefetch_capacity_limit is `0.5 * mem_pool_host.size` (cache_controller.py:729), and that size is not constant across a phase flip: sgl-project#905 measured the two host tiers at 703472 rows (PP) and 30518 (TP), 23x apart, putting the TP-phase budget near 15259 tokens -- under four prefetches of the 4096 this window actually completed, against 5 concurrent requests per flip. The order of magnitude fits 411/675, and I am deliberately not fitting it exactly: the PP phase has ample budget, so the number cannot come from there alone. If it holds, the same host-tier asymmetry behind sgl-project#905's double-free is also starving the TP-phase prefetch budget -- one defect, two effects, to be checked rather than assumed. The counter decides it on the next boot instead of re-arguing it. RED-FIRST, four mutants, each red, restored green (14 tests): revert to the shipped conjunction (no counter) ......... 4 failed eligible computed twice instead of derived from reason . 2 failed attempts not counted (denominator lost) ................ 2 failed no-observation reported as attempted=0 ................. 1 failed ONE SELF-INFLICTED DEFECT, found and fixed in the same pass, recorded because the form is evidently easy to commit. My first version of test_the_rate_limit_check_is_still_called_at_most_once counted occurrences in the function's RAW source and went red because a PRE-EXISTING comment at :2674 mentions `prefetch_rate_limited()`. That is the sgl-project#908 substring defect -- prose used to reach a verdict about code -- in a test of mine, an hour after I rejected the same form in someone else's patch. Comment lines are now stripped before counting. GATE test/registered/unit/mem_cache serial, CVD="": 3 failed, 1950 passed, 1658 skipped. Both axes agree (summary 3, ANSI-stripped extraction 3). Two are test_acceptance_emitters_758::RefillTiming, baseline-proven pre-existing earlier in this branch; the third is test_pool_invariant_double_owned_912, a concurrently-developed test belonging to sgl-project#912 and untouched by this commit. The sgl-project#904 census suite is green against this change (62 passed together with the sgl-project#913/sgl-project#914/sgl-project#915 suites), which is the regression that matters: note_refused gained an optional third argument and its existing callers must not care.
The census (/spinning/gpu-arb/CENSUS_1035_readthrough_chain.md) listed every gate on the read-through chain with its instrument or BLIND. Three BLIND rows needed no new mechanism, only exposure, and all three share one shape: a gate that can ZERO the read path while printing nothing, so a boot where it fired and a boot where storage was simply cold produce identical logs. That is how this campaign measured "#cached-token: 0" for weeks without being able to say which link produced it. R11 -- unified_radix_cache.py, sgl-project#841 host-only insert declined. This is the sgl-project#841 law itself: it marks the whole fetched tail unclaimed and returns. It logged at logger.DEBUG, i.e. INVISIBLE on every boot this strand runs. Raised to warning, rate-limited, and the line now says the declined tail is released rather than published. Cheapest row on the list: a level hub, no new logic. The counter it increments already existed. R13 -- hicache_storage.py, empty KV prefix. #1028B fires only when a COMPONENT cap moved the number (final_pages != kv_pages). When the KV prefix is itself 0 the claim is already zero and NOTHING printed, so "storage genuinely holds nothing for this key" and "everything is fine" were the same log. Now named as an honest cold miss, explicitly distinguished from a capped claim. W12 -- staging_write_ring.py, refused admission. refused/refused_tokens have been maintained all along, but stats() -- their only exposure -- has ZERO callers in the tree, so a write-through page refused for want of staging room left no trace anywhere. Nothing can be READ back that was never WRITTEN, which makes a silent write-side refusal indistinguishable from a read-side miss to whoever debugs the read path afterwards. Emitted at the refusal itself rather than waiting for a reporting site to be invented. NOT built, named only (register lines, per the census): R6 symmetric participation vote returning zero silently; R15/R16 match validators and dead-node stop (sgl-project#904 exists but is unarmed); W9 -- under write_through _evict_device_leaf DELETES a childless node outright, taking its mamba anchor with it, which is a correctness candidate in its own right and not an instrumenting job. Check: the failure class here is a NameError on a logger or on an uninitialised rate-limit attribute, so the check was py_compile on all three files, a grep asserting each module defines `logger`, and an AST scope check that every name the three new emitters reference (keys, tokens, matched_length, total_len, node) is genuinely bound in its enclosing function. All pass.
… per cutover wave The emission gate was `_dpc_emitted % every == 0` on a process-lifetime counter with `every` = the shared sgl-project#904 knob (default 64), so the FIRST [sgl-project#939 double-prefill] line of a boot needed the 64th recording call. Measured on boot_855_weg1b2 (2026-09-02): two readmit waves, a handful of calls, ZERO lines -- and the acceptance read that absence as "half B is not wired" instead of as a rate limit it could not clear. The section-5 contract is ONE CENSUS PER CUTOVER, so the floor is one LINE per cutover wave: `_dpc_wave_emitted` forces the first observation after `reset_double_prefill_census` (the only thing that ends a census) out, and the periodic sampling plus the always-on breach rule are unchanged on top. Also documents the launcher knob HICACHE_EXTRA (sgl-project#810 store watermark, default {"max_size":"200G"}) in docs/dev/WEG1_1068.md section 4; the tree refuses staging without a watermark (lru_file_evictor.py:151) and boot weg1b1 died on exactly that. Evidence (DESK-PROVEN, hermetic, CUDA_VISIBLE_DEVICES=""): red on 8bf12cf: 2 failed ("0 not greater than or equal to 2 ... from 2 waves") green with the fix: 2 passed mutant (drop the wave-flag reset in reset_double_prefill_census): 1 failed ruff check: All checks passed! ruff format --check: 2 files already formatted
…NG_ANTHROPIC_INLINE_SYSTEM_IN_PLACE, default off) Claude Code 2.1.280 sends mid-conversation role:"system" messages (api_system: tool additions/removals, output_config). With the Qwen3.8 template (head-only system turn) the Anthropic adapter hoisted them into the head system turn, so the first turn that carries one re-renders every token behind the system text. Measured on D: radix sgl-project#1420 WALK-STOP and the arena key chain (sgl-project#1439 leading_complete) both stop at the end of the system turn -- depth 3876 (dkr27bnvfp4bar1agent09252328, weg2-14-16/-17) and 4655 (dkr27bbar1final09260145, weg2-12-12), stored (13,248046)=".<|im_end|>", new (13,198)=".\n" (the "\n".join of the hoist). The sgl-project#904 census then reads MambaComponent:absent at that depth, X-GATE W31, X-REQUEUE, one P epoch. The front's request_text renders the same message in place and credits the whole previous turn (presence_span 19335/20168), so it routes SHORT. Switch on: non-leading inline system messages become a user turn at their position, wrapped in <system-reminder>; the leading run is still merged into the head. Must be set identically on P and D (handoff keys are P's tokenization). Off = byte-identical to before. Tests (test27b.sh, CPU): test_inline_system_in_place.py 5 passed (off pins the break at ".\n", on pins turn N as a rendered prefix of turn N+1); anthropic/test_serving.py + test_anthropic_557.py + test_anthropic_rid_h90.py + weg2/test_weg2_messages_wire_rc7_0925.py: 100 passed together. UNIFY (UN6, Operator 26.09.): pick of desk/27b-mambaend-0926 (2303948, MZ, base 0644c6f) after the sgl-project#49-rest port, clean. dup_defs_gate: 0 new. test27b.sh: test_inline_system_in_place + anthropic/test_serving + test_anthropic_557 + test_anthropic_rid_h90 91 passed, 9 subtests.
…fork, not at N-1 (--fork-anchor-token, default off) Boot dkr27brc10bar1agent09261821: 9 of 78 D-direct requests prefilled 1.2k-4.7k tokens (weg2-12-17, 16-24, 20-31, 20-32, 22-36, 22-42, 24-44, 26-55, 26-67). Each is a SIBLING of a request just served: same chat up to the predecessor's last message, then another ~155-token message instead of the generation prompt <|im_start|>assistant\n<think>\n (5 tokens). D's own match names the fork: sgl-project#904 reached=31441 against weg2-10-14's 31446 (41677/41682, 48867/48872, 73895/73900, 89902/89907 -- always N-5). P's only tail anchor is the END anchor at N-1 (P-TRIM-END-ANCHOR, trim=1), 4 tokens past the fork, so the read fell back to the previous anchor (#1028B FETCH CAP kv=31441 claimed=27384 caps={mamba: 27384}). 8 of 9 arrived while the predecessor was still decoding; no anchor at "prompt + generated" existed or would have helped (next turns re-render the output: 26-57 claimed 73898 = P's end anchor of 24-54, not D's finish anchor at 74752). 26-67: D's own prefill track of 26-66 landed at N-1 (extend 1025 = 16*64+1). Switch SGLANG_WEG2_FORK_ANCHOR_TOKEN=<id> on both groups (launcher --fork-anchor-token 248045, needs --p-trim-end-anchor; unset = byte-identical): * P intake cuts a leg-1 prompt at the last fork token of its final 16 ids (N-5 on Qwen3.x) instead of N-1 -- no forward added, D extends 6 tokens instead of 2 in the leg-2 extend it runs anyway; * D's store read ends at the same cut (pure function of ids + env, same on every rank): a fork-cut leg-2 read lands complete, no sgl-project#1324 deferral; * D's extend track of the step reaching the prompt end stays at or below the fork (h at (t-prefix)/64, relative grid; extend only, never decode). Tests (test27b.sh, CPU): new test_weg2_fork_anchor_0926.py 18 passed; red on c51591f (13 failed / 4 passed: the switch had no effect). Real UnifiedRadixCache uni+bigram: sibling claim 0 -> the fork with P's state; D leg 2 resumes at the fork. Affected files 173 passed (trim, tail fold, end anchor hold, store-short, deferral, hold-refetch, dormant hold, P intake, launcher argv). Broader sweep: 471 passed, 15 failed identically on base (gloo tests 580/943b/1290, pre-existing).
…lform (array statt list) -- Admit(0) fuer alle, wenn PP0s Anker am told fehlt rc12z2 (e8a2cd2, P-Log Z. 226312-226760, 03:26:31-41, rid weg2-44-76): PP0 las 47168 Tokens (8x 'sgl-project#1433 L3->L2 fill' -- der Rueckweg, den Runde 4 erst oeffnete), klemmte told auf den Store-Anker 37952 (sgl-project#1416 ANCHOR-CLAMP) und schickte ihn. Beim Zulassen: PP0 'sgl-project#904 ... reached=37952 refused=37952 why=MambaComponent:absent' -> '[sgl-project#928 anchor] REFUSING resume' -> Zulassung bei 0 PP2 dasselbe -> 0 PP1 'sgl-project#988 LOADBACK ... anchor_depth=37952 mamba_restored' -> 37952 PP1 bekam PP0s Zeilen 32768..49152 zu seinem Batch 70720..79688: 'sgl-project#1233 W27 PP WIDTH DIVERGENCE REFUSED ... cause=START-SPLIT', P tot. Genau dafuer gibt es TF (weg2_told_fidelity, Default an): vor dem Admit fragt PP0 den eigenen Baum, kann er am told nicht fortsetzen, geht told=0 mit PF-Fallback-Marke an ALLE Stufen (jede gibt ihren Read frei, alle prefillen ab 0). Auf NF lief TF nie: '#TF told-fidelity probe skipped ... AssertionError((<class 'array.array'>, <class 'list'>))' 10 von 10 auf diesem Boot -- _probe_key baute den Schluessel als list, der Baum vergleicht array('q')-Schluessel (RadixKey.match: assert type(t0) is type(t1)). Keine Aussage -> Admit blieb 37952, waehrend PP0 selbst 0 nahm. Fix (kein Schalter, bestehende Vereinbarung): _probe_key baut array('q'). Die #1416d-Klemme teilt den Schluessel; ein Array-Schluessel hasht dieselben Seiten (compute_node_hash_values hasht die Array-Schluessel des Baums). Test test_told_fidelity_array_key_rc12z2 (4; Fake-Baum vergleicht mit dem ECHTEN RadixKey.match gegen einen Array-Knoten, told 37952, Bigram): 3 rot auf e8a2cd2 (genau 'AssertionError((array.array, list))', Verdikt (37952, None) statt (0, 0)), gruen mit Fix; Hash-Gleichheit list/array gruen auf beiden. Die TF-Suite 0927 mass nur len(key) und erreichte den Vergleicher nie.
…), not at device+host NF rc12z22 D (dkrnfh91dprsavisnoadoptstbar1dauer09281447, 7d50735) TP0: the three READ-TRACE lines of 14:56:26 (asked=43/286/44 readable=0 why=no-file) are NOT a missing Mamba blob. weg2-10-16's wake read came back complete (prefetch success matched=18240 loaded=50944 = 69184, KV + Mamba state at 69184). Its admission registration read "sgl-project#823 HEAD-VOTE ANCHOR matched=50944 (device 0 + host 50944) anchor_depth=69184": the 18240-token device node carries no recurrent state, the mamba validator cut the device match to 0 (sgl-project#904 refusers=MambaComponent:absent=18240) and host_hit_length counted only 50944. _prefetch_kvcache took len(prefix_indices)+host_hit_length = 50944 as the span start but last_host_node (= best_match_node, depth 69184) for the hash chain -> 286 keys that exist nowhere. Same shape for weg2-4-11 (2560 device + 73472 host, the SAME first stem bf3b9106.. asked at 14:56:01 and 14:56:26), weg2-11-18 (2560 + 73344) and weg2-8-15 (18240 + 45312, asked=286). Each read ended store-short / prefetch_pending: 5 s of X-DEFER until the X gate priced the true remainder (uncached=42/161/204). The four hypotheses (write behind, off-grid anchor, KV/Mamba key schema, eviction of the blob) do not apply: PARK-DEMOTE had put every KV page on L3 (l3_pages=1081 of 1081). - scheduler: _weg2_prefetch_span_start -- on a group without PP and a tree with a controller, a state_anchor_depth deeper than device+host is the span start (the depth of last_host_node, where its last hash is); the HP1 span base follows. SGLANG_WEG2_PREFETCH_FROM_ANCHOR=0 = the old start. Group P's sgl-project#1400 told arithmetic is untouched (pp_size > 1 keeps the old start). Line "FA PREFETCH-FROM-ANCHOR rid= matched= (device + host) anchor=". - Nothing is recomputed that was stored: the read now asks exactly the tail beyond the stored prefix (weg2-10-16: 41 tokens instead of 18281 bogus ones). Tests (test27b.sh, hermetic, real _prefetch_kvcache on a recording tree): test_weg2_prefetch_from_anchor_0928 6 passed (weg2-10-16 asks [69184, 69225) under the hash of 69184; switch off reproduces the metal's 286-page ask from 50944; HP1 span_base = anchor; no deeper anchor / PP group / no controller keep the old start; the line), 3 red on a94aea5. Neighbours green: 946 clamp, hold_prefetch_sleep_h91e, dormant_hold_1445, prefetch_deferral_1068 (111).
…dit -- Bruecke f833fcb: D schrieb den Anker 37376 in 16-46s eigenem Extend ('sgl-project#1469 RETAIN rid=weg2-16-46 ... cache_len=37376', 19:50:38), 16-48 setzte darauf auf (19:50:51, '#cached-token: 37376'). Der D-Schlaf 19:51:15-16 verweigerte die Host-Sicherung ('sgl-project#1421 BACKUP-REFUSED why=mamba_pin ... pins=8/8', danach 'why=mamba_claim' hinter 'sgl-project#1427 ARENA-CLAIM REFUSED ... no free slot' der 32-Slot-Arena) und setzte den Baum zurueck ('sgl-project#1470 FLUSH-PUBLISH issued=13 unbacked_left=1', 'Cache flushed'). Nach dem Wake gab der Store den KV zurueck (store_hit=37376), aber keinen Anker ('[sgl-project#904 match-census] ... MambaComponent:absent=37376'); die Front preiste weg2-16-54 trotzdem SHORT (presence 35942 src=d_leg2_cached) -> 17 s D-Sitz, dann W31 am X-Gate (38291 Tok) und W50-Umweg ueber P (12 solche in rc12z26, 8 auf der Bruecke). Das X-Gate auf D preist schon die anker-nutzbare Tiefe (W31), die Front nicht. Fix: (a) Instrument: sgl-project#1421 BACKUP-REFUSED nennt depth= und rid= (weg2_node_depth, weg2_anchor_rid). (b) Der Flush zaehlt vor tree_cache.reset() jeden nur-Device-Mamba-Anker mit Tiefe (UnifiedRadixCache.weg2_unbacked_anchors, 'WEG2-ANCHOR-LOST at=flush n= depths= rids='); der Scheduler haelt die Tiefen bis zur Schlafantwort (weg2_take_anchors_lost, einmal lesen). (c) Die Gruppenzaun-Stimme des release-Legs traegt anchors_lost (Modulfunktion _weg2_anchors_lost_for_leg, [] auf jedem anderen Leg und ohne Scheduler-Ledger), die Reduktion vereinigt, ReleaseMemoryOccupationReqOutput.anchors_lost, der Tokenizer-Merge vereinigt (Feld fehlt, wenn nichts verloren -> alte Antwortform). (d) Front.flip liest nach completed_tags die Schlafantwort von D und nimmt jeden SpanLRU-Eintrag zurueck, dessen Kredit auf einer verlorenen Tiefe stand (depth_cap, sonst gemessene ct), 'WEG2 PRESENCE-ANCHOR-LOST'; ein Eintrag auf einem ueberlebenden Anker behaelt ihn. Zu viel Zuruecknehmen ist sicher (voller Preis = Preis des X-Gates). Die Rettung des Ankers ueber den Schlaf bleibt bei sgl-project#259 4c. Luecke: Arena-Verdraengung ohne L3 ausserhalb des Flushs erfasst der Ledger nicht (nur die Host-Kopie-Pruefung). Tests test_weg2_anchor_lost_presence_0928 (10): alle rot auf 1ea8133, gruen mit Fix. Nachbarn 18 Dateien (1424g, H113, H74, R12, sgl-project#59, X-exact, Spans, sgl-project#1470, sgl-project#49/P49, flip_cost_1235, 1273 x2, sched_fix3, W35, Profil, H91e, sgl-project#248): 298 gruen auf Fix und Basis, dieselben 5 Altlasten (VramCredit x3, sched_fix3 b4a/b4c).
…dit -- Bruecke f833fcb: D schrieb den Anker 37376 in 16-46s eigenem Extend ('sgl-project#1469 RETAIN rid=weg2-16-46 ... cache_len=37376', 19:50:38), 16-48 setzte darauf auf (19:50:51, '#cached-token: 37376'). Der D-Schlaf 19:51:15-16 verweigerte die Host-Sicherung ('sgl-project#1421 BACKUP-REFUSED why=mamba_pin ... pins=8/8', danach 'why=mamba_claim' hinter 'sgl-project#1427 ARENA-CLAIM REFUSED ... no free slot' der 32-Slot-Arena) und setzte den Baum zurueck ('sgl-project#1470 FLUSH-PUBLISH issued=13 unbacked_left=1', 'Cache flushed'). Nach dem Wake gab der Store den KV zurueck (store_hit=37376), aber keinen Anker ('[sgl-project#904 match-census] ... MambaComponent:absent=37376'); die Front preiste weg2-16-54 trotzdem SHORT (presence 35942 src=d_leg2_cached) -> 17 s D-Sitz, dann W31 am X-Gate (38291 Tok) und W50-Umweg ueber P (12 solche in rc12z26, 8 auf der Bruecke). Das X-Gate auf D preist schon die anker-nutzbare Tiefe (W31), die Front nicht. Fix: (a) Instrument: sgl-project#1421 BACKUP-REFUSED nennt depth= und rid= (weg2_node_depth, weg2_anchor_rid). (b) Der Flush zaehlt vor tree_cache.reset() jeden nur-Device-Mamba-Anker mit Tiefe (UnifiedRadixCache.weg2_unbacked_anchors, 'WEG2-ANCHOR-LOST at=flush n= depths= rids='); der Scheduler haelt die Tiefen bis zur Schlafantwort (weg2_take_anchors_lost, einmal lesen). (c) Die Gruppenzaun-Stimme des release-Legs traegt anchors_lost (Modulfunktion _weg2_anchors_lost_for_leg, [] auf jedem anderen Leg und ohne Scheduler-Ledger), die Reduktion vereinigt, ReleaseMemoryOccupationReqOutput.anchors_lost, der Tokenizer-Merge vereinigt (Feld fehlt, wenn nichts verloren -> alte Antwortform). (d) Front.flip liest nach completed_tags die Schlafantwort von D und nimmt jeden SpanLRU-Eintrag zurueck, dessen Kredit auf einer verlorenen Tiefe stand (depth_cap, sonst gemessene ct), 'WEG2 PRESENCE-ANCHOR-LOST'; ein Eintrag auf einem ueberlebenden Anker behaelt ihn. Zu viel Zuruecknehmen ist sicher (voller Preis = Preis des X-Gates). Die Rettung des Ankers ueber den Schlaf bleibt bei sgl-project#259 4c. Luecke: Arena-Verdraengung ohne L3 ausserhalb des Flushs erfasst der Ledger nicht (nur die Host-Kopie-Pruefung). Tests test_weg2_anchor_lost_presence_0928 (10): alle rot auf 1ea8133, gruen mit Fix. Nachbarn 18 Dateien (1424g, H113, H74, R12, sgl-project#59, X-exact, Spans, sgl-project#1470, sgl-project#49/P49, flip_cost_1235, 1273 x2, sched_fix3, W35, Profil, H91e, sgl-project#248): 298 gruen auf Fix und Basis, dieselben 5 Altlasten (VramCredit x3, sched_fix3 b4a/b4c).
…nn sein Span schon im Baum liegt (y5a weg2-19-28) NF-Operator 30.09. (Mamba/GDN-Praefixzustaende gehen verloren, obwohl L3 Platz hat). Kette aus dem D-Log y5a (0110f8e132_0930_174039, TP0): - 17:48:10 der Hold-Read von weg2-19-28 findet mit EIGENEN Schluesseln (sgl-project#1442 HANDOFF-KEYS NONE) den Sessions-Praefix im Store: "#1028B FETCH CAP kv=675 claimed=665 ... anchors_in_range mamba (9, 664)" = 42560 Token MIT Anker auf Seite 664. Schluessel-Identitaet ist NICHT die Luecke (18-24 las denselben Praefix mit eigenen Schluesseln: store_hit=42240; das "known=640" ist der Nachlese-Rest [42560, 43200) nach dem ersten Read). - 17:48:12 das Geschwister weg2-18-24 wird zuerst zurueckgeladen ("sgl-project#988 LOADBACK ... prefix moved to 43520 ... anchor_depth=43520"): sein KV liegt auf dem DEVICE, Zustand nur bei 43520. - 19-28's Read endet (completed=42560): _insert_helper_host findet den ganzen Span schon im Baum, legt keinen Host-Knoten an (inserted_host_node None: der Knoten hat keine Host-KV-Kopie), und MambaComponent.commit_hicache_transfer(PREFETCH) GIBT DEN ANKER FREI. - "[sgl-project#904 match-census] reached=43200 accepted=0 refusers=MambaComponent: absent=43200" -> "[sgl-project#928 anchor] REFUSING ... best_value_len=0 NONE-ON- THIS-PATH" -> W31 -> W50 x_refusal_midstream, 43396 Token zu P. Fix: _insert_helper_host nennt den Knoten, an dem sein Schluessel ENDET (InsertResult.matched_end_node; der Lauf hat ihn dort gespalten = Tiefe des Ankers). Traegt dieser Knoten keinen Zustand (weder Device noch Host), nimmt er den gelesenen Anker (host_value, Host-LRU) statt der Freigabe. Ein Knoten MIT Zustand behaelt seinen, die Kopie wird wie bisher freigegeben. Der naechste Match liest: best_match_node = der Knoten, state_anchor_depth = seine Tiefe, mamba_host_hit_length = 1 -> init_load_back stellt den Zustand in den Slot des Requests und nimmt den Device-KV darueber (der sgl-project#988-Weg). Marker "WEG2 PREFETCH-ANCHOR-ATTACH". Schalter SGLANG_WEG2_PREFETCH_ANCHOR_ATTACH (environ.py, default an). Tests: test_weg2_prefetch_anchor_attach_0930.py 6 (echter UnifiedRadixCache ueber echtem HybridReqToTokenPool; 5 rot auf 8406e68). Nachbarn (anchor_drop_l3_257ii, arena_mamba_direct_1427, prefetch_* 7 Dateien, short_read_anchor_sa, mamba_hicache_issuance_869b, mamba_gates_the_hit_706, unified_radix_hicache_dispatch, unified_mamba_views, boundary_anchor_792, mamba_anchor_seams_747): 152 gruen; 905 double_frees + 747 prepare_gates rot auch auf der Basis (fremd).
…asses over the sgl-project#243/sgl-project#248 keep (z30y12 epoch 50: six park resumes W31, 381k tokens to P) Metal z30y12 (df1f815, 30.09.): P PP0 "ARENA-EVICT n=11 want=256" at 23:35:32 (the L3 fill / arena write callers floor want at 256 slots); 3 s later the MAMBA arena read "ARENA-REF-CENSUS arena-78446592.bin slots=112 complete=0" on every P rank, D's MAMBA census went park_kept=6/12 -> 0/12. At the 23:35:44 wake all six D-parked requests read their KV back ("HiCache prefetch success ... loaded=53503") but no anchor ("[sgl-project#904 match-census] verdict=refused ... refusers=MambaComponent:53503 why=MambaComponent:absent"), the X gate priced each whole (X-GATE-TERMS head=0 store=0 floor=0 -> W31) and W50-REROUTE sent 381501 tokens to P -- the flip of epoch 50. weg2-35-112 came through 4 s later with uncached=150 after one anchor page was filled back from L3 (sgl-project#1433, TP1 23:35:47). The second collapse (D TP0-2 "ARENA-EVICT want=256" 23:42:10, MAMBA complete 110 -> 8) has the same shape. Root: HiCacheFile._arena_evict_to_disk evicts min(want, every unreferenced complete slot) with only the pins kept. want >= 256 is larger than the 112-slot MAMBA arena, and the sgl-project#248 park / sgl-project#243 hand-off pages are kept by ORDER, not by reference -- so one round took them all. The claim path (_evict_for_claim) has passed over that keep since sgl-project#243/sgl-project#248; the clock evict never did. Fix: (1) want is capped at an eighth of the arena's slots (FULL/draft arenas: 90112, i.e. unchanged for every caller; MAMBA: 14); (2) the host pools register themselves on the backend at bind (register_keep_pool), and the clock evict passes the pool's handoff_pending.keep_for keys as keep_lo (stage i only: kept pages are never its candidates). Tests (CUDA_VISIBLE_DEVICES="" nice -n 19 ionice -c3): test_evict_keep_mamba_arena_0930.py (real ShmArena): base 2 failed (the unregistered case is behavioural: 8 of 8 slots evicted), fix 2 passed. Mutant A (keep_lo dropped): 1 failed (the parked anchor leaves L2). Mutant B (cap dropped): 2 failed. Affected files (park_l3_248, arena_reset_release_h81, prefix_trace_in, probe_hold_257, w3_arena_spill, arena_host_pool_1424, arena_l3_fill_group_rc12z, arena_l3_fill_1433, l3_reuse_after_restart, hicache_shm_arena_1404, mamba_arena_displace_h19, new file): 94 passed, 4 failed -- the same 4 fail on the base df1f815/92c9c18020 sources (arena_host_pool_1424 x3, arena_l3_fill_group_rc12z x1), not touched here.
… done NF bc35462, P PP0, weg2-56-163 (lane 1, 142861 tokens): a short store read marks the operation terminated inside the KV transfer; the io thread reads the recurrent state at the deepest landed anchor only afterwards (_short_read_anchor_transfer, extra_ms=371) and then sets pool_transfers_done. can_terminate_prefetch reaped on `terminated` in between: _weg2_short_anchor_pages was still None, sgl-project#257 cut on the store's presence answer (89472), a stateless prefix was inserted, PP0's walk refused it (sgl-project#904 MambaComponent:absent), #TF put told=0 on the wire and P re-prefilled all 142861 tokens (36 s, cached_tokens=0). Measured in that boot: 3/3 #TF fallbacks (weg2-12-41, weg2-25-63, weg2-56-163) were reaped BEFORE their SA line; 7/7 SA-before-reap reads resumed at the anchor. Fix: while a started read's component pools are still in transfer, the termination is held via the H45 hold slot (group-uniform MAX), bounded by the operation's own timeout budget. Queued-never-started and pool-less reads are unaffected. Tests: py_compile OK; test_short_read_anchor_reap_race_1014.py (5 cases).
NF h13 84e787e (boot 1014_135753), P, weg2-10-82 (59163 tokens): all 921 KV pages landed (loaded=58944), yet sgl-project#904 MambaComponent:absent =58944 and #TF told=0 -> full prefill. The anchor read of the 32-slot mamba arena found it FULL on every PP rank. Each rank's ROOM-WAIT evicted and re-claimed; PP2 (pid854) won the freed slot ('ROOM-WAIT claimed=1'), PP0 and PP1 got status 1 ('L3-FILL JOINED … holders=[26/7ms/pid854/l3fill…]', 'ROOM-WAIT claimed=0 still_full=0'). _await_fill_room unclaimed the join and DROPPED it: a miss, the anchor never read on PP0, the stateless KV prefix was refused at admission. PP2's fill completed within ms. Fix: a status-1 re-claim inside the room wait is unclaimed and appended to `joined`, so the JOIN wait that runs right after awaits the peer's fill like every other join. Tests: py_compile OK; test_l3fill_join_wait_0930.py +1 (peer wins the freed slot, the fill gets it through the join wait).
…'s mamba anchor NF h13 84e787e (boot 1014_135753), D TP0 14:35:34, weg2-24-181 (lane 1, 149125 tokens): the LANE-PREEMPT park of six lane-0 requests made room for their park chains in the full 32-slot mamba arena and spilled the arriving lane-1 request's own resume anchor to L3 ('MAMBA-ARENA DISPLACE n=93 why=park_first node=673 depth=148992', 'PARK-END-ANCHOR-FIRST SPILL n=93 node=691 depth=70400 victim_depth=148992'). D's match then refused the request ('sgl-project#904 refusers=MambaComponent:148992 why=absent'), X-GATE head=0 store=0 W31, W50 to P, 12 s stall with 10 open requests. - a lane >= 1 request's finish insert stamps every node of its sequence with the highest lane that finished through it (weg2_lane); lane 0 stamps nothing (no walk); - a park chain carries the PARKED request's lane (weg2_park_lane, at PARK-END-ANCHOR-FIRST MARK) and claims with it; - pick_park_victim / pick_last_resort_victim / the flush spill displace only anchors of a lane <= the claimer's, lowest lane first, then the old order (shallowest, node id); a chain node that finds none goes KV-only as before -- the lower lane pays. SGLANG_WEG2_LANE_ANCHOR_RANK (default on); off = the old pick order byte for byte; lanes off = every node lane 0 = the old order. Tests: test_weg2_lane_anchor_rank_1014.py (6) + park_end_anchor_first_1303, mamba_last_resort_1008, mamba_arena_displace_h19, mamba_anchor_bypass_1014: 45 passed (pytest_gedeckelt.sh).
…'s mamba anchor NF h13 84e787e (boot 1014_135753), D TP0 14:35:34, weg2-24-181 (lane 1, 149125 tokens): the LANE-PREEMPT park of six lane-0 requests made room for their park chains in the full 32-slot mamba arena and spilled the arriving lane-1 request's own resume anchor to L3 ('MAMBA-ARENA DISPLACE n=93 why=park_first node=673 depth=148992', 'PARK-END-ANCHOR-FIRST SPILL n=93 node=691 depth=70400 victim_depth=148992'). D's match then refused the request ('sgl-project#904 refusers=MambaComponent:148992 why=absent'), X-GATE head=0 store=0 W31, W50 to P, 12 s stall with 10 open requests. - a lane >= 1 request's finish insert stamps every node of its sequence with the highest lane that finished through it (weg2_lane); lane 0 stamps nothing (no walk); - a park chain carries the PARKED request's lane (weg2_park_lane, at PARK-END-ANCHOR-FIRST MARK) and claims with it; - pick_park_victim / pick_last_resort_victim / the flush spill displace only anchors of a lane <= the claimer's, lowest lane first, then the old order (shallowest, node id); a chain node that finds none goes KV-only as before -- the lower lane pays. SGLANG_WEG2_LANE_ANCHOR_RANK (default on); off = the old pick order byte for byte; lanes off = every node lane 0 = the old order. Tests: test_weg2_lane_anchor_rank_1014.py (6) + park_end_anchor_first_1303, mamba_last_resort_1008, mamba_arena_displace_h19, mamba_anchor_bypass_1014: 45 passed (pytest_gedeckelt.sh).
…ore -- read just the blob NF h13 84e787e (boot 1014_135753), D 14:35:36, weg2-24-181 (lane 1, 149125 tokens): the match walked the KV to 148992 but refused every node for a missing Mamba state (sgl-project#904 MambaComponent:absent=148992) -- the lane-0 park had spilled the anchor to L3 (DISPLACE why=park_first on_disk=1) while the KV stayed in the tree. The store held the anchor but not the KV (D's pages were device-only, PUBLISH-SWEEP unbacked=13), so the ordinary prefetch from the root could not deliver; W31, W50, P prefilled the whole prompt. (c') removes that displacement; this is the net. UnifiedRadixCache.weg2_anchor_only_read(req), called by the scheduler's _prefetch_kvcache right after the re-match: when key_match_depth > state_anchor_depth, the node at key_match_depth carries no Mamba state and has a last-page hash, the GDN-holding rank reads ONLY that anchor through the existing mamba-pool transfer (PoolTransfer MAMBA, TRAILING_PAGES, alloc_read placeholder) and batch_get_v2 -- arena in place, L3 fill, or the ANCHOR-BYPASS staging row -- and attaches it as the node's host Mamba value (the PREFETCH-ANCHOR-ATTACH form, host LRU); the re-match then accepts it (MAMBA-HOST-RESUME) and the load-back moves the state only. No new prefetch op and no new collective: a rank without GDN state never reads (rank_holds_gdn_state), and the group's head vote stays the MIN of the ranks' matches, so a rank whose read failed keeps the group where it was. One read per depth per request. A failed read releases its row. SGLANG_WEG2_ANCHOR_ONLY_READ (default on). Tests: test_anchor_only_read_e1_1014.py (5: read+attach, non-GDN rank never reads, failed read releases and is not re-read, nothing to do / switch off, and a source-level MUTANT with the GDN guard removed that the rank-agreement invariant kills) + anchor_bypass_1014, prefetch_anchor_attach_0930 (+full_unbacked), prefetch_anchor_pin: 27 passed.
…ore -- read just the blob NF h13 84e787e (boot 1014_135753), D 14:35:36, weg2-24-181 (lane 1, 149125 tokens): the match walked the KV to 148992 but refused every node for a missing Mamba state (sgl-project#904 MambaComponent:absent=148992) -- the lane-0 park had spilled the anchor to L3 (DISPLACE why=park_first on_disk=1) while the KV stayed in the tree. The store held the anchor but not the KV (D's pages were device-only, PUBLISH-SWEEP unbacked=13), so the ordinary prefetch from the root could not deliver; W31, W50, P prefilled the whole prompt. (c') removes that displacement; this is the net. UnifiedRadixCache.weg2_anchor_only_read(req), called by the scheduler's _prefetch_kvcache right after the re-match: when key_match_depth > state_anchor_depth, the node at key_match_depth carries no Mamba state and has a last-page hash, the GDN-holding rank reads ONLY that anchor through the existing mamba-pool transfer (PoolTransfer MAMBA, TRAILING_PAGES, alloc_read placeholder) and batch_get_v2 -- arena in place, L3 fill, or the ANCHOR-BYPASS staging row -- and attaches it as the node's host Mamba value (the PREFETCH-ANCHOR-ATTACH form, host LRU); the re-match then accepts it (MAMBA-HOST-RESUME) and the load-back moves the state only. No new prefetch op and no new collective: a rank without GDN state never reads (rank_holds_gdn_state), and the group's head vote stays the MIN of the ranks' matches, so a rank whose read failed keeps the group where it was. One read per depth per request. A failed read releases its row. SGLANG_WEG2_ANCHOR_ONLY_READ (default on). Tests: test_anchor_only_read_e1_1014.py (5: read+attach, non-GDN rank never reads, failed read releases and is not re-read, nothing to do / switch off, and a source-level MUTANT with the GDN guard removed that the rank-agreement invariant kills) + anchor_bypass_1014, prefetch_anchor_attach_0930 (+full_unbacked), prefetch_anchor_pin: 27 passed.
… Mamba-Arena-Ablehnung, kein Spill auf P (PP>1), kein ANCHOR-LOST auf Form-A-Workern, Nachfolger nach dem Tombstone A1/B1: _1421_refused merkt den Grund (_k15_last_refusal); die Sicherung spillt nur nach mamba_claim / anchor_only_claim -- nach KV arena_claim, Pin-Budget oder write_none kann der zweite Versuch nicht gelingen (Boot _215812: D TP0 38 solche neben 161 mamba_claim, P 51x arena_claim). A2/B2: auf P (pp_size > 1) ist der Mamba-Slot zwischen den Stufen geteilt -- _weg2_evict_spill_room spillt dort nicht (auch nicht im P-FUND-Pfad), Zeile K15 EVICT-SPILL SKIPPED pp_size= pp_rank=; die Sicherung selbst bleibt. A3/B3: ein Form-A-Worker (_r12.role() == worker) sichert nicht und schreibt kein WEG2-ANCHOR-LOST (TP0 entscheidet und sichert). A4/B4: drive_eviction frischt den Nachfolger erst NACH _evict_component_and_detach_lru/_cascade_evict auf (wie der Blatt-Zweig). A5: die Review-Proben R1-R5 in die Suite uebernommen. Tests: test_weg2_k15_evict_anchor_save_1016.py 20 gruen, auf 29162a4 7 rot (A1 x4, A2, A3, A4). Mutanten: Spill bei jeder Ablehnung ROT 4, Spill auf PP>1 ROT 1, Worker-LOST ROT 1, Auffrischen vor dem Tombstone ROT 1, sgl-project#904-Guard weg (Review-M2) ROT 2.
…3 at the next hit, not recomputed (SOLL R7 R11) S109c (81037f0, _s109c_aux_device_round) releases from a device-KV node the Full host rows AND the mamba anchor once the write-behind's record holds the L3 copy of both (secure_rows_to_l3 + l3_complete, _secure_child_anchors None). The node keeps its device KV with no state anywhere (rc29 D TP0: 6x S109c W3-AUX-DEVICE SPILL 02:25:49-02:29:33Z). The next hit: the match refused it (sgl-project#904 MambaComponent:absent), fell back to the last state above, and e1 returned 0 silently for exactly this shape (AUX-HOST-LAW: no host state on a node without a Full host copy) -- every token behind the last state was recomputed although L3 held the anchor; e1 was asked only at intake and at the deferral retry. - e1 (_pdflip_aor_read_attach): device KV without a Full host copy -> S109R device read: the blob by the node's last page hash (arena slot / L3 fill / ANCHOR-BYPASS row) into a device mamba slot, S90b's device attach (note injectable). Only where no peer holds a mamba replica (Form A TP0 / local-PP floor); otherwise named how=peer_replicas (#639b). Nothing is written, no collective; like e1 it only raises this rank's match, the group MIN stays the verdict (R11). - Admission match asks too (group D, pp_size 1): _s109r_admission_anchor_read right after the match in _get_new_batch_prefill_raw, re-match after an attach (one read per depth per request, e1's mark). - H105g: anchor_dropped -- the matched host anchor (mamba_host_hit_length > 0) released by S109c inside the room step -> NO_TOKEN in the gathered vote; the next pass reads it back (S109R) instead of loading back without it and recomputing. - Live witness: "S109R ANCHOR-DEVICE-READ n= rid= depth= how= attached=" (attached=0 how=store_miss/ copy_failed/no_device_slot/peer_replicas = recomputed despite L3); the S109c line now names ids=<id>@<depth> for pairing. Tests (serial, pytest_gedeckelt.sh, CUDA_VISIBLE_DEVICES=""): test_nf_rc29_s109c_anchor_reread_s109r_1021.py 7/7 + test_nf_rc29_loadback_target_dropped_h105g_1021.py 9/9 green; RED on 7933168: 9 failing (key: after a real S109c release e1 returns 0, no read; anchor-in-window loads back with the anchor gone). Mutant: rank-alone guard removed -> reads on a replicated group (caught). Neighbours e1/S90b/S43/S104/ S109/S92/Form A: 294 passed. All 53 admission-loop test files: failure set identical to a773b4e (57 pre-existing failures, none new).
when running the latency benchmark on a L4 with 24GB of VRAM, the error was not well handled. See before and after below:
before:
after