Skip to content

fixed an error handling in bench_latency.py - #904

Merged
Ying1123 merged 2 commits into
sgl-project:mainfrom
etched-ai:main
Aug 4, 2024
Merged

Ying1123 merged 2 commits into
sgl-project:mainfrom
etched-ai:main

Conversation

@min-xu-et

Copy link
Copy Markdown
Contributor

when running the latency benchmark on a L4 with 24GB of VRAM, the error was not well handled. See before and after below:

before:

[rank0]: Traceback (most recent call last):
[rank0]:   File "<frozen runpy>", line 198, in _run_module_as_main
[rank0]:   File "<frozen runpy>", line 88, in _run_code
[rank0]:   File "/home/min/.cache/sglang/python/sglang/bench_latency.py", line 323, in <module>
[rank0]:     main(server_args, bench_args)
[rank0]:   File "/home/min/.cache/sglang/python/sglang/bench_latency.py", line 288, in main
[rank0]:     work_func(server_args, bench_args, 0)
[rank0]:   File "/home/min/.cache/sglang/python/sglang/bench_latency.py", line 272, in latency_test
[rank0]:     run_once(4)
[rank0]:   File "/home/min/miniconda3/envs/sgl/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
[rank0]:     return func(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/min/.cache/sglang/python/sglang/bench_latency.py", line 234, in run_once
[rank0]:     next_token_ids, _, batch = extend(reqs, model_runner)
[rank0]:                                ^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/min/.cache/sglang/python/sglang/bench_latency.py", line 157, in extend
[rank0]:     batch.prepare_for_extend(model_runner.model_config.vocab_size, None)
[rank0]:   File "/home/min/.cache/sglang/python/sglang/srt/managers/schedule_batch.py", line 383, in prepare_for_extend
[rank0]:     self.tree_cache.evict(extend_num_tokens, self.token_to_kv_pool.free)
[rank0]:     ^^^^^^^^^^^^^^^^^^^^^
[rank0]: AttributeError: 'NoneType' object has no attribute 'evict'

after

[gpu=0] Load weight end. type=LlamaForCausalLM, dtype=torch.float16, avail mem=9.30 GB
[gpu=0] Memory pool end. avail mem=4.27 GB
[gpu=0] Capture cuda graph begin. This can take up to several minutes.
max_total_num_tokens=10085
max_batch_size=13
Prefill out of memory. Try to lower your batch size.

@Ying1123 Ying1123 left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Currently, it is impossible to see if self.tree_cache is None. The bench_latency file is a hacky way of running model inference using low-level APIs, which skips the step of creating tree_cache.
However, it is also a concern that the --disable-radix option only skips the computation at the lowest level and does not remove everything like tree_cache. It might be better to turn off everything, but in that case, there might not be only one place to change. We are also planning to refactor the tree_cache in a more modular way.

Regardless, the current change is correct and could be part of the complete change. What do you think? @merrymercy

@min-xu-et

Copy link
Copy Markdown
Contributor Author

for benchmarking, maybe it is actually good to be able to run without the cache? That should make the benchmark more deterministic and not affected by caching? IMHO, it is better to have the ability to run things with least dependencies unless tree cache is such a core thing and srt simply can't run without it?

@Ying1123

Ying1123 commented Aug 3, 2024

Copy link
Copy Markdown
Contributor

for benchmarking, maybe it is actually good to be able to run without the cache? That should make the benchmark more deterministic and not affected by caching? IMHO, it is better to have the ability to run things with least dependencies unless tree cache is such a core thing and srt simply can't run without it?

Yes, agree.

@Ying1123 Ying1123 changed the title fixed an error handling fixed an error handling in bench_latency.py Aug 4, 2024
@Ying1123
Ying1123 merged commit 7dd8a7e into sgl-project:main Aug 4, 2024
timethink pushed a commit to timethink/sglang that referenced this pull request Mar 9, 2025
McZyWu pushed a commit to McZyWu/sgl-sglang that referenced this pull request Jul 2, 2026
Modify Multimodal Pressure Test Parameters in the Adaptation Scripts.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 26, 2026
…d never joined the transfer that fills it) into the flip train
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 26, 2026
…, so 411 declines have no verdict

Routing from the write-side agent (sgl-project#872) narrowed Posten 0 correctly: the store
is healthy, 24277 canonical pages live during the R7 window, and the historic
"store is empty" readings probed a path this boot does not use. So the 671
refused walks are a MATCH problem, not a content problem. The question that
follows immediately is the one nobody could answer: a walk that matches nothing
should fall through to an L3 prefetch, and prefetch was attempted on only 264
of 675 sampled walks.

THE OBVIOUS ANSWER IS WRONG, and the code says why. It is tempting to read the
411 as the same MambaComponent refusal declining twice -- one root, two numbers.
It is not:

  * The prefetch anchor IS mamba-gated, but by an aliasing the base contract
    does not describe. base_prefix_cache.py:201-210 separates `last_host_node`
    (the L3 prefetch anchor) from `best_match_node` (the validator-consensus L2
    anchor) -- two fields, two purposes. UnifiedRadixCache sets them equal under
    HiCache (unified_radix_cache.py:1443-1449). Deliberate and commented, but it
    contradicts the field's own documentation, and two fields that are
    documented different and implemented identical are a drift source (sgl-project#747's
    shape). Named, not changed here.
  * Even so, the refusal does NOT decline the prefetch. Total refusal leaves
    best_match_node at the root, and the caller admits the root on purpose:
    `locally_eligible = last_host_node.backuped or last_host_node is
    self.tree_cache.root_node` (scheduler.py:4933-4935). A cold request with no
    match is supposed to prefetch. The first term holds.
  * The second term (>= 256 new tokens) holds for agent prompts of thousands.

That leaves the rate limiter -- AND THERE THE EVIDENCE STOPS, which is the
actual finding. None of the three terms is counted or logged. The 411 declines
are silent. Nobody can attribute them from this boot, and no amount of
re-reading the log will change that.

    eligible = (locally_eligible
                and prefetch_length >= self.prefetch_threshold
                and not self.cache_controller.prefetch_rate_limited())

Three unrelated verdicts wearing one boolean, remedies in three different
files. The sgl-project#914 shape one module over: blame without a defect.

So this is an instrument and NOT a policy change. The three terms are the same
three, evaluated in the same order, with the same outcome; `eligible` is now
`reason is None`, one expression, so the predicate and its explanation cannot
drift apart the way sgl-project#747 records these very lineages drifting. The first
failing term is named rather than all of them -- a request can trip several,
and summing them would double-count exactly as
`refused_tokens_by_component` is documented to.

Not env-gated, unlike the sgl-project#904 match census. That one builds an object and
walks validators a second time, so it pays for itself only when armed. This is
one integer increment on a path that already builds a RadixKey and takes a host
lock -- and a counter you must remember to arm cannot answer "was it ever
tried", which is the whole question. `attempted` is counted alongside the
refusals so the denominator is local (sgl-project#873: a denominator reconstructed from
another log is how a narrowed candidate set reads as a decomposition), and an
unrecorded gate says "no observation" rather than a silent zero (sgl-project#829).

HYPOTHESIS, RECORDED AS ONE. prefetch_capacity_limit is `0.5 *
mem_pool_host.size` (cache_controller.py:729), and that size is not constant
across a phase flip: sgl-project#905 measured the two host tiers at 703472 rows (PP) and
30518 (TP), 23x apart, putting the TP-phase budget near 15259 tokens -- under
four prefetches of the 4096 this window actually completed, against 5
concurrent requests per flip. The order of magnitude fits 411/675, and I am
deliberately not fitting it exactly: the PP phase has ample budget, so the
number cannot come from there alone. If it holds, the same host-tier asymmetry
behind sgl-project#905's double-free is also starving the TP-phase prefetch budget -- one
defect, two effects, to be checked rather than assumed. The counter decides it
on the next boot instead of re-arguing it.

RED-FIRST, four mutants, each red, restored green (14 tests):
  revert to the shipped conjunction (no counter) ......... 4 failed
  eligible computed twice instead of derived from reason . 2 failed
  attempts not counted (denominator lost) ................ 2 failed
  no-observation reported as attempted=0 ................. 1 failed

ONE SELF-INFLICTED DEFECT, found and fixed in the same pass, recorded because
the form is evidently easy to commit. My first version of
test_the_rate_limit_check_is_still_called_at_most_once counted occurrences in
the function's RAW source and went red because a PRE-EXISTING comment at :2674
mentions `prefetch_rate_limited()`. That is the sgl-project#908 substring defect -- prose
used to reach a verdict about code -- in a test of mine, an hour after I
rejected the same form in someone else's patch. Comment lines are now stripped
before counting.

GATE test/registered/unit/mem_cache serial, CVD="": 3 failed, 1950 passed,
1658 skipped. Both axes agree (summary 3, ANSI-stripped extraction 3). Two are
test_acceptance_emitters_758::RefillTiming, baseline-proven pre-existing
earlier in this branch; the third is test_pool_invariant_double_owned_912, a
concurrently-developed test belonging to sgl-project#912 and untouched by this commit.
The sgl-project#904 census suite is green against this change (62 passed together with
the sgl-project#913/sgl-project#914/sgl-project#915 suites), which is the regression that matters: note_refused
gained an optional third argument and its existing callers must not care.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 30, 2026
The census (/spinning/gpu-arb/CENSUS_1035_readthrough_chain.md) listed every
gate on the read-through chain with its instrument or BLIND. Three BLIND rows
needed no new mechanism, only exposure, and all three share one shape: a gate
that can ZERO the read path while printing nothing, so a boot where it fired
and a boot where storage was simply cold produce identical logs. That is how
this campaign measured "#cached-token: 0" for weeks without being able to say
which link produced it.

R11 -- unified_radix_cache.py, sgl-project#841 host-only insert declined.
  This is the sgl-project#841 law itself: it marks the whole fetched tail unclaimed and
  returns. It logged at logger.DEBUG, i.e. INVISIBLE on every boot this strand
  runs. Raised to warning, rate-limited, and the line now says the declined
  tail is released rather than published. Cheapest row on the list: a level
  hub, no new logic. The counter it increments already existed.

R13 -- hicache_storage.py, empty KV prefix.
  #1028B fires only when a COMPONENT cap moved the number
  (final_pages != kv_pages). When the KV prefix is itself 0 the claim is
  already zero and NOTHING printed, so "storage genuinely holds nothing for
  this key" and "everything is fine" were the same log. Now named as an honest
  cold miss, explicitly distinguished from a capped claim.

W12 -- staging_write_ring.py, refused admission.
  refused/refused_tokens have been maintained all along, but stats() -- their
  only exposure -- has ZERO callers in the tree, so a write-through page
  refused for want of staging room left no trace anywhere. Nothing can be READ
  back that was never WRITTEN, which makes a silent write-side refusal
  indistinguishable from a read-side miss to whoever debugs the read path
  afterwards. Emitted at the refusal itself rather than waiting for a
  reporting site to be invented.

NOT built, named only (register lines, per the census): R6 symmetric
participation vote returning zero silently; R15/R16 match validators and
dead-node stop (sgl-project#904 exists but is unarmed); W9 -- under write_through
_evict_device_leaf DELETES a childless node outright, taking its mamba anchor
with it, which is a correctness candidate in its own right and not an
instrumenting job.

Check: the failure class here is a NameError on a logger or on an
uninitialised rate-limit attribute, so the check was py_compile on all three
files, a grep asserting each module defines `logger`, and an AST scope check
that every name the three new emitters reference (keys, tokens, matched_length,
total_len, node) is genuinely bound in its enclosing function. All pass.
efschu pushed a commit to efschu/htsglang that referenced this pull request Sep 2, 2026
… per cutover wave

The emission gate was `_dpc_emitted % every == 0` on a process-lifetime
counter with `every` = the shared sgl-project#904 knob (default 64), so the FIRST
[sgl-project#939 double-prefill] line of a boot needed the 64th recording call.
Measured on boot_855_weg1b2 (2026-09-02): two readmit waves, a handful of
calls, ZERO lines -- and the acceptance read that absence as "half B is not
wired" instead of as a rate limit it could not clear.

The section-5 contract is ONE CENSUS PER CUTOVER, so the floor is one LINE
per cutover wave: `_dpc_wave_emitted` forces the first observation after
`reset_double_prefill_census` (the only thing that ends a census) out, and
the periodic sampling plus the always-on breach rule are unchanged on top.

Also documents the launcher knob HICACHE_EXTRA (sgl-project#810 store watermark,
default {"max_size":"200G"}) in docs/dev/WEG1_1068.md section 4; the tree
refuses staging without a watermark (lru_file_evictor.py:151) and boot
weg1b1 died on exactly that.

Evidence (DESK-PROVEN, hermetic, CUDA_VISIBLE_DEVICES=""):
  red on 8bf12cf: 2 failed ("0 not greater than or equal to 2 ... from 2 waves")
  green with the fix: 2 passed
  mutant (drop the wave-flag reset in reset_double_prefill_census): 1 failed
  ruff check: All checks passed!   ruff format --check: 2 files already formatted
efschu pushed a commit to efschu/htsglang that referenced this pull request Sep 26, 2026
…NG_ANTHROPIC_INLINE_SYSTEM_IN_PLACE, default off)

Claude Code 2.1.280 sends mid-conversation role:"system" messages
(api_system: tool additions/removals, output_config). With the Qwen3.8
template (head-only system turn) the Anthropic adapter hoisted them into the
head system turn, so the first turn that carries one re-renders every token
behind the system text. Measured on D: radix sgl-project#1420 WALK-STOP and the arena
key chain (sgl-project#1439 leading_complete) both stop at the end of the system turn
-- depth 3876 (dkr27bnvfp4bar1agent09252328, weg2-14-16/-17) and 4655
(dkr27bbar1final09260145, weg2-12-12), stored (13,248046)=".<|im_end|>",
new (13,198)=".\n" (the "\n".join of the hoist). The sgl-project#904 census then reads
MambaComponent:absent at that depth, X-GATE W31, X-REQUEUE, one P epoch.
The front's request_text renders the same message in place and credits the
whole previous turn (presence_span 19335/20168), so it routes SHORT.

Switch on: non-leading inline system messages become a user turn at their
position, wrapped in <system-reminder>; the leading run is still merged into
the head. Must be set identically on P and D (handoff keys are P's
tokenization). Off = byte-identical to before.

Tests (test27b.sh, CPU):
  test_inline_system_in_place.py 5 passed (off pins the break at ".\n",
  on pins turn N as a rendered prefix of turn N+1);
  anthropic/test_serving.py + test_anthropic_557.py + test_anthropic_rid_h90.py
  + weg2/test_weg2_messages_wire_rc7_0925.py: 100 passed together.

UNIFY (UN6, Operator 26.09.): pick of desk/27b-mambaend-0926 (2303948, MZ, base 0644c6f) after
the sgl-project#49-rest port, clean. dup_defs_gate: 0 new. test27b.sh: test_inline_system_in_place +
anthropic/test_serving + test_anthropic_557 + test_anthropic_rid_h90 91 passed, 9 subtests.
efschu pushed a commit to efschu/htsglang that referenced this pull request Sep 26, 2026
…fork, not at N-1 (--fork-anchor-token, default off)

Boot dkr27brc10bar1agent09261821: 9 of 78 D-direct requests prefilled
1.2k-4.7k tokens (weg2-12-17, 16-24, 20-31, 20-32, 22-36, 22-42, 24-44,
26-55, 26-67). Each is a SIBLING of a request just served: same chat up to
the predecessor's last message, then another ~155-token message instead of
the generation prompt <|im_start|>assistant\n<think>\n (5 tokens). D's own
match names the fork: sgl-project#904 reached=31441 against weg2-10-14's 31446
(41677/41682, 48867/48872, 73895/73900, 89902/89907 -- always N-5). P's only
tail anchor is the END anchor at N-1 (P-TRIM-END-ANCHOR, trim=1), 4 tokens
past the fork, so the read fell back to the previous anchor (#1028B FETCH CAP
kv=31441 claimed=27384 caps={mamba: 27384}). 8 of 9 arrived while the
predecessor was still decoding; no anchor at "prompt + generated" existed or
would have helped (next turns re-render the output: 26-57 claimed 73898 =
P's end anchor of 24-54, not D's finish anchor at 74752). 26-67: D's own
prefill track of 26-66 landed at N-1 (extend 1025 = 16*64+1).

Switch SGLANG_WEG2_FORK_ANCHOR_TOKEN=<id> on both groups (launcher
--fork-anchor-token 248045, needs --p-trim-end-anchor; unset = byte-identical):
* P intake cuts a leg-1 prompt at the last fork token of its final 16 ids
  (N-5 on Qwen3.x) instead of N-1 -- no forward added, D extends 6 tokens
  instead of 2 in the leg-2 extend it runs anyway;
* D's store read ends at the same cut (pure function of ids + env, same on
  every rank): a fork-cut leg-2 read lands complete, no sgl-project#1324 deferral;
* D's extend track of the step reaching the prompt end stays at or below the
  fork (h at (t-prefix)/64, relative grid; extend only, never decode).

Tests (test27b.sh, CPU): new test_weg2_fork_anchor_0926.py 18 passed; red on
c51591f (13 failed / 4 passed: the switch had no effect). Real
UnifiedRadixCache uni+bigram: sibling claim 0 -> the fork with P's state; D
leg 2 resumes at the fork. Affected files 173 passed (trim, tail fold, end
anchor hold, store-short, deferral, hold-refetch, dormant hold, P intake,
launcher argv). Broader sweep: 471 passed, 15 failed identically on base
(gloo tests 580/943b/1290, pre-existing).
efschu pushed a commit to efschu/htsglang that referenced this pull request Sep 28, 2026
…lform (array statt list) -- Admit(0) fuer alle, wenn PP0s Anker am told fehlt

rc12z2 (e8a2cd2, P-Log Z. 226312-226760, 03:26:31-41, rid weg2-44-76):
PP0 las 47168 Tokens (8x 'sgl-project#1433 L3->L2 fill' -- der Rueckweg, den Runde 4
erst oeffnete), klemmte told auf den Store-Anker 37952 (sgl-project#1416 ANCHOR-CLAMP)
und schickte ihn. Beim Zulassen:
  PP0 'sgl-project#904 ... reached=37952 refused=37952 why=MambaComponent:absent'
      -> '[sgl-project#928 anchor] REFUSING resume' -> Zulassung bei 0
  PP2 dasselbe -> 0
  PP1 'sgl-project#988 LOADBACK ... anchor_depth=37952 mamba_restored' -> 37952
PP1 bekam PP0s Zeilen 32768..49152 zu seinem Batch 70720..79688:
'sgl-project#1233 W27 PP WIDTH DIVERGENCE REFUSED ... cause=START-SPLIT', P tot.

Genau dafuer gibt es TF (weg2_told_fidelity, Default an): vor dem Admit
fragt PP0 den eigenen Baum, kann er am told nicht fortsetzen, geht told=0
mit PF-Fallback-Marke an ALLE Stufen (jede gibt ihren Read frei, alle
prefillen ab 0). Auf NF lief TF nie: '#TF told-fidelity probe skipped ...
AssertionError((<class 'array.array'>, <class 'list'>))' 10 von 10 auf
diesem Boot -- _probe_key baute den Schluessel als list, der Baum
vergleicht array('q')-Schluessel (RadixKey.match: assert type(t0) is
type(t1)). Keine Aussage -> Admit blieb 37952, waehrend PP0 selbst 0 nahm.

Fix (kein Schalter, bestehende Vereinbarung): _probe_key baut array('q').
Die #1416d-Klemme teilt den Schluessel; ein Array-Schluessel hasht dieselben
Seiten (compute_node_hash_values hasht die Array-Schluessel des Baums).

Test test_told_fidelity_array_key_rc12z2 (4; Fake-Baum vergleicht mit dem
ECHTEN RadixKey.match gegen einen Array-Knoten, told 37952, Bigram): 3 rot
auf e8a2cd2 (genau 'AssertionError((array.array, list))', Verdikt
(37952, None) statt (0, 0)), gruen mit Fix; Hash-Gleichheit list/array
gruen auf beiden. Die TF-Suite 0927 mass nur len(key) und erreichte den
Vergleicher nie.
efschu pushed a commit to efschu/htsglang that referenced this pull request Sep 28, 2026
…), not at device+host

NF rc12z22 D (dkrnfh91dprsavisnoadoptstbar1dauer09281447, 7d50735) TP0: the
three READ-TRACE lines of 14:56:26 (asked=43/286/44 readable=0 why=no-file)
are NOT a missing Mamba blob. weg2-10-16's wake read came back complete
(prefetch success matched=18240 loaded=50944 = 69184, KV + Mamba state at
69184). Its admission registration read "sgl-project#823 HEAD-VOTE ANCHOR matched=50944
(device 0 + host 50944) anchor_depth=69184": the 18240-token device node
carries no recurrent state, the mamba validator cut the device match to 0
(sgl-project#904 refusers=MambaComponent:absent=18240) and host_hit_length counted only
50944. _prefetch_kvcache took len(prefix_indices)+host_hit_length = 50944 as
the span start but last_host_node (= best_match_node, depth 69184) for the
hash chain -> 286 keys that exist nowhere. Same shape for weg2-4-11 (2560
device + 73472 host, the SAME first stem bf3b9106.. asked at 14:56:01 and
14:56:26), weg2-11-18 (2560 + 73344) and weg2-8-15 (18240 + 45312, asked=286).
Each read ended store-short / prefetch_pending: 5 s of X-DEFER until the X gate
priced the true remainder (uncached=42/161/204). The four hypotheses (write
behind, off-grid anchor, KV/Mamba key schema, eviction of the blob) do not
apply: PARK-DEMOTE had put every KV page on L3 (l3_pages=1081 of 1081).
- scheduler: _weg2_prefetch_span_start -- on a group without PP and a tree
  with a controller, a state_anchor_depth deeper than device+host is the span
  start (the depth of last_host_node, where its last hash is); the HP1 span
  base follows. SGLANG_WEG2_PREFETCH_FROM_ANCHOR=0 = the old start. Group P's
  sgl-project#1400 told arithmetic is untouched (pp_size > 1 keeps the old start).
  Line "FA PREFETCH-FROM-ANCHOR rid= matched= (device + host) anchor=".
- Nothing is recomputed that was stored: the read now asks exactly the tail
  beyond the stored prefix (weg2-10-16: 41 tokens instead of 18281 bogus ones).

Tests (test27b.sh, hermetic, real _prefetch_kvcache on a recording tree):
test_weg2_prefetch_from_anchor_0928 6 passed (weg2-10-16 asks [69184, 69225)
under the hash of 69184; switch off reproduces the metal's 286-page ask from
50944; HP1 span_base = anchor; no deeper anchor / PP group / no controller keep
the old start; the line), 3 red on a94aea5. Neighbours green: 946 clamp,
hold_prefetch_sleep_h91e, dormant_hold_1445, prefetch_deferral_1068 (111).
efschu pushed a commit to efschu/htsglang that referenced this pull request Sep 28, 2026
…dit -- Bruecke f833fcb: D schrieb den Anker 37376 in 16-46s eigenem Extend ('sgl-project#1469 RETAIN rid=weg2-16-46 ... cache_len=37376', 19:50:38), 16-48 setzte darauf auf (19:50:51, '#cached-token: 37376'). Der D-Schlaf 19:51:15-16 verweigerte die Host-Sicherung ('sgl-project#1421 BACKUP-REFUSED why=mamba_pin ... pins=8/8', danach 'why=mamba_claim' hinter 'sgl-project#1427 ARENA-CLAIM REFUSED ... no free slot' der 32-Slot-Arena) und setzte den Baum zurueck ('sgl-project#1470 FLUSH-PUBLISH issued=13 unbacked_left=1', 'Cache flushed'). Nach dem Wake gab der Store den KV zurueck (store_hit=37376), aber keinen Anker ('[sgl-project#904 match-census] ... MambaComponent:absent=37376'); die Front preiste weg2-16-54 trotzdem SHORT (presence 35942 src=d_leg2_cached) -> 17 s D-Sitz, dann W31 am X-Gate (38291 Tok) und W50-Umweg ueber P (12 solche in rc12z26, 8 auf der Bruecke). Das X-Gate auf D preist schon die anker-nutzbare Tiefe (W31), die Front nicht.

Fix: (a) Instrument: sgl-project#1421 BACKUP-REFUSED nennt depth= und rid= (weg2_node_depth, weg2_anchor_rid). (b) Der Flush zaehlt vor tree_cache.reset() jeden nur-Device-Mamba-Anker mit Tiefe (UnifiedRadixCache.weg2_unbacked_anchors, 'WEG2-ANCHOR-LOST at=flush n= depths= rids='); der Scheduler haelt die Tiefen bis zur Schlafantwort (weg2_take_anchors_lost, einmal lesen). (c) Die Gruppenzaun-Stimme des release-Legs traegt anchors_lost (Modulfunktion _weg2_anchors_lost_for_leg, [] auf jedem anderen Leg und ohne Scheduler-Ledger), die Reduktion vereinigt, ReleaseMemoryOccupationReqOutput.anchors_lost, der Tokenizer-Merge vereinigt (Feld fehlt, wenn nichts verloren -> alte Antwortform). (d) Front.flip liest nach completed_tags die Schlafantwort von D und nimmt jeden SpanLRU-Eintrag zurueck, dessen Kredit auf einer verlorenen Tiefe stand (depth_cap, sonst gemessene ct), 'WEG2 PRESENCE-ANCHOR-LOST'; ein Eintrag auf einem ueberlebenden Anker behaelt ihn. Zu viel Zuruecknehmen ist sicher (voller Preis = Preis des X-Gates). Die Rettung des Ankers ueber den Schlaf bleibt bei sgl-project#259 4c. Luecke: Arena-Verdraengung ohne L3 ausserhalb des Flushs erfasst der Ledger nicht (nur die Host-Kopie-Pruefung).

Tests test_weg2_anchor_lost_presence_0928 (10): alle rot auf 1ea8133, gruen mit Fix. Nachbarn 18 Dateien (1424g, H113, H74, R12, sgl-project#59, X-exact, Spans, sgl-project#1470, sgl-project#49/P49, flip_cost_1235, 1273 x2, sched_fix3, W35, Profil, H91e, sgl-project#248): 298 gruen auf Fix und Basis, dieselben 5 Altlasten (VramCredit x3, sched_fix3 b4a/b4c).
efschu pushed a commit to efschu/htsglang that referenced this pull request Sep 28, 2026
…dit -- Bruecke f833fcb: D schrieb den Anker 37376 in 16-46s eigenem Extend ('sgl-project#1469 RETAIN rid=weg2-16-46 ... cache_len=37376', 19:50:38), 16-48 setzte darauf auf (19:50:51, '#cached-token: 37376'). Der D-Schlaf 19:51:15-16 verweigerte die Host-Sicherung ('sgl-project#1421 BACKUP-REFUSED why=mamba_pin ... pins=8/8', danach 'why=mamba_claim' hinter 'sgl-project#1427 ARENA-CLAIM REFUSED ... no free slot' der 32-Slot-Arena) und setzte den Baum zurueck ('sgl-project#1470 FLUSH-PUBLISH issued=13 unbacked_left=1', 'Cache flushed'). Nach dem Wake gab der Store den KV zurueck (store_hit=37376), aber keinen Anker ('[sgl-project#904 match-census] ... MambaComponent:absent=37376'); die Front preiste weg2-16-54 trotzdem SHORT (presence 35942 src=d_leg2_cached) -> 17 s D-Sitz, dann W31 am X-Gate (38291 Tok) und W50-Umweg ueber P (12 solche in rc12z26, 8 auf der Bruecke). Das X-Gate auf D preist schon die anker-nutzbare Tiefe (W31), die Front nicht.

Fix: (a) Instrument: sgl-project#1421 BACKUP-REFUSED nennt depth= und rid= (weg2_node_depth, weg2_anchor_rid). (b) Der Flush zaehlt vor tree_cache.reset() jeden nur-Device-Mamba-Anker mit Tiefe (UnifiedRadixCache.weg2_unbacked_anchors, 'WEG2-ANCHOR-LOST at=flush n= depths= rids='); der Scheduler haelt die Tiefen bis zur Schlafantwort (weg2_take_anchors_lost, einmal lesen). (c) Die Gruppenzaun-Stimme des release-Legs traegt anchors_lost (Modulfunktion _weg2_anchors_lost_for_leg, [] auf jedem anderen Leg und ohne Scheduler-Ledger), die Reduktion vereinigt, ReleaseMemoryOccupationReqOutput.anchors_lost, der Tokenizer-Merge vereinigt (Feld fehlt, wenn nichts verloren -> alte Antwortform). (d) Front.flip liest nach completed_tags die Schlafantwort von D und nimmt jeden SpanLRU-Eintrag zurueck, dessen Kredit auf einer verlorenen Tiefe stand (depth_cap, sonst gemessene ct), 'WEG2 PRESENCE-ANCHOR-LOST'; ein Eintrag auf einem ueberlebenden Anker behaelt ihn. Zu viel Zuruecknehmen ist sicher (voller Preis = Preis des X-Gates). Die Rettung des Ankers ueber den Schlaf bleibt bei sgl-project#259 4c. Luecke: Arena-Verdraengung ohne L3 ausserhalb des Flushs erfasst der Ledger nicht (nur die Host-Kopie-Pruefung).

Tests test_weg2_anchor_lost_presence_0928 (10): alle rot auf 1ea8133, gruen mit Fix. Nachbarn 18 Dateien (1424g, H113, H74, R12, sgl-project#59, X-exact, Spans, sgl-project#1470, sgl-project#49/P49, flip_cost_1235, 1273 x2, sched_fix3, W35, Profil, H91e, sgl-project#248): 298 gruen auf Fix und Basis, dieselben 5 Altlasten (VramCredit x3, sched_fix3 b4a/b4c).
efschu pushed a commit to efschu/htsglang that referenced this pull request Sep 30, 2026
…nn sein Span schon im Baum liegt (y5a weg2-19-28)

NF-Operator 30.09. (Mamba/GDN-Praefixzustaende gehen verloren, obwohl L3
Platz hat). Kette aus dem D-Log y5a (0110f8e132_0930_174039, TP0):
- 17:48:10 der Hold-Read von weg2-19-28 findet mit EIGENEN Schluesseln
  (sgl-project#1442 HANDOFF-KEYS NONE) den Sessions-Praefix im Store: "#1028B FETCH CAP
  kv=675 claimed=665 ... anchors_in_range mamba (9, 664)" = 42560 Token MIT
  Anker auf Seite 664. Schluessel-Identitaet ist NICHT die Luecke (18-24 las
  denselben Praefix mit eigenen Schluesseln: store_hit=42240; das "known=640"
  ist der Nachlese-Rest [42560, 43200) nach dem ersten Read).
- 17:48:12 das Geschwister weg2-18-24 wird zuerst zurueckgeladen ("sgl-project#988
  LOADBACK ... prefix moved to 43520 ... anchor_depth=43520"): sein KV liegt
  auf dem DEVICE, Zustand nur bei 43520.
- 19-28's Read endet (completed=42560): _insert_helper_host findet den
  ganzen Span schon im Baum, legt keinen Host-Knoten an (inserted_host_node
  None: der Knoten hat keine Host-KV-Kopie), und
  MambaComponent.commit_hicache_transfer(PREFETCH) GIBT DEN ANKER FREI.
- "[sgl-project#904 match-census] reached=43200 accepted=0 refusers=MambaComponent:
  absent=43200" -> "[sgl-project#928 anchor] REFUSING ... best_value_len=0 NONE-ON-
  THIS-PATH" -> W31 -> W50 x_refusal_midstream, 43396 Token zu P.

Fix: _insert_helper_host nennt den Knoten, an dem sein Schluessel ENDET
(InsertResult.matched_end_node; der Lauf hat ihn dort gespalten = Tiefe des
Ankers). Traegt dieser Knoten keinen Zustand (weder Device noch Host), nimmt
er den gelesenen Anker (host_value, Host-LRU) statt der Freigabe. Ein Knoten
MIT Zustand behaelt seinen, die Kopie wird wie bisher freigegeben. Der
naechste Match liest: best_match_node = der Knoten, state_anchor_depth =
seine Tiefe, mamba_host_hit_length = 1 -> init_load_back stellt den Zustand
in den Slot des Requests und nimmt den Device-KV darueber (der sgl-project#988-Weg).
Marker "WEG2 PREFETCH-ANCHOR-ATTACH". Schalter
SGLANG_WEG2_PREFETCH_ANCHOR_ATTACH (environ.py, default an).

Tests: test_weg2_prefetch_anchor_attach_0930.py 6 (echter UnifiedRadixCache
ueber echtem HybridReqToTokenPool; 5 rot auf 8406e68). Nachbarn
(anchor_drop_l3_257ii, arena_mamba_direct_1427, prefetch_* 7 Dateien,
short_read_anchor_sa, mamba_hicache_issuance_869b, mamba_gates_the_hit_706,
unified_radix_hicache_dispatch, unified_mamba_views, boundary_anchor_792,
mamba_anchor_seams_747): 152 gruen; 905 double_frees + 747 prepare_gates
rot auch auf der Basis (fremd).
efschu pushed a commit to efschu/htsglang that referenced this pull request Oct 1, 2026
…asses over the sgl-project#243/sgl-project#248 keep (z30y12 epoch 50: six park resumes W31, 381k tokens to P)

Metal z30y12 (df1f815, 30.09.): P PP0 "ARENA-EVICT n=11 want=256" at
23:35:32 (the L3 fill / arena write callers floor want at 256 slots); 3 s
later the MAMBA arena read "ARENA-REF-CENSUS arena-78446592.bin slots=112
complete=0" on every P rank, D's MAMBA census went park_kept=6/12 -> 0/12.
At the 23:35:44 wake all six D-parked requests read their KV back
("HiCache prefetch success ... loaded=53503") but no anchor ("[sgl-project#904
match-census] verdict=refused ... refusers=MambaComponent:53503
why=MambaComponent:absent"), the X gate priced each whole (X-GATE-TERMS
head=0 store=0 floor=0 -> W31) and W50-REROUTE sent 381501 tokens to P --
the flip of epoch 50. weg2-35-112 came through 4 s later with uncached=150
after one anchor page was filled back from L3 (sgl-project#1433, TP1 23:35:47). The
second collapse (D TP0-2 "ARENA-EVICT want=256" 23:42:10, MAMBA complete
110 -> 8) has the same shape.

Root: HiCacheFile._arena_evict_to_disk evicts min(want, every unreferenced
complete slot) with only the pins kept. want >= 256 is larger than the
112-slot MAMBA arena, and the sgl-project#248 park / sgl-project#243 hand-off pages are kept by
ORDER, not by reference -- so one round took them all. The claim path
(_evict_for_claim) has passed over that keep since sgl-project#243/sgl-project#248; the clock evict
never did.

Fix: (1) want is capped at an eighth of the arena's slots (FULL/draft arenas:
90112, i.e. unchanged for every caller; MAMBA: 14); (2) the host pools register
themselves on the backend at bind (register_keep_pool), and the clock evict
passes the pool's handoff_pending.keep_for keys as keep_lo (stage i only:
kept pages are never its candidates).

Tests (CUDA_VISIBLE_DEVICES="" nice -n 19 ionice -c3):
test_evict_keep_mamba_arena_0930.py (real ShmArena): base 2 failed (the
unregistered case is behavioural: 8 of 8 slots evicted), fix 2 passed. Mutant A
(keep_lo dropped): 1 failed (the parked anchor leaves L2). Mutant B (cap
dropped): 2 failed. Affected files (park_l3_248, arena_reset_release_h81,
prefix_trace_in, probe_hold_257, w3_arena_spill, arena_host_pool_1424,
arena_l3_fill_group_rc12z, arena_l3_fill_1433, l3_reuse_after_restart,
hicache_shm_arena_1404, mamba_arena_displace_h19, new file): 94 passed,
4 failed -- the same 4 fail on the base df1f815/92c9c18020 sources
(arena_host_pool_1424 x3, arena_l3_fill_group_rc12z x1), not touched here.
efschu pushed a commit to efschu/htsglang that referenced this pull request Oct 14, 2026
… done

NF bc35462, P PP0, weg2-56-163 (lane 1, 142861 tokens): a short store
read marks the operation terminated inside the KV transfer; the io thread
reads the recurrent state at the deepest landed anchor only afterwards
(_short_read_anchor_transfer, extra_ms=371) and then sets
pool_transfers_done. can_terminate_prefetch reaped on `terminated` in
between: _weg2_short_anchor_pages was still None, sgl-project#257 cut on the store's
presence answer (89472), a stateless prefix was inserted, PP0's walk
refused it (sgl-project#904 MambaComponent:absent), #TF put told=0 on the wire and
P re-prefilled all 142861 tokens (36 s, cached_tokens=0).

Measured in that boot: 3/3 #TF fallbacks (weg2-12-41, weg2-25-63,
weg2-56-163) were reaped BEFORE their SA line; 7/7 SA-before-reap reads
resumed at the anchor.

Fix: while a started read's component pools are still in transfer, the
termination is held via the H45 hold slot (group-uniform MAX), bounded by
the operation's own timeout budget. Queued-never-started and pool-less
reads are unaffected.

Tests: py_compile OK; test_short_read_anchor_reap_race_1014.py (5 cases).
efschu pushed a commit to efschu/htsglang that referenced this pull request Oct 14, 2026
NF h13 84e787e (boot 1014_135753), P, weg2-10-82 (59163 tokens):
all 921 KV pages landed (loaded=58944), yet sgl-project#904 MambaComponent:absent
=58944 and #TF told=0 -> full prefill.

The anchor read of the 32-slot mamba arena found it FULL on every PP
rank. Each rank's ROOM-WAIT evicted and re-claimed; PP2 (pid854) won
the freed slot ('ROOM-WAIT claimed=1'), PP0 and PP1 got status 1
('L3-FILL JOINED … holders=[26/7ms/pid854/l3fill…]', 'ROOM-WAIT
claimed=0 still_full=0'). _await_fill_room unclaimed the join and
DROPPED it: a miss, the anchor never read on PP0, the stateless KV
prefix was refused at admission. PP2's fill completed within ms.

Fix: a status-1 re-claim inside the room wait is unclaimed and appended
to `joined`, so the JOIN wait that runs right after awaits the peer's
fill like every other join.

Tests: py_compile OK; test_l3fill_join_wait_0930.py +1 (peer wins the
freed slot, the fill gets it through the join wait).
efschu pushed a commit to efschu/htsglang that referenced this pull request Oct 14, 2026
…'s mamba anchor

NF h13 84e787e (boot 1014_135753), D TP0 14:35:34, weg2-24-181 (lane 1,
149125 tokens): the LANE-PREEMPT park of six lane-0 requests made room for
their park chains in the full 32-slot mamba arena and spilled the arriving
lane-1 request's own resume anchor to L3 ('MAMBA-ARENA DISPLACE n=93
why=park_first node=673 depth=148992', 'PARK-END-ANCHOR-FIRST SPILL n=93
node=691 depth=70400 victim_depth=148992'). D's match then refused the
request ('sgl-project#904 refusers=MambaComponent:148992 why=absent'), X-GATE head=0
store=0 W31, W50 to P, 12 s stall with 10 open requests.

- a lane >= 1 request's finish insert stamps every node of its sequence
  with the highest lane that finished through it (weg2_lane); lane 0
  stamps nothing (no walk);
- a park chain carries the PARKED request's lane (weg2_park_lane, at
  PARK-END-ANCHOR-FIRST MARK) and claims with it;
- pick_park_victim / pick_last_resort_victim / the flush spill displace
  only anchors of a lane <= the claimer's, lowest lane first, then the old
  order (shallowest, node id); a chain node that finds none goes KV-only
  as before -- the lower lane pays.

SGLANG_WEG2_LANE_ANCHOR_RANK (default on); off = the old pick order byte
for byte; lanes off = every node lane 0 = the old order.

Tests: test_weg2_lane_anchor_rank_1014.py (6) + park_end_anchor_first_1303,
mamba_last_resort_1008, mamba_arena_displace_h19, mamba_anchor_bypass_1014:
45 passed (pytest_gedeckelt.sh).
efschu pushed a commit to efschu/htsglang that referenced this pull request Oct 14, 2026
…'s mamba anchor

NF h13 84e787e (boot 1014_135753), D TP0 14:35:34, weg2-24-181 (lane 1,
149125 tokens): the LANE-PREEMPT park of six lane-0 requests made room for
their park chains in the full 32-slot mamba arena and spilled the arriving
lane-1 request's own resume anchor to L3 ('MAMBA-ARENA DISPLACE n=93
why=park_first node=673 depth=148992', 'PARK-END-ANCHOR-FIRST SPILL n=93
node=691 depth=70400 victim_depth=148992'). D's match then refused the
request ('sgl-project#904 refusers=MambaComponent:148992 why=absent'), X-GATE head=0
store=0 W31, W50 to P, 12 s stall with 10 open requests.

- a lane >= 1 request's finish insert stamps every node of its sequence
  with the highest lane that finished through it (weg2_lane); lane 0
  stamps nothing (no walk);
- a park chain carries the PARKED request's lane (weg2_park_lane, at
  PARK-END-ANCHOR-FIRST MARK) and claims with it;
- pick_park_victim / pick_last_resort_victim / the flush spill displace
  only anchors of a lane <= the claimer's, lowest lane first, then the old
  order (shallowest, node id); a chain node that finds none goes KV-only
  as before -- the lower lane pays.

SGLANG_WEG2_LANE_ANCHOR_RANK (default on); off = the old pick order byte
for byte; lanes off = every node lane 0 = the old order.

Tests: test_weg2_lane_anchor_rank_1014.py (6) + park_end_anchor_first_1303,
mamba_last_resort_1008, mamba_arena_displace_h19, mamba_anchor_bypass_1014:
45 passed (pytest_gedeckelt.sh).
efschu pushed a commit to efschu/htsglang that referenced this pull request Oct 14, 2026
…ore -- read just the blob

NF h13 84e787e (boot 1014_135753), D 14:35:36, weg2-24-181 (lane 1,
149125 tokens): the match walked the KV to 148992 but refused every node
for a missing Mamba state (sgl-project#904 MambaComponent:absent=148992) -- the lane-0
park had spilled the anchor to L3 (DISPLACE why=park_first on_disk=1)
while the KV stayed in the tree. The store held the anchor but not the
KV (D's pages were device-only, PUBLISH-SWEEP unbacked=13), so the
ordinary prefetch from the root could not deliver; W31, W50, P prefilled
the whole prompt. (c') removes that displacement; this is the net.

UnifiedRadixCache.weg2_anchor_only_read(req), called by the scheduler's
_prefetch_kvcache right after the re-match: when key_match_depth >
state_anchor_depth, the node at key_match_depth carries no Mamba state
and has a last-page hash, the GDN-holding rank reads ONLY that anchor
through the existing mamba-pool transfer (PoolTransfer MAMBA,
TRAILING_PAGES, alloc_read placeholder) and batch_get_v2 -- arena in
place, L3 fill, or the ANCHOR-BYPASS staging row -- and attaches it as
the node's host Mamba value (the PREFETCH-ANCHOR-ATTACH form, host LRU);
the re-match then accepts it (MAMBA-HOST-RESUME) and the load-back moves
the state only. No new prefetch op and no new collective: a rank without
GDN state never reads (rank_holds_gdn_state), and the group's head vote
stays the MIN of the ranks' matches, so a rank whose read failed keeps
the group where it was. One read per depth per request. A failed read
releases its row. SGLANG_WEG2_ANCHOR_ONLY_READ (default on).

Tests: test_anchor_only_read_e1_1014.py (5: read+attach, non-GDN rank
never reads, failed read releases and is not re-read, nothing to do /
switch off, and a source-level MUTANT with the GDN guard removed that
the rank-agreement invariant kills) + anchor_bypass_1014,
prefetch_anchor_attach_0930 (+full_unbacked), prefetch_anchor_pin:
27 passed.
efschu pushed a commit to efschu/htsglang that referenced this pull request Oct 14, 2026
…ore -- read just the blob

NF h13 84e787e (boot 1014_135753), D 14:35:36, weg2-24-181 (lane 1,
149125 tokens): the match walked the KV to 148992 but refused every node
for a missing Mamba state (sgl-project#904 MambaComponent:absent=148992) -- the lane-0
park had spilled the anchor to L3 (DISPLACE why=park_first on_disk=1)
while the KV stayed in the tree. The store held the anchor but not the
KV (D's pages were device-only, PUBLISH-SWEEP unbacked=13), so the
ordinary prefetch from the root could not deliver; W31, W50, P prefilled
the whole prompt. (c') removes that displacement; this is the net.

UnifiedRadixCache.weg2_anchor_only_read(req), called by the scheduler's
_prefetch_kvcache right after the re-match: when key_match_depth >
state_anchor_depth, the node at key_match_depth carries no Mamba state
and has a last-page hash, the GDN-holding rank reads ONLY that anchor
through the existing mamba-pool transfer (PoolTransfer MAMBA,
TRAILING_PAGES, alloc_read placeholder) and batch_get_v2 -- arena in
place, L3 fill, or the ANCHOR-BYPASS staging row -- and attaches it as
the node's host Mamba value (the PREFETCH-ANCHOR-ATTACH form, host LRU);
the re-match then accepts it (MAMBA-HOST-RESUME) and the load-back moves
the state only. No new prefetch op and no new collective: a rank without
GDN state never reads (rank_holds_gdn_state), and the group's head vote
stays the MIN of the ranks' matches, so a rank whose read failed keeps
the group where it was. One read per depth per request. A failed read
releases its row. SGLANG_WEG2_ANCHOR_ONLY_READ (default on).

Tests: test_anchor_only_read_e1_1014.py (5: read+attach, non-GDN rank
never reads, failed read releases and is not re-read, nothing to do /
switch off, and a source-level MUTANT with the GDN guard removed that
the rank-agreement invariant kills) + anchor_bypass_1014,
prefetch_anchor_attach_0930 (+full_unbacked), prefetch_anchor_pin:
27 passed.
efschu pushed a commit to efschu/htsglang that referenced this pull request Oct 16, 2026
… Mamba-Arena-Ablehnung, kein Spill auf P (PP>1), kein ANCHOR-LOST auf Form-A-Workern, Nachfolger nach dem Tombstone

A1/B1: _1421_refused merkt den Grund (_k15_last_refusal); die Sicherung spillt nur nach mamba_claim /
anchor_only_claim -- nach KV arena_claim, Pin-Budget oder write_none kann der zweite Versuch nicht
gelingen (Boot _215812: D TP0 38 solche neben 161 mamba_claim, P 51x arena_claim).
A2/B2: auf P (pp_size > 1) ist der Mamba-Slot zwischen den Stufen geteilt -- _weg2_evict_spill_room
spillt dort nicht (auch nicht im P-FUND-Pfad), Zeile K15 EVICT-SPILL SKIPPED pp_size= pp_rank=; die
Sicherung selbst bleibt.
A3/B3: ein Form-A-Worker (_r12.role() == worker) sichert nicht und schreibt kein WEG2-ANCHOR-LOST
(TP0 entscheidet und sichert).
A4/B4: drive_eviction frischt den Nachfolger erst NACH _evict_component_and_detach_lru/_cascade_evict
auf (wie der Blatt-Zweig).
A5: die Review-Proben R1-R5 in die Suite uebernommen.

Tests: test_weg2_k15_evict_anchor_save_1016.py 20 gruen, auf 29162a4 7 rot (A1 x4, A2, A3, A4).
Mutanten: Spill bei jeder Ablehnung ROT 4, Spill auf PP>1 ROT 1, Worker-LOST ROT 1, Auffrischen vor
dem Tombstone ROT 1, sgl-project#904-Guard weg (Review-M2) ROT 2.
efschu pushed a commit to efschu/htsglang that referenced this pull request Oct 21, 2026
…3 at the next hit, not recomputed (SOLL R7 R11)

S109c (81037f0, _s109c_aux_device_round) releases from a device-KV node the Full host rows AND the
mamba anchor once the write-behind's record holds the L3 copy of both (secure_rows_to_l3 + l3_complete,
_secure_child_anchors None). The node keeps its device KV with no state anywhere (rc29 D TP0: 6x S109c
W3-AUX-DEVICE SPILL 02:25:49-02:29:33Z). The next hit: the match refused it (sgl-project#904 MambaComponent:absent),
fell back to the last state above, and e1 returned 0 silently for exactly this shape (AUX-HOST-LAW: no
host state on a node without a Full host copy) -- every token behind the last state was recomputed
although L3 held the anchor; e1 was asked only at intake and at the deferral retry.

- e1 (_pdflip_aor_read_attach): device KV without a Full host copy -> S109R device read: the blob by the
  node's last page hash (arena slot / L3 fill / ANCHOR-BYPASS row) into a device mamba slot, S90b's device
  attach (note injectable). Only where no peer holds a mamba replica (Form A TP0 / local-PP floor);
  otherwise named how=peer_replicas (#639b). Nothing is written, no collective; like e1 it only raises
  this rank's match, the group MIN stays the verdict (R11).
- Admission match asks too (group D, pp_size 1): _s109r_admission_anchor_read right after the match in
  _get_new_batch_prefill_raw, re-match after an attach (one read per depth per request, e1's mark).
- H105g: anchor_dropped -- the matched host anchor (mamba_host_hit_length > 0) released by S109c inside
  the room step -> NO_TOKEN in the gathered vote; the next pass reads it back (S109R) instead of loading
  back without it and recomputing.
- Live witness: "S109R ANCHOR-DEVICE-READ n= rid= depth= how= attached=" (attached=0 how=store_miss/
  copy_failed/no_device_slot/peer_replicas = recomputed despite L3); the S109c line now names
  ids=<id>@<depth> for pairing.

Tests (serial, pytest_gedeckelt.sh, CUDA_VISIBLE_DEVICES=""): test_nf_rc29_s109c_anchor_reread_s109r_1021.py
7/7 + test_nf_rc29_loadback_target_dropped_h105g_1021.py 9/9 green; RED on 7933168: 9 failing (key:
after a real S109c release e1 returns 0, no read; anchor-in-window loads back with the anchor gone).
Mutant: rank-alone guard removed -> reads on a replicated group (caught). Neighbours e1/S90b/S43/S104/
S109/S92/Form A: 294 passed. All 53 admission-loop test files: failure set identical to a773b4e
(57 pre-existing failures, none new).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants