Repository navigation
fix: not use the default port - #1068
Merged
Merged
Conversation
Contributor
Author
|
fix https://github.com/sgl-project/sglang/actions/runs/10358622990/job/28673239739 |
Contributor
Author
|
ref sglang/.github/workflows/unit-test.yml Lines 40 to 43 in 6a38efa Lines 18 to 19 in 6a38efa |
timethink
pushed a commit
to timethink/sglang
that referenced
this pull request
Mar 9, 2025
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Sep 1, 2026
…hot: an instrument may never gate Boot 1067park, second fence (tp_to_pp, 06:49:55Z): _1063_record_fence walks every page stem of every eligible node through 2-4 os.path.exists() against a 1.69-million-file store. The first fence ran warm (221862 stems, ~3 s); the second hit the re-prefill's fresh stems cold and never returned -- PP0's last line for 25 minutes was the FENCE-NODES header, no abandon could run (PP0 is the sole timeout carrier), and the deadman killed the boot 07:15:37. Fix: hard 2.0 s budget; an expired scan reports SCAN CAPPED (counts declared a sample, denominator law) and bumps fence_scan_capped. The two-point consumer (_1063_probe_since_fence) reads only the 4096-capped _1063_AT_FENCE dict and loses no coverage it ever had. Trap recorded in a comment: 'now' is a PARAMETER of the fence function, not a module name -- a bare now() here would NameError into the outer except and silently delete the snapshot (sgl-project#872 silent-zero shape). time.monotonic() used explicitly. Desk-proven by execution, both arms: uncapped emits normally; a slow clock trips SCAN CAPPED.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Sep 1, 2026
…and returns Against cbbb619: structural pin (_scan_deadline exists, set before its in-walk check, SCAN CAPPED self-report present) plus a behavioral arm -- a store whose stem probe costs 50 ms each (4096+ stems would take minutes unbudgeted) must return within the budget and still record a snapshot; measured green at ~2 s. Full file: 16 passed.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Sep 2, 2026
… (905 double, 1035c stub) Two test stand-ins that the branch's new production contracts outgrew (sgl-project#624/sgl-project#978 stand-in-drift class). Production is the reviewed, accepted contract in both cases and is NOT touched; only the stand-ins move. DRIFT 1 -- the sgl-project#905 census-scheduler double missed the A12.2 cutover clearer. test/registered/scheduler/seam_census_double.py:417 (FaithfulCensusScheduler) binds the REAL Scheduler.readmit_seam_residents, which since the slice-3 fix calls self._clear_prefetch_deferral_for_reissue(population) at python/sglang/srt/managers/scheduler.py:5810 (definition :6292, lifecycle CLEARER (c) in the table at :5969). The double bound the caller but not the helper -> AttributeError, surfacing as "retry() exceed maximum number of retries". Fix: bind BOTH real methods (_clear_prefetch_deferral_for_reissue and the _clear_prefetch_deferral_fields it calls) from the real class, not a no-op. A no-op stand-in would be the W29 direction again: the double would pass a cutover that never cleared, i.e. exactly the defect the clearer exists for. The A12.2 fields are deliberately NOT pre-planted on FaithfulReq -- the real Req does not declare them either (no occurrence in schedule_batch.py); they are planted dynamically by _apply_prefetch_deferral and every reader uses getattr(..., default). Absence is the faithful state. This is the same edit the slice-3 fix already made in the two sibling doubles (test/registered/unit/managers/test_seam_readmission_w31.py:87-89 and test_prefetch_deferral_1068.py:195-197); the 905 double was the one instance missed by that sweep. DRIFT 2 -- the #1035c stub missed the termination protocol. python/sglang/srt/mem_cache/hybrid_cache/hybrid_cache_controller.py:752 reads operation.is_terminated() DIRECTLY (branch commit 846c679, slice 2 fix 4). That override is a FULL replacement of the base probe -- one batch_exists call over the whole span, no per-batch loop -- so the base class's per-batch guard never reaches it, and the class-agnostic getattr form at managers/cache_controller.py:851 is a different pass and does not apply here. The stub at test/registered/unit/mem_cache/test_zero_answer_partition_1035c.py:172 built the operation without the protocol. Fix: the stub answers is_terminated() -> False (a live operation, which is what all four fixtures are about), modelling StorageOperation.is_terminated at managers/cache_controller.py:287. mark_terminate is deliberately NOT added: this path never calls it, and a stub method nothing drives is surface without a contract behind it. hybrid_cache_controller.py:752 is NOT weakened to a getattr. TALLY (hermetic: CUDA_VISIBLE_DEVICES="", PYTHONPATH=<worktree>/python, pytest -q -p no:cacheprovider, per file, run once each) test_seam_census_double_905.py before 6 failed / 14 passed after 0 failed / 20 passed test_zero_answer_partition_1035c.py before 4 failed / 5 passed after 0 failed / 9 passed THE FOUR 905 REDS THAT PRE-DATE THIS BRANCH test_every_resident_is_retracted_readmitted_and_off_the_live_set, test_no_row_belongs_to_nobody_afterwards, test_the_drop_evicted_before_it_ reset, test_the_seam_stamped_its_own_retractions were already RED at the pin 228a66d, so the branch gate counted only the other two as truly new. They go GREEN here with the same stand-in fix and no further edit. Their pin-era cause is therefore not present on this branch: the sgl-project#1068 slice work rewrote the production seam and this suite together (git diff 228a66d..HEAD touches seam_census_double.py +47/-33 and test_seam_census_double_905.py +60), and the clearer binding was the last piece missing. Not carried forward as debt -- the file is 20/20. Checks: both files run once each (above); import smoke of sglang.srt.managers.scheduler and sglang.srt.mem_cache.hybrid_cache.hybrid_cache_controller OK; ruff check and ruff format --check clean on both files. No production file touched.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Sep 3, 2026
…; one timeout source; the store witness replaces the seam stamp ROOT (boot weg1b3 @ 6980c75, rid 679e4568, 84,027 tokens, cutovers 3+4): _prefetch_timeout_check_linear_func charged base + len(hash_value) x per_page and hash_value is empty until the serial prefetch thread's store probe has run, so the every-pass drain reaped the LONG re-admission at the bare base (effective 1.0 s: the sgl-project#968/sgl-project#1065 re-pricing 2.0/1.0 was a pair of dead locals overwritten by the hybrid parse's 1/0.25) with completed=0. The store had the pages; nobody asked it in time. phase_purity then counted the retract stamp as 'restored' and licensed a P=0 TP prefill (6 x 4096 recomputed). FIX F1a unified_radix_cache.py: _prefetch_priced_pages = len(hash_value) or len(token_ids)//page_size; _prefetch_timeout_budget_s = base + pages x per_page; the linear check compares against it (hiradix sibling same form under its own cfg.max; hi_mamba unreachable, registry.py:106-107). The unified budget is deliberately UNCAPPED (no PrefetchTimeoutConfig.max): an unprobed 1M-token op prices ~1000 s and admission waits on it through the PP0 _prefetch_done_for hold. Named, not hidden. F1b one timeout source: hybrid_cache_controller.parse_storage_backend_ extra_config pops its defaults from PrefetchTimeoutConfig (2.0 s + 1.0 s per KiToken); dead 2.0/1.0 locals and the 3.0/0.25 re-attach literals deleted; 'sgl-project#1157 PREFETCH TIMEOUT base= per_ki_token= source=' printed once per attach. F1c the reap is a line: 'sgl-project#1157 PREFETCH REAPED req= probed= requested_pages= hit_pages= completed= elapsed= budget=' per termination that is not a local completion. PrefetchOperation.probed_hit_tokens is stamped by the prefetch thread after its MIN; the record the admission loop pops (prefetch_loaded_tokens_by_reqid) is a PrefetchOutcome(int) annotated with hit_tokens/probed, written at completion, reap and revoke. F1d the seam premise is a measured witness: store_witness() reads the tree's own prefetch state (pending / hit / bounded / cold / unprobed); the `cache_protected_len>0 or cached_prompt_tokens_at_retract>0` stamp arm is DELETED. StoreWitnessContradiction is raised from the premise and from both admission arms and re-raised through the three fail-open probes; it leaves get_next_batch_to_run uncaught and is the group STOP (run_scheduler_process -> SIGQUIT), never a P=0 admission. REVIEW FIX (operator decisions 2026-09-03): B1 PrefetchOutcome now round-trips pickle/copy.deepcopy: positional __new__(loaded, hit_tokens=None, probed=False) + __reduce__. The keyword-only form survived dumps but not loads and rode req.storage_hit_length -> cached_tokens_storage -> details['storage'] -> send_pyobj -> detokenizer death on the first partial store hit. Both admission writes store int(loaded_tokens). Int subclass kept over a dataclass: smaller diff, every existing int reader untouched. B2 beside a stamp the witness MEASURES the stamped span: 'hit' only if loaded > 0 and stamp - hit_tokens <= one HiCache chunk; allowance = tree._prefetch_chunk_tokens (chunked_prefill_size, the one-chunk bound the sgl-project#1068 L2 truncation line already prices; fallback prefetch_threshold when the tree has no term). A revoke (loaded=0, the 40-token chat header answering every probe) or a probed shortfall beyond the allowance beside a stamp is a contradiction. Cold requests (stamp 0) keep today's reading. B3 raise text renamed to 'sgl-project#1157 STORE WITNESS CONTRADICTION ...' and the class to StoreWitnessContradiction: 'SEAM RESTORE' tripped the sgl-project#1068 seam-copy zombie gate (test_968_seam_copy_deleted_1068 red on R1, green now). Functions renamed store_witness / assert_store_witness_at_admission. N1 rank-uniform reap annotation: probed (MIN) and hit_tokens (MIN) ride two extra slots of the EXISTING packed all_reduce in check_prefetch_progress (the one that reduces min_completed_tokens); the record and the REAPED line are derived from the reduced vector only, never the thread-timing stamp. No new collective. N2 phase_policy.py hold reason now names the store witness instead of 'premise verified on the retract credit'. POPULATION CHANGES NAMED: (a) 'bounded' state: a stamped request shorter than prefetch_threshold counts as restored (no read can exist; recompute bounded below the threshold; the /health_generate shape). (b) A stamped request with no record, no ongoing prefetch and a prompt >= threshold (rate-limited, storage off, deferral refused) is now 'cold' -> premise refused -> exemption closed, where the stamp used to open it. (c) A mixed queue still opens on the witnessed request and a cold sibling rides at P=0 (pre-existing); the boot-4 count 'sgl-project#1157 PREFETCH REAPED probed=False completed=0' = 0 for retract re-admissions is the metal proof no LONG rides at P=0. (d) PP followers (pp_rank>0) run assert_store_witness_at_admission too: a follower raising is a process death via SIGQUIT (a STOP), not a verdict; it is not gated to pp_rank==0 here. RANK UNIFORMITY of the witness inputs: ongoing_prefetch removed only after the group MIN / under the drain barrier / at the cutover sweep; prefetch_loaded_tokens_by_reqid written from the reduced vector (N1) or on the revoke path whose enqueue the prefetch thread's MIN precedes; prefetch_deferred set from a rank-local counter but cleared/retried under the group gate (sgl-project#1068 slice 3 fix). TESTS (hermetic, CUDA_VISIBLE_DEVICES="" PYTHONPATH=<tree>/python) test/registered/unit/mem_cache/test_1157_reaper_prices_requested_span.py (20) test/registered/unit/managers/test_1157_seam_witness.py (25) Red measured before the fix on the R1 build: pickle.loads/deepcopy TypeError; _witness_from_outcome(stamp=80009, PrefetchOutcome(0,hit=40,probed)) == 'hit'; (4096,hit=4096) beside 80009 == 'hit'; no _reap_annotation_from_packed; no StoreWitnessContradiction. On the parent e63fd08 both files fail at collection (PrefetchOutcome / StoreWitnessContradiction absent). Inverted tests carry withdrawal docstrings: 861j test_a_stamp_without_a_store_read_no_longer_opens_the_gate, W30 test_a_stamp_alone_no_longer_opens_the_exemption, 968 test_premise_with_stamps_and_host_content_no_longer_holds_alone. Mutants killed (5/5): M1a keyword-only __new__ restored -> 2F; M1b int() dropped at the TP admission site -> 1F; M2 shortfall term dropped -> 2F; M3 reap annotation read from the local stamp -> 1F; M4 'SEAM RESTORE' token restored -> 3F (incl. the sgl-project#1068 zombie gate). SUITE (bounded selection, identical on both trees: test/registered/unit/ mem_cache + distributed/test_prefetch_progress_symmetry_580.py + managers test_*phase_purity*/test_*phase_policy*/test_*seam* minus test_phase_flip_spec_seam_631.py + test_968_starvation_umbau.py + test_tp_decode_formation_861j.py; pytest -q -p no:cacheprovider -rf, single process; the two new files EXCLUDED from the parent run, appended on the fixed run): parent e63fd08 (scratch worktree): 125 failed / 3446 passed / 927 skipped (174 s) fixed (this tree): 125 failed / 3492 passed / 927 skipped (172 s) per-test FAILED-id diff (comm): 96 FAILED ids on each tree, IDENTICAL sets (comm -13 and comm -23 both empty): 0 new failures, 0 newly passing; +46 passed = the 45 new tests plus one (not attributable from -q output; FAILED, ERROR and skipped sets identical) pre-existing failing files (parent, top): test_phase_policy.py 35, test_pool_invariant_live_double_912b.py 12, distributed/test_prefetch_progress_symmetry_580.py 8, test_seam_abandon_cap_631.py 7, test_mamba_double_free_924.py 5. Ruff parity measured per touched file (fixed / parent): cache_controller 14/14, phase_policy 0/0, phase_purity 0/0, scheduler 102/102, hicache_storage 0/0, hiradix 1/1, hybrid_cache_controller 0/0, unified_radix_cache 1/1, 968 0/0, W30 5/5, 861j 9/9; new files 0/0; totals 132 = 132. Not claimed clean. EVIDENCE TIER: DESK-PROVEN. Boot 4 proof lines: 'sgl-project#1157 PREFETCH TIMEOUT base=2.00s per_ki_token=1.00s source=PrefetchTimeoutConfig' once per rank; 'sgl-project#1157 PREFETCH REAPED ... probed=False' = 0 lines for retract re-admissions; 'HiCache prefetch success req=<LONG> completed_local=N>0' at the first pp_to_tp cutover; 'sgl-project#788 ADMIT ... prefix_lens>0' for the LONG; 'sgl-project#1157 STORE WITNESS CONTRADICTION' genuine count 0 (trapsafe_count.py --marker 'sgl-project#1157 STORE WITNESS CONTRADICTION' --genuine-regex 'RuntimeError: sgl-project#1157 STORE WITNESS CONTRADICTION|StoreWitnessContradiction: sgl-project#1157'); a nonzero is a group STOP naming rid/stamped/probed_hit/loaded/ allowance/shortfall.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Sep 3, 2026
…follower speaks before it can be stuck
boot_855_weg1b5_cd5bb69607_0903_115008, rid 0c34259f, 11:55:13-11:55:30.
All three ranks issued the same re-admission prefetch (gen 4, keys=13224,
byte-identical FETCH CAP lines). PP0's completed in ~3 s, printed
completed_synced=12288, and ADMITted prefix_lens=12288. PP1's and PP2's had
not terminated. PP1 took the forwarded row into execute_scheduled_prefix and
raised the designed group STOP 14 s later. The number PP0 admitted on was
rank-local by construction: the MIN reduce behind "completed_synced" runs
only under tp_world_size > 1, and this boot is --tp-size 1 --pp-size 3 --
attn_reduce_world=1 on 307/307 HICACHE-ROUND lines of the whole log.
E1 managers/pp_prefetch_completion.py (new, pure, no I/O, no clock) plus a
per-rid completion carrier on the sgl-project#791 ring lap, modelled exactly on
sgl-project#968's parked-continuation table (stamp on the way up, absorb at PP0).
No new all_reduce, no new message, no new blocking point on the
admission path -- the PP0-authority order records a collective there as
fatal. PP0 admits only when the group floor covers what it wants to
schedule; a peer that is silent or still running contributes NO NUMBER
(INCOMPLETE, never a zero), otherwise the rid is DEFERRED by name while
other work proceeds. When the same length-priced bound the follower's
own wait uses expires, PP0 CLAMPS to the group floor through the
existing sgl-project#1059 note_observed_coverage channel instead of over-telling.
want <= 0 (no store span) admits unconditionally, so a boot with no
storage hit takes exactly the pre-sgl-project#1175 path. Kill switch
SGLANG_PP_GROUP_COMPLETION=0 restores the old admission for an A/B.
completed_synced now prints synced=yes|no and attn_reduce_world=N beside
itself, so the field can no longer claim a sync it did not do.
E2 pp_admission_congruence.execute_scheduled_prefix now DRIVES the writer it
used to suspend: after each poll it calls check_prefetch_progress when
the tree reports that path collective-free (world 1 = every all_reduce
skipped by construction), and the expiry text carries
prefetch_driven_in_loop=<bool> so an expiry can never be read as "the
bytes were not there" when nothing drove the sole writer. Bounds
unchanged. The sgl-project#1157 reaper's comment now states where it structurally
cannot fire (it reports terminations; a prefetch that never terminates
produces no line, so 0 REAPED is not evidence that nothing was reaped).
E3 Every follower prints one bounded "sgl-project#1175 PREFIX-EXEC UNDER-COVERAGE"
line at ENTRY, before any bound can expire, naming rid/local/scheduled/
deficit/bound. PP2 printed nothing at all about this rid and its silence
was indistinguishable from health. The line also states honestly that a
rank which never RECEIVES a decision row never calls this function and
therefore still emits nothing -- that case is what E1+E2 remove.
E4 (a) the healthy local == scheduled path no longer returns silently: one
bounded, counted "sgl-project#968 PREFIX-EXEC no-op" line, so "0 materialised lines
on the whole boot" reads as "never needed" rather than "never reached"
(INDIKATOR-GESETZ). (b) the sgl-project#939 census line now names its population
(population=retract_closure_only) -- SEAM_READMIT_ATTR is stamped only
by the retract closure, so the two 13225-token queue-occupant
re-admissions of this boot produced not one census line. The field is
placed BEFORE fence_proceeds so sgl-project#1068's tail pin still holds.
EVIDENCE (DESK-PROVEN; no boot, no GPU)
red-first, parent cd5bb69 + only the three new test files:
16 failed, 3 passed, 1 collection error (pp_prefetch_completion absent)
fixed tree: 36 passed (25 managers + 11 mem_cache)
mutants: 12/12 killed, >= 3 per decision (E1 x4, E2 x3, E3 x3, E4 x2)
matched checks re-run on the fixed tree: test_pp_admission_congruence_791,
test_968_starvation_umbau, test_968_deletion_falsifiers,
test_producer_phase_census_631, test_double_prefill_census_fence_1068,
test_producer_phase_census_wiring_1061 -> 114 passed, 1 failed
(test_the_module_imports_no_torch_at_all -- identical failure on the
parent, "import torch" count 1 on both trees: pre-existing)
bounded suite once, -n 8, hermetic, managers + touched mem_cache files:
parent 262 failed / 5435 passed / 12 errors
fixed 286 failed / 5472 passed / 12 errors
name-level comm: 35 "regressions" and 11 "cures", ALL of them inside
test_unified_radix_cache_unittest.py; 0 outside it, 0 involving the new
files. That module run SOLO and SERIAL is 642 passed / 860 skipped /
0 failed on BOTH trees -- the delta is the known -n 8 crowding family
(sgl-project#862/sgl-project#899), not this diff.
ruff --select=F401,F821,UP037: 2 errors on both trees, the same two
pre-existing ones; black/isort per file identical on both trees for all
five touched production files, all four new files clean.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Sep 4, 2026
…igure, C2 upstream scope Two text-only repairs handed over by the round-3 code reviewer. No behaviour change, no test re-run. The commit also carries the rest of the in-progress sgl-project#1189 slot-publish change (the test file is new, so its C1 edit cannot be committed on its own). C1 -- a boot-8 figure already retracted in production comment prose was still shipped verbatim by the test that a boot-9 agent reads when it goes red. Re-measured against /spinning/evidence-665-f1/boot_855_weg1b8_e9d1a719ac_0904_064622.log: grep -c 'running_bs=7768' -> 0 max of the running_bs= series -> 7771 grep -c ' running=7768' -> 1, at :458441, on a 'sgl-project#788 PP-ADMISSION verdict=DECLINE' line, and the max of that series grep -c 'distinct rid' -> 0 grep -c 'PP-ADMISSION verdict=ADMIT' -> 183 grep -c 'PP-ADMISSION verdict=DECLINE' -> 27127 grep -c '#969N ADMIT' -> 150 distinct rids on '#969N ADMIT' -> 16 distinct rids on 'PP-ADMISSION verdict=ADMIT'-> 19 (the 16 are a strict subset) Edits in test/registered/unit/managers/test_1189_slot_publish_lap.py at :39-46 (module docstring), :447-451 (lap-(A) assertion message) and :1036-1042 (the sgl-project#1189 OUTCOME failure message): 'running_bs 7768' becomes 'running=7768' with the :458441 anchor and the instrument named, matching the form the production comment already uses; 'with at most 3 distinct rids' is DELETED, no query reproduces it; the 183/27127 pair now names its instrument, because '#969N ADMIT' counts 150 for the same word. C2 -- the lap-(A) comment claimed a scope its own helper docstring refutes. scheduler_pp_mixin.py:5238-5240 closed with "This is COVERAGE FOR A STATE THE FORK'S OWN PURITY FEATURE INVENTED, not a break with an upstream invariant." The second clause is false. The new else also fires on the purity-INDEPENDENT idle lap: scheduler.py's 'ret = None' there is the outer else of 'if not running_batch.is_empty() and not running_batch.is_prefill_only:', not the _decode_blocked arm above it. Upstream reaches that identical lap (main:scheduler.py:2737 carries the same '# Run decode (skip for prefill-only batches)' arm, ret = None at :2742, and zero occurrences of phase_purity or phase_decode_blocked_here in that file) and preserves the previous last_mbs[slot] on it -- guard/publish/no-else three times over, at main scheduler_pp_mixin.py :148/:155, :320/:326 and :507/:514. So the fork now clears an entry upstream keeps. Edit at scheduler_pp_mixin.py:5233-5277: the true, one-step-verifiable first clause ("STRICT PHASE PURITY CREATED THIS LAP ON PURPOSE", :5225) stays untouched; only the false clause is replaced, by what _pp_record_slot_last_batch.__doc__ already says -- the wider scope is deliberate, toward the non-PP loops' unconditional 'self.last_batch = batch'. Anchors are written as grep recipes, since both files are under active edit. Checks: python -m py_compile clean on both files. ruff reports 3 findings in scheduler_pp_mixin.py (E402 :268, F841 carries_flip_arm, E741 'l'), all three present unchanged at HEAD on the same symbols; zero findings in the test file. No pytest run: both edits are text-only.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Sep 4, 2026
…ond copy
The conformance check of the final pass found the same sentence C2 retracted
in scheduler_pp_mixin.py still standing, un-retracted, in this test module's
own docstring -- the repaired-copy-beside-unrepaired-copy pattern, one more
level out. Text only, zero re-run.
The claim "this is coverage for a state the fork's own purity feature
invented, not a break with upstream" is false in its second half. The new
`else` also fires on the purity-INDEPENDENT idle lap, which upstream reaches
and where upstream PRESERVES last_mbs[slot] while the fork now clears it.
Replaced with what _pp_record_slot_last_batch.__doc__ already argues, plus
grep recipes (structural rule 2) instead of bare line numbers:
grep -n 'Run decode (skip for prefill-only batches)' scheduler.py
-> :9058 on the fork, :2737 on main = the two laps are one lap
grep -n 'BUT THIS `else` REACHES WIDER' scheduler_pp_mixin.py
-> :5239 = the production retraction this docstring now points at
py_compile clean. No pytest run (text-only by the reviewer's determination).
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Sep 9, 2026
…erisation, the acceptance takes the next boot's log The fixture is a recording of a boot that ran WITHOUT the fix, so no code change on this branch can turn it green. A test that can only ever be red is not an acceptance, it is a permanent red people learn to scroll past. * test_f3_the_walk_reproduces_the_recorded_defect_exactly pins the pre-fix number in executable form (37 truncated, 34 orphans) and proves the walk. * test_f3_invariant_holds_on_a_post_fix_log runs the SAME walk against WEG2_F3_LOG -- gate 4 of the boot ticket, the only place the invariant can honestly go green. Skips without the variable rather than passing on an empty walk, and keeps the population floor. Also: the landed assertion in the new shortfall arm matched _assert_unmarked, which is the FULL clearer's contract. sgl-project#1068's landed path clears the mark and arms the one-pass hold, leaving the timing fields as history. Asserted as sgl-project#1068 actually specifies it.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Sep 9, 2026
…s arm makes reachable for the first time _admission_held_for_deferred_prefetch records an OPEN hazard: in the TP phase the PP0 exemption is inert, so every rank holds on its own mark, and sgl-project#1068's mark is rank-local at both ends. On group D that path was previously unreachable -- no mark could exist there (symmetric_vote). This arm makes it reachable, so the precondition is checked rather than inherited: the shortfall mark comes from MIN-reduced values, so all three ranks hold and release together. Asserted both directions.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Sep 10, 2026
…d two pins that outlived their premise All eight NEW-only failures of the 7191379 gate, re-extracted per name WITH messages from the artefact itself (`897d9968ac` and `06d3ed02ad` give the identical eight; NEW-only 8, disappeared 0). THE THREE LEDGER FAILURES ARE **TEXT, NOT PRICE** -- classified, not assumed, which was the one thing that could have been the cycle's serious finding. The ARM line had gained `S_D` BETWEEN `S` and `M`, and three suites match the literal substring `ARM S=<s> M=<m>` (two by `in`, one by a list comprehension that then indexes `[0]` and raised IndexError). `S_D` now goes AFTER `M`, so the old reading is byte-identical and the new term is purely additive -- which is also the honest shape: nothing about S or M changed, D simply gained its own budget. All three pass with no pricing change. And the pricing was never wrong: `charge_terms(s_gb_d=None)` is byte-identical to the single-budget form (asserted), while the reclaimable term cannot book pinned host pools as reclaimable by construction -- `cg_reclaimable_bytes` is `(file - shmem) + slab_reclaimable` and its own docstring states "anon and unevictable are never subtracted: both are exactly what the reaper kills to recover". cudaHostAlloc rows are unevictable, so the 12.86 GiB of rings is a CHARGE, never a reclaimable credit. No ledger defect. THE GUARD IS OUT OF `main()`. `_derive_d_l2_budget` now carries the derivation and BOTH refusals. `main()` has a pinned law of zero try/except (the single funnel is `cli()`, sgl-project#1248) and a pinned inline-raise count; the first draft broke both (`[<ast.Try>] != []` and `6 != 4`). This is the class sgl-project#1248 exists to stop, and it cost a red gate. TWO PINS OUTLIVED THEIR PREMISE AND ARE REWRITTEN, NOT DELETED. Both asserted that `exempt_carrier_exceeds` must EXIST as D's cold-store fallback, with the reason that an above-carrier prompt whose write-through had not landed would otherwise have no route. That was true at a fixed 1 GB (30,518 rows, sgl-project#915 limit 27,466). With L2 derived from `--max-kv-per-request` the carrier is at or above the cap, so there are exactly two cases and both have a route: * ABOVE the cap -> the front refuses at admission. The cap is the law and 413 is the answer, not a hole. * BELOW the cap with an unlanded read -> the X-DEFER arm (sgl-project#1238 fix 7) holds it while the read is pending, so it is never priced at its whole extent. THAT ROUTE IS VERIFIED RATHER THAN ASSERTED IN PROSE, which the operator's ruling required before any pin was flipped: the new `test_t9b2_a_pending_store_read_DEFERS_it_never_refuses_it` pins `_weg2_store_read_is_pending` on BOTH pending shapes (an `ongoing_prefetch` record and the sgl-project#1068 A12.2 mark) plus the negative case, so the defer arm demonstrably covers the population the exemption used to. No replacement route was built and no bypass was added. AND THE FIFTH INSTANCE TODAY OF A SOURCE ASSERTION MATCHING PROSE, this one in a test I wrote after naming the class four times: my own absence check tripped on the `front.py` COMMENT that DOCUMENTS the deletion. The comment is right and stays; the check now strips comments AND string literals via `tokenize` (a deleted emitter's log text is a string literal, and keeping one would be the same false positive wearing quotes) -- and it carries a PLANT-PROOF, because a stripper that removes too much turns the suite into a rubber stamp: every name is planted as real code and must be caught, while the same token in a comment or docstring must not be. One process note, since both seats got the count right and the names wrong in different ways: the boot agent's reconciliation listed five of these as green and quoted messages that belong to BASE failures. Re-extracting name+message pairs from the junit settles it -- `extraktions-zaehlprobe` applied to a handover, in both directions. Neither list was dishonest; a name recalled from code reads exactly like a name read from a result.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Sep 26, 2026
…en, POST /weg2/park_running vor D's Schlaf -- Nutzer-Design 25.09. Stufe 1 Sitze: --profile nextflash ohne --d-bs faehrt D jetzt mit 2 Sitzen (weg2.DEFAULT_D_BS_NEXTFLASH, launcher.apply_profile_d_bs_default); ein gesagtes --d-bs gewinnt. Der D-FRACTION-SOLVE bepreist die Sitze: die eingebauten NF-Referenzen tragen max_running=1 (7 Mamba-Slots = ceil(1 x 5 x 1.25)), seat_rebook bucht 'mamba state pool' (7 -> 13 Slots, TP0 393.2 -> 730.2 MiB) und 'speculative intermediate state' (linear) auf die wirksamen --max-running-requests um, VOR H64 -- dessen per_req hatte bei bs2 den bs1-Posten durch 2 geteilt (halber Zustand je Request). KARTE: Mehr-Allokation je Rang (TP0 -355.1 MiB Kopfraum in der Ring-Form), der bs2-Decode-Graph ist NICHT GEBUCHT bis --d-seat-graph-mib (gemessen) gegeben ist. POOL-SCHRITT: bs2 routet 2 x 4 x top-10 = 80 Ids je Verify-Schritt; Scratch 48 auf den Workern wirft beim Capture des bs2-Graphen 'Step ids exceed the LRU rows plus the staging rows' -> W-SITZE-Verweigerung mit den Zahlen vor dem Boot. expert_pool_device: Plan-Breite folgt dem breitesten gefangenen Schritt (plan_width_for: bs1 bleibt 64, bs2 128), die Task-sgl-project#40-Schranke zaehlt min(Ids, E-R). Launcher-Befunde: --d-bs gegen wirksame --max-running-requests, --cuda-graph-bs-decode deckt 1..Sitze. Park (weg2/d_seats.py Politik, weg2/d_park_runtime.py Buchhaltung, Scheduler delegiert): Decode-Druck auf D retrahiert den JUENGSTEN (kvso session_priority_key; unter Spec nur das Batch-Ende, sonst benannt), die Spanne bleibt im Baum (retract_retain: KV, GDN-Anker, Draft-Zeilen), er steht am Queue-Kopf und kommt zurueck, wenn kein aelterer Request mehr lebt; Neuankoemmlinge warten solange (Barriere). POST /weg2/park_running {"epoch","reason"} -> 200 {"parked":[rid..],"held":[..],"epoch"}: laufende Requests retrahiert mit Retain + erzwungenem Host-Write-Through (der Schlaf leert den Baum, der Store ist die einzige Kopie, #969D/sgl-project#1068), Streams offen; am Dormant-Punkt des Schlafs stehen sie VOR allen anderen in der sgl-project#1443-Hold (Prefetch waehrend des Flips), die Wake gibt sie aelteste zuerst frei. Kommt der Schlaf nicht, reihen sie sich nach SGLANG_WEG2_D_PARK_AWAKE_REQUEUE_S (30 s, Gruppen-MIN) wieder am Kopf ein; Abort erreicht die Park-Liste. Schalter SGLANG_WEG2_D_PARK (Standard an auf Gruppe D), Stufe-2-Haken SGLANG_WEG2_D_PARK_RESUME_MARGIN_TOKENS (aus). kvso-Manager bleibt aus: kein eigener gepinnter Pool, ein Schreib- und ein Leseweg (HiCache). Tests: test_weg2_d_seats_h91b (12), test_weg2_d_seat_pricing_h91b (10), test_weg2_d_park_scheduler_h91b (13) -- rot auf 0d5a086 (2 Sammelfehler ModuleNotFoundError + 10 Fails), 35 gruen mit dem Fix; betroffene Suiten (29 Dateien) ohne neue Fails, die 12 Fails dort sind vorbestehend und auf der Basis gleich (launcher_teardown_1248 x2, sched_fix3_0907 x4, retract_decode_fcfs x6).
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Sep 28, 2026
…nach N frischen Marken ohne Zuwachs PREFETCH-DEFER-FALLBACK (Match nehmen, Rest auf D extenden) rc12y (ce041f0, D-Log weg2-16-42, 00:07:35-00:13:24): EIN Store-Read von 46850 Tokens endete 692-mal 192 kurz (sgl-project#1324 delivered=46656 deliverable=46848). Jeder Zyklus: Drain setzt eine FRISCHE Marke ('sgl-project#1068 PREFETCH DEFERRED reason=store_prefix_short attempt=1'), der Retry gibt den Read neu aus, das 'issued' raeumt die Marke als LANDED ('sgl-project#1068 PREFETCH LANDED after_passes=1 verdict=issued'), der Read endet wieder kurz. Der Fortschrittszeuge (_weg2_note_prefetch_progress) laeuft nur auf einer Marke, die eine zweite Beobachtung erlebt -- also nie; das X-Gate hielt den Request (X-DEFER bound_s=inf) bis zum DEADMAN BUSY-STARVED der Front (00:13:12Z). Kein Schreiber war unterwegs: P schlief, und D's eigener Vorturn weg2-13-41 hatte am Ende keinen rekurrenten Zustand behalten (sgl-project#1469 RETAIN is_finished=True cache_len=None value=False). Fix (Schranke, kein Schalter): _weg2_store_short_fallback zaehlt je Request die FRISCHEN store-short-Marken, deren geliefertes Praefix nicht gewachsen ist (am Request, also vom Raeumen der Marke unberuehrt; Paesse, keine Sekunden -- jeder Rang markiert im selben Pass aus dem gruppensynchronen Record; waehrend D schlaeft zaehlt nichts, xsn344). Ueber SGLANG_WEG2_STORE_SHORT_MAX_CYCLES (Default 4): Rest <= X (oder kein X-Riegel) -> Marke weg, Request zur Zulassung mit dem gematchten Praefix, D extendet den Rest, Zeile 'PREFETCH-DEFER-FALLBACK ... reason= no_writer_progress'; der kurze Record setzt danach keine neue Marke, bis ein Read MEHR liefert. Rest > X auf dem Fensterpfad -> benanntes W88 (reason=over_x), wie der Zeuge selbst. Unabhaengig vom xsn437-Schalter (NF-Profil store_short_tail aus). Test test_weg2_store_short_fallback_rc12y (5, Metallzahlen 46850/46656/ 46848, X=12288): 4 rot auf ce041f0 (50 Zyklen 'deferred', der Livelock), gruen mit Fix; wachsender Read wartet weiter (gruen auf beiden).
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Sep 28, 2026
…anz nicht mehr in die Host-Release-Queue -- TP0 sammelte ~1,26 Mio. Seiten-Eintraege, der Flush-Reset lief sie einzeln ab
Befund (D-Log boot_weg2_dkrnfh91dprsavisnoadopts0bar1dauer09281111_103bfdf29a_0928_111148.D.log):
- TP0 stand 11:22:57 -> 11:23:14 im Flush vor dem Schlaf ('sgl-project#1476 DISPATCH kind=FlushCacheReqInput dispatch_ms=16372'), zwischen 'R12 HOST-VERDICT reset drops' und 'sgl-project#1068 RESET JOIN ... joined_s=0.00'. TP1/TP2 resetteten 11:22:57.
- Das ist kein Warten auf Store-Ops/Backup-Acks, nicht der sgl-project#1068-Join (0,00 s) und nicht die W88-Schleife selbst. Es ist der Reset-Gang ueber die Host-Release-Queue (_weg2_release_queued_refs_before_reset, ein torch-Aufruf je Eintrag). Er gab nichts zurueck und schrieb deshalb keine Zeile.
- Gefuellt hat die Queue der W88-Retry-Sturm von weg2-10-50: 1299x 'sgl-project#915 PREFETCH TRUNCATED need=69952 got=5760 lost=64192' nur auf TP0 (Worker lost=0).
- Der Gruppen-Trim legte den abgeschnittenen Schwanz jedes Mal in die Queue, je 64-Id-Seite ein Eintrag, also ~1000 je Pass. Das sind Lese-Platzhalter aus alloc_read (kein Slot, keine Referenz); der Drain verwirft sie ohnehin (ARENA-QUEUE-REFS placeholders=64 je Aufruf, kumulativ 2,4 Mio.).
- Der Drain einigt sich auf das Gruppen-MIN der Queue-Groessen. Einen Ueberschuss, den nur TP0 hat, erreicht er nie.
- Derselbe Queue-Gang im Zensus-Thread: ARENA-REF-HOLDERS n=7 faellig 11:22:40, gedruckt 11:23:03 (snapshot=torn); HOST-ANON-STACK 11:22:53 zeigt ihn in unified_radix_cache.py:1639 (queue-Summe).
Fix:
- Der Schwanz eines Gruppen-Trims wird nur noch gequeued, wenn host_indices echte Zeilen sind (alloc). Platzhalter aus alloc_read werden gezaehlt (_weg2_trim_placeholders_dropped), nicht gequeued.
- Der Reset-Gang misst sich. '#1424g ARENA-REF RESET-QUEUE' traegt jetzt ms= und erscheint auch ohne Rueckgabe, sobald der Gang >= 100 ms dauert.
- Kein Schalter.
Test test_weg2_trim_placeholder_tail_rc12z17 (4), ueber das echte prefetch_from_storage:
- rc12z17-Form (need 69952, Gruppe 5760): 0 Ids gequeued.
- 64 Retry-Paesse: Queue bleibt leer.
- Staging-Schwanz (alloc): weiter freigegeben.
- Ohne Trim: keine Freigabe.
- Die ersten beiden sind rot auf 1bab093 und gruen mit Fix; die beiden Waechter sind auf beiden Seiten gruen.
Gate: 35 Testdateien, die prefetch_from_storage, _reset_full oder den Reset-Gang beruehren (ohne test_unified_radix_cache_unittest, OOM in der Cgroup).
- Fix: 33 failed / 396 passed.
- Basis 1bab093: 35 failed / 394 passed.
- Die 33 Altlasten sind identisch, 0 neue Rote.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Sep 29, 2026
…fällt benannt auf die Lesetiefe zurück (Umleitung über P statt 503) Metall NF dkrnfh91dprsavisnoadoptstcutvsyncbar1dauer09290232 (e7c0200), D TP0: 03:20:16 sgl-project#1068 DEFERRED weg2-122 span=96129 -> 03:20:17 DEFER RELEASED after_passes=11 (Zeuge 96000) 03:20:56 WEG2-D-PARK park_running ... parked=[... weg2-122-144 ... weg2-126-148] 03:21:06 HiCache prefetch INCOMPLETE req=weg2-122-144 ... deliverable=97024 shortfall=67328 03:21:06 W88 ... rid=weg2-122-144 arm=store_prefix_short span=97025 site=drain no_progress_passes=10 ... witness=(0, 0, 0, 0, 0, 0, 0, 96000) 03:21:06 W88 ... rid=weg2-126-148 ... no_progress_passes=1 ... witness=(..., 7872) 03:21:14 PREFETCH-DEFER-FALLBACK rid=weg2-122-144 delivered=29696 tail=67339 X=12288 cycles=5 bound=4 reason=over_x -- named W88 Front 03:21:06 LEG2-TERMINAL-NAMED rid=weg2-122-144/-126-148 reason=error:W88 (Client: 529) Wurzel 1 (warum 1 Pass reicht): der Park eröffnet einen neuen Lesezyklus und löschte nur den sgl-project#1324-Stempel. Der Fortschrittszeuge des alten Zyklus (_weg2_best_delivered 96000, _weg2_progress_terms, 9 Pässe, Wanduhr 49 s alt) und die Store-short-Schranke (cycle_best/cycles/fallback) überlebten. Der erste Drain nach dem Wake maß den frischen Read (29696) gegen 96000 = "kein Fortschritt", die 30-s-Wandschranke war längst abgelaufen -> terminal 0,6 s nach Lesebeginn. Fix: d_park_read.clear_read_cycle() an beiden Zyklusanfängen (park_running, RESUME-VIA-P keep_on_d) räumt Stempel, Zeugen und Schranke. Wurzel 2 (was fehlt im Store): Seite 464 (Token 29696) von weg2-122-144 lag weder im L2 noch im L3 (READ-STAGES pages=464 bei hit_pages=1500, zweiter Read l3fill 377 und wieder 464); am Park 03:20:56 meldeten die Form-A-Worker ARENA-CLAIM REFUSED pages=464 (no free slot) / sgl-project#1421 BACKUP-REFUSED tokens=29696. Lesbar war der Präfix 29696, der Rest 67339 > X. Fix 2: der Store-short-Arm über X (Schranke over_x UND Zeugen-Stillstand) endet nicht mehr im W88-503, sondern BENANNT in W88-REROUTE: Markierung geräumt, Request zur X-Gate mit dem gelieferten Präfix; die Gate (ohne Ausnahme) verweigert den Rest und leitet über P um (RESUME-VIA-P mitten im Stream / W50 vor dem ersten Byte), P liest den Store und prefillt nur den Rest. Kein Prefill über X auf D, kein Neurechnen von 0. W88 bleibt für den Arm host_pool_shortfall (eigener Staging-Pool, kein fehlender Präfix). Schwelle je Arm: store_prefix_short hat keinen Client-Tod mehr, die Pass-/Wandschranke entscheidet nur, wann auf den Schreiber nicht mehr gewartet wird, und zählt ab dem Zyklusanfang. Tests (CUDA_VISIBLE_DEVICES=): neu test_weg2_w88_cycle_0929.py 3/3 auf Basis rot, mit Fix grün; umgestellt rc12y/xsn437/1324 (W88-Vertrag -> Umleitung). Nachbarn 28 Dateien: 305 Fälle, nur die 3 absichtlich umgestellten Vertragsfälle unterscheiden sich von der Basis. Metall-Marker: 'W88-REROUTE rid=... cause=over_x|standstill', danach 'W50-REROUTE ... x_refusal_midstream' bzw. X-REQUEUE; 0x 'W88 Weg2StoreLoadNotProgressing arm=store_prefix_short'.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Oct 2, 2026
…D->P quiesce (WEG2-QUIESCE-PARKED-PREFETCH) N5d 1002_124821 epoch 6 (D->P begin 12:51:58.621, 14.3 s, layer 12.6 s): weg2-6-10 (SHORT on D, 171166 tokens, 169224 in L3) was admitted 12:51:48; its store read (READ-STAGES l3fill_pages=104379 total_ms=15886) still ran when the front parked D (park_running ... queued-behind=['weg2-6-10']). Every /flush_cache quiesce poll answered 400 'hicache_prefetch(1: weg2-6-1)' (547x), FLIP STALL stage=quiesce at 9.8 s -- and the request went into the sgl-project#1443 hold anyway once the read ended, re-read at the wake. Helper weg2_sleep_drain.parked_owned_prefetch(parked, ongoing_prefetch): the open reads of requests in sched.weg2_d_parked. They are a storage->HOST operation whose device half is the wake's load; the release flush's reset terminates and joins them (sgl-project#1068 RESET JOIN) and the sgl-project#248 hold intake defers the read to the wake. Exempt (like H91e's hold-owned reads) in: * Scheduler.group_idle_verdict -- the quiesce poll's and the release flush's idle (module helper _weg2_parked_owned_prefetch_of), marker WEG2-QUIESCE-PARKED-PREFETCH rids=.. n=..; * WeightUpdater._weg2_hold_owned_prefetch -- the sleep drain and the release leg's idle assert. Rank-uniform: the park list is replicated (park_running is a broadcast control request, every rank parks the same list) and prefetch registration is participation-voted. Switch SGLANG_WEG2_PARKED_PREFETCH_NOT_A_SLEEP_TERM (default on; 0 = the old blocker). Test (flock pytest.lock, systemd-run MemoryMax=4G, nice 19, ionice c3, -n 1; 17 files touching group_idle_verdict / idle_blockers / sleep drain): e03d95c: 13 failed (pre-existing: idle_round_fresh_1268, quiesce_fast_h111, tail_fold_store_h63d, ...), 175 passed this commit: the same 13 failed, 179 passed (+4 in test_pdflip_parked_prefetch_1002.py)
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Oct 2, 2026
…quest's store probe L15 boot dac8b62 D->P 14:43:49: drain+quiesce 1866 ms, D flush_cache 1820 ms, 'WEG2-SLEEP-SUB alloc_clear=1414' = 'sgl-project#1068 RESET JOIN terminated_ops=1 joined_s=1.29' (TP0) / 0.94 (TP1): the release reset joined the running probe of the parked weg2-6-6 (133k keys in one batch_exists_v2, no batch boundary inside, so the termination landed only at its end). HiCacheController.reset(): when every in-flight prefetch operation belongs to D's park list (the scheduler hands PDFLIP-A's exempt set to the controller around the flush's tree reset), no store->host page transfer is in flight and, after a 50 ms grace join, only the prefetch thread is alive, the operations are terminated as before and that thread is joined by a background reaper 'hicache-reset-reaper', which then discards the old control queues and restarts the pipeline. The probe reads the store only -- a terminated operation pins nothing (sgl-project#257) and transfers nothing (sgl-project#1068) -- so the host pool clear cannot collide with it. Until the reaper is done: the radix caches' storage-control drains skip their round (a release the old thread queues names slots of the cleared pool and is discarded with the queue, never freed into a reallocated slot); prefetch(), write_storage() and any further stop wait for it; a second reset (the release flush) merges. A reaper that cannot join within the 10 s bound takes the thread STOP path. Markers 'sgl-project#1068 RESET JOIN ... detached=prefetch rids=...', 'sgl-project#1068 RESET MERGED ...', 'sgl-project#1068 PARK-READ-DETACH reaped ...' / '... await why=...'. Switch SGLANG_WEG2_PARK_READ_DETACH (default on). Gate (pytest -n 1, capped, CUDA_VISIBLE_DEVICES empty) over the 27 files touching the storage reset/stop/drain: 278 passed, 10 failed; the same 10 fail on 446d67d (staging_write_ring_810 / flip_writeback_703 / tail_fold_store_h63d fixtures missing attributes). The 900-s test_unified_radix_cache_unittest.py did not finish inside the cap and is not counted. New tests: 4 red on 446d67d, all 6 green here.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Oct 2, 2026
…DETACH: the sleep flush's reset no longer joins a parked request's store probe) Conflict in managers/scheduler.py, the plain D flush: kept L15's form (_l15_pr0 clock, L15-PLAIN-RESET line, one _fsub 'alloc_clear' mark outside it -- PDFLIP-L) and wrapped its tree reset + req/alloc clears in the PARK-READ-DETACH hand-over (_weg2_reset_detach_rids set before, cleared in a finally). The L15-D2FIX seam test (test_weg2_l15_sleep_keep_clear_1002) finds the reset one indent deeper and reads a 5000-char window. Gate (pytest -n 1, capped, CUDA_VISIBLE_DEVICES empty): park_read_detach, pdflip_parked_prefetch, l15_keep_split, l15_sleep_keep_clear, l15_unbacked_refuse, sleep_drain_group_x105, the three sgl-project#1068 controller reset/stop files: 42 passed.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Oct 2, 2026
…es (instrument) N6a (..._4dd76c122a_1002_164400) P sleeps: WEG2-SLEEP-SUB rpc_flush tree_reset 31-47 ms on every PP rank (total 34-77), the block of the P>D quiesce. Inside UnifiedRadixCache._reset_full only the H81 RESET-RELEASE prints a time (9.5-11.5 ms); the storage threads' stop/start (sgl-project#1068 RESET JOIN joined_s=0.00), the host pool clear, the #1424g orphan give-back (torch passes over the arena ledger) and the at=reset holder census are unclocked. On the desk shell the same reset is 1.5 ms warm -- the cost scales with the real arenas/threads, so the split has to come from metal. - _reset_full stamps unpin / release_host / tree_rebuild / queued_refs / controller_reset / host_clear / orphans / census / tail and prints 'WEG2-TREE-RESET-SUB total= <steps> threads_start=' (only where a cache controller exists, i.e. HiCache trees); HiCacheController.reset clocks its _start_storage_threads into _weg2_last_start_ms (threads_start). No branch reads any of it. Tests: test/registered/unit/mem_cache/test_tree_reset_sub_1002.py (3, on the #1424g arena shell: steps in order and adding up to total, no controller no line, the restart clock). Gate, 14 reset/hicache files, base 5e14459 vs change: base 5 failed (2 new-file reds + 3 pre-existing: rebind_0828 window source, protected_residue_938 x2) / 121 passed, change 3 failed (the same 3) / 123 passed.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Oct 14, 2026
…eady holds decides at its first re-read without growth NF h16 a2cfc77 (boot ..._1014_151524), D TP0, weg2-8-68 (5 images, 176133 tokens, lane 1, routed SHORT -> D on credit=175744 src=d_served_anchor with no device copy left): 15:27:55 the read landed at 64576 ('sgl-project#257 BELOW-ANCHOR read=65536 anchored=64576', '#1324c ... store_hit=175872 delivered=64576'); 15:28:00/:06/:11 three identical re-reads (matched=64576 loaded=0, 5.2 s each, l3fill 730 pages), each 'sgl-project#1068 PREFETCH DEFERRED reason=store_prefix_short attempt=1' ("the store did not hold the requested prefix yet"); 15:28:16 the rc12y bound (4 cycles) fell back -> W31/W50 -> PARK-IMMEDIATE -> flip. The store HELD 175872: P loaded all of it at 15:28:22-24 ('sgl-project#988 LOADBACK ... anchor_depth=175872'). The store-short wait exists for a WRITER still filling the store. A read that ended short of the store's own hit (deliverable, #248h, the group-synced END reading of the drain) and did not grow on its re-read has no writer to wait for. SGLANG_WEG2_STORE_SHORT_HELD_DECIDE (default on): bound 1 instead of SGLANG_WEG2_STORE_SHORT_MAX_CYCLES for such a read; the fallback names reason=store_held_no_growth / store_held=<hit>. A growing read is waited for as before, a dormant D still does not count (xsn344), the settled-wake bound 0 is unchanged. Off: the cycle bound alone. Tests (pytest_gedeckelt.sh, no GPU): test_weg2_store_held_decide_1014 (new, 6, metal numbers of weg2-8-68 and rc12y); the three files that pin the cycle bound itself run with the switch off (one autouse fixture each); all 19 files that reference the store-short path: 188 passed. test_allreduce_admission_pin_1537::test_p5 is red at the base 38c6d40 already (resume_via_p.py reads 'priority' since the lanes), not touched.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Thank you for your contribution, we really appreciate it. The following instructions will help improve your pull request and make it easier to receive feedback. If there are any items you don't understand, don't worry. Just submit the pull request and ask the maintainers for help.
Motivation
Please explain the motivation behind this PR and the goal you aim to achieve with it.
Modification
Briefly describe the changes made in this PR.
Checklist
pre-commit run --all-filesor other linting tools are used to fix potential lint issues.