Skip to content

fix: not use the default port - #1068

Merged
zhyncs merged 1 commit into
sgl-project:mainfrom
zhyncs:port
Aug 13, 2024
Merged

zhyncs merged 1 commit into
sgl-project:mainfrom
zhyncs:port

Conversation

@zhyncs

@zhyncs zhyncs commented Aug 13, 2024

Copy link
Copy Markdown
Contributor

Thank you for your contribution, we really appreciate it. The following instructions will help improve your pull request and make it easier to receive feedback. If there are any items you don't understand, don't worry. Just submit the pull request and ask the maintainers for help.

Motivation

Please explain the motivation behind this PR and the goal you aim to achieve with it.

Modification

Briefly describe the changes made in this PR.

Checklist

  1. Ensure pre-commit pre-commit run --all-files or other linting tools are used to fix potential lint issues.
  2. Confirm that modifications are covered by complete unit tests. If not, please add more unit tests for correctness.
  3. Modify documentation as needed, such as docstrings or example tutorials.

@zhyncs

zhyncs commented Aug 13, 2024

Copy link
Copy Markdown
Contributor Author

fix https://github.com/sgl-project/sglang/actions/runs/10358622990/job/28673239739

Run models/test_generation_models.py


test_prefill_logits (models.test_generation_models.TestGenerationModels) ... The attention mask and the pad token id were not set. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.
Setting `pad_token_id` to `eos_token_id`:128001 for open-end generation.
The attention mask is not set and cannot be inferred from input because pad token is same as eos token. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
The attention mask and the pad token id were not set. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.
Setting `pad_token_id` to `eos_token_id`:[1280](https://github.com/sgl-project/sglang/actions/runs/10358622990/job/28673239739#step:4:1281)01 for open-end generation.
The attention mask and the pad token id were not set. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.
Setting `pad_token_id` to `eos_token_id`:128001 for open-end generation.
INFO 08-12 20:21:48 weight_utils.py:225] Using model weights format ['*.safetensors']
ERROR

Stderr:
/data/zhyncs/runner-v2/actions-runner/_work/sglang/sglang/python/sglang/srt/utils.py:190: DeprecationWarning: The 'warn' method is deprecated, use 'warning' instead
  logger.warn(
WARNING: Port 30000 is not available. Use port 10000 instead.

======================================================================
ERROR: test_prefill_logits (models.test_generation_models.TestGenerationModels)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/data/zhyncs/runner-v2/actions-runner/_work/sglang/sglang/test/srt/models/test_generation_models.py", line 67, in test_prefill_logits
    self.assert_close_prefill_logits_and_output_strs(
  File "/data/zhyncs/runner-v2/actions-runner/_work/sglang/sglang/test/srt/models/test_generation_models.py", line 44, in assert_close_prefill_logits_and_output_strs
    with SRTRunner(
  File "/data/zhyncs/runner-v2/actions-runner/_work/sglang/sglang/python/sglang/test/runners.py", line 183, in __init__
    self.runtime = Runtime(
  File "/data/zhyncs/runner-v2/actions-runner/_work/sglang/sglang/python/sglang/srt/server.py", line 510, in __init__
    raise RuntimeError(
RuntimeError: Initialization failed. Please see the error messages above.

Stderr:
/data/zhyncs/runner-v2/actions-runner/_work/sglang/sglang/python/sglang/srt/utils.py:190: DeprecationWarning: The 'warn' method is deprecated, use 'warning' instead
  logger.warn(
WARNING: Port 30000 is not available. Use port 10000 instead.

----------------------------------------------------------------------
Ran 1 test in 35.878s

FAILED (errors=1)
Fail. Time elapsed: 826.18s
Error: Process completed with exit code 255.

@zhyncs

zhyncs commented Aug 13, 2024

Copy link
Copy Markdown
Contributor Author

ref
https://github.com/sgl-project/sglang/tree/main/test/srt/models

- name: Test Backend Runtime
run: |
cd test/srt
python3 run_suite.py --suite minimal

"models/test_generation_models.py",
"models/test_embedding_models.py",

@zhyncs
zhyncs requested review from Ying1123 and hnyls2002 August 13, 2024 05:09
@zhyncs
zhyncs merged commit 65e89ba into sgl-project:main Aug 13, 2024
@zhyncs
zhyncs deleted the port branch August 13, 2024 05:12
timethink pushed a commit to timethink/sglang that referenced this pull request Mar 9, 2025
efschu pushed a commit to efschu/htsglang that referenced this pull request Sep 1, 2026
…hot: an instrument may never gate

Boot 1067park, second fence (tp_to_pp, 06:49:55Z): _1063_record_fence walks
every page stem of every eligible node through 2-4 os.path.exists() against
a 1.69-million-file store. The first fence ran warm (221862 stems, ~3 s);
the second hit the re-prefill's fresh stems cold and never returned -- PP0's
last line for 25 minutes was the FENCE-NODES header, no abandon could run
(PP0 is the sole timeout carrier), and the deadman killed the boot 07:15:37.

Fix: hard 2.0 s budget; an expired scan reports SCAN CAPPED (counts declared
a sample, denominator law) and bumps fence_scan_capped. The two-point
consumer (_1063_probe_since_fence) reads only the 4096-capped _1063_AT_FENCE
dict and loses no coverage it ever had.

Trap recorded in a comment: 'now' is a PARAMETER of the fence function, not
a module name -- a bare now() here would NameError into the outer except and
silently delete the snapshot (sgl-project#872 silent-zero shape). time.monotonic() used
explicitly.

Desk-proven by execution, both arms: uncapped emits normally; a slow clock
trips SCAN CAPPED.
efschu pushed a commit to efschu/htsglang that referenced this pull request Sep 1, 2026
…and returns

Against cbbb619: structural pin (_scan_deadline exists, set before its
in-walk check, SCAN CAPPED self-report present) plus a behavioral arm --
a store whose stem probe costs 50 ms each (4096+ stems would take minutes
unbudgeted) must return within the budget and still record a snapshot;
measured green at ~2 s. Full file: 16 passed.
efschu pushed a commit to efschu/htsglang that referenced this pull request Sep 2, 2026
… (905 double, 1035c stub)

Two test stand-ins that the branch's new production contracts outgrew
(sgl-project#624/sgl-project#978 stand-in-drift class). Production is the reviewed, accepted
contract in both cases and is NOT touched; only the stand-ins move.

DRIFT 1 -- the sgl-project#905 census-scheduler double missed the A12.2 cutover clearer.
  test/registered/scheduler/seam_census_double.py:417 (FaithfulCensusScheduler)
  binds the REAL Scheduler.readmit_seam_residents, which since the slice-3 fix
  calls self._clear_prefetch_deferral_for_reissue(population) at
  python/sglang/srt/managers/scheduler.py:5810 (definition :6292, lifecycle
  CLEARER (c) in the table at :5969). The double bound the caller but not the
  helper -> AttributeError, surfacing as "retry() exceed maximum number of
  retries".
  Fix: bind BOTH real methods (_clear_prefetch_deferral_for_reissue and the
  _clear_prefetch_deferral_fields it calls) from the real class, not a no-op.
  A no-op stand-in would be the W29 direction again: the double would pass a
  cutover that never cleared, i.e. exactly the defect the clearer exists for.
  The A12.2 fields are deliberately NOT pre-planted on FaithfulReq -- the real
  Req does not declare them either (no occurrence in schedule_batch.py); they
  are planted dynamically by _apply_prefetch_deferral and every reader uses
  getattr(..., default). Absence is the faithful state.
  This is the same edit the slice-3 fix already made in the two sibling
  doubles (test/registered/unit/managers/test_seam_readmission_w31.py:87-89
  and test_prefetch_deferral_1068.py:195-197); the 905 double was the one
  instance missed by that sweep.

DRIFT 2 -- the #1035c stub missed the termination protocol.
  python/sglang/srt/mem_cache/hybrid_cache/hybrid_cache_controller.py:752
  reads operation.is_terminated() DIRECTLY (branch commit 846c679, slice 2
  fix 4). That override is a FULL replacement of the base probe -- one
  batch_exists call over the whole span, no per-batch loop -- so the base
  class's per-batch guard never reaches it, and the class-agnostic getattr
  form at managers/cache_controller.py:851 is a different pass and does not
  apply here. The stub at
  test/registered/unit/mem_cache/test_zero_answer_partition_1035c.py:172 built
  the operation without the protocol.
  Fix: the stub answers is_terminated() -> False (a live operation, which is
  what all four fixtures are about), modelling
  StorageOperation.is_terminated at managers/cache_controller.py:287.
  mark_terminate is deliberately NOT added: this path never calls it, and a
  stub method nothing drives is surface without a contract behind it.
  hybrid_cache_controller.py:752 is NOT weakened to a getattr.

TALLY (hermetic: CUDA_VISIBLE_DEVICES="", PYTHONPATH=<worktree>/python,
pytest -q -p no:cacheprovider, per file, run once each)
  test_seam_census_double_905.py       before 6 failed / 14 passed
                                       after  0 failed / 20 passed
  test_zero_answer_partition_1035c.py  before 4 failed /  5 passed
                                       after  0 failed /  9 passed

THE FOUR 905 REDS THAT PRE-DATE THIS BRANCH
  test_every_resident_is_retracted_readmitted_and_off_the_live_set,
  test_no_row_belongs_to_nobody_afterwards, test_the_drop_evicted_before_it_
  reset, test_the_seam_stamped_its_own_retractions were already RED at the pin
  228a66d, so the branch gate counted only the other two as truly new.
  They go GREEN here with the same stand-in fix and no further edit. Their
  pin-era cause is therefore not present on this branch: the sgl-project#1068 slice work
  rewrote the production seam and this suite together (git diff
  228a66d..HEAD touches seam_census_double.py +47/-33 and
  test_seam_census_double_905.py +60), and the clearer binding was the last
  piece missing. Not carried forward as debt -- the file is 20/20.

Checks: both files run once each (above); import smoke of
sglang.srt.managers.scheduler and
sglang.srt.mem_cache.hybrid_cache.hybrid_cache_controller OK; ruff check and
ruff format --check clean on both files. No production file touched.
efschu pushed a commit to efschu/htsglang that referenced this pull request Sep 3, 2026
…; one timeout source; the store witness replaces the seam stamp

ROOT (boot weg1b3 @ 6980c75, rid 679e4568, 84,027 tokens, cutovers 3+4):
_prefetch_timeout_check_linear_func charged base + len(hash_value) x per_page
and hash_value is empty until the serial prefetch thread's store probe has
run, so the every-pass drain reaped the LONG re-admission at the bare base
(effective 1.0 s: the sgl-project#968/sgl-project#1065 re-pricing 2.0/1.0 was a pair of dead
locals overwritten by the hybrid parse's 1/0.25) with completed=0. The store
had the pages; nobody asked it in time. phase_purity then counted the retract
stamp as 'restored' and licensed a P=0 TP prefill (6 x 4096 recomputed).

FIX
F1a unified_radix_cache.py: _prefetch_priced_pages = len(hash_value) or
    len(token_ids)//page_size; _prefetch_timeout_budget_s = base + pages x
    per_page; the linear check compares against it (hiradix sibling same
    form under its own cfg.max; hi_mamba unreachable, registry.py:106-107).
    The unified budget is deliberately UNCAPPED (no PrefetchTimeoutConfig.max):
    an unprobed 1M-token op prices ~1000 s and admission waits on it through
    the PP0 _prefetch_done_for hold. Named, not hidden.
F1b one timeout source: hybrid_cache_controller.parse_storage_backend_
    extra_config pops its defaults from PrefetchTimeoutConfig (2.0 s + 1.0 s
    per KiToken); dead 2.0/1.0 locals and the 3.0/0.25 re-attach literals
    deleted; 'sgl-project#1157 PREFETCH TIMEOUT base= per_ki_token= source=' printed once
    per attach.
F1c the reap is a line: 'sgl-project#1157 PREFETCH REAPED req= probed= requested_pages=
    hit_pages= completed= elapsed= budget=' per termination that is not a
    local completion. PrefetchOperation.probed_hit_tokens is stamped by the
    prefetch thread after its MIN; the record the admission loop pops
    (prefetch_loaded_tokens_by_reqid) is a PrefetchOutcome(int) annotated
    with hit_tokens/probed, written at completion, reap and revoke.
F1d the seam premise is a measured witness: store_witness() reads the
    tree's own prefetch state (pending / hit / bounded / cold / unprobed);
    the `cache_protected_len>0 or cached_prompt_tokens_at_retract>0` stamp
    arm is DELETED. StoreWitnessContradiction is raised from the premise and
    from both admission arms and re-raised through the three fail-open
    probes; it leaves get_next_batch_to_run uncaught and is the group STOP
    (run_scheduler_process -> SIGQUIT), never a P=0 admission.

REVIEW FIX (operator decisions 2026-09-03):
B1 PrefetchOutcome now round-trips pickle/copy.deepcopy: positional
   __new__(loaded, hit_tokens=None, probed=False) + __reduce__. The
   keyword-only form survived dumps but not loads and rode
   req.storage_hit_length -> cached_tokens_storage -> details['storage'] ->
   send_pyobj -> detokenizer death on the first partial store hit. Both
   admission writes store int(loaded_tokens). Int subclass kept over a
   dataclass: smaller diff, every existing int reader untouched.
B2 beside a stamp the witness MEASURES the stamped span: 'hit' only if
   loaded > 0 and stamp - hit_tokens <= one HiCache chunk; allowance =
   tree._prefetch_chunk_tokens (chunked_prefill_size, the one-chunk bound the
   sgl-project#1068 L2 truncation line already prices; fallback prefetch_threshold when
   the tree has no term). A revoke (loaded=0, the 40-token chat header
   answering every probe) or a probed shortfall beyond the allowance beside
   a stamp is a contradiction. Cold requests (stamp 0) keep today's reading.
B3 raise text renamed to 'sgl-project#1157 STORE WITNESS CONTRADICTION ...' and the
   class to StoreWitnessContradiction: 'SEAM RESTORE' tripped the sgl-project#1068
   seam-copy zombie gate (test_968_seam_copy_deleted_1068 red on R1, green
   now). Functions renamed store_witness / assert_store_witness_at_admission.
N1 rank-uniform reap annotation: probed (MIN) and hit_tokens (MIN) ride two
   extra slots of the EXISTING packed all_reduce in check_prefetch_progress
   (the one that reduces min_completed_tokens); the record and the REAPED
   line are derived from the reduced vector only, never the thread-timing
   stamp. No new collective.
N2 phase_policy.py hold reason now names the store witness instead of
   'premise verified on the retract credit'.

POPULATION CHANGES NAMED: (a) 'bounded' state: a stamped request shorter
than prefetch_threshold counts as restored (no read can exist; recompute
bounded below the threshold; the /health_generate shape). (b) A stamped
request with no record, no ongoing prefetch and a prompt >= threshold
(rate-limited, storage off, deferral refused) is now 'cold' -> premise
refused -> exemption closed, where the stamp used to open it. (c) A mixed
queue still opens on the witnessed request and a cold sibling rides at P=0
(pre-existing); the boot-4 count 'sgl-project#1157 PREFETCH REAPED probed=False
completed=0' = 0 for retract re-admissions is the metal proof no LONG rides
at P=0. (d) PP followers (pp_rank>0) run assert_store_witness_at_admission
too: a follower raising is a process death via SIGQUIT (a STOP), not a
verdict; it is not gated to pp_rank==0 here.

RANK UNIFORMITY of the witness inputs: ongoing_prefetch removed only after
the group MIN / under the drain barrier / at the cutover sweep;
prefetch_loaded_tokens_by_reqid written from the reduced vector (N1) or on
the revoke path whose enqueue the prefetch thread's MIN precedes;
prefetch_deferred set from a rank-local counter but cleared/retried under
the group gate (sgl-project#1068 slice 3 fix).

TESTS (hermetic, CUDA_VISIBLE_DEVICES="" PYTHONPATH=<tree>/python)
test/registered/unit/mem_cache/test_1157_reaper_prices_requested_span.py (20)
test/registered/unit/managers/test_1157_seam_witness.py (25)
Red measured before the fix on the R1 build: pickle.loads/deepcopy TypeError;
_witness_from_outcome(stamp=80009, PrefetchOutcome(0,hit=40,probed)) == 'hit';
(4096,hit=4096) beside 80009 == 'hit'; no _reap_annotation_from_packed;
no StoreWitnessContradiction. On the parent e63fd08 both files fail at
collection (PrefetchOutcome / StoreWitnessContradiction absent).
Inverted tests carry withdrawal docstrings: 861j
test_a_stamp_without_a_store_read_no_longer_opens_the_gate, W30
test_a_stamp_alone_no_longer_opens_the_exemption, 968
test_premise_with_stamps_and_host_content_no_longer_holds_alone.
Mutants killed (5/5): M1a keyword-only __new__ restored -> 2F;
M1b int() dropped at the TP admission site -> 1F; M2 shortfall term dropped
-> 2F; M3 reap annotation read from the local stamp -> 1F;
M4 'SEAM RESTORE' token restored -> 3F (incl. the sgl-project#1068 zombie gate).

SUITE (bounded selection, identical on both trees: test/registered/unit/
mem_cache + distributed/test_prefetch_progress_symmetry_580.py + managers
test_*phase_purity*/test_*phase_policy*/test_*seam* minus
test_phase_flip_spec_seam_631.py + test_968_starvation_umbau.py +
test_tp_decode_formation_861j.py; pytest -q -p no:cacheprovider -rf,
single process; the two new files EXCLUDED from the parent run, appended
on the fixed run):
  parent e63fd08 (scratch worktree): 125 failed / 3446 passed / 927 skipped (174 s)
  fixed  (this tree):                    125 failed / 3492 passed / 927 skipped (172 s)
  per-test FAILED-id diff (comm): 96 FAILED ids on each tree, IDENTICAL sets (comm -13 and comm -23 both empty): 0 new failures, 0 newly passing; +46 passed = the 45 new tests plus one (not attributable from -q output; FAILED, ERROR and skipped sets identical)
  pre-existing failing files (parent, top): test_phase_policy.py 35,
  test_pool_invariant_live_double_912b.py 12,
  distributed/test_prefetch_progress_symmetry_580.py 8,
  test_seam_abandon_cap_631.py 7, test_mamba_double_free_924.py 5.
Ruff parity measured per touched file (fixed / parent): cache_controller
14/14, phase_policy 0/0, phase_purity 0/0, scheduler 102/102,
hicache_storage 0/0, hiradix 1/1, hybrid_cache_controller 0/0,
unified_radix_cache 1/1, 968 0/0, W30 5/5, 861j 9/9; new files 0/0;
totals 132 = 132. Not claimed clean.

EVIDENCE TIER: DESK-PROVEN. Boot 4 proof lines: 'sgl-project#1157 PREFETCH TIMEOUT
base=2.00s per_ki_token=1.00s source=PrefetchTimeoutConfig' once per rank;
'sgl-project#1157 PREFETCH REAPED ... probed=False' = 0 lines for retract
re-admissions; 'HiCache prefetch success req=<LONG> completed_local=N>0' at
the first pp_to_tp cutover; 'sgl-project#788 ADMIT ... prefix_lens>0' for the LONG;
'sgl-project#1157 STORE WITNESS CONTRADICTION' genuine count 0 (trapsafe_count.py
--marker 'sgl-project#1157 STORE WITNESS CONTRADICTION' --genuine-regex
'RuntimeError: sgl-project#1157 STORE WITNESS CONTRADICTION|StoreWitnessContradiction:
sgl-project#1157'); a nonzero is a group STOP naming rid/stamped/probed_hit/loaded/
allowance/shortfall.
efschu pushed a commit to efschu/htsglang that referenced this pull request Sep 3, 2026
…follower speaks before it can be stuck

boot_855_weg1b5_cd5bb69607_0903_115008, rid 0c34259f, 11:55:13-11:55:30.
All three ranks issued the same re-admission prefetch (gen 4, keys=13224,
byte-identical FETCH CAP lines). PP0's completed in ~3 s, printed
completed_synced=12288, and ADMITted prefix_lens=12288. PP1's and PP2's had
not terminated. PP1 took the forwarded row into execute_scheduled_prefix and
raised the designed group STOP 14 s later. The number PP0 admitted on was
rank-local by construction: the MIN reduce behind "completed_synced" runs
only under tp_world_size > 1, and this boot is --tp-size 1 --pp-size 3 --
attn_reduce_world=1 on 307/307 HICACHE-ROUND lines of the whole log.

E1  managers/pp_prefetch_completion.py (new, pure, no I/O, no clock) plus a
    per-rid completion carrier on the sgl-project#791 ring lap, modelled exactly on
    sgl-project#968's parked-continuation table (stamp on the way up, absorb at PP0).
    No new all_reduce, no new message, no new blocking point on the
    admission path -- the PP0-authority order records a collective there as
    fatal. PP0 admits only when the group floor covers what it wants to
    schedule; a peer that is silent or still running contributes NO NUMBER
    (INCOMPLETE, never a zero), otherwise the rid is DEFERRED by name while
    other work proceeds. When the same length-priced bound the follower's
    own wait uses expires, PP0 CLAMPS to the group floor through the
    existing sgl-project#1059 note_observed_coverage channel instead of over-telling.
    want <= 0 (no store span) admits unconditionally, so a boot with no
    storage hit takes exactly the pre-sgl-project#1175 path. Kill switch
    SGLANG_PP_GROUP_COMPLETION=0 restores the old admission for an A/B.
    completed_synced now prints synced=yes|no and attn_reduce_world=N beside
    itself, so the field can no longer claim a sync it did not do.

E2  pp_admission_congruence.execute_scheduled_prefix now DRIVES the writer it
    used to suspend: after each poll it calls check_prefetch_progress when
    the tree reports that path collective-free (world 1 = every all_reduce
    skipped by construction), and the expiry text carries
    prefetch_driven_in_loop=<bool> so an expiry can never be read as "the
    bytes were not there" when nothing drove the sole writer. Bounds
    unchanged. The sgl-project#1157 reaper's comment now states where it structurally
    cannot fire (it reports terminations; a prefetch that never terminates
    produces no line, so 0 REAPED is not evidence that nothing was reaped).

E3  Every follower prints one bounded "sgl-project#1175 PREFIX-EXEC UNDER-COVERAGE"
    line at ENTRY, before any bound can expire, naming rid/local/scheduled/
    deficit/bound. PP2 printed nothing at all about this rid and its silence
    was indistinguishable from health. The line also states honestly that a
    rank which never RECEIVES a decision row never calls this function and
    therefore still emits nothing -- that case is what E1+E2 remove.

E4  (a) the healthy local == scheduled path no longer returns silently: one
    bounded, counted "sgl-project#968 PREFIX-EXEC no-op" line, so "0 materialised lines
    on the whole boot" reads as "never needed" rather than "never reached"
    (INDIKATOR-GESETZ). (b) the sgl-project#939 census line now names its population
    (population=retract_closure_only) -- SEAM_READMIT_ATTR is stamped only
    by the retract closure, so the two 13225-token queue-occupant
    re-admissions of this boot produced not one census line. The field is
    placed BEFORE fence_proceeds so sgl-project#1068's tail pin still holds.

EVIDENCE (DESK-PROVEN; no boot, no GPU)
 red-first, parent cd5bb69 + only the three new test files:
   16 failed, 3 passed, 1 collection error (pp_prefetch_completion absent)
 fixed tree: 36 passed (25 managers + 11 mem_cache)
 mutants: 12/12 killed, >= 3 per decision (E1 x4, E2 x3, E3 x3, E4 x2)
 matched checks re-run on the fixed tree: test_pp_admission_congruence_791,
   test_968_starvation_umbau, test_968_deletion_falsifiers,
   test_producer_phase_census_631, test_double_prefill_census_fence_1068,
   test_producer_phase_census_wiring_1061 -> 114 passed, 1 failed
   (test_the_module_imports_no_torch_at_all -- identical failure on the
   parent, "import torch" count 1 on both trees: pre-existing)
 bounded suite once, -n 8, hermetic, managers + touched mem_cache files:
   parent 262 failed / 5435 passed / 12 errors
   fixed  286 failed / 5472 passed / 12 errors
   name-level comm: 35 "regressions" and 11 "cures", ALL of them inside
   test_unified_radix_cache_unittest.py; 0 outside it, 0 involving the new
   files. That module run SOLO and SERIAL is 642 passed / 860 skipped /
   0 failed on BOTH trees -- the delta is the known -n 8 crowding family
   (sgl-project#862/sgl-project#899), not this diff.
 ruff --select=F401,F821,UP037: 2 errors on both trees, the same two
   pre-existing ones; black/isort per file identical on both trees for all
   five touched production files, all four new files clean.
efschu pushed a commit to efschu/htsglang that referenced this pull request Sep 4, 2026
…igure, C2 upstream scope

Two text-only repairs handed over by the round-3 code reviewer. No behaviour
change, no test re-run. The commit also carries the rest of the in-progress
sgl-project#1189 slot-publish change (the test file is new, so its C1 edit cannot be
committed on its own).

C1 -- a boot-8 figure already retracted in production comment prose was still
shipped verbatim by the test that a boot-9 agent reads when it goes red.
Re-measured against
/spinning/evidence-665-f1/boot_855_weg1b8_e9d1a719ac_0904_064622.log:
  grep -c 'running_bs=7768'                    -> 0
  max of the running_bs= series                -> 7771
  grep -c ' running=7768'                      -> 1, at :458441, on a
                                                  'sgl-project#788 PP-ADMISSION verdict=DECLINE'
                                                  line, and the max of that series
  grep -c 'distinct rid'                       -> 0
  grep -c 'PP-ADMISSION verdict=ADMIT'         -> 183
  grep -c 'PP-ADMISSION verdict=DECLINE'       -> 27127
  grep -c '#969N ADMIT'                        -> 150
  distinct rids on '#969N ADMIT'               -> 16
  distinct rids on 'PP-ADMISSION verdict=ADMIT'-> 19 (the 16 are a strict subset)
Edits in test/registered/unit/managers/test_1189_slot_publish_lap.py at :39-46
(module docstring), :447-451 (lap-(A) assertion message) and :1036-1042 (the
sgl-project#1189 OUTCOME failure message): 'running_bs 7768' becomes 'running=7768' with
the :458441 anchor and the instrument named, matching the form the production
comment already uses; 'with at most 3 distinct rids' is DELETED, no query
reproduces it; the 183/27127 pair now names its instrument, because '#969N
ADMIT' counts 150 for the same word.

C2 -- the lap-(A) comment claimed a scope its own helper docstring refutes.
scheduler_pp_mixin.py:5238-5240 closed with "This is COVERAGE FOR A STATE THE
FORK'S OWN PURITY FEATURE INVENTED, not a break with an upstream invariant."
The second clause is false. The new else also fires on the purity-INDEPENDENT
idle lap: scheduler.py's 'ret = None' there is the outer else of 'if not
running_batch.is_empty() and not running_batch.is_prefill_only:', not the
_decode_blocked arm above it. Upstream reaches that identical lap
(main:scheduler.py:2737 carries the same '# Run decode (skip for prefill-only
batches)' arm, ret = None at :2742, and zero occurrences of phase_purity or
phase_decode_blocked_here in that file) and preserves the previous
last_mbs[slot] on it -- guard/publish/no-else three times over, at main
scheduler_pp_mixin.py :148/:155, :320/:326 and :507/:514. So the fork now
clears an entry upstream keeps.
Edit at scheduler_pp_mixin.py:5233-5277: the true, one-step-verifiable first
clause ("STRICT PHASE PURITY CREATED THIS LAP ON PURPOSE", :5225) stays
untouched; only the false clause is replaced, by what
_pp_record_slot_last_batch.__doc__ already says -- the wider scope is
deliberate, toward the non-PP loops' unconditional 'self.last_batch = batch'.
Anchors are written as grep recipes, since both files are under active edit.

Checks: python -m py_compile clean on both files. ruff reports 3 findings in
scheduler_pp_mixin.py (E402 :268, F841 carries_flip_arm, E741 'l'), all three
present unchanged at HEAD on the same symbols; zero findings in the test file.
No pytest run: both edits are text-only.
efschu pushed a commit to efschu/htsglang that referenced this pull request Sep 4, 2026
…ond copy

The conformance check of the final pass found the same sentence C2 retracted
in scheduler_pp_mixin.py still standing, un-retracted, in this test module's
own docstring -- the repaired-copy-beside-unrepaired-copy pattern, one more
level out. Text only, zero re-run.

The claim "this is coverage for a state the fork's own purity feature
invented, not a break with upstream" is false in its second half. The new
`else` also fires on the purity-INDEPENDENT idle lap, which upstream reaches
and where upstream PRESERVES last_mbs[slot] while the fork now clears it.
Replaced with what _pp_record_slot_last_batch.__doc__ already argues, plus
grep recipes (structural rule 2) instead of bare line numbers:

  grep -n 'Run decode (skip for prefill-only batches)' scheduler.py
      -> :9058 on the fork, :2737 on main = the two laps are one lap
  grep -n 'BUT THIS `else` REACHES WIDER' scheduler_pp_mixin.py
      -> :5239 = the production retraction this docstring now points at

py_compile clean. No pytest run (text-only by the reviewer's determination).
efschu pushed a commit to efschu/htsglang that referenced this pull request Sep 9, 2026
…erisation, the acceptance takes the next boot's log

The fixture is a recording of a boot that ran WITHOUT the fix, so no code
change on this branch can turn it green. A test that can only ever be red is
not an acceptance, it is a permanent red people learn to scroll past.

* test_f3_the_walk_reproduces_the_recorded_defect_exactly pins the pre-fix
  number in executable form (37 truncated, 34 orphans) and proves the walk.
* test_f3_invariant_holds_on_a_post_fix_log runs the SAME walk against
  WEG2_F3_LOG -- gate 4 of the boot ticket, the only place the invariant can
  honestly go green. Skips without the variable rather than passing on an
  empty walk, and keeps the population floor.

Also: the landed assertion in the new shortfall arm matched _assert_unmarked,
which is the FULL clearer's contract. sgl-project#1068's landed path clears the mark and
arms the one-pass hold, leaving the timing fields as history. Asserted as
sgl-project#1068 actually specifies it.
efschu pushed a commit to efschu/htsglang that referenced this pull request Sep 9, 2026
…s arm makes reachable for the first time

_admission_held_for_deferred_prefetch records an OPEN hazard: in the TP phase
the PP0 exemption is inert, so every rank holds on its own mark, and sgl-project#1068's
mark is rank-local at both ends. On group D that path was previously
unreachable -- no mark could exist there (symmetric_vote). This arm makes it
reachable, so the precondition is checked rather than inherited: the shortfall
mark comes from MIN-reduced values, so all three ranks hold and release
together. Asserted both directions.
efschu pushed a commit to efschu/htsglang that referenced this pull request Sep 10, 2026
…d two pins that outlived their premise

All eight NEW-only failures of the 7191379 gate, re-extracted per name WITH
messages from the artefact itself (`897d9968ac` and `06d3ed02ad` give the
identical eight; NEW-only 8, disappeared 0).

THE THREE LEDGER FAILURES ARE **TEXT, NOT PRICE** -- classified, not assumed,
which was the one thing that could have been the cycle's serious finding.
The ARM line had gained `S_D` BETWEEN `S` and `M`, and three suites match the
literal substring `ARM S=<s> M=<m>` (two by `in`, one by a list comprehension
that then indexes `[0]` and raised IndexError). `S_D` now goes AFTER `M`, so
the old reading is byte-identical and the new term is purely additive -- which
is also the honest shape: nothing about S or M changed, D simply gained its own
budget. All three pass with no pricing change.
And the pricing was never wrong: `charge_terms(s_gb_d=None)` is byte-identical
to the single-budget form (asserted), while the reclaimable term cannot book
pinned host pools as reclaimable by construction --
`cg_reclaimable_bytes` is `(file - shmem) + slab_reclaimable` and its own
docstring states "anon and unevictable are never subtracted: both are exactly
what the reaper kills to recover". cudaHostAlloc rows are unevictable, so the
12.86 GiB of rings is a CHARGE, never a reclaimable credit. No ledger defect.

THE GUARD IS OUT OF `main()`. `_derive_d_l2_budget` now carries the derivation
and BOTH refusals. `main()` has a pinned law of zero try/except (the single
funnel is `cli()`, sgl-project#1248) and a pinned inline-raise count; the first draft
broke both (`[<ast.Try>] != []` and `6 != 4`). This is the class sgl-project#1248 exists
to stop, and it cost a red gate.

TWO PINS OUTLIVED THEIR PREMISE AND ARE REWRITTEN, NOT DELETED. Both asserted
that `exempt_carrier_exceeds` must EXIST as D's cold-store fallback, with the
reason that an above-carrier prompt whose write-through had not landed would
otherwise have no route. That was true at a fixed 1 GB (30,518 rows, sgl-project#915
limit 27,466). With L2 derived from `--max-kv-per-request` the carrier is at
or above the cap, so there are exactly two cases and both have a route:
* ABOVE the cap -> the front refuses at admission. The cap is the law and 413
  is the answer, not a hole.
* BELOW the cap with an unlanded read -> the X-DEFER arm (sgl-project#1238 fix 7) holds
  it while the read is pending, so it is never priced at its whole extent.

THAT ROUTE IS VERIFIED RATHER THAN ASSERTED IN PROSE, which the operator's
ruling required before any pin was flipped: the new
`test_t9b2_a_pending_store_read_DEFERS_it_never_refuses_it` pins
`_weg2_store_read_is_pending` on BOTH pending shapes (an `ongoing_prefetch`
record and the sgl-project#1068 A12.2 mark) plus the negative case, so the defer arm
demonstrably covers the population the exemption used to. No replacement route
was built and no bypass was added.

AND THE FIFTH INSTANCE TODAY OF A SOURCE ASSERTION MATCHING PROSE, this one in
a test I wrote after naming the class four times: my own absence check tripped
on the `front.py` COMMENT that DOCUMENTS the deletion. The comment is right and
stays; the check now strips comments AND string literals via `tokenize` (a
deleted emitter's log text is a string literal, and keeping one would be the
same false positive wearing quotes) -- and it carries a PLANT-PROOF, because a
stripper that removes too much turns the suite into a rubber stamp: every name
is planted as real code and must be caught, while the same token in a comment
or docstring must not be.

One process note, since both seats got the count right and the names wrong in
different ways: the boot agent's reconciliation listed five of these as green
and quoted messages that belong to BASE failures. Re-extracting name+message
pairs from the junit settles it -- `extraktions-zaehlprobe` applied to a
handover, in both directions. Neither list was dishonest; a name recalled from
code reads exactly like a name read from a result.
efschu pushed a commit to efschu/htsglang that referenced this pull request Sep 26, 2026
…en, POST /weg2/park_running vor D's Schlaf -- Nutzer-Design 25.09. Stufe 1

Sitze: --profile nextflash ohne --d-bs faehrt D jetzt mit 2 Sitzen
(weg2.DEFAULT_D_BS_NEXTFLASH, launcher.apply_profile_d_bs_default); ein
gesagtes --d-bs gewinnt. Der D-FRACTION-SOLVE bepreist die Sitze: die
eingebauten NF-Referenzen tragen max_running=1 (7 Mamba-Slots = ceil(1 x 5 x
1.25)), seat_rebook bucht 'mamba state pool' (7 -> 13 Slots, TP0 393.2 ->
730.2 MiB) und 'speculative intermediate state' (linear) auf die wirksamen
--max-running-requests um, VOR H64 -- dessen per_req hatte bei bs2 den
bs1-Posten durch 2 geteilt (halber Zustand je Request). KARTE: Mehr-Allokation
je Rang (TP0 -355.1 MiB Kopfraum in der Ring-Form), der bs2-Decode-Graph ist
NICHT GEBUCHT bis --d-seat-graph-mib (gemessen) gegeben ist. POOL-SCHRITT:
bs2 routet 2 x 4 x top-10 = 80 Ids je Verify-Schritt; Scratch 48 auf den
Workern wirft beim Capture des bs2-Graphen 'Step ids exceed the LRU rows plus
the staging rows' -> W-SITZE-Verweigerung mit den Zahlen vor dem Boot.
expert_pool_device: Plan-Breite folgt dem breitesten gefangenen Schritt
(plan_width_for: bs1 bleibt 64, bs2 128), die Task-sgl-project#40-Schranke zaehlt
min(Ids, E-R). Launcher-Befunde: --d-bs gegen wirksame
--max-running-requests, --cuda-graph-bs-decode deckt 1..Sitze.

Park (weg2/d_seats.py Politik, weg2/d_park_runtime.py Buchhaltung, Scheduler
delegiert): Decode-Druck auf D retrahiert den JUENGSTEN (kvso
session_priority_key; unter Spec nur das Batch-Ende, sonst benannt), die
Spanne bleibt im Baum (retract_retain: KV, GDN-Anker, Draft-Zeilen), er steht
am Queue-Kopf und kommt zurueck, wenn kein aelterer Request mehr lebt;
Neuankoemmlinge warten solange (Barriere). POST /weg2/park_running
{"epoch","reason"} -> 200 {"parked":[rid..],"held":[..],"epoch"}: laufende
Requests retrahiert mit Retain + erzwungenem Host-Write-Through (der Schlaf
leert den Baum, der Store ist die einzige Kopie, #969D/sgl-project#1068), Streams offen;
am Dormant-Punkt des Schlafs stehen sie VOR allen anderen in der sgl-project#1443-Hold
(Prefetch waehrend des Flips), die Wake gibt sie aelteste zuerst frei. Kommt
der Schlaf nicht, reihen sie sich nach SGLANG_WEG2_D_PARK_AWAKE_REQUEUE_S (30 s,
Gruppen-MIN) wieder am Kopf ein; Abort erreicht die Park-Liste. Schalter
SGLANG_WEG2_D_PARK (Standard an auf Gruppe D), Stufe-2-Haken
SGLANG_WEG2_D_PARK_RESUME_MARGIN_TOKENS (aus). kvso-Manager bleibt aus: kein
eigener gepinnter Pool, ein Schreib- und ein Leseweg (HiCache).

Tests: test_weg2_d_seats_h91b (12), test_weg2_d_seat_pricing_h91b (10),
test_weg2_d_park_scheduler_h91b (13) -- rot auf 0d5a086 (2 Sammelfehler
ModuleNotFoundError + 10 Fails), 35 gruen mit dem Fix; betroffene Suiten (29
Dateien) ohne neue Fails, die 12 Fails dort sind vorbestehend und auf der Basis
gleich (launcher_teardown_1248 x2, sched_fix3_0907 x4, retract_decode_fcfs x6).
efschu pushed a commit to efschu/htsglang that referenced this pull request Sep 28, 2026
…nach N frischen Marken ohne Zuwachs PREFETCH-DEFER-FALLBACK (Match nehmen, Rest auf D extenden)

rc12y (ce041f0, D-Log weg2-16-42, 00:07:35-00:13:24): EIN Store-Read von
46850 Tokens endete 692-mal 192 kurz (sgl-project#1324 delivered=46656
deliverable=46848). Jeder Zyklus: Drain setzt eine FRISCHE Marke
('sgl-project#1068 PREFETCH DEFERRED reason=store_prefix_short attempt=1'), der Retry
gibt den Read neu aus, das 'issued' raeumt die Marke als LANDED
('sgl-project#1068 PREFETCH LANDED after_passes=1 verdict=issued'), der Read endet
wieder kurz. Der Fortschrittszeuge (_weg2_note_prefetch_progress) laeuft
nur auf einer Marke, die eine zweite Beobachtung erlebt -- also nie; das
X-Gate hielt den Request (X-DEFER bound_s=inf) bis zum DEADMAN BUSY-STARVED
der Front (00:13:12Z). Kein Schreiber war unterwegs: P schlief, und D's
eigener Vorturn weg2-13-41 hatte am Ende keinen rekurrenten Zustand
behalten (sgl-project#1469 RETAIN is_finished=True cache_len=None value=False).

Fix (Schranke, kein Schalter): _weg2_store_short_fallback zaehlt je Request
die FRISCHEN store-short-Marken, deren geliefertes Praefix nicht gewachsen
ist (am Request, also vom Raeumen der Marke unberuehrt; Paesse, keine
Sekunden -- jeder Rang markiert im selben Pass aus dem gruppensynchronen
Record; waehrend D schlaeft zaehlt nichts, xsn344). Ueber
SGLANG_WEG2_STORE_SHORT_MAX_CYCLES (Default 4): Rest <= X (oder kein
X-Riegel) -> Marke weg, Request zur Zulassung mit dem gematchten Praefix,
D extendet den Rest, Zeile 'PREFETCH-DEFER-FALLBACK ... reason=
no_writer_progress'; der kurze Record setzt danach keine neue Marke, bis
ein Read MEHR liefert. Rest > X auf dem Fensterpfad -> benanntes W88
(reason=over_x), wie der Zeuge selbst. Unabhaengig vom xsn437-Schalter
(NF-Profil store_short_tail aus).

Test test_weg2_store_short_fallback_rc12y (5, Metallzahlen 46850/46656/
46848, X=12288): 4 rot auf ce041f0 (50 Zyklen 'deferred', der
Livelock), gruen mit Fix; wachsender Read wartet weiter (gruen auf beiden).
efschu pushed a commit to efschu/htsglang that referenced this pull request Sep 28, 2026
…anz nicht mehr in die Host-Release-Queue -- TP0 sammelte ~1,26 Mio. Seiten-Eintraege, der Flush-Reset lief sie einzeln ab

Befund (D-Log boot_weg2_dkrnfh91dprsavisnoadopts0bar1dauer09281111_103bfdf29a_0928_111148.D.log):
- TP0 stand 11:22:57 -> 11:23:14 im Flush vor dem Schlaf ('sgl-project#1476 DISPATCH kind=FlushCacheReqInput dispatch_ms=16372'), zwischen 'R12 HOST-VERDICT reset drops' und 'sgl-project#1068 RESET JOIN ... joined_s=0.00'. TP1/TP2 resetteten 11:22:57.
- Das ist kein Warten auf Store-Ops/Backup-Acks, nicht der sgl-project#1068-Join (0,00 s) und nicht die W88-Schleife selbst. Es ist der Reset-Gang ueber die Host-Release-Queue (_weg2_release_queued_refs_before_reset, ein torch-Aufruf je Eintrag). Er gab nichts zurueck und schrieb deshalb keine Zeile.
- Gefuellt hat die Queue der W88-Retry-Sturm von weg2-10-50: 1299x 'sgl-project#915 PREFETCH TRUNCATED need=69952 got=5760 lost=64192' nur auf TP0 (Worker lost=0).
- Der Gruppen-Trim legte den abgeschnittenen Schwanz jedes Mal in die Queue, je 64-Id-Seite ein Eintrag, also ~1000 je Pass. Das sind Lese-Platzhalter aus alloc_read (kein Slot, keine Referenz); der Drain verwirft sie ohnehin (ARENA-QUEUE-REFS placeholders=64 je Aufruf, kumulativ 2,4 Mio.).
- Der Drain einigt sich auf das Gruppen-MIN der Queue-Groessen. Einen Ueberschuss, den nur TP0 hat, erreicht er nie.
- Derselbe Queue-Gang im Zensus-Thread: ARENA-REF-HOLDERS n=7 faellig 11:22:40, gedruckt 11:23:03 (snapshot=torn); HOST-ANON-STACK 11:22:53 zeigt ihn in unified_radix_cache.py:1639 (queue-Summe).

Fix:
- Der Schwanz eines Gruppen-Trims wird nur noch gequeued, wenn host_indices echte Zeilen sind (alloc). Platzhalter aus alloc_read werden gezaehlt (_weg2_trim_placeholders_dropped), nicht gequeued.
- Der Reset-Gang misst sich. '#1424g ARENA-REF RESET-QUEUE' traegt jetzt ms= und erscheint auch ohne Rueckgabe, sobald der Gang >= 100 ms dauert.
- Kein Schalter.

Test test_weg2_trim_placeholder_tail_rc12z17 (4), ueber das echte prefetch_from_storage:
- rc12z17-Form (need 69952, Gruppe 5760): 0 Ids gequeued.
- 64 Retry-Paesse: Queue bleibt leer.
- Staging-Schwanz (alloc): weiter freigegeben.
- Ohne Trim: keine Freigabe.
- Die ersten beiden sind rot auf 1bab093 und gruen mit Fix; die beiden Waechter sind auf beiden Seiten gruen.

Gate: 35 Testdateien, die prefetch_from_storage, _reset_full oder den Reset-Gang beruehren (ohne test_unified_radix_cache_unittest, OOM in der Cgroup).
- Fix: 33 failed / 396 passed.
- Basis 1bab093: 35 failed / 394 passed.
- Die 33 Altlasten sind identisch, 0 neue Rote.
efschu pushed a commit to efschu/htsglang that referenced this pull request Sep 29, 2026
…fällt benannt auf die Lesetiefe zurück (Umleitung über P statt 503)

Metall NF dkrnfh91dprsavisnoadoptstcutvsyncbar1dauer09290232 (e7c0200), D TP0:
  03:20:16 sgl-project#1068 DEFERRED weg2-122 span=96129 -> 03:20:17 DEFER RELEASED after_passes=11 (Zeuge 96000)
  03:20:56 WEG2-D-PARK park_running ... parked=[... weg2-122-144 ... weg2-126-148]
  03:21:06 HiCache prefetch INCOMPLETE req=weg2-122-144 ... deliverable=97024 shortfall=67328
  03:21:06 W88 ... rid=weg2-122-144 arm=store_prefix_short span=97025 site=drain
           no_progress_passes=10 ... witness=(0, 0, 0, 0, 0, 0, 0, 96000)
  03:21:06 W88 ... rid=weg2-126-148 ... no_progress_passes=1 ... witness=(..., 7872)
  03:21:14 PREFETCH-DEFER-FALLBACK rid=weg2-122-144 delivered=29696 tail=67339 X=12288
           cycles=5 bound=4 reason=over_x -- named W88
  Front 03:21:06 LEG2-TERMINAL-NAMED rid=weg2-122-144/-126-148 reason=error:W88 (Client: 529)

Wurzel 1 (warum 1 Pass reicht): der Park eröffnet einen neuen Lesezyklus und
löschte nur den sgl-project#1324-Stempel. Der Fortschrittszeuge des alten Zyklus
(_weg2_best_delivered 96000, _weg2_progress_terms, 9 Pässe, Wanduhr 49 s alt)
und die Store-short-Schranke (cycle_best/cycles/fallback) überlebten. Der erste
Drain nach dem Wake maß den frischen Read (29696) gegen 96000 = "kein
Fortschritt", die 30-s-Wandschranke war längst abgelaufen -> terminal 0,6 s
nach Lesebeginn. Fix: d_park_read.clear_read_cycle() an beiden Zyklusanfängen
(park_running, RESUME-VIA-P keep_on_d) räumt Stempel, Zeugen und Schranke.

Wurzel 2 (was fehlt im Store): Seite 464 (Token 29696) von weg2-122-144 lag
weder im L2 noch im L3 (READ-STAGES pages=464 bei hit_pages=1500, zweiter
Read l3fill 377 und wieder 464); am Park 03:20:56 meldeten die Form-A-Worker
ARENA-CLAIM REFUSED pages=464 (no free slot) / sgl-project#1421 BACKUP-REFUSED
tokens=29696. Lesbar war der Präfix 29696, der Rest 67339 > X.

Fix 2: der Store-short-Arm über X (Schranke over_x UND Zeugen-Stillstand)
endet nicht mehr im W88-503, sondern BENANNT in W88-REROUTE: Markierung
geräumt, Request zur X-Gate mit dem gelieferten Präfix; die Gate (ohne
Ausnahme) verweigert den Rest und leitet über P um (RESUME-VIA-P mitten im
Stream / W50 vor dem ersten Byte), P liest den Store und prefillt nur den
Rest. Kein Prefill über X auf D, kein Neurechnen von 0. W88 bleibt für den
Arm host_pool_shortfall (eigener Staging-Pool, kein fehlender Präfix).
Schwelle je Arm: store_prefix_short hat keinen Client-Tod mehr, die
Pass-/Wandschranke entscheidet nur, wann auf den Schreiber nicht mehr
gewartet wird, und zählt ab dem Zyklusanfang.

Tests (CUDA_VISIBLE_DEVICES=): neu test_weg2_w88_cycle_0929.py 3/3 auf Basis
rot, mit Fix grün; umgestellt rc12y/xsn437/1324 (W88-Vertrag -> Umleitung).
Nachbarn 28 Dateien: 305 Fälle, nur die 3 absichtlich umgestellten
Vertragsfälle unterscheiden sich von der Basis.
Metall-Marker: 'W88-REROUTE rid=... cause=over_x|standstill', danach
'W50-REROUTE ... x_refusal_midstream' bzw. X-REQUEUE; 0x 'W88
Weg2StoreLoadNotProgressing arm=store_prefix_short'.
efschu pushed a commit to efschu/htsglang that referenced this pull request Oct 2, 2026
…D->P quiesce (WEG2-QUIESCE-PARKED-PREFETCH)

N5d 1002_124821 epoch 6 (D->P begin 12:51:58.621, 14.3 s, layer 12.6 s):
weg2-6-10 (SHORT on D, 171166 tokens, 169224 in L3) was admitted 12:51:48; its
store read (READ-STAGES l3fill_pages=104379 total_ms=15886) still ran when the
front parked D (park_running ... queued-behind=['weg2-6-10']). Every
/flush_cache quiesce poll answered 400 'hicache_prefetch(1: weg2-6-1)' (547x),
FLIP STALL stage=quiesce at 9.8 s -- and the request went into the sgl-project#1443 hold
anyway once the read ended, re-read at the wake.

Helper weg2_sleep_drain.parked_owned_prefetch(parked, ongoing_prefetch): the
open reads of requests in sched.weg2_d_parked. They are a storage->HOST
operation whose device half is the wake's load; the release flush's reset
terminates and joins them (sgl-project#1068 RESET JOIN) and the sgl-project#248 hold intake defers
the read to the wake. Exempt (like H91e's hold-owned reads) in:
* Scheduler.group_idle_verdict -- the quiesce poll's and the release flush's
  idle (module helper _weg2_parked_owned_prefetch_of), marker
  WEG2-QUIESCE-PARKED-PREFETCH rids=.. n=..;
* WeightUpdater._weg2_hold_owned_prefetch -- the sleep drain and the release
  leg's idle assert.
Rank-uniform: the park list is replicated (park_running is a broadcast control
request, every rank parks the same list) and prefetch registration is
participation-voted. Switch SGLANG_WEG2_PARKED_PREFETCH_NOT_A_SLEEP_TERM
(default on; 0 = the old blocker).

Test (flock pytest.lock, systemd-run MemoryMax=4G, nice 19, ionice c3, -n 1;
17 files touching group_idle_verdict / idle_blockers / sleep drain):
  e03d95c: 13 failed (pre-existing: idle_round_fresh_1268, quiesce_fast_h111,
              tail_fold_store_h63d, ...), 175 passed
  this commit: the same 13 failed, 179 passed (+4 in
               test_pdflip_parked_prefetch_1002.py)
efschu pushed a commit to efschu/htsglang that referenced this pull request Oct 2, 2026
…quest's store probe

L15 boot dac8b62 D->P 14:43:49: drain+quiesce 1866 ms, D flush_cache
1820 ms, 'WEG2-SLEEP-SUB alloc_clear=1414' = 'sgl-project#1068 RESET JOIN
terminated_ops=1 joined_s=1.29' (TP0) / 0.94 (TP1): the release reset joined
the running probe of the parked weg2-6-6 (133k keys in one batch_exists_v2,
no batch boundary inside, so the termination landed only at its end).

HiCacheController.reset(): when every in-flight prefetch operation belongs to
D's park list (the scheduler hands PDFLIP-A's exempt set to the controller
around the flush's tree reset), no store->host page transfer is in flight and,
after a 50 ms grace join, only the prefetch thread is alive, the operations
are terminated as before and that thread is joined by a background reaper
'hicache-reset-reaper', which then discards the old control queues and
restarts the pipeline. The probe reads the store only -- a terminated
operation pins nothing (sgl-project#257) and transfers nothing (sgl-project#1068) -- so the host
pool clear cannot collide with it. Until the reaper is done: the radix
caches' storage-control drains skip their round (a release the old thread
queues names slots of the cleared pool and is discarded with the queue,
never freed into a reallocated slot); prefetch(), write_storage() and any
further stop wait for it; a second reset (the release flush) merges.
A reaper that cannot join within the 10 s bound takes the thread STOP path.

Markers 'sgl-project#1068 RESET JOIN ... detached=prefetch rids=...', 'sgl-project#1068 RESET
MERGED ...', 'sgl-project#1068 PARK-READ-DETACH reaped ...' / '... await why=...'.
Switch SGLANG_WEG2_PARK_READ_DETACH (default on).

Gate (pytest -n 1, capped, CUDA_VISIBLE_DEVICES empty) over the 27 files
touching the storage reset/stop/drain: 278 passed, 10 failed; the same 10
fail on 446d67d (staging_write_ring_810 / flip_writeback_703 /
tail_fold_store_h63d fixtures missing attributes). The 900-s
test_unified_radix_cache_unittest.py did not finish inside the cap and is
not counted. New tests: 4 red on 446d67d, all 6 green here.
efschu pushed a commit to efschu/htsglang that referenced this pull request Oct 2, 2026
…DETACH: the sleep flush's reset no longer joins a parked request's store probe)

Conflict in managers/scheduler.py, the plain D flush: kept L15's form
(_l15_pr0 clock, L15-PLAIN-RESET line, one _fsub 'alloc_clear' mark outside
it -- PDFLIP-L) and wrapped its tree reset + req/alloc clears in the
PARK-READ-DETACH hand-over (_weg2_reset_detach_rids set before, cleared in a
finally). The L15-D2FIX seam test (test_weg2_l15_sleep_keep_clear_1002)
finds the reset one indent deeper and reads a 5000-char window.

Gate (pytest -n 1, capped, CUDA_VISIBLE_DEVICES empty): park_read_detach,
pdflip_parked_prefetch, l15_keep_split, l15_sleep_keep_clear,
l15_unbacked_refuse, sleep_drain_group_x105, the three sgl-project#1068 controller
reset/stop files: 42 passed.
efschu pushed a commit to efschu/htsglang that referenced this pull request Oct 2, 2026
…es (instrument)

N6a (..._4dd76c122a_1002_164400) P sleeps: WEG2-SLEEP-SUB rpc_flush tree_reset
31-47 ms on every PP rank (total 34-77), the block of the P>D quiesce. Inside
UnifiedRadixCache._reset_full only the H81 RESET-RELEASE prints a time (9.5-11.5
ms); the storage threads' stop/start (sgl-project#1068 RESET JOIN joined_s=0.00), the
host pool clear, the #1424g orphan give-back (torch passes over the arena
ledger) and the at=reset holder census are unclocked. On the desk shell the
same reset is 1.5 ms warm -- the cost scales with the real arenas/threads, so
the split has to come from metal.

- _reset_full stamps unpin / release_host / tree_rebuild / queued_refs /
  controller_reset / host_clear / orphans / census / tail and prints
  'WEG2-TREE-RESET-SUB total= <steps> threads_start=' (only where a cache
  controller exists, i.e. HiCache trees); HiCacheController.reset clocks its
  _start_storage_threads into _weg2_last_start_ms (threads_start). No branch
  reads any of it.

Tests: test/registered/unit/mem_cache/test_tree_reset_sub_1002.py (3, on the
#1424g arena shell: steps in order and adding up to total, no controller no
line, the restart clock). Gate, 14 reset/hicache files, base 5e14459 vs
change: base 5 failed (2 new-file reds + 3 pre-existing: rebind_0828 window
source, protected_residue_938 x2) / 121 passed, change 3 failed (the same 3) /
123 passed.
efschu pushed a commit to efschu/htsglang that referenced this pull request Oct 14, 2026
…eady holds decides at its first re-read without growth

NF h16 a2cfc77 (boot ..._1014_151524), D TP0, weg2-8-68 (5 images, 176133 tokens, lane 1, routed
SHORT -> D on credit=175744 src=d_served_anchor with no device copy left): 15:27:55 the read landed at
64576 ('sgl-project#257 BELOW-ANCHOR read=65536 anchored=64576', '#1324c ... store_hit=175872 delivered=64576');
15:28:00/:06/:11 three identical re-reads (matched=64576 loaded=0, 5.2 s each, l3fill 730 pages), each
'sgl-project#1068 PREFETCH DEFERRED reason=store_prefix_short attempt=1' ("the store did not hold the requested
prefix yet"); 15:28:16 the rc12y bound (4 cycles) fell back -> W31/W50 -> PARK-IMMEDIATE -> flip.
The store HELD 175872: P loaded all of it at 15:28:22-24 ('sgl-project#988 LOADBACK ... anchor_depth=175872').

The store-short wait exists for a WRITER still filling the store. A read that ended short of the
store's own hit (deliverable, #248h, the group-synced END reading of the drain) and did not grow on
its re-read has no writer to wait for. SGLANG_WEG2_STORE_SHORT_HELD_DECIDE (default on): bound 1
instead of SGLANG_WEG2_STORE_SHORT_MAX_CYCLES for such a read; the fallback names
reason=store_held_no_growth / store_held=<hit>. A growing read is waited for as before, a dormant D
still does not count (xsn344), the settled-wake bound 0 is unchanged. Off: the cycle bound alone.

Tests (pytest_gedeckelt.sh, no GPU): test_weg2_store_held_decide_1014 (new, 6, metal numbers of
weg2-8-68 and rc12y); the three files that pin the cycle bound itself run with the switch off (one
autouse fixture each); all 19 files that reference the store-short path: 188 passed.
test_allreduce_admission_pin_1537::test_p5 is red at the base 38c6d40 already (resume_via_p.py
reads 'priority' since the lanes), not touched.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant