Repository navigation
Conversation
Collaborator
Author
|
This issue has been solved by flashinfer-ai/flashinfer#440, thanks to @yzh119 |
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 31, 2026
…nswer their own question sgl-project#1063 -- THE FENCE DECIDER, three states printed side by side rather than one saldo. Both candidate roots for the 92 % empty read-through (714/771 on boot 29) were refuted at the desk, so the next observation must separate what is left instead of summing it: * present_and_readable -- persisted at the fence and still readable now * complete_marker_absent -- a .part706/.slots706 with no .bin: the blob whose writer group never covered the last byte. `_set_canonical_slice` says it itself -- "a writer acting alone leaves nothing readable behind" -- and a cutover tearing an in-flight assembly apart is exactly that shape, forever invisible to every reader. * evicted_since_flip -- was readable at the fence, gone at re-admission. TWO OBSERVATION POINTS, NOT ONE, and that is the whole design: "gone now" and "never there" are the SAME stat() and have opposite fixes. The fence snapshot is what makes the eviction countable at all. Recorded at the fence before staging changes anything; re-read on the sgl-project#1060 cadence, which is the only place on the re-admission path holding a tree_cache. sgl-project#1062 -- THE KEY TRACE MOVES TO THE FUNNEL THAT IS ACTUALLY SYMMETRIC. The #969G probe sat on `_get_component_key`, whose comment claimed "The ONE funnel every store key goes through, read and write alike (6 call sites)". Statically there are FIVE sites and not one of them writes -- `_get_component_path`, `_collect_existing_component_keys` (x2), `batch_exists_v2` (x2), all lookup or existence. Confirmed on metal: across boot 30 the only callers it ever observed were `_collect_existing_component_keys`, `has_component` and a `<genexpr>`. So the instrument built to compare the re-admission's key against the retention's key COULD NEVER SEE A WRITE KEY. Its never-observed difference was a property of its placement. Indicator law, verbatim -- and the lying comment is replaced rather than amended, because it is the next reader's trap. The probe now lives on `_log_key`, which `_write_page` AND `_read_page` both call. The rate limit is gone: the old `_n <= 60 or _n % 1024` printed 602 lines against a population of 235520 on boot 30 (0.3 %), and an absence in that is worth nothing. What remains is an explicit LINE CAP with its own printed suppressed-count, so the trace stays bounded and always names its denominator. MATCHED CHECKS for both failure classes -- a census that collapses its states, and an instrument that never observes the direction it compares. sgl-project#1063 is driven through a real transition (a .bin removed between the two observation points must score evicted_since_flip=1 and NOT still_absent), and the torn-assembly case is asserted as its own state. sgl-project#1062 is asserted at the placement level, because placement is precisely what a behavioural call on one direction cannot show. 7/7 green. py_compile green, ruff clean on the added lines. MEASURE ONLY. Nothing is gated, refused or repaired by either. Order: coordinator (0a)+(0b). Evidence: boots 29/30, specimen /spinning/evidence-665-f1/specimen_1060cens_boot30_uniformwidth_crash/
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 31, 2026
…ompensation layer, and its bugs were not the finding Coordinator order (2), after boots 30 and 31 killed the same mechanism twice at the same site with the same numbers (`told prefix 12288 exceeds this rank's pinned span 0`). Upstream-minimal law: a defect found in a compensation layer is a DELETION candidate, never a repair order for that layer. Two repairs were already spent on it -- sgl-project#1061's epoch gate and #1061b's retract clear -- and the second only existed because the first was built on a clock whose tick was never checked against the act it fenced. DELETED: `pp_uniform_width.py` whole (`uniform_pass_geometry`, `UniformWidthPromiseBroken`, `report_local_coverage`, `min_told`, `PassGeometry`, `epoch_admits_row`, `set_epoch_source`, `current_epoch`); `Req.apply_uniform_pass_geometry_1059` and its call site; the `_1059_told_prefix/_extend/_epoch` stamps; `PPAdmissionEntry.decided_epoch` and its wire column; the publication-time epoch stamp; the epoch-source registration; the told clearing at the retract chokepoint; the `SGLANG_PP_UNIFORM_WIDTH` gate; three test files. WHAT DELIBERATELY STAYED, and each for a named reason: * THE EVICTION FLOOR in `pp_stamp_observed_coverage`. `report_local_coverage` was `max(0, int(x))`, but the same line ALSO raises `cache_protected_len`, which `mem_cache/common.py:82` honours as an eviction floor. Deleting a promise must not silently lower a protection, so the arithmetic is inlined and the floor is byte-identical. Only the promise semantics die. * THE CARRIER COUNTER, renamed to what survives. The #1060b ledger counted the told row at the apply site; that apply is gone, but the question underneath it is not. Boot 29 shipped a carrier whose send gate had been dead since sgl-project#1046 and reported `sgl-project#631=0` while nothing executed -- it cost a window and a census to notice. `sgl-project#1064 CARRIER CENSUS` counts `rows_received` on every rank, so that regression can never be silent again. PP0 never receives, so only a downstream zero is a finding, and the line says so. * `sgl-project#1060`/`sgl-project#1063` untouched: they measure, they never gated anything. WHAT THIS RETURNS TO: the pre-sgl-project#1059 behaviour, in which every rank derives its geometry from its own match. That is not a guess about safety -- boot 29 ran it for its ENTIRE life (`evaluated=87 absent=87`, the row never arrived) across 51 flips with no `sgl-project#631`, no divergence death and no crash. The two boots that DID die at this seam are the two that ran the promise layer. c1 (downstream takes the geometry as a VERDICT) is NOT in this commit, and the reason is a desk finding that inverts its safety argument: `truncate_prefix_to` is MIN, NEVER ASSIGN (schedule_batch.py, sgl-project#930/sgl-project#958) -- so "verbatim" cannot be expressed through the one helper both admission sites use, and a rank holding less than it was told silently keeps its own value, which is the divergence the verbatim form exists to remove. Reported to the coordinator with the file:line rather than worked around. 38/38 green across sgl-project#791/sgl-project#1060/sgl-project#1063/sgl-project#677, codec round-trip verified after the column removal, import smoke of every touched module, ruff clean, py_compile green. Order: coordinator (2). Evidence: boots 30/31, specimens specimen_1060cens_boot30_uniformwidth_crash/ and specimen_1061b_boot32_int8_triton_wedge/
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Sep 1, 2026
…hot: an instrument may never gate Boot 1067park, second fence (tp_to_pp, 06:49:55Z): _1063_record_fence walks every page stem of every eligible node through 2-4 os.path.exists() against a 1.69-million-file store. The first fence ran warm (221862 stems, ~3 s); the second hit the re-prefill's fresh stems cold and never returned -- PP0's last line for 25 minutes was the FENCE-NODES header, no abandon could run (PP0 is the sole timeout carrier), and the deadman killed the boot 07:15:37. Fix: hard 2.0 s budget; an expired scan reports SCAN CAPPED (counts declared a sample, denominator law) and bumps fence_scan_capped. The two-point consumer (_1063_probe_since_fence) reads only the 4096-capped _1063_AT_FENCE dict and loses no coverage it ever had. Trap recorded in a comment: 'now' is a PARAMETER of the fence function, not a module name -- a bare now() here would NameError into the outer except and silently delete the snapshot (sgl-project#872 silent-zero shape). time.monotonic() used explicitly. Desk-proven by execution, both arms: uncapped emits normally; a slow clock trips SCAN CAPPED.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
For debugging
With this shape
The diff is
With this shape
The diff is