Repository navigation
fix: Fix returned prefill logits and add output str test - #1046
Merged
Merged
Conversation
Ying1123
force-pushed
the
fix-prefill-logits
branch
from
August 12, 2024 05:17
45f7a05 to
f35ca09
Compare
Ying1123
force-pushed
the
fix-prefill-logits
branch
from
August 12, 2024 05:50
f35ca09 to
cdfbc5c
Compare
merrymercy
approved these changes
Aug 12, 2024
merrymercy
enabled auto-merge (squash)
August 12, 2024 06:11
timethink
pushed a commit
to timethink/sglang
that referenced
this pull request
Mar 9, 2025
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 31, 2026
…s; delete the load-back delivery chain OPTION A, in the hard form. Two edits that only work together: the memory-axis watchman moves onto the group availability floor FIRST, and only then does the delivery chain come out. sgl-project#1045 -- THE FLOOR IS PUBLISHED UNCONDITIONALLY ON A GROUP. It used to stay None whenever availability happened to be equal that iteration, and `unified_radix_cache`'s load-back fell into a RANK-LOCAL memory decision when it was None, guarded only by the delivery signal `pp_load_back_told`. A watchman on one axis was hanging off the delivery of another, and it stood down exactly when nothing looked wrong. PRICE VERIFIED BEFORE BUILDING, per the no-new-collective rule: at the live path (scheduler.py:6396) `min_avail` and `max_avail` are BOTH harvested from the same already-performed all-reduce (`t[0]`, `t[max_avail_at]`), so dropping the comparison removes work rather than adding it. The single-rank path now publishes the LOCAL value instead of None -- it IS the group min for a group of one -- which is what lets None become a construction violation everywhere downstream. Behaviourally identical on even pools: the published floor equals the live local value, and the sgl-project#694 ledger keeps it tracking through the iteration. FIX-6 IS NOW ONE BRANCH. `floor is None` at the load-back raises, naming rid, kv_tokens and local availability. A rank that cannot answer from a group number must not answer from a local one -- that is how ranks stop agreeing, and the 21:52:25 wedge is the measured price. sgl-project#1046 -- THE DELIVERY CHAIN IS DELETED. Consumption is local: `_pp_load_back_extent` returns this rank's own stamp, chosen at its own match by one shared expression over the same content. Uniformity comes from identical DERIVATION, not from one rank shipping a value. Deleted with it: `apply_pp_load_back_row` and both its call sites, PP0's self-row parking and the one-lap delay, `pp_load_back_told` and all four readers, the `sgl-project#1035` deferral ledger, the fact carriers, `holders_with_unspent_extent`, `_spend_for_entry`, the spend transition, the `sgl-project#1041` population counter and the `sgl-project#1044` ship/recv ledgers. EVERY READER WAS DECIDED, NOT SWEPT ALONG -- the enumeration is the point, because a reader falling silently into a rank-local branch is the class that found the edge in the first place: unified_radix_cache.py:2713 group-fact signal -> moved to the floor (sgl-project#1045) schedule_policy.py:107 delivery mechanics -> local stamp schedule_policy.py:2305 delivery gate -> deleted, no fact to await schedule_policy.py:2360 clamp gate -> KEPT, re-pointed. Its hazard (a GDN anchor adopted at `_applied` while the KV is cut) is created by the load-back itself and is independent of where the extent came from. KEPT AND UNTOUCHED: the admission row, its `prefix_len`/`extend_len` columns and the sgl-project#791 membership and ordering rules -- batch geometry, a different axis. The `load_back_len` FIELD stays on the wire for width tolerance and always travels None. sgl-project#1042's lifecycle contract stays, minus the spend transition that had no consumer left. CHECKS: check_1045_floor_watchman.py proves the guard CAN FIRE (a crash guard that cannot crash is worse than the branch it replaced), that a small floor is an ordinary refusal, that the rank-local fallback is DELETED rather than bypassed, and that publication is unconditional and counted. check_1042 updated: the row build must now LEAVE the field alone -- a build that still cleared it would resurrect the eraser through the back door. check_1040 green. The 1041, 1043 and 1044 checks are DELETED with their subjects; the cell's zombie test picks up their markers (ROW APPLIED, DEFERRED, EXTENT POPULATION, ROW SHIP) so a stale build cannot pass silently. py_compile + import smoke on all five modules; ruff clean. NOT YET BOOTED -- this commit is desk-proven only.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 31, 2026
…a dict the cut swallowed TWO EMITTERS MADE HONEST, both instances of the sampling trap this campaign has now hit four times. `sgl-project#988 LOADBACK` printed the FIRST event and then every 64th, with no suppression count. Three lines across three ranks were therefore indistinguishable from "one application per rank" and from "sixty-three per rank that were never printed" -- and a reproducibility run on boot 15 read exactly that ambiguity and could conclude NOTHING, in either direction. This is the quantity the whole slice is judged on, so it is now unconditional. `sgl-project#1042 EXTENT LIFECYCLE` sampled at n<=3 then every 256th. Boot 15's acceptance window contained ZERO such lines while the log held nine: the window fell between samples, and a reader would have concluded "the contract was never exercised". `set` and `retract` are rare and now print unconditionally; only `hitless_noop` is still sampled, and it prints how many it skipped. RUFF CAUGHT A REAL BUG, not a style nit: `_SECOND_CONTINUATION_REFUSALS` (F821) was swallowed by the sgl-project#1046 cut, which deleted the old `_pp_load_back_extent` body by slicing to the next `def` -- and that module-level line sat inside the span. The NameError would have fired only on the sgl-project#959 second-continuation refusal path, i.e. under specific traffic and probably not on the next boot. Restored with a note. This is the argument for keeping ruff in the loop even under speed mode: it is not a suite, it is one pass that can fail on exactly the class a text-slice deletion produces. check_1042 and check_1045 re-run green; import smoke confirms the restored ledger.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 31, 2026
…erted with the architecture BOOT 15 CRASHED AT 10:59:42, all three schedulers, and the cause is the sgl-project#1046 cut -- reported as mine, not as a pre-existing condition. sgl-project#968 LOAD-BACK EXTENT UNREACHABLE for rid=c9d14e69...: PP0 published an extent of 4618 token(s) and this rank's load-back yielded only 0 1448 times on one rid, refusing the forwarded schedule on every rank every pass, until the ring wedged (sgl-project#789 PROXY READINESS TIMEOUT) and the schedulers died. ROOT: sgl-project#1042's "a hitless match must never clear" was CORRECT while the extent was consumed A LAP LATER by the delivery row -- the fact had to outlive the match that made it. sgl-project#1046 moved consumption into the SAME match and I carried the old rule across unchanged. The extent then outlived its own validity: rid c9d14e69 kept extent=4618 through a readmit whose match read `host_hit=0`, and the clamp demanded 4618 from a tree that could serve 0. THE LESSON IS THE LIFECYCLE LAW ITSELF: a table is valid for ONE architecture. It must be RE-DERIVED when the consumer moves, never inherited. I built the table, then broke it by moving the consumer without re-deriving it. TWO FIXES, both following from the same inversion: 1. A hitless match now CLEARS (`hitless_clear`, was `hitless_noop`). Under local consumption the extent is only valid for the match that produced it. 2. `_applied < _lb_extent` is no longer a PPScheduleRefused. That raise existed because `_lb_extent` was PP0's PUBLISHED number, so a rank that could not reach it would diverge from peers who could. Since sgl-project#1046 the extent is this rank's own, so a shortfall means only that this rank's stamp went stale -- local staleness, not divergence. It now takes what its own tree can serve and says so (`sgl-project#1048 EXTENT STALE`), which is uniform because every rank re-derives from its own tree. Leaving it a group refusal is what turned a stale number into a ring wedge. ALSO REPORTED: the boot-15 reproducibility run is CONFOUNDED, not negative. Two cell runs landed on a server that was dying and then dead (deadman CRASH 10:59:47); the repro window's `sgl-project#988 genuine=0` measures the death, not the mechanism. It is not evidence either way and is not counted as such. check_1042 updated to pin the INVERTED rule and re-run green; ruff clean apart from pre-existing E402.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 31, 2026
…fork turns on: told prefix vs local Boot 27 died because two ranks admitted the same pass at different widths (PP0 extend=4096, PP1 extend=2596; same slot, same fwd_ct, same rids; exactly one divergent (slot,fwd_ct) pair in the whole boot). The fix form is a real value fork and BOTH arms already killed a boot: per-rank refusal on an uncoverable told value -- boot 15 (sgl-project#1048): PP0 published extent 4618, a rank yielded 0, 1448 refusals on one rid until the ring wedged and the schedulers died. sgl-project#1048's own fix deleted the raise. rank-local derivation -- boot 27, above. So this commit changes NOTHING and only sizes the open quantity: how often does PP0's published prefix exceed what a peer can cover, and is it concentrated at flip boundaries? The row already arrives and is already relayed verbatim (632ba36); sgl-project#1046 removed only its consumer. The receive point therefore stashes the told prefixes into a dict that NOTHING but the census reads. IT MUST NEVER WRITE _pp_admission_incoming_effective. That field is the armed predicate of the clamp in _get_new_batch_prefill_raw; populating it would arm PP0-published widths together with the refusal that killed boot 15. A separate dict cannot. The matched check is an AST scan proving the new function writes only _1058_* attributes, plus a census that the armed predicate still has exactly its 4 None/{} writers -- py_compile is structurally blind to this failure class and would have passed either way. DENOMINATOR LAW BUILT IN, not remembered: five counters, and the ABSENT case is one of them. A census counting only comparisons would report '0 overshoots' identically whether PP0 never overshot or no told value ever arrived. The census prints all five on a fixed cadence, and the per-event line is rate limited with its occurrence number. Desk: py_compile green, AST matched check passed, import smoke binds the method, ruff clean on the added lines (the file's 105 findings are all pre-existing and untouched). Behaviour: none -- the caller does not read a return value and the clamp predicate is unchanged. Boot 28 is a MEASUREMENT boot and may die at the known blocker; the harvest is this distribution up to and including the death. Analysis: DESIGN_968_PP0_AUTHORITATIVE_STATIC_HALF.md 5c-CORRECTION.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 31, 2026
…oject#1059 chain inert Boot 29 ran with the gate on and its own census read evaluated=87 absent=87 -- no told prefix ever arrived. So sgl-project#631=0 and UniformWidthPromiseBroken=0 across 51 flips were NOT evidence the fix works; they were evidence it never executed. That is exactly what the #1058b census was built to make impossible to misread, and it earned its keep on its first real boot. ROOT: the row is put on the wire only 'if any(load_back_len)', and sgl-project#1046 deleted that field's last producer. The condition has been permanently false since, so the row never reached the wire, observed_local never came home, PP0's MIN had no input, and the apply had nothing to apply. PRESENT-WIRED-NEVER-POPULATED, ONE LAYER FURTHER OUT THAN I FOUND IT. I verified the codec round-trips in both directions (:315 write, :358 read), concluded 'site 3 needs no work', and wired five sites onto it -- without asking whether anything ever SENDS. A carrier is not wired until something sends on it. Site 3 did need work, and it was never in the codec. The gate is now the honest one: send when the decision has entries. The row is a tuple of ints per request riding a message that is already sent, so the cost is bytes on an existing frame, not a new frame; load_back_len staying None reads back as None at the far end exactly as before. SECOND HALF -- the gate now announces itself once per process. On boot 29 I tried to derive 'the gate is on' from /proc/<rank>/environ, read ABSENT on all three ranks, and was about to kill a healthy boot; the control refuted it (SGLANG_UNEVEN_DCP reads ABSENT there too and works every boot -- /proc/environ is the exec-time environment, spawn ranks rebuild os.environ at runtime). Wrong instrument. A line from inside the process is the right one, and its absence now means the gate is off rather than leaving it to inference. Matched check for THIS edit's failure class (a gate that still consults the dead field would look identical and stay inert): an AST/source assertion that the send condition no longer reads load_back_len. py_compile green, 36 tests green, ruff clean on the added lines. Evidence: /spinning/evidence-665-f1/specimen_1059uw_boot29_prefill_loop/
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 31, 2026
…ompensation layer, and its bugs were not the finding Coordinator order (2), after boots 30 and 31 killed the same mechanism twice at the same site with the same numbers (`told prefix 12288 exceeds this rank's pinned span 0`). Upstream-minimal law: a defect found in a compensation layer is a DELETION candidate, never a repair order for that layer. Two repairs were already spent on it -- sgl-project#1061's epoch gate and #1061b's retract clear -- and the second only existed because the first was built on a clock whose tick was never checked against the act it fenced. DELETED: `pp_uniform_width.py` whole (`uniform_pass_geometry`, `UniformWidthPromiseBroken`, `report_local_coverage`, `min_told`, `PassGeometry`, `epoch_admits_row`, `set_epoch_source`, `current_epoch`); `Req.apply_uniform_pass_geometry_1059` and its call site; the `_1059_told_prefix/_extend/_epoch` stamps; `PPAdmissionEntry.decided_epoch` and its wire column; the publication-time epoch stamp; the epoch-source registration; the told clearing at the retract chokepoint; the `SGLANG_PP_UNIFORM_WIDTH` gate; three test files. WHAT DELIBERATELY STAYED, and each for a named reason: * THE EVICTION FLOOR in `pp_stamp_observed_coverage`. `report_local_coverage` was `max(0, int(x))`, but the same line ALSO raises `cache_protected_len`, which `mem_cache/common.py:82` honours as an eviction floor. Deleting a promise must not silently lower a protection, so the arithmetic is inlined and the floor is byte-identical. Only the promise semantics die. * THE CARRIER COUNTER, renamed to what survives. The #1060b ledger counted the told row at the apply site; that apply is gone, but the question underneath it is not. Boot 29 shipped a carrier whose send gate had been dead since sgl-project#1046 and reported `sgl-project#631=0` while nothing executed -- it cost a window and a census to notice. `sgl-project#1064 CARRIER CENSUS` counts `rows_received` on every rank, so that regression can never be silent again. PP0 never receives, so only a downstream zero is a finding, and the line says so. * `sgl-project#1060`/`sgl-project#1063` untouched: they measure, they never gated anything. WHAT THIS RETURNS TO: the pre-sgl-project#1059 behaviour, in which every rank derives its geometry from its own match. That is not a guess about safety -- boot 29 ran it for its ENTIRE life (`evaluated=87 absent=87`, the row never arrived) across 51 flips with no `sgl-project#631`, no divergence death and no crash. The two boots that DID die at this seam are the two that ran the promise layer. c1 (downstream takes the geometry as a VERDICT) is NOT in this commit, and the reason is a desk finding that inverts its safety argument: `truncate_prefix_to` is MIN, NEVER ASSIGN (schedule_batch.py, sgl-project#930/sgl-project#958) -- so "verbatim" cannot be expressed through the one helper both admission sites use, and a rank holding less than it was told silently keeps its own value, which is the divergence the verbatim form exists to remove. Reported to the coordinator with the file:line rather than worked around. 38/38 green across sgl-project#791/sgl-project#1060/sgl-project#1063/sgl-project#677, codec round-trip verified after the column removal, import smoke of every touched module, ruff clean, py_compile green. Order: coordinator (2). Evidence: boots 30/31, specimens specimen_1060cens_boot30_uniformwidth_crash/ and specimen_1061b_boot32_int8_triton_wedge/
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.