Skip to content

fix: Fix returned prefill logits and add output str test - #1046

Merged
merrymercy merged 1 commit into
mainfrom
fix-prefill-logits
Aug 12, 2024
Merged

merrymercy merged 1 commit into
mainfrom
fix-prefill-logits

Conversation

@Ying1123

@Ying1123 Ying1123 commented Aug 12, 2024 •

Copy link
Copy Markdown
Contributor
  1. Fix the bug of missing logits softcapping when return full prefill logits.
  2. Add output string alignment into model correctness test.

@Ying1123
Ying1123 force-pushed the fix-prefill-logits branch from 45f7a05 to f35ca09 Compare August 12, 2024 05:17
@Ying1123
Ying1123 force-pushed the fix-prefill-logits branch from f35ca09 to cdfbc5c Compare August 12, 2024 05:50
@Ying1123 Ying1123 changed the title Fix returned prefill logits and add output str test fix: Fix returned prefill logits and add output str test Aug 12, 2024
@merrymercy
merrymercy enabled auto-merge (squash) August 12, 2024 06:11
@merrymercy
merrymercy merged commit 32f6144 into main Aug 12, 2024
@merrymercy
merrymercy deleted the fix-prefill-logits branch August 12, 2024 06:13
timethink pushed a commit to timethink/sglang that referenced this pull request Mar 9, 2025
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 31, 2026
…s; delete the load-back delivery chain

OPTION A, in the hard form. Two edits that only work together: the memory-axis
watchman moves onto the group availability floor FIRST, and only then does the
delivery chain come out.

sgl-project#1045 -- THE FLOOR IS PUBLISHED UNCONDITIONALLY ON A GROUP.
It used to stay None whenever availability happened to be equal that iteration,
and `unified_radix_cache`'s load-back fell into a RANK-LOCAL memory decision
when it was None, guarded only by the delivery signal `pp_load_back_told`. A
watchman on one axis was hanging off the delivery of another, and it stood down
exactly when nothing looked wrong.
PRICE VERIFIED BEFORE BUILDING, per the no-new-collective rule: at the live path
(scheduler.py:6396) `min_avail` and `max_avail` are BOTH harvested from the same
already-performed all-reduce (`t[0]`, `t[max_avail_at]`), so dropping the
comparison removes work rather than adding it. The single-rank path now
publishes the LOCAL value instead of None -- it IS the group min for a group of
one -- which is what lets None become a construction violation everywhere
downstream. Behaviourally identical on even pools: the published floor equals
the live local value, and the sgl-project#694 ledger keeps it tracking through the
iteration.
FIX-6 IS NOW ONE BRANCH. `floor is None` at the load-back raises, naming rid,
kv_tokens and local availability. A rank that cannot answer from a group number
must not answer from a local one -- that is how ranks stop agreeing, and the
21:52:25 wedge is the measured price.

sgl-project#1046 -- THE DELIVERY CHAIN IS DELETED.
Consumption is local: `_pp_load_back_extent` returns this rank's own stamp,
chosen at its own match by one shared expression over the same content.
Uniformity comes from identical DERIVATION, not from one rank shipping a value.
Deleted with it: `apply_pp_load_back_row` and both its call sites, PP0's
self-row parking and the one-lap delay, `pp_load_back_told` and all four
readers, the `sgl-project#1035` deferral ledger, the fact carriers,
`holders_with_unspent_extent`, `_spend_for_entry`, the spend transition, the
`sgl-project#1041` population counter and the `sgl-project#1044` ship/recv ledgers.

EVERY READER WAS DECIDED, NOT SWEPT ALONG -- the enumeration is the point,
because a reader falling silently into a rank-local branch is the class that
found the edge in the first place:
  unified_radix_cache.py:2713  group-fact signal -> moved to the floor (sgl-project#1045)
  schedule_policy.py:107       delivery mechanics -> local stamp
  schedule_policy.py:2305      delivery gate     -> deleted, no fact to await
  schedule_policy.py:2360      clamp gate        -> KEPT, re-pointed. Its hazard
                               (a GDN anchor adopted at `_applied` while the KV
                               is cut) is created by the load-back itself and is
                               independent of where the extent came from.

KEPT AND UNTOUCHED: the admission row, its `prefix_len`/`extend_len` columns and
the sgl-project#791 membership and ordering rules -- batch geometry, a different axis. The
`load_back_len` FIELD stays on the wire for width tolerance and always travels
None. sgl-project#1042's lifecycle contract stays, minus the spend transition that had no
consumer left.

CHECKS: check_1045_floor_watchman.py proves the guard CAN FIRE (a crash guard
that cannot crash is worse than the branch it replaced), that a small floor is
an ordinary refusal, that the rank-local fallback is DELETED rather than
bypassed, and that publication is unconditional and counted. check_1042 updated:
the row build must now LEAVE the field alone -- a build that still cleared it
would resurrect the eraser through the back door. check_1040 green. The 1041,
1043 and 1044 checks are DELETED with their subjects; the cell's zombie test
picks up their markers (ROW APPLIED, DEFERRED, EXTENT POPULATION, ROW SHIP) so
a stale build cannot pass silently. py_compile + import smoke on all five
modules; ruff clean.

NOT YET BOOTED -- this commit is desk-proven only.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 31, 2026
…a dict the cut swallowed

TWO EMITTERS MADE HONEST, both instances of the sampling trap this campaign has
now hit four times.

`sgl-project#988 LOADBACK` printed the FIRST event and then every 64th, with no suppression
count. Three lines across three ranks were therefore indistinguishable from "one
application per rank" and from "sixty-three per rank that were never printed" --
and a reproducibility run on boot 15 read exactly that ambiguity and could
conclude NOTHING, in either direction. This is the quantity the whole slice is
judged on, so it is now unconditional.

`sgl-project#1042 EXTENT LIFECYCLE` sampled at n<=3 then every 256th. Boot 15's acceptance
window contained ZERO such lines while the log held nine: the window fell
between samples, and a reader would have concluded "the contract was never
exercised". `set` and `retract` are rare and now print unconditionally; only
`hitless_noop` is still sampled, and it prints how many it skipped.

RUFF CAUGHT A REAL BUG, not a style nit: `_SECOND_CONTINUATION_REFUSALS` (F821)
was swallowed by the sgl-project#1046 cut, which deleted the old `_pp_load_back_extent`
body by slicing to the next `def` -- and that module-level line sat inside the
span. The NameError would have fired only on the sgl-project#959 second-continuation
refusal path, i.e. under specific traffic and probably not on the next boot.
Restored with a note. This is the argument for keeping ruff in the loop even
under speed mode: it is not a suite, it is one pass that can fail on exactly the
class a text-slice deletion produces.

check_1042 and check_1045 re-run green; import smoke confirms the restored
ledger.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 31, 2026
…erted with the architecture

BOOT 15 CRASHED AT 10:59:42, all three schedulers, and the cause is the sgl-project#1046
cut -- reported as mine, not as a pre-existing condition.

    sgl-project#968 LOAD-BACK EXTENT UNREACHABLE for rid=c9d14e69...: PP0 published an
    extent of 4618 token(s) and this rank's load-back yielded only 0

1448 times on one rid, refusing the forwarded schedule on every rank every pass,
until the ring wedged (sgl-project#789 PROXY READINESS TIMEOUT) and the schedulers died.

ROOT: sgl-project#1042's "a hitless match must never clear" was CORRECT while the extent
was consumed A LAP LATER by the delivery row -- the fact had to outlive the
match that made it. sgl-project#1046 moved consumption into the SAME match and I carried
the old rule across unchanged. The extent then outlived its own validity: rid
c9d14e69 kept extent=4618 through a readmit whose match read `host_hit=0`, and
the clamp demanded 4618 from a tree that could serve 0.

THE LESSON IS THE LIFECYCLE LAW ITSELF: a table is valid for ONE architecture.
It must be RE-DERIVED when the consumer moves, never inherited. I built the
table, then broke it by moving the consumer without re-deriving it.

TWO FIXES, both following from the same inversion:

1. A hitless match now CLEARS (`hitless_clear`, was `hitless_noop`). Under local
   consumption the extent is only valid for the match that produced it.

2. `_applied < _lb_extent` is no longer a PPScheduleRefused. That raise existed
   because `_lb_extent` was PP0's PUBLISHED number, so a rank that could not
   reach it would diverge from peers who could. Since sgl-project#1046 the extent is this
   rank's own, so a shortfall means only that this rank's stamp went stale --
   local staleness, not divergence. It now takes what its own tree can serve
   and says so (`sgl-project#1048 EXTENT STALE`), which is uniform because every rank
   re-derives from its own tree. Leaving it a group refusal is what turned a
   stale number into a ring wedge.

ALSO REPORTED: the boot-15 reproducibility run is CONFOUNDED, not negative. Two
cell runs landed on a server that was dying and then dead (deadman CRASH
10:59:47); the repro window's `sgl-project#988 genuine=0` measures the death, not the
mechanism. It is not evidence either way and is not counted as such.

check_1042 updated to pin the INVERTED rule and re-run green; ruff clean apart
from pre-existing E402.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 31, 2026
…fork turns on: told prefix vs local

Boot 27 died because two ranks admitted the same pass at different widths
(PP0 extend=4096, PP1 extend=2596; same slot, same fwd_ct, same rids; exactly
one divergent (slot,fwd_ct) pair in the whole boot). The fix form is a real
value fork and BOTH arms already killed a boot:

  per-rank refusal on an uncoverable told value -- boot 15 (sgl-project#1048): PP0
    published extent 4618, a rank yielded 0, 1448 refusals on one rid until
    the ring wedged and the schedulers died. sgl-project#1048's own fix deleted the raise.
  rank-local derivation -- boot 27, above.

So this commit changes NOTHING and only sizes the open quantity: how often does
PP0's published prefix exceed what a peer can cover, and is it concentrated at
flip boundaries?

The row already arrives and is already relayed verbatim (632ba36); sgl-project#1046
removed only its consumer. The receive point therefore stashes the told
prefixes into a dict that NOTHING but the census reads.

IT MUST NEVER WRITE _pp_admission_incoming_effective. That field is the armed
predicate of the clamp in _get_new_batch_prefill_raw; populating it would arm
PP0-published widths together with the refusal that killed boot 15. A separate
dict cannot. The matched check is an AST scan proving the new function writes
only _1058_* attributes, plus a census that the armed predicate still has
exactly its 4 None/{} writers -- py_compile is structurally blind to this
failure class and would have passed either way.

DENOMINATOR LAW BUILT IN, not remembered: five counters, and the ABSENT case
is one of them. A census counting only comparisons would report '0 overshoots'
identically whether PP0 never overshot or no told value ever arrived. The
census prints all five on a fixed cadence, and the per-event line is rate
limited with its occurrence number.

Desk: py_compile green, AST matched check passed, import smoke binds the
method, ruff clean on the added lines (the file's 105 findings are all
pre-existing and untouched). Behaviour: none -- the caller does not read a
return value and the clamp predicate is unchanged.

Boot 28 is a MEASUREMENT boot and may die at the known blocker; the harvest is
this distribution up to and including the death. Analysis:
DESIGN_968_PP0_AUTHORITATIVE_STATIC_HALF.md 5c-CORRECTION.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 31, 2026
…oject#1059 chain inert

Boot 29 ran with the gate on and its own census read evaluated=87 absent=87 --
no told prefix ever arrived. So sgl-project#631=0 and UniformWidthPromiseBroken=0 across 51
flips were NOT evidence the fix works; they were evidence it never executed.
That is exactly what the #1058b census was built to make impossible to misread,
and it earned its keep on its first real boot.

ROOT: the row is put on the wire only 'if any(load_back_len)', and sgl-project#1046 deleted
that field's last producer. The condition has been permanently false since, so
the row never reached the wire, observed_local never came home, PP0's MIN had no
input, and the apply had nothing to apply.

PRESENT-WIRED-NEVER-POPULATED, ONE LAYER FURTHER OUT THAN I FOUND IT. I verified
the codec round-trips in both directions (:315 write, :358 read), concluded
'site 3 needs no work', and wired five sites onto it -- without asking whether
anything ever SENDS. A carrier is not wired until something sends on it. Site 3
did need work, and it was never in the codec.

The gate is now the honest one: send when the decision has entries. The row is a
tuple of ints per request riding a message that is already sent, so the cost is
bytes on an existing frame, not a new frame; load_back_len staying None reads
back as None at the far end exactly as before.

SECOND HALF -- the gate now announces itself once per process. On boot 29 I
tried to derive 'the gate is on' from /proc/<rank>/environ, read ABSENT on all
three ranks, and was about to kill a healthy boot; the control refuted it
(SGLANG_UNEVEN_DCP reads ABSENT there too and works every boot -- /proc/environ
is the exec-time environment, spawn ranks rebuild os.environ at runtime). Wrong
instrument. A line from inside the process is the right one, and its absence now
means the gate is off rather than leaving it to inference.

Matched check for THIS edit's failure class (a gate that still consults the dead
field would look identical and stay inert): an AST/source assertion that the
send condition no longer reads load_back_len. py_compile green, 36 tests green,
ruff clean on the added lines.

Evidence: /spinning/evidence-665-f1/specimen_1059uw_boot29_prefill_loop/
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 31, 2026
…ompensation layer, and its bugs were not the finding

Coordinator order (2), after boots 30 and 31 killed the same mechanism twice at
the same site with the same numbers (`told prefix 12288 exceeds this rank's
pinned span 0`). Upstream-minimal law: a defect found in a compensation layer is
a DELETION candidate, never a repair order for that layer. Two repairs were
already spent on it -- sgl-project#1061's epoch gate and #1061b's retract clear -- and the
second only existed because the first was built on a clock whose tick was never
checked against the act it fenced.

DELETED: `pp_uniform_width.py` whole (`uniform_pass_geometry`,
`UniformWidthPromiseBroken`, `report_local_coverage`, `min_told`, `PassGeometry`,
`epoch_admits_row`, `set_epoch_source`, `current_epoch`);
`Req.apply_uniform_pass_geometry_1059` and its call site; the
`_1059_told_prefix/_extend/_epoch` stamps; `PPAdmissionEntry.decided_epoch` and
its wire column; the publication-time epoch stamp; the epoch-source
registration; the told clearing at the retract chokepoint; the
`SGLANG_PP_UNIFORM_WIDTH` gate; three test files.

WHAT DELIBERATELY STAYED, and each for a named reason:

* THE EVICTION FLOOR in `pp_stamp_observed_coverage`. `report_local_coverage`
  was `max(0, int(x))`, but the same line ALSO raises `cache_protected_len`,
  which `mem_cache/common.py:82` honours as an eviction floor. Deleting a
  promise must not silently lower a protection, so the arithmetic is inlined and
  the floor is byte-identical. Only the promise semantics die.
* THE CARRIER COUNTER, renamed to what survives. The #1060b ledger counted the
  told row at the apply site; that apply is gone, but the question underneath it
  is not. Boot 29 shipped a carrier whose send gate had been dead since sgl-project#1046
  and reported `sgl-project#631=0` while nothing executed -- it cost a window and a census
  to notice. `sgl-project#1064 CARRIER CENSUS` counts `rows_received` on every rank, so
  that regression can never be silent again. PP0 never receives, so only a
  downstream zero is a finding, and the line says so.
* `sgl-project#1060`/`sgl-project#1063` untouched: they measure, they never gated anything.

WHAT THIS RETURNS TO: the pre-sgl-project#1059 behaviour, in which every rank derives its
geometry from its own match. That is not a guess about safety -- boot 29 ran it
for its ENTIRE life (`evaluated=87 absent=87`, the row never arrived) across 51
flips with no `sgl-project#631`, no divergence death and no crash. The two boots that DID
die at this seam are the two that ran the promise layer.

c1 (downstream takes the geometry as a VERDICT) is NOT in this commit, and the
reason is a desk finding that inverts its safety argument: `truncate_prefix_to`
is MIN, NEVER ASSIGN (schedule_batch.py, sgl-project#930/sgl-project#958) -- so "verbatim" cannot be
expressed through the one helper both admission sites use, and a rank holding
less than it was told silently keeps its own value, which is the divergence the
verbatim form exists to remove. Reported to the coordinator with the file:line
rather than worked around.

38/38 green across sgl-project#791/sgl-project#1060/sgl-project#1063/sgl-project#677, codec round-trip verified after the
column removal, import smoke of every touched module, ruff clean, py_compile
green.

Order: coordinator (2). Evidence: boots 30/31, specimens
specimen_1060cens_boot30_uniformwidth_crash/ and
specimen_1061b_boot32_int8_triton_wedge/
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants