Skip to content

minor: some potential bugs - #1044

Merged
merrymercy merged 3 commits into
mainfrom
fix-accuracy
Aug 12, 2024
Merged

merrymercy merged 3 commits into
mainfrom
fix-accuracy

Conversation

@hnyls2002

Copy link
Copy Markdown
Collaborator

Thank you for your contribution, we really appreciate it. The following instructions will help improve your pull request and make it easier to receive feedback. If there are any items you don't understand, don't worry. Just submit the pull request and ask the maintainers for help.

Motivation

This PR solves two potential bugs.

Checklist

  1. Ensure pre-commit pre-commit run --all-files or other linting tools are used to fix potential lint issues.
  2. Confirm that modifications are covered by complete unit tests. If not, please add more unit tests for correctness.
  3. Modify documentation as needed, such as docstrings or example tutorials.

@hnyls2002
hnyls2002 requested a review from merrymercy August 12, 2024 04:43
@merrymercy
merrymercy enabled auto-merge (squash) August 12, 2024 05:00
@merrymercy
merrymercy merged commit fb7421d into main Aug 12, 2024
@merrymercy
merrymercy deleted the fix-accuracy branch August 12, 2024 05:35
timethink pushed a commit to timethink/sglang that referenced this pull request Mar 9, 2025
hhhh1252023 pushed a commit to hhhh1252023/sglang_public that referenced this pull request Jul 28, 2026
test_npu_serving_transcription/test_npu_encoder_dp/test_npu_token_id_retokenize_e2e/test_npu_vision_chunked_prefill
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 31, 2026
… of the apply swept

Boot 14 published a row and no rank stamped one, and the receiver's four ways to
decline were one indistinguishable silence. Instrumented as a sweep of the whole
function rather than the one branch under suspicion -- this campaign has paid
three times for instrumenting a branch and leaving its neighbour mute.

SENDER: `sgl-project#1044 ROW SHIP` counts posted_with_row / posted_no_row /
posted_no_decision / skipped_with_row / skipped_no_row / entries_shipped. The
SKIPPED passes are counted too: a frame that is never posted is exactly the case
a posted-only counter cannot see.

RECEIVER: `sgl-project#1044 ROW RECV` names every exit of `apply_pp_load_back_row` --
no_entries, entry_no_extent, entry_no_rid, no_wanted, unknown_rid, stamped.
`unknown_rid` is candidate (D), the branch that could absorb the entire chain
without a line of evidence.

Together they partition in ONE boot: entries shipped with wanted 0 everywhere is
(C) delivery; wanted with unknown_rid is (D) lap; wanted with neither is a THIRD
branch, which is why the whole function is swept.

Emission is unconditional whenever a row is involved; where it is sampled, the
suppression count is printed -- a zero from a rate-limited emitter is not a
zero, which is this campaign's fourth denominator instance and now a standing
rule.

A DEAD COUNTER WAS REMOVED RATHER THAN SHIPPED: `raised` had no except path to
fire it, so it would have read 0 forever and been indistinguishable from
evidence.

CHECK: devtools/check_1044_rowtrip.py drives EVERY named counter and asserts it
moved; "no dead counters" is its own assertion, as is the denominator (`calls`).
Named failure class: a counter that cannot move reproduces the silence this edit
removes while appearing to fix it. 15/15 PASS; 1040/1041/1042/1043 re-run green;
ruff clean; import smoke.

NOT IN THIS COMMIT: the whole-cut to local consumption. Its edge check found a
prohibiting edge and it is reported rather than worked around -- see handover
section 16.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 31, 2026
…s; delete the load-back delivery chain

OPTION A, in the hard form. Two edits that only work together: the memory-axis
watchman moves onto the group availability floor FIRST, and only then does the
delivery chain come out.

sgl-project#1045 -- THE FLOOR IS PUBLISHED UNCONDITIONALLY ON A GROUP.
It used to stay None whenever availability happened to be equal that iteration,
and `unified_radix_cache`'s load-back fell into a RANK-LOCAL memory decision
when it was None, guarded only by the delivery signal `pp_load_back_told`. A
watchman on one axis was hanging off the delivery of another, and it stood down
exactly when nothing looked wrong.
PRICE VERIFIED BEFORE BUILDING, per the no-new-collective rule: at the live path
(scheduler.py:6396) `min_avail` and `max_avail` are BOTH harvested from the same
already-performed all-reduce (`t[0]`, `t[max_avail_at]`), so dropping the
comparison removes work rather than adding it. The single-rank path now
publishes the LOCAL value instead of None -- it IS the group min for a group of
one -- which is what lets None become a construction violation everywhere
downstream. Behaviourally identical on even pools: the published floor equals
the live local value, and the sgl-project#694 ledger keeps it tracking through the
iteration.
FIX-6 IS NOW ONE BRANCH. `floor is None` at the load-back raises, naming rid,
kv_tokens and local availability. A rank that cannot answer from a group number
must not answer from a local one -- that is how ranks stop agreeing, and the
21:52:25 wedge is the measured price.

sgl-project#1046 -- THE DELIVERY CHAIN IS DELETED.
Consumption is local: `_pp_load_back_extent` returns this rank's own stamp,
chosen at its own match by one shared expression over the same content.
Uniformity comes from identical DERIVATION, not from one rank shipping a value.
Deleted with it: `apply_pp_load_back_row` and both its call sites, PP0's
self-row parking and the one-lap delay, `pp_load_back_told` and all four
readers, the `sgl-project#1035` deferral ledger, the fact carriers,
`holders_with_unspent_extent`, `_spend_for_entry`, the spend transition, the
`sgl-project#1041` population counter and the `sgl-project#1044` ship/recv ledgers.

EVERY READER WAS DECIDED, NOT SWEPT ALONG -- the enumeration is the point,
because a reader falling silently into a rank-local branch is the class that
found the edge in the first place:
  unified_radix_cache.py:2713  group-fact signal -> moved to the floor (sgl-project#1045)
  schedule_policy.py:107       delivery mechanics -> local stamp
  schedule_policy.py:2305      delivery gate     -> deleted, no fact to await
  schedule_policy.py:2360      clamp gate        -> KEPT, re-pointed. Its hazard
                               (a GDN anchor adopted at `_applied` while the KV
                               is cut) is created by the load-back itself and is
                               independent of where the extent came from.

KEPT AND UNTOUCHED: the admission row, its `prefix_len`/`extend_len` columns and
the sgl-project#791 membership and ordering rules -- batch geometry, a different axis. The
`load_back_len` FIELD stays on the wire for width tolerance and always travels
None. sgl-project#1042's lifecycle contract stays, minus the spend transition that had no
consumer left.

CHECKS: check_1045_floor_watchman.py proves the guard CAN FIRE (a crash guard
that cannot crash is worse than the branch it replaced), that a small floor is
an ordinary refusal, that the rank-local fallback is DELETED rather than
bypassed, and that publication is unconditional and counted. check_1042 updated:
the row build must now LEAVE the field alone -- a build that still cleared it
would resurrect the eraser through the back door. check_1040 green. The 1041,
1043 and 1044 checks are DELETED with their subjects; the cell's zombie test
picks up their markers (ROW APPLIED, DEFERRED, EXTENT POPULATION, ROW SHIP) so
a stale build cannot pass silently. py_compile + import smoke on all five
modules; ruff clean.

NOT YET BOOTED -- this commit is desk-proven only.
efschu pushed a commit to efschu/htsglang that referenced this pull request Sep 3, 2026
…mainders; quiescence is an arm precondition; followers execute every launched pass under the arm or STOP

weg1b4 08:07:50Z: PP0 launched fwd_ct=81 slot=1 for rid b64dc1cb (extend
1912), posted its sgl-project#631 PROXY-SEND t6, and ONE LINE LATER armed pp_to_tp on
a DRAINED verdict reading "0 tok remaining" -- the sgl-project#1153 starvation shape
with no designed STOP.

D1 THE DRAINED PREMISE COUNTS EVERYTHING STILL TO PREFILL.
Scheduler._pending_prefill_tokens now carries an in-flight microbatch term:
for every request the PP ring launched but has not returned, it prices
len(origin_input_ids) - extend_range.start (NEVER - end: end is advanced at
PREPARE time, so pricing from end reads 0 for exactly the request that is
still running). Deduped against _queued_ids so a request cannot be billed
twice; the chunked and resident terms skip _inflight_ids for the same
reason. _pending_prefill_terms records the breakdown and its producer, and
all four pp-exit arms print it: "[sgl-project#1173 pending terms: queue=.. inflight=..
chunked=.. arriving=.. resident=.. total=.. (producer
Scheduler._pending_prefill_tokens, include_health=..)]". On the weg1b4
numbers the ring reads 1912 for b64dc1cb where it read 0 before.

D2a QUIESCENCE IS A PRECONDITION OF THE ARM, NOT A HOLD AFTER IT.
PhaseFlipRuntime.arm() takes launched_passes_fn (build_launched_passes_fn
reads the same _pp_launched_pending the sgl-project#1020 void guard reads, so arm and
guard cannot disagree about what is outstanding). With any slot
outstanding the arm is DEFERRED, not taken-and-held:
"sgl-project#1173 ARM DEFERRED: launched passes outstanding slots=[..] fwd_ct=..
direction=.. deferrals=..". The check sits after the rank!=0 early return
(PP0-only by construction, no new rank-local verdict, no new sync point)
and before the side-effecting _prearm_floor_relief. An unreadable probe
never defers. The mbs quiescence hold is KEPT, not deleted: it stays
reachable on followers, which take the arm as an order and never run this
precondition, and for a pass launched around the order's arrival -- its
reason text now says so.

D2b FOLLOWERS EXECUTE WHAT PP0 LAUNCHED, ARM OR NOT.
_pp_flip_hold_slot released the ring only when every slot was None, so a
proxy frame for a launched pass sitting in the typed inbox parked the
follower on an order it could not obey while the launcher waited for a
frame that never came. The hold now releases while such a frame is stashed
(_pp_flip_stashed_frame_forces_advance reads the typed inbox ONLY -- never
the "provably in flight" counter half of _pp_row_any_proxy_signal, which a
transit frame would false-positive during the armed spin) so the loop walks
the ring to the slot the frame names. A frame that never leaves is a group
STOP through the launcher, never a park and never a silent spin:
RuntimeError("sgl-project#1173 LAUNCHED PASS UNEXECUTED UNDER ARM STOP rank=%d
slot=%d fwd_ct=%d rid=%s arm_epoch=%d reason=%s") after 2*ring+2 armed
visits.

Small item: the phase_policy hold that says the premise was verified on the
STORE WITNESS now prints the witness STATE census
(pending|hit|bounded|cold|unprobed) via phase_purity.store_witness_census.

EVIDENCE
new test/registered/unit/managers/test_arm_quiescence_1173.py: 14 passed on
this tree, 11 failed / 1 passed on parent f58a71b (scratch worktree,
same file copied in) = red-first.
Mutants, each applied to this tree and reverted: price the in-flight pass
from extend_range.end -> 2 red (D1); neutralise the ARM DEFERRED block ->
1 red (D2a); restore the unconditional park in _pp_flip_hold_slot -> 3 red
(D2b). All three died.
test_pp_flip_slot_hold_631.py: the drained-ring pin is NARROWED, not
withdrawn (its docstring says why) and gains the release companion
test_does_not_hold_while_a_launched_passs_frame_is_stashed.
Touched test files, fixed vs parent: 54 failed / 152 passed vs 54 failed /
151 passed, comm diff EMPTY both directions.
test/registered/unit/managers bounded once each: fixed 247 failed / 4684
passed / 5 errors, parent 246 failed / 4670 passed / 5 errors; comm names
exactly one extra failure, test_load_snapshot_backends.py::TestZmqRoundTrip
::test_read_returns_latest, which passes SOLO on BOTH trees = the sgl-project#899 box
-timing class, not this diff. The 5 collection errors are the pre-existing
sgl-project#1044 import-broken *_631 modules on both sides.
ruff on the touched files: 104 errors before, 104 after.
efschu pushed a commit to efschu/htsglang that referenced this pull request Sep 4, 2026
…it in a named stop

BOOT weg1b7 (03:49:20 -> 03:50:24Z) died in a circular PP ring wait: PP0
parked in recv_object[src=2], PP1 tore at the 60 s chain-stall bound, both
downstream ranks ran their two ring-cut drain turns and the group died.

ROOT (built against, and it is NOT the ring cut). The sgl-project#631 row probe peeks
the head proxy frame's admission row and, when the row names a rid this rank
cannot locate, answers False and leaves the frame in the inbox. That
protection is right (boot 631row14: a frame consumed before its request hop
plans nothing and the ring dies upstream-waiting) and is untouched here.
What it lacked is a BOUND, and the defer is not free: PP0 has ALREADY
launched the pass the frame belongs to and is ALREADY parked on its output,
so an unbounded defer closes the ring BY CONSTRUCTION.

The defer's own premise -- "an unlocatable rid PROVES a chain send is in
flight, bounded by rendezvous+transfer" -- is false for both populations
measured in weg1b7:

  * a rid PP0 RETRACTED after sealing the row. PP0 shipped sgl-project#631 PROXY-SEND
    t40 naming 772368da and c25108f3, then the discretionary #888b parked-
    carrier relief retracted c25108f3 in the NEXT scheduling iteration
    (#969AD RETRACT site=_retract_decode_and_requeue fwd_ct=157). A sealed
    row cannot be amended, so _missing can never empty.
  * a re-admitted resident (sgl-project#968 READMIT CACHED, last_queued_as=cutover-
    requeue) requeued rank-locally by _add_request_to_queue -- no chain hop
    for it exists or ever will.

Ranks disagreeing about a rid is RAENGE-NIE-UNEINS, and the sanctioned
answer there is a bounded, NAMED stop -- never a silent unbounded wait.
sgl-project#1071's occupant horizon is the precedent this follows, down to "0 disables".

CHANGE
  python/sglang/srt/managers/pp_row_defer_horizon.py  (new, 300 lines)
      RowDeferHorizon: per-slot clock over the (frame stamp, missing-rid set)
      identity. First sighting defers. A CHANGED missing set is a hop landing
      and restarts the clock -- the boot 631row15 world (43k hot defers on a
      TURNING ring) therefore never reaches the bound. A DIFFERENT frame also
      restarts it, so a slot whose head alternates cannot inherit a
      predecessor's age. Only the same frame with the same missing set, held
      past SGLANG_PP_ROW_DEFER_HORIZON_S (default 20 s), answers defer=False
      with a message naming slot, wait, occurrences, rids, BOTH measured
      shapes, what is deliberately not repaired, and the remedy knob.
      PURE: no logger, no wire, no scheduler attribute -- the caller owns the
      raise, as parked_carrier_relief and min_free_slots_delayer are built.
      20 s is chosen against three measured neighbours: orders of magnitude
      above the healthy case (weg1b7's three defers of one frame land in the
      same log second), BELOW the 60 s chain stall bound so THIS names the
      contradiction before the ring cut misreports it as "closed ring on the
      request chain", and BELOW the 90 s sgl-project#1071 horizon.

  scheduler_pp_mixin.py:9184-9243   the defer consults the horizon and raises
      PpRowDeferHorizonLapsed past the bound. It does NOT consume the frame
      (that revives boot 631row14) and does NOT emit a void (that revives the
      sgl-project#801 reverse corpse). The DEFER log line now carries held=%.1fs.
  scheduler_pp_mixin.py:9266-9271   a satisfied or unreadable row clears the
      slot's clock, so a resolved wait cannot age a later one into the bound.
  scheduler_pp_mixin.py:8912-8941   _pp_row_chain_pending's docstring
      corrected: its _pp_row_chain_owed arm returns True on an INFERENCE, not
      on proof of availability, and the receive it opens is bounded by this
      horizon rather than by transfer time. Do not read that True as uniform
      evidence of a posted message.

EVIDENCE
  RED FIRST on parent 2615514 (scratch worktree /spinning/wt-1180-red,
  only the new test file copied in): 12 failed / 2 passed. Two of the reds
  are BEHAVIOURAL, not import errors --
    test_the_defer_is_not_unbounded: "the row probe deferred the same
      unsatisfiable row 4096 times over an hour of its own clock without ever
      stopping" (imports nothing new, on purpose);
    test_a_satisfied_row_clears_the_clock_its_own_defer_armed: "the defer
      must have armed a clock".
  FIXED: 14 passed / 0 failed.

  MUTANTS 6/6 died, each patch verified applied:
    M1 never raise (fix inert)                                    DIED
    M2 bound never lapses                                         DIED
    M3 a changed missing set does not restart the clock           DIED
    M4 satisfied row leaves its clock behind                      DIED
    M5 no frame token in the identity                             DIED
    M6 DANGER DIRECTION: stop on the FIRST defer                  DIED
  M4 survived the first round -- the locatable-row test was vacuous (no
  horizon object existed to inspect). The recovery test above was added to
  make the clear observable, and M4 then died.

  ruff parity: 3 pre-existing errors on the parent, the same 3 after
  (F841 carries_flip_arm, E741 l x2). Import smoke hermetic (CVD="").

  SUITE ONCE, both trees, same command, sequentially (crowding-free box,
  load ~2.9), test/registered/unit/managers with the five pre-existing
  collection-error modules ignored (the same five on both trees: the sgl-project#1044
  import-broken *_631 quartet plus test_quiescence_no_carry_858):
    parent 2615514 : 240 failed / 4767 passed / 2 skipped / 638 subtests
    fixed             : 240 failed / 4781 passed / 2 skipped / 638 subtests
  Delta = +14 passed, exactly the new file; the FAILED NAME SETS are byte
  identical (diff empty, 236 names each side). Counting probe declared
  honestly: 236 extracted names vs the summary's 240 on BOTH trees -- a
  symmetric extraction gap of 4 that is not investigated here and cannot
  affect the A/B, since both sides carry the same 236 names and the same
  summary.

RESIDUALS, deliberately NOT fixed here
  * The stop is RANK-LOCAL, exactly like sgl-project#1071's: the deferring rank raises,
    the others then die on their own collectives or the deadman. A
    group-uniform stop would need a collective on the wedged path, which is
    what the ring cut already refuses.
  * #888b's discretionary yield is untouched. Making it protect published
    rids is refused: the entry decision and the victim count must be
    rank-uniform (the sgl-project#583 precondition stated in _maybe_yield_parked_carrier
    itself), and only PP0 builds rows, so a PP0-only term would make ranks
    pop different numbers of victims -- a silent rank divergence traded for
    this deadlock.
  * The ring cut's arc asymmetry (pp_flip_drain_leftover_dicts receives only
    from the draining rank's own upstream, scheduler_pp_mixin.py:6796) is left
    as it is: it is fork-own compensation, and under upstream-minimal a defect
    inside a compensation layer is a deletion candidate, not a repair order --
    and widening it could not clear PP1's _missing anyway.
efschu pushed a commit to efschu/htsglang that referenced this pull request Sep 4, 2026
…ll-clock seconds

Review round 1 blocked the horizon build on reachability: a wall-clock
horizon's lapse was unreachable on the weg1b7 specimen, so the guard could
not have fired on the boot it was built for. The bound is now counted in
ring laps, which is the quantity the deadlock is actually made of.

The premise the defer rests on is named in the module and is FALSE for two
populations measured in weg1b7:

  * a re-admitted resident (sgl-project#968 READMIT CACHED, last_queued_as=
    cutover-requeue) is requeued rank-locally -- no chain hop exists to
    wait for;
  * a retracted-after-sealing rid (the #888b parked-carrier relief runs
    after the admission row was sealed) names a rid that will never arrive.

Both make "an unlocatable rid PROVES a chain send is in flight" untrue, and
a frame deferred for ever closes the ring by construction: PP0 has already
launched the pass and is already parked in the receive for its output.

pp_row_defer_horizon.py -> pp_row_defer_cap.py (the name now says what it
counts). The sgl-project#631 row-probe protection itself is unchanged and unweakened;
this only gives it an end.

EVIDENCE

Red-first (falsifiers copied onto parent 8f45927, scratch worktree):
  17 failed / 3 passed of the 20 pins in
  test/registered/unit/managers/test_pp_row_defer_cap_1180.py.
  Green on this tree: 20 passed.

Mutants: 8/8 died (harness /spinning/wt-1180-mut/run_mutants.py, every
  patch byte-verified as applied before its run). They cover both danger
  directions: M6 turns the documented `0 disables` escape into an instant
  killer, M8 stops the group on the ordinary first-sighting hedge. M3 (fork
  the cap into a local literal) was the mutant that exposed a vacuous test
  and forced the AST pin described under B3. One further mutant was DELETED
  rather than reported as a survivor: it edited a docstring sentence only,
  which no behavioural suite can distinguish, so it tested the harness, not
  the coverage. Its behavioural twin M4b dies.

Suites, once, both trees, same five ignores (sgl-project#1044 import-broken modules:
  test_phase_flip_spec_seam_631, test_quiescence_no_carry_858,
  test_phase_flip_decode_relay_631, test_phase_flip_draft_bootstrap_631,
  test_pp_proxy_stamp_631 -- ImportError x4 + AttributeError
  pp_flip_drain_tensor_dicts, identical on both trees):

  parent 8f45927 : 264 failed, 4761 passed, 18 skipped, 5 errors (890s)
  this tree         : 247 failed, 4764 passed, 18 skipped, 0 errors (938s)

  The whole difference is accounted for, by name, not by tally:
  - FAILED/ERROR name sets diffed. Names appearing ONLY on the fixed tree:
    ZERO. This change introduces no new failure.
  - The 17 fewer failures are exactly the 17 red-first falsifiers above.
  - The 5 errors are the collection errors of the five ignored modules; the
    parent scratch worktree collected them, this tree does not.
  - Collected node-ID sets diffed (collect-only, both trees): 5039 vs 5025.
    The 14 collected only on the parent are the entire retired
    test_pp_row_defer_horizon_1180.py -- the file this commit RENAMES. Each
    of its 14 names has a 1:1 successor among the 20; six pins are new.
    No behaviour lost coverage.

  One retired name is a deliberate SEMANTIC INVERSION, not a rename:
  test_the_bound_is_exclusive_at_its_own_value became
  test_the_cap_is_inclusive_at_its_own_value. This is B3 doing its work:
  pp_admission_congruence.py:1036 escalates only after "more than
  UNRESOLVED_DEFER_CAP consecutive passes", i.e. CAP passes are permitted
  and the stop falls on CAP+1 -- inclusive. `occurrence <= bound` matches
  the admission layer; the horizon's exclusive reading did not. Adopting
  the shared object without adopting its arithmetic would have been a
  second accounting wearing the first one's name.

ruff: parity. The same 3 pre-existing errors on both trees; none in the
  files this commit touches.

Not run: no boot, no GPU. This is desk work only.

FINDINGS CONFIRMED AND CONSCIOUSLY LEFT

B4 (the stop is rank-local; a defer taken by one rank alone is itself a
  divergence) CONFIRMED VERBATIM, NOT FIXED. The text is not a review
  opinion, it is the admission layer's own docstring at
  pp_admission_congruence.py:443-446: "ON THE WIRE because the rank that
  OBSERVES the miss is never the rank that can ACT on it: only PP0 chooses
  `told`. Downstream ranks report; PP0 owns the count and the escalation
  (`UNRESOLVED_DEFER_CAP`). A defer that only one rank takes IS the next
  divergence, so no rank takes one alone."

  This cap is rank-local and therefore stands in real tension with that
  rule. Left deliberately, for a reason that is itself the RAENGE-NIE-UNEINS
  law: the remedy for a rank-local decision is to put it on the wire, and
  the wire here is inside the region this stop exists to protect. Adding a
  collective to the row-probe path would give the deadlock a new place to
  live -- the failure this ticket is closing. The honest shape is: the cap
  raises, it does not compensate; every rank on the same unresolvable
  identity reaches the same bound at the same lap, so the stop is expected
  to be group-wide in practice, but that is an ARGUMENT, not a measurement.
  Making it a group fact is sgl-project#968's PP0-authoritative rebuild, not this fix.

B5 (a wall-clock horizon can lapse from scheduling noise alone) DISSOLVED BY
  CONSTRUCTION rather than fixed: the unit is now ring laps, the identity
  restarts on a new frame stamp, there is no wall clock anywhere in
  pp_row_defer_cap.py, and the zero-call-site clear_all() of the horizon
  module is gone (grep count 0). A finding about a clock cannot survive the
  removal of the clock.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants