Repository navigation
minor: some potential bugs - #1044
Merged
Merged
Conversation
merrymercy
requested changes
Aug 12, 2024
merrymercy
approved these changes
Aug 12, 2024
merrymercy
enabled auto-merge (squash)
August 12, 2024 05:00
timethink
pushed a commit
to timethink/sglang
that referenced
this pull request
Mar 9, 2025
4 of 5 tasks
hhhh1252023
pushed a commit
to hhhh1252023/sglang_public
that referenced
this pull request
Jul 28, 2026
test_npu_serving_transcription/test_npu_encoder_dp/test_npu_token_id_retokenize_e2e/test_npu_vision_chunked_prefill
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 31, 2026
… of the apply swept Boot 14 published a row and no rank stamped one, and the receiver's four ways to decline were one indistinguishable silence. Instrumented as a sweep of the whole function rather than the one branch under suspicion -- this campaign has paid three times for instrumenting a branch and leaving its neighbour mute. SENDER: `sgl-project#1044 ROW SHIP` counts posted_with_row / posted_no_row / posted_no_decision / skipped_with_row / skipped_no_row / entries_shipped. The SKIPPED passes are counted too: a frame that is never posted is exactly the case a posted-only counter cannot see. RECEIVER: `sgl-project#1044 ROW RECV` names every exit of `apply_pp_load_back_row` -- no_entries, entry_no_extent, entry_no_rid, no_wanted, unknown_rid, stamped. `unknown_rid` is candidate (D), the branch that could absorb the entire chain without a line of evidence. Together they partition in ONE boot: entries shipped with wanted 0 everywhere is (C) delivery; wanted with unknown_rid is (D) lap; wanted with neither is a THIRD branch, which is why the whole function is swept. Emission is unconditional whenever a row is involved; where it is sampled, the suppression count is printed -- a zero from a rate-limited emitter is not a zero, which is this campaign's fourth denominator instance and now a standing rule. A DEAD COUNTER WAS REMOVED RATHER THAN SHIPPED: `raised` had no except path to fire it, so it would have read 0 forever and been indistinguishable from evidence. CHECK: devtools/check_1044_rowtrip.py drives EVERY named counter and asserts it moved; "no dead counters" is its own assertion, as is the denominator (`calls`). Named failure class: a counter that cannot move reproduces the silence this edit removes while appearing to fix it. 15/15 PASS; 1040/1041/1042/1043 re-run green; ruff clean; import smoke. NOT IN THIS COMMIT: the whole-cut to local consumption. Its edge check found a prohibiting edge and it is reported rather than worked around -- see handover section 16.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 31, 2026
…s; delete the load-back delivery chain OPTION A, in the hard form. Two edits that only work together: the memory-axis watchman moves onto the group availability floor FIRST, and only then does the delivery chain come out. sgl-project#1045 -- THE FLOOR IS PUBLISHED UNCONDITIONALLY ON A GROUP. It used to stay None whenever availability happened to be equal that iteration, and `unified_radix_cache`'s load-back fell into a RANK-LOCAL memory decision when it was None, guarded only by the delivery signal `pp_load_back_told`. A watchman on one axis was hanging off the delivery of another, and it stood down exactly when nothing looked wrong. PRICE VERIFIED BEFORE BUILDING, per the no-new-collective rule: at the live path (scheduler.py:6396) `min_avail` and `max_avail` are BOTH harvested from the same already-performed all-reduce (`t[0]`, `t[max_avail_at]`), so dropping the comparison removes work rather than adding it. The single-rank path now publishes the LOCAL value instead of None -- it IS the group min for a group of one -- which is what lets None become a construction violation everywhere downstream. Behaviourally identical on even pools: the published floor equals the live local value, and the sgl-project#694 ledger keeps it tracking through the iteration. FIX-6 IS NOW ONE BRANCH. `floor is None` at the load-back raises, naming rid, kv_tokens and local availability. A rank that cannot answer from a group number must not answer from a local one -- that is how ranks stop agreeing, and the 21:52:25 wedge is the measured price. sgl-project#1046 -- THE DELIVERY CHAIN IS DELETED. Consumption is local: `_pp_load_back_extent` returns this rank's own stamp, chosen at its own match by one shared expression over the same content. Uniformity comes from identical DERIVATION, not from one rank shipping a value. Deleted with it: `apply_pp_load_back_row` and both its call sites, PP0's self-row parking and the one-lap delay, `pp_load_back_told` and all four readers, the `sgl-project#1035` deferral ledger, the fact carriers, `holders_with_unspent_extent`, `_spend_for_entry`, the spend transition, the `sgl-project#1041` population counter and the `sgl-project#1044` ship/recv ledgers. EVERY READER WAS DECIDED, NOT SWEPT ALONG -- the enumeration is the point, because a reader falling silently into a rank-local branch is the class that found the edge in the first place: unified_radix_cache.py:2713 group-fact signal -> moved to the floor (sgl-project#1045) schedule_policy.py:107 delivery mechanics -> local stamp schedule_policy.py:2305 delivery gate -> deleted, no fact to await schedule_policy.py:2360 clamp gate -> KEPT, re-pointed. Its hazard (a GDN anchor adopted at `_applied` while the KV is cut) is created by the load-back itself and is independent of where the extent came from. KEPT AND UNTOUCHED: the admission row, its `prefix_len`/`extend_len` columns and the sgl-project#791 membership and ordering rules -- batch geometry, a different axis. The `load_back_len` FIELD stays on the wire for width tolerance and always travels None. sgl-project#1042's lifecycle contract stays, minus the spend transition that had no consumer left. CHECKS: check_1045_floor_watchman.py proves the guard CAN FIRE (a crash guard that cannot crash is worse than the branch it replaced), that a small floor is an ordinary refusal, that the rank-local fallback is DELETED rather than bypassed, and that publication is unconditional and counted. check_1042 updated: the row build must now LEAVE the field alone -- a build that still cleared it would resurrect the eraser through the back door. check_1040 green. The 1041, 1043 and 1044 checks are DELETED with their subjects; the cell's zombie test picks up their markers (ROW APPLIED, DEFERRED, EXTENT POPULATION, ROW SHIP) so a stale build cannot pass silently. py_compile + import smoke on all five modules; ruff clean. NOT YET BOOTED -- this commit is desk-proven only.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Sep 3, 2026
…mainders; quiescence is an arm precondition; followers execute every launched pass under the arm or STOP weg1b4 08:07:50Z: PP0 launched fwd_ct=81 slot=1 for rid b64dc1cb (extend 1912), posted its sgl-project#631 PROXY-SEND t6, and ONE LINE LATER armed pp_to_tp on a DRAINED verdict reading "0 tok remaining" -- the sgl-project#1153 starvation shape with no designed STOP. D1 THE DRAINED PREMISE COUNTS EVERYTHING STILL TO PREFILL. Scheduler._pending_prefill_tokens now carries an in-flight microbatch term: for every request the PP ring launched but has not returned, it prices len(origin_input_ids) - extend_range.start (NEVER - end: end is advanced at PREPARE time, so pricing from end reads 0 for exactly the request that is still running). Deduped against _queued_ids so a request cannot be billed twice; the chunked and resident terms skip _inflight_ids for the same reason. _pending_prefill_terms records the breakdown and its producer, and all four pp-exit arms print it: "[sgl-project#1173 pending terms: queue=.. inflight=.. chunked=.. arriving=.. resident=.. total=.. (producer Scheduler._pending_prefill_tokens, include_health=..)]". On the weg1b4 numbers the ring reads 1912 for b64dc1cb where it read 0 before. D2a QUIESCENCE IS A PRECONDITION OF THE ARM, NOT A HOLD AFTER IT. PhaseFlipRuntime.arm() takes launched_passes_fn (build_launched_passes_fn reads the same _pp_launched_pending the sgl-project#1020 void guard reads, so arm and guard cannot disagree about what is outstanding). With any slot outstanding the arm is DEFERRED, not taken-and-held: "sgl-project#1173 ARM DEFERRED: launched passes outstanding slots=[..] fwd_ct=.. direction=.. deferrals=..". The check sits after the rank!=0 early return (PP0-only by construction, no new rank-local verdict, no new sync point) and before the side-effecting _prearm_floor_relief. An unreadable probe never defers. The mbs quiescence hold is KEPT, not deleted: it stays reachable on followers, which take the arm as an order and never run this precondition, and for a pass launched around the order's arrival -- its reason text now says so. D2b FOLLOWERS EXECUTE WHAT PP0 LAUNCHED, ARM OR NOT. _pp_flip_hold_slot released the ring only when every slot was None, so a proxy frame for a launched pass sitting in the typed inbox parked the follower on an order it could not obey while the launcher waited for a frame that never came. The hold now releases while such a frame is stashed (_pp_flip_stashed_frame_forces_advance reads the typed inbox ONLY -- never the "provably in flight" counter half of _pp_row_any_proxy_signal, which a transit frame would false-positive during the armed spin) so the loop walks the ring to the slot the frame names. A frame that never leaves is a group STOP through the launcher, never a park and never a silent spin: RuntimeError("sgl-project#1173 LAUNCHED PASS UNEXECUTED UNDER ARM STOP rank=%d slot=%d fwd_ct=%d rid=%s arm_epoch=%d reason=%s") after 2*ring+2 armed visits. Small item: the phase_policy hold that says the premise was verified on the STORE WITNESS now prints the witness STATE census (pending|hit|bounded|cold|unprobed) via phase_purity.store_witness_census. EVIDENCE new test/registered/unit/managers/test_arm_quiescence_1173.py: 14 passed on this tree, 11 failed / 1 passed on parent f58a71b (scratch worktree, same file copied in) = red-first. Mutants, each applied to this tree and reverted: price the in-flight pass from extend_range.end -> 2 red (D1); neutralise the ARM DEFERRED block -> 1 red (D2a); restore the unconditional park in _pp_flip_hold_slot -> 3 red (D2b). All three died. test_pp_flip_slot_hold_631.py: the drained-ring pin is NARROWED, not withdrawn (its docstring says why) and gains the release companion test_does_not_hold_while_a_launched_passs_frame_is_stashed. Touched test files, fixed vs parent: 54 failed / 152 passed vs 54 failed / 151 passed, comm diff EMPTY both directions. test/registered/unit/managers bounded once each: fixed 247 failed / 4684 passed / 5 errors, parent 246 failed / 4670 passed / 5 errors; comm names exactly one extra failure, test_load_snapshot_backends.py::TestZmqRoundTrip ::test_read_returns_latest, which passes SOLO on BOTH trees = the sgl-project#899 box -timing class, not this diff. The 5 collection errors are the pre-existing sgl-project#1044 import-broken *_631 modules on both sides. ruff on the touched files: 104 errors before, 104 after.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Sep 4, 2026
…it in a named stop BOOT weg1b7 (03:49:20 -> 03:50:24Z) died in a circular PP ring wait: PP0 parked in recv_object[src=2], PP1 tore at the 60 s chain-stall bound, both downstream ranks ran their two ring-cut drain turns and the group died. ROOT (built against, and it is NOT the ring cut). The sgl-project#631 row probe peeks the head proxy frame's admission row and, when the row names a rid this rank cannot locate, answers False and leaves the frame in the inbox. That protection is right (boot 631row14: a frame consumed before its request hop plans nothing and the ring dies upstream-waiting) and is untouched here. What it lacked is a BOUND, and the defer is not free: PP0 has ALREADY launched the pass the frame belongs to and is ALREADY parked on its output, so an unbounded defer closes the ring BY CONSTRUCTION. The defer's own premise -- "an unlocatable rid PROVES a chain send is in flight, bounded by rendezvous+transfer" -- is false for both populations measured in weg1b7: * a rid PP0 RETRACTED after sealing the row. PP0 shipped sgl-project#631 PROXY-SEND t40 naming 772368da and c25108f3, then the discretionary #888b parked- carrier relief retracted c25108f3 in the NEXT scheduling iteration (#969AD RETRACT site=_retract_decode_and_requeue fwd_ct=157). A sealed row cannot be amended, so _missing can never empty. * a re-admitted resident (sgl-project#968 READMIT CACHED, last_queued_as=cutover- requeue) requeued rank-locally by _add_request_to_queue -- no chain hop for it exists or ever will. Ranks disagreeing about a rid is RAENGE-NIE-UNEINS, and the sanctioned answer there is a bounded, NAMED stop -- never a silent unbounded wait. sgl-project#1071's occupant horizon is the precedent this follows, down to "0 disables". CHANGE python/sglang/srt/managers/pp_row_defer_horizon.py (new, 300 lines) RowDeferHorizon: per-slot clock over the (frame stamp, missing-rid set) identity. First sighting defers. A CHANGED missing set is a hop landing and restarts the clock -- the boot 631row15 world (43k hot defers on a TURNING ring) therefore never reaches the bound. A DIFFERENT frame also restarts it, so a slot whose head alternates cannot inherit a predecessor's age. Only the same frame with the same missing set, held past SGLANG_PP_ROW_DEFER_HORIZON_S (default 20 s), answers defer=False with a message naming slot, wait, occurrences, rids, BOTH measured shapes, what is deliberately not repaired, and the remedy knob. PURE: no logger, no wire, no scheduler attribute -- the caller owns the raise, as parked_carrier_relief and min_free_slots_delayer are built. 20 s is chosen against three measured neighbours: orders of magnitude above the healthy case (weg1b7's three defers of one frame land in the same log second), BELOW the 60 s chain stall bound so THIS names the contradiction before the ring cut misreports it as "closed ring on the request chain", and BELOW the 90 s sgl-project#1071 horizon. scheduler_pp_mixin.py:9184-9243 the defer consults the horizon and raises PpRowDeferHorizonLapsed past the bound. It does NOT consume the frame (that revives boot 631row14) and does NOT emit a void (that revives the sgl-project#801 reverse corpse). The DEFER log line now carries held=%.1fs. scheduler_pp_mixin.py:9266-9271 a satisfied or unreadable row clears the slot's clock, so a resolved wait cannot age a later one into the bound. scheduler_pp_mixin.py:8912-8941 _pp_row_chain_pending's docstring corrected: its _pp_row_chain_owed arm returns True on an INFERENCE, not on proof of availability, and the receive it opens is bounded by this horizon rather than by transfer time. Do not read that True as uniform evidence of a posted message. EVIDENCE RED FIRST on parent 2615514 (scratch worktree /spinning/wt-1180-red, only the new test file copied in): 12 failed / 2 passed. Two of the reds are BEHAVIOURAL, not import errors -- test_the_defer_is_not_unbounded: "the row probe deferred the same unsatisfiable row 4096 times over an hour of its own clock without ever stopping" (imports nothing new, on purpose); test_a_satisfied_row_clears_the_clock_its_own_defer_armed: "the defer must have armed a clock". FIXED: 14 passed / 0 failed. MUTANTS 6/6 died, each patch verified applied: M1 never raise (fix inert) DIED M2 bound never lapses DIED M3 a changed missing set does not restart the clock DIED M4 satisfied row leaves its clock behind DIED M5 no frame token in the identity DIED M6 DANGER DIRECTION: stop on the FIRST defer DIED M4 survived the first round -- the locatable-row test was vacuous (no horizon object existed to inspect). The recovery test above was added to make the clear observable, and M4 then died. ruff parity: 3 pre-existing errors on the parent, the same 3 after (F841 carries_flip_arm, E741 l x2). Import smoke hermetic (CVD=""). SUITE ONCE, both trees, same command, sequentially (crowding-free box, load ~2.9), test/registered/unit/managers with the five pre-existing collection-error modules ignored (the same five on both trees: the sgl-project#1044 import-broken *_631 quartet plus test_quiescence_no_carry_858): parent 2615514 : 240 failed / 4767 passed / 2 skipped / 638 subtests fixed : 240 failed / 4781 passed / 2 skipped / 638 subtests Delta = +14 passed, exactly the new file; the FAILED NAME SETS are byte identical (diff empty, 236 names each side). Counting probe declared honestly: 236 extracted names vs the summary's 240 on BOTH trees -- a symmetric extraction gap of 4 that is not investigated here and cannot affect the A/B, since both sides carry the same 236 names and the same summary. RESIDUALS, deliberately NOT fixed here * The stop is RANK-LOCAL, exactly like sgl-project#1071's: the deferring rank raises, the others then die on their own collectives or the deadman. A group-uniform stop would need a collective on the wedged path, which is what the ring cut already refuses. * #888b's discretionary yield is untouched. Making it protect published rids is refused: the entry decision and the victim count must be rank-uniform (the sgl-project#583 precondition stated in _maybe_yield_parked_carrier itself), and only PP0 builds rows, so a PP0-only term would make ranks pop different numbers of victims -- a silent rank divergence traded for this deadlock. * The ring cut's arc asymmetry (pp_flip_drain_leftover_dicts receives only from the draining rank's own upstream, scheduler_pp_mixin.py:6796) is left as it is: it is fork-own compensation, and under upstream-minimal a defect inside a compensation layer is a deletion candidate, not a repair order -- and widening it could not clear PP1's _missing anyway.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Sep 4, 2026
…ll-clock seconds Review round 1 blocked the horizon build on reachability: a wall-clock horizon's lapse was unreachable on the weg1b7 specimen, so the guard could not have fired on the boot it was built for. The bound is now counted in ring laps, which is the quantity the deadlock is actually made of. The premise the defer rests on is named in the module and is FALSE for two populations measured in weg1b7: * a re-admitted resident (sgl-project#968 READMIT CACHED, last_queued_as= cutover-requeue) is requeued rank-locally -- no chain hop exists to wait for; * a retracted-after-sealing rid (the #888b parked-carrier relief runs after the admission row was sealed) names a rid that will never arrive. Both make "an unlocatable rid PROVES a chain send is in flight" untrue, and a frame deferred for ever closes the ring by construction: PP0 has already launched the pass and is already parked in the receive for its output. pp_row_defer_horizon.py -> pp_row_defer_cap.py (the name now says what it counts). The sgl-project#631 row-probe protection itself is unchanged and unweakened; this only gives it an end. EVIDENCE Red-first (falsifiers copied onto parent 8f45927, scratch worktree): 17 failed / 3 passed of the 20 pins in test/registered/unit/managers/test_pp_row_defer_cap_1180.py. Green on this tree: 20 passed. Mutants: 8/8 died (harness /spinning/wt-1180-mut/run_mutants.py, every patch byte-verified as applied before its run). They cover both danger directions: M6 turns the documented `0 disables` escape into an instant killer, M8 stops the group on the ordinary first-sighting hedge. M3 (fork the cap into a local literal) was the mutant that exposed a vacuous test and forced the AST pin described under B3. One further mutant was DELETED rather than reported as a survivor: it edited a docstring sentence only, which no behavioural suite can distinguish, so it tested the harness, not the coverage. Its behavioural twin M4b dies. Suites, once, both trees, same five ignores (sgl-project#1044 import-broken modules: test_phase_flip_spec_seam_631, test_quiescence_no_carry_858, test_phase_flip_decode_relay_631, test_phase_flip_draft_bootstrap_631, test_pp_proxy_stamp_631 -- ImportError x4 + AttributeError pp_flip_drain_tensor_dicts, identical on both trees): parent 8f45927 : 264 failed, 4761 passed, 18 skipped, 5 errors (890s) this tree : 247 failed, 4764 passed, 18 skipped, 0 errors (938s) The whole difference is accounted for, by name, not by tally: - FAILED/ERROR name sets diffed. Names appearing ONLY on the fixed tree: ZERO. This change introduces no new failure. - The 17 fewer failures are exactly the 17 red-first falsifiers above. - The 5 errors are the collection errors of the five ignored modules; the parent scratch worktree collected them, this tree does not. - Collected node-ID sets diffed (collect-only, both trees): 5039 vs 5025. The 14 collected only on the parent are the entire retired test_pp_row_defer_horizon_1180.py -- the file this commit RENAMES. Each of its 14 names has a 1:1 successor among the 20; six pins are new. No behaviour lost coverage. One retired name is a deliberate SEMANTIC INVERSION, not a rename: test_the_bound_is_exclusive_at_its_own_value became test_the_cap_is_inclusive_at_its_own_value. This is B3 doing its work: pp_admission_congruence.py:1036 escalates only after "more than UNRESOLVED_DEFER_CAP consecutive passes", i.e. CAP passes are permitted and the stop falls on CAP+1 -- inclusive. `occurrence <= bound` matches the admission layer; the horizon's exclusive reading did not. Adopting the shared object without adopting its arithmetic would have been a second accounting wearing the first one's name. ruff: parity. The same 3 pre-existing errors on both trees; none in the files this commit touches. Not run: no boot, no GPU. This is desk work only. FINDINGS CONFIRMED AND CONSCIOUSLY LEFT B4 (the stop is rank-local; a defer taken by one rank alone is itself a divergence) CONFIRMED VERBATIM, NOT FIXED. The text is not a review opinion, it is the admission layer's own docstring at pp_admission_congruence.py:443-446: "ON THE WIRE because the rank that OBSERVES the miss is never the rank that can ACT on it: only PP0 chooses `told`. Downstream ranks report; PP0 owns the count and the escalation (`UNRESOLVED_DEFER_CAP`). A defer that only one rank takes IS the next divergence, so no rank takes one alone." This cap is rank-local and therefore stands in real tension with that rule. Left deliberately, for a reason that is itself the RAENGE-NIE-UNEINS law: the remedy for a rank-local decision is to put it on the wire, and the wire here is inside the region this stop exists to protect. Adding a collective to the row-probe path would give the deadlock a new place to live -- the failure this ticket is closing. The honest shape is: the cap raises, it does not compensate; every rank on the same unresolvable identity reaches the same bound at the same lap, so the stop is expected to be group-wide in practice, but that is an ARGUMENT, not a measurement. Making it a group fact is sgl-project#968's PP0-authoritative rebuild, not this fix. B5 (a wall-clock horizon can lapse from scheduling noise alone) DISSOLVED BY CONSTRUCTION rather than fixed: the unit is now ring laps, the identity restarts on a new frame stamp, there is no wall clock anywhere in pp_row_defer_cap.py, and the zero-call-site clear_all() of the horizon module is gone (grep count 0). A finding about a clock cannot survive the removal of the clock.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Thank you for your contribution, we really appreciate it. The following instructions will help improve your pull request and make it easier to receive feedback. If there are any items you don't understand, don't worry. Just submit the pull request and ask the maintainers for help.
Motivation
This PR solves two potential bugs.
Checklist
pre-commit run --all-filesor other linting tools are used to fix potential lint issues.