Repository navigation
fix: resolve correctness_test issue - #1002
Merged
Merged
Conversation
timethink
pushed a commit
to timethink/sglang
that referenced
this pull request
Mar 9, 2025
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 29, 2026
Boot 51 died on "sgl-project#789 OUTPUT READINESS TIMEOUT: mb_id=2: upstream (rank 1) posted 9 dict message(s) on dict|output (entered 9), this rank has consumed 9". Nine posted, nine consumed, nothing outstanding -- and the rank waited for a tenth, then killed the instance for its absence. The arithmetic in the readiness gate is right; the caller is not. `_do_recv` enters the blocking output receive on `mbs[next_mb_id] is not None and not prebuilt and not can_skip` -- this rank's OWN slot state -- while the upstream forwards on `if pp_outputs:`. Two predicates on one stream, and the raise's own hint says so: "this rank decided to receive from its OWN slot state while its upstream decided to forward on `if pp_outputs:`, and nothing publishes this rank's per-slot expectation to that upstream". So the gate now reports its verdict instead of only ever killing on it. When `consumed == posted == attempted`, the upstream has stated in-band that it forwarded nothing for this slot, and the caller takes the SAME no-output exit it already takes for `target is None` two lines up -- an existing, handled path, not a new one. What this deliberately is not: nothing is dropped, because nothing was sent; nothing is refused, because there is nothing to refuse. Those two forms are both metal-falsified on this path already (corpse R 2026-08-09, and sgl-project#995 in this window, boot 15). And no token is fabricated -- `_pp_make_skip_output_ result` is NOT reused here, because its zero placeholder is legitimate only for a chunk that really ran and produced no token, which is not this case. Opt-in, default unchanged. The proxy alias keeps raising: a missing proxy is a different fact, since those hidden states are the input to this rank's own forward and declining them would compute on nothing. Evidence: desk, executed. py_compile; both `soft` parameters present; the proxy alias reaches no soft path; the declined case emits a named line with posted/entered/consumed so a decline can never become quiet. Belegstufe: DESK-BEWIESEN -- the metal proof is the boot.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 31, 2026
… visible Boot 631row3 wedged in a three-arc ring 60 s after READY: the chain hop precedes the frame by the upstream's whole forward, so the probe was correctly False at plan time, the loop then parked in the BLOCKING chain receive, and PP0 parked in the output receive downstream never fed (py-spy: PP1 in PpChainReceiver.recv <- recv_requests, PP0 in _do_recv <- _pp_commit_send_output_work). The pre-row world never hit this because a downstream rank with work planned locally and parked on the PROXY wire. Row mode now does the same parking on the same wire: probe False + visible work (queue/chunked/slots/running) -> the budgeted soft readiness wait (30 s, declines loudly per sgl-project#1002), then re-probe and route by stamp. An idle rank has no visible work and still parks on the chain -- the PP3-solo shape stays deadlock-free. Matched check: py_compile. Metal proof is the boot.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
as titled cc @merrymercy @Ying1123 @hnyls2002 @min-xu-et
tested with
Modification
Briefly describe the changes made in this PR.
Checklist
pre-commit run --all-filesor other linting tools are used to fix potential lint issues.