Repository navigation
Fix streaming bug - #820
Merged
Merged
Conversation
timethink
pushed a commit
to timethink/sglang
that referenced
this pull request
Mar 9, 2025
amote-i
pushed a commit
to amote-i/sglang
that referenced
this pull request
Dec 8, 2025
…model (sgl-project#820) <!-- Thank you for your contribution! Please follow these guidelines to enhance your pull request. If anything is unclear, submit your PR and reach out to maintainers for assistance. Join our Slack community at https://slack.sglang.ai to discuss further. --> ## Motivation <!-- Describe the purpose and goals of this pull request. --> ## Modifications <!-- Detail the changes made in this pull request. --> ## Accuracy Tests <!-- If this pull request affects model outputs (e.g., changes to the kernel or model forward code), provide accuracy test results. --> ## Benchmarking and Profiling <!-- If this pull request impacts inference speed, provide benchmarking and profiling results. --> ## Checklist - [ ] Format your code according to the [Format code with pre-commit](https://docs.sglang.ai/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.ai/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.ai/developer_guide/contribution_guide.html#write-documentations). - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.ai/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.ai/developer_guide/contribution_guide.html#benchmark-the-speed).
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 23, 2026
Carries [sgl-project#790] carry-instrument gating and [sgl-project#777] threshold honesty. feat/797 is an ancestor of this head, so the previous stage is subsumed; kept as its own merge for per-ticket attribution. Clean merge. Touches phase_policy.py, which sgl-project#817/sgl-project#820 also touch -- those branch from 587e4c2 and are merged after, see their stage notes.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 23, 2026
[sgl-project#820] removes the hold's TP mirror, which had no reachable caller. sgl-project#817 is an ancestor of this head, so the previous stage is subsumed; kept as its own merge for per-ticket attribution. Clean merge.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 23, 2026
…t#816, sgl-project#810, sgl-project#806, sgl-project#797, sgl-project#790/sgl-project#777, sgl-project#817/sgl-project#820, sgl-project#818 Merge-checklist duty for the nine stages on this branch. Entries follow each section's own house style, and the mechanism text is taken from the commits' measured evidence rather than restated from the ticket titles. §3 KV backing relief + the allocator cap -- UPDATED IN PLACE rather than given a second bullet, because sgl-project#814 and sgl-project#816 are follow-on defects OF the KvRowCap mechanism that bullet already describes: the census reading the withheld block as a leak (340262 of 465190 ids), the lift being reachable only from a cutover (one boot at 26.8% of its id space for the life of the process), and exposure exceeding the backing (417850 rows over 105413 committed, the device-side assert in masked_set_kv_buffer_kernel). §3 HiCache staging write-through ring (sgl-project#810), new bullet, plus its two companion refusals -- the unbounded-file-tier refusal and the boot preflight ledger entry, the latter being why 22.01 GB of MHATokenToKVPoolHost across three PP ranks previously reached the preflight as nothing. §7 BAR1 deadline + loud abort -- appended the sgl-project#818 peer-liveness half to the existing narrative: the gate could wait forever on a peer that no longer exists, and neither Bar1CollectiveStalled (reset by every resolved read) nor defer_stall_for_building_peer (900 s off a build marker) caught it. §12 Robustness canon -- three new families: contradictory-flag (sgl-project#806), read-back-after-construction (sgl-project#797), denylist-of-reasons (sgl-project#817, sgl-project#820). §18.3 hicache staging sizing (sgl-project#810) -- §18's own rule is that a merge adding a reusable module adds its entry in the SAME merge, and this module had none. Records the removal of fits_pinned_host_budget so it is not reintroduced. §18.6 mamba carry instrument (sgl-project#767, gated by sgl-project#790) and flip break-even N (sgl-project#777). No existing entry was contradicted. Checked before writing: none of sgl-project#814, sgl-project#810, sgl-project#806, sgl-project#772, sgl-project#797, sgl-project#790, sgl-project#777, sgl-project#817, sgl-project#820, sgl-project#818 had a catalog entry, and the one sgl-project#677 line (§19.2, RESTORE-NEVER-REBUILD) describes a different mechanism than the sgl-project#677 layout hold, so it is not stale and was left alone. Gates for the tree this documents (hermetic, CUDA_VISIBLE_DEVICES=""): battery test/registered/unit/{managers,planner,server_args,mem_cache} baseline integ @ 78d27da 44 failed c56d238 (through sgl-project#818) 30 failed, 8452 passed 0 new ids, 14 fixed 1c4eadb (through sgl-project#816) 30 failed, 8461 passed 0 new ids, same set test_barlink_abort_gate_liveness_818.py (outside the battery dirs) 10 passed ruff --select=F401,F821,UP037 and codespell: 0 new findings vs the same file set on 78d27da (16 ruff / 6 codespell exist identically on base) docs-only change; codespell clean on the catalog itself
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 23, 2026
…g said it once The #791b ballot ALREADY detects that the TP replicas' waiting-queue heads have parted -- it rides an all_reduce(MIN) that runs once per TP-loop iteration, so the detector and its carrier both exist. On a mismatch it latched _prefetch_ballot_mismatch_logged, logged ONCE PER PROCESS, and fell back to the rank-local verdict. Its own comment called a divergent queue head "a deeper breakage than a divergent prefetch verdict" and then recorded it exactly once. WHAT THAT COST, from specimen /spinning/evidence-816-18f/wedge_0823_055757 (boot 0516, PP=3 with --enable-phase-flip): 05:55:19 sgl-project#797 void storm begins on PP1 05:55:38 this warning fires -- one line per rank, the last the process ever emits about it. PP0 digest 887126098, PP1+PP2 1471852626. 05:55:53 void storm core: 7710 #797d/sgl-project#798 lines in seven seconds 05:56:13 PHASE-FLIP DONE pp_to_tp (epoch 5) 05:56:15 flip back ABANDONED: "live slot set divergence cannot be repaired this round: the group's union reaches row 240831 and the poorest rank has only 1208" 05:56:15 last real forward progress 05:56:18 the three ranks each build a DIFFERENT prefill batch (#new-seq 1 vs 3, #cached-token 0 vs 16384, #queue-req 6 vs 3) 05:57:57 py-spy: PP0+PP1 in the spec VERIFY arm (eagle_worker_v2.py:2246), PP2 in the EXTEND arm (:2151); all three GPUs pinned at 100%, stacks frozen across three samples 06:00:43 external SIGTERM -- the wedge never resolved itself That 05:55:38 line was the earliest evidence in the whole chain, and the latch made it a single event with no duration attached. "It healed after N passes" and "it never healed" are the same silence to a latched logger, and they call for opposite responses. THIS CHANGES NO DECISION. The fall-back to the rank-local prefetch verdict is untouched; prefetch_done_under_ballot is not modified and its behaviour is pinned by test, because "only logging" is a claim rather than a licence. What changes is that the divergence is counted, its onset is reported, a persisting one is re-reported on a bounded cadence, and the RECOVERY EDGE is reported with the length of the divergence that just ended. NOT UNBOUNDED LOGGING, and the same boot is the argument: 7710 void lines in seven seconds is what a per-iteration warning on a hot loop becomes. The cadence reports streaks 1, 2, 4, 8, ... to a cap and then every cap-many passes -- a handful of lines per minute for a divergence that never heals. THE DECISION IS A PURE FUNCTION, AND A SURVIVING MUTANT IS WHY. advance_mismatch_streak lives in prefetch_ballot.py rather than inline. With it inline the recovery edge could only be checked by grepping the shipped source, and a source grep cannot tell a live branch from `elif False:` -- measured: mutant M2 (recovery edge disabled) left all 14 tests GREEN. After extraction the same mutant kills two tests that drive the machine directly. Same lesson as the sgl-project#820 M5 survivor: a probe that measures presence is not measuring reachability. Tests, hermetic (CUDA_VISIBLE_DEVICES=""): 20 passed, 7.7s. RED-FIRST on unchanged 587e4c2 in a scratch worktree: 10 of 14 red (before the state-machine class was added). The 4 green are the "nothing decides differently" pins plus the detector's own can-fail arm, which hold on base by construction. Noted as a limit, not a strength. Mutants, after extraction, each killed by the arm that owns it: M1 latch restored DEAD (1) M2 recovery edge disabled DEAD (2) -- SURVIVED before extraction M3 cadence logs every pass DEAD (4) M4 total resets across recovery DEAD (1) Regression: every registered unit test touching prefetch_ballot or the uniform reduce -- 40 failed / 65 passed on branch, 40 failed / 45 passed on unchanged base (the 20 difference is this commit's new file). Identical failures, all pre-existing holder drift. Zero regressions. black clean; ruff (F401,F821,E731) and codespell report the same pre-existing hits on base and branch, none in the changed region. isort wanted to renormalise this file's whole import block, which is pre-existing disorder and not this change; reverted, so the diff is the block and nothing else (0 import lines touched). Root cause of the wedge remains OPEN. This makes the earliest signal in the chain legible; it does not stop the divergence. Acting on the mismatch (refusing admission, or a uniform named abort off the already-agreed reduce) needs metal and belongs to the 18 lane.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Thank you for your contribution, we really appreciate it. The following instructions will help improve your pull request and make it easier to receive feedback. If there are any items you don't understand, don't worry. Just submit the pull request and ask the maintainers for help.
Motivation
Fix streaming error according to the following case:
Code:
Expected Result:
When
var_name="type"andvar_name="value",the result ofsgl.gen()should be output.Current Result:
Streaming stucks when
var_name="type", and program raises exception whenvar_name="value".Modification
Set streaming event when using
sgl.gen(choice=[]), and check event existence beforeevent.wait().Checklist
pre-commit run --all-filesor other linting tools are used to fix potential lint issues.