Skip to content

Fix streaming bug - #820

Merged
Ying1123 merged 1 commit into
sgl-project:mainfrom
objnf-dev:main
Jul 30, 2024
Merged

Ying1123 merged 1 commit into
sgl-project:mainfrom
objnf-dev:main

Conversation

@objnf-dev

Copy link
Copy Markdown
Contributor

Thank you for your contribution, we really appreciate it. The following instructions will help improve your pull request and make it easier to receive feedback. If there are any items you don't understand, don't worry. Just submit the pull request and ask the maintainers for help.

Motivation

Fix streaming error according to the following case:

Code:

@sgl.function
def test(s):
    s += sgl.system("You are a helpful assistant.")
    s += sgl.user("Choose a fruit and introduce it: apple, orange")
    with s.assistant():
        s += "Let's choose " + sgl.gen(choices=["apple", "orange"], name="type")
        if s["type"] == "apple":
            s += "The colour of apple is " + sgl.gen(max_tokens=100, stop=".", name="value")
        else:
            s += "The shape of orange is " + sgl.gen(max_tokens=100, stop=".", name="value")

state = test.run(stream=True)
for out in state.text_iter(var_name="value"):
    print(out, end="", flush=True)

Expected Result:

When var_name="type" and var_name="value",the result of sgl.gen() should be output.

Current Result:

Streaming stucks when var_name="type", and program raises exception when var_name="value".

Modification

Set streaming event when using sgl.gen(choice=[]), and check event existence before event.wait().

Checklist

  1. Ensure pre-commit pre-commit run --all-files or other linting tools are used to fix potential lint issues.
  2. Confirm that modifications are covered by complete unit tests. If not, please add more unit tests for correctness.
  3. Modify documentation as needed, such as docstrings or example tutorials.

@Ying1123
Ying1123 merged commit daf593a into sgl-project:main Jul 30, 2024
timethink pushed a commit to timethink/sglang that referenced this pull request Mar 9, 2025
amote-i pushed a commit to amote-i/sglang that referenced this pull request Dec 8, 2025
…model (sgl-project#820)

<!-- Thank you for your contribution! Please follow these guidelines to
enhance your pull request. If anything is unclear, submit your PR and
reach out to maintainers for assistance. Join our Slack community at
https://slack.sglang.ai to discuss further. -->

## Motivation

<!-- Describe the purpose and goals of this pull request. -->

## Modifications

<!-- Detail the changes made in this pull request. -->

## Accuracy Tests

<!-- If this pull request affects model outputs (e.g., changes to the
kernel or model forward code), provide accuracy test results. -->

## Benchmarking and Profiling

<!-- If this pull request impacts inference speed, provide benchmarking
and profiling results. -->

## Checklist

- [ ] Format your code according to the [Format code with
pre-commit](https://docs.sglang.ai/developer_guide/contribution_guide.html#format-code-with-pre-commit).
- [ ] Add unit tests according to the [Run and add unit
tests](https://docs.sglang.ai/developer_guide/contribution_guide.html#run-and-add-unit-tests).
- [ ] Update documentation according to [Write
documentations](https://docs.sglang.ai/developer_guide/contribution_guide.html#write-documentations).
- [ ] Provide accuracy and speed benchmark results according to [Test
the
accuracy](https://docs.sglang.ai/developer_guide/contribution_guide.html#test-the-accuracy)
and [Benchmark the
speed](https://docs.sglang.ai/developer_guide/contribution_guide.html#benchmark-the-speed).
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 23, 2026
 sibling is

test_hold_allowlist_817.py -- the file this one continues -- calls
register_cpu_ci, and 117 of the 245 suites in this directory do. Without it
these tests exist but never run where it matters, which is the inert-lever
shape the change itself is about.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 23, 2026
Carries [sgl-project#790] carry-instrument gating and [sgl-project#777] threshold honesty. feat/797
is an ancestor of this head, so the previous stage is subsumed; kept as its own
merge for per-ticket attribution. Clean merge. Touches phase_policy.py, which
sgl-project#817/sgl-project#820 also touch -- those branch from 587e4c2 and are merged after, see
their stage notes.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 23, 2026
[sgl-project#820] removes the hold's TP mirror, which had no reachable caller. sgl-project#817 is an
ancestor of this head, so the previous stage is subsumed; kept as its own merge
for per-ticket attribution. Clean merge.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 23, 2026
…t#816, sgl-project#810, sgl-project#806, sgl-project#797, sgl-project#790/sgl-project#777, sgl-project#817/sgl-project#820, sgl-project#818

Merge-checklist duty for the nine stages on this branch. Entries follow each
section's own house style, and the mechanism text is taken from the commits'
measured evidence rather than restated from the ticket titles.

§3 KV backing relief + the allocator cap -- UPDATED IN PLACE rather than given
a second bullet, because sgl-project#814 and sgl-project#816 are follow-on defects OF the KvRowCap
mechanism that bullet already describes: the census reading the withheld block
as a leak (340262 of 465190 ids), the lift being reachable only from a cutover
(one boot at 26.8% of its id space for the life of the process), and exposure
exceeding the backing (417850 rows over 105413 committed, the device-side
assert in masked_set_kv_buffer_kernel).

§3 HiCache staging write-through ring (sgl-project#810), new bullet, plus its two
companion refusals -- the unbounded-file-tier refusal and the boot preflight
ledger entry, the latter being why 22.01 GB of MHATokenToKVPoolHost across
three PP ranks previously reached the preflight as nothing.

§7 BAR1 deadline + loud abort -- appended the sgl-project#818 peer-liveness half to the
existing narrative: the gate could wait forever on a peer that no longer
exists, and neither Bar1CollectiveStalled (reset by every resolved read) nor
defer_stall_for_building_peer (900 s off a build marker) caught it.

§12 Robustness canon -- three new families: contradictory-flag (sgl-project#806),
read-back-after-construction (sgl-project#797), denylist-of-reasons (sgl-project#817, sgl-project#820).

§18.3 hicache staging sizing (sgl-project#810) -- §18's own rule is that a merge adding a
reusable module adds its entry in the SAME merge, and this module had none.
Records the removal of fits_pinned_host_budget so it is not reintroduced.

§18.6 mamba carry instrument (sgl-project#767, gated by sgl-project#790) and flip break-even N
(sgl-project#777).

No existing entry was contradicted. Checked before writing: none of sgl-project#814,
sgl-project#810, sgl-project#806, sgl-project#772, sgl-project#797, sgl-project#790, sgl-project#777, sgl-project#817, sgl-project#820, sgl-project#818 had a catalog entry, and
the one sgl-project#677 line (§19.2, RESTORE-NEVER-REBUILD) describes a different
mechanism than the sgl-project#677 layout hold, so it is not stale and was left alone.

Gates for the tree this documents (hermetic, CUDA_VISIBLE_DEVICES=""):

  battery test/registered/unit/{managers,planner,server_args,mem_cache}
    baseline integ @ 78d27da       44 failed
    c56d238 (through sgl-project#818)         30 failed, 8452 passed   0 new ids, 14 fixed
    1c4eadb (through sgl-project#816)         30 failed, 8461 passed   0 new ids, same set
  test_barlink_abort_gate_liveness_818.py (outside the battery dirs)  10 passed
  ruff --select=F401,F821,UP037 and codespell: 0 new findings vs the same
    file set on 78d27da (16 ruff / 6 codespell exist identically on base)
  docs-only change; codespell clean on the catalog itself
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 23, 2026
…g said it once

The #791b ballot ALREADY detects that the TP replicas' waiting-queue heads
have parted -- it rides an all_reduce(MIN) that runs once per TP-loop
iteration, so the detector and its carrier both exist. On a mismatch it
latched _prefetch_ballot_mismatch_logged, logged ONCE PER PROCESS, and fell
back to the rank-local verdict. Its own comment called a divergent queue
head "a deeper breakage than a divergent prefetch verdict" and then recorded
it exactly once.

WHAT THAT COST, from specimen /spinning/evidence-816-18f/wedge_0823_055757
(boot 0516, PP=3 with --enable-phase-flip):

  05:55:19  sgl-project#797 void storm begins on PP1
  05:55:38  this warning fires -- one line per rank, the last the process
            ever emits about it. PP0 digest 887126098, PP1+PP2 1471852626.
  05:55:53  void storm core: 7710 #797d/sgl-project#798 lines in seven seconds
  05:56:13  PHASE-FLIP DONE pp_to_tp (epoch 5)
  05:56:15  flip back ABANDONED: "live slot set divergence cannot be
            repaired this round: the group's union reaches row 240831 and
            the poorest rank has only 1208"
  05:56:15  last real forward progress
  05:56:18  the three ranks each build a DIFFERENT prefill batch
            (#new-seq 1 vs 3, #cached-token 0 vs 16384, #queue-req 6 vs 3)
  05:57:57  py-spy: PP0+PP1 in the spec VERIFY arm (eagle_worker_v2.py:2246),
            PP2 in the EXTEND arm (:2151); all three GPUs pinned at 100%,
            stacks frozen across three samples
  06:00:43  external SIGTERM -- the wedge never resolved itself

That 05:55:38 line was the earliest evidence in the whole chain, and the
latch made it a single event with no duration attached. "It healed after N
passes" and "it never healed" are the same silence to a latched logger, and
they call for opposite responses.

THIS CHANGES NO DECISION. The fall-back to the rank-local prefetch verdict is
untouched; prefetch_done_under_ballot is not modified and its behaviour is
pinned by test, because "only logging" is a claim rather than a licence. What
changes is that the divergence is counted, its onset is reported, a
persisting one is re-reported on a bounded cadence, and the RECOVERY EDGE is
reported with the length of the divergence that just ended.

NOT UNBOUNDED LOGGING, and the same boot is the argument: 7710 void lines in
seven seconds is what a per-iteration warning on a hot loop becomes. The
cadence reports streaks 1, 2, 4, 8, ... to a cap and then every cap-many
passes -- a handful of lines per minute for a divergence that never heals.

THE DECISION IS A PURE FUNCTION, AND A SURVIVING MUTANT IS WHY.
advance_mismatch_streak lives in prefetch_ballot.py rather than inline. With
it inline the recovery edge could only be checked by grepping the shipped
source, and a source grep cannot tell a live branch from `elif False:` --
measured: mutant M2 (recovery edge disabled) left all 14 tests GREEN. After
extraction the same mutant kills two tests that drive the machine directly.
Same lesson as the sgl-project#820 M5 survivor: a probe that measures presence is not
measuring reachability.

Tests, hermetic (CUDA_VISIBLE_DEVICES=""): 20 passed, 7.7s.
  RED-FIRST on unchanged 587e4c2 in a scratch worktree: 10 of 14 red
    (before the state-machine class was added). The 4 green are the
    "nothing decides differently" pins plus the detector's own can-fail arm,
    which hold on base by construction. Noted as a limit, not a strength.
  Mutants, after extraction, each killed by the arm that owns it:
    M1 latch restored               DEAD (1)
    M2 recovery edge disabled       DEAD (2)  -- SURVIVED before extraction
    M3 cadence logs every pass      DEAD (4)
    M4 total resets across recovery DEAD (1)
  Regression: every registered unit test touching prefetch_ballot or the
    uniform reduce -- 40 failed / 65 passed on branch, 40 failed / 45 passed
    on unchanged base (the 20 difference is this commit's new file).
    Identical failures, all pre-existing holder drift. Zero regressions.
  black clean; ruff (F401,F821,E731) and codespell report the same
    pre-existing hits on base and branch, none in the changed region.
  isort wanted to renormalise this file's whole import block, which is
    pre-existing disorder and not this change; reverted, so the diff is the
    block and nothing else (0 import lines touched).

Root cause of the wedge remains OPEN. This makes the earliest signal in the
chain legible; it does not stop the divergence. Acting on the mismatch
(refusing admission, or a uniform named abort off the already-agreed reduce)
needs metal and belongs to the 18 lane.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants