Repository navigation
Bump version to 0.2.6 - #779
Merged
Merged
Conversation
timethink
pushed a commit
to timethink/sglang
that referenced
this pull request
Mar 9, 2025
amote-i
pushed a commit
to amote-i/sglang
that referenced
this pull request
Dec 8, 2025
<!-- Thank you for your contribution! Please follow these guidelines to enhance your pull request. If anything is unclear, submit your PR and reach out to maintainers for assistance. Join our Slack community at https://slack.sglang.ai to discuss further. --> ## Motivation <!-- Describe the purpose and goals of this pull request. --> ## Modifications <!-- Detail the changes made in this pull request. --> ## Accuracy Tests <!-- If this pull request affects model outputs (e.g., changes to the kernel or model forward code), provide accuracy test results. --> ## Benchmarking and Profiling <!-- If this pull request impacts inference speed, provide benchmarking and profiling results. --> ## Checklist - [ ] Format your code according to the [Format code with pre-commit](https://docs.sglang.ai/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.ai/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.ai/developer_guide/contribution_guide.html#write-documentations). - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.ai/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.ai/developer_guide/contribution_guide.html#benchmark-the-speed).
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 20, 2026
The sgl-project#779 spec-accept boot gate exists because a healthy-looking boot can serve every request with a dead drafter: on this vocab-quantized checkpoint the draft dequantized the target's int8 rows with an uninitialized scale, accept length sat at 1.02, and health returned 200 throughout (fixed in 4e1b940). A gate against that failure is only worth its exit code if the failure can be reproduced on demand. Asserting that the gate "would" fail is the same class of claim as the one that hid the original defect for weeks. So the defect gets a switch. SGLANG_774_NEUTER_VOCAB_COMPANIONS=1 makes _share_vocab_quant_companions return before it shares anything, which restores the pre-4e1b940900 behaviour exactly: the draft keeps its own never-loaded scale. Booting with it armed drives the drafter into the collapsed regime, and the gate must report FAIL. Booting without it is byte-identical to today -- the branch is a single env lookup on a path that runs once per worker init. The hook is permanent rather than a throwaway patch on purpose. The boot runner refuses a dirty tree, so a temporary patch cannot be booted without polluting history anyway; and the can-fail property is not a one-off claim to be made once and cited forever, it is a property that must stay re-provable as the sharing code changes underneath it. It logs a warning at WARNING level naming itself, and the gate it exists to prove is precisely the thing that catches it -- an armed injector cannot pass acceptance unnoticed.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 21, 2026
…the send The proxy readiness gate refused to enter the blocking receive until the upstream's CHAN_DICT `sent` count exceeded this rank's `consumed` count. `sent` is published strictly after the post, so it says nothing while the upstream is still inside the send call. That window is empty for an ordinary isend -- but not for the first point-to-point op on a torch NCCL process group, which creates the 2-rank communicator lazily, and that creation is a rendezvous: the isend does not return until the peer enters the matching receive. Every boot therefore died on its first real prefill: PP0 cannot bump `sent` until the isend returns the isend cannot return until PP1 enters the receive PP1 will not enter the receive until `sent` bumps with the gate itself as one arc of the cycle. Boots instr7 and instr8 (2026-08-21) produced this identically; py-spy caught PP0 in isend -> send_tensor_dict -> _pp_send_dict_to_next_stage 33 s into a send it would never leave, while both downstream ranks raised "sgl-project#789 PROXY READINESS TIMEOUT ... posted 1681, consumed 1681". The message was not missing. It was being posted to the rank that refused to collect it. The gate's docstring asserted a stuck sender was "covered identically" to a sender that scheduled nothing; that assertion was the defect. Add a second counter, `attempted`, published on the line BEFORE the send call, and read only by the gate that would otherwise raise. "The upstream has irrevocably entered a send for me" is as positive a presence signal as "the upstream posted", and during a rendezvous it is the only one that exists. The module's ordering rule is unchanged and still governs `sent`; the new counter is not the phantom-message hazard in reverse, because the decision to send is already taken and unconditional at the bump, so the receiver waits on transfer and rendezvous time rather than on peer scheduling. Drain loops keep reading `sent`: they may take off the wire only what is provably already on it. Cost is one extra /dev/shm publish per dict message, measured at 13.6 us. Tests, all on this branch, measured not assumed: test_pp_proxy_readiness_rendezvous_789.py, 5 passed -- three live spawned processes, real gloo, the shipped functions, and a wire whose send blocks in a real dist.recv until the downstream enters recv, i.e. the NCCL lazy-init property in gloo primitives. The wire supplies a hazard rather than a guarantee, closing the direction of the trap where a test transport is more careful than production. Can-fail: neutering only PhaseFlipCounters.attempted, in the child, reproduces sgl-project#789 on both downstream ranks -- the metal specimen. test/registered/unit/managers full directory: 47 failed, 2753 passed. HEAD (96df16d) measured with the identical command: 47 failed, 2748 passed. The failing set is identical in both directions; the 5 added passes are the new file. The five holder/stub repairs in existing tests are interface drift only -- no assertion touched. Boot instr9, PP=3 on this rig with the reference instrumented configuration: health reached, 15 min 12 s uptime, one request served, the 8-request burst 8/8 in 3.1 s with per-request spec_accept_length mean 2.38 (sgl-project#779 gate PASS), 6 further back-to-back bursts under sustained load, 48 more requests, 0 failures, still serving afterwards. Zero sgl-project#789 raises, zero wedge markers, zero scheduler exceptions, and sgl-project#788 admission byte-identical on all three ranks including the 7-request batch. The first surviving boot of this series. Corridor is NOT passed on that boot and is not addressed here: NVML free sat flat at 5229/6612/5475 MiB per card under load against the 819-1229 MiB band, i.e. 4.4-5.0 GiB per card unclaimed against grants of 31800/18800/19800 MiB. That is a planner capacity item, and it is the first boot in this series on which the number means anything, because no earlier one ever served a request.
YAMY1234
added a commit
to YAMY1234/sglang
that referenced
this pull request
Sep 26, 2026
…L_BLOCK_PAD=1): length-bucketed block graphs for other singleton lengths (<=8K by 512, <=32K by 4096; padding rows g=0/beta=0 leave the state unchanged), receipts and bytewise check mode; test Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
YAMY1234
added a commit
to YAMY1234/sglang
that referenced
this pull request
Sep 26, 2026
…ormed in fp32 and cast (Triton interpreter has no bf16 constants); non-token tensors plain copies Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
YAMY1234
added a commit
to YAMY1234/sglang
that referenced
this pull request
Sep 26, 2026
- join once per stream per side-stream record; per-layer pool methods no longer wrapped (they run inside a target forward that init_forward_metadata joined) - cached env lookups and row indices; commit-graph key/size from the first layer - capture the singleton densify graphs at the first extend forward (server warmup) instead of the first prefix-hit prefill Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
YAMY1234
added a commit
to YAMY1234/sglang
that referenced
this pull request
Sep 26, 2026
…evice Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
YAMY1234
added a commit
to YAMY1234/sglang
that referenced
this pull request
Sep 26, 2026
…s from the real last row in padded graphs (state-kernel variant with a real-length device scalar written by the bind kernel; chunks past it leave the state untouched); default kernel unchanged Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
YAMY1234
added a commit
to YAMY1234/sglang
that referenced
this pull request
Sep 26, 2026
…IT_DEFER=1): the final commit group (same code) starts at the pool's first reader instead of beside the last layers before sampling; readers already join via _pending_commit Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
YAMY1234
added a commit
to YAMY1234/sglang
that referenced
this pull request
Sep 26, 2026
…6 copy of b_v under its own name inside the runtime branch (Triton type check) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
YAMY1234
added a commit
to YAMY1234/sglang
that referenced
this pull request
Sep 27, 2026
… to 8192 tokens (longer chunks are GPU-bound and eager); real-end state kernel loops over the real chunks only (no runtime branch in the chunk loop) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
YAMY1234
added a commit
to YAMY1234/sglang
that referenced
this pull request
Sep 27, 2026
…IT_SPLIT=1): unfinished-prompt commit split -- exact ring copy and slot flags on the main stream, the captured commit graph (factors, tracked states, prefix publish) on the side stream in parallel with the next chunk; extend metadata/plan do not wait for it, factor readers and the next commit do Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
YAMY1234
added a commit
to YAMY1234/sglang
that referenced
this pull request
Sep 27, 2026
… receipt (first use and every 50th), logging only Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
YAMY1234
added a commit
to YAMY1234/sglang
that referenced
this pull request
Sep 27, 2026
…NG_GDN_OPUS_COMMIT_QUEUE=1): queue unfinished-prompt chunk-end commits for the first other pool reader At a mid-prompt chunk end only the exact ring copy and slot flags (what the next chunk reads) are written on the main stream; the unchanged commit graph (factors, tracked prefix copy, slot copy) is queued and runs, in order, at the pool's first other reader -- decode metadata, slot copies, or a plan that is not the queued prompts' own continuation -- i.e. after a lone prompt's first token. Same kernels and inputs; only when the commit graphs run moves. Receipts (first queued commit, every 50th flush) are written off the forward thread. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
YAMY1234
added a commit
to YAMY1234/sglang
that referenced
this pull request
Sep 27, 2026
…idle (SGLANG_GDN_OPUS_IDLE_DRAIN, sgl-project#779) A one-output-token prompt has no second decode step, so with OPUS_COMMIT_TRACK_ONLY its queued tracked-copy jobs drained in the next request's first chunk (~5 ms host per chunk end; 895603 trace19 second cold turn +25 ms). The scheduler's fully-idle hook now runs the same drain in the forward stream's context. Default off. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.