Skip to content

Bump version to 0.2.6 - #779

Merged
merrymercy merged 1 commit into
mainfrom
version
Jul 28, 2024
Merged

merrymercy merged 1 commit into
mainfrom
version

Conversation

@merrymercy

Copy link
Copy Markdown
Collaborator

No description provided.

@merrymercy
merrymercy merged commit bc1154c into main Jul 28, 2024
@merrymercy
merrymercy deleted the version branch July 28, 2024 03:29
timethink pushed a commit to timethink/sglang that referenced this pull request Mar 9, 2025
amote-i pushed a commit to amote-i/sglang that referenced this pull request Dec 8, 2025
<!-- Thank you for your contribution! Please follow these guidelines to
enhance your pull request. If anything is unclear, submit your PR and
reach out to maintainers for assistance. Join our Slack community at
https://slack.sglang.ai to discuss further. -->

## Motivation

<!-- Describe the purpose and goals of this pull request. -->

## Modifications

<!-- Detail the changes made in this pull request. -->

## Accuracy Tests

<!-- If this pull request affects model outputs (e.g., changes to the
kernel or model forward code), provide accuracy test results. -->

## Benchmarking and Profiling

<!-- If this pull request impacts inference speed, provide benchmarking
and profiling results. -->

## Checklist

- [ ] Format your code according to the [Format code with
pre-commit](https://docs.sglang.ai/developer_guide/contribution_guide.html#format-code-with-pre-commit).
- [ ] Add unit tests according to the [Run and add unit
tests](https://docs.sglang.ai/developer_guide/contribution_guide.html#run-and-add-unit-tests).
- [ ] Update documentation according to [Write
documentations](https://docs.sglang.ai/developer_guide/contribution_guide.html#write-documentations).
- [ ] Provide accuracy and speed benchmark results according to [Test
the
accuracy](https://docs.sglang.ai/developer_guide/contribution_guide.html#test-the-accuracy)
and [Benchmark the
speed](https://docs.sglang.ai/developer_guide/contribution_guide.html#benchmark-the-speed).
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 20, 2026
The sgl-project#779 spec-accept boot gate exists because a healthy-looking boot can serve
every request with a dead drafter: on this vocab-quantized checkpoint the draft
dequantized the target's int8 rows with an uninitialized scale, accept length
sat at 1.02, and health returned 200 throughout (fixed in 4e1b940).

A gate against that failure is only worth its exit code if the failure can be
reproduced on demand. Asserting that the gate "would" fail is the same class of
claim as the one that hid the original defect for weeks.

So the defect gets a switch. SGLANG_774_NEUTER_VOCAB_COMPANIONS=1 makes
_share_vocab_quant_companions return before it shares anything, which restores
the pre-4e1b940900 behaviour exactly: the draft keeps its own never-loaded
scale. Booting with it armed drives the drafter into the collapsed regime, and
the gate must report FAIL. Booting without it is byte-identical to today -- the
branch is a single env lookup on a path that runs once per worker init.

The hook is permanent rather than a throwaway patch on purpose. The boot runner
refuses a dirty tree, so a temporary patch cannot be booted without polluting
history anyway; and the can-fail property is not a one-off claim to be made once
and cited forever, it is a property that must stay re-provable as the sharing
code changes underneath it.

It logs a warning at WARNING level naming itself, and the gate it exists to
prove is precisely the thing that catches it -- an armed injector cannot pass
acceptance unnoticed.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 21, 2026
…the send

The proxy readiness gate refused to enter the blocking receive until the
upstream's CHAN_DICT `sent` count exceeded this rank's `consumed` count.
`sent` is published strictly after the post, so it says nothing while the
upstream is still inside the send call. That window is empty for an
ordinary isend -- but not for the first point-to-point op on a torch NCCL
process group, which creates the 2-rank communicator lazily, and that
creation is a rendezvous: the isend does not return until the peer enters
the matching receive.

Every boot therefore died on its first real prefill:

  PP0 cannot bump `sent`  until the isend returns
  the isend cannot return until PP1 enters the receive
  PP1 will not enter the receive until `sent` bumps

with the gate itself as one arc of the cycle. Boots instr7 and instr8
(2026-08-21) produced this identically; py-spy caught PP0 in
  isend -> send_tensor_dict -> _pp_send_dict_to_next_stage
33 s into a send it would never leave, while both downstream ranks raised
"sgl-project#789 PROXY READINESS TIMEOUT ... posted 1681, consumed 1681". The message
was not missing. It was being posted to the rank that refused to collect
it. The gate's docstring asserted a stuck sender was "covered identically"
to a sender that scheduled nothing; that assertion was the defect.

Add a second counter, `attempted`, published on the line BEFORE the send
call, and read only by the gate that would otherwise raise. "The upstream
has irrevocably entered a send for me" is as positive a presence signal as
"the upstream posted", and during a rendezvous it is the only one that
exists. The module's ordering rule is unchanged and still governs `sent`;
the new counter is not the phantom-message hazard in reverse, because the
decision to send is already taken and unconditional at the bump, so the
receiver waits on transfer and rendezvous time rather than on peer
scheduling. Drain loops keep reading `sent`: they may take off the wire
only what is provably already on it. Cost is one extra /dev/shm publish per
dict message, measured at 13.6 us.

Tests, all on this branch, measured not assumed:

  test_pp_proxy_readiness_rendezvous_789.py, 5 passed -- three live spawned
  processes, real gloo, the shipped functions, and a wire whose send blocks
  in a real dist.recv until the downstream enters recv, i.e. the NCCL
  lazy-init property in gloo primitives. The wire supplies a hazard rather
  than a guarantee, closing the direction of the trap where a test
  transport is more careful than production. Can-fail: neutering only
  PhaseFlipCounters.attempted, in the child, reproduces sgl-project#789 on both
  downstream ranks -- the metal specimen.

  test/registered/unit/managers full directory: 47 failed, 2753 passed.
  HEAD (96df16d) measured with the identical command: 47 failed, 2748
  passed. The failing set is identical in both directions; the 5 added
  passes are the new file. The five holder/stub repairs in existing tests
  are interface drift only -- no assertion touched.

  Boot instr9, PP=3 on this rig with the reference instrumented
  configuration: health reached, 15 min 12 s uptime, one request served,
  the 8-request burst 8/8 in 3.1 s with per-request spec_accept_length mean
  2.38 (sgl-project#779 gate PASS), 6 further back-to-back bursts under sustained
  load, 48 more requests, 0 failures, still serving afterwards. Zero sgl-project#789
  raises, zero wedge markers, zero scheduler exceptions, and sgl-project#788 admission
  byte-identical on all three ranks including the 7-request batch. The
  first surviving boot of this series.

Corridor is NOT passed on that boot and is not addressed here: NVML free
sat flat at 5229/6612/5475 MiB per card under load against the 819-1229 MiB
band, i.e. 4.4-5.0 GiB per card unclaimed against grants of
31800/18800/19800 MiB. That is a planner capacity item, and it is the first
boot in this series on which the number means anything, because no earlier
one ever served a request.
YAMY1234 added a commit to YAMY1234/sglang that referenced this pull request Sep 26, 2026
…L_BLOCK_PAD=1): length-bucketed block graphs for other singleton lengths (<=8K by 512, <=32K by 4096; padding rows g=0/beta=0 leave the state unchanged), receipts and bytewise check mode; test

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
YAMY1234 added a commit to YAMY1234/sglang that referenced this pull request Sep 26, 2026
…ormed in fp32 and cast (Triton interpreter has no bf16 constants); non-token tensors plain copies

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
YAMY1234 added a commit to YAMY1234/sglang that referenced this pull request Sep 26, 2026
- join once per stream per side-stream record; per-layer pool methods no longer
  wrapped (they run inside a target forward that init_forward_metadata joined)
- cached env lookups and row indices; commit-graph key/size from the first layer
- capture the singleton densify graphs at the first extend forward (server warmup)
  instead of the first prefix-hit prefill

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
YAMY1234 added a commit to YAMY1234/sglang that referenced this pull request Sep 26, 2026
…evice

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
YAMY1234 added a commit to YAMY1234/sglang that referenced this pull request Sep 26, 2026
…s from the real last row in padded graphs (state-kernel variant with a real-length device scalar written by the bind kernel; chunks past it leave the state untouched); default kernel unchanged

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
YAMY1234 added a commit to YAMY1234/sglang that referenced this pull request Sep 26, 2026
…IT_DEFER=1): the final commit group (same code) starts at the pool's first reader instead of beside the last layers before sampling; readers already join via _pending_commit

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
YAMY1234 added a commit to YAMY1234/sglang that referenced this pull request Sep 26, 2026
…6 copy of b_v under its own name inside the runtime branch (Triton type check)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
YAMY1234 added a commit to YAMY1234/sglang that referenced this pull request Sep 27, 2026
… to 8192 tokens (longer chunks are GPU-bound and eager); real-end state kernel loops over the real chunks only (no runtime branch in the chunk loop)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
YAMY1234 added a commit to YAMY1234/sglang that referenced this pull request Sep 27, 2026
…IT_SPLIT=1): unfinished-prompt commit split -- exact ring copy and slot flags on the main stream, the captured commit graph (factors, tracked states, prefix publish) on the side stream in parallel with the next chunk; extend metadata/plan do not wait for it, factor readers and the next commit do

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
YAMY1234 added a commit to YAMY1234/sglang that referenced this pull request Sep 27, 2026
… receipt (first use and every 50th), logging only

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
YAMY1234 added a commit to YAMY1234/sglang that referenced this pull request Sep 27, 2026
…NG_GDN_OPUS_COMMIT_QUEUE=1): queue unfinished-prompt chunk-end commits for the first other pool reader

At a mid-prompt chunk end only the exact ring copy and slot flags (what the next chunk reads) are written on the
main stream; the unchanged commit graph (factors, tracked prefix copy, slot copy) is queued and runs, in order, at the
pool's first other reader -- decode metadata, slot copies, or a plan that is not the queued prompts' own continuation
-- i.e. after a lone prompt's first token. Same kernels and inputs; only when the commit graphs run moves. Receipts
(first queued commit, every 50th flush) are written off the forward thread.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
YAMY1234 added a commit to YAMY1234/sglang that referenced this pull request Sep 27, 2026
…idle (SGLANG_GDN_OPUS_IDLE_DRAIN, sgl-project#779)

A one-output-token prompt has no second decode step, so with OPUS_COMMIT_TRACK_ONLY its queued tracked-copy jobs
drained in the next request's first chunk (~5 ms host per chunk end; 895603 trace19 second cold turn +25 ms). The
scheduler's fully-idle hook now runs the same drain in the forward stream's context. Default off.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant