Repository navigation
Update version to 0.1.17 - #515
Merged
Merged
Conversation
timethink
pushed a commit
to timethink/sglang
that referenced
this pull request
Mar 9, 2025
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 5, 2026
…why no mm guard ships Deterministic inference is legal under breakable (it is in the tc_piecewise rule list at server_args.py:8455 but not the breakable one at :8487) and sets enforce_disable_flashinfer_allreduce_fusion=True (:11515), pinning the collective reduction order. That makes the ED/GD pair a direct test of the NOTE_515 §4 mechanism claim rather than just an extra data point: agreement under determinism confirms collective amplification and puts a byte-strict gate within reach; continued divergence places the epsilon before the collectives and makes the distribution-level gate the ceiling. Also records the untested gap: can_run_graph has no per-batch multimodal guard, so force-enabling the backend on this model would route image batches through the captured path too. contains_mm_inputs() already exists as the predicate, but with no measurable throughput to win there is no case for shipping an unexercised guard.
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 5, 2026
…nflation, and invalidate B2's oracle Went back to the primary artifacts instead of the summaries, and they are more decisive than my own GPU run. /spinning/gpu-battery-results/2026-08-03_452_arms/B2_RESULTS.md:90 states that the treatment fold was the DECODE backend (disabled -> full) and that prefill=PhaseConfig(backend='disabled') was byte-identical in both arms, on Qwen3.6-35B-A3B-AWQ-4bit. No sgl-project#452 arm ever captured a prefill graph. The prefill backend was never on trial, which is the conflation in one line. That window's oracle was also invalid, and its own §9 says so: the greedy-text half of B2 is recorded as UNANSWERABLE because three identical greedy requests produced 3/3 distinct hashes in BOTH arms. Recomputing from the raw greedy_* files: eager_1 vs eager_2 diverges at char 568 and eager_1 vs eager_3 at char 582, while eager_1 vs graphs_1 diverges at char 748 -- the cross-arm difference is LATER than the within-arm noise. One long prompt at 192 new tokens sits well above the ~109-token GDN prefill reproducibility ceiling, which is why the NOTE_515 harness keeps every probe underneath it. The 'systematic from character 5' headline therefore traces to the 2026-08-02 window, which carried no A-vs-A floor at all -- decode_eager1.json and decode_graphs.json, with no eager repeat. §9 prescribed --enable-deterministic-inference plus a pinned random_seed as the fix; window3_boot_floor.sh already runs exactly that, which is independent support for its design.
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 5, 2026
…d, plus the power-cap caveat User directive: the live hypothesis is that a captured prefill only pays IN COMBINATION WITH barlink. The mechanism is falsifiable and specific -- NCCL's collectives are host-driven, so the launch train a graph would tighten is not the part that costs, while barlink's device-side collectives can sit inside the captured graph and put the 68-75% collective share of the prefill window (sgl-project#252) within reach. If that holds, the flat NCCL result is a property of the transport, not of prefill graphs. window3_boot_floor.sh now stages STAGE=nccl|barlink|all over prefill eager prefill graphs NCCL E1, E2 G barlink BE1, BE2 BG Each transport carries its OWN eager-vs-eager floor, a delta counts only against the floor of its own row, and the reported swing counts only if it clears both. Points are the long-prompt 1900 and the bs>1 concurrency mix (256 tokens, 4 in flight), the latter scored on aggregate_tok_s because per-request rates understate a concurrent regime. The barlink row gets its own content gate too -- a transport change is exactly the kind of thing that moves reduction order. Both preconditions are enforced by the script rather than assumed: the barlink stage refuses to run unless b001d10 (the sgl-project#583 tripped-spin-kernel / CUDA context fix, merged here) is an ancestor of HEAD, and unless the operator passes BARLINK_VERDICT=confirmed from the live-load repro window. Absent arms report 'hypothesis UNTESTED at this point' instead of being guessed. Reporting was executed against the window-2 artifacts before commit, exercising the missing-floor, missing-transport and missing-point skip paths. Power caps were reduced on this rig (3080 320->200 W, 5090 525->400 W; verified power.limit 200/400/200 against default 320/575/320). Same-boot A/B arms share the caps and are unaffected, but comparisons against pre-change archive numbers -- the sgl-project#320 Messbuendel tables in particular -- are confounded. The script now prints the measured caps in its report header and NOTE_515 carries the caveat.
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 5, 2026
…hput claim is withdrawn NCCL half of the 2x2, on GPU 2026-08-05. BARLINK_VERDICT=inconclusive from barlink-583 (fix carried thousands of collectives but deadlines were never approached, abort path unexercised), so the barlink row stays UNTESTED and the script's precondition refused it, as designed. Artifacts in /spinning/gpu-battery-results/2026-08-05_prefill_graphs_w3/. CONTENT -- the control windows 1 and 2 both lacked, now taken, and it holds: E1 vs E2 (boot-to-boot floor) PASS, 8/8 byte-identical E1 vs G FAIL, 4/8 text + 3/8 logprob-only E2 vs G FAIL, same prompts, same char offsets Two independent eager boots are byte-identical and the graph boot diverges from BOTH at identical positions. The §4 divergence is real and attributable to the prefill graph backend, not boot noise. NOTE_452 §2 experiment 2 is closed. THROUGHPUT -- WITHDRAWN. Rebuilt to the sustained standard first (warmup discarded, ~5 s measured window of back-to-back draws, aggregate tok/s, SM clock and P-state sampled during the window), which produced what looked like a clean negative: -4.3/-1.8/-3.0% at 256/900/1900 against eager floors of 0.6/0.8/0.4%. The clock telemetry the directive required then falsified it. The 3080 median SM clock decays monotonically with ARM ORDER -- E1 1886, E2 1838 (-2.6%), G 1774 (-6.0%) at 256, same shape at every point -- a thermal-soak signature on 200 W-capped cards, aliased onto the treatment because G ran last every time. The clock deficit is larger than every throughput delta measured, and G's throughput deficit (-4.0/-2.1/-3.2%) is consistently SMALLER than its clock deficit (-6.0/-7.1/-7.2%), so per-MHz the graph arm is not behind at all. Block arm order on a power-capped card is not a recoverable design; the follow-up must interleave (E,G,E,G,E,G). The 256c4 point is separately unusable -- its eager floor is +9.1%. The content result is unaffected: byte-identity does not depend on clock rate, which is why it survived the confound that took the throughput number down. ED/GD did not run -- both determinism arms OOM'd during capture (boot_ED.log: avail mem=0.05 GB on TP1, then "multimem all-gather disabled (CUDA driver error: out of memory)"). --enable-deterministic-inference shifts the capture footprint past the 5500,3800,3800 reserve. Needs a larger 3080 reserve, not a code change. Serving restored pgid 189192, health 200 after 30s, corridor 1688/4069/2558 MiB, smoke content-sane and identical to the eager arm.
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 5, 2026
…e derived from the ledger Desk work, no cards touched (barlink-583 holds them). MEASUREMENT RULE, adopted and applied. At a fixed power limit -- and the caps were identical across every run -- a lower clock usually means HIGHER load, not a disadvantage: a power-limited card downclocks precisely when it does more work per cycle, so low clock with high power is a busy card while low clock with low power is an idle one. My earlier "per-MHz the graph arm is not behind" inference is therefore invalid and is RETRACTED in NOTE_515 §5a. The withdrawal of the throughput claim stands, but only because the numbers are confounded -- not because of any per-MHz argument. Clock is never a validity criterion and never a normaliser. HARNESS (tests/prefill_graphs/): * prefill_perf.py rescored on ms per FIXED unit of work -- an identical prompt set, same count, same order, from a fixed seed, verified equal across arms in the smoke (8 prefills / 15200 tokens both arms). Clock AND power.draw are sampled together over the measured span and reported as diagnostic annotation only. * window4_interleaved.sh runs E,G,E,G,E,G. Interleaving's only job here is removing slow drift; it is not a clock argument. Two workload points only (1900 single-stream, 256c4 concurrent), each sized into the 5-20 s band where the spread is already saturated, content gate on pair 1 only. Time plan is ~9-10 min, of which ~5.5 min is irreducible boot cost and ~2 min is actual measurement. * report_interleaved.py pairs G_i against E_i (adjacent in time), combines the pair deltas, and validates on the A-vs-A floor alone. Clock/power annotate load character and can no longer reject a point. * mock_server.py added so all of this is exercisable without a GPU. The 256c4 +9.1% "floor" is diagnosed and it was MY harness, not the server: E1 and E2 had done byte-identical work (32 draws, 8523 tokens) and differed only by 0.471 s of drain tail inside a tokens-over-wall denominator. On fixed work both give exactly 1704.6 tok/s -- a 0.0% floor. Point kept, denominator fixed. Smoke-tested end-to-end against the mock with no cards: fixed-work probe, identical-work equality check, concurrent point, content gate PASS and FAIL paths, and every report verdict branch (REPORTABLE / INSIDE FLOOR / INVALID / NO FLOOR) driven by synthetic replicates. ED/GD RESERVE, derived not guessed. --enable-deterministic-inference rewrites the flashinfer workspace from the 384 MiB default (environ.py:1270) to 2048 MiB (flashinfer_backend.py:1007); the served arch Qwen3_5ForConditionalGeneration is NOT on the 512 MiB Qwen list at flashinfer_backend.py:987, so the baseline really is 384 MiB. Delta = 1664 MiB per rank process, and since TERM_ATTN_WORKSPACE is not budget-funded (engine.py:105) it comes out of reserve headroom. The failed boot confirms the derivation: E1 consumed 2.49 -> 1.91 GB between pool end and capture begin, ED 2.48 -> 0.30 GB, an extra 1649 MiB against 1664 derived, with identical KV pools. => RESERVE="7164,5464,5464" (5500+1664, 3800+1664, 3800+1664). No code change. WHAT THE LEDGER CANNOT EXPRESS YET, stated instead of padded: engine.py:293 reads SGLANG_FLASHINFER_WORKSPACE_SIZE at ledger-build time, but the backend rewrites that variable afterwards, so TERM_ATTN_WORKSPACE does not move when enable_deterministic_inference moves -- exactly the property the ledger's own hermetic test asserts for a MODELED term. Missing piece is a resolver replicating the backend rule (2048 if deterministic; 512 for the listed Qwen archs; else the env default) plus enable_deterministic_inference and hf_config.architectures in the term's declared inputs. Ticket for the follow-up window: docs/dev/TICKET_prefill_graphs_window4.md (boot commands, arms, sizing, acceptances, time plan, power caps in header).
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 5, 2026
… reserve was stale Checked the window-4 ticket against the CURRENT production boot script. Two defects found, both silent, both fixed. 1. TRANSPORT LEAK. Production now exports SGLANG_BARLINK=1 (user decision 2026-08-05: production runs barlink, usage is the soak). window4_interleaved.sh never mentioned SGLANG_BARLINK, so it would have inherited the transport from whatever shell launched it. Launched from a production-flavoured shell it would have compared prefill-graphs-on-barlink against window 3's prefill-graphs-on-NCCL and reported the difference as a prefill-graph result. DECISION, stated explicitly: the window-4 E/G arms run NCCL. Holding the transport at window 3's value is what isolates the prefill-graph variable and keeps the two windows comparable. TRANSPORT is now set explicitly (nccl -> unset SGLANG_BARLINK; barlink -> export SGLANG_BARLINK=1), any other value is refused, the transport is stamped into EVERY artifact JSON plus transport.txt, printed in the report header, and report_interleaved.py ABORTS when arms disagree. Proven by fixture: a mixed-transport artifact set exits 1 with "arms used DIFFERENT transports -- not comparable". CONSEQUENCE, recorded rather than buried: production no longer runs NCCL, so an NCCL-only result is no longer by itself a production rollout argument. The decision-relevant cell is barlink x prefill-graphs, which still lacks its own eager floor. NOTE_515 section 7 is rescoped accordingly. 2. STALE RESERVE. Production raised the 3080 term 3800 -> 4200 on 2026-08-05: its own demand model derives 4160 MiB (activation + graph capture + GDN prefill scratch) and warned 3800 was 360 MiB short with the 96-slot mamba pool. Booting the arms at 3800 would have sized a different KV pool than production runs. Default is now 5500,4200,4200. The determinism vector rebases with it: 4200+1664 = 5864, not 5464. So RESERVE="7164,5864,5864". The two corrections are independent -- 4200 fixes a previously-diagnosed activation/capture/scratch shortfall, +1664 is the deterministic flashinfer workspace delta (384 -> 2048 MiB, flashinfer_backend.py:1007) on top of it. Every arm's transport and reserve now appear in the report header, and no artifact can be read without knowing which transport produced it.
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 5, 2026
…IRMED, with a regime split
TRANSPORT=barlink (production's transport), interleaved E,G,E,G,E,G, ms per
fixed unit of work, own barlink eager floor. Artifacts in
/spinning/gpu-battery-results/2026-08-05_prefill_graphs_w4{,det}/.
point eager graphs paired delta floor verdict
1900 single-stream 1196.3 ms 1320.6 ms +10.25% slower 0.12% REPORTABLE
256 x4 concurrent 230.1 ms 219.0 ms -4.62% faster 1.99% REPORTABLE
All three pairs agree in sign at both points and both deltas clear their own
A-vs-A floor by a wide margin (85x and 2.3x).
The user's hypothesis holds, and only in the regime it predicted: a captured
prefill pays where the work is launch-train bound (many small prefills, several
in flight) and loses where it is GEMM bound. With barlink's device-side
collectives inside the captured graph the concurrent short-prompt point turns a
real -4.6%, while the single long prefill has no launch overhead to recover and
pays the padding and capture cost instead. The +10.25% at 1900 is larger than
bucket padding alone explains (~3.7% for a 1974-token prefill in the 2048
bucket), so something else in the captured long-prefill path costs on top; not
localised here.
Consequence for the recommendation, now rescoped: the answer is not on/off but
workload-dependent, and a blanket flag is the wrong shape. can_run_graph is
already a per-batch decision (prefill_cuda_graph_runner.py:615), which is where
a regime gate would live. Enabling globally would trade 10% on long prefills
for 4.6% on short concurrent ones, and production's traffic mix decides whether
that nets out -- unmeasured.
DETERMINISM ARMS booted first try at the derived RESERVE 7164,5864,5864,
validating the §5a ledger derivation end to end -- right vector, no padding, no
retry. Same directional split (+7.44% / -6.31%) but REPS=1, so no floor and
therefore NO VERDICT on those two numbers, as the reporter states. Determinism
did NOT close the content divergence: 1/8 with it, 1/8 without.
CONTENT under barlink is 1/8 text (char 124) against 4/8 under NCCL, but the
gate ran on pair 1 only, so there is NO barlink eager-vs-eager content floor.
The 1/8 is not yet distinguishable from barlink boot noise, and is recorded
that way rather than as an improvement.
SOAK: 8 barlink arms booted and completed with no fault -- a clean data point
for sgl-project#583.
Serving restored pgid 231310, health 200 after 40s, corridor 1708/4131/2578 MiB,
smoke content-sane.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.