Skip to content

Update version to 0.1.17 - #515

Merged
merrymercy merged 3 commits into
mainfrom
update-version
Jun 8, 2024
Merged

merrymercy merged 3 commits into
mainfrom
update-version

Conversation

@merrymercy

Copy link
Copy Markdown
Collaborator

No description provided.

@merrymercy
merrymercy merged commit e8a2327 into main Jun 8, 2024
@merrymercy
merrymercy deleted the update-version branch June 8, 2024 02:49
timethink pushed a commit to timethink/sglang that referenced this pull request Mar 9, 2025
efschu added a commit to efschu/htsglang that referenced this pull request Aug 5, 2026
…why no mm guard ships

Deterministic inference is legal under breakable (it is in the tc_piecewise
rule list at server_args.py:8455 but not the breakable one at :8487) and sets
enforce_disable_flashinfer_allreduce_fusion=True (:11515), pinning the
collective reduction order. That makes the ED/GD pair a direct test of the
NOTE_515 §4 mechanism claim rather than just an extra data point: agreement
under determinism confirms collective amplification and puts a byte-strict
gate within reach; continued divergence places the epsilon before the
collectives and makes the distribution-level gate the ceiling.

Also records the untested gap: can_run_graph has no per-batch multimodal
guard, so force-enabling the backend on this model would route image batches
through the captured path too. contains_mm_inputs() already exists as the
predicate, but with no measurable throughput to win there is no case for
shipping an unexercised guard.
efschu added a commit to efschu/htsglang that referenced this pull request Aug 5, 2026
…nflation, and invalidate B2's oracle

Went back to the primary artifacts instead of the summaries, and they are more
decisive than my own GPU run.

/spinning/gpu-battery-results/2026-08-03_452_arms/B2_RESULTS.md:90 states that
the treatment fold was the DECODE backend (disabled -> full) and that
prefill=PhaseConfig(backend='disabled') was byte-identical in both arms, on
Qwen3.6-35B-A3B-AWQ-4bit. No sgl-project#452 arm ever captured a prefill graph. The
prefill backend was never on trial, which is the conflation in one line.

That window's oracle was also invalid, and its own §9 says so: the greedy-text
half of B2 is recorded as UNANSWERABLE because three identical greedy requests
produced 3/3 distinct hashes in BOTH arms. Recomputing from the raw greedy_*
files: eager_1 vs eager_2 diverges at char 568 and eager_1 vs eager_3 at char
582, while eager_1 vs graphs_1 diverges at char 748 -- the cross-arm difference
is LATER than the within-arm noise. One long prompt at 192 new tokens sits well
above the ~109-token GDN prefill reproducibility ceiling, which is why the
NOTE_515 harness keeps every probe underneath it.

The 'systematic from character 5' headline therefore traces to the 2026-08-02
window, which carried no A-vs-A floor at all -- decode_eager1.json and
decode_graphs.json, with no eager repeat.

§9 prescribed --enable-deterministic-inference plus a pinned random_seed as the
fix; window3_boot_floor.sh already runs exactly that, which is independent
support for its design.
efschu added a commit to efschu/htsglang that referenced this pull request Aug 5, 2026
…d, plus the power-cap caveat

User directive: the live hypothesis is that a captured prefill only pays IN
COMBINATION WITH barlink. The mechanism is falsifiable and specific -- NCCL's
collectives are host-driven, so the launch train a graph would tighten is not
the part that costs, while barlink's device-side collectives can sit inside the
captured graph and put the 68-75% collective share of the prefill window (sgl-project#252)
within reach. If that holds, the flat NCCL result is a property of the
transport, not of prefill graphs.

window3_boot_floor.sh now stages STAGE=nccl|barlink|all over

               prefill eager   prefill graphs
  NCCL         E1, E2          G
  barlink      BE1, BE2        BG

Each transport carries its OWN eager-vs-eager floor, a delta counts only
against the floor of its own row, and the reported swing counts only if it
clears both. Points are the long-prompt 1900 and the bs>1 concurrency mix
(256 tokens, 4 in flight), the latter scored on aggregate_tok_s because
per-request rates understate a concurrent regime. The barlink row gets its own
content gate too -- a transport change is exactly the kind of thing that moves
reduction order.

Both preconditions are enforced by the script rather than assumed: the barlink
stage refuses to run unless b001d10 (the sgl-project#583 tripped-spin-kernel / CUDA
context fix, merged here) is an ancestor of HEAD, and unless the operator
passes BARLINK_VERDICT=confirmed from the live-load repro window. Absent arms
report 'hypothesis UNTESTED at this point' instead of being guessed.

Reporting was executed against the window-2 artifacts before commit, exercising
the missing-floor, missing-transport and missing-point skip paths.

Power caps were reduced on this rig (3080 320->200 W, 5090 525->400 W; verified
power.limit 200/400/200 against default 320/575/320). Same-boot A/B arms share
the caps and are unaffected, but comparisons against pre-change archive numbers
-- the sgl-project#320 Messbuendel tables in particular -- are confounded. The script now
prints the measured caps in its report header and NOTE_515 carries the caveat.
efschu added a commit to efschu/htsglang that referenced this pull request Aug 5, 2026
…hput claim is withdrawn

NCCL half of the 2x2, on GPU 2026-08-05. BARLINK_VERDICT=inconclusive from
barlink-583 (fix carried thousands of collectives but deadlines were never
approached, abort path unexercised), so the barlink row stays UNTESTED and the
script's precondition refused it, as designed. Artifacts in
/spinning/gpu-battery-results/2026-08-05_prefill_graphs_w3/.

CONTENT -- the control windows 1 and 2 both lacked, now taken, and it holds:
  E1 vs E2 (boot-to-boot floor)  PASS, 8/8 byte-identical
  E1 vs G                        FAIL, 4/8 text + 3/8 logprob-only
  E2 vs G                        FAIL, same prompts, same char offsets
Two independent eager boots are byte-identical and the graph boot diverges from
BOTH at identical positions. The §4 divergence is real and attributable to the
prefill graph backend, not boot noise. NOTE_452 §2 experiment 2 is closed.

THROUGHPUT -- WITHDRAWN. Rebuilt to the sustained standard first (warmup
discarded, ~5 s measured window of back-to-back draws, aggregate tok/s, SM
clock and P-state sampled during the window), which produced what looked like a
clean negative: -4.3/-1.8/-3.0% at 256/900/1900 against eager floors of
0.6/0.8/0.4%. The clock telemetry the directive required then falsified it. The
3080 median SM clock decays monotonically with ARM ORDER -- E1 1886, E2 1838
(-2.6%), G 1774 (-6.0%) at 256, same shape at every point -- a thermal-soak
signature on 200 W-capped cards, aliased onto the treatment because G ran last
every time. The clock deficit is larger than every throughput delta measured,
and G's throughput deficit (-4.0/-2.1/-3.2%) is consistently SMALLER than its
clock deficit (-6.0/-7.1/-7.2%), so per-MHz the graph arm is not behind at all.
Block arm order on a power-capped card is not a recoverable design; the
follow-up must interleave (E,G,E,G,E,G). The 256c4 point is separately unusable
-- its eager floor is +9.1%.

The content result is unaffected: byte-identity does not depend on clock rate,
which is why it survived the confound that took the throughput number down.

ED/GD did not run -- both determinism arms OOM'd during capture (boot_ED.log:
avail mem=0.05 GB on TP1, then "multimem all-gather disabled (CUDA driver
error: out of memory)"). --enable-deterministic-inference shifts the capture
footprint past the 5500,3800,3800 reserve. Needs a larger 3080 reserve, not a
code change.

Serving restored pgid 189192, health 200 after 30s, corridor 1688/4069/2558 MiB,
smoke content-sane and identical to the eager arm.
efschu added a commit to efschu/htsglang that referenced this pull request Aug 5, 2026
…e derived from the ledger

Desk work, no cards touched (barlink-583 holds them).

MEASUREMENT RULE, adopted and applied. At a fixed power limit -- and the caps
were identical across every run -- a lower clock usually means HIGHER load, not
a disadvantage: a power-limited card downclocks precisely when it does more
work per cycle, so low clock with high power is a busy card while low clock
with low power is an idle one. My earlier "per-MHz the graph arm is not behind"
inference is therefore invalid and is RETRACTED in NOTE_515 §5a. The withdrawal
of the throughput claim stands, but only because the numbers are confounded --
not because of any per-MHz argument. Clock is never a validity criterion and
never a normaliser.

HARNESS (tests/prefill_graphs/):
* prefill_perf.py rescored on ms per FIXED unit of work -- an identical prompt
  set, same count, same order, from a fixed seed, verified equal across arms in
  the smoke (8 prefills / 15200 tokens both arms). Clock AND power.draw are
  sampled together over the measured span and reported as diagnostic
  annotation only.
* window4_interleaved.sh runs E,G,E,G,E,G. Interleaving's only job here is
  removing slow drift; it is not a clock argument. Two workload points only
  (1900 single-stream, 256c4 concurrent), each sized into the 5-20 s band where
  the spread is already saturated, content gate on pair 1 only. Time plan is
  ~9-10 min, of which ~5.5 min is irreducible boot cost and ~2 min is actual
  measurement.
* report_interleaved.py pairs G_i against E_i (adjacent in time), combines the
  pair deltas, and validates on the A-vs-A floor alone. Clock/power annotate
  load character and can no longer reject a point.
* mock_server.py added so all of this is exercisable without a GPU.

The 256c4 +9.1% "floor" is diagnosed and it was MY harness, not the server: E1
and E2 had done byte-identical work (32 draws, 8523 tokens) and differed only
by 0.471 s of drain tail inside a tokens-over-wall denominator. On fixed work
both give exactly 1704.6 tok/s -- a 0.0% floor. Point kept, denominator fixed.

Smoke-tested end-to-end against the mock with no cards: fixed-work probe,
identical-work equality check, concurrent point, content gate PASS and FAIL
paths, and every report verdict branch (REPORTABLE / INSIDE FLOOR / INVALID /
NO FLOOR) driven by synthetic replicates.

ED/GD RESERVE, derived not guessed. --enable-deterministic-inference rewrites
the flashinfer workspace from the 384 MiB default (environ.py:1270) to 2048 MiB
(flashinfer_backend.py:1007); the served arch Qwen3_5ForConditionalGeneration
is NOT on the 512 MiB Qwen list at flashinfer_backend.py:987, so the baseline
really is 384 MiB. Delta = 1664 MiB per rank process, and since
TERM_ATTN_WORKSPACE is not budget-funded (engine.py:105) it comes out of
reserve headroom. The failed boot confirms the derivation: E1 consumed
2.49 -> 1.91 GB between pool end and capture begin, ED 2.48 -> 0.30 GB, an
extra 1649 MiB against 1664 derived, with identical KV pools.
  => RESERVE="7164,5464,5464"  (5500+1664, 3800+1664, 3800+1664). No code change.

WHAT THE LEDGER CANNOT EXPRESS YET, stated instead of padded: engine.py:293
reads SGLANG_FLASHINFER_WORKSPACE_SIZE at ledger-build time, but the backend
rewrites that variable afterwards, so TERM_ATTN_WORKSPACE does not move when
enable_deterministic_inference moves -- exactly the property the ledger's own
hermetic test asserts for a MODELED term. Missing piece is a resolver
replicating the backend rule (2048 if deterministic; 512 for the listed Qwen
archs; else the env default) plus enable_deterministic_inference and
hf_config.architectures in the term's declared inputs.

Ticket for the follow-up window: docs/dev/TICKET_prefill_graphs_window4.md
(boot commands, arms, sizing, acceptances, time plan, power caps in header).
efschu added a commit to efschu/htsglang that referenced this pull request Aug 5, 2026
… reserve was stale

Checked the window-4 ticket against the CURRENT production boot script. Two
defects found, both silent, both fixed.

1. TRANSPORT LEAK. Production now exports SGLANG_BARLINK=1 (user decision
   2026-08-05: production runs barlink, usage is the soak). window4_interleaved.sh
   never mentioned SGLANG_BARLINK, so it would have inherited the transport from
   whatever shell launched it. Launched from a production-flavoured shell it
   would have compared prefill-graphs-on-barlink against window 3's
   prefill-graphs-on-NCCL and reported the difference as a prefill-graph result.

   DECISION, stated explicitly: the window-4 E/G arms run NCCL. Holding the
   transport at window 3's value is what isolates the prefill-graph variable and
   keeps the two windows comparable. TRANSPORT is now set explicitly
   (nccl -> unset SGLANG_BARLINK; barlink -> export SGLANG_BARLINK=1), any other
   value is refused, the transport is stamped into EVERY artifact JSON plus
   transport.txt, printed in the report header, and report_interleaved.py ABORTS
   when arms disagree. Proven by fixture: a mixed-transport artifact set exits 1
   with "arms used DIFFERENT transports -- not comparable".

   CONSEQUENCE, recorded rather than buried: production no longer runs NCCL, so
   an NCCL-only result is no longer by itself a production rollout argument. The
   decision-relevant cell is barlink x prefill-graphs, which still lacks its own
   eager floor. NOTE_515 section 7 is rescoped accordingly.

2. STALE RESERVE. Production raised the 3080 term 3800 -> 4200 on 2026-08-05:
   its own demand model derives 4160 MiB (activation + graph capture + GDN
   prefill scratch) and warned 3800 was 360 MiB short with the 96-slot mamba
   pool. Booting the arms at 3800 would have sized a different KV pool than
   production runs. Default is now 5500,4200,4200.

   The determinism vector rebases with it: 4200+1664 = 5864, not 5464. So
   RESERVE="7164,5864,5864". The two corrections are independent -- 4200 fixes a
   previously-diagnosed activation/capture/scratch shortfall, +1664 is the
   deterministic flashinfer workspace delta (384 -> 2048 MiB,
   flashinfer_backend.py:1007) on top of it.

Every arm's transport and reserve now appear in the report header, and no
artifact can be read without knowing which transport produced it.
efschu added a commit to efschu/htsglang that referenced this pull request Aug 5, 2026
…IRMED, with a regime split

TRANSPORT=barlink (production's transport), interleaved E,G,E,G,E,G, ms per
fixed unit of work, own barlink eager floor. Artifacts in
/spinning/gpu-battery-results/2026-08-05_prefill_graphs_w4{,det}/.

  point                 eager      graphs     paired delta   floor    verdict
  1900 single-stream    1196.3 ms  1320.6 ms  +10.25% slower  0.12%   REPORTABLE
  256 x4 concurrent      230.1 ms   219.0 ms   -4.62% faster  1.99%   REPORTABLE

All three pairs agree in sign at both points and both deltas clear their own
A-vs-A floor by a wide margin (85x and 2.3x).

The user's hypothesis holds, and only in the regime it predicted: a captured
prefill pays where the work is launch-train bound (many small prefills, several
in flight) and loses where it is GEMM bound. With barlink's device-side
collectives inside the captured graph the concurrent short-prompt point turns a
real -4.6%, while the single long prefill has no launch overhead to recover and
pays the padding and capture cost instead. The +10.25% at 1900 is larger than
bucket padding alone explains (~3.7% for a 1974-token prefill in the 2048
bucket), so something else in the captured long-prefill path costs on top; not
localised here.

Consequence for the recommendation, now rescoped: the answer is not on/off but
workload-dependent, and a blanket flag is the wrong shape. can_run_graph is
already a per-batch decision (prefill_cuda_graph_runner.py:615), which is where
a regime gate would live. Enabling globally would trade 10% on long prefills
for 4.6% on short concurrent ones, and production's traffic mix decides whether
that nets out -- unmeasured.

DETERMINISM ARMS booted first try at the derived RESERVE 7164,5864,5864,
validating the §5a ledger derivation end to end -- right vector, no padding, no
retry. Same directional split (+7.44% / -6.31%) but REPS=1, so no floor and
therefore NO VERDICT on those two numbers, as the reporter states. Determinism
did NOT close the content divergence: 1/8 with it, 1/8 without.

CONTENT under barlink is 1/8 text (char 124) against 4/8 under NCCL, but the
gate ran on pair 1 only, so there is NO barlink eager-vs-eager content floor.
The 1/8 is not yet distinguishable from barlink boot noise, and is recorded
that way rather than as an improvement.

SOAK: 8 barlink arms booted and completed with no fault -- a clean data point
for sgl-project#583.

Serving restored pgid 231310, health 200 after 40s, corridor 1708/4131/2578 MiB,
smoke content-sane.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant