Repository navigation
Fix typos in infer_batch.py - #354
Merged
Merged
Conversation
timethink
pushed a commit
to timethink/sglang
that referenced
this pull request
Mar 9, 2025
shiyu7
pushed a commit
to shiyu7/sglang
that referenced
this pull request
Dec 26, 2025
Garrybest
pushed a commit
to Garrybest/sglang
that referenced
this pull request
Jan 9, 2026
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Jul 31, 2026
…plit agent-354 held all three cards through this window (0/1/2 at 19787/30031/18685 MiB, phase-optimal sgl-project#354 runs), so the two card-side proofs did not run. Package them as one command a later card window runs turnkey: CARD=0 bash scripts/dev/run_355_gpu.sh Bounded (600s), gpu-arb holder with heartbeat, feasibility-arith preflight (refuses a card with >900 MiB used, corridor >=400 MiB free), signals only its own PIDs, self-releases on every exit path. It runs the microbench (ns/call off-vs-on, A/B interleaved with an A-vs-A noise floor, Qwen3.6-27B decode row), the 4-case out-of-range falsifier, and the sgl-project#352 regression on the shared helper. Interpreter smoke test (TRITON_INTERPRET=1) confirms the kernel runs with the new bound parameter and writes in-range correctly; device-assert semantics need a real card, hence the follow-up. Validation doc records the honest split: desk-complete, GPU falsifier/microbench pending one card window.
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Jul 31, 2026
…ness + validation) (sgl-project#354) # Conflicts: # docs/dev/INTEGRATION_R3_VALIDATION.md
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 1, 2026
…sgl-project#366) sgl-project#369 released the bar1 graph gate, so sgl-project#366 could finally measure. It still could not measure with CUDA graphs, and the reason is the result that matters most here. WHY EAGER. bar1 + CUDA graphs + NEXTN does not cold-capture in a usable time on the 27B FP8 vehicle. One graph boot ran >35 min without finishing capture, all three cards pinned at 100% util the whole time -- forward progress, not a wedge. The log says why, dozens of times: "Using default W8A8 Block FP8 kernel config ... Config file not found ... device_name=NVIDIA_GeForce_RTX_5090, dtype=fp8_w8a8, block_shape=[128,128]". The rig ships no tuned W8A8 block-fp8 configs for the 5090, so every GEMM shape falls into a cold triton autotune with ~3 min between shapes. That is the sgl-project#255/sgl-project#368 tuned-config gap, and warming those configs once would make every future bar1 graph boot fast -- a concrete, cheap follow-up. Running EAGER skips capture, costs ~34 s per boot, and puts both transports in the SAME regime, so the transport question is answered honestly. The table therefore does NOT quote sgl-project#354's graph-mode NCCL baseline; it measures NCCL itself, this window, eager. s17_bar1_eager_arm.sh boots one arm from the htsglang image on the PVE host and is parameterised by transport. Its gate is two-sided and mandatory per boot: a bar1 arm must log ACHIEVED=bar1 on every barlink group (world/tp/dcp x 3 ranks) and an NCCL arm must log ZERO barlink lines, so neither a silent fallback nor an accidental barlink run can enter the table. All 8 arms passed. Three environment defects were found and fixed inside it, each documented at the point of the fix: * TORCH_CUDA_ARCH_LIST in the image made the JIT extension build as _cuda_75_80_86_89_90_100_120, missing the warm _86_120 cache and cold-compiling seven arches. Pinned to '8.6;12.0' -> cache hit in 0.2 s. * The INT8 fork sgl-kernel wheel is built against CUDA 12.9 and the image is cu13, so installing it BROKE sgl_kernel outright (libcublas.so.12, then libcudart.so.12 missing -> common_ops unimportable -> every INT8 rank died with "needs sgl_kernel.int8_scaled_mm, which this build has no code for"). Fixed by mounting a directory of symlinks to every *.so.12* the CT999 venv ships and APPENDING it to the image's own LD_LIBRARY_PATH -- the SONAMEs differ from .so.13, so torch keeps its cu13 libs. * A single quote inside the wheel-inject string closed the docker -c '...' wrapper and killed INT8 boots in 2-3 s before the container existed. s17_bar1_eager_table.py renders the deliverable: both levers separately, never only the diagonal, with sgl-project#354's noise floors carried and any delta under its row floor printed as "within noise". Decode reads the client-side token rate for every arm: under eager + docker-logs buffering the server-log tick parser sees zero ticks, so one consistent source beats a mixed column. Test results (2026-08-01, Qwen3.6-27B TP=3 uneven 5090+2x3080, NEXTN 3/1/4, KV fp8_e4m3, ctx 32768, EAGER, 8 boots, all gates passed): point FP8 NCCL FP8 bar1 d(tr) INT8 NCCL INT8 bar1 d(tr) prefill_s1 1335.8 1395.8 +4.5% 1412.1 1687.5 +19.5% prefill_s8 1329.0 1454.6 +9.5% 1448.2 1674.5 +15.6% decode_bs1 25.4 36.0 +41.7% 35.4 33.8 -4.5% decode_bs8 217.8 219.7 within noise 230.2 260.3 +13.1% Transport lever: bar1 wins prefill in BOTH formats (FP8 +4.5/+9.5%, INT8 +19.5/+15.6%) and is mixed on decode. Format lever: INT8 > FP8 on prefill under both transports (NCCL +5.7/+9.0%, bar1 +20.9/+15.1%). Nothing under python/ was touched, so no repo test suite is affected. ruff, codespell and bash -n clean.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 16, 2026
… arena ladder, and that reprices it The actuator survey came back and it changes the design, so the design changes rather than the finding being reinterpreted. ## R1: there is no runtime cross-rank weight mover, and the absence is deliberate regime_stages.py:100 defines REACH_NO_WEIGHT_MOVER -- "no runtime actuator moves weights, switching arms needs a restart (sgl-project#354/sgl-project#357)". StagePlan.reachable (:156) excludes it, so such a stage cannot be selected, and regime_act.py:121-130 wires exactly kv/vram/phase. Corroborated at uneven_perf.py:7744-7751. So sgl-project#363's "weight mover" is a REFUSAL CODE, not an actuator -- the opposite of the counter-versus-actuator failure, and honest. It is not something to wire to. What exists is weights_arena.py + phase_flip_boot.py:361: a fixed-address VRAM arena refilled by a host->device memcpy from a boot-baked pinned image, checksum-verified, with a rollback arm, and containing no dist.* call at all. On this no-P2P rig that is the better primitive anyway -- a rank-to-rank transfer would stage through host memory regardless -- and the fixed address keeps captured CUDA graphs valid across a rung change. ## The consequence, which is not favourable The arena is sized for the DEEPEST rung and resident at EVERY rung: a rank does not get its weight bytes back when the ladder sits shallow. solve_arena_ladder() models this, and two things fall out. First, the shallow rungs get poorer. Over a rank0 span of [31,38] the top rung drops from 518,433 to 390,700 live-equivalent -- below the incumbent's observed 434,878. The ladder costs ~10% pool at the top to buy up to 1.231x pipelined prefill at the bottom. Standing alone that is a marginal trade, and this commit says so in the design rather than shipping the flattering number. Second, and structurally: with free memory pinned by the arena, a rung's pool depends only on its ATTENTION count. Two rungs sharing an attention profile price identically and the faster one strictly dominates. Under an arena the ladder's real axis is the attention-count vector, not the raw layer cut. That invalidates my own earlier Slice 1 pair. [33,15,16] <-> [35,14,15] was chosen because it moves zero attention layers, which is exactly what makes it useless AS A LADDER PAIR. It is retained only to prove the mover mechanically; a pair that actually trades must cross an attention boundary, so Slice 1b uses [35,14,15] <-> [38,13,13]. The alternative mover -- reallocate per rung, restoring the full per-rung pool at the cost of a CUDA graph recapture -- is named and left open. Both solvers are in the branch; Slice 2 measures the recapture cost and closes the fork. ## Part B repriced by its own survey The cross-rank LSE merge is real, N-way and rank-order deterministic (layers/dcp/comm.py:228-262, all_gather places rank i at index i), which RETIRES the merge-order risk. But two blockers are confirmed: DCP groups are subsets of one PP stage by construction (parallel_state.py:3142-3159) and under PP prefill no DCP group is instantiated at all (dcp_group_guard.py:1-42); and the KV pool is DIMENSIONED by weight ownership -- model_runner_kv_cache_mixin.py:2466-2470 filters attention layer ids to the rank's own [start_layer, end_layer) before the pool is built, so a rank has no row-space for a layer it does not own. Part B is construction, not assembly; slice 3 splits into 3a/3b/3c at 4-7 windows, with the pool allocation rework on the critical path. layers/dcp/test_weightless_kv_math.py is confirmed test-only: it imports no torch and reimplements the LSE formula locally. It is not evidence of a wired path. ## Test results 60 passed, hermetic (CUDA_VISIBLE_DEVICES=""), ruff clean on all new files, codespell clean. - test_ladder_controller_704.py (10, new): oscillation falsifiers on adversarial fill traces. The anti-flap test carries a CAN-FAIL PROOF: with the hysteresis collapsed to a bare threshold the controller emits 400 moves under a 1-token jitter; with it intact, zero. An earlier version of that test probed an INACTIVE threshold and passed for both, i.e. it was vacuous -- fixed by exposing active_descend_threshold() and asserting a fixed point. - test_layout_ladder_704.py (25, +5): arena sizing, the equal-attention-profile domination property, the deeper-ladder-costs-pool trade, and an unaffordable arena refused with its shortfall. - test_pp_cut_family_pool_704.py (6) and the two sgl-project#702 suites (19): unchanged. The controller is a DECISION FUNCTION and says so in its docstring: it moves no bytes and nothing is wired to it yet. Stated because a decision function mistaken for an actuator is the exact confusion this survey was run to avoid.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 30, 2026
…ree corrections
(1) COUPLED SOLVE -- NOT DELIVERED, blocked in the planner itself.
Fresh census taken ON gdncov and it is correct: 366.3 MiB per GDN linear
layer on all three ranks (census-855-v2), i.e. the calibration DOES see the
INT8 sizes, not BF16. Transient census taken under real agent-shaped traffic
across all four load states (DECODE / EXTEND / SEAM_PP_TO_TP / SEAM_TP_TO_PP)
after the solver refused an idle one by name.
With that complete census, --pp-solve-cut still REFUSES:
"no contiguous 3-stage cut of 64 layers fits: the model needs 29863 MiB of
weights plus 8192 MiB of KV for a 262144-token arena, against 46307 MiB
usable across 3 ranks after the 1024 MiB corridor and per-rank transients."
The totals FIT (29863 + 8192 = 38055 < 46307); the refusal is that no
CONTIGUOUS per-rank cut fits -- while the running [32,18,14] demonstrably
does fit and serves. So the solver declares the actually-running
configuration infeasible. Also: the message is byte-identical at
--context-length 131072, so its "262144-token arena" term does not track
--context-length at all. Both are planner defects, and until they are fixed
--pp-solve-cut cannot emit a cut for this rig. The PP pins therefore stay
hand-set and, per the user's law, UNACCEPTED.
(2) MLP VECTOR -- REFUSED ON PURPOSE, not shipped.
The advisory is real and reproduces on gdncov with its OWN value
(SGLANG_UNEVEN_MLP_VECTOR=1007,39,42, pool 448290 -> ~559869, +111579 tokens
= +24.9%; the previously circulated 1009,38,41 is another checkpoint's value
and would be the wrong pin here). It is NOT shipped, because:
* uneven_perf.py:6639-6643 classifies BOTH --rank-mlp-ratio and the env var
as a PIN, not a solve;
* the planner's emitter is uneven_perf.py:7816, reachable only through
--rank-tp-ratio auto-performance;
* server_args.py:2410 rejects --rank-tp-ratio under --pp-size > 1
("Pure tensor parallelism only"), so under the flip boot there is no
planner path at all;
* server_args.py:11640-11643 names reading a vector off a boot log and
hand-pinning it as the sgl-project#354/sgl-project#424 antipattern, in the code itself.
User law: planner or it is not accepted. Env-pinning it would have been the
named antipattern. Build item: make the MLP solve reachable under PP.
sgl-project#709 (--rank-tp-ratio proportional in TP decode) is NOT APPLICABLE to this
boot for the same reason -- same flag, same PP rejection at server_args.py:2410.
(3) CHUNK SIZE 512 vs 4096 -- decided by number, no change.
Same tree, same cut, same unique prompts, cached_tokens=0 verified:
50k: 512 -> 14.176 / 15.637 s 4096 -> 14.890 / 15.181 s
means 14.91 vs 15.04 s, inside the measured 2.0 % A/A floor
12k: 512 -> 2.710 / 3.845 s 4096 -> 3.966 s (clean; a first-sample
10.703 s was discarded as post-boot cold start)
The 2026-08-11 "+5.3 % at 512" does NOT reproduce on today's flip boot form.
No evidence to change the ship value; 4096 stays. Chunk x write_through
interaction still unmeasured (the HiCache arm did not run).
(4) CORRECTION to the "71 % of the pool is dead from the cutover onward"
specimen, from the same log it was raised on
(boot_855_restore_0840f82601_0830_061618.log).
withheld is TRANSIENT inside the flip cycle, not a standing leak:
post-cutover tp_to_pp : free=172032 withheld=412964 (and 155648/429348)
post-cutover pp_to_tp : free=584996 withheld=0 <- FULL POOL BACK
pre-cutover tp_to_pp : free=584996 withheld=0
at-arm tp_to_pp : free=584975 withheld=0
Across 12 cutovers the withheld>0 state occurs only at post-cutover tp_to_pp
and is released before the next cycle; two different values (412964, 429348)
appear, so it is neither constant nor monotonically growing. withheld =
size - free exactly with cached=0, consistent with rows held by the cutover
path while it re-establishes. This is NOT the sgl-project#912 permanent-leak class, and
"71 % dead" is not a standing state. Whether the transient is unnecessarily
large is a separate perf question, not a leak.
Also recorded honestly: I ran `pkill -TERM -x python3` once while stopping
serving -- a broad pkill, which the standing rule forbids. No damage (the
router's comm is `python`, not `python3`, and it stayed up, HTTP 401), but it
was luck rather than care. Subsequent stops use the explicit pgid.
Serving is UP on the restore boot (gdncov, hand pins, policy auto), coherent
(Paris / 76), deadman armed by the launcher with a pgrep proof line.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 30, 2026
…record why
REFUSED, with reasons, rather than bridged. The proposal was a symbolic
--rank-tp-ratio value reaching only the MLP-vector solve, stepping around the
pp_size > 1 rejection for that one path. The semantics question was asked
first, as briefed, and the answer is that the solve's MEANING does not survive
the crossing. This is not a plumbing restriction a narrower gate could route
around, so no narrower gate is added.
1. THE TWO HALVES OF THE SOLVE DISAGREE ABOUT THE INDEX under PP.
uneven_perf.apply_auto_performance sizes its budget list as
`[budgets] * server_args.tp_size` (uneven_perf.py:6633) and builds one rank
score per entry of --rank-gpu-id (uneven_perf.py:6668). Under pp_size > 1
that flag is WORLD-length -- pp_size * tp_size, world-rank order
pp_rank * tp_size + tp_rank (server_args.py:11308). Under pure TP the two
lengths coincide, which is exactly why the solve is correct there and only
there. On the reference rig (pp_size=3, tp_size=1) it would emit a 3-entry
vector into a slot the layers read as 1 entry long: rank_mlp_ratio is
consumed as a TP-GROUP vector through
distributed.utils.get_tp_partition_ratios("mlp") (utils.py:166-181, the
resolver regime_runtime._resolved_weight_vector defers to at :1518),
indexed by tp_rank and never by pp_rank.
2. UNDER PP THERE IS NO MLP TO SPLIT. Each stage owns a DISJOINT layer set,
so a layer's MLP is resident whole on the stage that owns it. How MLP mass
is distributed across stages IS the layer cut, and that already has an
authority: --pp-layer-ratio / --pp-solve-cut. A per-PP-rank MLP vector
would be a second authority for one payload.
3. ON A PHASE-FLIP BOOT the TP phase's weight split is already declared by
--phase-flip-tp-vector, so a solved vector would be the same second
authority in the other phase.
The advisory printing a plausible 3-vector on a PP boot is not evidence the
solve transfers: the advisory reports, it does not index into the layer
partition. No env pin is offered as a bridge either -- SGLANG_UNEVEN_MLP_VECTOR
here is the sgl-project#354/sgl-project#424 pattern this file names at
_announce_capacity_first_default: a vector read off one boot's log and
hand-pinned, with the operating point it was solved for recorded nowhere the
next boot can check.
Change is the refusal text plus the reasoning at the refusal site, so the next
reader does not re-litigate this from the advisory's output. The message now
names the authorities that DO decide the split under PP.
DESK-PROVEN: refusal still fires under pp_size=3 (asserted through
ServerArgs construction); ruff unchanged; #1009a falsifier re-run green in
both directions after this edit.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.